‘I hope I’m wrong’: Why some experts see doom in AI

One of the things that caused Geoffrey Hinton to completely change some of his core beliefs about artificial intelligence was a joke. Well, many jokes. Last year, before the emeritus University of Toronto professor announced his retirement from Google in May and started airing his concerns about the technology that has been his life’s work, he was experimenting with a Google large language model called PaLM.

Prof. Hinton, 75, has thought of the ability to understand humour as a benchmark for AI models that produce and parse text. He’s not sure why, other than a sense that grasping humour must be an indicator of something. So he peppered PaLM with quips (one was about whales), pushing for explanations. The model, he said, got it.

He hasn’t stopped testing chatbots. Ever since he raised the spectre that superintelligent AI could subjugate humanity, he has been swamped with media requests, including some from Fox News, which he does not care for. “Fox News is an oxymoron,” he wrote back to one e-mail from a morning show.

Open this photo in gallery:

He decided to put that quip into GPT-4, the large language model developed by San Francisco-based OpenAI that powers ChatGPT, except he added a space between “oxy” and “moron.” Prof. Hinton asked the model to explain the joke, and again, it did. Then he asked about the space, clarifying it was not a typo. “It says something like, ‘Ah, that’s an extra layer of humour, because oxy is an abbreviation for Oxycontin, and so it implies that Fox News is a drug,’” he told me on a Zoom call from his Toronto home.

To him, this is one sign that language models have leapfrogged over past iterations and are more capable than many people assume. Other things happened, too. Prof. Hinton had been thinking about how well AI models absorb information.

The human brain has around 100 trillion connections between billions of neurons, he explained. GPT-4 has the equivalent of just one trillion connections, by his understanding. “But it knows probably hundreds and maybe thousands of times more than any one of us, which implies it is getting knowledge into the connections much more efficiently,” he said. Also, copies of the same AI models could run on different machines and learn independently, but share that knowledge instantly.

Prof. Hinton used to believe that an AI system that transcends human intelligence was possible in 30 to 100 years. But owing to these recent insights, he now puts it at between five and 20 years – and that is a worrying development. “You have to ask the question: How many cases do I know where something more intelligent is controlled by something less intelligent?” he said.

The peril and promise of artificial intelligence

In his view, intelligence wins, and will seek to control. He’s not willing to game out how exactly AI could threaten our existence, though. “It’s such unknown territory and there are so many ways it could happen that I don’t like to speculate on the details,” he said.

But Prof. Hinton has raised the possibility repeatedly in recent weeks. “It’s quite conceivable that humanity is just a passing phase in the evolution of intelligence,” he said at a conference in May. And he’s not alone. About 350 AI scientists, engineers, and executives recently signed a statement contending that “mitigating the risk of extinction from AI” should be a global priority.

The notion is not new, but it has more recently been confined to science fiction and heady philosophy debates. Some would like it to push it back there. “All the conversation about the risk of extinction, we can’t have any way to evaluate the probability of that,” said Joelle Pineau, a McGill University computer science professor and vice-president of AI research at Facebook parent Meta Platforms Inc. “Suddenly, we’ve made this so big that we can’t talk about anything else.”

Marc Andreesen, the U.S. venture capitalist, wrote that people worried about existential risks have the hallmarks of an “apocalypse cult” with “extreme beliefs” rooted not in science but something more akin to religious faith.

Prof. Hinton, of course, disagrees. “The reason I went public is that the idea that this is science fiction is a big mistake,” he said. So what is everyone so afraid of?

Open this photo in gallery:

The extinction statement came about in part because of Dan Hendrycks. Or maybe it’s more accurate to say it came about because of a book he read around the time he graduated high school in 2014. Superintelligence: Paths, Dangers, Strategies, written by Swedish philosopher Nick Bostrom and published that same year, argues that massively smart AI is not only possible, but potentially dangerous. A superintelligence could outsmart us, slip from our control or have unintended consequences that we can’t fathom.

For Mr. Hendrycks, who grew up in a Christian fundamentalist household in Missouri, the prospect of this digital deity seemed reasonable enough to address. Earlier this year, after completing a PhD in computer science, he co-founded the Center for AI Safety in California with the goal of reducing “societal-scale risks.” Mr. Hendrycks noticed a lot of his peers were worried that AI could cause harm to humanity, but they weren’t open about it.

Jeff Clune, an associate computer science professor at the University of British Columbia who previously worked at OpenAI and Google DeepMind, said he’s been concerned about how AI could threaten our existence for more than a decade, but feared getting laughed out of the room. “It was almost dangerous from a career perspective to admit you were worried,” he said.

Will AI take over the world? And other questions Canadians are asking Google about the technology

It was only later, when computer scientist Stuart Russell talked about the long-term dangers of AI, that Prof. Clune opened up. “I remember how liberating and great it felt,” he said, “because I could point to somebody who is extremely established.”

Mr. Hendrycks set about organizing the statement to show the public that “this is where many people who have been building these systems over the past decades are now at,” he said. Among the first experts he contacted were two godfathers of AI, Prof. Hinton and Yoshua Bengio, the scientific director of the Mila artificial intelligence institute in Montreal. “If they weren’t going to sign on to it, then that would have axed it,” Mr. Hendrycks said.

Both men are about as high-profile as AI scientists can get, having pushed forward the concept of artificial neural networks – AI systems that learn to complete tasks, such as identifying objects in pictures, by ingesting massive amounts of data – which form the basis of today’s technologies. Mr. Hendrycks said that some people squirmed at the word “extinction,” but he wanted a clarifying sentence acknowledging the odds of this most extreme scenario were, at very least, not zero.

I spoke to more than a dozen people who endorsed the statement, and some added a little clarification. “‘Extinction’ is perhaps too dramatic,” said Gillian Hadfield, a law professor at U of T. “But I do believe that we are courting societal scale risks of economic or social collapse if we don’t get regulatory infrastructure in place.”

Eric Horvitz, chief scientific officer at Microsoft Corp., told me that, “I’m not happy with the way the statement was worded, but I signed as I believe we need to continue to invest in efforts and intellect to minimize potential risks.”

Foutse Khomh, a Polytechnique Montreal computer engineering professor, seemed torn about deflecting focus from nearer term AI risks if he signed the statement, but went ahead with it. Florian Shkurti, an associate professor of computer science at U of T, declined an interview. “I’m not the most avid supporter of this statement,” he said.

Many experts have contemplated the issues for years, though. In 2009, Mr. Horvitz co-led a panel that studied the implications of AI on society, including some of the more fanciful concepts from science fiction and futurist thinkers such as the “singularity,” meaning the hypothetical point at which we lose control of technology. The panelists were skeptical of these “radical views,” but deemed them worthy of more research.

Open this photo in gallery:

Sheila McIlraith, now a computer science professor at the U of T, participated in that panel, and said that the recent pace of development and the dramatic improvements in generative AI prompted her to think more seriously about catastrophic outcomes. “We really need to look at these longer-term risks now,” she said. Once more powerful AI systems become widely integrated into various aspects of our lives, it could be too late to mitigate the risks. “In some respects, the genie is already out of the bottle.”

Some academics are worried enough to change their entire research focus. David Duvenaud, an associate computer science professor at U of T, used to work on building deep learning algorithms. A few years ago, he came to believe that AI systems of human-level intelligence (or smarter) were getting closer, an insight that coincided with him securing tenure.

Opinion: AI may be coming for your job, but all is not lost

“That’s when I pulled the trigger,” he said. “I’m not going to start new projects under my old research agenda, even though I love it, and try to find projects that will somehow help our civilization deal with this big change.”

As it turns out, most people I spoke to were not immediately concerned with murderous superintelligent AI (one even called it a “red herring”) and reframed “extinction” as a catch-all for all manner of dystopian scenarios that don’t necessarily result in wiping humanity off the Earth.

More likely is that bad actors will intentionally deploy AI for nefarious means: AI models trained on virology studies coaxed to engineer new pathogens; systems that can pump out malware or hack into critical infrastructure; and disinformation campaigns to stoke violent outrage. (As always, the problem is us, not technology.)

Where the debate starts to get abstract is if you accept that a superintelligence is feasible. Maybe it stays a futurist fantasy – if there’s one field where expectations often exceed reality, it’s technology. But if you believe it, as some experts do, then anything is possible.

Algorithms don’t always work as intended. In 2020, Prof. Clune and some colleagues published a paper featuring examples of how algorithms can misbehave and defy expectations in research settings. A Tic Tac Toe program learned to make erratic moves that disabled its opponent and forced it to crash, thus winning the game. In another example, digital organisms evolved to climb over walls constructed to contain them.

Separately, researchers at OpenAI tasked an algorithm with playing a boat-racing game. The AI figured out it could win by driving in circles and knocking over targets to earn points despite “repeatedly catching on fire, crashing into other boats and going the wrong way.”

For some, these kinds of bizarre results become all the more concerning as companies develop increasingly powerful AI models and deploy them in consequential settings – financial systems, military applications and critical infrastructure. “It is the rule, not the exception, that any noble thing we try to do with it, AI will probably find loopholes and exceptions and unanticipated ways to accomplish the goal that are not what we want,” Prof. Clune said.

Broadly, this is what some researchers call rogue AI. To illustrate this point, some AI researchers use childlike parables that are maddeningly short on details. Prof. Hinton gave me the example of AI tasked with helping humanity. “It might decide the big problem with humanity is climate change, and the best way to help humanity is to get rid of 90 per cent of them so they don’t ruin the Earth,” he said.

Such thought exercises, however insufficient, have helped give rise to an entire subfield of AI research called alignment, which seeks to create technical solutions to steer AI models to follow our values, reducing unintended consequences.

It has led to some weird places: Can an AI system prevent itself from being switched off? The question has nothing to do with sentience. Stuart Russell at the University of California has argued that a rational AI system may conclude it cannot complete its programmed objective if someone turns it off, and as a result, disable the switch. He speculated that giving AI systems “an appropriate level of uncertainty about their objectives” could lead to safer outcomes.

Some researchers are not convinced, though, since language opens up the possibility of a superintelligent AI convincing humans to do its bidding, such as ensuring nobody flips the off switch. “As long as it can talk, as long as it can output text, you’re screwed because it can manipulate people,” Prof. Hinton said.

Prof. Clune, for one, sees all kinds of dystopian possibilities as a result, such as an AI model that could financially blackmail people into not switching it off, or politicians into not passing legislation that could reign in AI. He can foresee cults and political movements arising around AI, convincing people to make copies of itself, amassing more power. And then there are possibilities we might not even be considering. “Chimpanzees probably never anticipated all the ways we would subjugate the planet,” he said. “We’re literally talking about superintelligence. It probably can figure out all sorts of ways to take power from us.”

But even assuming superintelligence is possible, why would AI system endeavour to take power at all? One theory posits that if AI is given a goal to complete, the system could determine it first needs to amass more power and resources in order to finish the job.

Open this photo in gallery:

Prof. Bengio in Montreal recently wrote a lengthy article on his website about how rogue AI models can emerge, much of which has to do with competitive pressures that incentivize developers to neglect safety, incorporate more autonomy and remove humans from decision-making roles.

“As the stakes become higher, the ethical lines might become thinner,” he told me. Unless we see serious breakthroughs in alignment efforts, Prof. Bengio says the safety of more powerful AI models in the future cannot be guaranteed, which is why he’s been vocal in recent months about regulation.

Even then, powerful AI systems could prove damaging without going rogue. “The scenarios that are the most plausible and most inevitable to me are just about humans being pushed out of our own civilization, gradually over time, as we become economically redundant,” said Prof. Duvenaud at U of T.

Corporations, driven by competitive pressures, replace larger and larger portions of their workforces with AI, and whole swaths of the population become marginalized. AI doesn’t necessarily need to be better than humans at all tasks, just reliable and cheaper.

“If we want to stop this, we have to stop competition and move to a more planned economy, which, every time it’s been tried in human history, has been a complete disaster,” he said.

Prof. Bengio has also thought about how we might need to reorder society to prepare for all that AI could bring. That includes better education and critical thinking skills that would make people less susceptible to disinformation campaigns, improved mental health services so that individuals are less motivated by fear and hate, and ways to reduce geopolitical conflict given military adoption of AI.

“Is there a vision of a society where AI is useful to us, and somehow we have managed to control the risks?” he said. “The way we are currently organized, in the long term, is not safe.”

Open this photo in gallery:

Whenever Katrina Ingram gives a presentation about AI ethics, she’ll mention existential risks, but doesn’t spend time there. “I don’t think it’s beneficial in helping us to grapple with the ways in which AI is causing harms today,” said Ms. Ingram, who runs an Edmonton-based consultancy called Ethically Aligned AI.

There can be consequences when we put too much attention hypothetical risks, too. “People who are doing really good work on addressing the real and present risks and dangers aren’t necessarily being supported, either through funding or even the message that gets out,” she said.

Ms. Ingram is more concerned about how companies such as OpenAI have collected data – scraped from the internet, without consent – and biases that may lurk there and reinforce negative outcomes. “What about people who are being impacted right now, because a system decided they shouldn’t get a loan, or access to housing?” she said.

But could there be common ground between people who are worried about the present and those worried about the future? Whether you’re concerned about biased decision-making or rogue AI, pushing companies to develop models that are transparent, explainable and work as intended seems like a reasonable goal. Prof. Clune, for example, recently published a paper that outlines a method to monitor an AI system’s thought process (or the equivalent of it, anyway) and intervene before something goes awry.

I put that notion to Nick Frosst, a co-founder of Cohere in Toronto, which develops large language models and competes with OpenAI. “The jump you just made is the jump I want to push back on,” he said. “We have models that take in the beginning of text and write more text. To personify it and say, ‘Well, what if it discovers it has an off switch and then seeks to do something?’ Then we’re just in the world of sci-fi.”

Open this photo in gallery:

Mr. Frosst didn’t sign the “extinction” letter; instead, his co-founder wrote an opinion piece arguing those claims were misguided and risk derailing promising AI developments. “I just really want to make sure we’re not convincing the populace that, technologically, we’re at a place where we’re not,” Mr. Frosst said.

Part of what’s happened, he believes, is that large language models such as GPT-4 have performed far better than a lot of people anticipated, even researchers who helped design the foundations, stoking misplaced concerns. “Where I’ve seen the public discourse go is into a lot of fearmongering and exaggeration that is misleading the public,” he said.

Mr. Frosst is a Geoffrey Hinton protégé, and they worked together at Google. Prof. Hinton was among the first investors in Cohere, and they remain close friends who occasionally face-off in chess. (Prof. Hinton wins most of the time.) Though the two disagree on what’s at stake when it comes to AI, Mr. Frosst does not begrudge his mentor for speaking up. “I learned pretty much everything I know about research from him,” he said. “He’s exactly the type of mind that I’m glad is thinking about the long-term future.”

Before the world really knew what a computer was, an American mathematician named Norbert Wiener published a book called Cybernetics in 1948. He mapped out the similarities between animal nervous systems and electronic machines, laying the groundwork for a new wave of automation.

Cybernetics proved an unlikely hit, and Prof. Wiener became a public intellectual. But he was worried. He warned that “every degree of independence we give the machine is a degree of possible defiance of our wishes.” Humanity had to control its creations, and unchecked automation could wipe out jobs, necessitating a seismic policy response. “We can no longer fear the word ‘socialism,’” he said in 1950.

Roughly a decade later, experts were anticipating devices that could hold all the knowledge in the world in memory, and a machine-learning checkers program had been created. Prof. Wiener, though, wrote about a looming “catastrophe.” An electronic mind could think much faster than us and “we may not know, until too late, when to turn it off.” What if a fully autonomous bottle factory were programmed to maximize production? The owner could be bankrupted by an “enormous inventory of unsalable bottles” because he trusted the machine. “Disastrous results are to be expected not merely in the world of fairy tales,” Prof. Wiener wrote. When he died in 1964, a headline in The Globe and Mail called him the “father of automation.”

Today, critics might call Prof. Wiener a worrywart whose fears never came true. Or maybe he was prescient. The parallels to the present are striking, though. Whereas Prof. Wiener’s concerns deepened with advent of computer memory and checkers, today’s godfathers of AI are sounding the alarm as the technology takes on language. Another step change in computing has arrived, and no one knows what to make of it.

It’s also hard not to wonder whether people such as Prof. Hinton and Prof. Bengio are succumbing to the inverse of AI hype. Rather than overestimating the capabilities of a technology to tout its glories, they’re extrapolating from today’s AI to imagine a potential apocalypse. Hype and doom are two sides of the same coin, in a way.

Open this photo in gallery:

Joelle Pineau of Meta suggested some of the loudest voices may not fully appreciate the wide volume of scholarly work into making AI systems safe and reliable. “Geoffrey is someone who’s very knowledgeable about AI and machine learning, but he is not necessarily one who is an expert in terms of safety of AI systems,” she said.

I couldn’t help thinking about the jokes Prof. Hinton had put to chatbots. Was he fooled?

The prevailing view of large language models comes from a 2021 paper written by a group of researchers including Timnit Gebru, who previously co-headed Google’s ethical AI team. The paper argues that language models are “stochastic parrots” that possess no understanding of the language they spit out. These algorithms look at a group of words and predict the next one based on the vast quantities of data that has been pumped into them. To infer anything more is a mistake.

When I asked Prof. Hinton about this, he appeared to start bouncing on his feet at his standing desk, girding for a challenge. He launched into a rebuttal that began with a joke. “I had a pet parrot, and I rather object to the kind of people denigrating parrots,” he said. Language models, he continued, can do more than finish sentences; GPT-4 has solved simple logic puzzles Prof. Hinton has given it. “I don’t see how it can do that if it’s a stochastic parrot,” he said.

The inner workings are more sophisticated than the term implies, too. Prof. Hinton explained that language models convert words into numbers, with each digit representing a clue of sorts as to the word’s meaning, and compares those figures to surrounding strings of digits to hone in on context and derive meaning, layer by layer, to arrive at the next word.

“It is understanding,” Prof. Hinton said. “That’s what understanding data means, and it’s what we do, too. It’s what our brains do.”

He has long looked at things differently. He went to a Christian school, and despite the insistence of his instructors, never believed God existed. He pursued artificial neural networks even though other researchers had long ago abandoned the concept and considered it a waste of time – until he helped prove that it wasn’t.

His fears today about the direction of AI could make him an outsider again, but if so, he’s not relishing his position. “I hope I’m wrong,” he said. “I really hope I’m wrong.”