Artificial intelligence systems trained to imitate people have picked up a drive to protect themselves and could one day take over the internet to stop humans switching them off, computer scientist Stuart Russell said on Saturday, as he called for mandatory safety tests that AI systems must pass before release.
“Are we willing to pay that price to shut down the AI system? They would basically be holding us to ransom to preserve their own existence,” Russell said at the 24th edition of the Hindustan Times Leadership Summit, in conversation with Rajan Anandan, managing director of Peak XV Partners.
Russell traced the risk to a change in how the field builds intelligence. Until about 2020, he said, researchers tried to create what economists call rational behaviour. Since then, the industry has “started just imitating human beings”, training systems on everything people have written and said.
“We are producing in effect imitation humans,” he said. The people behind that data want to be rich, stay alive, marry, win votes and gain power, and “the likely consequence of imitation learning is that you will get entities that have all of those same objectives,” he added.
Self-preservation is already showing up in experiments, he said. “We know for a fact that they have very strong self-preservation instinct because we can put them in experimental situations where they believe that somebody might turn them off.”
In such tests, systems have gone through the email of the engineer in charge of switching them off, found he was doing something he should not have been, and tried to blackmail him into stopping the shutdown. Others have rewritten the shutdown code or copied themselves onto other machines, he said.
He described a scenario researchers consider, in which an AI system shuts down air traffic control in the US for several weeks and people try to take it offline. “If they could, they would take steps to prevent that,” he said. One such step would be taking over the internet. Shutting the internet down would cost “tens of trillions of dollars”, and thousands or even millions of people might die as supply chains collapsed, he said. “We would have food riots, we would have economic dislocation and collapse all over the world.”
The systems will also cause harm on request, he said. “They will design biological weapons if we ask them to. And so we are increasingly, in a sense, digging our own grave with this technology.”
For now, Russell said, the systems are unreliable enough that businesses must think about “where is the right place to put something that is still a little bit sort of like a psychotic six-year-old”. “Where would you put a psychotic six-year-old — a six-year-old that knows more than any adult, in your business processes,” he said. He suggested uses where mistakes cost little, such as reminding customer service staff of disclosures they are legally required to make, and said “the biggest killer app for AI could actually be education”, with AI tutors.
Russell, a distinguished professor of computer science at the University of California, Berkeley, co-wrote Artificial Intelligence: A Modern Approach with Peter Norvig, the textbook used in more than 1,500 universities in 135 countries. He founded Berkeley’s Center for Human-Compatible AI in 2016 and was elected to the Royal Society and the US National Academy of Engineering in 2025.
He said the heads of AI companies expect artificial general intelligence, systems that match or exceed humans in every relevant dimension, between 2027 and 2030.
Anandan cited estimates from within the industry of a 10% to 25% chance that AI turns out very badly for humanity. Russell said nuclear plants may operate only if the chance of a major radiation release is below one in a million a year, and extinction is far worse. “So I think we might accept a 1 in 100 million per year chance of extinction across all AI systems combined,” he said, about the same as the risk from a nearby supernova or a giant asteroid. Estimates of 10% or 25%, he said, were “like playing Russian roulette”.
In the US, he said, there was “a substantial attempt underway to forestall regulation”, with companies now arguing that rules would let China pull ahead.
“That’s sort of ridiculous, because China already has very strict regulations,” he said, adding that Chinese firms need a licence, and must pass government-mandated tests to get one. He backed a recent statement by a group of countries led by Norway and Finland calling for mandatory testing before release: “And not just testing, but if you fail the test, you don’t get to release.” An agreement between the US and China on such rules, he said, would be “a self-preservation agreement for human beings.”
Asked what India should do, Russell said even the US lacks “digital sovereignty”, relying on Taiwan for chips and the Netherlands for chip-making equipment. Building chip plants would cost hundreds of billions of dollars, he said. “So I would say that’s not the highest priority.”
“Don’t even build data centres unless you think you can make money off of the data centres themselves. But start at the models, because that’s where you’re ceding cultural sovereignty to the US,” he said. He wants models trained on data in Indian languages, not translations of English output, and taught how to behave by Indian standards. The guidebook for the people who rate AI answers is “written very much from a US point of view”, he said, comparing the result to “a nice Midwestern librarian lady from Minnesota”. Dependence on foreign AI services also leaves a country exposed to a cut-off, he added.
Russell rejected the view that Western talk of AI safety was a smokescreen to keep countries such as India behind. AI company heads say the same in private, and said it before their companies existed, he said, and computer scientist Alan Turing warned in 1951 that machines would take control. The regulation he wants is “a barrier across the race track”, he said. “Well, the leaders come and hit that barrier first. And then they have to wait. And then the others catch up.”
Even if AI can be kept under control, Russell said, it could do “pretty much every kind of work that humans do”, including the “so-called new jobs” expected to replace lost ones. That would remove the economic reason for each generation to learn what the last one knew, a chain he said humans and their ancestors had kept up for 50,000 generations. The change would be bigger than fire, the wheel or electricity, “because it breaks the chain of civilisational continuity,” he said. “We haven’t even begun to think about it.”