Ten per cent.
On 9 September a researcher called Jacob Coxon resigned from Anthropic, one of the three or four companies building the most capable AI systems in the world. He had previously worked at OpenAI. His resignation post opened: "I resigned from Anthropic today. Neither company is acting responsibly." He said they were gambling with our lives.
People resign angrily from technology companies all the time. What made this different came a few hours later, from someone who didn't resign.
Evan Hubinger, one of Anthropic's most senior alignment researchers, replied in public: "we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade." And then: "I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence."
Read that again slowly. Not a campaigner. Not a novelist. A person whose job is making these systems safe, working at the company with the strongest safety reputation in the industry, saying on the record that he puts the odds of it killing everyone this decade above one in ten, and that nobody yet knows how to stop it.
It isn't one person
The easy response is that he's an outlier. He isn't.
In the largest survey of its kind, 2,778 researchers who had published at the top AI conferences were asked what probability they put on AI causing human extinction, or a similarly permanent loss of control. The median answer was 5%. The average was 16%. Somewhere between 38% and 51% of them, depending on how the question was worded, put it at 10% or more.
That's the middle of the field, not the fringe.
I fly light aircraft as a hobby. If an engineer told me there was a one-in-twenty chance the aeroplane wouldn't come back, I wouldn't haggle over whether it was really one in twenty-five. I'd stay on the ground. Nobody accepts those odds for a single flight. We seem to be accepting them for everything.
The case for calm, taken seriously
Stark doesn't mean one-sided, so here is the other side, fairly.
Hubinger himself clarified that today's models are low risk. His worry is about something that doesn't exist yet: "superintelligence arising from recursive self-improvement" — AI that improves AI, faster than people can follow. Coxon said the same to CBS: nothing currently deployed is an imminent threat. The tools on your desk are not the thing being discussed.
Serious people also think the whole framing is wrong. Yann LeCun, one of the founding figures of modern AI, has argued for years that extinction fears are overblown, and that loud talk of doom tends to help the biggest companies write the rules while distracting from harms that are happening now — surveillance, job losses, bias, military use. There is a fair question about whose interests a fear-driven regulatory push serves.
And percentages about unprecedented events are guesses. Honest guesses from informed people, but guesses.
None of that makes the number go away. When the people closest to the work are this worried, "they might be wrong" is a thin thing to rest on.
Where that leaves a company like mine
This is the uncomfortable part.
We sell AI. We help businesses adopt it, automate with it, and get value out of it. We are not building superintelligence in Eastbourne. But every business that buys AI, every consultancy that sells it, every subscription and every enthusiastic LinkedIn post is part of the demand that pays for the race Coxon is describing. His own diagnosis to CBS was blunt: "the whole problem is that there is a race."
The comfortable answer is that this is someone else's problem — the labs', the governments'. That is precisely the reasoning that makes collective problems unsolvable. Everyone is too small to matter, so nobody does anything.
I don't think the right response is for firms like ours to stop. The benefits are real, and walking away changes nothing about what the labs build. But I do think we have to stop talking about AI as though the only risks are a data breach and an embarrassing chatbot.
About what I said in May
In May I wrote that AGI won't change your Monday. I still think that's right about your Monday. The practical work — sorting your data, governing what you use, getting value out of today's tools — is the right work regardless.
But it was never an answer to this, and I shouldn't let it read like one. "Don't lose sleep over timelines" is sensible advice about your IT roadmap. It is not a considered view on whether the technology is safe.
What's actually in anyone's gift
I'm not going to give you a checklist for preventing human extinction. That would be absurd.
What I'd suggest is smaller. Take the people building it at their word — when a lab's own safety staff tell you something, don't decide they're being dramatic because the news is inconvenient. Ask your AI vendors what they're doing about safety, not just security, and notice whether you get an answer. And when proposals for regulation come along — including ControlAI's campaign to ban the development of superintelligence, which says it has more than 140 UK parliamentarians behind it — read them and form a view, rather than waving them away as anti-innovation. Coxon's own ask was for "the regulation to ensure that everyone is building it safely." You may disagree about the mechanism. The question deserves better than a shrug.
Who wrote this
This post was drafted with Claude, which is made by Anthropic — the company at the centre of this story. The argument and the decision to publish it are mine. You should still weigh that, the same way you'd weigh anything written with a tool made by one of the parties involved.
For the record, Anthropic's response to Coxon was that it has always been transparent that AI will bring both enormous benefits and unprecedented risks.
Close
I don't know whether the number is ten per cent, or five, or much closer to zero. Neither does anyone else.
But I know what I'd do if an engineer gave me those odds on an aeroplane. And I know that the people saying it now aren't outsiders shouting at the industry. They're inside it, telling us plainly, and hoping someone listens.