In the summer of 1942, Edward Teller raised a terrible question. Could an atomic bomb ignite the atmosphere and burn the whole planet? Robert Oppenheimer drove to Michigan to find Arthur Compton, who ran the project’s Chicago laboratory. The two men argued on a lakeshore for hours.
Compton later told the writer Pearl S. Buck where his line was. If the chance that the bomb would ignite the atmosphere was more than about three in a million, he would not go ahead. Better, he said, “to accept the slavery of the Nazis than to run the chance of drawing the final curtain on mankind.” The calculations came in below the line. The project continued.
Eighty-one years later, Dario Amodei, the CEO of Anthropic, gave his own number. The chance that AI goes “quite catastrophically wrong on the scale of human civilization” was “somewhere between 10 and 25%.” Anthropic then went on to build the most capable models it could.
The two numbers measure different things. Compton’s was the chance that one test ends the world. Amodei’s is the chance for AI as a whole, and Anthropic argues that its presence makes that number smaller. That argument is the subject of this post. It is a consequentialist argument: the right act is the one with the best expected result. When the people who make that argument also control world-scale power, history shows a pattern. The calculation always finds a reason to go ahead.
The nuclear spreadsheets
The first generation of nuclear strategists were brilliant, numerate and sincere. Their calculations are still hard to read.
Herman Kahn, RAND, 1960. On Thermonuclear War has a table called “Tragic but Distinguishable Postwar States”. It lists outcomes from 2 million American dead with 1 year of economic recovery to 160 million dead with 100 years of recovery. Kahn’s point was that these outcomes differ, so a planner should choose among them. Stanley Kubrick put the phrase almost word for word into General Turgidson’s mouth in Dr. Strangelove.
John von Neumann, around 1950, on a preventive nuclear strike on the Soviet Union: “If you say why not bomb them tomorrow, I say why not today? If you say today at five o’clock, I say why not one o’clock?” The logic was clean. The Soviets would get the bomb. A war later would be worse than a war now. So strike now.
The Joint Chiefs of Staff, 1961. Daniel Ellsberg drafted a question for President Kennedy: how many people would die if the US carried out its general war plan, SIOP-62? The answer came back as a graph. 325 million dead in the Soviet Union and China in six months, 100 million more in Eastern Europe, 100 million in Western Europe from fallout, and more in neighbouring countries. About 600 million in total, roughly one in five people alive at the time. The plan existed and was ready to execute.
Each of those calculations was also wrong, and wrong in the same direction:
- They left out fire. US war plans counted only blast damage. Lynn Eden showed in Whole World on Fire that mass fires would roughly double the damage, with fire radii 2 to 5 times larger than blast. Because the planners underestimated damage, they built more and larger warheads to reach their targets.
- The physics surprised them. The 1954 Castle Bravo test was predicted to yield 6 megatons. It yielded 15. Fallout fell on the inhabited Rongelap and Utirik atolls and on a Japanese fishing boat, and a fisherman died. Historians still argue about the exact cause of the miss.
- The enemy was a guess. In 1960, Kennedy campaigned on a “missile gap”. Agency estimates had put Soviet ICBMs on launchers at close to 250 by mid-1961. The real number, even counting prototypes, was four.
These people had the best data, the best models and real moral seriousness. Their numbers still came from inside organisations whose budgets, careers and identities depended on the answer. Every estimate of the adversary justified more weapons. Every estimate of damage left out what was too horrible to model.
What actually saved us
The moments that kept the Cold War cold were refusals.
In October 1962, a Soviet submarine near Cuba was under depth charges. Its captain wanted to fire a nuclear torpedo. Vasily Arkhipov, the flotilla’s second-in-command, refused to give the consent the launch required. In September 1983, Stanislav Petrov saw his early-warning screen report five incoming American missiles. His procedure said to report an attack. He reported a malfunction instead.
Neither man ran an expected-value model. Each held a line in the moment when the system in front of him was ready to go.
SBF, times a hundred
In March 2022, Tyler Cowen asked Sam Bankman-Fried about a game. With 51% probability, the world doubles. With 49%, it disappears. Would he play, and keep playing double or nothing? He said he would, and he would keep playing.
Eight months later, FTX collapsed with about $8 billion of customer money missing. Bankman-Fried had applied the same rule to other people’s deposits: if the expected value is positive, the risk to others is one more cost on the spreadsheet.
That same spring, FTX led Anthropic’s Series B with $500 million. Both came out of the same intellectual movement, effective altruism, which taught a generation of very smart people to multiply probabilities by stakes and follow the result.
Anthropic has committed no fraud. The comparison is about the decision rule. On 10 November 2026, Anthropic launched its IPO roadshow at a valuation of up to $1.4 trillion. That is more than 40 times FTX’s peak valuation. The rest of the “times a hundred” is in what is at stake. Bankman-Fried bet customer deposits. Anthropic’s bet, in its CEO’s own words, is on a technology with up to a one-in-four chance of a civilisational catastrophe.
The Anthropic calculation
Anthropic’s founding argument is a calculation. Powerful AI will come anyway, so it is better to have safety-focused labs at the frontier. Look at what that rule has produced so far:
- Gulf money, 2025. Anthropic had declined Gulf state investment on national security grounds. In a leaked Slack message, Amodei explained the reversal: “No bad person should ever benefit from our success” is “a pretty difficult principle to run a business on.” Without that capital, “it is substantially harder to stay on the frontier.”
- The pause, 2026. The original Responsible Scaling Policy promised that Anthropic would pause training if its safety measures could not keep up. In February 2026, version 3.0 removed that pledge. Anthropic now delays only if it believes it has a significant lead over competitors and it sees catastrophic risk. The safety stop now depends on a competitive calculation. Chief science officer Jared Kaplan: “We felt that it wouldn’t actually help anyone for us to stop training AI models.”
- Military superiority. In Machines of Loving Grace, Amodei proposes an “entente strategy”: a coalition of democracies uses AI to get “robust military superiority” and blocks adversaries’ access to chips. That is von Neumann’s preventive logic with a better press team. It may even be the right strategy. It is also the strategy of every arms race: our side must win first, because their side winning is worse.
Each step is defensible inside the frame. That is the problem. A consequentialist frame, run by the organisation that benefits from the answer, approves each next step. The constraint moves every time the calculation needs it to.
To be fair to Anthropic: its best moment in this history was a rule. In February 2026, the Pentagon demanded that Anthropic drop two red lines, no mass domestic surveillance of Americans and no fully autonomous weapons. Anthropic refused. The government designated it a supply-chain risk, a label normally used for foreign adversaries, and in September a federal appeals court upheld the designation. Anthropic paid a real price for a line it would not cross. That is the Arkhipov move. It also shows that Anthropic knows how to hold a hard line when it wants to. It chose not to hold the pause.
What to ask of organisations this powerful
Consequentialism is a fine tool for a person choosing between charities. It is a dangerous operating system for an organisation that controls nuclear weapons or frontier AI, for three reasons. The organisation estimates the probabilities itself. It benefits from the answer. And the downside falls on people who never agreed to the bet.
Four things would change that:
- Name the stop number before the market does. Compton said a number and a rule: above three in a million, we stop. Frontier labs should publish the risk level at which they stop training, and who measures it.
- Outsiders estimate the risk. In 1961, the insiders counted the enemy’s missiles and got close to 250. The real number was four. Risk estimates for frontier models should come from people who do not hold the equity.
- Commitments that the company cannot rewrite alone. A safety policy that the company can edit in a blog post is a press release. Changes to a lab’s core safety commitments should need sign-off from a body outside the company. Anthropic’s Long-Term Benefit Trust was meant to be that body, but a large enough supermajority of shareholders can still change its powers, and after the IPO the shareholders will be the public markets.
- Red lines, more of them. The Pentagon refusal worked because it was a line. Pausing when safety cannot keep up should be a line of the same kind.
Compton’s three in a million may have been the wrong number. He still had a number, a rule and a willingness to stop. The people who now hold the most dangerous technology since the bomb have a spreadsheet instead.
