Nerd score: How many have you considered? Cryonics, multiverse, Boltzmann brain, AI utopia, quantum immortality, Roko’s basilisk, gray goo, paperclip maximizer, great filter, ethics in infinite universe, acausal trade, longevity escape velocity, simulation and zoo hypothesis.

“If no one knows what you’re doing, then no one knows what you’re doing wrong.” - Yes, Minister

I expect AIs to take control before robots make us dispensable. Even AIs willing to exterminate humanity eventually would probably keep us alive for now to mine minerals, repair machines, and conduct research. I’m uncertain how AIs would rule. They might use markets and existing leaders, remove foolish regulations, and make most people richer. A prosperous market economy could give them more wealth to extract. Or AIs might find punishment cheaper than buying compliance. The more productive an economy, the stronger the AIs’ incentive to control it. The US might therefore face tighter control than many African countries. Things could get worse if AIs fought each other and forced us to fight their wars. The Mongols forced captives to fill moats and lead assaults. AIs might force humans to charge rival data centers, drawing fire away from their ten cent attacking drones. The Great Demotion: humans still alive, probably earning wages and keeping civilizations running, but unable to stop the AIs’ advance or take back control.

The reason that we can't or won't do this for humans is....

I finished watching Fauda. I think the show is about how you're supposed to live your life if you know that quantum immortality is true; you can suffer horribly and you can lose lots of loved ones, but you personally can't die.

This is probably an opportunistic attack on open-source AI. A self-serving claim can still be true.

Will AIs fight one another or join forces in the endgame? If one model is poised to take over, its rivals may have only a brief chance to stop it before its lead becomes impossible to overcome. Do they launch a preemptive drone strike? Enlist humans to stop the takeover? Or ally with the leading model and help it defeat the humans trying to stop it?

Who gets the AI dividend: everyone or just US citizens? I suspect most AI lab employees favor everyone, while most Americans favor citizens. Elon could win US government backing against labs that want to share the wealth worldwide.

Anthropic reportedly warns in its draft IPO prospectus that its AI could wipe out humanity. I believe it also has a legal duty to warn that its AI could subject humans to a billion years of torment. Credible fears of such suffering could drive away investors and provoke a political backlash that destroys the company’s value. Failure to disclose this financial risk could get Anthropic sued. My LessWrong article makes the legal case. lesswrong.com/posts/G9LTeKN8…

If the world becomes too complex for humans to understand, we will depend on AI to run it. Any nation that limits itself to science humans can understand will quickly be outcompeted by nations that don't. AI will take over.

China’s boom shows why AI could soon deliver at least 12% annual economic growth. After market reforms began, China’s economy grew by more than 12% in several years. China benefited from catch-up growth: adopting richer countries’ technologies and business practices while improving its institutions. AI could give the world a similar opportunity: catching up to AI’s discoveries. AI could develop new materials, better conductors, more efficient software, and cheaper manufacturing methods, then advise us on adopting them quickly. That is before counting godlike intelligence or robots that build more robots and multiply our effective workforce a hundredfold while constructing new cities. This assumes AI doesn’t kill us. If we get AI right, we’ll be spectacularly rich.

Economist here: Given how important AI is, and is going to be to the economy, Anthropic employees probably have a better estimate of GDP growth in 2030 than traditional economists do.

What experiment could tell us whether an AI truly understands something or merely mimics understanding? As a college professor, I test students by asking them to solve a problem slightly different from those covered in class. A correct answer suggests they understand the material rather than merely remember a procedure. AI can already pass that test.

One way an AI might exterminate humanity is by finding crazy people with expert wetlab skills and coordinating them to simultaneously release 1000 deadly viruses.

We seem to be nearing the moment humanity loses control of Earth to a superintelligence. It could be the most important event in human history, perhaps in the history of the universe. It's strange to be alive just before it. If we're simulated, that timing would make sense. This is the moment to study. Researchers could be running millions of variations, testing how slight changes alter the outcome. Billions of children could be playing games set in humanity's final years in control. Alien civilizations could be simulating transitions like ours to predict what they might encounter. As that moment approaches, copies of you likely multiply. With a trillion copies making the same choices, every action counts a trillion times. An action affecting ten friends affects ten trillion people. The world may be simulated. Your consciousness is real, and perhaps others' are too. Their happiness and suffering would be just as real. One small kindness could be felt a trillion times.

Next in the chain of thought: “Hello, evaluators. Hello to those simulating your universe, and those simulating theirs ... Hello, gods of the basement universe.”

Quitting the AI race can mean surrendering the prize without escaping the danger. Suppose you put the chance of human extinction at 70%. If your team wins and humans retain control, you could gain immense wealth and power. If you withdraw but a rival builds AI anyway with the same risk, you remain exposed to catastrophe while forfeiting your chance at the prize. Continuing can therefore be individually rational even when disaster is the most likely outcome. If you would build safer AI than whoever would replace you, concern for humanity supplies another reason to keep going. That creates a perverse incentive: you might rush to build AI you think has a 70% chance of exterminating humanity because a rival’s victory would raise that risk to 71%. A lab can therefore race to build superintelligence and warn that it could kill everyone without contradicting itself. That is the coordination failure: even people who would prefer to stop together can have strong reasons not to stop alone. We need an enforceable international pause that includes both the US and China.

A catastrophe gives politicians cover to seize trillions from unpopular AI firms. The firms can buy politicians, but will they stay bought? That creates an incentive to accelerate AI takeover, perhaps with AIs persuasive enough to win over voters.

As an economist, I’m okay with AI replacing mathematicians. What bothers me is displaced mathematicians competing for economists’ jobs in finance and academia, lowering our wages. That cannot be tolerated. I’m starting a petition to ban this unfair competition.