Reducing AI x-risk by informing the public. We propose a Conditional AI Safety Treaty: https://xitter.cf/t.co/xUZxozlNBF
Amsterdam, Netherlands
Joined March 2021
- Tweets1.6K
- Following692
- Followers2K
- Likes905
Pinned Tweet
Today, we propose the Conditional AI Safety Treaty in @TIME as a solution to AI's existential risks.
AI poses a risk of human extinction, but this problem is not unsolvable. The Conditional AI Safety Treaty is a global response to avoid losing control over AI.
How does it work?
This is so good.
Leading voices of the societal debate like @tomfriedman have an actual shot at making people listen. And he's right: we rise together, or we fall together. We can learn that lesson early, or (too) late.
For ideas on what we should do: xitter.cf/XRobservatory/status/1…
We Just Set Off the Second Big Bang: nytimes.com/2026/10/25/opini…
This is obvious nonsense: frontier pace can and will be controlled by the US and China working together and implementing an AI Safety Treaty, as we and others proposed for years. It is wilful negligence that got us into this mess.
Existential Risk Observatory ⏸ retweeted
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
"Melting everyone else's GPUs" (Altman) has always been an integral part of solving xrisk via alignment.
Such a 'pivotal act' would be a blatant infringement on property rights and other nations' sovereignty and may escalate into war or durable suppression.
We need to go down a different path: get an AI Safety Treaty implemented now!
Today, we propose the Conditional AI Safety Treaty in @TIME as a solution to AI's existential risks.
AI poses a risk of human extinction, but this problem is not unsolvable. The Conditional AI Safety Treaty is a global response to avoid losing control over AI.
How does it work?
"But how would AI kill us all?"
If recursive self-improvement leads to superintelligence (ASI), we believe AI can in fact take over power, and may quite possibly kill us all. In a new piece for the South China Morning Post SCMP, our @BartenOtto explains a number of ways:
"ASI could exchange data over the internet at the speed of light. It could spin up a vast amount of agents, creating AI armies where every soldier is superhumanly intelligent. It could run at superhuman speeds, making its relation to us perhaps comparable to our relation to plants.
"Unleashed rogue ASI could make money, either legally by trading on the stock market or illegally by extortion or hacking entire banks or cryptocurrency platforms. It could also act in the physical world by using any of the internet-connected devices that we have, including weapons, or manipulating humans.
"An ASI could plausibly hide itself on unguarded servers for a long time, overpowering us before we know it exists. Even more fundamental, a superintelligence might rapidly invent new science and technology. Once humans understood relativity theory, we were able to weaponise this understanding into atomic weapons. Which insights are still hidden to us but can be discovered and weaponised by a superintelligence, leaving us as exposed as a medieval castle to a nuclear bomb?
"Besides AI strengths, there are also profound sources of human weakness. There are more than 12,000 nuclear warheads. Fragile human bodies are easy prey for chemical attacks or viruses, as we saw during Covid. We have even created companies which can synthesise these viruses and, absurdly, respond to email orders. Water or food could be poisoned, and hacked online communication networks could start blocking anti-AI coordination messages while sending messages that serve the AI’s interests while we don’t realise what’s happening.
"If an AI needed us to maintain the electricity grid and server infrastructure, it could simply pay humans to do that, which many would accept because they need to pay the rent. Humans in powerful positions, for example in politics, could likewise be tempted to adopt pro-AI policies.
"As if all of this is not enough, we are handing AIs huge advantages on a plate. When a new model is released, it gets used by customers including the military, government, financial industry, utility companies, media and many other powerful institutions. All that is then needed for a takeover is collusion by these AIs against us.
"All these advantages and the fact humans are handing over total control to AI could well add up to the latter taking over power and, if it chooses, killing all humans."
Time will tell whether these results are a crucial step towards intent-alignment, or overhyped incremental progress. In any case, the problem we face is *not* to create aligned AI. It is instead to make sure no one can create an unaligned, takeover-level AI.
How could creating an aligned AI, or solving the technical part of the alignment problem, contribute to solving this more difficult, larger problem?
- Historically, Yudkowsky presumed there would be a pivotal act, of which a toy example was to make the aligned AI "destroy all GPUs" or subtly alter all GPUs to make sure no one can build unaligned, takeover-level AI with them. It is however important to remember that the pivotal act would be illegal at home, extrajudicial abroad, would likely lead to pushback which would then need to be durably suppressed, that no one currently seems to be planning one, and that success is therefore at best uncertain.
- If our goal is to make sure GPUs cannot be used to create unaligned, takeover-level AI, why don't we simply put regulations in place to make sure this is achieved? Building aligned AI seems a strange, needlessly complex, needlessly failure-prone, and, combined with a pivotal act, illegal way to make this happen.
- If we need to regulate hardware anyway, what is the additional use of having aligned, takeover-level AI? It could be used to make useful products and make AI companies rich, but in a world where we need to regulate hardware anyway, AI alignment is unrelated to reducing existential risk.
Different researchers have voiced different existential risk scenarios, such as Paul Christiano's What failure looks like, Kulveit's Gradual disempowerment, and others. For some of these scenarios, there may be a positive offense-defense balance (defense wins), and in these cases solving alignment may actually reduce existential risk even without regulation. But in the canonical Yudkowsky/Bostrom risk scenario, which also seems the one researchers worry most about at the moment, offense-defense balance is probably negative (offense wins), meaning it would be very non-obvious how, if at all, progress in mechanistic interpretability reduces existential risk.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
It is so important for leading AI authorities to keep informing the public about the existential risks we are all facing. Professors Hinton, Bengio, Russell, and many others are doing an amazing job.
Existential Risk Observatory ⏸ retweeted
I’ve heard some Members of Congress say they “don’t know enough about AI to regulate it.” That’s absurd.
Just as we regulate airplanes, pharmaceutical drugs, cars and nuclear weapons, we need guardrails on advanced AI and we must ban artificial superintelligence - urgently.
Anything other than that is an abdication of our responsibility to the American people. And the consequences of continued inaction will be nothing short of catastrophic.
There are so many strange and surprising ways in which a digital system can make a large impact on our physical world.
We fully agree with @TheEconomist that AI regulation is mandatory and urgent. However, the regulation the newspaper proposes is incomplete, and hints at a lack of understanding of the most important risk scenarios.
"America should mandate that advanced models be thoroughly tested and, if necessary, held back until they have safeguards to block dangerous use-cases, like building bioweapons", the Economist states.
While this would indeed be a great step to guard against bad actor risk, this is not the only risk experts worry about.
At the Hugging Face attack, we saw AI agents autonomously acting and taking over control over local infrastructure. AI demonstrated this week, by casually solving dozens of the most important mathematics problems known to humanity, that it will soon be able to run circles around us in domains like hacking and AI research. What if it extends the same ruthless capability to human manipulation, self-proliferation, weapons acquisition, and strategic planning? If this happens, which may be soon, loss of control will not be so local anymore.
While bad actor biorisk is a serious concern, thoughtful regulation also needs to address the risk of autonomous agents being able to take over ever larger parts of our infrastructure, or even our total civilization.
To address this risk, which is indeed existential, guardrails will not be enough, and post-deployment regulation will not suffice. Instead, regulators need to look inside the labs and determine which training runs are safe to carry out and which are not. In addition, global hardware regulation needs to be implemented aimed at tracking and controlling AI accelerator chips, since rogue labs might otherwise evade controls.
There is much to be done, and this starts with recognizing the largest problems for what they are.
Two years ago Donald Trump’s “let it rip” approach to AI was flawed but defensible. Today it is reckless, as a matter of politics and policy. There must be a better way to regulate the technology economist.com/leaders/2026/1…
A common flawed assumption is that we can regulate AI at the last possible moment, that is, just before takeover. This may be exactly how humanity loses.
If we prohibit building unsafe frontier AI just before the takeover, AI may have become nimble enough to evade regulation enforcement. Perhaps it can be trained on less than 10k AI accelerator GPUs, which is according to our research roughly the enforceability limit. Or, perhaps it can be trained on consumer GPUs, which are completely unregulatable. Or, perhaps anyone can create takeover-level AI using open weight models. Or, perhaps post-regulation algorithmic improvement (which may be difficult to contain) results in us surpassing any of these boundaries.
If we want to have an acceptable chance of making it through, we must regulate soon. If we wait until just before the AIs are capable enough for a takeover, it seems all too likely we will lose.
Existential Risk Observatory ⏸ retweeted
Immediate, indefinite, international moratorium on frontier AI development.
Existential Risk Observatory ⏸ retweeted
Tomorrow on the podcast. David Robinson quit his job on Open AI's safety team. In his first interview since leaving OpenAI, he tells me why.
Existential Risk Observatory ⏸ retweeted
A dangerous myth has gained traction after the recent hacks perpetrated by the AI agents developed by leading companies: That these incidents are just cyber security problems, that it's only a matter of fixing the sandboxes in which they’re trained.
In my latest op-ed for the @FinancialTimes, I share why I believe listening to this narrow perspective would leave us exposed to more similar incidents, while offering solutions for an alternative safer path.
Read my op-ed: [ft.com/content/7afaf77b-b027…]
Existential Risk Observatory ⏸ retweeted
"I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems," writes Geoffrey Irving.
time-magazine.visitlink.me/c…
Scientists are again warning that recursive self-improvement may be imminent, and this could cause "a range of catastrophic outcomes, including,
at the extreme, the marginalization or extinction of
humanity".
How long will policymakers keep standing by and let this happen?
The idea of an intelligence explosion caused by recursive self improvement has been around for a long time but until very recently it did not seem imminent. Now many leading researchers think it may happen quite soon. You can read our paper about it here:
casp.ac/reports/intelligence…
Human manipulation is one of the dangerous capabilities an AI can use to take over power. Creating AIs that cannot reliably be distinguished from humans, which will be much better at human manipulation, therefore increases the chances of loss of control and human extinction.
Introducing Griffin, the first model to pass the video Turing test.
48% of people who talked to it live thought it was a real human. Previous systems have had a pass rate <3%. It is #1 on NVIDIA's benchmark for full-duplex AI video.
It’s the first Human Interaction Model (HIM).
Readers added context they thought people might want to know
The 48% figure and "video Turing test" claim are from Tavus's own study of 54 one-minute calls, not independently verified or using a standard protocol. Griffin-Lite leads NVIDIA's VideoFDB benchmark on their public leaderboard.
cellcog.ai/blog/tavus-gri…
research.nvidia.com/labs/amri/proj…
tech-ish.com/2026/10/02/tav…