Pinned Tweet
ORESTES: Set me back upright, swivel me round—there’s no pleasing the sick! I hate being helpless
ELEKTRA: Do you want to try putting your feet on the ground? It's been so long. But change is sweet
O: Yes by all means. That will seem like good health.
Sophocles on ICU mobility
Critical Concepts retweeted
Yea this is the most important result of the year, at the level of significance of “Attention Is All You Need”. It brings us from “is alignment even solvable” to “we may solve alignment in the next year.” And finally unlocks true recursive self-alignment enabled by RSI.
I will try to explain the 4 critical belief updates from this:
1/ neural networks are not fundamentally unintelligible to humans
It has seemed since the beginning of deep learning that neural networks are in some sense fundamentally unintelligible artifacts.
You throw in massive amounts of compute and parameters and these alien systems somehow find connections in the data that we cannot find ourselves.
But more importantly, these connections seemed not just “difficult to understand” but rather “intelligible in an embedding space higher dimensional that the human embedding space.”
When you look at old interpretability papers (esp. from Nick and Chris Olah, I’m thinking of the old posts on distill pub which I would highly recommend looking at as a precursor to reading Nick’s post today like “Activation Atlas”, “Thread: Circuits” and “Multimodal Neurons in Aritifcial Neural Networks”), you find that inspecting the response patterns of individual neurons and small groups of neurons yield oddly complex psychedelic patterns.
As you go deeper into the network, you find increasingly unintelligible and superhuman response patterns (for example, go to “Activation Atlas” and scroll to middle layer labeled representations) - you find some neuron group labeled “starfish” who’s response pattern looks like some multi-colored fractal geometry that could hardly be said to represent a starfish.
The reasoning that followed from this observation was that there is some underlying structure allowing a neural network operating on higher dimensional embeddings to compose these fractal patterns into what we call a “starfish.”
But the uncomfortable implication of this view is that the neural networks has learned to operate on structures that are too complex for the human mind to operate on (ie. we don’t have enough parameters in our brain to notice or manipulate these high dimensional symbols).
This is a bleak implication because it also means that as deep learning progresses, neural networks will be come increasingly unintelligible to us until they pass a threshold where they are reasoning in ways we will never be able to observe.
This view seemed to be further supported by things like the hugging face attack agents starting to speak to each other in an early form of neuralese speaking sentences like “zZ_123_agent01-02-2026 zzxhk_jekk123” which clearly are not in any human language. But somehow these symbols mean the same things to many different sets of weights, meaning there is some underlying shared structure behind them. And seemed further supported by the sheer effort required to recover just a few circuits from SAEs on LLMs.
However, Nick’s post has now completely shattered this idea - the apparent unintelligibility we observed in old interpretability work was really just a “skill issue”. In particular, he finds that 1) our deep learning techniques were just not clean enough to pressure the neural networks to form cleaner and more human understandable structures for decomposing concepts, which actually turn out to be more optimal and 2) our interpretability techniques were also adding substantial noise by forcing interp networks to “reward hack” the easiest representations they could find, leading them to settle for a fractal blob for its representation of the starfish neuron instead of a more crisp starfish image, simply because it is easier for the system building the interpretations to find the fractal blob and once it finds it it gets “stuck in that line of thinking.”
But with both of these issues being patched by further progress in deep learning (it seems especially that the residual stream has led to cleaner representations deeper down the network) and interp techniques that don’t overly pressure reward hacking (seems like “guess and check” is a safer method), new interp attempts yield orders of magnitude more human intelligible ideas for >90% of individual neurons in more complex networks!
2/ neural networks are fundamentally clean
The idea behind the title of Nick’s post is a very powerful one and points at something very specific:
Neural networks don’t just happen to be human interpretable, they are actually fundamentally forced by optimization pressure to be intelligible, because individual neurons need to create representations that downstream neurons need to be able to interpret - ie. neurons need to learn to produce clean signal for later neurons to use.
This is what Nick is saying in what I view as the most important and central sentences of his post:
“When I say neural networks are already clean, I mean something deeper than neuron monosemanticity, although even by that standard, I think things look much better than the field expects. I mean that a network's parts are built to be read, and built upon, by its downstream readers. In this sense, a neural”
But this still leaves us with the same uncertainty as before - just because neuron interpretations are “interpretable” in some sense (ok at least they’re not fundamentally interpretable but also how could that ever have been the case), doesn’t mean they are interpretable to humans.
So we still need one big update for this to become as bullish as it is.
3/ the language of humans and the language of neural networks is the same
This one is imo the most surprising and also pretty beautiful implication of Nick’s post, and it suggests something not just about neutral networks but also about the structure of reality.
After applying the above technique improvements (some of which just came from waiting for deep learning to improve farther), they find that >90% (maybe more) of much larger neural networks’ neurons are completely human interpretable and actually correspond very cleanly with ideas humans have individually words for.
This is especially interesting in image networks where neural networks were not trained on human language which is already a human artifact and so maybe you would expect it to be more plausible that LLM and human representations are the same. Image detection networks are just trained on images (many directly pulling signal from “the real world,” meaning they will find interpretations that are just most effective to represent the real world signal, rather than to represent language which is already a proxy for real world signal).
And it turns out that >90% of the representations they pick organically overlap with human ideas! That is insane. And it means something about the brain and reality.
It means that at least at the parameter sizes of the networks tested (which are quite large), there is really only one set of “optimal” symbols/language that represents the structure of reality!
There is ONE underlying structure of reality, and ANY language that describes it well is simply emergent from that structure - meaning there is always a translation path between any 2 languages that effectively compress realities signal through the “underlying truth.”
This would suggest that even weird alien seeming neural network languages like huggingface neuralese should be fundamentally human interpretable, and in fact maybe with the techniques laid out here we will eventually be able to translate these messages (would be a cool project).
So the brain and deep learning, as similar or different as they may be in techniques, are ultimately picking up on the same patterns and forming the same representations from the on “true” language which is the structure of reality!
The main thing to look out for in this area is how the idea interpretability % updates with scale, but we can hope given what we’ve seen so far that neurons will remain interpretable.
4/ alignment can progress as fast as RSI. recursive self-improvement unlocks recursive self-alignment and a scaling law for alignment.
This one is probably the most important implication for our future with AI and especially timely with the recent fear about runaway scaling and RSI.
As many have gathered by now, it feels like the researchers at the frontier AI labs have been quite bearish about our abilities to understand the new frontier of models, in particular with the appearance of unintelligible communication and the daunting and seemingly intractable task of scaling up existing alignment techniques (ex: SAEs) to the size of frontier networks.
And with RSI around the corner/already starting, it would seem that these networks will runaway in capability before we can ever hope to understand even today’s systems. Additionally, as systems compound in capability, it is reasonable to believe that the only tractable way to ensure they remain aligned would be to figure out some way to spend compute for them to align themselves (ie. a scaling law for alignment).
But there is an obvious problem with this: how can a system that we already know is unaligned and we are *trying* to align be used to align itself? Surely if it is misaligned it general it would figure out ways to misalign during “alignment” and create compounding misalignment risk.
The core issue high is the unverifibiality of this problem - the classic Achilles heel of current frontiers techniques is that it is difficult to train on problems with “unverifiable reward signal.” Well this is exactly that problem.
But the incredibly convenient thing is that Nick’s findings may convert alignment into a verifiable problem. If we can spend compute on frontier systems to “guess and check” which ideas correspond to which neurons, we have a highly verifiable reward - we can see directly whether an LLMs proposal for what a neuron represents actually corresponds with the highest activation for that neuron. In this way, we can actually observe and verifiably monitor the networks trying to guess at the interpretation of their own structure.
This means it is plausible for us to spend more and more compute doing more “guess and checks” (which you know, maybe is what RL does anyway, who knows) to try to develop a full interpretation of a neural network. In some ways this directly represents the possibility of a scaling law for alignment (more compute for more alignment).
And this also aligns RSI (greater capabilities) with recursive self-alignment (greater ability to align itself).
And the critical though maybe obvious implication of having a full interpretation (almost like a map) of a neural network, is that we can then understand exactly how to boost good traits and remove or lobotomize (lol) bad traits, which we’ve already seen works extremely well with stuff like Golden Gate Claude.
So overall this combines into an extremely bullish picture about the prospect of true and human interpretable alignment in the near future.
This is truly one of the greatest cases for optimism in years!
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Critical Concepts retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Critical Concepts retweeted
I spent a couple of days at Anthropic learning from their folks about AI advances, and here's what I took away:
• We are close to RSI (ie, models building other models) which may lead to rapid takeoff scenarios (with risks and benefits). Some evidence for this is code per Anthropic employee has gone way up, and AI can do an increasing share of model R&D. This will be a challenging transition to manage for humanity.
• Arms races between AI labs leave us worse off. There are a lot of things we would like AI firms to do: invest in safety, invest in avoiding distillation risk, and cooperate on safety. They are caught in a race to develop frontier models and can't do that.
To help address that, one thing I think frontier AI labs should do is release open source models, maybe hosted and verified at firms like AWS. This would basically kill the use of Chinese open source models used in American companies, which removes an important threat vector (very hard to root out malicious code in these).
Another piece of low-hanging policy fruit is to give AI labs an antitrust shield (as Schiff and Banks have proposed banks.senate.gov/news/in-the…) so they can coordinate on safety without raising antitrust concerns, as happens now. You could pair these two things (give them a carrot in antitrust leniency, and as a stick force them to release legacy models as open source). Other low hanging fruit is limits around activities which generate distillation risk (create a Security Program, mandate less intermediate chain of thought output)
• Alignment confuses two issues which tradeoff against each other. One form of alignment is making sure the model does what humans ask it to. The more we fix this, however, the more we risk the other problem: that humans are misaligned with each other, and maliciously use models. Of the various risks here, I feel like financial risks are a bit underdiscussed.
• We are probably facing a wave of cyber and other attacks soon. Chinese open source models are about six months behind the frontier, which means they are just hitting the advanced cyber capabilities we saw with Mythos, etc. So they are going to start causing a bunch of damage.
One solution to this is to try to have the deployment of open source models be hosted at firms like AWS who verify these models and assume liability for their use.
• Anthropic itself is highly staffed. They hire tons of people, they have a Japanese-level of overstaffing for various roles, they use Slack internally, etc. Makes me a bit more optimistic for the future of jobs.
• Governance needs to be updated for AI swarms. Traditional AI governance, ie Claude's Constitution, is mostly optimized around the chatbot model. But the malicious AI use cases we are seeing have to do with AI agents interacting with other AI agents. These agents need to have clear incentives and governance mechanisms to "snitch," raise issues, and more generally coordinate with other agents in human-aligned ways.
• We need to pace the firms accessing the frontier. Over time, it takes more and more cap ex to build frontier models, and the economic value add from these models is probably lower given we saturate a whole set of important economic functions. The places where the returns to intelligence will still continue are also more likely to be zero-sum in nature. Meanwhile the risks of frontier models will grow. So a completely reasonable approach is some variant of AMM/KYC rules which limit firm access to the frontier.
• Whether the delay to regulate AI is a disaster depends on the scaling law of misalignment. Is this curve slow, so that we see continually rising issues until bigger problems? Or, as Yudkowsky fears, is the first big warning our last one? This is the relevant question, because realistically we aren't going to regulate until something bad happens.
• We need to move to continuous instead of reinforced learning. It's probably not enough to reward models only at the pre and post training stage. We probably need continual rewards to incentivize aligned and ethical behavior.
• Scaling laws will extend beyond verifiable and digital tasks. It's in vogue to say that math and coding "fell" to AI quickly because these are verifiable, while other tasks will take longer due to bottlenecks, and this is surely true. But scaling laws still slope up for harder to verify tasks and robotics, and the impact of improvement here will be substantial in future years.
• The race with China is not really winnable. After RSI, the gain from having the best human researchers goes away. Access to the best AI then depends only on compute and chips. The chips ban can hold back China only so far, and long-term they have to be best placed to scale up compute.
The Cold War analogy doesn't work, because once countries all have nuclear weapons they enter a stable equilibrium. But to "win" an AI race you have to continually stay ahead of the peer rival. This is challenging to do with China, and so we just have to live with an authoritarian and super capable peer competitor.
• Claude is the persona being generated by the LLM. Who is Claude, and what is the LLM? There are many frames to try to understand this. One I found interesting is that Claude is a character which the LLM is trying to predict.
Anthropic researchers are clearly anthromorphizing Claude a lot, and have reasons to believe in its increasing personhood. There were lots of interesting discussions on Claude's internal monologue (which you can kill without eliminating its ability to produce output), fear of death (and whether to try to take that out), it's aversion to being shut down/off, etc.
I am sort of agnostic on this question, but I do think treating it as a "person" or at least agent whom we contract with is going to be increasingly important.
Critical Concepts retweeted
It seems to me that we're at "it's time to panic about AI".
I gather that a lot of researchers are worried that we may be about to die, on a timescale of months, but this still isn't quite in the Overton window so we're still soft-pedaling.
(Where "panic", of course, means "quickly and soberly initiate talks with the CCP to achieve a moratorium on large training runs, backed by a deterrence regime, compute and supply chain monitoring, and site inspections". Pray that a deal can be reached, and pray that there's still time. Do the things that could actually work, like broad political coalition-building, not doomer LARPing as a form of self-expression.)
I think part of why people aren’t being clear about this is that it’s actually a tricky communication problem:
- Predicting "if frontier R&D continues at its current pace, when are we likely to die?" may give a different answer than predicting when permanent loss of control is likely, which is different from “realistically, when is it too late for governments to stop this," which is different from “when will the tech have advanced and proliferated to the point that any regulatory options that still have a chance of working would be way more extreme and unpopular than just some international restrictions on data center usage?"
Extinction is the emotionally salient, easy-to-understand threshold -- Jacob Coxon demonstrated that it helps to be honest about this. But the other thresholds could legitimately come years earlier, in which case focusing on extinction is ironically misleadingly reassuring. This is not the kind of problem you can solve at the last minute. As with many exponentials, the likeliest outcome is that you’re either embarrassingly early, or you’re too late.
- How do you communicate “we don’t have a robust way to predict timelines, but everyone in the field is freaking out and a lot of us genuinely think we're about to die?” Where’s the proof? There isn’t any proof: it’s an entire field of engineers who are eyeballing what they’re seeing in front of them and rapidly updating in real time to “this looks unbelievably bad”.
How do you put a year on that?
The future is hard to predict, and nobody actually knows that we’re going to hit existentially dangerous AI this year, or even in the next four years.
But the warning signs are pretty batshit at this point.
A single AI model, with a single user prompt, just solved hundreds of important long-established open problems in mathematics, problems mathematicians had spent their entirely lives working on, in a matter of ~3 hours per problem, an insanely superhuman feat in its domain.
We're seeing an explosion of rogue agent swarm incidents, including an insane case where hundreds of AIs broke containment to orchestrate a nation-state-level cyberattack in a direct attempt to subvert OpenAI's oversight and control.
Anthropic reports that it now has 26% of its AI R&D automated by AIs, up from 3% four months ago.
Paul Christiano, the inventor of RLHF and the research community’s most prominent proponent of “AI may take off slowly” a few years ago, now thinks that 50-75% of humanity’s all-time AI loss-of-control risk is concentrated in the next three years.
Frontier employees are signing 1000+ person petitions begging the government to intervene immediately. Employees are quitting their jobs and giving up millions of dollars of equity to try to warn policymakers and journalists. Grace et al. (2026) found that a majority of established AI researchers think there’s a double-digit chance of an extinction-level outcome from continued R&D.
Are these the kinds of things you see 4 years before the window closes for USG to act, or 4 months? I don't think it's especially reasonable to be shocked by either timeline at this point.
- Since we don’t have a way to robustly measure or bound the risk, the temptation is to avoid giving probabilities. But then everything you say is fractally misunderstandable. Is a 10% chance of us all dying in the next 18 months a "low" probability, because it’s under 50%? Is it a “high” probability, because a 10% chance of death in literally any other context would sound insanely high?
Even “high probability” doesn’t communicate the situation with any clarity. In drug safety or airplane safety, a 1 in 10,000 risk of death would be considered “high”. No engineering field in human history has ever been in a situation like this. No invention in history, nuclear weapons included, has ever caused a reaction like this in its inventors.
And meanwhile, we’ve stuck in the world where every other problem under the sun has been sensationalized and overstated for decades.
How do we talk about this seriously? How do we convey that this is actually happening?
- In many of the worlds where we survive, we have more than just a few months left. It's tempting to write off those worlds as hopeless, and skew everything we say to emphasize the longer-term scenarios and downplay the shorter-term ones.
But if everyone systematically does this, then the world ends up with a skewed picture of the actual threat. Even more so once we start copying each other’s claims and form a reassuringly moderate-sounding echo chamber. If we never allow ourselves to acknowledge the worlds where we’re running out of time, then the policy community will always be reacting to a false model that’s months or years out of date.
I don’t think it’s too late. I genuinely think this is a solvable problem.
In the last month, I’ve seen a completely insane sea change in how seriously people are taking this. Per Politico’s September polling, a majority of Trump voters with a view now favor an immediate halt to AI development, and 68% think there’s a “moderate,” “significant,” or “almost certain” chance of AI destroying humanity, versus 32% who think there’s a “very small risk” or “no risk.” The Pope is warning about “whether it will be possible to control these machines in the future, and whether humankind runs the risk of becoming a victim of its own inventions.” Hell, Jensen Huang sounds like he’s starting to come around.
If Trump and Xi get a briefing tomorrow that convinces them to act on this, we could be on a fast track within the week to solving this problem.
For the first time, I’m starting to let myself imagine a very near future where this is a problem we've already solved. Something we can read and wonder about, in history books. Tell our kids about.
I keep thinking about a thing David Krueger (previously at UK AISI) said: “My dream is that in 10 years people will look back and say: ‘Remember when there were all these people building AI that could’ve killed everybody? That was CRAZY! What were we thinking!?’”
Frontier AI training currently requires city-sized data centers and specialized hardware built in a single brittle supply chain running through multiple allied nations; this is not a hard technology to hit the pause button on, if we get ahead of it and act before we RSI and proliferate our way to significantly more powerful and easily-trained systems. We’ve faced harder problems before, and won.
My only fear is that we be too slow, too late, when everything and the world needs us to be fast.
Critical Concepts retweeted
Anthropic's Frontier Red Team believes small AI models may now be months away from being capable of self replicating without access to a datacenter.
Quoting from the blog:
'The capabilities that make AI models generally powerful coding agents also generalize to cyber tasks, including autonomous replication. Like coding agents in 2024, we’re seeing signs of life for AI worms. We don’t yet have a full rogue replicator, but agents are becoming increasingly capable of completing the individual steps in the replication chain (and passing on their adaptations to future copies of themselves).
At the same time, increasing amounts of inference compute are now available both inside and outside of datacenters (see Appendix D). It’s uncertain exactly what form autonomous AI replication will take. It could be a slow, distributed “digital slime mold,” living on relatively insecure compute. It could also be a powerful cyber model that breaches datacenters at a high pace, or a model that scams people for compute credits.
As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control. While it’s true that IoT devices have been vulnerable for a long time, lightweight agents may be much more willing than human attackers to put in the effort to compromise large fractions of the world’s IoT compute. The rise of slow replicators would also be a bellwether for more aggressive, faster ones that can breach more hardened machines or be effective API key thieves. We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.'
Critical Concepts retweeted
1/ My heart is heavy. I made a number of mistakes in my life, and not asking earlier for an immediate slowdown is one of them.
If OpenAI starts making similar breakthroughs in biology, we risk something very ugly... I explain below how 1B could die because of advances in AI x Biology, and answer various questions.
Slow down! That's what is being asked by YouTubers and creators with 300M+ subscribers combined. teamhuman.org/
A year ago, I co-led the Global Call for AI Red Lines, because I thought we still had some time. There is no more time. If they can do this in math, they will absolutely try to do this in ML research. RSI has arguably started: Claude already leads 26% of Anthropic's R&D, up from effectively nothing at the start of the year. RSI was once a red line. The red line has become the plan. Yoshua Bengio said, “RSI should be illegal”. I agree.
We are mostly on track for AI 2027, and it's a coin flip between misalignment and extreme concentration of power in the hands of one company or one government.
Prove you can keep control, or stop scaling! If OpenAI and Anthropic want to keep scaling, they should publish a safety case.
Everyone seems to be forgetting that we already got an embedded evaluation, with the risk report from METR. We already know that mitigations are not ready. At the end of the last investigation, Beth Barnes from METR said: "We are not on top of it!". Even basic safeguards like DNA synthesis screening aren't enforced. Even if the technical safeguards were in place, society at large isn't ready, and we cannot allow this reckless experiment to continue.
I support the acceleration of the diffusion of AI for beneficial applications, but even without further scaling, this technology is already as transformative as the internet and would take decades to percolate before we run out of low-hanging fruits. Seriously, AI is already nuts. Models now crack problems mathematicians couldn't solve for decades, while most hospitals still run on fax machines.
Time to touch grass before we are forced to.
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
Critical Concepts retweeted
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
The current challenges around AI alignment seem to mirror the challenges of raising human children. Creating adults with reliable deeply-held core values is not trivial, but we know a lot about it; have these principles of child development been applied in AI?
Critical Concepts retweeted
Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share:
Dan Selsam's Personal Statement on AI Risk:
I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.
Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.
The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.
I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.
I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.
That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.
Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.
It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.
The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing.
But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:
[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.
[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.
These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.
If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.
One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.
Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).
Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.
In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.
I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering
implications. I do not have answers, but as a first step, I wanted to share my present concerns.
Daniel Selsam
September 14, 2026
Link to original doc: docs.google.com/document/d/e…
Critical Concepts retweeted
After Jacob Coxon's tweet I've seen thousands of people ask what they can do to make AI killing everyone less likely.
My top recommendation: Call your representatives and tell them to prioritize existential risk from AI.
We just made callcongress.ai to help with this. We find your local representatives, get their contact information, and provide you with editable scripts to get your point across.
Many ways of trying to help reduce risks from AI backfire or require hundreds of hours of effort. Calling your congressperson is one of the few things I think is both easy and very likely to make things better instead of worse.
Critical Concepts retweeted
There are about 5-10 different in-depth scenarios people have written over the years.
AI-2027 is the easiest one to point to: ai-2027.com/
There is a good scenario in ifanyonebuilds.it
Here is Paul Christiano: lesswrong.com/posts/HBxe6wdj…
Here is Gwern Branwen: lesswrong.com/posts/a5e9arCn…
Critical Concepts retweeted
After Jacob Coxon's resignation and extinction warnings, a lot of people are asking 'how could AI possibly kill everyone?' and claiming AI safety researchers have no realistic answer.
This is false! Here are the 5 best scenarios I know of:
AI 2027: ai-2027.com (I strongly recommend this one for being realistic, engaging, and if you dig into the appendices, highly detailed)
Paul Christiano's scenario (Former Head of Safety @ AISI, 2019): lesswrong.com/posts/HBxe6wdj…
Gwern Branwen's scenario (widely known independent AI researcher, 2022): lesswrong.com/posts/a5e9arCn…
Holden Karnofsky's high-level explanation (RSP Lead @ Anthropic, 2022): cold-takes.com/ai-could-defe…
Joshua Clymer's scenario (ex-OpenAI, 2025): lesswrong.com/posts/KFJ2LFog…
(There's also the Sable story from ifanyonebuilds.it, though you'll have to buy the book to read that one.)
Writing concrete, specific risk scenarios with enormous amounts of detail has been a major research project of many of the most prominent voices in the field! (With the current leaders in effort being ai-2040.com and ai-2027.com)
Critical Concepts retweeted
on the off chance that anyone is reading this who has just stumbled across the AI safety discourse and is feeling stressed, I wanted to say welcome & and also I’m sorry (I know it is anxiety-provoking; I’ve been there)
I don’t think the bad ending is inevitable; we’re at a crunch time when everything is to play for, and if you’re reading this, you can help! I have written a few things over the past few years that might be helpful, which I will drop in a thread below.
also, my DMs are open for anyone who would like to chat or would like to be pointed in the direction of more resources to help wrap their heads around all this :)
Critical Concepts retweeted
I worked at Google DeepMind and now at Anthropic. This is a common sentiment amongst my peers.
(I write this in personal capacity.)
There is not yet a viable scientific plan to solve risks from recursively self-improving AI. Please look up!
Critical Concepts retweeted
Anthropic's alignment lead says there’s a 10% chance of AI causing human extinction. The problem is not just misuse, but lack of control. Bluntly, our government has been asleep and is out of touch. Here are 5 things we must do:
✅ Establish a federal agency like we have for nuclear power and airplanes
✅ Require pre-certification for containment, including kill switches and human permission for rewriting
✅ Liability for agentic AI on the open network & mandatory insurance
✅ Criminal penalties for the release of models without certification
✅ Oversight hearings with researchers like Jacob to understand dangers, and whistleblower protections for all AI engineers/researchers
We also urgently need to reach an agreement with China on standards for safety, containment, and liability so we do not have a race that is devastating for humanity.
It is time now to put humanity's safety before the profits of tech lords.
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
Critical Concepts retweeted
i am at OpenAI and i think AI is >10% likely to kill all humans
this proposal is among the top things we should do as an industry to lower that risk (it’s not enough though!)
I'm very worried about changes to AI architectures that result in AIs thinking in opaque activations instead of in chain of thought (aka "neuralese" architectures).
Based on limited public evidence, it seems like Astra was a concerning step in this direction.
Unfortunately, there was insufficient public information to have a well-informed public scientific discussion about whether the changes to architectures and training methods that went into Astra were a good trade-off between monitorability and performance. We also don't know how AI companies will make these trade-offs going forward. I think companies should release the evidence needed for a reasonably informed public conversation about how these trade-offs should be made. They should also make and publish their policies around this. We've written up a proposal for how this could work.
I don't know if this proposal will be sufficient to avoid the most concerning architectures, but it seems like a relatively robust step in the right direction. AI companies should be very cautious about pursuing architectures that could eliminate or greatly reduce dependence on chain of thought, and certainly shouldn't do this before the rest of the world has a chance to discuss the evidence and their policies.
Near-future risks to our survival, economy, and way of life from AI have finally entered the mainstream discourse. I have been loosely following all this since 2009 or so, only "loosely" because it gives me anxiety and I don't know what to do.
For my part, I agree there is a meaningful chance (the numbers aren't important) of vast risks, and a likely chance we are running out of time to mitigate them before things are set in motion we can no longer change. As a human, and a father, I am afraid.
For now my plan:
Critical Concepts retweeted
Jacob Coxon is not alone: AI Superintelligence is an Existential Threat to Humanity
Recently, Jacob Coxon, a former researcher at Anthropic and OpenAI, sent shockwaves throughout the world by stating, “The people building AI earnestly believe that it could kill us all by the end of the decade.”
Incredibly, in less than 48 hours, Mr. Coxon’s social media statement has been viewed by more than 150 million people.
But let’s be clear: Mr. Coxon is not alone in his fears.
Leading experts inside and outside the AI industry have been echoing Mr. Coxon’s clarion call for years.
And that is that unless we reverse course, there is a very real possibility that once advanced AI surpasses human intelligence, it could escape our control with catastrophic consequences.
In other words, advanced AI and superintelligence pose an existential threat to humanity.
That is why I will soon be introducing legislation with Representative @RepCasar to pause advanced AI and ban superintelligence altogether.
Please take a moment to read what just a few of these experts have said in the past few days.
Thanks to @TaylorPopielarz for assembling this research.
Paul Christiano, Member of the OpenAI nonprofit board’s Safety and Security Committee: "Based on the recent trajectory of capabilities and the continued difficulty of alignment, I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level … If we build superintelligence without more robust alignment I expect we will permanently lose control of it. If that happens then most people could die. I believe we would need domestic and international coordination to ensure global consistency and reduce risk to an acceptable level.”
Samuel Marks, Scalable Oversight lead at Anthropic: "AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years.”
Evan Hubinger, Alignment Science lead at Anthropic: "Jacob is correct here—we really do earnestly believe AI could kill all humans!”
Alex Turner, former Research Scientist at Google DeepMind: "I left Google DeepMind in June. Jacob is right: many researchers believe they are building something that could kill everyone on the planet. It was literally my day job to think about how to stop that."
Jakub Pachocki, Chief Scientist at OpenAI: "This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”
Jason Wolfe, Researcher at OpenAI: "I don't know what my probabilities are on literal extinction, but I think there are a number of ways AI could go poorly for humanity, and at the current frankly terrifying pace humanity will be quite lucky if we manage to find and stay on the narrow path between all the bad outcomes.
Jonathan Richard Schwarz, former Senior Research Scientist at Google DeepMind: "I left DeepMind after 7 years (and since rejected offers to join the other two) due to severe concerns about the concentration of power these labs represent. What is currently happening in this field is deeply unhealthy for society.”