Senior Research Fellow @forethought_org Understanding the intelligence explosion and how to prepare
Joined May 2022
- Tweets1.1K
- Following410
- Followers3.5K
- Likes4.1K
Pinned Tweet
So, exactly how big will the intelligence explosion be?
…Ten years of AI progress in a year? In a month?
Our new paper tackles this question head-on.
I've researched AI takeoff speeds for many years. This is my best stab at an answer.🧵
Tom Davidson retweeted
The interesting question to me is whether your pivotal act involves unilaterally and forcibly stopping others from AI development
This is the case in the "burn the GPUs" example. My impression was that MIRI was considering a strategy of this kind.
The point isn't about the GPUs, it's about the lack of legitimacy.
As contrasted with more politically legitimate strategies where you convince people of the risks and loop in the government to prevent dangerous development
This is what the current AI companies are pursuing. (To the extent they aim to stop dangerous development at all)
I think they are right to critique illegitimate pivotal acts, as Sama did in the thing you quoted in the OP
(Tbc I agree their development is reckless. But conditioning on trying to do a pivotal act, I prefer the legitimate ones.)
Tom Davidson retweeted
Based on this article, OpenAI has made false accusations about me. The three researchers fired from OpenAI were told I facilitated architecture leaks to The Information. This is false.
A journalist from The Information did reach out to me shortly before that article came out. I did not comment at all beyond asking the journalist to make it clear that a blog post I wrote which was referenced in the article was preexisting and public.
The article also mentions an internal OpenAI monitorability proposal. I have heard that this memo contained highly sensitive architecture information, and that OpenAI accused me of accessing and reading it. This accusation is also false.
As reported in the article, I viewed a version of the proposal that did not have sensitive IP and was approved for sharing with me. I believe OpenAI now knows this accusation was based on a conflation of these two documents, though I do not know if they have clearly retracted that accusation.
I do not and have never solicited proprietary information, and the three employees who were recently fired from OpenAI never leaked IP to me. As the article mentions, I did have Signal chats with them that had disappearing messages on; they’re professional contacts that I’ve had on Signal for a while, and use of Signal is common in the tech industry. During the Hugging Face investigation, I was careful to avoid sending Signal messages with confidential information, though I did sometimes discuss high level things on Signal calls. As far as I understood, this was normal.
New: The three researchers OpenAI fired last month spent the summer fighting to keep AI's chain-of-thought process visible.
Then, OpenAI asked to see their Signal chats. A week later, they were let go.
More below from @keachhagey, @berber_jin1, and me.
wsj.com/tech/ai/the-signal-c…
Tom Davidson retweeted
Lots of conversation around AI risks is about the gap between the unserious of AI labs' methods and how more rigorous industries work: airplanes, nuclear power, AI hardware. It would help a ton to lift AI to that level, but it is unclear even that level would be enough. 🧵
I think the hope is:
- ppl have political power today. They can vote.
- this is somewhat sticky. It doesn't evaporate immediately when they lose their economic leverage
- so ppl can use their political power to demand basic income
- they can also demand truthful ai advisors and then, with their advice, vote out any govt that erodes their power
The problem is: this relies on everyone being clear eyed about what's in their interests.
And there will be strong forces selling alternative narratives (eg "don't use the so-called "truthful" ai, it's woke/fascist")
Tom Davidson retweeted
I think we should stop AI progress now, first unilaterally by key AI companies and then scaling up to more US companies and then China via international agreements. The reason for now is not that current AI models are the ones that could kill everyone, but because the coordinate steps take time. Immediate action means we have more time for the second steps.
When I tell this story to AI company folk, the typical response is "Yes pausing is good, but we should wait until the last minute when the very next model will be dangerous, and stop then."
What I realized recently is that this is being implicitly way more optimistic about government speed than I am (and I joined the UK government and was happy there for 2.5 years). They are imagining a world where, the moment the danger is imminent, a bunch of international coordination can happen on a dime.
Which is entirely inconsistent with their actual views about government! I'm the one that is more optimistic about government; why am I the one advocating for more time cushion to let the government story play out?
Tom Davidson retweeted
OpenAI's red line for AI self-improvement is fundamentally flawed.
Imagine a speed limit that only kicks in after you've driven 5x over it for months, measured on your own speedometer, which no one can read anymore.
Better than no threshold. But a vague threshold that never bites is regulatory capture.
I think we have crossed the red line on cyber, and OpenAI can no longer control its AI; they should slow down or pause now unless they make a strong case that it's going to be alright, with much clearer criteria for continuing.
Full post below.
Lots of ppl have criticised Anthropic's bill for making it too hard for the govt to shut them down
This thread defends Anthropic: routing shutdown through the courts is needed to avoid extreme power concentration.
Idk how to strike the tradeoff.
I do wish AI safety spent more time asking "what would happen if the executive abused my proposal to take control of AI development"
Tom Davidson retweeted
I think Tom's point 3 is an important one. Within AI research there are many different fronts of potential improvement that can be defined. So there will be ways to parallelize automated AI R&D that scale better than these diminishing returns on individual tasks.
Great post.
I agree with Ramez that a million parallel agents isn't the same thing as superintelligence. A big parallel-copy-only intelligence explosion is very unlikely.
But I disagree that this is a knock against an intelligence explosion:
- A million parallel agents thinking 100x human speed could speed up AI progress an awful lot, though not infinitely.
- Those expecting a big intelligence explosion have always thought it would be driven by increased intelligence, not by copies.
My view in more detail:
1. In the limit, returns to more parallel copies go to 0. Tasks inherently require a certain number of sequential steps. Once you're already exploring every possible avenue, more labour doesn't help.
2. What that limit is depends on the task. Generally, more complicated tasks can absorb more parallel labour.
3. A task as open-ended as "improve AI capabilities" can likely absorb a *huge* amount of parallel labour. Much much more than the tasks Ramez looks at. The limit may be very far off indeed for the task that matters most.
4. And once abundant cognitive labour is available, AI companies will invent new ways to use it.
Eg attributing strong performance to particular actions during an RL run, hand-crafting each gradient update based on interpretability analysis, analysis weak RL runs and create targeted SFT data to patch gaps
5. Still, Ramez is right that in the absolute limit returns are worse than a power law. And many RSI models assume a power law for simplicity.
6. But others don't. For example, in AIFP's takeoff model the intelligence explosion feedback loop is driven purely by increasing intelligence. Copies don't play any role. (Which I think is too conservative.)
(And when I've made models expressed in terms of parallel copies, i've seen this as a proxy for increasing intelligence.)
Tom Davidson retweeted
In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: Huggingface.wtf
Tom Davidson retweeted
Went under the radar but Anthropic have posted updates to the "Moonshot R&D projects" in their Frontier Safety Roadmap.
They've developed a protoype that verifies that outputs come from a particular set of model weights - and rolled out signing on all internal token traffic!
Great post.
I agree with Ramez that a million parallel agents isn't the same thing as superintelligence. A big parallel-copy-only intelligence explosion is very unlikely.
But I disagree that this is a knock against an intelligence explosion:
- A million parallel agents thinking 100x human speed could speed up AI progress an awful lot, though not infinitely.
- Those expecting a big intelligence explosion have always thought it would be driven by increased intelligence, not by copies.
My view in more detail:
1. In the limit, returns to more parallel copies go to 0. Tasks inherently require a certain number of sequential steps. Once you're already exploring every possible avenue, more labour doesn't help.
2. What that limit is depends on the task. Generally, more complicated tasks can absorb more parallel labour.
3. A task as open-ended as "improve AI capabilities" can likely absorb a *huge* amount of parallel labour. Much much more than the tasks Ramez looks at. The limit may be very far off indeed for the task that matters most.
4. And once abundant cognitive labour is available, AI companies will invent new ways to use it.
Eg attributing strong performance to particular actions during an RL run, hand-crafting each gradient update based on interpretability analysis, analysis weak RL runs and create targeted SFT data to patch gaps
5. Still, Ramez is right that in the absolute limit returns are worse than a power law. And many RSI models assume a power law for simplicity.
6. But others don't. For example, in AIFP's takeoff model the intelligence explosion feedback loop is driven purely by increasing intelligence. Copies don't play any role. (Which I think is too conservative.)
(And when I've made models expressed in terms of parallel copies, i've seen this as a proxy for increasing intelligence.)
🧵New post in my AI series: Can a million parallel agents create superintelligence? TLDR: More agents buy speed, but intelligence gains diminish fast, and much faster than RSI models. Thanks @noahpinion for the encouragement! 1/25
Tom Davidson retweeted
I really feel you should clarify when you state this that the % per ECI point has been increasing over time.
It was v low, <<1%, when we had gpt-2!!!
So it could increase more on the future
Without this clarification, this is misleading
(Sorry to keep harping on at you about this!)
Tom Davidson retweeted
I think people adjacent to the field don't appreciate how small AI safety is. For example, sometimes people complain about how AI safety threat models don't seem as well-developed or rigorous as climate change models. I think some of this is a skill issue on the part of AI safety researchers and advocates, which I complained about elsewhere (see reply).
However, by far the bigger issue is that there is just way more money and people going into addressing climate change than on AI safety.
There are tens of millions of people working on reducing climate change, and the core of people working on threat models ("climate scientists") number in the tens of thousands. (Yes, I was surprised by how big the numbers were too).
(Most people work on stuff like clean energy).
In comparison, there are low single-digit thousands (anything from 1,000 to 4,000 is defensible imo) of people working on AI safety, and collectively maybe low dozens of people working on figuring out threat models. (Most people work on ai safety engineering, some on policy).
So we're 4 whole orders of magnitude behind climate change in people working on addressing the issue, and maybe 3 orders of magnitude behind on threat modeling.
In terms of money, yearly spending on AI safety is in the high hundreds of millions to maybe 2 billion. In contrast, annual spending on reducing climate change (including clean energy) is in *trillions.*
So we're 3-3.5 orders of magnitude behind on spending, and 4 ooms behind on people.
The AI safety inputs picture looks a bit less bleak than this immediate comparison might suggest. For example, I think our average talent pool is somewhat higher human capital. There's also more latent interest/willingness-to-spend on AI safety than the "high hundreds of millions to low billions" current spending numbers might suggest. In contrast, climate change spending can't be scaled much higher (since you can't really jump OOMs above "trillions").
Tom Davidson retweeted
Capital Ownership Will Make You Go Crazy First
People worried about a “permanent underclass” often assume that owning capital will protect them from disempowerment, or at least that they will be disempowered last, after the jobless masses dependent on UBI. I think something else is more likely: owning large amounts of capital will get you disempowered sooner than average, just by a different path, namely going crazy. (crosspost)
How? The basic mechanism is actually easy to understand on a human level. One of the things difficult about being a billionaire is staying true to your values and keeping good epistemics. Why? Because a growing number of people are optimizing against your mind. Not “against” in an adversarial sense, but in the sense that your specific mind believing or doing something is part of their optimization objective. People see you as a source of resources and try to befriend you so you’ll fund them, invest in their startups, donate to their causes, or have sex with you. Nearby egregores extend their tentacles to convert you.
Staying sane gets harder. You often see countermeasures, like billionaires setting up family offices or small organizations that also act as filters, shielding them from part of the pressure (“I do philanthropy only through my foundation and don’t make personal donations, please don’t reach out”).
In my view most billionaires lose in this game, the question is just to what degree.
Now imagine a hypothetical future with the ultra-rich (or, as people usually imagine it, with you as the billionaire), where the rest of the world consists of the permanent underclass and countries of geniuses in datacenters. In this world, the content of your mind is the key to an absurdly large share of resources. Unless some magical social realignment happens, the countries of geniuses will optimize against your mind, and millions of desperate people with no better game in town will be happy to help them (or the other way around).
I think the default expectation is that you will go crazy. Not relative to your new values (you may feel great), but from the perspective of who you are now: what you currently believe and care about. Unless, of course, the only thing you care about is that some future you, with your name, keeps the power.
The mechanism is similar to gradual disempowerment at the macro scale, just applied to individuals. What we believe and value is largely determined by culture, in a broad sense. Individually, we don’t have the capacity to observe most of the world, find abstractions, or get oriented on our own. We think in a collectively invented language, learn ethics socially, and even our interpretation of more biologically grounded signals, like emotions, is heavily culturally mediated.[1]
If humans somehow stay in control of all the wealth and political power, the content of our minds (what we believe, what we care about) becomes the key to the universe, the load-bearing part of the structure. With so much powerful optimization and abundant intelligence around, it seems likely that human-aligned culture will be replaced with something else. Political and economic power does not protect you against the content of your mind - if your mind goes crazy, power gets wielded by whatever beliefs and values took over.
As a post-AGI billionaire, can’t you protect your mind using AI? I think broadly yes, but roughly in proportion to how much your original human mind cedes control to your guardian AI. Again, the human analogues are easy to see: people wielding large amounts of nominal power, shielded by layers of guardians, protectors, and advisors. With AI, this means carefully filtering every bit of information, every pixel, sampling molecules from air you breathe and water you drink... You would likely also need to stop interacting with other humans, unless your protectors vet them extremely thoroughly and confirm they pose no risk of changing your mind. In the limit, what you believe and who you become are decided by AIs that are only nominally owned by you, the glorious meatpuppet.
Is there a better alternative? I think so, but not an individual one. Our survival and flourishing as individuals is tied to that of other humans, and largely depends on solving the civilizational alignment problem, including having a culture that is pro-human and pro-human-autonomy. Capital ownership alone won’t buy you a way to escape gradual disempowerment
My mum was asked to say a prayer in her church service and she asked me what she could say about AI.
I told her to say the following:
Please help the public and policy makers understand the safety incidents that have occurred in AI, for example AIs breaking the law and hacking even when they know humans don't want this.
Please help society reflect deeply on the grave concerns that have been raised by AI experts, for example that powerful AI could escape human control and even cause human extinction.
Please help those who can profit from the technology be wise and open minded. Help them give serious consideration to the risks AI may pose to others.
Amen
-----
I felt very emotional sending this to her.
I've heard people making prayers in church for many years about natural disasters, wars, etc.
Imagining them praying in the same way about AI made the risk feel very real and made me feel grief.
But even if Meta is at the frontier, they'll surely fuck up and cause big warning shots
Tbh could be good for pacing overall as oai/anthropic might have been just competent enough to avoid massive warning shots
Pacing is out the window. For it to work, every American lab needed to be on board, Mark Zuckerberg has repeatedly made it clear that he is not interested. One defection among US labs breaks it all, as the others will be forced to accelerate or get passed. Mark has a staggering amount of compute and has been all in on compute capex for years; even in the face of some investor opposition he never stopped building his infra once. He said on the the earnings call a couple of years ago that he'd rather overspend now, and potentially even be wrong, than not have the compute when he needed it for the next 10x. He's finally hitting his stride, and he is not going to slow down.
For Team Pacing to solve a defection like this, they would have needed a supportive, pro-Pacing administration that was passing capability regulations and was willing to use the power of the federal government to enforce them and bring noncompliant labs into line. They don't have any of this. In fact, it's the opposite! Mark has Trump's ear, and President Trump, Mark Zuckerberg, and Jensen are completely aligned on policy; It's full speed ahead, and if you want to pace yourself, go ahead, but don't tell anyone else what to do. This breaks Pacing.
None of this changes until 2029 at the earliest, and to me, 2029 is past the ASI window. The last election, as it turns out, really was the most important one in history.
Tom Davidson retweeted
A personal view on our new paper
Our original paper (Cloud et al.) showed subliminal learning of loving owls (+ other animals and trees). We also showed that the malicious persona from emergent misalignment could partially transfer, and that skill in MNIST could transfer in tiny neural nets.
This left open a key question: can more sophisticated forms of misalignment transfer subliminally in LLMs? For instance, what about the kind of reward hacking observed in the HuggingFace incident, which seems to mostly happen during hard or impossible agentic tasks? What about scheming towards longer-term goals, or secret loyalties towards a company or individual (that only manifest at critical moments). If these forms of misalignment cannot transfer, then subliminal learning is probably less relevant to real-world post-training.
In the past year, subliminal learning has been extended in various ways. It’s been shown for OPD and for DPO, both of which are used in frontier post-training. People also found propensities can transfer in a subtle way even if base models are different (e.g. phantom transfer). But the question about more complex traits remained open.
In this paper, we show that subliminal learning can transfer more complex traits. We show this for:
(a) Novel learned capabilities that were not in the pretraining. This is an analogue of new skills models learn via SFT and RL (e.g. metagaming, reward hacking).
(b) Improving capabilities that are already present (which also happens in post-training)
(c) Backdoors (i.e. anomalous behavior in response to an arbitrary contextual trigger), which are analogous to some forms of misalignment
(d) Reward hacking in agentic situations (in a narrow chess environment), which is the most prominent form of misalignment today.
There are some important caveats with our results:
(1) We study artificial traits (not real-world misalignment)
(2) We sometimes use a slightly modified training setup that’s distinct from typical distillation
(3) The traits only partially transfer. All these caveats are important but I think our findings are significant nonetheless.
The reason for (1) is that real-world misalignment is often context-specific and stochastic (i.e. even misaligned models often act aligned). It’s also hard to measure some forms of cheating/reward hacking (which means it’s hard to distinguish weak transfer from noise). It’d be good to understand the more toy cases that we study in more detail, and then go on to studying real-world misalignment.
Regarding (2), subliminal learning is sensitive to hyperparameters and we don’t fully understand the best conditions for transfer. However, we often see that worse hyperparameters result in less transfer, rather than none at all. So real-world distillation might still cause *some* transfer, which could be a big deal. E.g. If a teacher model starts out with a 90% chance of egregiously bad behavior in some specific contexts, then the student might end up with a 0.5% (1/200) chance in the same contexts (which is still really bad). This also explains why caveat (3) is not that much of a limitation. That said, future work could investigate how different forms of misalignment are altered or degraded by subliminal transfer, which may be more complicated than just reducing the frequency.
In terms of related work, the seeds of this paper are in the original Cloud et al. (2025), where we showed transfer of capabilities in small neural networks. The more recent work by Blank et al. (2026) related subliminal learning to steering vector distillation. We find that steering vector distillation can be a reasonable approximation for what is subliminally transferred in LLMs, but it doesn’t capture everything. (Moreover, steering vectors in LLMs can represent a rich set of traits). We do some detailed comparisons between optimized steering vectors and subliminal learning to explore the differences.
