AI strategy researcher @BlueDotImpact, previously @Stanford. I tweet about economics, futurism, and philosophy. Views my own.
San Francisco
Joined March 2017
- Tweets816
- Following490
- Followers962
- Likes4.9K
Pinned Tweet
(This is the first tweet in a series of thoughts related to Scott Alexander’s recent piece “Against SURA.”) I support the U.S. attempting to unilaterally “pace the frontier” of AI progress.
For one, some combination of diffusion, distillation, and derisking some kinds of frontier R&D means that it would take longer for China to catch up after a slowdown or pause (the catch-up rate would probably be fairly quick early on, but then gradually slow down).
Even if China catches up (e.g., in the case of a full pause), I expect the U.S. to grow faster once it resumes progress, so I’d expect the U.S. to eventually regain much of its lead (although maintain less of a lead than it counterfactually would have had it not slowed down). So I think buying time and perhaps increased Chinese willingness for a sensible U.S.–China treaty (and better verification technology), making progress on alignment or low-stakes control research (to help elicit useful alignment research from AIs), or drafting a more sensible domestic policy regime seems worth the risk of China catching up. Labs can also use slowdowns or pauses to build up security, which might strengthen the U.S. lead relative to China more on net.
Additionally, I think the “point of no return” might be pretty soon after we get superhuman AI researchers, which in turn might be pretty soon in calendar time, and in those worlds it’d be foolish to keep racing.
Tejas Subramaniam retweeted
METR's report on AI acceleration is heavily redacted. Can we learn anything from it?
Yes. Using public data, I ask
1. is the acceleration real? (yes)
2. is the acceleration because of AI? (probably)
3. is the acceleration going to continue? (who the fuck knows)
🧵
Scrappy summary I wrote for myself to understand the Twitter discourse on the METR report (my loose impression of tweets by Ryan Greenblatt, Ajeya Cotra, and others), in case helpful for anyone else:
Suppose effective compute = training compute x software efficiency.
Let’s say growth in software efficiency (g), d log(software efficiency)/dt, is a simple equal-weight Cobb-Douglas production function A*sqrt(L)*sqrt(E), where L = effective research labor, E = experimental compute per unit time, and A = research productivity at a given software frontier. Suppose also that baseline progress in AI capabilities is two-thirds software progress and one-third training compute, so 2x overall progress requires 2.5x faster software progress (holding growth in training compute constant).
So g_(with AI)/g_base = A_(with AI)/A_base * sqrt(L_AI / L_base) * sqrt(E_AI/E_base). Holding A and E fixed, this gives us L_AI/L_base = (g_AI/g_base)^2. Hence, 2.5x faster software progress means 2.5^2 = 6.25x research labor uplift.
But accelerated progress has probably also reduced A. Once you account for this “ideas getting harder to find” effect, the implied research labor uplift could be much higher.
forethought.org/research/can…
I recommend this report by Avi Parrack for some more detailed thinking related to the thoughts I give in my tweet below (I hadn’t seen the report at the time of writing this tweet).
(1) I think a temporary pause primarily makes sense as a stopgap to implement other measures, like security and tightening export controls. My preferred operationalization of pacing would be different and have longer-lasting effects (e.g., requiring that only models of some age are used in AI R&D). I was using a pause as a simple illustration of my guess for how Chinese catch-up would work (e.g., different growth rates meaning the U.S. would rebuild its lead).
(2) I’m not sure, probably not a dramatic improvement but not completely trivial. Suppose a pause is operationalized as 100% of compute going to external inference. My central guess is that it takes about 12–15 months for China to catch up if the U.S. pauses now, so (assuming a ~6-month lead) burning 5 months of lead means a ~9–10-month pause. Assuming datacenter construction continues (so about 3x more compute is available at the end of 10 months) but research runs at maybe ~5% of normal pace (almost no experiments or coding agents), I’d guess labs’ underlying capacity is ~6 months behind by the end of the pause. Actual models would at first be further behind (due to the length of training pipelines, de-risking, etc.), peaking at ~10 months behind a few months after the pause ends. But then the gap closes (~5 months a year later, ~2.5 two years later), because the research lost in the pause becomes a falling share of the growing research stock. So for near-term capability levels, the pause buys ~10 months, but for the more dangerous levels reached a couple of years later, it buys only a few months of extra alignment/control work. I think this could be used well, but I’d rather put effort into more durable pacing efforts.
(3) The burned lead probably wouldn’t last, though. China’s catch-up is substantially fueled by the gap (e.g., distillation, diffusion), so, as noted above, it would slow as the lead becomes smaller. Once the U.S. resumes, it would outpace China’s slow near-frontier speed and the gap would reopen, until China’s copying-boosted speed matched U.S. speed again at ~about the old gap. In a simple model, I’d guess the lead bottoms out at ~1 month shortly after the pause ends, is back to ~4–5 months about half a year later, and is roughly fully back within a year (as long as there’s no model weight theft or similar).
Tejas Subramaniam retweeted
I used an AI for two hours and found a zero-day that would let a nation-state hack any server running stock Ubuntu. It was patched this morning. CVE-2026-109518. 🧵
🚨 BREAKING: A flaw in APT, the tool that installs software on Debian Linux, lets an attacker on the network tamper with a package after it passes verification and run code as root.
The bug, CVE-2026-109518, sits in resumed HTTP downloads. APT can merge two different downloads when a redirect gives them the same address, and a man-in-the-middle can then alter the file after its check.
Debian fixed it in apt 3.0.4, shipped to Debian 13 security updates on 4 November. Users should upgrade now.
Per Debian's changelog, the fix is four words long. "Open target files exclusively."
If anyone working on (1) reducing misalignment risks from powerful AIs or (2) “better futures”-flavored work wants to give me an info-dense mini-lecture on your work or your takes on AI strategy/futurism where I ask annoying questions and take scrappy notes, I’d be grateful!
Tejas Subramaniam retweeted
Conversation with someone from a chip startup on a recent flight:
Me: oh I'm at Epoch. We make basically every graph on the trajectory of AI, except one.
Them: Oh! Yeah, I've heard of you. You make that graph with how long of a task AI can do over time!
Tejas Subramaniam retweeted
the "curse of cryonics" is when a problem is both weird and very important, but it's sitting right next to other weird problems that are even more important, so everyone who's able to notice weird problems works on something else instead.
Tejas Subramaniam retweeted
Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.
Two parts that stand out as particularly infuriating when taken together:
1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.
2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.
First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.
This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??
There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.
Big news. My take: Anthropic deserves some credit for actually proposing a pretty reasonable AI safety bill; that's more than other frontier labs have done.
But they don't deserve *too* much credit, IMO; this bill is weaker than the Trahan-Obernolte FRONTIER Act in some key ways, and since the political momentum has shifted significantly in the direction of safety since FRONTIER was introduced back in July, I think new bills introduced today should ideally go further than FRONTIER was able to go rather than regressing.
To be fair to Anthropic, the main places where this bill is significantly weaker than FRONTIER are (a) rulemaking (very strong in FRONTIER, almost nonexistent in Anthropic's bill) and (b) the emergency shutdown authority. Even if you take safety seriously, it might make some sense to dial those authorities back if you didn't trust the executive branch to wield them responsibly, because those are the easiest provisions for a hypothetical President who hypothetically might not like a certain frontier AI company to abuse.
There are some improvements over FRONTIER as well; for instance, this bill does specify that auditors are employed on a continuing basis and has better provisions regarding auditor access to information. Happy to see that from Anthropic (although I think their access provision is still a little undercooked and has Fourth Amendment problems).
Overall, I think the embedded auditor bills that get introduced in the coming months are going to be significantly better and more ambitious than this, and I hope Anthropic supports those bills instead of opposing anything that treats catastrophic risks like the serious national security issue they are. Still, it's encouraging to see a frontier lab come this far.
British Parliamentary debate (the largest collegiate debate format in the world) has started taking takeover risk pretty seriously.
Tejas Subramaniam retweeted
We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵
Time to re-read “That Which Is Seen and That Which Is Not Seen.”
I just fixed my dishwasher with the help of ChatGPT. A trivial task. I had been about to order a new one. So this software has increased the country's real wealth yet decreased the measured GDP. The main economic indicator is structurally incapable of registering the thing that actually makes people better off, namely the growth of knowledge
Three thoughts on this post. First, the “endogenous response” was also partly endogenous to this pessimism. Franck, Szilard, and later Oppenheimer’s advocacy, on the margins, helped with arms control and the intellectual framework surrounding deterrence.
marginalrevolution.com/margi…
Second, as Tyler has argued elsewhere, the risk of nuclear war remains significant. Even a small probability of nuclear war in any given year means a substantial probability with enough time. In his Stanford talk, Tyler suggested this meant humanity has 5–7 centuries left.
I recently started working as a strategy researcher @BlueDotImpact. My aim is to reduce the risk of takeover by misaligned AI, because humanity isn’t on the ball yet. I’m working on figuring out what concrete projects would be best-positioned to absorb talent in this field.
Especially if you have a lot of context about AI safety, consider applying to work with us. I think figuring out how to best navigate the third wave of American philanthropy will be crucial in shaping both sensible AI policy and our best alignment plans: bluedot.org/join-us
