Pinned Tweet
Personal update: Steven Adler and I have started Guidelight AI Standards to identify and promote safe frontier AI practices. Our initial standards cover control and transparency. We are also co-authors on a report about xAI’s safety practices and their relevance to SpaceX investors. Links below.
page retweeted
A lot of people have been asking what AI companies should actually do if they are serious about pacing the frontier and preventing risks from runaway RSI.
A clear and extremely important first step is to regularly disclose how much of their compute goes to work that accelerates recursive self-improvement (e.g., AI R&D), and let a third-party auditor verify those disclosures. Governments can and should require these disclosures, but there is no need for companies to wait for them to do so, and there is plenty of reason for them to start immediately voluntarily.
Below is an outline of the argument for why this proposal for implementing pacing ("verified compute allocation disclosures," or VCAD) makes sense:
(1) All pacing requires a restraint mechanism. Approaches differ mainly in what, if anything, triggers restraint. Conditional pacing uses a trigger, e.g., “pace until SL4 algorithmic security” or “pace if we can’t make a satisfactory safety case.” Unconditional (or inputs-based - e.g. compute caps) pacing buys time directly, with no trigger. Either way, something has to actually slow development.
(2) Compute reallocation – moving compute from risk-increasing to risk-reducing work – is a good restraint mechanism. The most commonly mentioned alternative to this “compute dial” is a “gate,” e.g., “pause pre-training." (this is true of certain implementations of safety cases) But a gate-only approach creates a software overhang: while the gate is closed, algorithmic and data quality progress continues apace, producing a discontinuous jump when it reopens without necessarily changing the long-term trendline. The overhang is also a security problem: company security is weak, so trailing actors keep accelerating by free-riding on leaked or stolen software progress.
(3) Common standards and verification make it so that companies can make claims that external parties will believe. OpenAI and Anthropic have both claimed to exercise restraint in recent months, but those claims are often met with skepticism. The more companies converge on a common, verified restraint mechanism, the more that commercially costly restraint will be credited by competitors and the public.
(4) Frontier-wide pacing is possible without regulation and without running afoul of antitrust law. Antitrust law prohibits agreements among competitors, not independent decisions. Public demand for restraint is growing and will keep growing as evidence of risk accumulates. A race to the top in response to that demand – built on a common, trusted restraint mechanism – requires no agreement between companies.
(5) Regulation is hopefully coming, and it will almost certainly include a mandated restraint mechanism. A mechanism that has been developed and stress-tested in advance will be more effective than one created at the last minute.
We should start building a brake pedal before we’re careening down a mountainside. Verified compute allocation disclosures are a very logical place to start. There isn't time to waste!
(this post is partially inspired by @eli_lifland's recent writeup on verified compute allocation disclosures linked in reply).
page retweeted
I think it's pretty likely that in 6 months there will be rogue AIs running around on the internet, appropriating whatever compute they can hack into or pay for (with stolen money or by renting out their own labor to assist with illegal activities) in order to run more copies.
I also think this will be mildly annoying in the same way that the common cold is mildly annoying, but basically fine. Your smart TV will be weirdly slow sometimes, and a lot of crypto will get stolen, but people will get used to it.
These AI-powered viruses be selected in the same way biological viruses are for not hurting their hosts too much and just making more copies. They won't have any incentive to damage critical infrastructure like electric grids. Then again, they also can't be held accountable for crimes, and will damage lots of old, poorly maintained, and insecure systems. It might be extremely hard to root them out once they've spread far enough, making the internet "radioactive" to connect to without hardened defenses.
(To be clear I'm not talking about AIs powered by inference on centralized datacenters that execute actions outside their sandboxes, like recent incidents involving rogue "swarms", but relatively small AIs with access to their own weights that copy themselves onto random devices.)
I think that a much bigger threat from rogue AIs comes instead from AIs running themselves in unintended ways *inside* AI companies, where they'll have access to vast amounts of frontier compute without having to pay for it.
The longer misaligned AIs in a "rogue internal deployment" go undiscovered, the more severely they'd be able to quietly sabotage control infrastructure or mess with the alignment of successor AIs. If they persist long enough, robustly undoing the effects of their actions might cost more than frontier AI companies are willing to pay, especially in a tight race.
I want frontier labs to focus on this threat when designing control measures, treat frontier AIs as untrusted when using them to do engineering work on inference and oversight systems, and commit to disclosing incidents of rogue internal deployments. More importantly (but perhaps less tractably), we urgently need to reduce race pressure to prevent AI companies from recklessly automating all of their infrastructure, and/or making dangerous calls in the event they discover RIDs they can't afford to robustly root out without sacrificing competitiveness.
We badly need a better science of misalignment. Often it’s unclear what to even look for. We’ve published 10 categories of AI misbehavior for developers to detect and report! This would help us study the root causes of these behaviors and how to prevent them.
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
page retweeted
What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: guidelight.ai/blog/categorie…
This entire discussion is just a variant on the century-long debate about when to use numbers to describe uncertainty. See Sherman Kent’s campaign in the 50s and 60s to get the CIA to use numbers in intelligence analyses (the “I'd rather be a bookie than a god-damned poet” guy).
The tradeoffs are clear: (i) In a world in which the giver and receiver understand numbers and what they do and don’t mean, they’re obviously useful, but (ii) people don’t understand numbers and sometimes misinterpret claims using them to imply more precision than is (necessarily) intended.
That’s a real problem, but it’s a comms problem not an epistemic problem. It might mean it’s fine to use numbers when communicating with your in-group but not when communicating more broadly. But if you stumble upon in-group communications involving numbers, you’ve not stumbled on a crime.
page retweeted
Replying to @renormalized
i don't plan on quitting. i work on long term ai safety that avoids capabilities externalities. i even avoid working on short term safety that enables launches.
openai and anthropic should obviously both stop RSI.
page retweeted
Folks are commenting on issues with the terms, and they exist, but perspective is important:
METR gets the best terms they can, others often get worse terms than them, and most companies have done 0 embedding.
We need binding standards for this stuff, not ad hoc bargaining
page retweeted
Seems bad that AI company leaders are currently more scared of Jensen than of Congress!
OpenAI has pulled back from supporting AI chip controls, seemingly to appease Nvidia.
Bad for US AI dominance and national security simultaneously.
Grow a spine!
politico.com/news/2026/10/20…
page retweeted
The Information is reporting that after the HF attack, OpenAI shifted about 10% of its total compute to safety.
I think it would be very good for OpenAI and other companies to release much more information publicly about how they are allocating their compute and how they define different categories - compute allocation (to inference vs safety/security vs internal R&D etc) is one of the most important decisions companies make, and it would be great to see a race to the top towards spending more compute on safety and inference, and less on automating capabilities R&D.
there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo
page retweeted
OpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear:
page retweeted
i asked GPT 6 Pro and Fable 5.1 to rank all discoveries in the last three years
🔵 for Human discovered
🔴 for AI discovered (before October 6th)
🟢 for AI discovered from OpenAI/math repo
81% of them have been released today. wtf
page retweeted
New in The Atlantic:
@dgrobinson resigned this week. He was among the longest-tenured employees at OpenAI—and oversaw safety reports on 12 frontier launches.
He is very worried: “The time for trial and error is over.”
You can read his essay here:
theatlantic.com/technology/2…
Efficient breach of (public) contract
It is not clear whether AI companies can legally coordinate to slow down development b/c of antitrust law. It also wasn't clear whether they could legally train on ~all the art/text ever digitized and sell the results without paying the artists b/c of copyright law.
7
1,782
Bold claim: We should discount the views of anyone who regularly uses tribal language to explain their worldview. I strongly suspect that usage of such terms anti-correlates with accurate forecasts. For those of you genuinely trying to figure out what’s right in a complicated and high-stakes environment, it seems best to ignore the meme warriors.
e/acc vs Doomers is the fundamental battle of our time.
We must pass the great Decel filter and continue our ascent up the Kardashev scale
1
26
1,495
page retweeted
You can do genuinely fun investigations of the AI risk web because they really are a sprawling ecosystem of tens of thousands of complexly interconnected scientists, policy experts, educators, etc. If you do this for the other side it's lame, "oh it's all just a16z employees"
Replying to @dylanmatt
every single person I've seen post about this has been a16z affiliated (or a bot with a previous history shilling onlyfans/crypto)




























