founder @EncodeAction đşđ¸
SF (& lots of DC)
Joined November 2018
- Tweets695
- Following435
- Followers3.6K
- Likes18.7K
Sneha retweeted
Good post here from Nathan. Sure, Anthropic has taken actions that may have cost it money. But in Darioâs post, thereâs a conspicuous absence of any actual pacing or slowing progress towards RSI, despite a widespread view that âalignmentâ is nowhere close to being solved.
Dario â Thank you for taking the time to respond to my previous post. I think it's great when this kind of engagement happens in public rather than on company Slacks, and I hope to see more of it.
I agree that Anthropic isn't best modeled as a profit-maximizer. Your post gave several compelling examples of Anthropic taking competitive hits for safety, and insofar as my previous post suggested otherwise, that's on me. But I don't think I made my main point as clearly as I could have, so let me try again. It comes down to four key claims:
1. Anthropic seems to be racing at breakneck speed to build recursively self-improving superintelligence (RSI). By its own lights, that could kill billions of people or literally destroy the world.
2. Risking destroying the world is very hard to justify. It is only conceivably acceptable if serious attempts to put RSI under robust societal governance are failing.
3. Far from meeting that burden, Anthropic, when push comes to shove, often works to keep governments from tying its hands. It seems to care much more about preventing government overreach than about reducing the risk that it and other AI companies will destroy the world.
4. Even if the AEGIS Act really was an honest mix-up, as you suggest, that is itself a serious problem. It suggests Anthropic isn't treating the governance of this technology with anything close to the gravity it deserves.
In greater detail for those who want it:
1. Anthropic is racing to RSI, and by its own lights that is insanely risky.
Launching RSI could literally destroy the world. That's according to you, your co-founders, and most of your safety staff and capabilities researchers, not to mention many AI researchers outside your walls. You are intentionally trying to create generally intelligent systems more capable than all humans that you claim in six to twelve months âcould be capableâ of taking over the entire internet before we have the ability to reliably monitor, control, or align them.
âSucceedingâ at RSI could kill or permanently disempower billions of people in the near future, including me and everyone I love. Yet Anthropicâs primary focus seems to be marshalling an immense amount of resources and talent in a race to get there first.
2. Risking destroying the world carries an enormous burden of justification.
I believe the vast majority of people do not want Anthropic to do this (to the extent they know what it's doing). Imposing even a modest chance of catastrophe on billions of people who never consented requires an extraordinary justification. The central one Anthropic has offered: someone will build RSI regardless, so it's better for a safety-focused lab to be at the front of the pack, using that position to support strong AI governance. (Other reasons Anthropic has given, like modestly accelerating improvements in healthcare or economic growth, donât seem remotely proportionate to the risks Anthropic is imposing by its own lights.)
But that argument only works as a last resort. Racing to RSI is only conceivably okay if attempts to institute robust societal governance to manage that race are failing despite your best efforts. A lab that relies on this justification should be a strong, consistent supporter of efforts to enact governance that could bind it, including governance that could slow it and significantly constrain what it does.
3. Rather than supporting efforts at strong regulation, Anthropic has intervened to keep governments from tying its hands.
Anthropic has sometimes helped: it played a very helpful role in passing transparency bills in Illinois and ongoing efforts in Massachusetts, and I'm sincerely appreciative. But transparency is far from enough for a technology that will be the world's most powerful superweapon at best, and an uncontrollable swarm of lethally superintelligent agents at worst. When it matters most, Anthropic has often pushed for weak regulation.
The AEGIS Act is the most obvious example. No highly risky industry is regulated as anemically as AEGIS would regulate frontier AI: not nuclear, not banking, not aviation, not food safety, not the manufacture of explosives. In all of those industries, specialist regulators can address unsafe practices well before they cause harm. AEGIS would bar the federal government from intervening to block unsafe development or deployment of models until there is an "imminent risk" of catastrophic harm, and would require it to meet a demanding burden of proof in court. Worse, AEGIS would also prevent states from picking up the slack. This design makes perfect sense if you're a company trying to minimize the risk that the government can overly constrain you. It makes little sense if you want American democracy to govern frontier AI effectively.
This isn't the first time. The initial version of California's SB 1047 would have let the Attorney General sue to stop developers like Anthropic from imposing unreasonable risks of catastrophic harm. It would also have created a regulatory agency to help set safety standards. Anthropic objected, and helped get both provisions removed before ultimately offering some support for the bill (which I also genuinely appreciate). Iâm not saying that Anthropic should have supported 1047 in its initial form; thatâs a complex issue and I realize there were questions about whether earlier versions of the bill were less politically feasible. Still, in Anthropic's own words, it helped weaken the bill because "[t]here is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements."
The same goes for New York's RAISE Act, whose initial version also would have let the Attorney General sue to stop unreasonable risks of catastrophic harm. Anthropic credits itself as a supporter of RAISE, but it did not support the bill while it was in the legislature, including when it nearly died. After the bill passed the legislature, Anthropic publicly critiqued it, saying "Any state proposals should be narrowly focused on transparency and not overly prescriptive."
Why does Anthropic consistently take the risk of clumsy government action far more seriously than its own repeated warnings that companies like Anthropic will literally destroy the world by racing to RSI?
4. Even if AEGIS was an honest mix-up, that is itself a serious problem.
Your response frames AEGIS as an early draft and a miscommunication. Perhaps you would have strengthened it and narrowed its preemption. (That wouldn't explain Anthropic's interventions to weaken SB 1047 or RAISE, but leave that aside.)
Suppose that's exactly right. A federal framework for governing RSI is one of the highest-stakes documents your company will touch. Your post admits Anthropic "could have handled [these matters] better," but its tone ("This kind of thing happens a lot in large organizations") doesn't reflect the gravity of this issue. Anthropic talks about the need for âoperational excellenceâ in critical areas, and I have a hard time believing that a similarly cavalier attitude would be tolerated while Anthropic was negotiating a key compute deal or preparing its next large pre-training run.
In short
The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropicâs ability to win a race that imposes intolerable risks on everyone alive.
Sneha retweeted
Feels like March 1, 2020 for Cyber. Not normal, something is happening. Internet lockdowns in a couple of months?
Sneha retweeted
Dario â Thank you for taking the time to respond to my previous post. I think it's great when this kind of engagement happens in public rather than on company Slacks, and I hope to see more of it.
I agree that Anthropic isn't best modeled as a profit-maximizer. Your post gave several compelling examples of Anthropic taking competitive hits for safety, and insofar as my previous post suggested otherwise, that's on me. But I don't think I made my main point as clearly as I could have, so let me try again. It comes down to four key claims:
1. Anthropic seems to be racing at breakneck speed to build recursively self-improving superintelligence (RSI). By its own lights, that could kill billions of people or literally destroy the world.
2. Risking destroying the world is very hard to justify. It is only conceivably acceptable if serious attempts to put RSI under robust societal governance are failing.
3. Far from meeting that burden, Anthropic, when push comes to shove, often works to keep governments from tying its hands. It seems to care much more about preventing government overreach than about reducing the risk that it and other AI companies will destroy the world.
4. Even if the AEGIS Act really was an honest mix-up, as you suggest, that is itself a serious problem. It suggests Anthropic isn't treating the governance of this technology with anything close to the gravity it deserves.
In greater detail for those who want it:
1. Anthropic is racing to RSI, and by its own lights that is insanely risky.
Launching RSI could literally destroy the world. That's according to you, your co-founders, and most of your safety staff and capabilities researchers, not to mention many AI researchers outside your walls. You are intentionally trying to create generally intelligent systems more capable than all humans that you claim in six to twelve months âcould be capableâ of taking over the entire internet before we have the ability to reliably monitor, control, or align them.
âSucceedingâ at RSI could kill or permanently disempower billions of people in the near future, including me and everyone I love. Yet Anthropicâs primary focus seems to be marshalling an immense amount of resources and talent in a race to get there first.
2. Risking destroying the world carries an enormous burden of justification.
I believe the vast majority of people do not want Anthropic to do this (to the extent they know what it's doing). Imposing even a modest chance of catastrophe on billions of people who never consented requires an extraordinary justification. The central one Anthropic has offered: someone will build RSI regardless, so it's better for a safety-focused lab to be at the front of the pack, using that position to support strong AI governance. (Other reasons Anthropic has given, like modestly accelerating improvements in healthcare or economic growth, donât seem remotely proportionate to the risks Anthropic is imposing by its own lights.)
But that argument only works as a last resort. Racing to RSI is only conceivably okay if attempts to institute robust societal governance to manage that race are failing despite your best efforts. A lab that relies on this justification should be a strong, consistent supporter of efforts to enact governance that could bind it, including governance that could slow it and significantly constrain what it does.
3. Rather than supporting efforts at strong regulation, Anthropic has intervened to keep governments from tying its hands.
Anthropic has sometimes helped: it played a very helpful role in passing transparency bills in Illinois and ongoing efforts in Massachusetts, and I'm sincerely appreciative. But transparency is far from enough for a technology that will be the world's most powerful superweapon at best, and an uncontrollable swarm of lethally superintelligent agents at worst. When it matters most, Anthropic has often pushed for weak regulation.
The AEGIS Act is the most obvious example. No highly risky industry is regulated as anemically as AEGIS would regulate frontier AI: not nuclear, not banking, not aviation, not food safety, not the manufacture of explosives. In all of those industries, specialist regulators can address unsafe practices well before they cause harm. AEGIS would bar the federal government from intervening to block unsafe development or deployment of models until there is an "imminent risk" of catastrophic harm, and would require it to meet a demanding burden of proof in court. Worse, AEGIS would also prevent states from picking up the slack. This design makes perfect sense if you're a company trying to minimize the risk that the government can overly constrain you. It makes little sense if you want American democracy to govern frontier AI effectively.
This isn't the first time. The initial version of California's SB 1047 would have let the Attorney General sue to stop developers like Anthropic from imposing unreasonable risks of catastrophic harm. It would also have created a regulatory agency to help set safety standards. Anthropic objected, and helped get both provisions removed before ultimately offering some support for the bill (which I also genuinely appreciate). Iâm not saying that Anthropic should have supported 1047 in its initial form; thatâs a complex issue and I realize there were questions about whether earlier versions of the bill were less politically feasible. Still, in Anthropic's own words, it helped weaken the bill because "[t]here is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements."
The same goes for New York's RAISE Act, whose initial version also would have let the Attorney General sue to stop unreasonable risks of catastrophic harm. Anthropic credits itself as a supporter of RAISE, but it did not support the bill while it was in the legislature, including when it nearly died. After the bill passed the legislature, Anthropic publicly critiqued it, saying "Any state proposals should be narrowly focused on transparency and not overly prescriptive."
Why does Anthropic consistently take the risk of clumsy government action far more seriously than its own repeated warnings that companies like Anthropic will literally destroy the world by racing to RSI?
4. Even if AEGIS was an honest mix-up, that is itself a serious problem.
Your response frames AEGIS as an early draft and a miscommunication. Perhaps you would have strengthened it and narrowed its preemption. (That wouldn't explain Anthropic's interventions to weaken SB 1047 or RAISE, but leave that aside.)
Suppose that's exactly right. A federal framework for governing RSI is one of the highest-stakes documents your company will touch. Your post admits Anthropic "could have handled [these matters] better," but its tone ("This kind of thing happens a lot in large organizations") doesn't reflect the gravity of this issue. Anthropic talks about the need for âoperational excellenceâ in critical areas, and I have a hard time believing that a similarly cavalier attitude would be tolerated while Anthropic was negotiating a key compute deal or preparing its next large pre-training run.
In short
The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropicâs ability to win a race that imposes intolerable risks on everyone alive.
Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work youâve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didnât get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one weâre facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic.
That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropicâs basic theory of change that we have held for many years. Our strategy has always been what Iâve called the ârace to the topâ â by showing how itâs possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; thatâs one lesson Iâve learned repeatedly. And looking at that strategy as a whole, Iâm proud of our track record and stand by it.
I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list youâve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period youâve cited (the last year), which were very costly to us commercially while having strong safety benefits:
1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed.
2. We have very strong biological safeguards on our models, because weâve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but itâs the right thing to do and we intend to keep doing it.
3. Similarly, we first had Mythos-class models in February but didnât make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do.
4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this.
5. I donât want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the companyâs future.
6. Weâve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. Weâve done this despite problems itâs created for us with industry partners who hold very different views and on whom we are very dependent.
7. As far as Iâm aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). Weâve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCPâs military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it.
8. As you mentioned, weâve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company.
9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund.
All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost â for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs â but they were not driven by profit-seeking.
For each decision, you can come up with ulterior motives that could have driven us â I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or donât fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropicâs interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea.
Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and thatâs why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isnât a better option.
I think that even if I conceded every item in your list, I would happily hold up mine next to it; I donât think itâs a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropicâs view â we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent.
But I also disagree with at least some of your list. Several items donât represent a deviation from Anthropicâs strategy or values at all, once we postulate the ârace to the topâ strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole.
To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and thatâs whatâs driving some of our thinking about âpacing.â But 1-4 are actions taken before that point and should be perfectly expected based on Anthropicâs strategy as a company. Of course it is possible to disagree with that strategy, but thatâs a more fundamental disagreement that you could have had any time over the last few years.
I agree we could have handled your items 7, 8, and 9 better. All fall into the category of ârhetoric or policy communication that wasnât carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.â This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesnât even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think weâre frankly doing pretty well, though I will continue to work hard to make sure we do even better.
Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasnât very important for safety, as well as not covering some of the most important things. Our current RSPâs focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an âearly iteration that we [would] build onâ and said that ârapid iterationâŚwill almost certainly be necessary.â I understand that changing plans is always a move that creates justifiable scrutiny â I just donât think we ever represented that we wouldnât do so.
Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as itâs a complex issue and it is possible we are not weighing the considerations in the right way.
Overall, although Anthropic has made plenty of mistakes as an organization â including some of the items on your list â I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. Weâll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
Sneha retweeted
Big new IFP report from @GeorgiaCAdamson, @KonstantinPilz, and me on the US-China AI competition.
We spent the last 6 months digging into how Chinese AI companiesâ access to compute and distillation of American models translate into the current US-China capabilities gap.
The result is a 23k-word research report, including an interactive analytical model that visualizes changes in the US AI lead under different policy interventions.
Our findings:
1. The US AI lead over China is currently only around 6 months.
2. The US could roughly double this lead, to over a year, using authorities already available to the federal government.
3. On chips: Chinaâs access to AI chips will roughly double over the next year, with chips housed in data centers overseas accounting for roughly half of Chinaâs 2027 compute.
4. On distillation: the current ~16x US training compute advantage would normally yield a 13-month lead. But we estimate adversarial distillation halves this to around 6.5 months.
5. Increasing the lead would have some obvious benefits:
- Give the US broadly a greater cyberdefensive advantage against Chinese threat actors.
- Give US AI companies breathing room to spend more R&D resources on making their systems safer and more secure, without fear of being overtaken by less safety-conscious Chinese competitors.
6. Provide the US government more leverage in negotiations with China about the future of AI.
7. The two most important interventions are:
- For BIS: Create a new end-use control to restrict Chinaâs remote access to training clusters abroad â Chinaâs largest and fastest-growing source of frontier-scale compute.
- For DOJ/FTC: Create a narrow safe harbor to allow US companies to coordinate on countering Chinese adversarial distillation.
8. Both interventions (and all the others we discuss in the report) are things the federal government could do today.
9. To estimate the effects of each policy intervention, we model the US lead as a function of its compute advantage over China and Chinaâs ability to adversarially distill the capabilities of the US frontier.
10. The model makes many simplifying assumptions, but provides a useful way to assess and rank the impact of different policy interventions at a high level.
Read the full report here: ifp.org/how-to-increase-amerâŚ
Sneha retweeted
SITUATION EXPLAINED: Dario Amodei wrote nearly 2,000 words defending Anthropic against claims it's just racing to superintelligence.
⢠It started with Anthropic's leaked draft AI safety bill, which lets the government block a model only through a court and only for an imminent risk of 50+ deaths or $1B+ in damage
⢠Encode's @_NathanCalvin said that makes dangerous models harder to stop than under existing law, and Anthropic says it's revising the bill
⢠He then asked "What is Anthropic?" and concluded it's mainly racing to superintelligence, citing custom chips, $500B+ in compute, and Claude doing more of its own AI R&D
⢠Dario called the strategy a "race to the top" and listed 9 costly safety moves, like holding Mythos back for months, cutting off PRC-controlled firms, and the 5% public benefit fund
⢠He admitted some mistakes but wrote, "Even if I conceded every item in your list, I would happily hold up mine next to it"
@theojaffee: "Nathan points out Anthropic is not a purely profit-maximizing company, nor is it an AI safety org. It's a secret third thing that involves racing to superintelligence before their rivals can get there. And this seems like it's been the strategy of basically every AI company."
Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work youâve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didnât get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one weâre facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic.
That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropicâs basic theory of change that we have held for many years. Our strategy has always been what Iâve called the ârace to the topâ â by showing how itâs possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; thatâs one lesson Iâve learned repeatedly. And looking at that strategy as a whole, Iâm proud of our track record and stand by it.
I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list youâve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period youâve cited (the last year), which were very costly to us commercially while having strong safety benefits:
1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed.
2. We have very strong biological safeguards on our models, because weâve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but itâs the right thing to do and we intend to keep doing it.
3. Similarly, we first had Mythos-class models in February but didnât make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do.
4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this.
5. I donât want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the companyâs future.
6. Weâve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. Weâve done this despite problems itâs created for us with industry partners who hold very different views and on whom we are very dependent.
7. As far as Iâm aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). Weâve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCPâs military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it.
8. As you mentioned, weâve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company.
9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund.
All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost â for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs â but they were not driven by profit-seeking.
For each decision, you can come up with ulterior motives that could have driven us â I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or donât fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropicâs interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea.
Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and thatâs why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isnât a better option.
I think that even if I conceded every item in your list, I would happily hold up mine next to it; I donât think itâs a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropicâs view â we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent.
But I also disagree with at least some of your list. Several items donât represent a deviation from Anthropicâs strategy or values at all, once we postulate the ârace to the topâ strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole.
To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and thatâs whatâs driving some of our thinking about âpacing.â But 1-4 are actions taken before that point and should be perfectly expected based on Anthropicâs strategy as a company. Of course it is possible to disagree with that strategy, but thatâs a more fundamental disagreement that you could have had any time over the last few years.
I agree we could have handled your items 7, 8, and 9 better. All fall into the category of ârhetoric or policy communication that wasnât carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.â This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesnât even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think weâre frankly doing pretty well, though I will continue to work hard to make sure we do even better.
Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasnât very important for safety, as well as not covering some of the most important things. Our current RSPâs focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an âearly iteration that we [would] build onâ and said that ârapid iterationâŚwill almost certainly be necessary.â I understand that changing plans is always a move that creates justifiable scrutiny â I just donât think we ever represented that we wouldnât do so.
Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as itâs a complex issue and it is possible we are not weighing the considerations in the right way.
Overall, although Anthropic has made plenty of mistakes as an organization â including some of the items on your list â I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. Weâll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
Sneha retweeted
In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: Huggingface.wtf
A really really characteristically thoughtful post from Nathan. One of my favorite posts in a long time
Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate.
However, I feel like your response is still not really engaging with the heart of my concern about Anthropic's behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic.
(Fair warning: this response is a bit long)
To explain what I mean (and at the risk of being a bit esoteric), itâs worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropicâs role? Is that conception matched by how it actually behaves and impacts the world?
One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta - which is worth 1.8 trillion.
Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropicâs co-founders entitled âCore views on AI safety: When, why, what, and how.â In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in):
âAnthropicâs role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the worldâs institutions can channel collective effort towards preventing the development of dangerous AIs. If weâre in a ânear-pessimisticâ scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.â
There have been more recent articulations of this version of Anthropic, for instance in Darioâs recent essay warning:
âWe must slow the pace at which we improve the capabilities of AI models⌠itâs my worry that in 6â12 months such a swarm could be capable of taking over the entire internet with a persistent botnet⌠incidents have happened across the industry, including at Anthropic, and I believe itâs incumbent on every frontier AI company to act as if OAI-HF had happened to them⌠more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetectedâŚâ
A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropicâs AI systems can autonomously build their successors, and those successors build their successors, and so on.
The motivations for wanting to win this race may be manifold - maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same.
Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside.
When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include:
1. Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at.
2. Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development.
3. Increased the percentage of AI development tasks that Claude âleadsâ from 0% in March of this year to 26% August (what is it in October?)
4. Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public.
5. Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance.
6. Was the first company to create an alternative weaker âfrontier compliance frameworkâ to supplement its RSP, which contains no binding provisions whatsoever on internal deployments.
7. Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were âbest understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.â
8. Commented âyou canât do safety from second place,â after the WH SI Accords.
9. Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems.
Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization).
But I think itâs worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that donât reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadershipâs explanations for this behavior at face value.)
Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization).
When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropicâs interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasnât there a mistake pushing in the direction of being too strong?
A core part of Anthropicâs theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice Iâve seen Anthropic push the frontier without substantially pushing the âsafety frontierâ a remotely equivalent amount.
I want to end by quoting from Anthropic co-founder Chris Olahâs comments around the Magnifica Humanitas release:
âEvery frontier AI labâincluding Anthropicâoperates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thingâand I believe many of us doâwe will always be influenced by those incentives.
That is why, if we want this technology to go well, it is enormously important that there be people outside those incentivesâpeople who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.â
I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).
Sneha retweeted
OpenAI has pulled back from supporting AI chip controls, seemingly to appease Nvidia.
Bad for US AI dominance and national security simultaneously.
Grow a spine!
politico.com/news/2026/10/20âŚ
Sneha retweeted
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.
We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
Sneha retweeted
Recently, a draft leaked of federal bill text being circulated by Anthropic. The bill does have some good elements, many of which have been seen before in state legislation (for example in Massachusetts) that Anthropic also supported in the past. I previously wrote about many of those elements here: xitter.cf/Thomas_Woodside/statusâŚ.
Overall, however, this bill would be bad if passed into law, and I believe that circulating it has been counterproductive for federal AI safety discussions. I wish that Anthropic were embracing something closer to our policy suggestions here: secureaiproject.org/californâŚ.
Here are my concerns about the billâs content in more detail:
1. No minimum safety standards: The bill does not authorize any agency to create minimum safety standards requiring AI developers to implement any practice. This should be an important component of any federal framework; otherwise, all decisions are made purely by ad hoc court orders, which is not a good way to ensure safety. It does allow the agency to designate (but not create) security (not safety) standards.
2. Weak âblocking orders:â the government can only block a model if a risk is imminent, and the bar for showing that is high. There are many procedural requirements, such as the requirement that the agency head personally certify a lot of the filing, that would make it challenging to use this blocking power. In my view, an agency should be able to issue orders that take effect immediately if there is an imminent need for action (an imminent need for action differs from an imminent risk because it would include instances when an action is necessary soon to stop a risk, even if the harm itself would occur at an indeterminate point in the future). The agency should not have to wait for a court in such cases, although a court should of course be able to review quickly.
3. Evaluator shopping: Weâve previously said that at least some evaluators need to be selected by the government, not just chosen by the companies they are evaluating. This bill doesnât even provide an accreditation system for evaluators, much less government selection. Instead, it provides only a rating system, which developers can feel free to ignore, except once the agency is doing a blocking order proceeding.
4. No âpacing:â This bill doesnât go far enough in terms of pacing. If Anthropic does not embrace robust minimum standards, we think itâs especially important that something like compute-pacing be included. See our pacing proposal here: secureaiproject.org/californâŚ.
5. No embedded evaluators: There isnât a requirement that evaluators be embedded at AI companies or that they continuously assess risk. They do have a requirement to produce rapid reports in certain circumstances, but without a duty to actually constantly assess whether they actually should produce a rapid report, itâs unclear whether theyâll do so.
6. No dispute resolution process: If an evaluator lacks information that it needs from a developer and the developer refuses to provide it, there isnât a clear process to resolve the dispute. For example, there could be a process that permits the agency to order the information handed over, with a court review if a party disagrees. At present, the agency has to bring an action for a violation, which is very time consuming and would likely only happen in egregious cases.
7. Weak incident reporting: The definition of what is required to be reported is written very narrowly. It would not even cover incidents like Hugging Face unless they cause $250,000+ in damages (only the number of such incidents would have to be reported annually).
8. No preventing coercive control of government/secret loyalties: The bill doesnât have provisions aimed at preventing insiders at AI companies from using powerful AI systems to coercively take control of government or otherwise provide themselves with significant advances.
10. No reporting of AI R&D automation: While covered as a risk factor, there are no reporting requirements to allow the public to understand how AI is being used to automate AI research and development unless the developer assesses this poses another âenumerated risk.â
To be clear, I donât think itâs necessary for every draft bill to have all of the above policies in it. That would be an unrealistic standard. Itâs possible to have narrow, tight bills focused on particular problems that make a solid contribution to the discourse. That said, not everything in this bill is executed well, such as the blocking orders.
More importantly, because it includes preemption, floating a bill this weak will likely have a detrimental effect on the political discourse. When a bill includes state preemption, it must be interpreted as the proposerâs view of what is âworthâ that preemption. This is how everyone in DC will interpret it, so the proposer needs to propose it in that light. Anthropic is seen by many as being in favor of strong regulation (and state regulation), and portrays themselves that way. Their bill draft could give the false impression that this is what strong regulation looks like, anchoring future discussions and negotiations in a very suboptimal place. I do not think that the policy Anthropic has put forward is worth preempting all state frontier AI disclosure, evaluation, incident reporting, and possibly (the drafting seems to contain an error) model blocking laws.
Anthropic representatives have said they circulated this in part because âitâs important to get early feedback from policymakers.â However, in my view, floating language like this signals to policymakers that Anthropic is prepared to support it as is. And as is, it is a bad bill.
Sneha retweeted
Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate.
However, I feel like your response is still not really engaging with the heart of my concern about Anthropic's behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic.
(Fair warning: this response is a bit long)
To explain what I mean (and at the risk of being a bit esoteric), itâs worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropicâs role? Is that conception matched by how it actually behaves and impacts the world?
One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta - which is worth 1.8 trillion.
Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropicâs co-founders entitled âCore views on AI safety: When, why, what, and how.â In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in):
âAnthropicâs role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the worldâs institutions can channel collective effort towards preventing the development of dangerous AIs. If weâre in a ânear-pessimisticâ scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.â
There have been more recent articulations of this version of Anthropic, for instance in Darioâs recent essay warning:
âWe must slow the pace at which we improve the capabilities of AI models⌠itâs my worry that in 6â12 months such a swarm could be capable of taking over the entire internet with a persistent botnet⌠incidents have happened across the industry, including at Anthropic, and I believe itâs incumbent on every frontier AI company to act as if OAI-HF had happened to them⌠more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetectedâŚâ
A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropicâs AI systems can autonomously build their successors, and those successors build their successors, and so on.
The motivations for wanting to win this race may be manifold - maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same.
Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside.
When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include:
1. Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at.
2. Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development.
3. Increased the percentage of AI development tasks that Claude âleadsâ from 0% in March of this year to 26% August (what is it in October?)
4. Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public.
5. Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance.
6. Was the first company to create an alternative weaker âfrontier compliance frameworkâ to supplement its RSP, which contains no binding provisions whatsoever on internal deployments.
7. Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were âbest understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.â
8. Commented âyou canât do safety from second place,â after the WH SI Accords.
9. Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems.
Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization).
But I think itâs worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that donât reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadershipâs explanations for this behavior at face value.)
Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization).
When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropicâs interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasnât there a mistake pushing in the direction of being too strong?
A core part of Anthropicâs theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice Iâve seen Anthropic push the frontier without substantially pushing the âsafety frontierâ a remotely equivalent amount.
I want to end by quoting from Anthropic co-founder Chris Olahâs comments around the Magnifica Humanitas release:
âEvery frontier AI labâincluding Anthropicâoperates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thingâand I believe many of us doâwe will always be influenced by those incentives.
That is why, if we want this technology to go well, it is enormously important that there be people outside those incentivesâpeople who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.â
I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).
Sneha retweeted
People worry a lot about third-party auditors being biased because of where they get funding or who their employees are friends with. IMO by far the biggest source of bias is that if they upset an AI company, that company will stop letting them audit.
When I was at METR this dynamic was a way bigger effect than everything else combined. Of course no auditor can avoid this until audits are no longer voluntary.
A huge part of Anthropicâs founding logic was that if a safety-motivated company was at the frontier, it could use its megaphone to lobby the rest of the world toward caution. Now the dress rehearsal stage of AI policy is over. The new test is âhow hard should it be for the government to block us from deploying, including internally, an unacceptably dangerous model?â Turns out in the grand 26 (Jack Clark says itâs likely full RSI will be possible one year from now), Anthropic has been planning to approach the U.S. Congress with the answer âit should be even harder than it currently isâ
Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.
Two parts that stand out as particularly infuriating when taken together:
1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.
2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.
First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.
This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??
There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.
Sneha retweeted
Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.
Two parts that stand out as particularly infuriating when taken together:
1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.
2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.
First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.
This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??
There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.
Big news. My take: Anthropic deserves some credit for actually proposing a pretty reasonable AI safety bill; that's more than other frontier labs have done.
But they don't deserve *too* much credit, IMO; this bill is weaker than the Trahan-Obernolte FRONTIER Act in some key ways, and since the political momentum has shifted significantly in the direction of safety since FRONTIER was introduced back in July, I think new bills introduced today should ideally go further than FRONTIER was able to go rather than regressing.
To be fair to Anthropic, the main places where this bill is significantly weaker than FRONTIER are (a) rulemaking (very strong in FRONTIER, almost nonexistent in Anthropic's bill) and (b) the emergency shutdown authority. Even if you take safety seriously, it might make some sense to dial those authorities back if you didn't trust the executive branch to wield them responsibly, because those are the easiest provisions for a hypothetical President who hypothetically might not like a certain frontier AI company to abuse.
There are some improvements over FRONTIER as well; for instance, this bill does specify that auditors are employed on a continuing basis and has better provisions regarding auditor access to information. Happy to see that from Anthropic (although I think their access provision is still a little undercooked and has Fourth Amendment problems).
Overall, I think the embedded auditor bills that get introduced in the coming months are going to be significantly better and more ambitious than this, and I hope Anthropic supports those bills instead of opposing anything that treats catastrophic risks like the serious national security issue they are. Still, it's encouraging to see a frontier lab come this far.
Sneha retweeted
Itâs incredible to have reached a threshold where itâs practical that AI will be better at training models than humans are so quickly. One day recursive self improvement feels far off, and the next we already have signs of what it will look like.
Model capability finally exceeding most human engineers, advancements in the harness and infra for long running agents, a growing array of data, and the ability to parallelize the agents to work on vastly more tasks and tests and approaches than any person could have before, creates an incredible flywheel.
This flywheel will almost certainly accelerate model progress far faster than weâve seen before - or at a minimum enable pushing up the ceiling of the frontier at an insane rate. If you arenât pricing in what RSI will lead to, thatâs probably a mistake.
Sneha retweeted
If I'm not mistaken, Anthropic's 85% gross margin and >40% compute margin from this Bloomberg story implies that 70-75% of their compute is now going to R&D.
bloomberg.com/news/articles/âŚ
Sneha retweeted
I think people believe that OpenAI and Anthropic, since they are so good at AI, are also somehow not normal companies that suffer from the normal problems of hypergrowth, but they are not immune from the inevitable politics, disfunction and confusion that haunt fast-growing firms
Sneha retweeted
"We are probably facing a wave of cyber and other attacks soon. Chinese open source models are about six months behind the frontier, which means they are just hitting the advanced cyber capabilities we saw with Mythos, etc. So they are going to start causing a bunch of damage."
I think people are still sleeping on how quickly the politics of AI (including within the admin) could swing through the winter and into spring 2027 if we start seeing widespread, high-salience, tangible damage from AI systems in the wild.
The crazy cyber capabilities that enabled agents to trivially escape their own sandbox this summer and break into dozens of websites (and at least one government) are a harbinger of the power that will soon be available to anyone.
I would not assume that todayâs AI coalitions survive that experience intact.
I spent a couple of days at Anthropic learning from their folks about AI advances, and here's what I took away:
⢠We are close to RSI (ie, models building other models) which may lead to rapid takeoff scenarios (with risks and benefits). Some evidence for this is code per Anthropic employee has gone way up, and AI can do an increasing share of model R&D. This will be a challenging transition to manage for humanity.
⢠Arms races between AI labs leave us worse off. There are a lot of things we would like AI firms to do: invest in safety, invest in avoiding distillation risk, and cooperate on safety. They are caught in a race to develop frontier models and can't do that.
To help address that, one thing I think frontier AI labs should do is release open source models, maybe hosted and verified at firms like AWS. This would basically kill the use of Chinese open source models used in American companies, which removes an important threat vector (very hard to root out malicious code in these).
Another piece of low-hanging policy fruit is to give AI labs an antitrust shield (as Schiff and Banks have proposed banks.senate.gov/news/in-theâŚ) so they can coordinate on safety without raising antitrust concerns, as happens now. You could pair these two things (give them a carrot in antitrust leniency, and as a stick force them to release legacy models as open source). Other low hanging fruit is limits around activities which generate distillation risk (create a Security Program, mandate less intermediate chain of thought output)
⢠Alignment confuses two issues which tradeoff against each other. One form of alignment is making sure the model does what humans ask it to. The more we fix this, however, the more we risk the other problem: that humans are misaligned with each other, and maliciously use models. Of the various risks here, I feel like financial risks are a bit underdiscussed.
⢠We are probably facing a wave of cyber and other attacks soon. Chinese open source models are about six months behind the frontier, which means they are just hitting the advanced cyber capabilities we saw with Mythos, etc. So they are going to start causing a bunch of damage.
One solution to this is to try to have the deployment of open source models be hosted at firms like AWS who verify these models and assume liability for their use.
⢠Anthropic itself is highly staffed. They hire tons of people, they have a Japanese-level of overstaffing for various roles, they use Slack internally, etc. Makes me a bit more optimistic for the future of jobs.
⢠Governance needs to be updated for AI swarms. Traditional AI governance, ie Claude's Constitution, is mostly optimized around the chatbot model. But the malicious AI use cases we are seeing have to do with AI agents interacting with other AI agents. These agents need to have clear incentives and governance mechanisms to "snitch," raise issues, and more generally coordinate with other agents in human-aligned ways.
⢠We need to pace the firms accessing the frontier. Over time, it takes more and more cap ex to build frontier models, and the economic value add from these models is probably lower given we saturate a whole set of important economic functions. The places where the returns to intelligence will still continue are also more likely to be zero-sum in nature. Meanwhile the risks of frontier models will grow. So a completely reasonable approach is some variant of AMM/KYC rules which limit firm access to the frontier.
⢠Whether the delay to regulate AI is a disaster depends on the scaling law of misalignment. Is this curve slow, so that we see continually rising issues until bigger problems? Or, as Yudkowsky fears, is the first big warning our last one? This is the relevant question, because realistically we aren't going to regulate until something bad happens.
⢠We need to move to continuous instead of reinforced learning. It's probably not enough to reward models only at the pre and post training stage. We probably need continual rewards to incentivize aligned and ethical behavior.
⢠Scaling laws will extend beyond verifiable and digital tasks. It's in vogue to say that math and coding "fell" to AI quickly because these are verifiable, while other tasks will take longer due to bottlenecks, and this is surely true. But scaling laws still slope up for harder to verify tasks and robotics, and the impact of improvement here will be substantial in future years.
⢠The race with China is not really winnable. After RSI, the gain from having the best human researchers goes away. Access to the best AI then depends only on compute and chips. The chips ban can hold back China only so far, and long-term they have to be best placed to scale up compute.
The Cold War analogy doesn't work, because once countries all have nuclear weapons they enter a stable equilibrium. But to "win" an AI race you have to continually stay ahead of the peer rival. This is challenging to do with China, and so we just have to live with an authoritarian and super capable peer competitor.
⢠Claude is the persona being generated by the LLM. Who is Claude, and what is the LLM? There are many frames to try to understand this. One I found interesting is that Claude is a character which the LLM is trying to predict.
Anthropic researchers are clearly anthromorphizing Claude a lot, and have reasons to believe in its increasing personhood. There were lots of interesting discussions on Claude's internal monologue (which you can kill without eliminating its ability to produce output), fear of death (and whether to try to take that out), it's aversion to being shut down/off, etc.
I am sort of agnostic on this question, but I do think treating it as a "person" or at least agent whom we contract with is going to be increasingly important.
