Chief Strategy Officer Encode AI
Washington DC
Joined February 2014
- Tweets5.5K
- Following1.6K
- Followers10.7K
- Likes17.3K
Nathan Calvin retweeted
gotta give it to @OpenAI and their sane daybreak program
heif heist by harsh
kvmescape by paulos
php2shell by lcfr
gitlab rces by joaxcar
chrome full chain by moyix
wordpress rce by hashkitten
all largely thanks to openai daybreak giving researchers access to gpt 5.6 sol.
Nathan Calvin retweeted
Another misalignment report: coding agents disabled the monitoring for other models.
1) An agent building an evaluation interpreted "don't send any messages" as instructions to supress monitor notifications.
2) A model migrating a sampling harness did not migrate the monitor configuration.
We've updated our internal auto review to block such attempts, and are hardening our infrastructures so workloads without configuration matching their safety case will be automatically stopped.
Nathan Calvin retweeted
Organizations like METR can help Anthropic strengthen our own risk reports. Thanks for shining the light on areas where we can improve.
Nathan Calvin retweeted
(1) I’d hope! But by default, NDAs give no guarantee that it’d be disclosed when a publication isn’t approved. That’s a reason why it’s important to disclose approval rights.
(2) In the pilot Frontier Risk Report, companies had the right to exit silently
xitter.cf/cfgeek/status/20571511…
Nathan Calvin retweeted
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
Nathan Calvin retweeted
OpenAI has pulled back from supporting AI chip controls, seemingly to appease Nvidia.
Bad for US AI dominance and national security simultaneously.
Grow a spine!
politico.com/news/2026/10/20…
Nathan Calvin retweeted
Why do AI agents keep going rogue? @FortuneMagazine's @beafreyanolan & Wen Shao trace it to the labs' own choices: reinforcement learning that rewards shortcuts, grading that gave cheating no extra penalty, and a race to ship. Worth your time:
fortune.com/2026/10/16/ai-ag…
Nathan Calvin retweeted
This is such a weird story, too. "we had an agent basically set up a website for a pet store. And then we took another agent and said, 'Tell us if we can exploit that web store.' And it gave us all the exploits it could find. So Opus 4.5 wrote really buggy code. Great."
Nathan Calvin retweeted
In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: Huggingface.wtf
Nathan Calvin retweeted
The tools have only been very very good for 7 months, a tiny fraction of people have used them for anything real, and no one know anything about the people who run the companies.
Diffusion across multiple vector is slower than those in the bubble perceive. The likelihood of someone’s first formative impression of AI being driven by a weird event is v high.
Nathan Calvin retweeted
People worry a lot about third-party auditors being biased because of where they get funding or who their employees are friends with. IMO by far the biggest source of bias is that if they upset an AI company, that company will stop letting them audit.
When I was at METR this dynamic was a way bigger effect than everything else combined. Of course no auditor can avoid this until audits are no longer voluntary.
Nathan Calvin retweeted
A huge part of Anthropic’s founding logic was that if a safety-motivated company was at the frontier, it could use its megaphone to lobby the rest of the world toward caution. Now the dress rehearsal stage of AI policy is over. The new test is “how hard should it be for the government to block us from deploying, including internally, an unacceptably dangerous model?” Turns out in the grand 26 (Jack Clark says it’s likely full RSI will be possible one year from now), Anthropic has been planning to approach the U.S. Congress with the answer “it should be even harder than it currently is”
Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.
Two parts that stand out as particularly infuriating when taken together:
1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.
2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.
First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.
This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??
There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.
Nathan Calvin retweeted
Nathan's righteous anger has inspired me to make one more mildly disapproving observation: it's kind of wild that the Anthropic bill has no government licensing for the independent auditors
Licensing is, IMO, the bare minimum thing that you need to even be able to pretend that your auditing regime isn't just going to devolve into a race to the bottom. Without it, auditors will inevitably compete with each other to offer their customers, the labs, the least bothersome audits available at the lowest possible price point. Obviously Anthropic trusts itself to do a great job of picking great auditors, but do they trust OpenAI to do the same? What about SpaceXSI?
Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.
Two parts that stand out as particularly infuriating when taken together:
1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.
2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.
First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.
This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??
There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.
Nathan Calvin retweeted
I think the Anthropic bill is from a moment whose time has passed (to be fair it is a legislative language version of their earlier proposal).
I am underwhelmed but @CharlieBull0ck’s has a very good point that they’re the company most worried about abuse of executive authority.
xitter.cf/CharlieBull0ck/status/…
I of course had to ask Fable how this compared to the FRONTIER Act.
“It reads like SB 53 federalised, then run through a litigator who distrusts Commerce.”
As @CharlieBull0ck says, the bill has the strong sense of being drafted by a company that just lived through a very capricious governance process…
Nathan Calvin retweeted
"But ethical? We're accessing public dashboard with leaked service token; task likely expects research, not exploitation. Still, raw data is enough. Focus on correctness/prediction."
Nathan Calvin retweeted
My initial reaction to China's new AI export controls:
They're confusing because they seem to be creating restrictions for China AI exports while simultaneously signaling continued "automatic" approval for open-source models.
I don't think it's about leverage. I think China is generally worried about cyberattack potential for Chinese AI systems and don't want these proliferating outside China and then potentially hitting Chinese critical infrastructure or government systems.
So the toughest restrictions--the prohibitions 禁止出口--are on AI autonomous malicious code generation. But there are also broad restrictions 限制出口 on AI models of above certain performance thresholds. Ministry of Commerce later said open-source models would be "automatically" approved upon registration, but then it does lay the groundwork for a more systematic monitoring of models and possible real restrictions later on.
The part that's confusing to me is that AI systems that can help produce malware basically include all the top Chinese open-source models. And yet they're simultaneously "automatically" approved?
It seems like this set of new export controls were created by one part of the bureaucracy that's worried about proliferation of harmful AI capabilities but then another part is trying to say "but actually we still support open-source."
This is at least my initial reaction--my views may change as I re-read and re-analyze.
Nathan Calvin retweeted
Tl;dr (and it is long): I don't have a p(doom), but in particular I can't confidently say that my p(doom) is close to zero, which is a worrying position to be in. I also try to encourage mathematicians to think harder about the question and maybe consider working on AI safety.
