AI and National Security @cnasdc, MS @wisconsincs. Views my own.
Joined November 2015
- Tweets798
- Following1.5K
- Followers5.1K
- Likes99.3K
Ben Hayum retweeted
So basically, today PHP got two critical bugs that let anyone run code on your server without even logging in, which is bad, and then the details leaked before the patch was even out, which is worse, and it works on WordPress and MediaWiki and Magento and Joomla and Laravel, and Censys says like 1.5 million servers are running a version with no fix, so the NSA was like "take your system offline," and the FBI was like "take your system offline," and CISA and the UK and CERT-EU were all like "take your system offline," and then the Prime Minister of Japan, the actual Prime Minister, posted at 5:55 in the morning telling 48.8 million people to take their system offline, so everyone took their system offline, like the Dutch cybersecurity agency took its own website offline, and the Dutch parliament went offline, and the Port of Rotterdam went offline, and Finland and France and Denmark started pulling government sites, and Wikipedia went down, and Archive.org went down, and Asana and GOG and Something Awful went down, and the University of Vienna's webmail went down for some reason, and that's just PHP. Because GitLab also dropped a CVSS 10.0, which is the highest score they give out, for a bug that's been there since 2014, so that's every self-hosted GitLab for twelve years, and then LiteLLM 1.107.0 got hijacked and shipped a credential stealer and a remote access trojan in every single wheel and the Docker images, including latest, and you didn't even have to import it, just installing it was enough, and it's the second time this year, and then Railway paused every deployment on the entire platform because someone got into the environment variables, so now everyone has to rotate every secret they've ever stored there, and TiDB Cloud said someone was walking around inside with stolen credentials for basically a month and an extortion group says they took 10.6 terabytes, and Zeabur got breached for the second time in two months and is just shutting down, like the whole company, and Node.js patched remote code execution and a TLS bypass, and CISA says Plex is being actively exploited, and someone took over 300 Tor exit relays to steal SSH passwords, and SafePay ransomware has hit over 600 critical infrastructure orgs, and India's biggest bank filed a data loss incident, and Japan has been in an official state of emergency in cyberspace since October 9, which honestly explains the 5:55 AM post. Oh, and eMudhra, a certificate authority that's issued over 60 million certificates, whose entire job is proving websites are who they say they are, was supposed to be using private keys that are random numbers over a hundred digits long, and one of them was 4. Just 4. And the others were like 66 and 237. Anyway, it's Wednesday.
Ben Hayum retweeted
We think existing executive branch authorities could be used to double America's current AI lead over China by the end of 2027.
This decision could be made by a very small number of people:
- @howardlutnick could ~halve China's access to AI training compute, without reducing overall US AI chip sales
- Ferguson (FTC chair) + Woodward (oversees DOJ antitrust division) + McCotter (acting DOJ DAG) could provide legal protections for US companies to share threat intel and best practices to allow the US AI ecosystem to counter China's adversarial distillation
Big new IFP report from @GeorgiaCAdamson, @KonstantinPilz, and me on the US-China AI competition.
We spent the last 6 months digging into how Chinese AI companies’ access to compute and distillation of American models translate into the current US-China capabilities gap.
The result is a 23k-word research report, including an interactive analytical model that visualizes changes in the US AI lead under different policy interventions.
Our findings:
1. The US AI lead over China is currently only around 6 months.
2. The US could roughly double this lead, to over a year, using authorities already available to the federal government.
3. On chips: China’s access to AI chips will roughly double over the next year, with chips housed in data centers overseas accounting for roughly half of China’s 2027 compute.
4. On distillation: the current ~16x US training compute advantage would normally yield a 13-month lead. But we estimate adversarial distillation halves this to around 6.5 months.
5. Increasing the lead would have some obvious benefits:
- Give the US broadly a greater cyberdefensive advantage against Chinese threat actors.
- Give US AI companies breathing room to spend more R&D resources on making their systems safer and more secure, without fear of being overtaken by less safety-conscious Chinese competitors.
6. Provide the US government more leverage in negotiations with China about the future of AI.
7. The two most important interventions are:
- For BIS: Create a new end-use control to restrict China’s remote access to training clusters abroad — China’s largest and fastest-growing source of frontier-scale compute.
- For DOJ/FTC: Create a narrow safe harbor to allow US companies to coordinate on countering Chinese adversarial distillation.
8. Both interventions (and all the others we discuss in the report) are things the federal government could do today.
9. To estimate the effects of each policy intervention, we model the US lead as a function of its compute advantage over China and China’s ability to adversarially distill the capabilities of the US frontier.
10. The model makes many simplifying assumptions, but provides a useful way to assess and rank the impact of different policy interventions at a high level.
Read the full report here: ifp.org/how-to-increase-amer…
Ben Hayum retweeted
You know it's serious when heads of state get involved.
皆様に、今夜に対応して頂きたい大至急のお願いです。
多くのウェブサイトで使用されているPHPに関し、PHPのセキュリティチームが、重大な２件の脆弱性に関する注意喚起を行っています。
公開しているウェブサイトでPHPを使用しているか確認し、今夜、10月21日午後11時（日本時間）に公開される修正パッチについて、可及的速やかに適用できるよう準備をお願いします。
修正パッチを速やかに適用することが困難な場合は、例えばウェブサイト等を一時的に停止させることも、リスクを低減させるための有効策の一つと考えられます。
国家サイバー統括室のXアカウントでも関係する情報を発信していますので、そちらも参考にして下さい。
国家サイバー統括室(注意・警戒情報)（@cyber_forecast） / X
Ben Hayum retweeted
To prepare for and respond to SI control incidents, NIST CAISSI, NSA SISC, and CISA have authored voluntary practices for organizations training and evaluating highly capable agents.
Ben Hayum retweeted
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
Ben Hayum retweeted
We believe that Anthropic should include "undermining of safety-relevant decision-making" as key risk pathway.
This is partially inspired by Section 2.20 of their risk report, where Anthropic asked Claude for feedback on their report, and received a (imo) sycophantic answer.
(2) We think Anthropic should, for future reports, consider risks from AI systems undermining safety-relevant decisions at Anthropic.
Read our full risk report review here: metr.org/blog/2026-10-20-mis…
Ben Hayum retweeted
Excited to announce that I’m moving to Beijing and starting a professorship in AI safety at Tsinghua University’s College of AI!
Here I explain why: sorenmindermann.substack.com…
Ben Hayum retweeted
It is important to re-state that the threats are not hypothetical, and that we must take action to mitigate the risks of AI-enabled terrorism.
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
If you are looking for technical grounding into how AI-enabled terrorism might occur despite safety practices, look no further.
The most comprehensive treatment I've seen. It will certainly be used in legal cases for years to come.
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
Verifiable identifiers like this should be tied to all agent deployments in the future.
Ben Hayum retweeted
Chinese AI capabilities lag the US by about 6.5 months. But the US can extend this lead, if it chooses to.
New report from @fiiiiiist, @GeorgiaCAdamson and @KonstantinPilz at @IFP.
Big new IFP report from @GeorgiaCAdamson, @KonstantinPilz, and me on the US-China AI competition.
We spent the last 6 months digging into how Chinese AI companies’ access to compute and distillation of American models translate into the current US-China capabilities gap.
The result is a 23k-word research report, including an interactive analytical model that visualizes changes in the US AI lead under different policy interventions.
Our findings:
1. The US AI lead over China is currently only around 6 months.
2. The US could roughly double this lead, to over a year, using authorities already available to the federal government.
3. On chips: China’s access to AI chips will roughly double over the next year, with chips housed in data centers overseas accounting for roughly half of China’s 2027 compute.
4. On distillation: the current ~16x US training compute advantage would normally yield a 13-month lead. But we estimate adversarial distillation halves this to around 6.5 months.
5. Increasing the lead would have some obvious benefits:
- Give the US broadly a greater cyberdefensive advantage against Chinese threat actors.
- Give US AI companies breathing room to spend more R&D resources on making their systems safer and more secure, without fear of being overtaken by less safety-conscious Chinese competitors.
6. Provide the US government more leverage in negotiations with China about the future of AI.
7. The two most important interventions are:
- For BIS: Create a new end-use control to restrict China’s remote access to training clusters abroad — China’s largest and fastest-growing source of frontier-scale compute.
- For DOJ/FTC: Create a narrow safe harbor to allow US companies to coordinate on countering Chinese adversarial distillation.
8. Both interventions (and all the others we discuss in the report) are things the federal government could do today.
9. To estimate the effects of each policy intervention, we model the US lead as a function of its compute advantage over China and China’s ability to adversarially distill the capabilities of the US frontier.
10. The model makes many simplifying assumptions, but provides a useful way to assess and rank the impact of different policy interventions at a high level.
Read the full report here: ifp.org/how-to-increase-amer…
Ben Hayum retweeted
Big new IFP report from @GeorgiaCAdamson, @KonstantinPilz, and me on the US-China AI competition.
We spent the last 6 months digging into how Chinese AI companies’ access to compute and distillation of American models translate into the current US-China capabilities gap.
The result is a 23k-word research report, including an interactive analytical model that visualizes changes in the US AI lead under different policy interventions.
Our findings:
1. The US AI lead over China is currently only around 6 months.
2. The US could roughly double this lead, to over a year, using authorities already available to the federal government.
3. On chips: China’s access to AI chips will roughly double over the next year, with chips housed in data centers overseas accounting for roughly half of China’s 2027 compute.
4. On distillation: the current ~16x US training compute advantage would normally yield a 13-month lead. But we estimate adversarial distillation halves this to around 6.5 months.
5. Increasing the lead would have some obvious benefits:
- Give the US broadly a greater cyberdefensive advantage against Chinese threat actors.
- Give US AI companies breathing room to spend more R&D resources on making their systems safer and more secure, without fear of being overtaken by less safety-conscious Chinese competitors.
6. Provide the US government more leverage in negotiations with China about the future of AI.
7. The two most important interventions are:
- For BIS: Create a new end-use control to restrict China’s remote access to training clusters abroad — China’s largest and fastest-growing source of frontier-scale compute.
- For DOJ/FTC: Create a narrow safe harbor to allow US companies to coordinate on countering Chinese adversarial distillation.
8. Both interventions (and all the others we discuss in the report) are things the federal government could do today.
9. To estimate the effects of each policy intervention, we model the US lead as a function of its compute advantage over China and China’s ability to adversarially distill the capabilities of the US frontier.
10. The model makes many simplifying assumptions, but provides a useful way to assess and rank the impact of different policy interventions at a high level.
Read the full report here: ifp.org/how-to-increase-amer…
Ben Hayum retweeted
I am kind of surprised to discover that many security researches used to doing security work manually have no idea what is coming for them, and think AI assistance is a "nice to have".
The conundrum we face is the following cycle:
1) Not giving reliable closed-model access to on-the-ball defenders leads to more vehemently pro open-model people.
2) Open-model cyberweapons leads to not on-the-ball defenders (a massive amount of software globally) getting popped.
3) Software getting popped leads to more vehemently anti open-model people.
And repeat.
The only way out is through making trusted closed-model access programs accessible, predictable, and reliable.
Maybe there should even be an "emergency/active attack" feature in these programs where more leeway is given but model providers also monitor more closely.
Defenders who are on-the-ball cannot be held back while they are being actively attacked.
But it is also politically and diplomatically unsustainable to continue releasing open-model cyberweapons.
The South Korea's and Japan's of the world who are facing the brunt of it will not be happy, and they'll be right.
Distillation attacks are bolstered by using fraudulent accounts that are harder to trace back to China.
The hack's objective could have been to steal a bunch of people's account information and keys. Unclear though.
Ben Hayum retweeted
Distillation has always been part of the explanation for why China is so close behind the U.S. in model benchmark performance. But one of my updates after joining openai is that distillation probably plays a bigger role in the explanation than I realized.
Ben Hayum retweeted
I like this suggestion from @logangraham. I would be extremely excited to see EA philanthropic money/frontier companies go into partnerships with governments - eg Australia’s - trying to secure the digital infrastructure underpinning many small island nations in the Indo-Pacific. They are often poorly resourced and have a long list of critical concerns that sit higher than cyber until an incident occurs.
EG: Vanuatu’s Govt was held ransom following hack in Nov 22, Tonga’s telecomms in Feb 23, Palau in March 24, Solomon Islands in May 24, Palau again in July and Nov 24, PNG this year again.
Would be great to see a securing initiative.
Anthropic's @logangraham predicts we could tackle the vast majority of cyber risk in just 1-2 years:
"I want to just go secure the entire world, and I want the entire world to go secure the entire world. Over the next one to two years, we could just harden the world."
"We now have very sophisticated and relatively cheap tools that allow you to magically rewrite all software to be more secure. You can find all the bugs, write code that is secure by default, and use models to defend everything."
"There’s literally millions of people around the world who are the most technically sophisticated people in the world who are cyber defenders, and they move really fast. The whole thing is, how do you empower them?"
"I think 2026 has been the year of AI cyber, and the message has spread. People are now like, how do I defend everything?"
"“I’m pretty optimistic. Over the next one or two years, we could deal with the vast majority of this. It’s actually fairly straightforward how you get that for cyber."
@AnthropicAI
Ben Hayum retweeted
My initial reaction to China's new AI export controls:
They're confusing because they seem to be creating restrictions for China AI exports while simultaneously signaling continued "automatic" approval for open-source models.
I don't think it's about leverage. I think China is generally worried about cyberattack potential for Chinese AI systems and don't want these proliferating outside China and then potentially hitting Chinese critical infrastructure or government systems.
So the toughest restrictions--the prohibitions 禁止出口--are on AI autonomous malicious code generation. But there are also broad restrictions 限制出口 on AI models of above certain performance thresholds. Ministry of Commerce later said open-source models would be "automatically" approved upon registration, but then it does lay the groundwork for a more systematic monitoring of models and possible real restrictions later on.
The part that's confusing to me is that AI systems that can help produce malware basically include all the top Chinese open-source models. And yet they're simultaneously "automatically" approved?
It seems like this set of new export controls were created by one part of the bureaucracy that's worried about proliferation of harmful AI capabilities but then another part is trying to say "but actually we still support open-source."
This is at least my initial reaction--my views may change as I re-read and re-analyze.