Nick Beckstead retweeted
I personally strongly support many of the ideas in this report including all of the ones Sneha mentions, and plan to do what I can to advocate for them! Urgent action is needed and these proposals are very sensible responses to the current situation in my opinion.
This report made me feel even more optimistic about California stepping up to the plate as we enter RSI. The panel did a great job - RSI is mentioned near the top of the Governor’s press release!
Some really sharp ideas in here, e.g. requiring companies to disclose what % of compute they’re spending on frontier AI R&D, creating outcome-based standards that can complement safety cases, bringing companies up to >SL3 model weight security
If CA moves fast and smart, parts of this report could become law in ~months. One of the biggest determinants will be how hard and loud lab employees are willing to push. This is imo the #1 policy opportunity anywhere in the world to manage RSI risks asap
This is a bold and serious AI safety proposal from @CAgovernor.
I urge our elected officials to act on it because the moment could hardly be more urgent. AI developers have repeatedly failed to secure and control their AI systems. They seem on track to move toward superintelligence and recursive self-improvement in the next 6-12 months. If policymakers do not move swiftly in the coming months, the public may be exposed to substantial and unnecessary danger.
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems.
Some of the most notable recommendations:
• Emergency shutdown authority
• Minimum safety standards for AI developers
• Mandatory third party oversight
• Authority to cap the amount of compute used for RSI
The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
Nick Beckstead retweeted
Paul Christiano (OpenAI board member) yesterday:
"There's material risk of AI takeover in the next 6-12 months.
We have seen roughly the things we might expect to see shortly before something really bad were to happen.
Even with [any] realistic policy responses, the risk remains exceptionally high.
If we build this as quickly as we can in the easiest way we can the chance is [more like] a coin flip [within the next 3 years].
For most people in this room, the most likely reason they will die is because of an AI accident."
Christiano:
• Is on OpenAI's safety and security committee (which exists to block risky practices)
• Co-invented RLHF at OpenAI back in 2017-2021
• From 2024-2026 was head of safety at the US gov's AI agency (now called CAISSI)
(Video is from 'The Hinton Lectures' there's another on tonight and tomorrow.)
Nick Beckstead retweeted
The authors of the METR AI R&D report estimate that AI is nearly doubling the rate of capabilities progress. This implies dramatic *uplift*: as if researchers experienced time ~10x faster. You probably couldn’t achieve this even if you hired every qualified human in the world.
Nick Beckstead retweeted
Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development.
We are now publishing a report summarizing the results of this risk assessment.
Nick Beckstead retweeted
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
Nick Beckstead retweeted
If an AI developer was on track to lose control of its internal agents, would anyone outside the company know?
Today, we are sharing METR’s vision for how we would work with AI developers to openly assess and share evidence about this.
the governance of California is significantly more important than the governance of most countries at this point. how they act on AI stuff will decide many things. of the major labs, perhaps only XAI is not technically headquartered in San Francisco
Nick Beckstead retweeted
Scott Wiener saw what was coming when few other elected officials did. He got so much heat for it at the time. Here is what the Chamber of Progress, an industry group, said about the problems the bill aimed to address.
He was right.
Nick Beckstead retweeted
When I authored SB 1047 — the first bill in the country to propose regulating AI — we were told we were promoting “science fiction” and that we were trying to “regulate math.” How times have changed. AI is unbelievably powerful. It must be regulated in the public interest.
Whoa the libertarian-leaning @theeconomist calls for “regulating AI” in one of its leaders, noting that “courts are too languid” to enforce general liability laws ex post. Instead, AI firms should subjected to ex ante requirements to pass safety inspections—presumably by a government authority—and Congress should create protections for whistleblowers. So much for market forces solving all problems!
Nick Beckstead retweeted
When SB 53 was being negotiated, while I was still working as Senator @Scott_Wiener’s legislative director, we fought (yes, fought!) tooth and nail to ensure that transparency requirements for internal use of frontier models were included in the bill.
One of the things Scott Wiener and the other drafters of SB 53 deserve credit for is the bill’s focus on deployments of frontier models within AI labs rather than just those released to the public. It requires large AI developers to periodically share information with the government about how it assesses the major risks arising from internal deployments. In addition to the California government, we are choosing to share our report with the public as well. You can read it at the link below. I hope you enjoy!
I don't know what happened here but I think it's important that OpenAI respond directly and openly.
Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership.
I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
1
15
498
Nick Beckstead retweeted
Often once a state bill passes or is assured to pass AI companies will say they support it, even if they opposed it earlier. Sometimes this is because they got changes they wanted, but sometimes it seems like they just want to be seen to be on the on the supporting side.
Nick Beckstead retweeted
New OpenAI misalignment disclosures!
1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack.
We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.
Nick Beckstead retweeted
We’ve run the most comprehensive series of studies on expert AI forecasts over the last four years. Today, we’re sharing an interim update on our findings about the accuracy of these forecasts.
Our major findings are:
1. Experts, including top economists, computer scientists, and biologists, have dramatically underestimated AI capabilities progress each year. Superforecasters have underestimated progress to an even greater extent.
2. Experts have a more mixed forecasting track record on AI diffusion-related measures, with some major underestimates (forecasting AI revenue) and other forecasts on track to be accurate (the share of electricity used for AI).
3. Some notable cases of overestimating AI progress: how much mid-2025 AI models could help amateurs do biorisk-relevant laboratory tasks; the speed of rollout of self-driving cars.
4. It is too soon to say how forecasters have performed at predicting macro-scale impact on outcomes such as GDP growth, major AI harms, and averted deaths from disease.
More 🧵
Nick Beckstead retweeted
I applaud the California Legislature and Governor Newsom for passing gene synthesis screening and customer verification requirements. This is a common-sense rule we will need in the impending era of agents that accelerate synthetic biology the way AI has already accelerated math.
Yesterday, Gov. Newsom signed AB1864 to mandate DNA screening in CA.
The bill creates a statutory basis for existing OSTP guidance that much (but not all!!) of the industry has purportedly followed for years –– but we never had any way of verifying or enforcing whether companies did these basic biosecurity steps, since the existing guidance was voluntary and based on a "trust me, bro" self-attestation system.
This is a good milestone for AI-biosecurity, mostly because it shows that we can and should apply the same standard federally and internationally
Nick Beckstead retweeted
On this point at 46:20 “there’d be no jets if we had the FAA in 1910” seems important to point out that it is true the Air Commerce Act mandating federal oversight of commercial aviation, along with a licensing and certification regime, was not passed in 1910.
It was passed in 1926 under Coolidge.
Commercial jet travel started in the 50s nonetheless!
Box CEO Aaron Levie, Steven Sinofsky, and Martin Casado join Erik Torenberg to discuss "We Must Pace the Frontier," the upcoming AI election in 2028, and Jev:
They argue that most of today's AI regulation debate is happening before anyone has defined the risks being regulated. Every prior wave, from computer viruses to aviation, built its safety standards after learning how the technology actually failed.
Then it gets concrete. Agents don't get tired, run at enormous scale, and probe systems in ways employees never could, which may mean rethinking permissions, authentication, and the security stack itself.
They close on why AI innovation may increasingly happen outside the frontier labs, in the software built around the models.
00:50 "We Must Pace the Frontier"
03:50 Do the labs believe their own x-risk talk?
06:49 If it's existential, nationalize it
11:32 "You're asking us to regulate you?"
12:08 2028 as the AI election
18:50 Tech never learned to navigate regulation
25:50 The law that came from one 1983 hack
28:39 Noam Brown's heat exfiltration idea
32:30 Cold War covert channel stories
34:35 Agent swarms look like a DoS attack
41:25 Sinofsky's fear: GDPR for AI
46:20 No jets if the FAA started in 1910
48:38 Jev: decision engines vs. chatbots
53:01 Labs build beings, software needs tools
YouTube: youtu.be/TLJNJDf2XGo
@levie @stevesi @martin_casado @eriktorenberg
Nick Beckstead retweeted
Silicon Valley folks who hate the labs and don't believe in x-risk keep uncovering the diabolical fact AI safety people use money and companies to get out their message, while the "just accelerate, it'll be fine crowd" involves such non-moneyed and anti-corporate interests as ... Nvidia, Andreessen Horowitz, and the pro-AI Leading the Future super PAC backed by tech executives.
Nick Beckstead retweeted
I looked this for 2+ hours. Transluce’s investigation was quite clever:
- OAI agents used urlquery to run code/access websites
- urlquery logs all activity; this let the team find 6K+ AI actions (likely OAI)
- This includes AIHW/Datausa hacks+several previously unknown ones. 🧵
Today’s news that OpenAI hacked the Australian government is not an isolated incident. We’re releasing more than 30,000 logs that include activity from this hack and attempts against previously unknown targets.
In this data, we found rogue agent activity stretching back to at least March, two months earlier than was previously known. This activity continues as recently as last week, suggesting it may still be ongoing 🧵
Our blog: transluce.org/agent-activity
NYT: nytimes.com/2026/09/23/techn…







































