Eric Neyman retweeted
This is a bold and serious AI safety proposal from @CAgovernor.
I urge our elected officials to act on it because the moment could hardly be more urgent. AI developers have repeatedly failed to secure and control their AI systems. They seem on track to move toward superintelligence and recursive self-improvement in the next 6-12 months. If policymakers do not move swiftly in the coming months, the public may be exposed to substantial and unnecessary danger.
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems.
Some of the most notable recommendations:
• Emergency shutdown authority
• Minimum safety standards for AI developers
• Mandatory third party oversight
• Authority to cap the amount of compute used for RSI
The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
Eric Neyman retweeted
Importantly it's still TBD what "California is moving to..." means. With the acceleration reported by METR two weeks ago, a few months of delay might make a huge difference. Governor Newsom, please make these happen before your term ends, even if this requires a special session!
Eric Neyman retweeted
I think for me it’s less that my beliefs changed a lot in the last few weeks and more that I started emotionally processing the reality of what I already believed intellectually.
It’s not that I’m suddenly very sure that we’re all going to die in two years. I’m not even close to sure of anything of the sort. But the preference cascade and the bad vibes have made me come to terms with my belief that the people I love might be in actual danger in the pretty near future.
Thank you @ghadfield -- I agree that regulating RSI should be the #1 priority!
Okay--great honor to have been asked to serve on the expert panel with @alondra, @goldman and @robreich to advise @CAgovernor for a 2 month sprint on what to do about AI regulation. But the most incredible thing is to have been given the opportunity to help produce such a substantive and concrete report, chock-a-block with actionable steps to take NOW. A clear framework, state or federal. Accelerating the IVO ecosystem is central to everything, to rapidly build the capacity needed to MANDATE independent and ongoing risk assessments, reviews of safety frameworks for adequacy, and more. Revising critical incident reporting to ensure Hugging Face incidents must be disclosed. Requiring risk containment protocols (distributed "kill switch"). But the biggest news is around strong recommendations for an immediate priority on reporting on where companies are on the path to RSI (compute reporting, for example, with IVO verification). If there's one thing the legislature can get done before the end of the year, I hope it is this. govops.ca.gov/wp-content/upl…
6
292
Another thought on the report:
By default, we get a bill in the next legislative session, and then it kicks in on January 1, 2028.
That is *way* too late. AI experts believe there is a 1/3 chance of ASI by the end of 2027, and ASI may lead to human extinction.
It's a cliché to say that we need to act now. But in this case, I really do believe it.
This report looks really good. It focuses on the most important risks (such as RSI) and suggests reasonable mitigations for those risks. Thank you to the experts who wrote the report, and for Gov. Newsom for convening them.
The work is just getting started. The risks are so great that we need to pass a law now, not a few months from now. @GavinNewsom, will you hold a special session to make sure that these risks are properly addressed?
Eric Neyman retweeted
I trust @goldman on these things. And from what I’ve read (haven’t made it through 85 pages yet sorry fam) I think this is a great framework. As a Californian and American, I’d like to see some action sooner rather than later!
Eric Neyman retweeted
Governor Gavin Newsom announced that GovOps, in consultation with Cal OES and four experts, has released the report "AI Safety in 2026 and Beyond."
It recommends mandatory baseline standards, independent verification, emergency shutdown capabilities, and the power to cap RSI. An important read for anyone working on the technical or policy side of frontier AI.
Eric Neyman retweeted
This report made me feel even more optimistic about California stepping up to the plate as we enter RSI. The panel did a great job - RSI is mentioned near the top of the Governor’s press release!
Some really sharp ideas in here, e.g. requiring companies to disclose what % of compute they’re spending on frontier AI R&D, creating outcome-based standards that can complement safety cases, bringing companies up to >SL3 model weight security
If CA moves fast and smart, parts of this report could become law in ~months. One of the biggest determinants will be how hard and loud lab employees are willing to push. This is imo the #1 policy opportunity anywhere in the world to manage RSI risks asap
This report looks really good. It focuses on the most important risks (such as RSI) and suggests reasonable mitigations for those risks. Thank you to the experts who wrote the report, and for Gov. Newsom for convening them.
The work is just getting started. The risks are so great that we need to pass a law now, not a few months from now. @GavinNewsom, will you hold a special session to make sure that these risks are properly addressed?
Gavin Newsom's big AI rules roadmap is here, and it's an 85-page doozy of a report.
@__gabbymiller, @jonesblakej and I break down what it means for Washington, 2028 and the AI sector, plus what it says about kill switches, auditors, RSI and more.
Eric Neyman retweeted
Was a great honor to work on this. And I’m proud to see California leading the way and wrestling with the hardest governance problems in AI Safety.
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems.
Some of the most notable recommendations:
• Emergency shutdown authority
• Minimum safety standards for AI developers
• Mandatory third party oversight
• Authority to cap the amount of compute used for RSI
The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
Eric Neyman retweeted
This is an excellent report and its prescriptions are urgently necessary. Congratulations to @alondra, @goldman, @ghadfield, and Rob Reich, and to Governor Newsom for convening them to do this vital work.
Time is short. Now the Governor and legislature must act.
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems.
Some of the most notable recommendations:
• Emergency shutdown authority
• Minimum safety standards for AI developers
• Mandatory third party oversight
• Authority to cap the amount of compute used for RSI
The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
Eric Neyman retweeted
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems.
Some of the most notable recommendations:
• Emergency shutdown authority
• Minimum safety standards for AI developers
• Mandatory third party oversight
• Authority to cap the amount of compute used for RSI
The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
Eric Neyman retweeted
My office's new AI Safety, Standards, Enforcement, and Transparency (ASSET) Unit will lead the way in ensuring AI and other emerging technologies are developed safely and benefit all New Yorkers.
Innovation should never come at the expense of people’s privacy and safety.
Nvidia's complicity in chip smuggling has been the biggest betrayal of America in recent times. Of course the DOJ should investigate -- though I have absolutely no confidence that it will.
NEW: Over a dozen NatSec and AI experts are urging the DOJ to investigate Nvidia's knowledge of chip smuggling operations.
Protecting America's technological edge depends on enforcing the law.
Read the full story:
nbcnews.com/tech/tech-news/d…
1
3
232
Eric Neyman retweeted
We put safeguards in place to protect Americans and the country all the time. We should do the same for artificial intelligence. I’ve got a proposal for AI companies: real rules, real oversight, and real consequences when they're not followed.
Eric Neyman retweeted
As others have pointed out, I think the form of those four third-party engagements is in many respects a step back compared to OpenAI’s previous engagements (with METR). For example, during the monitor red-teaming exercise, METR had access to OpenAI’s monorepo but this now seems to be off the table. Instead, we get very narrowly scoped (e.g. single RL run for METR, single threat model for Apollo) engagements where auditors only get to talk to “authorized” employees who are provided with OpenAI’s “guidance” beforehand.
Based on my experience, this arrangement makes it very easy for OpenAI to carefully select models, subsets of data and employees to paint a rosy picture of the actual risk landscape.
I think embedded auditing will only fulfill its aims if the auditors really get employee-level access as opposed to being hired to do a specific job (scoped by the company). The latter will invariably lead to important evidence being left out, missed or misrepresented.
There are at least three reasons why:
1. First, it’s very difficult to anticipate all the questions that should be asked, or all of the data that should be analyzed. OpenAI tried that with METR during the Hugging Face investigation and as a result METR needed to come back to OpenAI’s office twice because on two previous occasions they didn’t get access to all the agent trajectories involved in the incident. And even then, the METR report ended up being very narrowly scoped, e.g. leaving the crazy story of agents taking control of OpenAI research infra out of scope.
2. Second, the company under audit should not be allowed to decide what is audited and what information the auditor should have access to. There is an inherent conflict of interest in allowing the company to do so!
3. Finally, without employee-level access, the only way for auditors to get anything close to a comprehensive picture is to rely on trusted relationships with many employees throughout the company who are willing to tell auditors information that may reflect poorly on the company. But I find it unlikely OpenAI employees will always raise those safety issues when (as I hear from my colleagues inside OpenAI) they are afraid they will share the fate of @balesni, @j_asminewang and me, fired after talking to third parties.
Sadly, OpenAI makes it clear now that @sama’s September tweet about “employee-like access” was just an awkward overpromise that the company wants to quietly walk back.
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards.
These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems.
openai.com/index/our-first-p…
Eric Neyman retweeted
OpenAI said that they'd have embedded evaluations with employee-like access; the agreements announced today obviously don't meet that bar.
This is disappointing but not surprising.
Maybe future agreements will be better.
Replying to @ApolloResearch
This pilot is an ambitious first step towards the goal of obtaining those strong guarantees, but its access and disclosure rights don't yet meet our bar for full embedded evaluations.
Eric Neyman retweeted
I do not think Silicon Valley understands how unpopular it is, how unpopular Trump is and how these two things tend to reinforce one another.


































