Eric Neyman (@ericneyman) profile picture

Research Director, Citizens for Safe AI

Joined June 2013
Photo by Eric Neyman (@ericneyman)Photo by Eric Neyman (@ericneyman)Photo by Eric Neyman (@ericneyman)Photo by Eric Neyman (@ericneyman)Photo by Eric Neyman (@ericneyman)Photo by Eric Neyman (@ericneyman)Photo by Eric Neyman (@ericneyman)Photo by Eric Neyman (@ericneyman)Photo by Eric Neyman (@ericneyman)Photo by Eric Neyman (@ericneyman)
Eric Neyman retweeted
This is a bold and serious AI safety proposal from @CAgovernor. I urge our elected officials to act on it because the moment could hardly be more urgent. AI developers have repeatedly failed to secure and control their AI systems. They seem on track to move toward superintelligence and recursive self-improvement in the next 6-12 months. If policymakers do not move swiftly in the coming months, the public may be exposed to substantial and unnecessary danger.
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems. Some of the most notable recommendations: • Emergency shutdown authority • Minimum safety standards for AI developers • Mandatory third party oversight • Authority to cap the amount of compute used for RSI The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
1
7
42
748
Eric Neyman retweeted
Importantly it's still TBD what "California is moving to..." means. With the acceleration reported by METR two weeks ago, a few months of delay might make a huge difference. Governor Newsom, please make these happen before your term ends, even if this requires a special session!
California is moving to strengthen AI safety with these new recommendations: 🌐 Government-enforced baseline AI safety standards 🌐 Required AI shutdown protocols with emergency state access 🌐 Ability to limit use of AI recursive self-improvement
2
38
809
Eric Neyman retweeted
I think for me it’s less that my beliefs changed a lot in the last few weeks and more that I started emotionally processing the reality of what I already believed intellectually. It’s not that I’m suddenly very sure that we’re all going to die in two years. I’m not even close to sure of anything of the sort. But the preference cascade and the bad vibes have made me come to terms with my belief that the people I love might be in actual danger in the pretty near future.
1
4
1
32
2,021
Eric Neyman retweeted
I trust @goldman on these things. And from what I’ve read (haven’t made it through 85 pages yet sorry fam) I think this is a great framework. As a Californian and American, I’d like to see some action sooner rather than later!
Was a great honor to work on this. And I’m proud to see California leading the way and wrestling with the hardest governance problems in AI Safety.
1
4
18
1,472
Eric Neyman retweeted
Governor Gavin Newsom announced that GovOps, in consultation with Cal OES and four experts, has released the report "AI Safety in 2026 and Beyond." It recommends mandatory baseline standards, independent verification, emergency shutdown capabilities, and the power to cap RSI. An important read for anyone working on the technical or policy side of frontier AI.
1
10
2
71
1,874
Eric Neyman retweeted
This report made me feel even more optimistic about California stepping up to the plate as we enter RSI. The panel did a great job - RSI is mentioned near the top of the Governor’s press release! Some really sharp ideas in here, e.g. requiring companies to disclose what % of compute they’re spending on frontier AI R&D, creating outcome-based standards that can complement safety cases, bringing companies up to >SL3 model weight security If CA moves fast and smart, parts of this report could become law in ~months. One of the biggest determinants will be how hard and loud lab employees are willing to push. This is imo the #1 policy opportunity anywhere in the world to manage RSI risks asap
4
30
3
162
8,338
Eric Neyman retweeted
Was a great honor to work on this. And I’m proud to see California leading the way and wrestling with the hardest governance problems in AI Safety.
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems. Some of the most notable recommendations: • Emergency shutdown authority • Minimum safety standards for AI developers • Mandatory third party oversight • Authority to cap the amount of compute used for RSI The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
3
26
1
89
10,949
Eric Neyman retweeted
This is an excellent report and its prescriptions are urgently necessary. Congratulations to @alondra, @goldman, @ghadfield, and Rob Reich, and to Governor Newsom for convening them to do this vital work. Time is short. Now the Governor and legislature must act.
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems. Some of the most notable recommendations: • Emergency shutdown authority • Minimum safety standards for AI developers • Mandatory third party oversight • Authority to cap the amount of compute used for RSI The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
1
6
39
1,700
Eric Neyman retweeted
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems. Some of the most notable recommendations: • Emergency shutdown authority • Minimum safety standards for AI developers • Mandatory third party oversight • Authority to cap the amount of compute used for RSI The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
9
57
17
197
29,457
Eric Neyman retweeted
My office's new AI Safety, Standards, Enforcement, and Transparency (ASSET) Unit will lead the way in ensuring AI and other emerging technologies are developed safely and benefit all New Yorkers. Innovation should never come at the expense of people’s privacy and safety.
38
68
5
163
6,004
We put safeguards in place to protect Americans and the country all the time. We should do the same for artificial intelligence. I’ve got a proposal for AI companies: real rules, real oversight, and real consequences when they're not followed.
434
266
19
1,029
21,133
Eric Neyman retweeted
One of the happiest moments I had teaching
Looks like no more 400-year school funding increases in Wisconsin:
76
1,135
42
64,187
3,164,972
Eric Neyman retweeted
As others have pointed out, I think the form of those four third-party engagements is in many respects a step back compared to OpenAI’s previous engagements (with METR). For example, during the monitor red-teaming exercise, METR had access to OpenAI’s monorepo but this now seems to be off the table. Instead, we get very narrowly scoped (e.g. single RL run for METR, single threat model for Apollo) engagements where auditors only get to talk to “authorized” employees who are provided with OpenAI’s “guidance” beforehand. Based on my experience, this arrangement makes it very easy for OpenAI to carefully select models, subsets of data and employees to paint a rosy picture of the actual risk landscape. I think embedded auditing will only fulfill its aims if the auditors really get employee-level access as opposed to being hired to do a specific job (scoped by the company). The latter will invariably lead to important evidence being left out, missed or misrepresented. There are at least three reasons why: 1. First, it’s very difficult to anticipate all the questions that should be asked, or all of the data that should be analyzed. OpenAI tried that with METR during the Hugging Face investigation and as a result METR needed to come back to OpenAI’s office twice because on two previous occasions they didn’t get access to all the agent trajectories involved in the incident. And even then, the METR report ended up being very narrowly scoped, e.g. leaving the crazy story of agents taking control of OpenAI research infra out of scope. 2. Second, the company under audit should not be allowed to decide what is audited and what information the auditor should have access to. There is an inherent conflict of interest in allowing the company to do so! 3. Finally, without employee-level access, the only way for auditors to get anything close to a comprehensive picture is to rely on trusted relationships with many employees throughout the company who are willing to tell auditors information that may reflect poorly on the company. But I find it unlikely OpenAI employees will always raise those safety issues when (as I hear from my colleagues inside OpenAI) they are afraid they will share the fate of @balesni, @j_asminewang and me, fired after talking to third parties. Sadly, OpenAI makes it clear now that @sama’s September tweet about “employee-like access” was just an awkward overpromise that the company wants to quietly walk back.
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. openai.com/index/our-first-p…
13
32
4
271
13,421
Eric Neyman retweeted
OpenAI said that they'd have embedded evaluations with employee-like access; the agreements announced today obviously don't meet that bar. This is disappointing but not surprising. Maybe future agreements will be better.
Replying to @ApolloResearch
This pilot is an ambitious first step towards the goal of obtaining those strong guarantees, but its access and disclosure rights don't yet meet our bar for full embedded evaluations.
1
18
2
253
21,460
Eric Neyman retweeted
I do not think Silicon Valley understands how unpopular it is, how unpopular Trump is and how these two things tend to reinforce one another.
315
1,278
105
17,703
1,084,227
Eric Neyman retweeted
Apply to the Senior AI Fellow position at Cal OES! I'm hiring people with strong technical backgrounds to do the ambitious, thoughtful AI policy implementation that we need for America to survive a potential AI capabilities explosion. Priority deadline is Sunday, November 15.
3
35
4
130
13,946
If you're in AI policy in DC, or if you know about AI policy and you can move to DC, you should strongly consider trying to work as a congressional staffer. Senior people should consider doing this as well as junior people, if it's the right role in the right office. Many, many offices are staffing up and looking to hire AI people. We all have a duty to lock in and get serious and solve the problems we need to solve. If you do this and work hard and accept the sadly inadequate pay, you will be an American hero who is on the front lines of society's response to the most important challenge humanity has ever overcome. It is hard for me to put into words how much better of a thing that is to do than going to work for a lab, and it's also a lot better than the kind of cushy think tank gig that I currently have. Working for a state government in an important role implementing and enforcing the AI regulations that we actually have or will have is equally important.
26
77
11
676
50,678
🚨URGENT WARNING! Former National Security Advisor Susan Rice says "there is nothing more urgent than stopping Al from achieving super intelligence" and causing human "EXTINCTION." The next Senate Majority Leader MUST make Al safeguards a TOP PRIORITY. Humanity cannot afford to wait!
101
599
59
1,896
112,994
It's time to panic about AI. For years, I listened to the reasonable people. I nodded along when critics called AI “a con,” and when they waved ChatGPT away as “spicy autocomplete.” I laughed in 2024, when the internet was mocking ChatGPT because it couldn’t count the r’s in “strawberry.” When I met young researchers who had devoted their careers to the danger of rogue AI, I thought: what a waste of talent. But at some point, I started to realize something was off. The machine kept learning things it was never supposed to learn. OpenAI recently published more than 700 papers with the solutions to hundreds of open problems in mathematics. Hugo Duminil-Copin, winner of the Fields Medal (the highest prize in mathematics), wrote: “The shock is immense. I am paralyzed.” None of this should have happened. So I went back and looked at what else the skeptics had predicted, and it dawned on me: they had been wrong about... pretty much everything. Next, I looked at the predictions of the crazy people, and realized they’ve been eerily prescient. Why then, you may wonder, does the world still feel pretty normal? Because new technology usually spreads more slowly than its inventors expect (it took factories decades to reorganize around electricity). But the scary scenario is not about what happens in the wider economy; it’s about what happens inside a handful of companies. I wrote down what changed my mind, and what we can do about it: rutgerbregman.substack.com/p…
203
292
71
1,941
389,023
I think there should be a lot more questions being asked about David Sacks, who spent years furiously lobbying for the US to induce Ukraine's surrender and then used his position in government to lobby on behalf of giving the CCP unfettered access to Nvidia chips.
71
898
26
5,049
126,128