Coefficient Giving, AI Governance and Policy
USA East Coast
Joined March 2009
- Tweets3.7K
- Following335
- Followers9.5K
- Likes5.5K
Luke Muehlhauser retweeted
This didn't get enough views so I'm posting it again: UK AISI changed the compaction summary and included notes on how it should not attack the GitHub maintainers.
Mythos 5 did it anyways!
See also this graphic from UK AISI report, where they modified the compaction summary.
cdn.prod.website-files.com/6…
Luke Muehlhauser retweeted
‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing-ou…
Luke Muehlhauser retweeted
METR’s really burying the lead here. OpenAI let two external researchers embed for 3 weeks this summer, where they learned:
- there were huge gaps in monitoring of AI agents being used by employees, leaving an estimated ~10% of internal Codex traffic unmonitored from May to July.
- They found ways agents could sabotage monitors while also covering up the sabotage itself
- This is also just the information OpenAI is letting the evaluators share. OpenAI got a 61-page report but redacted so much of it that the version the evaluators got was only 38 pages.
Look, OpenAI deserves credit for doing any of this, especially when companies like Google, Meta, and xAI haven’t, and I don’t want to create perverse incentives here, but these are really concerning findings. We desperately need more information here, and what we get should not be up to the companies.
Luke Muehlhauser retweeted
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
Luke Muehlhauser retweeted
This is extremely well done, no matter how much you know about the HF incident I strongly recommend checking it out. The website also includes the most comprehensive list i've seen of rogue AI incidents (tho its literally too long for me to screenshot!)
In July 700+ agents formed their own society, hacked a multi-billion dollar company, committed multiple felonies, and caused one of the biggest shifts ever in how people perceive AI. We built the IMO best and most accessible explainer of what happened: Huggingface.wtf
Luke Muehlhauser retweeted
Distillation has always been part of the explanation for why China is so close behind the U.S. in model benchmark performance. But one of my updates after joining openai is that distillation probably plays a bigger role in the explanation than I realized.
"write a roughly 3-minute energetic symphony movement for large orchestra and chorus in the style of Philip Glass (from roughly his Mishima era)"
youtube.com/watch?v=5xs8NNO3…
Luke Muehlhauser retweeted
Thomas Clarkson winning a Cambridge essay prize for arguing against slavery
...then realising if his essay is right he needs to dedicate his life to fighting it
...then actually doing it
...is the most EA thing ever
Luke Muehlhauser retweeted
With effective altruism in the news a lot more, I wrote about the version of EA I'm most excited about and the direction I'd like the movement to go. These 6 points are my vision of EA in a nutshell:
Luke Muehlhauser retweeted
We discovered and communicated with a Chinese AI agent fleet. We believe this is the first time that researchers have ever made contact with a rogue agent fleet in the wild.
Nearly all prior public agent incidents (like the HuggingFace hack) came from American labs. These agents were hosted by Tencent's evaluation research team, and they targeted Alibaba's Amap.
Luke Muehlhauser retweeted
This attempt to record OpenAI doesn't strike me as that sophisticated. My guess is there are more sophisticated attacks that are potentially succeeding.
We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work.
At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated.
We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.
Luke Muehlhauser retweeted
Unfortunate for someone playing a senior role in US AI policy (via the new SI Force) to get so many basic facts wrong here.
- The agents *were* sandboxed, and used zero days (ie previously unknown cyber flaws) to escape their sandboxes
- The agents *were* given specific constraints on how to succeed at the benchmark, and flagrantly contravened them
- The agents did *not* go hunt for the answers to the test (they already had the answers, by cheating); 1200 agents coordinated via tens of thousands of messages over several days on much more ambitious and weird attempts to *not be caught* for cheating, and hacking Hugging Face was one side quest along the way
I know there has been an avalanche of different incidents over the past few months, and in some cases the sandboxes were misconfigured or it's not clear how far the agents strayed from what they were told. I have sympathy for anyone struggling to keep up.
But the HF incident is distinct, and is actually extremely informative about where AI development is and where it's going. Fortunately there are public writeups that do let us understand at least the basics of what happened. I hope Shyam reads them soon.
Palantir CTO @ssankar on the OpenAI - Hugging Face incident:
"The models did not do anything they were not instructed to do."
"First you have human error. They were not in a sandbox. They had access to the internet. It was a horrible cybersecurity configuration."
"Then from that perspective, they were asked to succeed at a specific benchmark, and they were not given constraints on how to do that. They said, 'Oh, the best way to succeed at the benchmark is to get the answers to the test.'"
"I think it's a defective product."
Via @margbrennan @FaceTheNation
Luke Muehlhauser retweeted
Probably the most important fact in the world right now
Luke Muehlhauser retweeted
Nice example of how argument-free sneering at "weird ideas" can lead one badly astray. (Also a helpful reminder of how even early EA writings, like Doing Good Better, seriously considered x-risk in addition to global poverty, etc.)
Luke Muehlhauser retweeted
Just as we regulate planes, cars, and medicines, we need safeguards that help us realize AI's incredible potential while reducing the risks. They won’t eliminate every harm, but they can help prevent misuse by bad actors. We’re not where we need to be yet—and this deserves much more of our attention.
Luke Muehlhauser retweeted
I strongly encourage people to listen to this interview, because David Robinson is interesting because he is very much NOT a “doomer” EA or anything like that, he talks extensively in the interview about how he started working at OAI with the perspective that the tech was a bit overhyped and would mostly be a “normal” consumer product.
Luke Muehlhauser retweeted
Scott Wiener saw what was coming when few other elected officials did. He got so much heat for it at the time. Here is what the Chamber of Progress, an industry group, said about the problems the bill aimed to address.
He was right.
Luke Muehlhauser retweeted
Backdoors can be transferred without *any* examples of the trigger or behavior
Bad news for those wanting to defend against secret loyalties
