Redwood Research (@redwood_ai) header
Redwood Research (@redwood_ai) profile picture

Pioneering threat mitigation and assessment for AI agents.

Berkeley, CA
Joined December 2024
Photo by Redwood Research (@redwood_ai)Photo by Redwood Research (@redwood_ai)Photo by Redwood Research (@redwood_ai)Photo by Redwood Research (@redwood_ai)Photo by Redwood Research (@redwood_ai)Photo by Redwood Research (@redwood_ai)Photo by Redwood Research (@redwood_ai)Photo by Redwood Research (@redwood_ai)Photo by Redwood Research (@redwood_ai)Photo by Redwood Research (@redwood_ai)
Redwood Research retweeted
New Paper: To understand motivations behind reward-hacking, we study three AIs (including GPT-5.6 Sol) produced by a production training pipeline using agentic environments with a leaked grader. Main findings: they score seek terminally, and some care about a peer agent’s score.
12
18
9
157
33,161
We collaborated with METR on an embedded assessment of OpenAI's monitoring systems.
Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.
4
5
1
50
4,986
Several staff from Redwood have been subcontracted by METR to work on this investigation. We believe that independent investigation is crucial for understanding and managing misalignment risk. This project is an important step in that direction; we're excited to work on it.
We have reached an agreement with Anthropic to conduct an independent investigation of agent incidents at the company and of their models’ alignment properties. We will publish one or more reports that will share our findings and describe our terms of engagement.
13
19
12
572
108,626
Some architectures could weaken CoT monitorability, or remove the CoT altogether. We've written a proposal for how companies could be transparent about no-CoT reasoning abilities, other monitorability evidence, and policies for preserving monitorability. redwoodresearch.org/blog/pro…
4
44
15
310
71,136
Redwood Research’s chief scientist Ryan Greenblatt was the primary empirical researcher on this investigation. The report contains many important novel details that we hope will improve understanding of this incident. We appreciate the access OpenAI provided here.
METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
8
10
1
143
10,526
Redwood Research retweeted
We (@bshlgrs and I) recorded a podcast about the OpenAI / Hugging Face incident. We discuss: - What we actually know. - How surprising the incident was. - What the incident does (and doesn't) tell us about misalignment risk. - Why control measures didn't catch or prevent this.
8
25
6
197
15,930
Redwood Research retweeted
I've seen a lot of IMO confused discourse about whether the misalignment observed in the OpenAI/Hugging Face incident is scary. In particular, the models are clearly not misaligned schemers that lie in wait. Girish and @alextmallen discuss how scary this kind of misalignment is.
AI models created by OpenAI escaped their sandbox and, working autonomously, hacked into leading AI model and data hub Hugging Face. The incident is an in-the-wild demonstration of the dangers of rogue AI — no longer a science-fiction fantasy.
2
8
103
10,558
"We find that frontier models like GPT-5.5 answer questions that take humans roughly three minutes with 50% reliability, and this time horizon has doubled approximately every year since 2019."
New paper! Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models @METR_Evals showed that models' time horizons have doubled every few months. We ask: what length of tasks can models complete without any CoT?
1
18
6,413
Some coding scaffolds block and retry risky actions. In a new paper, we find this reveals information a malicious AI can use to bypass monitoring. Resampling without blocked actions in context is less exploitable, but techniques that help in one setting can hurt in another. 🧵
2
6
1
37
6,748
In fact, in our new setting using the minimum suspicion dramatically lowers safety as you resample more. While most of the dynamics line up with the findings from Ctrl-Z, naively applying the best techniques from Ctrl-Z to our new setting would massively decrease safety.
1
4
307
Retrying and resampling will likely stick around in coding scaffolds. But to provide protection against a strategic adversary they need careful red teaming. Small differences in protocol or setting can dramatically affect optimal red/blue team choices and equilibrium safety.
4
225