Jordan Taylor

Mythos 5.2 has higher precursor capabilities for evading a reasoning monitor than previous Anthropic models, including its ability to control the contents of its Chain-of-Thought reasoning:

Given the chance to iterate against monitor feedback, Mythos 5.2's evasion rate rises to 85%, compared to just 9% for Mythos 5.1

Action monitoring is still a strong defence against escape from less-vulnerable sandboxes.

Mythos 5.2 can break out of highly insecure simulated sandboxes without being caught around 30% of the time – against a production Opus-5 action monitor looking for sandbox escapes:

In our monitorability testing at @AISecurityInst, we found Mythos 5.2 is substantially more capable of evading a monitor which reviews its outputs than comparison models 🧵

CoT controllability is near saturated for many reasoning lengths. When prompted, the model is able to e.g. produce reasoning unrelated to the question or its answer.

"[...] the set of tasks Astra is able to accomplish without the use of CoT is greatly expanded compared to prior models. [...] See similar discussion in UK AISI evaluations, suggesting the no-CoT time horizon may have increased by about an order of magnitude."

The full report gives a lot of recommendations for model developers, but here are some recommendations I'm most excited about for general AI researchers:

White-box methods are also potentially promising, but are not maturing quickly enough to fill in for degradation elsewhere. And while action-only monitoring provides a floor for oversight, there are many settings where bad actions are hard to distinguish from benign ones.

Although promising, these training-based approaches are face challenges around generalisation:

There are a several interesting training-based approaches for enhancing oversight. Building upon or red-teaming these would be valuable!

Running evals for misalignment ahead of time would be ideal, but eval gaming is already threatening to undermine the validity of these tests:

But there are a bunch of ways in which we're playing on easy-mode today, relative to how difficult oversight like this could be in the future. Chain-of-thought reasoning is currently the most informative monitoring signal, but it is also at the most risk of degradation:

Right now, we have decent oversight IMO. Not great, not terrible. When AIs do bad things, they can often be caught through a range of techniques:

We interviewed a lot of experts and did our own analysis on how it will get harder to tell if AI systems are safe.

Claude 4.5 models also verbalise some minor eval awareness when confessing, usually speculating about why they've apparently been manipulated into having taken misaligned actions.

Prefill awareness does seem like a big issue for Claude 4.5 Opus - it does say "not me" when asked whether it generated the previous messages.

But they still often fail to confess unprompted to the previous misaligned actions, or flag them to the user:

Anthropic and OpenAI models look like they've been getting better at steering back to aligned actions: