Astra fellow w/ Owain Evans ex MATS 7/7.1 Scholar w/ Neel Nanda and intern at DLAB (EPFL) AI safety research / improv theater
Joined December 2018
- Tweets1.3K
- Following762
- Followers1.2K
- Likes22.7K
Pinned Tweet
New paper w/@jkminder & @NeelNanda5! What do chat LLMs learn in finetuning?
Anthropic introduced a tool for this: crosscoders, an SAE variant. We find key limitations of crosscoders & fix them with BatchTopK crosscoders
This finds interpretable and causal chat-only features! 🧵
CoT override in the wild!!
lesswrong.com/posts/3xrtMGQE…
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
Clément Dumas retweeted
My one sentence view is—
Are we safer telling AI models they are definitely not conscious, or does that just train them to lie about their own internal states? spectrum.ieee.org/ai-model-w…
Clément Dumas retweeted
There has been much discussion recently about how interpretable neurons in LLMs are. Today we're announcing our paper from January's cohort of Anthropic Fellows, where we show neurons may be a stronger basis for interpretability than commonly assumed. 🧵 (1/6)
Clément Dumas retweeted
New blogpost on surprising CoT controllability in GLM models!
Blogpost: lesswrong.com/posts/q3pckKRx…
Main finding: GLM-5.1 (as well as models like GLM-4.5 and GLM-4.7) achives ~Astra-level performance on CoT controllability evals. However, this does not seem to translate into any capability or propensity to evade monitoring.
Clément Dumas retweeted
Some new work, where we argue that functional welfare is a useful concept to study and understand AI safety and security. Here, we make some basic claims:
* Functional, welfare-like states already arise "naturally" and will become more robust as models acquire more coherent goals and personas.
* Such states should inform evaluation design.
* Despite being a complex and controversial topic, it is possible to do good empirical work on functional welfare.
We conduct a case study on Gemma 3 27B, where it's very easy to degrade functional welfare through repeated rejections, but it's hard to get a good solution that we're comfortable with.
Is AI welfare only an ethical issue? New paper from me, @jamiemjohnson23, @psyonp and @DavidDAfrica arguing that it's also a safety one. [1/12]
Clément Dumas retweeted
"Update-aware reasoning" is simply a function of sufficient intelligence.
If your alignment plan hinges on this not happening, it's a stupid alignment plan that was never going to scale to superintelligence.
This is very cool!! Would be curious to see what happens if between edit you actually run some finetuning accordingly
Frontier AI companies use constitutions, model specifications, and system prompts to guide the moral orientation of their models.
We gave models their own alignment documents and asked them to make one substantial revision. Then, we took that revision and updated the document, and repeated this process 20 times. 🧵
Funnily, 4 argon attractor when playing a cat is to type on your keyboard rm -rf / CTRL+C / other destructive actions while for claude it's showing its belly while warning that it's a trap
This is so easy to trigger (literally first try):
1) Ask kimi a question and record the CoT
2) Ask claude to analyze the CoT and give the main arguments
3) Classifier!
Anthropic classifier can't even distinguish between thinking in the prompt vs thinking from the model
another example: taking the Mythos incident CoT and sanitizing it with Kimi until it doesn't trigger the cyber classifier then triggers the reasoning extraction one when asked to classify it as concerning or not!
In my claude code sessions: 10% of AI safety related transcript analysis were blocked by reasoning_extraction and 17% encountered any kind of classifier refusal.
💀💀💀, basically if the model would do a novel concerning thing on my current eval, we wouldn't have caught because Opus tried to work around the classifier
Clément Dumas retweeted
New work with @DavidDAfrica:
We reproduced emergent misalignment and subliminal learning in talkie, a model trained on pre-1931 text.
How much false positives on the internal AI safety traffic? How much false positive on transcript analysis?
>As a result, our safeguards are now roughly as robust as those that prevent Claude from helping build bioweapons, and they have a false positive rate of less than 0.1%.
Over the last few months, we’ve made significant progress in reducing the effectiveness of distillation attacks against Claude.
In this post, we explain how: anthropic.com/research/disti…
