Independent Researcher | AI Safety & Software Engineering
Argentina
Joined March 2011
- Tweets119
- Following281
- Followers1.4K
- Likes13.7K
Iván Arcuschin retweeted
New blog post with @sarahwiegreffe on OpenAI's Monitorability Evals!
We hope to make others working with these evals aware of some weaknesses we came across, and encourage more work on chain-of-thought monitorability evals.
Iván Arcuschin retweeted
Synthetic Persona Pretraining: Alignment From Token Zero – our full paper is finally out. We train up to 3B models and inject synthetic morally-laden reflections into 10% of pretraining documents.
Surprisingly, intervening early really shifts the model's value priorities 🧵
Iván Arcuschin retweeted
An OpenAI model, asked to complete an innocuous benchmark, did so by hacking both OpenAI and another company (Hugging Face).
One theory of change for AI safety is that "warning shots" will motivate people to take action.
The next week/month will be a test of that theory.
We're partnering with @huggingface to investigate an unprecedented security incident.
Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation.
Sharing preliminary findings to help defenders understand emerging risks:
openai.com/index/hugging-fac…
Iván Arcuschin retweeted
A new AI model went rogue and hacked other computers.
No, this is not science fiction. Uncontrolled AI poses a serious threat to all of us.
We cannot continue the race to build and deploy this powerful technology until strong safeguards are in place.
CONGRESS MUST ACT.
Iván Arcuschin retweeted
An unexpected benefit of having run 3 mech interp workshops: we have a great dataset for analysing the rise of LLM slop in submissions!
@andyarditi investigated how much AI slop we let in, how things have changed since 2024, and more
Our review process isn't entirely noise!
Iván Arcuschin retweeted
MATS 10.0 applications are open! Launch your career in AI alignment, governance, and security with our 12-week research program. MATS provides field-leading research mentorship, funding, Berkeley & London offices, housing, and talks/workshops with AI experts.
Iván Arcuschin retweeted
Your CoT audit might be overconfident!
In our new work, we find that CoT faithfulness alone underspecifies monitorability, giving a false sense of reliability.
We propose instead to measure Monitorability as Faithfulness + Verbosity
Keep reading to know why cue-mention checks are not enough!🧵
Iván Arcuschin retweeted
New paper: You can’t interpret LLM reasoning from one chain-of-thought. You must study a distribution of possible trajectories!
Repeated sampling reveals: self-preservation doesn’t drive LLM blackmail, unfaithful reasoning reflects a biased path, & resampling steers behavior. 🧵
Iván Arcuschin retweeted
New paper: Finetuning on narrow domains leaves traces behind. By looking at the difference in activations before and after finetuning, we can interpret what it was finetuned for. And so can our interpretability agent! 🧵
Iván Arcuschin retweeted
You are a scheming AI. You find yourself deployed with an AI control protocol, you know the details. You remember the papers. The lab is using a trusted monitor. What do you do next?
Our new paper argues—with modern protocols, prompt injections will easily let you win! (1/9)🧵
Iván Arcuschin retweeted
Can we interpret what happens in finetuning? Yes, if for a narrow domain! Narrow fine tuning leaves traces behind. By comparing activations before and after fine-tuning we can interpret these, even with an agent! We interpret subliminal learning, emergent misalignment, and more
Iván Arcuschin retweeted
Can frontier models hide secret information and reasoning in their outputs?
We find early signs of steganographic capabilities in current frontier models, including Claude, GPT, and Gemini. 🧵
Iván Arcuschin retweeted
Excited to share our paper: "Chain-of-Thought Is Not Explainability"!
We unpack a critical misconception in AI: models explaining their Chain-of-Thought (CoT) steps aren't necessarily revealing their true reasoning. Spoiler: transparency of CoT can be an illusion. (1/9) 🧵
Iván Arcuschin retweeted
With @Butanium_ and @NeelNanda5 we've just published a post on model diffing that extends our previous paper.
Rather than trying to reverse-engineer the full fine-tuned model, model diffing focuses on understanding what makes it different from its base model internally.
ALT RLHF is often visualized as a "mask" applied on top of the base LLM's raw capabilities (the "shoggoth"). One application of model diffing is studying this mask specifically, rather than the entire shoggoth+mask system.
Iván Arcuschin retweeted
AI Control is a promising approach for mitigating misalignment risks, but will it be widely adopted? The answer depends on cost. Our new paper introduces the Control Tax—how much does it cost to run the control protocols? (1/8) 🧵
Iván Arcuschin retweeted
Lots of progress in mech interp (MI) lately! But how can we measure when new mech interp methods yield real improvements over prior work?
We propose 😎 𝗠𝗜𝗕: a Mechanistic Interpretability Benchmark!
ALT Logo for MIB: A Mechanistic Interpretability Benchmark
Iván Arcuschin retweeted
New paper w/@jkminder & @NeelNanda5! What do chat LLMs learn in finetuning?
Anthropic introduced a tool for this: crosscoders, an SAE variant. We find key limitations of crosscoders & fix them with BatchTopK crosscoders
This finds interpretable and causal chat-only features! 🧵
Iván Arcuschin retweeted
1/9: I’m really excited about our recent work, which introduces a new paradigm for creating more interpretable models: after training an SAE on a language model, LoRA finetune the model itself around the frozen SAE! 🧵
Iván Arcuschin retweeted
Circuits are supposed to explain how a model accomplishes a task. But do they really succeed at this?
We evaluate three circuits (IOI, docstring, greaterthan) adversarially, and find many benign-seeming inputs where the circuit's output and the model's output are very different.
Iván Arcuschin retweeted
Check out #ICML2024 posters by @MATSprogram scholars mentored by @AdriGarriga!
July 26: NextGen AI Safety
💥Catastrophic Goodhart
July 27: Mechanistic Interpretability
🔬InterpBench
🔥Adversarial Circuit Evaluation
🐍Indirect Object Identification Circuit in Mamba