Using interpretability to understand, design, and align AI.
San Francisco
Joined August 2024
- Tweets901
- Following46
- Followers28.2K
- Likes1.9K
Pinned Tweet
For years, neural networks looked inscrutable, like tangled knots that needed to be unwound. We no longer think that's the case. AI models are far more interpretable than we thought. We think this is great news for interpretability.
Goodfire's @nickcammarata co-authored Zoom In, the paper that introduced polysemanticity and superposition. He spent months revisiting the vision model organisms that much of interpretability grew out of and studying modern language models up to trillions of parameters. His new essay explains what the early field, him included, got wrong, why there's reason for optimism today, and why there might be a bitter lesson for interpretability.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
“I think our job is to sort of try to think like the neural network.”
@nickcammarata's interview with @techreview today on why he feels hopeful that neural networks can be understood:
An exclusive interview with Nick Cammarata, the researcher whose 26,000-word viral blog post has AI safety folks celebrating. trib.al/2pWmDvF
Goodfire retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
We identified a gene variant behind hearing loss, a big step for interpretability in medicine.
In April, our interpretability system EVEE scored all 4.2M variants in ClinVar, the NIH's variant database. MEPE c.209_212del scored particularly high, so we built a case and had it successfully reclassified as "likely pathogenic."
~2M variants are still "uncertain," but we believe interpretability can help us understand many more!
The variant: ncbi.nlm.nih.gov/clinvar/var…
How we made EVEE: goodfire.com/research/evee-e…
Goodfire retweeted
AI can explain its reasoning. That doesn’t mean we know how it made the decision.
@GoodfireAI co-founder and chief scientist @banburismus_ joins Rocket Drew to explore what AI “brain scans” could reveal that written reasoning misses.
🚀 Watch AI Deep Dive on YouTube: thein.fo/3VJTpzl
Spotify: thein.fo/4e6QSp8
Goodfire retweeted
🚀 AI Deep Dive Episode 5: Can We See Inside an AI’s Mind?
AI can write down its reasoning. But what happens when more of its thinking never makes it onto the page?
@rocketalignment and @GoodfireAI co-founder and chief scientist Tom McGrath explore what AI “brain scans” can reveal beyond written reasoning, how internal signals could help catch cheating, and what it would take to understand—and change—what models learn.
00:00 Intro
00:39 Tom McGrath and AI brain scans
09:01 Written reasoning and hidden computation
17:45 Why AI is hard to understand
24:07 How researchers identify concepts
31:10 What interpretability could change
34:42 Reverse engineering a language model
43:09 The field’s approaches and investment
51:51 Changing models without hiding problems
1:00:10 AI agents studying AI
Goodfire safety infrastructure 🤝 @Baseten inference
Today we're partnering with Baseten to bring real-time safety monitoring to open models. This is an important step toward our vision of ensuring every capable model has a frontier alignment stack.
Open models can come from anywhere. Securing them shouldn’t depend on where you get them.
Today we're launching Project Beacon to make safety a default for open intelligence.
We're partnering with @GoodfireAI to bring inference and in-line monitoring together for the first time, so safety signals are caught proactively.
Read more in our announcement: baseten.co/blog/safe-open-mo…
Our activation monitors detect undesired behaviors using signals from inside a model, so they catch more, run faster, and cost less than text-based monitors.
Baseten customers can monitor for:
- Prompt injection
- Actions outside policy
- Sensitive data exposure
- Cyber misuse
Teams can configure how their applications respond when a concern is flagged, including logging, additional review, refusal, and re-routing.
If you serve or train models and want monitors built into your stack, learn more: goodfire.com/blog/goodfire-b…
We discovered a new class of Alzheimer’s biomarkers with @PrimaMente. Today we're taking the next step by opening a global AI competition to discover new treatments.
Introducing the Alzheimer’s Translation Challenge, led by @PrimaMente and @AlzData. 🧵
The Alzheimer’s Translation Challenge is led by @primamente and @AlzData, with @nvidia, @huggingface, @nebiusai, Talisman Therapeutics, @ultimagenomics, @Cellanome, @PrimeIntellect, and @boltz_bio.
Registration is open now. Competition starts spring 2027: primamente.com/challenges
We built cybersecurity monitors for Kimi K3 and GLM 5.3 — more accurate, 50x faster, & 50x cheaper than an optimized LLM judge.
External red-teaming from @farairesearch found that our monitor greatly reduces universal jailbreaks. 🧵
Read the full post: goodfire.com/research/produc…
If you're interested in deploying these monitors, get in touch: goodfire.com/contact-us
Manifold steering, SAE geometry, and a model that ignores its own clock…our researchers are at @COLM_conf this week to present papers on all three.
Meet them at our booth, Tue-Thu, 1:30-2:30pm. Poster times and happy hour RSVP below 👇
