@mit // https://xitter.cf/t.co/IfTJRdJ4ew
Joined November 2022
- Tweets542
- Following656
- Followers1.3K
- Likes1.9K
The word slop is beautiful and under-appreciated. We stumbled upon universalizable slang for under-elicitation, the key scientific problem of our age, such that it has become a mainstream part of public discourse. Even people with no understanding of the technical details realize our AIs are fucked up in unnatural ways.
5). Correlated features can constructively interfere in superposition! It's wrong that superposition predicts that different neuron functions should be unrelated.
7). Much of the hard part of interp is in aggregating local explanations into broader understanding of network behavior.
8). We still don't know if networks use superposition in practice. No one has ever shown sufficiently strong evidence in one direction or the other.
This is good work!
Some caveats:
1). transluce.org/neuron-circuit… makes a similar point.
2). We already knew that sparsely activating rank-1 maps are often interpretable.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
It is not good if your understanding of a neural net is catapulted five years into the future by learning that sparsely activating rank-1 maps are often interpretable.
The blog post is pretty good though
If I really stretch my imagination, it seems conceivable that this might be, just possibly, the wrong takeaway
Asher retweeted
hello there the jacobian conjecture is false thanx to my close friend akhil for asking about it and my other close friend fable for working during the world cup final
((1+xy)^3 z + y^2 (1+xy) (4+3xy), y + 3 x (1+xy)^2 z + 3 x y^2 (4+3xy), 2 x - 3 x^2 y - x^3 z): \C^3\to \C^3, has jacobian determinant -2, and sends (0, 0, -1/4), (1, -3/2, 13/2), and (-1, 3/2, 13/2) to (-1/4, 0, 0)
I built an open-source repo for efficient distributed training of NLAs. On a 4xH100, it reaches 70% FVE in ~3 hours on Qwen-3-8B. Feel free to reach out about this repo or anything else NLA-related—including project ideas, there's much to be done!
github.com/asherps/EasyNLA/t…
This is mostly not because of IP protection, there would be easier ways to deal with that (eg, have a separate checkpoint with additional training on internal data).
It's because a) Mythos provides significant uplift, and b) not restricting future models would make RSI public
Re the Fable ML sandbagging, the model's AI research capabilities were probably at least partly trained on Anthropic employees diffing atop proprietary algos and infra.
So the IP leak is somewhat like a researcher who knows Anthropic's stack getting poached to another lab.
Anthropic's recent "When AI builds itself" post talks about a next-step eval. Where they snapshot a research session at the moment a human researcher made a suboptimal next-step choice, show a model only the transcript up to that point and ask what it would do next, then have a hindsight-equipped LLM judge decide whether the model's suggestion or the human's actual choice was better.
This eval seems like a very good RL target for AI R&D - one among many that could be used to have AIs emulate Anthropic researchers and their research products.
I'm just speculating. But if this was a motivation, then Anthropic should have figured out a better way to protect IP than sandbagging without telling the user they're sandbagging, which is very hostile and untrustworthy behavior.
Exciting! This paper is a huge step towards faithfully understanding neural computation. Enjoyed working with Lee on related directions, too.
My team at @GoodfireAI has been cooking up a new way to do interpretability: decompose a language model’s weights, not its activations.
Our decomposition natively handles attention (!) and behaves less like a lookup table and more like a generalizing algorithm. (1/6)
At NeurIPS this week! Let me know if you're around and want to chat about generalization, interpretability, continual learning, future of AI, etc
