repo.ing

@treetowntree and i created neuron lens: a database of every single neuron and attention head in qwen 3 8b and gemma 4 12b read using the R-lens (a J-lens variant) and what we call the "S-lens". you can find it at neuron-lens.com/, along with the tools we used to make it. using the R-lens to read neurons' input and output projections turns out to be very effective at determining what a neuron does. we're able to legibly read what a majority of neurons do when fired: the lens qualitatively predicts the result of steering over 90% of the time! the same trick lets us read attention heads' QK circuits and OV circuits, showing us both what they attend to and what they do with it. however, the R-lens has two important flaws: its accuracy degrades in early layers, and it can't separate the MLP block and the attention block. we found ways around both these issues. we first modified the r-lens by fitting it on input tokens rather than output tokens to get the s-lens, which is accurate in early layers but inaccurate in late layers. we then fitted R- and S-lenses to the activations after attention but before the MLP to get "mid-lenses", which are much better at deciphering MLP inputs and attention outputs. using both of these techniques, we were able to interpret most of the models' layers.

There is a single neuron that, when knocked out in fruit flies, makes them unable to stop having sex (Crickmore & Vosshall, Cell 2013)

Aha we found the meth recipe neuron in Talkie 1930.