some thoughts on nick's interpretability post 1) nick argues that most neurons represent a single concept or a generalization of a single concept in well trained networks 2) he says that the field generally believed that neurons stored many concepts because people didn't know what kinds of features to look for 3) he says the better interpretation of neurons is as flow control which separate incoming data points toward different downstream neurons 4) so, you should look for a neuron that separates different types of dogs or separates objects in different poses; rather retain a dog neuron or a train neuron 5) and, that early efforts were hampered by poorly trained networks that had neurons without sufficient gradient applied / reward hackable tools 6) he thinks that most modern networks have clean features and that this will get better over time just as a result of better training techniques 7) it's an interesting idea and i am at least interested, i look forward to reading his longer paper; i do think he presents some interesting evidence 8) one early theory was that there are fewer neurons than concepts, so models need to pack a lot of ideas in one neuron, and they do so with nearly orthogonal vectors; so, we should expect many per neuron 9) he shows that on smaller vision models, you can crush different parts of the network, like make the residual stream smaller, and you don't seem to get this 10) instead, you get more general representations in each neuron, more coarse grained features, rather than more polysemanticity; with many ideas in one neuron 11) and, he shows that these representations evolve over the process of training the network, where the model learns some initial feature, then related features on the same neuron 12) one question i had though reading this was, if he is right and most networks are clean, why don't things like saes work better? 13) probably, the best answer that i would have, if he is right, is really just that goodfire can afford to allow claude to be a research scientist over their networks 14) and, because claude can investigate all the neurons, try to trace the circuits, etc... more than any human, and they are willing to just go crazy on it, they can find the circuits 15) and, as claude gets labels more neurons with confidence, it becomes easier to figure out what the remaining neurons do and so on 16) nick seems to gesture to this, even if he isn't super explicit, his future vision seems to be that claude as research scientist investigates circuits like an experimenter over the network 17) and that, rather than trying to label absolutely everything, you let claude investigate certain problems, like a human biological experimenter 18) a lot of ai scientist ideas seem to be trending in this direction; because the cost of discovering everything in these fields is high 19) and so you want a scientist to figure out specific things you care about with tool usage, small experiments, etc... and research hypotheses 20) you might notice a tension between nick's as we label everything, it gets easier to label the remainder, and we need a scientist over the network that investigates hypothesis 21) and, i think this tension is real, but i resolve it as he means that you can still do constraint satisfaction within some particular problem space (interested in how he would solve it though) 22) so, i think it's possible that a combination of training getting better, tools getting better, and abundant intelligence really is going to make a big difference here 23) but, i still don't know how to square this with saes being hard to get to work; i've looked at some of these features and they really do look impossible to label fairly often 24) just to clarify this a bit more, let's say we make saes using the residual stream, with a top-k reconstruction loss so each sae represents ~1 thing 25) if the neurons are clean, and the sae is wider than the neurons, shouldn't that also end up clean with enough optimization? 26) same with transcoders, where you replace the mlp layer with a wider layer and do reconstruction loss; if the neurons are clean, shouldn't the transcoder be clean? 27) this is my immediate trepidation in the post... but maybe it's just that these were not labeled with a sufficient claude research scientist budget? or we were not looking for the right thing 28) oh, he also says he doesn't quite know what attention does and doesn't think anyone has a clear read on it, but seeing which neurons connect through attention enough 29) this doesn't seem satisfying to me; i would like a more detailed description of attention and what it is doing if possible; this seems like a gap 30) i think it would be interesting for other people to try this on a smaller scale; train a small mnist model and see what claude can do if you spend $1k tokens 31) but, i think this is possible; i do think it's possible that if you begin to really throw intelligence at the models with the right framing, you can label things 32) oh, he seems to think having access to the checkpoints is very helpful because you can see features generalizing 33) so like, first there is an "iy" neuron, which then triggers for "diy" which then triggers for "do it yourself" and that this is hard to understand without seeing the generalization through the checkpoints 34) one final issue, a lot of people seem to work in interpretability through these canonical examples, like the dog <> truck neuron nick describes (which ended up being a single pose concept) 35) and, this seems untenable to me; a field like interpretability, with access to whole minds, should not need to work through a bunch of canonical examples; 36) they are good for understanding, but they don't feel like science to me; you need more comprehensive sweeps, automated grading, etc... 37) like, there was one very good example where based on an english language description a model could craft a sentence that could trigger a neuron more than any single example in the training corpus 38) that seems to me to be a good example that you probably have a decent (at least starting) understanding of what that neuron does 39) and so, maybe it would be good to see a sweep of that over a network of some size to see how well we can actually say what each neuron does 40) and get a better example of how much stronger automated research agents have gotten in interpreting neurons and how clean their features are 41) nick says in his blog post that an article is forthcoming so it may cover some more of these things