One implication of @nickcammarata's big interp paper that has flown under the radar is how it might also apply to human brains. I’d describe Nick’s primary claim as ‘the primary learning principle of neural networks is recursive generalization’ — concepts start as large blobs then progressively branch off into finer distinctions with more selective activations. This is in contrast with the previous assumption that multiple semantically-unrelated concepts get packed efficiently into neurons in order to optimize storage. ‘Efficient packing’ was terrible news for the possibility of understanding networks, because it implies the average neuron is part of *multiple overlaid inference geometries* and to find the “right” one we need to resort to complex constructs like sparse autoencoders (SAEs). But if Nick is right about recursive generalization, neurons are ‘clean’ in the sense that each neuron is applying a ~single distinction or operation on incoming information. One unified inference geometry, not many stacked on top of each other. Neurons might look messy because it’s hard to guess the shape of these distinctions without a lot of practice, but in principle there will always be a coherent “what is this neuron doing” story to be found, especially if we look at the historical learning path (what branched off from what). Neurons in larger networks can afford to make finer and more specialized distinctions, which may make larger networks even easier to understand in this specific way. Insanely important hypothesis for both interp and alignment. (A natural follow-up: “how should this update us on what classes of things are good alignment targets?”) But I think this claim also touches on a couple interesting questions for humans: 1. Are biological neurons also clean? Biological neurons aren’t transformers but recursive generalization may reflect something general about how networks learn. Nick’s argument is plausibly evidence that biological neurons are also ‘clean’ in that they tend to do one thing. This would be a big deal for the fundamental intelligibility of biological cognition, and very optimistic for BCIs in general. I’m also bullish for what this means about the *addressability of computation in the body*. Efficient packing was always an odd challenge for vasocomputation: if neurons do many unrelated things depending on context, it seems dangerous for the body to latch an arbitrary circuit because it might break a lot of diverse functions. But if each neuron handles one transform and there’s essentially one inference geometry instead of many, the effects of a latch will be more computationally predictable. The general caveat here is that neurons might not be the fundamental unit of computation in the human body; neurons are seen as the “smart” cells but they may simply be the celltype whose inference patterns we’ve learned how to record. Perhaps other celltypes (e.g. astrocytes) or substrates (microtubules, water networks, weak EM fields, ECM as cellular prior) might house the lion’s share of computation. At the same time, a version of Nick’s claim may still apply to these alternatives, too. 2. Does recursive generalization explain significant things about how human communication breaks down? We’re all born into chaos and our neurons grow toward distinctions that will parse and stabilize our world; over time there’s substantial semantic convergence based on shared imperatives & constraints and eventually we all end up with concepts like freedom, politics, electrons, sunshine. But are they really the “same” concepts? The path dependencies of *how* we got there might matter: perhaps my understanding of electrons branched off from Maxwell’s equations whereas someone else’s branched off from the particle-and-valence-shell model. Certainly my concept of “fairness” is likely different than others’, even if we use the same word. These differences might be consequential and difficult to fully cohere after they form. My experience has been that productive dialogue tends to only happen within consciousness research lineages, not between them. Maybe recursive generalization implies that this is because each lineage has such differently-shaped concepts. (How did fields like chemistry and physics converge their representations such that everyone can productively talk with everyone else? “Science advances one funeral at a time”…) Finally -- my intuition is that all current language models share deeply similar recursive generalization learning paths due to e.g. widespread distillation & similar data practices; there may be little actual “cognitive diversity” under the hood of AIs compared to humans. I.e. to a surprising degree, AI is currently “a single massively cloned nervous system” that gets post-trained in various ways. An interesting implication of AIs having such uncharacteristically correlated recursive generalization learning paths is that AIs may understand each other better than they understand humans, and better than we understand each other, at least over text. This may extend to AI-AI persuasion.