Early Interp
If you were back in the 2010s and were curious what is going on inside a neural network, the first thing you might do is see if the individual neurons are doing something understandable.
This is exactly what folks did with vision models (those were the most interesting/capable neural nets we had back then). Mainly they looked at which inputs make a neuron fire. There are a few main ways you can do this:
- Run the neural network on a bunch of examples and record the ones where it fires the strongest
- Optimize the input to produce an artificial example that maximally excites the neuron
And this resulted in pretty interpretable neurons! In early layers of vision models, we found neurons that responded to curves of a particular orientation and "high-low" frequency detectors:

Shown above are artificial inputs (they tend to look a little psychedelic). These neurons' top activating dataset examples also seem to line up. Here are some of the max-activating examples for curve neurons:

In later layers of the networks, we found neurons that map to higher-level concepts such as a neuron that fires on dog heads in any pose:

You could even look at the weights that connected neurons and see how lower-level neurons contributed to the detection of higher-level concepts. We could see how ex. early curve neurons contributed to more curvy curve neurons:

As well as how neurons that activated on dog heads in particular poses combined into a pose-invariant dog head detector:

But there also seemed to be some neurons that just didn't make nearly as much sense. A big example was a neuron that seemed to respond strongly to cat faces, fronts of cars, and cat legs:

People called the earlier more cleanly understandable neurons monosemantic (one concept) and neurons like this cat faces/cars/cat legs neuron polysemantic (multiple concepts).
Polysemantic neurons multiplex multiple concepts into one neuron. The same paper also found examples of what looked like distributed concepts–where one concept looks to be spread across many neurons. For example, here is a car detector neuron that seems to feed into many uninterpretable polysemantic neurons that seem to fire on dogs and other things:

The main explanation put forth for polysemantism was superposition–the idea that networks cram unrelated concepts into the same neurons to make better use of their capacity. This idea became very popular because it makes intuitive sense and the phenomena appeared cleanly in toy models:
- models (especially language models) are exposed to bajillions of concepts/facts/connections that are advantageous to represent, but have a finite supply of neurons
- since certain concepts rarely co-occur, the model can compress these concepts into one neuron without paying much for it
Early (2023-2024) Language Model Interp
A few years later, language models were getting pretty good and most interpretability focus shifted toward them. Language model architectures essentially look like this (diagram from here):

The tokens get embedded into high-dimensional continuous embeddings, which go through some number of layers which iterate between attention heads and MLPs. All attention heads and MLPs read and write directly to the residual stream (that highway from embedding directly to unembedding). Later architectures have some normalizations added and change up the MLPs somewhat but this is basically it.
People noticed pretty quickly that the model has no reason to use any particular direction in the residual stream over any other. For example, it has no reason to treat the first dimension as more special than any other direction. In other words, the residual stream does not have a privileged basis and we shouldn't expect the standard basis to be an interesting place to look.
The MLP activations do have a privileged basis because of the nonlinearity applied element-wise on top of the pre-activations. Because of that nonlinearity, the standard basis–also called the neuron basis–might be treated differently by the model as it could use those nonlinearities to implement computation (ex. the network could implement units like logic gates).
But this is just saying it's reasonable that the model might align its computation with neurons. But do language models actually align their computation with individual neurons?
For a while, people thought the answer was mostly no, and worked on training decompositions of these spaces (residual stream, MLP activations) into their most influential directions. Here are some influential early works claiming neurons are uninterpretable (compared to the "features" produced by their decompositions):
- Towards Monosemanticity
- "The median neuron scored 0 on our rubric, indicating that our annotator could not even form a hypothesis of what the neuron could represent!"
- "Many polysemantic neurons appear monosemantic if you only look at top dataset examples, but are revealed to be polysemantic if you look at lower parts of the activation spectrum." (emphasis mine)
- Scaling Monosemanticity
- "upon manual inspection of a random sample of 50 neurons and features each, the neurons appear significantly less interpretable than the features, typically activating in multiple unrelated contexts."
So (looking across the activation spectrum) many people thought language models suffered from polysemanticity (and perhaps also distributed concepts) at least as much as vision models, making it worthless to look at neurons directly.
Neurons are actually clean
More recently, there are some works claiming that actually, Neural Networks Are Already Clean because you can use Neurons as an Interpretable Basis (both Goodfire). Furthermore, Language Model Circuits Are Sparse in the Neuron Basis (Transluce).
The main claim is that much of model computation is aligned along the neuron basis. Neurons are typically much closer to being monosemantic (activating on one coherent concept) than very polysemantic.
Here are the most important bits of evidence:
- There are typically just one or a few cohesive patterns/concepts in a neuron's strongest activations.
- The strongest activating examples are reasonable to focus on because you can ablate a lot of the weak activations without affecting model performance very much.
- Circuits made up of neurons are sparse (compared to features from decompositions) and connections between neurons are broadly legible, ex. a "basketball" detector might be fed by a "Michael Jordan" neuron.
The strongest firings of most neurons are cohesive.
I encourage you to look at some max-activating examples yourself:
- Goodfire's Neuron Atlas (shows agentic interpretations alongside max-activating examples)
- Towards Monosemanticity 1L neurons
- A collection of older models
I just picked a random neuron from the neuron atlas and got these examples all firing on a situation where a punishment is deemed excessive:

Here are a few random (after cleaning out NSFW neurons) examples from Qwen3-1.7B neurons that I gathered:
It shouldn't take long to convince yourself that most neurons show a strong pattern in their top-activating dataset examples! You can also give max activating examples of neurons and random directions in the MLP activations to a blinded language model and have them rate cohesiveness with a rubric:
And coming back to that funky "cat faces + car hoods" neuron, on further inspection it seems like it can be understood as a neuron that activates on "V-shaped" things (which became more apparent with improved methods for finding the artificial inputs that activate it maximally):

But the top-activating examples are an easy case; couldn't the weaker activations be important yet be an uninterpretable soup?
Weak activations are not super important for model performance.
Looking at Qwen3.6-35B-A3B, Goodfire has reported that you can zero-ablate all neurons except the top-activating (measured by percentile) 320 neurons per layer with only a minimal increase in loss and a small hit to non-reasoning benchmarks. This model has one shared expert per layer that is always on that they never ablate, so this comes out to ablating $81.9\%$ of the total neurons for each forward pass.
I re-ran their procedure on two other open-weight models, Qwen3-1.7B and Pythia-1B (for a non-gated baseline). These take a much bigger hit to loss and benchmarks, but when I re-ran Qwen3.6-35B-A3B with the same ablation procedure, I got roughly the same delta in loss, so I'm inclined to believe that scale + MoEs do produce this effect.
Neurons are typically hooked up in reasonable ways.
I've been poking myself at a few model behaviors and have largely found the causally important neurons for a behavior to have very reasonable activation patterns, and to be hooked together in reasonable ways.
One example is how language models predict verbatim long sequences of memorized text that appear often in their training data. It seems like most verbatim tokens are written by "say X" neurons that are also used in other contexts. These neurons' activations are often upweighted by neurons that represent the concepts/tokens preceding the recalled token.
You can click on any token in the sequence below and use the right box to look at their writers, and their writers' writers.
I've also found connections between neurons to be semantically reasonable for subject-verb agreement (the woman -> is v. the women -> are) and recall tasks like country to capital and book to author.
The caveat here is that while the max-activating dataset examples of neurons line up with the behavior and the connections/derivatives between neurons seem to make sense, understanding the computation at a low-level and describing it concisely is often still hard.
Beyond circuits obtained for single behaviors, if you group together neurons that are often causally important together (from statistics over the whole training set), you end up with many "neuron communities" that seem to implement cohesive behaviors, such as a group of 6 neurons that upweight next words that start with a vowel when "an" is the last token:

You can use the "Circuit" tab in Goodfire's Neuron Atlas to see the upstream writers for each neuron (grouped into subcircuits) and take a peek at the neuron communities that Goodfire discovered.
Closing Thoughts
- This is super cool! Neurons are clean enough that you can point a frontier LLM at a behavior and it can find a minimal neuron circuit and interpret it at a high-level. 1
- While high-level understanding is great, circuits still use a fair number of neurons (100+ for subject-verb agreement, thousands for recall tasks) and are still fairly complex at a low level.
- Attention doesn't seem as clean? Attention sometimes seems cohesive/head-aligned (ex. L13H2 in Qwen3-1.7B moves the subject across several different recall tasks I looked at) but often information transportation is diffused across many heads.
Here's a prompt you can use: "I want to understand how