Mechanistic interpretability: reverse engineering neural networks
Mechanistic interpretability seeks to understand neural networks by breaking them into interpretable components, such as features and circuits. Techniques like feature visualization, dictionary learning, and attribution graphs allow researchers to trace how information flows through a model. This approach aims to explain why a network behaves as it does, supporting safety and reliability.
.jpg){kind=link}
What is mechanistic interpretability?
Mechanistic interpretability is the attempt to reverse engineer neural networks by studying the internal components that produce behavior. Distill's introduction to circuits explains that by examining the connections between neurons, researchers can find meaningful algorithms in the weights of a network. This approach treats individual neurons and their weights as objects worthy of investigation, rather than viewing the network as an opaque black box.
Early work on visualizing convolutional networks demonstrated that intermediate layers could be inspected to understand feature extraction. The study Visualizing and Understanding Convolutional Networks introduced a visualization technique that reveals how different layers respond to input patterns, helping researchers see what each layer has learned. This laid groundwork for later interpretability tools.
The goal is not merely to describe behavior but to identify the specific algorithms a model implements. By zooming into the details, researchers hope to build a picture of how a network represents facts, performs reasoning, and potentially makes errors.
Feature visualization and attribution
Feature visualization generates examples that maximize the activation of a neuron or channel, showing what the network is looking for. The feature visualization article on Distill explains that optimization can reveal the causes of behavior by separating them from correlational patterns. Starting from random noise, the process iteratively tweaks an input to activate a target unit.
Attribution studies which parts of an input are responsible for a particular activation. This complementary approach answers questions about what the network pays attention to when making a decision. Together, feature visualization and attribution form two major threads in interpretability research.
These tools can be applied at different scales, from individual neurons to entire layers. For example, a whole layer can be studied using a DeepDream style objective that searches for images the layer finds interesting.
Polysemanticity and superposition
A major obstacle is polysemanticity: many neurons respond to mixtures of unrelated inputs. The Anthropic report on monosemantic features notes that a single neuron in a vision model may respond to both cat faces and car fronts, making it hard to reason in terms of individual neurons.
Superposition is the hypothesized phenomenon where a network represents more features than it has neurons, by assigning each feature a linear combination of neurons. The Toy Models of Superposition paper provides a toy model where polysemanticity arises as a result of storing sparse features in superposition, demonstrating a phase change and a connection to the geometry of uniform polytopes.
To address this, researchers use sparse autoencoders to decompose activations into a larger set of interpretable features. The same Anthropic report describes training sparse autoencoders on a one-layer transformer, extracting thousands of features that are more monosemantic than the original neurons. This dictionary learning approach yields features that can support circuit analysis.
Circuit tracing and attribution graphs
Attribution graphs are a tool for tracing the chain of intermediate steps a model uses to transform an input into an output. The study of a large language model's biology uses attribution graphs to hypothesize mechanisms and then test them with perturbation experiments. This allows researchers to see, for instance, how a model plans rhyming words in poetry or performs multi-hop reasoning.
These graphs map connections between features, akin to a wiring diagram. By following the paths that contribute most to an output, one can identify the internal representations that drive behavior, such as the model internally representing "Texas" before naming Austin as the capital.
Mechanistic interpretability remains an active field with many open questions. The ability to trace circuits in large models is a step toward understanding and validating their reasoning, which is important for safety and reliability.
| Technique | Primary question | Output | Example context |
|---|---|---|---|
| Feature visualization | Which input maximally activates a unit? | Optimized images | Neuron and channel analysis |
| Attribution | Which parts of an input cause activation? | Saliency maps | Locating evidence in input |
| Sparse autoencoder | What are the underlying features? | Interpretable feature vectors | Decomposing polysemantic activations |
| Attribution graph | How do features interact? | Causal graph | Tracing reasoning paths |
Frequently asked questions
What is a polysemantic neuron?
A polysemantic neuron responds to multiple unrelated concepts, which makes interpreting individual neurons difficult.
How do sparse autoencoders help in mechanistic interpretability?
They decompose neuron activations into a larger set of interpretable features using dictionary learning, providing a more monosemantic unit of analysis.
What is an attribution graph?
An attribution graph is a mapping of the intermediate steps a model uses to compute an output from an input, showing which features contribute most.
Why is mechanistic interpretability important?
It helps explain why a model behaves as it does, which can support safety, debugging, and trust in model outputs.
How does feature visualization work?
It uses optimization to create an input that maximizes the activation of a specific unit, revealing what that unit detects.