A researcher reports good news about the nature of neural networks. After many months studying neurons and circuits in image and language models, they have come to believe a network's own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than feared. The canonical evidence of polysemanticity and superposition that convinced much of the field was misread. What looked like arbitrary packing was mostly a motif called recursive generalization, which is much less concerning. This means researchers can study networks directly rather than first untangling them.
There are two main approaches to mechanistic interpretability. The first is rapid normal science on the network itself: examining neurons and circuits, watching data flow through them, and testing explanations experimentally. The second starts from hypothesized features and tries to extract them, often ignoring the original network. The field began with the first approach but fell back to the second because neural networks seemed to be tangled knots. The most famous polysemantic neuron in an image model responded to both cat faces and the fronts of cars.
The leading explanation, superposition, held that networks fold unrelated concepts together when they run out of neurons. Studying such a network required training a second model, such as a sparse autoencoder, to approximate it unfolded. However, the researcher found that superposition was the wrong explanation for Inception V1. Architectural problems made later layers overflow, and weak interpretability tools amplified the apparent incoherence. Ordinary architectural improvements, with no change to model size or training data, made 99-100% of the last layer's neurons easily interpretable.
Trained versions of the model across a 300× range in size remained broadly legible at every scale. With less room, neurons cover coarser concepts; with more room, they differentiate into specific ones. The researcher concludes the problem was an ordinary learning failure that is easy to fix, and that the canonical vision examples of polysemanticity and superposition were errors too.
Source: Hacker News · Summarized by HeadlinesBriefing