I wasn't expecting geometry to show up. I was reproducing a small piece of Anthropic's 2022 interpretability paper, "Toy Models of Superposition" [1], mostly because the central claim sounded implausible enough that I wanted to check it myself rather than take it on faith. The claim: a neural network can represent more features than it has dimensions to work with, by packing them in at angles to each other and tolerating a bit of interference.

Ask it to compress five things into two dimensions under the right conditions, and it doesn't pick two winners and give up on the rest. It arranges all five into a perfect pentagon. I didn't have PyTorch or any autograd library available, and no internet access to install one either, so everything below is plain NumPy, and I derived the backward pass by hand.

That turned out to be the right kind of annoying. Deriving the gradients yourself forces you to actually understand what the model is doing to the data, rather than trusting a .backward() call you've never had to think about. If you have a math background and you've never hand-derived backprop through even a tiny network, I'd genuinely recommend it as an exercise.

It's ten minutes of chain rule that makes everything downstream click. All the data in this post is synthetic. I generate it myself in code, there's no external dataset involved, which is also how the original paper does it.

All images, unless otherwise noted, are by the author. The problem this is trying to explain Here's the motivating puzzle, and it's a real one in interpretability research. If you look inside a trained neural network hoping to find individual neurons that cleanly represent individual concepts, one neuron for "is this a dog," one for "is this red," you mostly don't find that.

Instead you find neurons that seem to respond to several unrelated things at once, a neuron that fires for both cat faces and the front ends of cars, say. This is called polysemanticity, and it makes interpretability much harder, because you can't just read off what a network "believes" by inspecting individual units. The paper's proposal is that polysemanticity isn't noise or failure.

It's a real strategy the network uses on purpose, because it has more concepts to represent than it has neurons to represent them with, and most of those concepts are rarely active at the same time. If two features are almost never "on" simultaneously, the network can afford to let them share a direction in activation space, since the interference only costs something on the rare occasions both happen to fire together. This packing strategy is what the paper calls superposition, and its toy model is designed to be the simplest possible setting where you can watch it happen and actually measure it.

The model, and the math I had to work out to train it The setup is small on purpose. You have n synthetic features, each one a number between 0 and 1 that's zero most of the time (that's the sparsity) and nonzero the rest of the time. You compress them down through a bottleneck of m hidden dimensions, where m is smaller than n, and then try to reconstruct the original features on the way back out through a ReLU.

Concretely, with a single weight matrix W of shape (m, n) used for both the compression and the reconstruction:h=W⋅xx^=ReLU(W⊤⋅h+b)h = W{cdot}x\ \hat{x} = \text{ReLU}(W^\top{\cdot} h + b)Training minimizes a weighted squared error between xxand x^\hat{x}, where each feature i gets an importance weight Ii I_i, so the network is told some features matter more to get right than others: L=∑i Ii⋅(xi−x^i)2\m...