This paper presents initial steps toward reverse-engineering transformer language models through mechanistic interpretability. As models like GPT-3, LaMDA, Codex, Meena, and Gopher scale, their open-endedness creates risks of unexpected harmful behaviors. Mechanistic interpretability aims to reverse engineer detailed computations, similar to reverse engineering binaries into source code, potentially providing systematic approaches to safety problems.

The authors study attention-only transformers with two layers or fewer, contrasting with GPT-3's 96 layers alternating attention and MLP blocks. By conceptualizing transformer operations in a mathematically equivalent framework, they gain significant understanding of internal operations. Key findings include: zero-layer transformers model bigram statistics accessible directly from weights; one-layer models are ensembles of bigram and "skip-trigram" (sequences like "A… B C") models with directly accessible tables; and specific "induction heads" explain in-context learning in small models, developing only in models with at least two attention layers.

The framework includes concepts like virtual weights, residual stream as communication channel, independent additive attention heads, path expansion tricks, and QK/OV circuit analysis. While not yet applied to larger models in this first paper, forthcoming work shows the framework and induction heads remain partially relevant for larger models, though full reverse engineering remains distant.