Many Transformer explainers start with the finished architecture. We ask why it looks the way it does.

The post explains that keys, queries, and values arise from symmetry problems and that attention heads emerge when we replace a large dynamic weight matrix with reusable transformations. It traces the evolution from 2014’s RNNs, whose fixed‑memory design can’t grow with input, to attention‑based models that keep an expanding memory of past states.

Vaswani’s 2017 paper removed recurrent dependencies, allowing all layers to compute in parallel—only two compute steps regardless of sequence length—while still needing a way to generate weights for unseen input sizes. The solution is dynamic weights: a function that produces weights on the fly, guided by input and position information. Skip connections, inspired by ResNet, further optimize the architecture.

The result is a Transformer that is not arbitrary but a consequence of design pressures: dynamic memory, parallelism, and reusable transformations. Understanding these forces accelerates the search for the next generation beyond Transformers.