TokenTown offers a unique isometric city visualization to demystify the workings of transformer language models (LLMs). Each district in the city represents a stage in the LLM process, with a _hidden state_ convoy traveling through. The convoy is broken down into tokens at the docks, converted into vectors at the foundry, stamped with positional information, and then circulates through the 'layer ring'—comprising attention, residual, feed-forward, and residual connections—for each layer.

Key components are computed live in the browser: tokenization, embedding lookup, positional encoding, Layer Norm, multi-head scaled dot-product attention with causal masking, residual adds, GELU feed-forward, and sampling. While actual LLMs use thousands of dimensions and dozens of layers, TokenTown scales these down to 12 dimensions, 2 attention heads, and 2–12 layers for clarity. The weights are randomized, and the output is guided by a bigram prior and sharpened attention scores to mimic trained model patterns, making the mechanism, not the generated text, the primary lesson.

The initial tour explains each district, with stops lasting between 9 and 26 seconds. Once explained, the city runs at a faster pace. Controls include Space to pause, S to advance stage by stage, R to reset, and a speed slider. Users can drag to pan, scroll to zoom, and click districts for detailed explanations. The camera can follow the convoy or be manually controlled.