Transformer-based LLMs can draw information from any previous token during generation, but need to be selective. This mechanism can be visualized by tapping or hovering over generated tokens to see which past tokens influenced them.

The visualization calculates attention weight scaled by value vector magnitude, aggregated across heads and layers, then controls opacity of previous tokens. Despite simplification, interesting patterns emerge. For example, in the "Office Move Summary" prompt, copied text like addresses and dates stand out because the generated token draws heavily from source data.

This addresses a puzzle: if LLMs predict tokens probabilistically, why are they so good at copy-pasting? The mechanism shows the model doesn't predict entire sequences from limited internal states; it accesses all past tokens and decides which to draw from, reducing error probability.

In another example, hovering over "remain" shows it draws from "work" and "stay the same," combining information from different phrases. The visualization uses a React app with Transformers.js, but required modifying the .onnx file to expose internal values, and pre-generated prompts for instant viewing.