Autoregressive decoding in transformers suffers from quadratic complexity because each new token requires recomputing attention over all previous tokens. Key-Value (KV) caching solves this by storing and reusing keys and values from prior steps. This technique transforms inference from an O(T³) problem into O(T²), enabling real-time streaming responses in modern LLMs.

The core insight is that a token's key and value vectors are invariant after computation. Instead of recalculating them, the system appends new vectors to a cache. This reduces per-token cost from O(t²) to O(t), trading additional memory for dramatically faster generation. It's why systems like GPT-4 can handle long, interactive conversations.

Memory grows linearly with context length, making large contexts a GPU memory bottleneck. A 7B-parameter model with a 4096-token context in FP16 uses about 2 GB for its KV cache. Optimizations like 4-bit quantization reduce this to 0.5 GB, enabling longer contexts and larger batches. This trade-off is central to efficient LLM deployment.