A mid-sized model’s weights fit comfortably, yet concurrent traffic triggered CUDA out-of-memory errors while GPU utilization stayed low. The key-value cache, not model size, consumed the headroom. Before adding a GPU, the author enabled paged allocation and prefix caching, lifting the ceiling and exposing a memory-layout problem.

Naive serving stacks reserve one contiguous block per request for the maximum sequence length. A request producing 200 tokens against a 4,096-token reservation leaves most of it unused. A v LLM analysis found that naive KV cache management wastes 60 to 80 percent of reserved memory. The cache grows with concurrent requests, so traffic spikes—not longer prompts—push servers over the memory cliff.

GPU memory is shared by fixed model weights, transient activations, and the variable KV cache. Each token costs roughly 320 KB for Llama 3.1 70B at BF16, with 80 layers, 8 key/value heads, and a head dimension of 128. Grouped-query attention reduces the key/value head count and cache size.

At 128 concurrent requests holding 4K-token contexts, the cache alone needs about 160 GB. The 70B weights need 140 GB at BF16, so cache can outgrow the model. v LLM reserves cache as max concurrent sequences times max sequence length; raising either multiplies the reservation and can trigger startup OOM.