Large Language Models (LLMs) face escalating costs with longer prompts due to the KV cache, a form of working memory storing key and value vectors for input tokens. This cache grows with each token, consuming significant GPU memory, especially for models like Llama 3 70B with context lengths of 128,000 tokens, requiring approximately 40 gigabytes.

The KV cache optimizes token generation by storing previously computed vectors, avoiding redundant calculations. However, accessing this cache during the 'decoding' phase, where the model generates output token by token, becomes a bottleneck. Each token requires reading the entire cache, making the process memory-bound and costly as the cache size increases.

Optimizations include architectural changes like Grouped-Query Attention, which reduces the number of key-value heads, and Multi-Head Latent Attention, which compresses vectors. For existing models, Quantization reduces the bits used to store vectors, significantly shrinking cache size with minimal quality loss. These methods aim to mitigate the memory demands and associated expenses of long-context LLM processing.