A new approach to transformer key-value cache compression claims to shatter the Shannon entropy limit by treating KV caches as sequences rather than isolated vectors. Gregory Magarshak introduces sequential KV compression, which exploits the fact that KV cache tokens are samples from the exact formal language the model was trained on. This method achieves theoretical compression ratios over 914,000x compared to existing per-vector quantization methods like TurboQuant.

The technique uses two complementary layers: probabilistic prefix deduplication that identifies semantically equivalent shared prefixes across sessions using trie metrics, and predictive delta coding that stores only residuals from the model's own predictions. At typical language model perplexity of 10-20 for fluent English, this achieves 3.3-4.3 bits per token position versus TurboQuant's 3 bits per vector component. The method remains effective even with pessimistic worst-case overhead, maintaining approximately 914x compression over TurboQuant as context length grows.

This represents a fundamental shift in compression strategy by recognizing that KV cache data follows predictable language patterns rather than being arbitrary floating-point values. The orthogonal layers can compose with existing quantization methods, suggesting practical implementation potential beyond theoretical bounds.