An enterprise running a Kubernetes cluster for LLM serving started with 64 H100 SXM GPUs across eight nodes, all managed by vLLM in a monolithic layout. During prompt prefill, tensor cores hit 92% utilization, but milliseconds later the same cards fell to roughly 30% during decode. The mismatch left the fleet idle for most of each request, inflating the GPU bill to training‑level costs.

Prefill expands the prompt into a key‑value cache, requiring matrix multiplications that saturate tensor cores. On an H100 this phase delivers 200‑400 ops per byte and keeps compute utilization above 90%, while memory bandwidth stays around 30%. Decode generates tokens pulling the cache from HBM; arithmetic intensity falls to 60‑80 ops/byte and compute usage drops to 20‑40%, with memory bus near capacity.

The remedy is disaggregated inference, which isolates prefill and decode onto separate GPU pools linked by a KV‑aware router. A prefill cluster of compute‑optimized cards builds the cache, then hands it to a memory‑bandwidth‑optimized decode cluster for token generation. Frameworks such as DistServe, vLLM, SGLang and NVIDIA’s Dynamo already support this split, letting large enterprises cut inference spend by up to 75% without sacrificing latency.