Workers AI is running large, open models like Moonshot's Kimi and Z.ai's GLM on GPUs in Cloudflare data centers. These models are demanding due to their size and long context, posing memory challenges. To serve them efficiently, Workers AI employs three key techniques layered on top of prefill and decode separation: quantizing the KV cache, compressing model weights, and protecting the shared cache.

Quantizing the KV cache from 16-bit to 8-bit floating point (FP8) halves its size, doubling the context capacity for models like Kimi K2.6. This allows for significantly more concurrent requests on a single GPU, improving cost-efficiency without impacting accuracy. Similarly, compressing GLM's weights from 8-bit to 4-bit integers (INT4) reduces their size by about 40%, freeing up GPU memory for larger caches and speeding up decode times. For both optimizations, prefill is kept in higher precision where it's compute-bound, while decode uses the optimized formats.

To handle the increased concurrency from these optimizations, Workers AI implemented KV cache integrity checking. This safeguards against errors in shared cache pages, ensuring data accuracy. The overhead for this check is minimal, under 1%, and can be disabled when not needed. These advancements enable Workers AI to support more users at lower costs with no loss in model accuracy, using frameworks like SGLang for benchmarking.