Routing requests between LLMs to save money sounds simple: send easy tasks to a cheap model and hard ones to a frontier model. But switching models mid-session carries a hidden cost. The new model has never seen the conversation history, so it must reprocess the entire session from scratch at full price, a cold-cache prefill that can be one of the most expensive operations in LLM serving.
A August 2026 study by Microsoft Azure Research, covering 13.5 million GitHub Copilot sessions and 760 million model calls, illustrates the problem. Within a turn, about 90% of the prompt could be reused from cache. Crossing a turn boundary on the same model dropped that to 55%, and switching models dropped it to just 8%. Roughly 6.4% of sessions switch models, often because of errors or rate limits, meaning many users pay this cost unintentionally.
To measure the effect directly, the author ran identical 20-turn agent sessions through three routing policies on a vLLM server with prefix caching: sticky routing, per-request alternation, and phase-boundary switching. Per-request routing at the gateway destroyed almost twice as much cache as sticky routing and delivered only 13 of the 40 percentage points of savings implied by the raw price gap. Phase-boundary routing, which switches only when one stage of the agent workflow ends, captured 29 of those points. A cache-blind policy on a frontier model cost 4.88× more than the cache-aware baseline despite processing the same tokens.
The takeaway for anyone operating LLM agents: routing savings depend not only on model prices but on how routing interacts with state and cache reuse. Routing is as much a state-management decision as a pricing one, and the harness is where that state lives.
Source: Towards Data Science · Summarized by HeadlinesBriefing