The trend of building LLM routers, which dynamically select models for requests to reduce costs, is being questioned. While initially promising, the company behind the Manifest LLM router, launched in March, deprecated it in June and shut it down by September 1st. Their router classified requests into four complexity tiers: simple, standard, complex, and reasoning, aiming for cost-effectiveness by avoiding powerful, expensive models for simple tasks.

However, after four months of use by 7000 cloud users, the results were mixed. A primary issue identified is that prompt complexity alone is insufficient; context discovered through tool calls and web searches significantly impacts the task's difficulty. For instance, a request to "evaluate and improve tests for a repo" could be simple for a personal website but incredibly complex for the Linux kernel.

Furthermore, caching proved more effective for cost reduction than routing. Cache reads are significantly cheaper than uncached inputs, and prefix caching works well for system prompts and conversation history. The company argues that LLM routers break behavioral consistency, likening engineers' model selection to a painter choosing brushes. Unpredictability introduced by routers also adds hidden costs in maintenance, evals, and observability, making isolated requests with specific models, parameters, and prompts generally superior. While acknowledging potential use cases, their experience suggests LLM routing is often not worth the trade-offs.