When an application adopts a large language model (LLM), they generally choose the most capable model possible. This means that every single request is sent to that expensive model. While this approach is easier to implement, it can become quite expensive in the long run. For example, a request such as “classify this support ticket as billing, technical, or account-related” doesn’t require the same level of reasoning as “investigate why these financial records don’t match properly and explain the likely cause.”

With smart model routing, we can solve this problem. In such a routing approach, we choose a specific model for each request. In other words, simple work is sent to a small model that might be less expensive, and difficult work is routed to a more capable model. If most requests are simple, this approach can reduce the total cost in a big way, sometimes by even around 10 times. Also, the quality of the response doesn’t go down noticeably.

However, cost reduction isn’t a given. It also depends on the types of requests the application receives, the price difference between models, and how well the routing system performs. The total cost of using an LLM API usually depends on the number of tokens processed. Input tokens include the user’s message, system instructions, conversation history, and any documents supplied to the model. Output tokens are the tokens generated within the response. Larger and more capable models generally cost more because they require more computing resources.

For example, imagine a customer-support application that has to process a million requests per month. If each request goes to the most powerful model, the company has to pay a premium price even for work that is quite simple. You could think of this as hiring a senior software architect to rename files, sort support tickets, and format dates—a waste of capability and poor resource management.