AI labs are rapidly releasing more capable models, but cost-efficiency is paramount. OpenAI engineers shared insights into optimizing systems like Codex and ChatGPT. A user request doesn't go directly to the Large Language Model (LLM); instead, it passes through several layers.

The harness layer orchestrates the process, managing conversation history, tool execution, and assembling context for the LLM. It acts as the agent loop, interacting with the LLM until the task is complete.

An API layer sits between the harness and the inference endpoint. It handles authentication, rate limits, and the crucial tokenization of text into IDs the LLM understands, performing these tasks in parallel with inference.

The inference layer, backed by GPUs, runs the LLM's computation. Optimizations here include cache-aware routing, KV cache management, speculative decoding, and separating prefill from decode. Techniques like persistent Web Sockets and deferred tool discovery in the harness layer also reduce repeated work, significantly cutting down the cost per successful task. For example, GPT 5.6 Sol is noted as being more capable and less expensive than Fable 5.