Caching is a core system design technique for large-scale systems. It stores data that is expensive to compute so it can be reused later without paying the same cost again. A website, for example, might cache a database query result rather than running it on every page load.
The same principle applies to large language models. Caching in LLM applications means storing work done during earlier requests so it can be reused, which lowers processing costs and response times, especially when requests share the same instructions, documents, or questions. There are two main approaches: prompt caching and semantic caching. Both aim to reduce repeated work, but they operate at different stages of a request and follow different rules.
LLM costs come mainly from two sources: input tokens and output tokens. Output tokens often cost more, but input tokens can also be expensive because requests frequently include much more than the user's question. A customer-support application, for instance, may send behavioral instructions, example answers, product documentation, and earlier conversation messages along with a 10-word question, producing thousands of input tokens. Applications that send a long product manual with every request repeat that same content each time, even though different users ask different questions about it.
Repetition also appears in the questions themselves. Hundreds of users may ask how to change a profile picture, each phrasing it differently, so the model often generates roughly the same answer again and again. This creates two opportunities to cut costs: reusing input content that has already been processed, such as the product manual, and reusing answers to questions that are essentially the same.
Source: ByteByteGo · Summarized by HeadlinesBriefing