A new technical blog post from November 2025 explains continuous batching from first principles, starting with attention mechanisms and KV caching. The authors, Rémi Ouazan Reboul, Arthur Zucker, and Luc Georges, break down how AI chatbots like Qwen and Claude generate responses token-by-token, revealing why the first word takes time while subsequent words appear rapidly.

At the core of large language models lies the attention mechanism, where tokens interact through query, key, and value projections. The authors explain how attention layers create connections between words, with the attention mask controlling which tokens can influence each other. They demonstrate how causal masking ensures tokens only interact with previous ones, maintaining logical flow in generated text. The quadratic complexity of attention operations makes optimization crucial for practical applications.

Understanding prefill and decoding phases reveals why continuous batching matters. During prefill, the entire input sequence passes through attention layers to compute the next token prediction. The decoding phase then leverages cached computations for faster generation. By processing multiple conversations in parallel and swapping them out when complete, continuous batching maximizes throughput in high-load serving scenarios, making AI chatbots more responsive and efficient.