Large language models thrive on context, but recent research flips that assumption. A 2025 study by Chroma tested 18 top models, including GPT‑4.1, Claude, and Gemini, and found every model’s accuracy dropped sharply when input crossed a certain length. The decline reached more than 30% for some in practice.

Tokens are the building blocks LLMs read; a single token averages three‑quarters of a word. The context window, the total tokens a model can process at once, now stretches from 128,000 to over 2 million. Yet larger windows don’t guarantee better performance because of attention decay in real world scenarios.

Attention mechanisms compare every token against every other, creating a decay that favors beginning and end of the input. This “lost‑in‑the‑middle” effect means vital data buried mid‑document often slips through. Engineers now must orchestrate what appears in the window, a discipline called context engineering to optimize model outputs effectively today.

Context engineering extends beyond tweaks; it manages system instructions, user input, conversation history, retrieved knowledge, tool descriptions, and tool outputs—all vying for limited token space. By prioritizing relevant content and trimming noise, developers keep models focused, cut computational cost, and avoid the accuracy cliffs highlighted by Chroma’s 2025 findings today.