When an internal knowledge base returned a confident but wrong answer about contractor onboarding, the fault traced back to how the text was split. The exception clause lived in a paragraph that the retrieval layer sliced at the boundary, leaving two orphaned fragments. Neither fragment could be retrieved for every query in production environments, exposing chunking as the most fragile design choice in a RAG stack.

The team first tried fixed-size chunking, breaking documents into 512‑token windows with a 50‑token overlap. While quick to implement, this method ignored sentence boundaries, causing critical qualifiers to be split. An evaluation with RAGAS showed a 0.72 context‑recall score, meaning roughly one in four queries missed existing information in our internal corpus—a compliance risk they could not ignore.

Switching to sentence windows via LlamaIndex’s SentenceWindowNodeParser solved the problem. Each sentence became a retrievable node, while surrounding sentences were stored as metadata for generation. The previously hidden exception surfaced immediately, allowing the LLM to answer accurately. The episode demonstrates that chunking decisions dictate downstream retrieval quality and must be treated as a core architectural concern across large knowledge bases.