A developer built a RAG system for an enterprise knowledge base with 800 pages of documentation. Retrieval accuracy hit 96%, but answer accuracy was only 32%. The root cause was fixed-size chunking, which split procedures and tables mid-context, leaving the LLM with incomplete instructions.

The problem wasn't retrieval quality but semantic boundaries. A procedure to reset an API key was split into three chunks, and the system retrieved only the middle paragraph. Without setup and follow-up steps, the LLM hallucinated or gave vague answers. Increasing chunk size or adding overlap created other issues like reduced precision and bloat.

The solution was semantic chunking, grouping content by structure—procedures, tables, and lists stayed intact. Adding hierarchical metadata (page title, section headings) gave the LLM crucial context. This pushed answer accuracy to 87% and support ticket deflection to 63%, proving that preserving meaning is more important than optimizing for token count.