HeadlinesBriefing favicon HeadlinesBriefing.com

Loop Engineering for Listing Questions in RAG

Towards Data Science •
×

Listing questions, like “list every exclusion in this policy,” break the core assumption of retrieval: that the answer is the top passage. Here the answer is every passage. Naive RAG pipelines retrieve top-k chunks, and the LLM confidently lists five out of six items, missing one that is phrased differently. This silent failure isn't fixed by long-context models; they hit the same wall because the question asks for every item, not the top one.

Top-k is structurally wrong for listing. Items in a list are similar, so embedding scores cluster, and a fixed k either misses items or dilutes attention. The fix is to detect listing intent via regex or an LLM classifier, then use retrieval strategies designed to find all items. Three aggregation strategies are presented, along with a completeness signal that says when the list is done.

Examples come from the NIST Cybersecurity Framework (GOVERN categories, including GV.SC) and the Attention Is All You Need paper (Vaswani et al. 2017). A runnable companion notebook demonstrates the approach. The pipeline shape differs from factual lookup: it enforces completeness by construction or iteration, and the expert ratifies the result.