Kapa.ai introduced a listwise pruner that sits between retriever and generator in their RAG pipeline. A small LLM grades all retrieved chunks together on a five-level scale, dropping 68% of context while preserving 96% recall and cutting query cost by 34% net of pruner overhead.

The pruner solves two failures of pointwise rerankers: uncalibrated scores across queries and inability to judge chunk sets. By seeing the question and all chunks simultaneously, it captures partial and indirect relevance that cross-encoders miss. Three parameters control behavior: model choice, threshold, and keep-top-k protection for the strongest reranked chunks.

Testing on labeled production data showed the scoring strategy dominates budget-select and direct-ask baselines at every compression level. At the chosen operating point, each dropped chunk saves roughly 4% of query spend, with latency around 0.7 seconds per query.

The approach demonstrates that cheap listwise judgment outperforms expensive reranker calibration tricks. For teams running large-scale RAG over technical docs, inserting a fast pruner recovers context budget for agent tool calls while cutting inference spend — provided recall loss stays within acceptable bounds for the domain.