In January 2025, Aditi Singh and coauthors published a survey on agentic retrieval-augmented generation, or agentic RAG. They describe it as a search system with an AI agent inside it. The agent can plan its next step, check its own results, and use other tools, such as a calculator or a search box, while it looks for information. The same survey lists evaluation and efficiency among the open challenges, which is the part this test set out to measure.
Retrieval-augmented generation, or RAG, means a language model answers a question using text it first retrieved from a document collection. Plain search looks the question up once, as written. Agentic RAG adds an agent that reads the first results and decides whether to stop or write a new search. More searches should find better evidence, but each extra search needs another language model call, adds delay for the user, and gives the agent another chance to write a worse query than the one it started with.
The author ran four retrieval methods over the same 298 questions and 2,000 recent arXiv papers, with the AI coding assistant Claude Code building and running the test. The methods were plain search, a reranker that reorders the first 50 results, a question rewriter that produces three alternative queries, and an agent that searches up to three times. Each was scored on how often it placed the right paper among its top five results.
The agent put the right paper in its top five for 96.87 percent of questions, against 95.64 percent for plain search. The gap of +1.2 percentage points had a plausible range from -0.8 to +3.5, which includes zero, so the test could not tell the two apart. The agent found papers plain search missed, but it also lost some that plain search had found.
Source: Towards Data Science · Summarized by HeadlinesBriefing