A team boasts about their RAG application hitting 97% evaluation scores after repeatedly identifying issues and fixing them on the same test set. While this sounds like responsible development, the author argues it reveals a fundamental problem: the evaluation set has quietly become part of the training process.

The issue mirrors classic machine learning overfitting, where models perform well on familiar data but fail on new inputs. Unlike traditional regression models with clear x-y pairs, RAG systems are harder to intuit. Developers often tune prompts, cherry-pick questions, or build tests from indexed documents without realizing they're compromising their evaluation integrity.

The article identifies three common overfitting patterns in RAG evaluation: prompt tuning on evaluation data, selecting only questions the system already handles well, and crafting test queries from indexed knowledge base documents. Each creates an illusion of performance rather than genuine capability measurement.

The fix requires discipline: maintain a truly held-out test set used minimally, construct questions independently of known system behavior, and view suspiciously high metrics with skepticism. A RAG system excelling on reused evaluation sets resembles a student who memorized past exams but crumbles when facing novel questions.