A significant challenge in evaluating AI models is the "contamination problem," where benchmark answers inadvertently leak into the models being tested. This occurs because models are trained on vast datasets, including historical information with known outcomes. For instance, when testing a model's ability to predict drug trial success, the model might simply recall information from public records of past trials rather than demonstrating genuine reasoning.

This "leakage" can happen through three main routes: input leak (the model accesses information after a decision date), benchmark leak (the model is trained on the benchmark dataset itself, as seen with GPT-3.5 and GPT-4), and outcome leak (the model learns public outcomes from its training data, such as a drug's success, which is already widely reported).

While the ideal solution is to test on future, unresolved events using "live benchmarks" like CT Open, this is not always feasible. Consequently, researchers are developing methods to prevent or detect these leaks, with a particular focus on addressing the outcome leak for benchmarks based on real-world, historical data. Methods like date-gated tools and fresh test sets aim to mitigate these issues, ensuring more reliable AI performance evaluations.