A systematic study by authors including Mubashara Akhtar and Irene Solaiman reveals that artificial intelligence benchmarks, crucial for tracking model progress and deployment, are rapidly becoming saturated. This saturation makes it difficult to distinguish between different models and reduces their long-term utility.

The researchers defined benchmark saturation and analyzed 60 language model benchmarks using 14 properties related to this phenomenon. They discovered that approximately half of the evaluated benchmarks show signs of saturation, with the rate of saturation increasing over time.

Furthermore, the study indicates that expert curation, rather than the availability of public test data, influences a benchmark's resilience to saturation. These findings suggest that thoughtful design choices can prolong the usefulness of benchmarks and lead to more robust evaluation strategies for AI models.