A team of 49 mathematicians spent six weeks creating a rigorous benchmark for mathematical reasoning, culminating in a workshop at the Max Planck Institute in Leipzig. The group assembled 100 research-level questions with verified answers, rigorously testing whether large language models could match human problem-solving capabilities on abstract mathematical challenges.

The evaluation unfolded in three stages using different model configurations. Five state-of-the-art LLMs attempted the questions once, leaving 41 unsolved. Three of those models then ran 20 attempts each, reducing the unsolved count to 16. Finally, two heavy-thinking models completed three runs each, leaving only 2 questions unanswered. This progressive improvement shows how iterative prompting boosts performance on complex reasoning tasks.

The results demonstrate that LLMs are rapidly advancing in mathematical reasoning, though gaps remain for truly research-level problems. The 100-question dataset provides a valuable benchmark for tracking progress in this domain. The workshop format—bringing together pure mathematicians and AI researchers—produced a methodology that could inform future evaluations across other technical fields.

This work establishes concrete metrics for measuring AI mathematical capability while revealing both current strengths and persistent weaknesses in automated reasoning.