OpenAI's GPT-5.6 Sol initially struggled on the ARC-AGI-3 benchmark, scoring only 7.8%, a stark contrast to its success in other complex tasks. This puzzle benchmark presented a unique challenge, prompting an investigation into the model's performance.

Researchers discovered that the benchmark's generic harness discarded private reasoning after each action and used rolling truncation, effectively limiting the model's memory. By enabling two specific API settings—retained reasoning and compaction—OpenAI saw a significant improvement. These settings, used in products like ChatGPT and Codex, allow the model to remember its past thinking and actions.

With retained reasoning and compaction, GPT-5.6 Sol's score on the ARC-AGI-3 public set jumped from 13.3% to 38.3%, nearly tripling its performance. This enhancement also reduced output tokens by 6x. The findings highlight that benchmark scores are influenced not just by the model, but also by harness design and API settings, suggesting developers should utilize retained reasoning and compaction for optimal results.