People solved 98.0% of the grid puzzles in this study, taking about 11 seconds for an easy one and a little over two minutes for the hardest. The best model tested, GPT-6.1 Sol, solves 83.6% of them. That sounds close to human, but on the hardest level it solves 49.4%, while people solve 94.5%.
Solving one of these puzzles requires several separate skills, and a single accuracy number hides which of them a model lacks. The researchers therefore split the task into four questions: Can models read the board, especially from an image? Can they keep track of their own line? Can they follow the rules? Do they try harder when a puzzle is harder? The post draws on three papers, including SPa RC, a spatial pathfinding reasoning challenge published at EMNLP 2025.
Each puzzle is a board of green cells joined by grey lanes. The player draws one non-crossing line from a large circle on the edge to an exit stub. Symbols decide which lines are valid. Dots must lie on the line, gaps block lane segments, and stones of different colors must end up in separate regions closed off by the line. A line that passes both dots but leaves a purple, black and red stone in one region breaks the stone rule, and the correct solution takes a longer route through the board.
People solved the puzzles in a drawing app, but models usually receive the board as a text grid with coordinates. Models explain their reasoning and then write the line as a list of positions, step by step. The full benchmark contains seven rule types, and the gap between people and models on the hardest puzzles shows where current systems still struggle.
Source: Towards Data Science · Summarized by HeadlinesBriefing