A language model can write 'let me double-check that' and still get the multiplication wrong. It can even write 'wait, let me reconsider,' and land on a different wrong answer. Neither sentence is evidence of thinking.

If the goal is solving the problem, only the number at the end counts. Deep Seek’s R1-Zero brought considerable attention to reinforcement learning without a preliminary supervised fine-tuning stage. Its researchers reported behaviors such as revisiting an approach and spending more tokens on difficult problems.

Deep Seek-R1, in contrast, used a broader training pipeline, including cold-start data, so we shouldn’t treat the two training processes as interchangeable. What interests me the most about Group Relative Policy Optimization, or GRPO as most of us know it, is how little feedback the basic setup needs. A model can generate several attempts at a question, receive a score for each, and use the differences to adjust its behavior.

We can even check the final answer without writing out the solution we want it to imitate. Combined with techniques that reduce training memory, this makes smaller reasoning experiments more accessible. That leaves two things worth understanding before choosing a model or a GPU: how those scores guide an update, and what happens when the scoring rule rewards the wrong thing.

A simple arithmetic example helps explain both. Consider four possible responses to the same question and how a reward function would score each one. A problem with an answer we can check: A shop has 6 boxes with 8 items in each box.

It sells 6 items. How many items remain? The answer is 42. A Python expression can verify it.

Now, that simple check gives us something valuable: an independent way to score the final answer. We do not need another language model to decide whether the response sounds convincing. In a training dataset, the question goes to the model while the expected answer stays with the verifier.

The model generates a response, the verifier scores it, and the training algorithm uses that score to adjust the model’s behavior. Easy. We can therefore use question-and-answer pairs without providing worked solutions as supervised targets.

This is the appeal of reinforcement learning with verifiable rewards: the feedback can come from checking an outcome. For a real inventory system, ordinary arithmetic is still the sensible solution. The language model would be an expensive calculator with opinions.

Here, the example makes the learning mechanism easy to inspect. This also shows where our verifier falls short, because checking that the final integer is correct doesn’t tell us whether the reasoning behind it makes sense. GRPO compares attempts at the same question.

GRPO was introduced in the Deep Seek Math work as an alternative to the memory requirements of conventional PPO-based training. Common actor-critic implementations of PPO train a value estimator, often called a critic, alongside the policy. That estimator supplies a baseline for judging an action’s outcome.

GRPO obtains its baseline from a group of responses sampled for the same prompt. Instead of asking a separate critic to estimate how well the model should do, it compares how well several attempts actually did. Imagine the model produces these four answers: Attempt Final answer Correctness reward 1 42 1, 2 24 0, 3 42 1, 4 48 0.

The group’s average reward is 0.5. Attempts 1 and 3 performed better than that average; attempts 2 and 4 performed worse. The original outcome-reward formulation standardizes that comparison: [Ai=ri−rˉσr+ϵ][A_i = rac{r_i - a...