A research paper claims reward labels barely change how agents learn. I tested that on three models, and one of them did exactly what the wrong labels told it to. In my test, each trial gave a model a made up shipping order and asked it to choose one of four delivery routes, named Kiwi, Plum, Fig, and Lime.
A hidden rule decided which route was right, and the model had to work that rule out from its memory. By the last six trials, GPT-6 Sol answered 94.4 out of 100 correctly with true labels and 7.8 with flipped ones. Claude Haiku 5.5 rose from 21.1 to 37.2 with true labels.
Qwen3 4B Instruct stayed near 32.8. A paper submitted on 8 October 2026, Do LLMs Learn from Rewards in Context?, reports something different. Its abstract says that flipping, randomizing, or removing rewards changed improvement curves very little across four benchmarks and six models.
My test does not contradict it, because my task is different and easier in one way. I hid a rule that only the labels could reveal, so a model that ignored the labels had nothing to learn from. I ran the same job on three models and report what each did.
I also report what went wrong. A check I wrote in advance, which I call a stop rule, fired for one model. I added a supplement afterwards, and I found that my first confidence ranges were too narrow.
All the code, saved replies, and cost logs are in a public repository. The whole test used about 2.55 US dollars of API fees and 0.65 Colab compute units.
Source: Towards Data Science · Summarized by HeadlinesBriefing