From pairwise reward modeling to calibrated, multiway decisions Jev looks mysterious when viewed as an alternative to a language model. It becomes much simpler when viewed as the next step in reward modeling. The core idea is: RLCD = multiway preference modeling + probability calibration. More specifically, RLCD is a schema-conditioned Plackett–Luce objective. Jev turns that objective into a product by adding typed outputs and parallel inference. That is the secret: the reward model is no longer hidden behind a generator. The reward model becomes the model.

Reward modeling started with a scalar. A conventional reward model receives a context and a candidate answer, then produces a scalar. Outcome reward models score the final answer. Process reward models score individual reasoning steps. In both cases, the learned object is an absolute-looking number. The problem is that this number is not actually absolute. A reward of 0.8 does not have a stable meaning across problems, candidate pools, checkpoints, or model families. It is mainly useful for comparing candidates generated under similar conditions. The operational signal was always relative preference.

PPRM made the preference explicit. LLaMA-Berry’s Pairwise Preference Reward Model, or PPRM, exposes the comparison directly. Given a problem and two solutions, PPRM answers: Is the first answer better than the second answer? This is the Bradley–Terry model. LLaMA-Berry implements the comparison as a constrained language-model decision over Yes and No tokens. It trains the evaluator on almost 7.8 million mathematical-solution pairs and uses DPO to improve the pairwise prediction task. The essential change is conceptual: reward modeling becomes preference-probability modeling.

Plackett–Luce is the multiway PPRM. PPRM compares two candidates. A real decision interface usually receives more than two. Assign each candidate a context-dependent utility, then normalize all candidates together. This is the Luce choice model, also known as multinomial logit. When K=2, it reduces exactly to Bradley–Terry. PPRM is therefore the binary case of the same choice geometry. If the supervision contains a complete ranking, the full Plackett–Luce likelihood repeatedly selects the next-best remaining candidate.