Ranking agent configurations by average score misses critical interaction effects between model, prompt, and tool choices. The article demonstrates a Best-Worst Scaling protocol using MaxDiff comparisons and Plackett-Luce utility modeling to surface which configurations actually win head-to-head. Instead of scoring every output in isolation, a judge selects only the single best and worst response from anonymized batches, producing cleaner preference signal.

The experiment tested eight pipeline configurations on the shubh303/Invoice-to-Json dataset (100 invoices, 499 runs). Three binary factors defined the space: Claude Haiku 4.5 versus GPT-5.4-mini, Systematic Planner versus Contextual Leaper prompts, and stream-table-rows versus semantic-search tools. Each invoice was processed by five random configs, then judged by claude-sonnet-4-6.

Raw win rates placed Config 7 (GPT + Leaper + Semantic) at 29.6%, but the Plackett-Luce model with effects coding revealed the dominant signal: the three-way triple interaction (Model × Prompt × Tool) at 0.31 importance, dwarfing main effects. Model alone scored 0.25, while prompt main effect was negligible at 0.03. This means no single component upgrades reliably — the combination dictates performance.

For agent-first teams, the protocol plugs directly into automated eval loops: generate candidate configs, run factorial comparisons, fit utility rankings, and propose the next configuration grid. The human sets priorities; the system delivers evidence that accounts for competitive context, not isolated averages.