Imagine evaluating a retrieval-augmented-generation system where multiple LLM judges evaluate the same passage. Eight say 'relevant'; two say 'not relevant'. Eight out of 10 feels convincing, but agreement strength depends on judge independence.

If judges share prompt templates, training lineage, model families, or blind spots, they may repeat the same mistake, making vote counts misleadingly strong. The paper 'Dependence-aware label aggregation for LLM-as-a-judge via Ising models,' coauthored with Shiva Kasiviswanathan and presented at ICML, presents a method for assessing correlations between judges' outputs and adjusting aggregate scores to ensure opinion diversity. In tests on three tasks, the method outperformed the best-performing baseline — a panel weighted by historical accuracy — by 9% to 14% on standard metrics.

The majority vote assumes judges make errors independently, which is often too optimistic for LLM-as-a-judge systems. Two judges may fail together due to similar rubric interpretation, shared prompt examples, or shared model sensitivity. The aggregator models the panel as a network where each judge has a reliability profile and pairs have relationships, agreeing more often than predicted or providing complementary perspectives.

The method models judge skill and similarity using an Ising model, a statistical model representing pairwise dependence between binary variables. Two modeling variants exist: one treats relationship patterns as consistent across positive and negative labels, adjusting weights for correlation; the other, a class-dependent model, lets relationship patterns change with labels, useful when judges agree on clear items but cluster on ambiguous ones. The unsupervised method learns from judge outputs without human reference labels, treating true labels as latent variables to jointly infer judge reliability and dependence.