Preprint. Under review.
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations Manan Gupta1
Dhruv Kumar1
1 BITS Pilani, Pilani Campus, India
{f20241231, dhruv.kumar}@pilani.bits-pilani.ac.in
arXiv:2604.15302v1 [cs.AI] 16 Apr 2026
Abstract LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval: (1) a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates (ρ̄ = 0.8–4.1%), with 33–67% of documents exhibiting at least one directed 3-cycle; and (2) split conformal prediction sets over 1–5 Likert scores providing theoretically-guaranteed ≥ (1−α) coverage, with set width serving as a per-instance reliability indicator (rs = +0.576, N =1,918, p < 10−100 , pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement (r̄ = 0.32–0.38), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size ≈ 3.0) and coherence moderately so (avg. set size ≈ 3.9), while fluency and consistency remain unreliable (avg. set size ≈ 4.9). We release all code, prompts, and cached results.
1
Introduction
Automatic evaluation of natural language generation (NLG) has become a cornerstone of modern NLP research. LLM-as-judge systems, where a large language model scores or ranks system outputs, have gained rapid adoption as scalable proxies for human annotation (Zheng et al., 2023; Liu et al., 2023; Fu et al., 2023). A single LLM call can replace expensive crowd-sourced annotation pipelines, and practitioners increasingly treat these scores as ground truth. Yet a critical question goes largely unasked: when should you trust an LLM judge? Aggregate metrics such as system-level Kendall’s τ or Pearson correlation with human scores look impressive, but they average over hundreds of instances. A judge that is right 90% of the time can be spectacularly wrong on the 10% that matters most. This paper develops and evaluates two complementary diagnostics for per-instance reliability. Contribution 1: Transitivity diagnostic. Pairwise LLM judges are assumed to produce transitive preferences, yet tournament theory tells us that directed cycles (A≻B, B≻C, C≻A) arise naturally when alternatives are near-equally preferred (Young, 1988). We measure directed 3-cycle violation rates across four judges on SummEval (Fabbri et al., 2021) and, crucially, disaggregate by input document. Aggregate rates are low (ρ̄ < 5%), but 33–67% of documents exhibit at least one violation, with per-document rates reaching 30.4% for Mistral-Small-3.1. We also test whether Minimum Feedback Arc Set (MFAS) ranking repair (Ailon et al., 2008) improves agreement with human rankings.It does not, establishing that violations are sparse noise rather than systematic directional bias. Contribution 2: Conformal prediction diagnostic. We apply split conformal prediction (Vovk et al., 2005; Angelopoulos & Bates, 2021) to direct Likert scores, producing pre1
Preprint. Under review.
Judges
PAIRWISE PATH
Dataset
SummEval
Analysis
Pairwise
Transitivity
Protocol 40,320 calls k = 3 reps
Analysis 3-cycle detection MFAS repair
Direct
Conformal
Scoring 3,840 calls Likert 1 5
Prediction Split CP ( = 0.10) Nonconf. scores
LLM Judges
Findings
DIAGNOSTIC A Violation rate (x) Per-doc distribution MFAS ranking repair
GPT-4o-mini LLaMA-3.1-70B Qwen-2.5-72B Mistral-Small
SCORING PATH
30 docs ×8 systems ×4 criteria
Protocol
DIAGNOSTIC B Prediction sets C(x) Width as trust signal Cross-judge agreement
Unified finding: criterion matters more than judge (coherence & relevance reliable; fluency & consistency unreliable)
Figure 1: Two-pronged diagnostic pipeline. SummEval documents are evaluated by four LLM judges under two protocols. The pairwise protocol (40,320 API calls) feeds the transitivity diagnostic, which measures directed 3-cycle violation rates ρ( x ) per input and tests whether MFAS ranking repair improves agreement with human rankings. The direct scoring protocol (3,840 API calls) feeds the conformal diagnostic, which produces prediction sets C( x ) with guaranteed ≥ (1−α) coverage; set width serves as a per-instance trust signal.
diction sets with finite-sample, distribution-free coverage guarantees. The central finding is that prediction set width is a statistically robust per-instance reliability indicator: pooling 1,918 observations across all judges and criteria yields Spearman rs = +0.576 (p < 10−100 ) between width and actual judge–human disagreement. Moreover, different judges assign wide sets to the same documents (r̄ = 0.32–0.38 across judge pairs for fluency, consistency, and relevance), demonstrating that width measures inherent document difficulty rather than judge-specific noise. Unified finding. Both diagnostics independently identify the same axis of variation: criterion explains reliability more than judge. This is actionable: a practitioner deploying LLM judges should trust coherence and relevance scores more than fluency and consistency scores, regardless of which judge is used.
2
Related Work
LLM-as-judge reliability. Zheng et al. (2023) introduced MT-Bench and Chatbot Arena, establishing LLM judges as scalable evaluation tools. Liu et al. (2023) showed that GPT4-based G-Eval correlates strongly with human judgment on SummEval. Known biases include position (Wang et al., 2023), verbosity (Saito et al., 2023), and self-enhancement effects. Fernandes et al. (2023) and Koo et al. (2023) audit LLM judges at scale, finding systematic weaknesses on specific input types, consistent with our per-document view. Concurrent to our work, Ye et al. (2024) study fine-grained reliability of LLM judges across skill categories, but without formal uncertainty guarantees. Transitivity and ranking consistency. Condorcet cycles in pairwise preferences have been studied in social choice theory since de Condorcet (1785) and are known to be ubiquitous when alternatives are near-equal (Young, 1988; Moon, 1968). MFAS-based ranking repair has been applied to preference aggregation (Ailon et al., 2008) and recently to LLM-generated ranked lists (Qin et al., 2024). We are the first to measure directed 3-cycle rates in LLM judges at the per-document level and connect them to conformal uncertainty. 2
Preprint. Under review.
Conformal prediction in NLP. Split conformal prediction (Vovk et al., 2005; Papadopoulos et al., 2002) provides distribution-free coverage guarantees; Angelopoulos & Bates (2021) give a modern treatment. Applications to NLP include uncertainty for machine translation (Fomicheva et al., 2020), text classification (Maltoudoglou et al., 2020), and question answering (Quach et al., 2024). Kumar et al. (2023) apply conformal methods to LLM generation quality, while Kuhn et al. (2023) propose semantic entropy as a complementary uncertainty signal. Our work is the first to apply conformal prediction to LLM-as-judge scores and to interpret prediction set width as a per-instance deployment signal.
3
Methods
Figure 1 illustrates our pipeline. Both diagnostics share the same four judges, dataset, and criteria, enabling direct comparison of their findings. 3.1
Transitivity Diagnostic
Tournament formulation. For each input document x and set of n system outputs S = {s1 , . . . , sn }, a pairwise judge produces a tournament G = (S , E) where (si , s j ) ∈ E iff the judge prefers si over s j . A transitivity violation is a directed 3-cycle: si ≻ s j , s j ≻ sk , sk ≻ si . We define the violation rate for document x as: # directed 3-cycles in Gx ρ( x ) = , (1) (n3 ) normalized by the total number of possible triples. We report the aggregate mean ρ̄, the fraction of documents with ρ( x ) > 0, and the full distribution {ρ( x )}.
Per-document violation rate.
Ranking methods. We compare five ranking methods: Win Rate (fraction of pairwise wins), Bradley-Terry (Bradley & Terry, 1952) (maximum-likelihood strength scores), Schulze (Schulze, 2011) (beatpath method), MFAS-ILP (exact Minimum Feedback Arc Set via integer linear programming), and MFAS-Copeland (Copeland scores as a fast MFAS approximation). We measure agreement with human rankings via Kendall’s τ. Repetitions. Each pair is queried k =3 times per judge to measure win-rate confidence (0, 1/3, 2/3, or 1). 3.2
Conformal Prediction Diagnostic
Setup. We use split conformal prediction (Vovk et al., 2005) in the direct scoring setting: the judge assigns a Likert score ŷ ∈ {1, . . . , 5}, and the calibration target is the rounded average human score y∗ ∈ {1, . . . , 5}. Nonconformity score and prediction set. We use the absolute residual as the nonconformity score: si = |ŷi − yi∗ |. Given calibration set {( xi , yi∗ )}in=1 , the conformal threshold is: q̂ = s(⌈(1−α)(n+1)⌉) , (2) the appropriate empirical quantile ensuring marginal coverage P(y∗ ∈ C( x )) ≥ 1 − α (Tibshirani et al., 2019). The prediction set for a new instance with judge score ŷ is: C( x ) = y ∈ {1, . . . , 5} : |ŷ − y| ≤ q̂ . (3) Set width w( x ) = |C( x )| ranges from 1 (maximally confident) to 5 (full uncertainty). Evaluation protocol. We evaluate (1) empirical coverage vs. target 1−α; (2) average set size as an informativeness measure; (3) Spearman rs (w, |ŷ − y∗ |) to quantify how well set width predicts actual judge error; and (4) inter-judge width agreement, the Spearman correlation between widths assigned by two different judges to the same document, which tests whether width reflects document-level difficulty or judge-specific noise. All metrics are averaged over 20 random 50/50 calibration/test splits for stable estimates. 3
Preprint. Under review.
Judge
Agg. ρ̄
% docs ≥1
Max ρ
Med. ρ
LL A MA-3.1-70B Q WEN -2.5-72B M ISTRAL -S MALL GPT-4 O - MINI
0.008 0.022 0.041 0.014
33.3% 50.0% 50.0% 46.7%
3.6% 14.3% 30.4% 7.1%
0.0% 1.8% 1.8% 0.0%
Table 1: Per-document transitivity violation statistics (coherence). Aggregate rates ρ̄ appear low (<5%), yet 33–50% of documents exhibit at least one directed 3-cycle, and per-document rates reach 30.4% for M ISTRAL -S MALL. Median = 0 for all judges: most documents are violation-free, but a minority expose severe judge inconsistency.
4
Experimental Setup
Dataset. SummEval (Fabbri et al., 2021) contains 100 documents × 16 systems (= 1,600 outputs) with human Likert scores (1–5) on coherence, consistency, fluency, and relevance, averaged over three annotators. We subsample to 30 documents × 8 systems (systems 0, 2, 4, 6, 9, 11, 13, 15) for cost efficiency, rounding averaged human scores to the nearest integer for conformal calibration. Judges. We evaluate four instruction-tuned LLMs accessed via OpenRouter: GPT4 O - MINI (gpt-4o-mini), LL A MA-3.1-70B (meta-llama/llama-3.1-70b-instruct), Q WEN -2.5-72B (qwen/qwen-2.5-72b-instruct), and M ISTRAL -S MALL -3.1 (mistralai/mistral-small-3.1-24b-instruct). All responses are cached in SQLite.
5
Results
5.1
Transitivity Violations: Aggregate Rates Mask Per-Document Heterogeneity
Table 1 reveals a fundamental measurement problem. Aggregate violation rates ρ̄ range from 0.8% to 4.1% across judges, statistics that would reassure any practitioner. But the per-document view tells a different story: between 33% and 50% of documents carry at least one directed 3-cycle, and the worst document for M ISTRAL -S MALL has ρ( x ) = 30.4%, meaning nearly a third of all triples form preference cycles on that input. This is not random noise. Figure 2 shows that the per-document distribution is heavily righttailed: the median is zero across all judges, but a small fraction of documents, typically those where system outputs differ subtly in quality, drive the aggregate statistic. A practitioner who relies on ρ̄ alone will miss this concentrated inconsistency. Appendix C (Table 6) provides violation statistics across all four criteria. Fluency and consistency show the highest fraction of documents with ≥ 1 violation (up to 66.7% for LLaMA–fluency), converging with the conformal results below. MFAS ranking repair does not help. Table 2 compares five ranking methods against the SummEval human gold standard on coherence. MFAS-ILP achieves the highest Kendall’s τ on one judge but matches or falls below Win Rate on the other three. No method consistently dominates. This non-result is informative: when violations are sparse and concentrated in a few documents, the tournament graph is nearly acyclic and MFAS repair finds little to improve. The diagnostic value of transitivity analysis lies in identifying unreliable documents, not in repairing global rankings. 5.2
Conformal Prediction Sets: Guaranteed Coverage with a Trust Signal
Coverage guarantee satisfied. The conformal guarantee holds for all 16 judge×criterion combinations across all four α levels tested (α ∈ {0.05, 0.10, 0.15, 0.20}). Figure 6 confirms that empirical coverage tracks the theoretical 1−α line, meeting or exceeding it at every operating point. Full coverage and set-size results are in Appendix B. 4
Preprint. Under review.
Per-Input Transitivity Violation Rates by Judge and Criterion GPT-4o-mini
Coherence
Per-input violation rate
0.7
LLaMA-3.1-70B
Qwen-2.5-72B
Consistency
0.7
Mistral-Small
Fluency
0.7
0.6
0.6
0.6
0.6
0.5
0.5
0.5
0.5
0.4
0.4
0.4
0.3
random baseline
0.3
random baseline
0.4
0.3
random baseline
0.3
0.2
0.2
0.2
0.2
0.1
0.1
0.1
0.1
0.0
0.0
0.0
0.1
=0.01 47% 1
=0.01 33% 1
=0.02 50% 1
=0.04 50% 1
GPT
LLaMA
Qwen
Mist.
0.1
=0.01 43% 1
=0.01 37% 1
=0.03 57% 1
=0.04 63% 1
GPT
LLaMA
Qwen
Mist.
0.1
Relevance
0.7
random baseline
0.0 =0.03 53% 1
=0.03 67% 1
=0.02 60% 1
=0.03 53% 1
GPT
LLaMA
Qwen
Mist.
0.1
=0.02 53% 1
=0.01 37% 1
=0.02 50% 1
=0.03 53% 1
GPT
LLaMA
Qwen
Mist.
Figure 2: Per-document violation rate distributions. Each violin shows the distribution of ρ( x ) across 30 documents for one judge–criterion pair. Dashed horizontal line: randombaseline rate (0.25). All distributions are right-tailed with median = 0, but the upper tails, where a single document can expose > 30% violation rates, are practically significant. Fluency consistently shows the widest tails. Method
LLaMA Qwen Mistral GPT
Win Rate Bradley-Terry Schulze MFAS-Copeland MFAS-ILP†
0.571 0.571 0.643 0.571 0.643
0.643 0.643 0.643 0.643 0.643
0.071 0.071 0.071 0.071 0.071
0.714 0.714 0.643 0.714 0.643
† Exact ILP solution; bold = best per judge.
Table 2: Ranking agreement (Kendall’s τ) vs. SummEval human gold standard (coherence). MFAS does not consistently outperform Win Rate or Bradley-Terry. When violations are sparse, standard aggregation already captures the signal; MFAS repair adds noise rather than structure.
Criterion drives reliability, not judge. Table 3 and Figure 3 show the dominant pattern: the criterion column explains far more variance in average set size than the judge row. Coherence and relevance receive small sets (2.82–4.51 labels) while fluency and consistency receive near-maximal sets (4.47–4.99) across all four judges. This is not a failure of the method. SummEval neural summaries are uniformly fluent, leaving little variance for a judge to exploit (Fabbri et al., 2021). Consistency requires crossdocument factual reasoning that 24–72B scale models perform inconsistently. The wide prediction sets are the method correctly reporting that it cannot reliably grade these criteria. Set width predicts judge error. Figure 4 shows pooled reliability diagrams for each criterion: mean absolute error (MAE) vs. prediction set width, pooled across all four judges. Thirteen of sixteen judge×criterion combinations show a perfectly monotonic width–error relationship. Pooling all 1,918 observations, Spearman rs = +0.576 (p < 10−100 ). Percriterion pooled correlations are rs = +0.34 (consistency), rs = +0.16 (coherence), rs = +0.15 (fluency), and rs = −0.01 (relevance, n.s.). The three non-monotonic cases, LLaMA fluency, LLaMA relevance, and Mistral coherence, share a common structure: only two distinct width values are observed (widths 2 and 3), leaving insufficient variance for a meaningful per-judge correlation. The width–error relationship is real but requires pooling to emerge against this discrete noise floor. Width reflects document difficulty, not judge noise. A potential confound: perhaps wide sets indicate that a specific judge is unreliable, rather than that the document is inherently hard to evaluate. Table 4 addresses this directly by measuring inter-judge agreement on 5
Preprint. Under review.
Average Prediction Set Size and Coverage at =0.10 (Smaller = More Informative / Reliable)
LLaMA-3.1-70B
Qwen-2.5-72B
Mistral-Small
4.51
4.76
4.99
3.17
cov=0.971
cov=0.942
cov=0.995
cov=0.928
4.5
3.04
4.97
4.93
2.82
4.0
cov=0.947
cov=0.992
cov=0.967
cov=0.961
3.62
4.48
4.76
2.99
cov=0.934
cov=0.962
cov=0.967
cov=0.961
4.47
4.98
4.99
3.06
cov=0.950
cov=0.996
cov=1.000
cov=0.935
Coherence
Consistency
Fluency
Relevance
Avg. Prediction Set Size
GPT-4o-mini
5.0
3.5
3.0
2.5
Figure 3: Average prediction set size at α=0.10 (green = small = reliable; red = large = unreliable). Each cell shows average set size (larger text) and empirical coverage (smaller text). The criterion axis drives variation far more than the judge axis: coherence and relevance (left two columns) are reliably judged (≈3.0), while fluency and consistency are near-maximally uncertain (≈5.0). All 16 cells meet the 90% coverage guarantee. Judge
Coh. Rel. Con. Flu.
Size GPT-4 O - MINI Cov. Size LL A MA-3.1-70B Cov. Size Q WEN -2.5-72B Cov. Size M ISTRAL -S MALL Cov.
4.51 .971 3.04 .947 3.62 .934 4.47 .950
3.17 .928 2.82 .961 2.99 .961 3.06 .935
4.76 .942 4.97 .992 4.48 .962 4.98 .996
4.99 .995 4.93 .967 4.76 .967 4.99 1.00
rs (w,e)
+0.18 / +0.25 −0.03 / +0.08 +0.14 / +0.65 −0.02 / +0.17
Table 3: Conformal results at α=0.10 (90% target coverage). Columns ordered: reliable criteria then unreliable criteria . rs (w,e) = Spearman correlation of set width with absolute error, reported as Coh./Con. (coherence and consistency). Q WEN -2.5-72B achieves the strongest width–error signal for consistency (rs = +0.65). prediction set width: the Spearman correlation between widths assigned by two different judges to the same documents. For fluency, consistency, and relevance, 15 of 18 judge pairs show significant positive width agreement (p < 0.05), with mean correlations of r̄ = 0.38, 0.32, and 0.36 respectively. The standout pairs, GPT/Mistral fluency (r = +0.81) and GPT/Qwen relevance (r = +0.70), confirm that different model families converge on the same documents as hard to judge. Coherence is the exception (r̄ = 0.10), consistent with its smaller and less variable set sizes. Figure 5 visualizes the full inter-judge agreement matrices, making the qualitative pattern clear: coherence shows near-zero off-diagonal entries while fluency and relevance show predominantly warm (positive) off-diagonal colors. 5.3
Convergent Evidence: Criterion Drives Reliability
Both diagnostics, applied independently, reach the same conclusion. Fluency and consistency show the highest fraction of documents with ≥ 1 transitivity violation (LLaMA: 66.7% for fluency; Qwen: 60%; Table 6) and the widest prediction sets (avg. size > 4.9 for three of four 6
Preprint. Under review.
Width Error Reliability: Wider Sets Indicate Higher Uncertainty ( =0.10) n=365 n=147
Mean Absolute Error
1.5
n=2
Coherence
n=204
n=126
=+0.158 p=0.001
2.5 2.0 1.5 1.0 0.5 0.0 0.5
1.0 0.5 0.0 0.5
2
3 4 Prediction Set Width
5
n=110
n=5
=+0.341 p=0.000
Consistency 2.0
=+0.148 p=0.001
Fluency
n=443 n=31
0.8
n=35
0.5
n=1
0.4
0.0 4 Prediction Set Width
5
n=449
0.6
1.0
3
Relevance
0.7
1.5
0.5
=-0.008 p=0.860
3
4 Prediction Set Width
5
2
Prediction Set Width
3
Figure 4: Pooled reliability diagrams (all four judges, α=0.10). x-axis: prediction set width; y-axis: mean absolute error (MAE) vs. human score. Error bars: 95% CI. Annotations: sample count per width. Spearman rs and p-value shown per panel. Consistency shows the clearest signal (rs = +0.34, p<0.0001); relevance is the exception (rs ≈ 0, p=0.86). Judge Pair
Coh.
Con.
Flu.
Rel.
GPT / LLaMA GPT / Qwen GPT / Mistral LLaMA / Qwen LLaMA / Mistral Qwen / Mistral
+0.12 +0.27‡ +0.31‡ −0.08 +0.08 −0.09
+0.22† +0.38‡ +0.32‡ +0.17 +0.56‡ +0.24‡
+0.45‡ +0.23† +0.81‡ +0.21† +0.23† +0.33‡
+0.28‡ +0.70‡ +0.29‡ +0.20† +0.25† +0.44‡
+0.10 +0.32 +0.38 ‡ p<.01 or better bold r >0.50
+0.36
Mean † p<.05
Table 4: Inter-judge agreement on prediction set width (Spearman r, α=0.10). Positive values indicate different judges assign wider sets to the same documents, width reflects document-level difficulty, not judge-specific noise. judges). Coherence and relevance show lower violation rates and narrower prediction sets (avg. size < 4.0 for three of four judges). This cross-diagnostic agreement is strong evidence that criterion difficulty is a fundamental property of the evaluation task, not an artifact of any single measurement approach. A document that causes preference cycles in pairwise ranking also receives a wide prediction interval in direct scoring, both signals point to the same underlying difficulty.
6
Discussion
The masked-heterogeneity problem. A practitioner who reports only ρ̄ < 5% or systemlevel τ > 0.5 will conclude that their LLM judge is reliable. Our results show this conclusion is premature: nearly half of all documents expose judge inconsistency at the per-document level. We recommend that evaluation studies report, at minimum: (a) the fraction of documents with ≥ 1 violation; and (b) the distribution of per-document violation rates, not just the aggregate mean. Prediction sets as a deployment signal. Our conformal results have an immediate practical implication: before accepting an LLM judge score, compute the prediction set. • If |C( x )| ≤ 2: the judge is likely reliable for this instance, proceed. • If |C( x )| = 5 (full scale): the judge expresses maximum uncertainty, consider human annotation. This selective escalation strategy is principled: the coverage guarantee ensures C( x ) contains the human score with at least 1−α probability. The cross-judge agreement results further 7
Preprint. Under review.
Consistency r=0.32
Fluency r=0.38
Relevance r=0.36
1.0
1.0
+0.12
+0.27
+0.31
1.0
+0.22
+0.38
+0.32
1.0
+0.45
+0.23
+0.81
1.0
+0.28
+0.70
+0.29
LLaMA +0.12
1.0
-0.08
+0.08
+0.22
1.0
+0.17
+0.56
+0.45
1.0
+0.21
+0.23
+0.28
1.0
+0.20
+0.25
Qwen +0.27
-0.08
1.0
-0.09
+0.38
+0.17
1.0
+0.24
+0.23
+0.21
1.0
+0.33
+0.70
+0.20
1.0
+0.44
Mist. +0.31
+0.08
-0.09
1.0
+0.32
+0.56
+0.24
1.0
+0.81
+0.23
+0.33
1.0
+0.29
+0.25
+0.44
1.0
LLaMA Qwen
Mist.
GPT
LLaMA Qwen
Mist.
GPT
LLaMA Qwen
Mist.
GPT
LLaMA Qwen
Mist.
GPT
GPT
0.8 0.6 0.4 0.2
Spearman r
Pairwise Inter-Judge Agreement on Prediction Set Width
Coherence r=0.10
0.0 0.2
Figure 5: Inter-judge width agreement matrices (Spearman r, α=0.10). Rows/columns: the four judges. Diagonal forced to 1.0. Coherence (leftmost) shows predominantly near-zero off-diagonal entries; fluency and relevance show consistently positive agreement, confirming that prediction width tracks document-level difficulty across model families. justify this approach: a wide set from any judge is a warning about the document, not about that specific model. Why MFAS does not help. When violations are sparse (ρ̄ < 5%) and concentrated in a minority of documents, the tournament graph is nearly acyclic. MFAS repair finds at most a handful of edges to reverse, insufficient to change aggregate rankings. Detecting these problematic documents for human follow-up is a more effective use of the transitivity signal than attempting automated repair. The coherence exception. Coherence is the one criterion where inter-judge width agreement is weak (r̄ = 0.10). We hypothesize two reasons: (1) neural summaries in SummEval vary substantially in coherence, making it a more discriminable dimension; and (2) different model families may have different internal representations of “coherence,” leading to idiosyncratic scoring patterns that do not generalize across judges.
7
Limitations
Scale and generalization. We use 30 documents × 8 systems from SummEval. Results may differ on larger subsets, other summarization datasets, or non-summarization tasks (dialogue, translation, etc.). Marginal vs. conditional coverage. Split conformal guarantees marginal coverage P(y∗ ∈ C( x )) ≥ 1−α, not per-document conditional coverage. Harder documents may receive tighter-than-justified sets in practice; conditional conformal methods (Angelopoulos & Bates, 2021) could address this. Fixed nonconformity score. We use the absolute residual |ŷ − y∗ |. Learned nonconformity scores (e.g., based on judge confidence or LLM log-probabilities) could produce tighter, more informative sets. Prompt sensitivity. Each judge uses a single prompt template per criterion. Different prompts may yield different violation rates and set widths; we leave prompt-robustness analysis to future work. Human score rounding. SummEval provides averaged annotations; rounding to integers introduces a small discretization error in calibration targets.
8
Conclusion
We presented two complementary, low-cost diagnostics for LLM judge reliability. Transitivity analysis reveals that per-document inconsistency is dramatically higher than aggregate statistics suggest, a pattern invisible to standard evaluation metrics. Conformal prediction 8
Preprint. Under review.
sets provide finite-sample coverage guarantees and a practical deployment signal: prediction set width predicts actual judge error (pooled rs = +0.576, N =1,918) and tracks document-level difficulty rather than judge-specific noise. Both diagnostics independently converge on the same finding: criterion matters more than judge. Coherence and relevance can be judged reliably by any of the four models tested; fluency and consistency should be treated with skepticism regardless of the model. We recommend that LLM evaluation pipelines adopt per-instance uncertainty reporting as standard practice, and we release all code, prompts, and cached API responses to support reproducibility.
Ethics Statement This work develops diagnostic tools for assessing the reliability of LLM-as-judge systems, with the goal of reducing uncritical reliance on automated evaluation. All experiments use publicly available data (SummEval) and commercially available LLMs accessed via a standard API. We do not collect human annotations, and the research poses no direct risk of harm to individuals. The broader impact is positive: surfacing systematic failure modes of LLM judges helps practitioners deploy automated evaluation more responsibly and identify when human oversight is needed.
References Nir Ailon, Moses Charikar, and Alantha Newman. Aggregating inconsistent information: Ranking and clustering. Journal of the ACM, 55(5):1–27, 2008. Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021. Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. Marie Jean Antoine Nicolas Caritat de Condorcet. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Imprimerie Royale, 1785. Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409, 2021. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F.T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Machine Translation, pp. 1066–1083. Association for Computational Linguistics, 2023. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. In Transactions of the Association for Computational Linguistics, volume 8, pp. 539–555, 2020. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. GPTScore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309.17012, 2023. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023. 9
Preprint. Under review.
Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam. Conformal prediction with large language models for multi-choice question answering. arXiv preprint arXiv:2305.18404, 2023. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511–2522. Association for Computational Linguistics, 2023. Lysimachos Maltoudoglou, Andreas Paisios, and Harris Sakkas. BERT-based conformal predictor for intent classification. In Proceedings of the Ninth Symposium on Conformal and Probabilistic Prediction and Applications, pp. 178–193, 2020. John W. Moon. Topics on Tournaments. Holt, Rinehart and Winston, 1968. Harris Papadopoulos, Kostas Proedrou, Vladimir Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European Conference on Machine Learning, pp. 345–356. Springer, 2002. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Shen, Tianyi Liu, Jiaming Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large language models are effective text rankers with pairwise ranking prompting. In Findings of the Association for Computational Linguistics: NAACL 2024, 2024. Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In The Twelfth International Conference on Learning Representations, 2024. Keita Saito, Saku Sugawara, and Kentaro Inui. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076, 2023. Markus Schulze. A new monotonic, clone-independent, reversal symmetric, and condorcetconsistent single-winner election method. Social Choice and Welfare, 36(2):267–303, 2011. Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in Neural Information Processing Systems, 32, 2019. Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005. Chenhui Wang, Yutai Yang, Chenghao Dang, and Wanxiang Che. Large language models are not yet human-level evaluators for abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 4215–4233. Association for Computational Linguistics, 2023. Seonghyeon Ye, Doyoung Kim, Sungdong Jang, Hyungjoo Shin, Youngjae Baek, Juho Song, Dongha Park, and Minjoon Seo. FLASK: Fine-grained language model evaluation based on alignment skill sets. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, 2024. H. Peyton Young. Condorcet’s theory of voting. American Political Science Review, 82(4): 1231–1244, 1988. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36, 2023.
10
Preprint. Under review.
A
Prompt Templates
Pairwise Preference Prompt You are evaluating the {criterion} of two summaries of the following article. Article: {document} Summary A: {system a} Summary B: {system b} Which summary is better in terms of {criterion}? Answer with exactly ‘‘A’’ or ‘‘B’’. No explanation. Direct Scoring Prompt You are evaluating the {criterion} of the following summary of an article. Article: {document} Summary: {system output} Rate the {criterion} on a scale of 1--5 where: 1 = Very Poor, 2 = Poor, 3 = Fair, 4 = Good, 5 = Excellent Respond with a single integer between 1 and 5. No explanation needed.
B
Full Conformal Results
Table 5 reports conformal coverage and average set size at all four α levels tested. Every entry exceeds the 1−α target, confirming the theoretical guarantee holds robustly across all operating points. 11
Preprint. Under review.
Table 5: Full conformal prediction results across all α levels. Mean coverage and set size across 20 random splits. All coverages meet or exceed the 1−α target (bolded).
Judge
Criterion
α = 0.05
α = 0.10
α = 0.15
α = 0.20
Cov. Size Cov. Size Cov. Size Cov. Size Coherence Consistency LL A MA-3.1-70B Fluency Relevance Q WEN -2.5-72B
4.29 4.97 4.95 2.98
.947 .992 .967 .961
3.04 4.97 4.93 2.82
.945 .889 .967 .961
2.98 4.26 4.93 2.82
.945 .889 .967 .961
2.98 4.26 4.93 2.82
Coherence 1.000 4.55 Consistency .963 4.53 Fluency .967 4.76 Relevance .975 3.63
.934 .962 .967 .961
3.62 4.48 4.76 2.99
.905 .962 .967 .961
2.99 4.48 4.76 2.99
.905 .962 .967 .961
2.99 4.48 4.76 2.99
.993 .992 .971 .964
Coherence .980 4.83 .950 4.47 .950 4.47 .901 4.10 Consistency .996 4.98 .996 4.98 .996 4.98 .970 4.94 M ISTRAL -S MALL Fluency 1.000 4.99 1.000 4.99 1.000 4.99 1.000 4.99 Relevance .995 4.15 .935 3.06 .931 2.93 .931 2.93 GPT-4 O - MINI
C
Coherence .979 Consistency .996 Fluency .995 Relevance 1.000
4.59 4.99 4.99 4.37
.971 .942 .995 .928
4.51 4.76 4.99 3.17
.899 .899 .995 .919
3.46 4.54 4.99 2.96
.876 .899 .984 .919
2.98 4.54 4.97 2.96
Violation Rates Across All Criteria
Table 6 reports aggregate and per-document transitivity violation rates for all four judges across all four criteria. Fluency consistently exhibits the highest fraction of documents with at least one violation.
Judge
Criterion
Agg. ρ̄
% docs ≥1
LL A MA-3.1-70B
Coherence Consistency Fluency Relevance
0.008 0.012 0.033 0.011
33.3% 36.7% 66.7% 36.7%
Q WEN -2.5-72B
Coherence Consistency Fluency Relevance
0.022 0.025 0.024 0.023
50.0% 56.7% 60.0% 50.0%
M ISTRAL -S MALL
Coherence Consistency Fluency Relevance
0.041 0.036 0.034 0.029
50.0% 63.3% 53.3% 53.3%
GPT-4 O - MINI
Coherence Consistency Fluency Relevance
0.014 0.013 0.026 0.020
46.7% 43.3% 53.3% 53.3%
Table 6: Aggregate and per-document violation rates across all four criteria. Fluency consistently shows the highest fraction of documents with ≥ 1 violation, reaching 66.7% for LL A MA-3.1-70B. 12
Preprint. Under review.
D
Coverage vs. α Curves
Empirical Coverage
1.00
1.00 0.950
0.95
0.925
0.907
0.90
0.85
0.85
0.80
1.00
0.80
Coherence (mean±std) 1 guarantee
0.05
0.10
0.983
0.982
0.15
0.20
0.982
0.980
(miscoverage level)
Fluency
0.75
1.00
0.95
0.95
0.90
0.90
0.85
0.85
0.80 0.75
0.80
Fluency (mean±std) 1 guarantee
0.05
0.10
0.15
(miscoverage level)
0.987
0.973
0.937
0.95
0.90
0.75
Empirical Coverage
0.988
Empirical Coverage vs. Level Validity of Conformal Guarantee (shaded band = ±1 std across 4 judges) Coherence Consistency
0.75
0.20
0.930
Consistency (mean±std) 1 guarantee
0.05
0.984
0.10
0.15
(miscoverage level)
0.20
Relevance
0.946
0.943
0.943
0.15
0.20
Relevance (mean±std) 1 guarantee
0.05
0.10
(miscoverage level)
Figure 6: Empirical coverage vs. α. Shaded bands: ±1 std across the four judges. Dashed line: theoretical guarantee 1−α. Coverage meets or exceeds the guarantee at every operating point for all four criteria.
13