When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability Zongyou Yang
Yinghan Hou
Xiaokun Yang∗
Dyson School of Design Engineering, Imperial College London London, United Kingdom [email protected]
Department of Electrical and Electronic Engineering, Imperial College London London, United Kingdom [email protected]
School of Electronic Information, Nanchang Institute of Technology Nanchang, China [email protected]
arXiv:2607.08535v1 [cs.CL] 9 Jul 2026
Abstract An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurementvalidity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2–M2.7 released APIs. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B→4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and verbosity bias. Repeated-sample juries add little when errors are correlated. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation. We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.
CCS Concepts • Computing methodologies → Natural language processing; Machine learning.
Keywords LLM-as-judge, automatic evaluation, model scaling, evaluation reliability, bias, jury aggregation
1
Introduction
LLM-as-judge evaluation is now often used as a measuring instrument for model quality. The difficulty is that the instrument is itself a model. When a system’s score changes after replacing the evaluator, the change is ambiguous: it may indicate that the new judge is more capable, that it is biased differently, that it fails on a different slice of the benchmark, or that the evaluation pipeline parsed and aggregated its outputs differently. This ambiguity is not a minor implementation detail. It determines whether an LLM-as-judge score can be interpreted as evidence about the candidate systems at all. We call this the evaluator-replacement ambiguity: when the measured preference outcome changes after replacing the judge, the source of the change is not identifiable from accuracy alone. This paper studies the ambiguity through two observable interventions available in evaluation practice. The first is a parameter-scaling decision, represented by Qwen3 dense judges from 1.7B to 32B parameters [21]. The second is a released-model upgrade path, represented by MiniMax M2–M2.7 APIs evaluated as released [15]. ∗ Corresponding author.
These axes are evidence sources rather than the paper’s object of explanation. The MiniMax M2 report documents the released series and its agent-oriented training pipeline, but it does not make our API sequence a controlled ablation; consequently, this paper does not make a causal claim about MiniMax internals. This formulation leads to three research questions. RQ1 asks whether judge reliability improves similarly along a parameter axis and a released-model upgrade path. If evaluator capability is the dominant factor, both axes should show consistently positive adjacent gains across datasets; if reliability is workload-dependent, gains should vary by dataset and significance should not transfer uniformly. RQ2 asks whether higher aggregate accuracy also reduces known judge biases. Higher-capability judges should be less bias-sensitive, but nonzero flip rates would show that capability does not eliminate measurement artifacts. RQ3 asks whether protocol-level upgrades, such as juries and debate, change reliability beyond single-judge scaling. Jury gains should be limited when error correlation is high, while debate shifts should depend on capability asymmetry and protocol auditability. The central thesis is that evaluator capability is important but incomplete: the same judge can look different across workloads, bias probes, correlated votes, and protocol implementations. This paper contributes an auditable measurement framework for LLM-as-judge reliability, separating model capability from workload dependence, bias behavior, correlated voting, and measurement artifacts. First, it presents a two-axis evaluator study that compares parameter scaling with released-model generation while explicitly avoiding causal claims about MiniMax internals. Second, it decomposes reliability into single-judge accuracy, bias probes, correlated jury errors, and debate auditability. Third, it distills a reporting standard: benchmark slices, parse-shared tests, A/B randomization, 𝜌 estimates, and protocol logs. The study yields three main results. First, judge upgrades are not interchangeable: parameter scaling produces one robust low-end gain, while adjacent MiniMax releases do not. Second, higher accuracy reduces but does not remove bias. Third, aggregation protocols are interpretable only when their assumptions are auditable: jury voting is bounded by correlated errors, and debate shifts cannot be attributed to deliberation without parser logs.
2
Related Work
LLM judges and preference evaluation. MT-Bench, Chatbot Arena, PandaLM, JudgeLM, G-Eval, Auto-J, Prometheus, and RewardBench have helped establish LLM judges as practical evaluators for open-ended generation [8, 9, 11, 12, 14, 19, 23, 24]. These studies
Z. Yang, Y. Hou, and X. Yang
Score changes are produced by the whole measurement pipeline The judge is one component of the instrument, not an external observer.
Candidate responses
Benchmark slice
Judge model
Prompt / decoding
Parser
Aggregation / debate
model capability changes validity
order and prompt change bias
parser and fallback create artifacts
repeated samples create dependence
Reported score
Figure 1: LLM-as-judge as a measurement pipeline. A reported score depends not only on candidate responses, but also on the benchmark slice, judge model, prompt and decoding choices, parser, and any aggregation or debate protocol. show that LLM judges can correlate with human preferences, but also that correlations depend on task, prompt, scoring scale, and model family. Our work follows this empirical tradition but treats the judge itself as the object of scaling analysis. The gap is that prior reports often present evaluator performance as a one-dimensional model ordering, leaving unclear whether gains arise from capability, benchmark slice, bias reduction, or protocol design. Judge bias and protocol effects. Prior work documents position bias, verbosity bias, sensitivity to rating format, and reliability changes caused by evaluation design choices [5, 16, 17, 20]. These effects matter because they can change rankings without changing candidate responses. What remains under-specified is how bias changes jointly with evaluator scaling rather than under isolated prompt perturbations. Unlike a full prompt-factorial study, our experiments use a fixed main prompt and treat prompt sensitivity as a robustness check; the main focus is how model capability, bias, aggregation, and debate interact under a controlled reporting protocol. Aggregation and debate. Classical Condorcet-style results predict gains from independent voters, while work on correlated voters shows that dependence can sharply reduce those gains [3, 10]. In LLM evaluation, self-consistency, multi-agent debate, and ChatEvalstyle protocols offer analogous routes to aggregation or deliberation [2, 4, 7, 13, 18]. Existing studies often emphasize final accuracy; our focus is whether the aggregation assumptions required for such gains are empirically satisfied. Our jury experiment directly estimates error correlation, and our debate experiment is reported as an auditability case study.
Table 1: Operationalizing LLM-as-judge reliability. Construct
Operational measure
Exp.
Judgment validity Bias robustness
Accuracy, Cohen’s 𝜅 , Spearman Position flip, verbosity bias, granularity sensitivity Error correlation 𝜌 , jury gain Parse status, fallback behavior, intermediate verdicts
1 2
Aggregation independence Protocol auditability
3.2
We treat LLM-as-judge reliability as a multi-dimensional measurement property rather than a single accuracy score. In this study, reliability consists of four observable components: judgment validity, bias robustness, aggregation independence, and protocol auditability. Table 1 maps each construct to an operational measure and experiment.
Models and Datasets
The main panel contains eight evaluated judges: Qwen3-1.7B, Qwen34B, Qwen3-14B, Qwen3-32B, MiniMax-M2, MiniMax-M2.1, MiniMaxM2.5, and MiniMax-M2.7. This panel separates an approximate parameter intervention from an observed released-model intervention. GLM-5.1 and mimo-v2-pro are used as cross-family reference judges in selected experiments, but are not members of either axis. Four datasets cover complementary judgment settings so that reliability is not inferred from a single task slice. LLMBar is an adversarial pairwise benchmark with 419 examples [22]. PandaLM testset-v1 provides broad-domain pairwise judgments; after removing tie-majority examples, 894 valid pairwise examples remain [19]. A 2,000-example seed-42 sample from Chatbot Arena provides broad human preference comparisons, with 1,997 valid pairwise examples [23]. Our Judge’s Verdict slice uses 200 TechQA-derived examples with three-level pointwise labels, supporting exact-match and rank-based evaluation [1, 6].
3.3
3 Experimental Design 3.1 Operationalizing Judge Reliability
3 4
Metrics and Protocols
Pairwise experiments report accuracy and Cohen’s 𝜅 on parseable verdicts; Judge’s Verdict additionally reports Spearman rank correlation. Main single-judge runs use near-greedy decoding (𝑇 = 0.1) to reduce sampling noise when estimating model-level reliability. Homogeneous juries sample one judge multiple times at 𝑇 = 0.7 because repeated low-temperature calls would understate the dependence structure among stochastic jurors; heterogeneous juries combine different judges at 𝑇 = 0.1 to isolate model composition from sampling variance. Adjacent model comparisons use exact two-sided McNemar tests on parse-shared examples, so significance reflects paired judgment changes rather than differences in parser coverage. Holm correction controls the family of 18 adjacent tests.
When the Judge Changes, So Does the Measurement
The experiments instantiate the constructs in Table 1. Experiment 1 measures single-judge validity along the two axes. Experiment 2 measures LLMBar position, verbosity, and granularity robustness. Experiment 3 evaluates majority-vote juries under measured error correlation. Experiment 4 evaluates a structured debate protocol as an auditability case study. Four robustness checks cover human-ceiling calibration, corrected adjacent significance, limited prompt sensitivity, and Arena seed stability.
4 Results 4.1 Finding 1: Judge Upgrades Are Not Interchangeable A capability-only account would predict similar adjacent gains along both upgrade paths; the paired tests do not show that pattern. Table 3 summarizes the main single-judge pattern. Qwen3 improves sharply from 1.7B to 4B on LLMBar (0.463 to 0.617) and more modestly on Arena, but later parameter steps are smaller and not uniformly monotone. MiniMax release generations show no reliable adjacent improvement in this panel: adjacent pairwise differences are at most 0.022 accuracy, and none of the nine MiniMax adjacent tests reaches uncorrected 𝑝 < 0.05. After Holm correction across all 18 adjacent tests, only the Qwen3 1.7B to 4B steps on LLMBar and Arena remain significant. The relevant conclusion is not that one family is universally stronger. No judge wins all datasets: GLM-5.1 leads LLMBar, MiniMaxM2.7 leads PandaLM, and mimo-v2-pro leads the sampled Arena slice. This shifts the design question from one-dimensional evaluator ordering to slice-specific measurement validity.
4.2
Finding 2: Stronger Judges Are Less Biased, Not Unbiased
On LLMBar, the bias probes move in the same direction as capability, but they do not disappear. The position-flip rate falls from 0.320 for Qwen3-1.7B to 0.117–0.147 for MiniMax releases. Verbosity bias under a fixed generic and innocuous padding string falls from 0.547 for Qwen3-1.7B to roughly 0.13 for MiniMax. Granularity sensitivity is also smaller for the stronger judges. This pattern is best interpreted as a capability–fairness association rather than a mechanism. Across the eight evaluated judges, the correlation between LLMBar accuracy and position-flip rate is strong (Pearson 𝑟 = −0.957), but it is measured on one dataset and one model panel. Stronger judges are also not unbiased: MiniMaxM2.7 still changes 14.7% of verdicts under A/B reversal. Thus position randomization and slice-level bias reporting remain necessary even when using high-accuracy judges.
4.3
Finding 3: Jury Size Matters Less Than Error Dependence
The jury experiment fails for a familiar statistical reason: the votes are not independent. Majority voting is therefore not a generic reliability amplifier for LLM judges. For homogeneous juries, we estimate the intra-class error correlation 𝜌 from the vote matrix and compare three quantities: the independence prediction, a 𝜌corrected beta-binomial prediction, and observed jury accuracy. Let 𝑝 be the single-judge baseline accuracy, 𝑞 ∼ Beta(𝛼, 𝛽), 𝛼 = 𝑝𝑠,
𝛽 = (1 − 𝑝)𝑠, and 𝑠 = 1/𝜌 − 1. The jury accuracy is then the beta-binomial probability that a majority of 𝐾 votes are correct. The correction is empirically necessary. Across homogeneous juries, the independence prediction has median absolute error 0.078 on LLMBar and 0.093 on PandaLM. The 𝜌-corrected prediction reduces these errors to 0.008 and 0.004, with maximum error below 0.02. Estimated correlations are high: Qwen3 homogeneous juries have 𝜌 = 0.944–0.972 on LLMBar, while MiniMax juries are lower but still correlated at 𝜌 = 0.664–0.706. As a result, increasing jury size barely moves accuracy; for Qwen3-1.7B on LLMBar, 𝐾 = 1, 3, 5 yields 0.463, 0.475, and 0.482. Heterogeneous juries also underperform Poisson-binomial independence predictions. Family mixing alone therefore does not restore independent errors under a shared prompt. This shifts the relevant design question from “how many jurors should be sampled?” to “how independent are their errors?”
4.4
Finding 4: Debate Shifts Require Audit Trails
The debate runs produce the largest protocol-level shifts in the paper, but they are also the least auditable. We include debate not as evidence for a deliberation effect, but as a stress test for protocol auditability. The structured-debate experiment pairs two judges for up to three rebuttal rounds on LLMBar. Cross-capability pairs show the largest final-vs-round-1 accuracy shifts: Qwen31.7B paired with GLM-5.1, MiniMax-M2.7, or mimo-v2-pro changes final decisions toward higher measured accuracy by +0.317, +0.305, and +0.289; Qwen3-4B paired with GLM-5.1 shifts decisions by +0.243 and reaches 0.897. Same-family and top-vs-top shifts are much smaller, ranging from +0.005 to +0.050. These changes show that protocol-level interventions can alter judge outcomes substantially; the missing parser audit trail shows why such shifts should not be accepted without measurement logs. The implementation records round verdicts and final verdicts, but not raw responses or parse-success flags. Round-1 parse failures fall back to A, and later parse failures retain the previous verdict. Since fallback rates cannot be audited without rerunning the experiment, the debate results are best read as a suggestive protocol pattern and a reporting requirement, not as a clean estimate of deliberation.
4.5
Robustness Checks
The robustness checks support the direction of the main findings while keeping their scope narrow. Human-ceiling calibration shows that PandaLM retains headroom: the best judge has matched leave-one-annotator-out 𝜅 = 0.753 versus a human ceiling of 0.920. Judge’s Verdict is closer to its noisy ceiling, with best-judge 𝜅 = 0.620 versus human 𝜅 = 0.562. Prompt-sensitivity checks on LLMBar preserve the cross-family ordering and show greater crossprompt stability for stronger judges. Arena seed perturbations produce 0.004–0.032 accuracy spread, smaller than the robust Qwen3 1.7B to 4B Arena gap. These checks do not establish cross-family generality, but they reduce several obvious alternative explanations.
5
Discussion
Principle. A judge score is interpretable only when the evaluated slice, perturbation sensitivity, error dependence, and protocol state
Z. Yang, Y. Hou, and X. Yang
Evaluator-replacement interventions Qwen3 parameter axis MiniMax release path
Datasets
Cross-family references
1.7b
4b
14b
32b
0.463
0.617
0.647
0.678
M2
M2.1
M2.5
M2.7
mimo-v2-pro (0.883)
0.829
0.839
0.832
0.832
numbers: LLMBar accuracy
glm-5.1 (0.900)
LLMBar 419 | PandaLM 894 | Arena 1,997 | Judge's Verdict 200
Measurements validity | bias probes | jury rho correction | debate auditability | robustness
Figure 2: Study design. The paper uses a Qwen3 parameter axis and a MiniMax release-generation axis as observable evaluatorreplacement interventions, then tests whether single-judge scaling, bias probes, jury aggregation, and structured debate answer the same reliability question. Table 2: Main answers by research question. Research question
Answer
Key evidence
Implication
RQ1: Do scaling and release upgrades similarly improve reliability? RQ2: Does higher accuracy reduce bias?
No.
RQ3: Do jury and debate protocols add reliability?
Conditionally.
Qwen3 1.7B→4B is robustly significant; MiniMax adjacent releases are not significant. Position flips decline, but MiniMax-M2.7 still changes 14.7% under A/B reversal. Jury gains are small under high 𝜌 ; debate shifts are large but lack an audit trail.
Model upgrading is not a uniform reliability intervention. High-accuracy judges still need bias probes. Aggregation protocols must report dependence and audit trails.
Partly.
Table 3: Single-judge aggregate results. Values are accuracies except Judge’s Verdict, where exact match is reported for comparability; brackets give Wilson 95% intervals.
LLMBar PandaLM Arena Judge’s Verdict
Qwen3 1.7B → 32B 0.463 [0.416,0.511] → 0.678 [0.632,0.721] 0.779 [0.751,0.805] → 0.769 [0.740,0.795] 0.625 [0.604,0.646] → 0.688 [0.667,0.708] 0.595 [0.526,0.661] → 0.530 [0.461,0.598]
Adjacent McNemar tests by dataset (* = uncorrected p < 0.05)
p<.001*
4b 14b
0.203
0.816
0.164
0.148
0.032*
0.696
LLMBAR
PANDALM
ARENA
3 2 1 0
m2 m2.1
0.488
0.716
1.000
m2.1 m2.5
0.883
1.000
0.099
m2.5 m2.7
0.644
0.450
0.949
LLMBAR
PANDALM
ARENA
4 3 log10 p
0.090
MiniMax (release-generation axis)
4
log10 p
1.7b 4b
p<.001*
14b 32b
Qwen3 (parameter axis)
2 1
Q1.7B
Accuracy and position sensitivity on LLMBar Pearson r = 0.957 slope = 0.49
0.300 Position-bias flip rate (Exp2)
Best judge GLM-5.1, 0.900 [0.868,0.925] MiniMax-M2.7, 0.857 [0.833,0.878] mimo-v2-pro, 0.742 [0.722,0.761] GLM-5.1, 0.680 [0.612,0.741]
Dataset
0.325 0.275
Q4B
0.250 0.225 0.200
Q14B Q32B
0.175 M2.7 M2.5
0.150 0.125
Qwen3 MiniMax
M2 M2.1
0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 LLMBar judge accuracy (Exp1)
0
Figure 3: Adjacent-pair McNemar tests for the two axes. After Holm correction over 18 adjacent tests, only the Qwen3 1.7B→4B steps on LLMBar and Arena remain significant; no MiniMax adjacent release step is significant in this panel.
Figure 4: Capability–fairness association on LLMBar. Higher single-judge accuracy co-varies with lower position-flip rate across the eight evaluated judges, but the non-zero flip rates show that stronger judges still require A/B randomization and bias reporting.
are jointly reported. The practical recommendation is therefore simple: an LLM-as-judge result should not be accepted as reliable unless it reports what slice was judged, how biased the judge was
under simple perturbations, how correlated repeated judgments were, and whether protocol-level decisions were auditable.
When the Judge Changes, So Does the Measurement (a) Theory, rho-corrected, observed
0.8
Homogeneous, theory Homogeneous, rho-corrected Heterogeneous y=x
0.7
(c) Homogeneous-jury dependence
0.47
m2 m2.1 m2.5
cross family mid qwen3 4b 14b 32b
0.6
(b) Jury size under dependence Theory (rho=0) rho-corrected Observed
0.48
Accuracy
Observed accuracy
0.9
0.45 0.44
0.5
0.6 0.7 0.8 Predicted accuracy
0.9
0.43
0.97
Q4B
0.96
Q1.7B
0.46
qwen3 1.7b 4b 14b
0.5
Q14B
1
3 Jury size k (Qwen3-1.7B)
0.94
M2.7
0.71
M2.1
0.67
M2
0.67
M2.5
0.66
5
0.0
0.2
0.4 0.6 0.8 Error correlation rho
Qwen3 MiniMax
1.0
Figure 5: Jury behavior on LLMBar. Majority voting provides little gain because sampled jurors make correlated errors; the 𝜌-corrected beta-binomial model tracks observed jury accuracy much better than the independence assumption.
Structured-debate decision shifts same-family cross-family top cross-capability
Final - round-1 accuracy
0.30 0.25
q1.7b glm-5.1 q1.7b m2.7 q1.7b mimo q4b glm-5.1
0.20 0.15 0.10 0.05 0.00
0.0 0.1 0.2 0.3 0.4 Single-judge capability gap |acca accb| (Exp1 LLMBar)
Figure 6: Structured-debate accuracy shift versus single-judge capability gap on LLMBar. Cross-capability pairs produce the largest shifts, but the debate implementation’s parse-fallback behavior makes the result suggestive rather than causal.
remains measurable. Jury experiments show that aggregation gains depend on error correlation. Debate experiments show that protocol changes can be large enough to rival scaling effects, but only if the implementation preserves an audit trail for parse failures, tie-breaking, and abstentions. For pairwise datasets, this means parseable-subset definitions, parse-shared significance tests, and A/B randomization. For juries, it means reporting 𝜌 in addition to 𝐾. For debate or multi-agent protocols, it means storing raw responses, parser status, and every intermediate vote before interpreting final-score shifts. These requirements are operationally simple, but without them the apparent reliability of a judge can be an artifact of the evaluation pipeline.
6
7 Table 4: A minimal audit trail for LLM-as-judge reports. Reliability risk
Required report item
Dataset-slice shift Parser artifacts Position/verbosity bias Correlated jury errors Debate protocol artifacts
Valid 𝑁 , parseable 𝑁 , tie handling Hidden sample mismatch Parser success rate, fallback rule Artificial accuracy changes A/B reversal, padding probe Ranking artifacts 𝜌 and 𝐾 , not 𝐾 alone Overestimated voting gains Raw outputs, parser status, round ver- Unauditable protocol gains dicts Paired tests, Holm correction, uncer- False adjacent-step claims or overtainty intervals where estimable stated precision
Statistical overclaiming
Failure mode avoided
The results suggest two methodological lessons for LLM-as-judge evaluation. First, judge scaling should be reported as a design space rather than a single evaluator ordering. In this panel, the clearest parameter-axis gain occurs at the low end, while the MiniMax release-generation axis changes little in the measured adjacent comparisons. This should not be read as a claim that all release generations lack meaningful gains, or that the MiniMax releases isolate one training stage. The supported conclusion is narrower: under the same datasets and prompts, the two observed upgrade paths have different adjacent-pair patterns. Second, reliability reporting has to go beyond single-number accuracy and name the unit of analysis. Bias probes show that stronger judges can be less brittle, but they also show that bias
Threats to Validity
Construct validity. Reliability is operationalized through selected proxies: accuracy and agreement, bias probes, error correlation, and auditability logs. These do not exhaust all reliability dimensions, such as calibration, long-form rationale quality, or domain-specific consistency. Internal validity. The Qwen3 axis is closer to a parameter sweep, whereas the MiniMax axis is an observed release sequence rather than a controlled ablation. The debate experiment is especially limited by missing raw outputs and parse-success logs. Statistical conclusion validity. McNemar tests are adjacentpair tests and do not constitute a formal between-axis test. Multiple comparisons are controlled through Holm correction, but sample sizes and parser coverage differ across datasets. External validity. Results are conditioned on two model families, two reference judges, four datasets, a small number of prompts, and one primary decoding regime per experiment. Additional domains, languages, and instruction distributions are needed before turning these observations into general judge-selection rules.
Conclusion
This paper presents a two-axis empirical study of LLM-as-judge reliability. The results show that evaluator capability is important but incomplete: reliability is also shaped by dataset slice, bias behavior, correlated errors, and protocol implementation. In the tested panel, the strongest robust adjacent effect is the Qwen3 1.7B to 4B transition; no MiniMax adjacent release step is significant; majorityvote juries are limited by high error correlation; and debate with a strong reference changes outcomes substantially but lacks sufficient evidence for causal interpretation under the current logs. These findings recast LLM-as-judge reliability as a measurement-validity problem rather than a model-selection problem alone.
Acknowledgments This work was supported by the Open Research Project of the State Key Laboratory of Industrial Control Technology, China (Grant No. ICT2025B70); the Jiangxi Provincial Natural Science Foundation (Grant Nos. 20242BAB20041 and 20232BAB212006); the Hubei Provincial Natural Science Foundation of China (Grant Nos. 2023AFB474 and 2024AFB881); the Anhui Provincial Special Project for Special Needs in Humanities and Social Sciences (Grant No. 2025AHGXSK50067); and the Postgraduate Quality Engineering Project of Anhui Province (Grant No. 2024jyjxggyjY232).
Z. Yang, Y. Hou, and X. Yang
References [1] Vittorio Castelli, Rishav Chakravarti, Saswati Dana, Anthony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, Dinesh Khandelwal, Scott McCarley, Mike McCawley, Mohamed Nasr, Lin Pan, Cezar Pendus, John Pitrelli, Saurabh Pujar, Salim Roukos, Andrzej Sakrajda, Avirup Sil, Rosario Uceda-Sosa, Todd Ward, and Rong Zhang. 2019. The TechQA Dataset. arXiv preprint arXiv:1911.02984 (2019). arXiv:1911.02984 [2] Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate. arXiv preprint arXiv:2308.07201 (2023). arXiv:2308.07201 [3] Marie Jean Antoine Nicolas Caritat de Condorcet. 1785. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Imprimerie Royale, Paris. [4] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In International Conference on Machine Learning (ICML). arXiv:2305.14325 [5] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv preprint arXiv:2404.04475 (2024). arXiv:2404.04475 [6] Steve Han, Gilberto Titericz Junior, Tom Balough, and Wenfei Zhou. 2025. Judge’s Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement. arXiv preprint arXiv:2510.09738 (2025). arXiv:2510.09738 [7] Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with More Persuasive LLMs Leads to More Truthful Answers. International Conference on Machine Learning (ICML) (2024). arXiv:2402.06782 [8] Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing Fine-grained Evaluation Capability in Language Models. International Conference on Learning Representations (ICLR) (2024). arXiv:2310.08491 [9] Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. arXiv preprint arXiv:2405.01535 (2024). arXiv:2405.01535 [10] Krishna K. Ladha. 1992. The Condorcet Jury Theorem, Free Speech, and Correlated Votes. American Journal of Political Science 36, 3 (1992), 617–634. doi:10.2307/2111584 [11] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. RewardBench: Evaluating Reward Models for Language Modeling. arXiv preprint arXiv:2403.13787 (2024). arXiv:2403.13787 [12] Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023. Generative Judge for Evaluating Alignment. arXiv preprint arXiv:2310.05470 (2023). arXiv:2310.05470 [13] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2023. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. arXiv preprint arXiv:2305.19118 (2023). arXiv:2305.19118 [14] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. In Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2303.16634 [15] MiniMax. 2026. The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence. arXiv preprint arXiv:2605.26494 (2026). arXiv:2605.26494 [16] Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity Bias in Preference Labeling by Large Language Models. arXiv preprint arXiv:2310.10076 (2023). arXiv:2310.10076 [17] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large Language Models are not Fair Evaluators. arXiv preprint arXiv:2305.17926 (2023). arXiv:2305.17926 [18] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. International Conference on Learning Representations (ICLR) (2023). arXiv:2203.11171 [19] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2023. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. arXiv preprint arXiv:2306.05087 (2023). arXiv:2306.05087 [20] Yusuke Yamauchi, Taro Yano, and Masafumi Oyamada. 2025. An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability. arXiv preprint arXiv:2506.13639 (2025). arXiv:2506.13639 doi:10.48550/arXiv.2506.13639
[21] An Yang et al. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 (2025). arXiv:2505.09388 [22] Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating Large Language Models at Evaluating Instruction Following. In International Conference on Learning Representations (ICLR). arXiv:2310.07641 [23] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MTBench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. arXiv:2306.05685 [24] Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. JudgeLM: Fine-tuned Large Language Models are Scalable Judges. arXiv preprint arXiv:2310.17631 (2023). arXiv:2310.17631