Cherry-pick Override: Unsafe Directional Commitment in LLM Judges under Mixed Evidence Haoran Xu University of Glasgow [email protected]
arXiv:2606.07834v1 [cs.SE] 5 Jun 2026
Abstract LLM judges increasingly turn verdicts into system commitments—logged answers, routing decisions, moderation actions. Under mixed evidence (claims with both supporting and refuting sources) this is unsafe: when the benchmark schema exposes C ONFLICTING as the authorized non-directional verdict, returning S UPPORTS/R EFUTES is an unauthorized directional commitment, a failure mode we name Cherry-pick Override (CCO). We define CCO under an explicit task contract and report it with a same-denominator diagnostic protocol that tracks coverage, selective error, conditional and same-denominator CCO rates, pure-S/R accuracy, and conflict recall, paired with matched-coverage bootstrap and a random-veto null. On the AVeriTeC Conflicting/Cherrypicking subset (NC =150), three-option judges (a pre-contract baseline that has no C ONFLICTING verdict available) return a directional verdict on more than 84% of mixed-evidence claims; under the incontract typed schema, three-judge majority voting amplifies direction-on-conflict on AVeriTeC (0.887 vs. 0.840; 95% CI [+0.013, +0.080]) but does not replicate on the cross-dataset VitaminC-Mixed control (CI [−0.060, +0.060]). Walking an intervention ladder of common single-channel fixes (typed vocabulary, panel aggregation, confidence thresholding, validator-only filtering), we find that each leaves a distinct residual failure: typed vocabulary leaves a structurally avoidable residual; panel aggregation by majority vote suppresses single-judge C ONFLICTING dissent in 48% of CCO cases; the panel is well-calibrated for direction (ECE = 0.07 on pure-S/R) so confidence cannot operationally separate CCO from correct directional commits at any tested threshold; validator-as-classifier collapses pure-evidence accuracy by nearly half. The minimal two-channel reference probe we study reaches operating points neither single channel reaches, but matched-coverage magnitude differences vs the confidence baseline are modest and bootstrap CIs straddle zero; under an apples-to-apples random Stage-1 null, the probe’s promotion to C ONFLICTING is structurally targeted on AVeriTeC (empirical p < 1/2001; AccS/R and RecC are algebraically coupled under fixed k, so this is one selection event expressed on two axes, not two independent tests) and weaker but in the same direction on VitaminCMixed—which is a selectivity result, not a magnitude one. We argue, on the basis of the ladder analysis, for an external commitment-control layer that separates verdict generation from commitment authorization, with structural-evidence and confidence as orthogonal channels and N O -C OMMIT as a routed controller state—not a new verification verdict. We
present the probe as a design probe, not a solution.
1
Introduction
A long-standing concern in the literature on selective prediction, and more broadly in the philosophy of testimony, is that holding a judgment is not the same as being entitled to commit to it: a classifier may be permitted to abstain when its evidence is structurally inadequate, and a witness may be required to qualify testimony when the underlying record is divided. Contemporary LLM-as-judge pipelines inherit this distinction in an acute form, because a judge’s verdict is often recorded as the system’s commitment with no intervening warrant. The question becomes sharp under mixed evidence—claims with both supporting and refuting sources, where the structurally correct outcome is the nondirectional verdict C ONFLICTING—and motivates a design question the judge alone cannot answer: when is an AI system not authorized to commit a final verdict under mixed evidence? We diagnose one specific way the entitlement fails in current pipelines, a failure we name Cherry-pick Override (CCO): the case in which an LLM judge issues a directional verdict on mixed evidence when the task schema exposes C ONFLICTING as the authorized non-directional verdict. CCO is wrong commitment type, not wrong label— distinct from accuracy error, hallucination, and calibration error—and matters because the verdict, once committed, propagates downstream as if settled. On AVeriTeC’s Conflicting/Cherrypicking subset, three-option judges overcommit on more than 84% of mixed-evidence claims. An illustrative case. For the claim “face masks cause hypoxia,” retrieved evidence states that masks do not restrict oxygen in healthy individuals, while a separate strand notes prolonged N-95 use in patients with preexisting lung disease could raise CO2 . A typed panel reads both strands and issues R EFUTES at high mean self-reported confidence; no confidence threshold within the explored range blocks it, because the model is genuinely confident in the direction it picked. The structural property that should have withheld the commitment—a material conflict on a load-bearing subclaim—is visible to an evidence-state validator but invisible to a confidence score. The example shows that confidence and structural evidence carry orthogonal information.
An analysis-paper contract. This paper is diagnostic rather than prescriptive. We do not claim to solve mixedevidence verification, nor to propose an optimal commitment controller. Instead, we identify a failure mode, define the task contract under which it is measurable, and use an intervention ladder to show why common single-channel fixes are insufficient. The two-channel controller we instantiate is included only to make the separation principle operational; it is a design probe, not a solution claim. The strongest claim the paper supports is that the structural and confidence channels are orthogonal and that each common single-channel fix leaves a distinct residual failure, not that any controller we implement is generally superior. Contributions. The paper makes four connected moves. We first name and operationalize Cherry-pick Override under a task contract that exposes C ONFLICTING as the authorized non-directional verdict, fixing an output taxonomy that distinguishes verification verdicts from controller routing states. We then introduce a same-denominator diagnostic protocol—joint metrics on a single denominator, matchedcoverage paired bootstrap, and an apples-to-apples random Stage-1 null that promotes k random commits to C ONFLICTING so that the null can in principle reach any of the controller’s metric values. Using this protocol, we walk an intervention ladder over the common single-channel fixes— typed vocabulary, panel aggregation, confidence thresholding, validator-only filtering—and show that each leaves a distinct residual failure (Table 3); CCO persists across the configurations we examine. On the basis of the ladder, we argue for an architectural separation between verdict generation and commitment authorization, with structural evidence and confidence treated as orthogonal authorization channels and N O -C OMMIT routed as a controller state rather than a verification verdict; we instantiate this separation as a minimal two-channel reference probe, included to make the principle operational rather than as a method claim.
2
Related Work
LLM-as-judge and evidence-grounded fact verification. Large language models are increasingly deployed as judges or verification components, evaluated either by their finallabel accuracy on fact-checking benchmarks or by aggregating multiple LLM judgments into a panel decision (Verga et al. 2024; Zheng et al. 2023; Liu et al. 2023). Substrates such as AVeriTeC (Schlichtkrull, Guo, and Vlachos 2023), FEVER and its successors (Thorne et al. 2018), and VitaminC (Schuster, Fisch, and Barzilay 2021) provide claim– evidence pairs with explicit C ONFLICTING and contrastiveevidence labels, and recent work documents that LLMs can be confidently wrong even when given the relevant documents (Hagar, Agustianto, and Diakopoulos 2025; Ji et al. 2023). Our paper builds on this substrate and on the AVeriTeC lineage of QA-style subclaim decomposition, but differs in target: where existing evaluations report label accuracy or judge reliability, we study commitment authorization—when a judge should not have committed at all— and isolate Cherry-pick Override as unauthorized directional commitment on materially mixed evidence under a task con-
tract that exposes C ONFLICTING as a first-class verdict. Selective prediction and abstention from Chow’s rule to LLM judges. The canonical formulation of abstention is Chow’s rule (Chow 1957), formalized in the learning-withrejection framework (Cortes, DeSalvo, and Mohri 2016; Bartlett and Wegkamp 2008), which licenses a reject option whenever no class posterior crosses a confidence threshold. Modern selective classification inherits and generalizes this template: a learned scoring function ranks inputs by reliability and a coverage–risk curve traces the gain from ceding low-confidence items (El-Yaniv and Wiener 2010), with neural-network instantiations such as SelectiveNet jointly training a predictor and a reject head (Geifman and El-Yaniv 2019). Carried into the LLM-as-judge regime, this lineage becomes a confidence-thresholded escalation policy: Jung, Brahman, and Choi (2025) selectively decide when to trust an LLM judge’s preference using a confidence-based criterion with provable agreement-rate bounds, and verifierscaling at test time (Lifshitz, McIlraith, and Du 2025) composes multiple aspect verifiers via binary True/False approvals to filter candidate outputs. Self-consistency aggregation (Wang et al. 2023) provides a related test-time scaling tool that operates on the answer side rather than the verification side. We adopt the risk–coverage and matched-coverage primitives from this tradition, but argue that Cherry-pick Override is a failure the tradition treats as out of scope: the judge is confident, a directional answer is reachable from the inputs, and the reason to withhold lives in the structure of the evidence—supporting and refuting strands coexist on a load-bearing subclaim—rather than in the judge’s posterior. Our same-denominator protocol and random-veto null adapt selective primitives to a setting where the abstention threshold should reflect structural evidence as much as model uncertainty; we do not claim to invent selective prediction or abstention. Calibration, conformal prediction, and distributionfree coverage. A complementary line treats overconfident commitment as a calibration problem, diagnosed by expected calibration error and related reliability diagnostics (Guo et al. 2017), and a more recent strand replaces parametric calibration with distribution-free coverage guarantees. Conformal prediction, in particular, supplies prediction sets whose marginal coverage is controlled at a userchosen level without distributional assumptions on the underlying scorer (Vovk, Gammerman, and Shafer 2005; Angelopoulos and Bates 2021). Our matched-coverage protocol is morally close to conformal coverage control—both fix a target coverage and compare downstream behaviour at that budget—and our random-veto Stage-1 null functions as a non-parametric, distribution-free reference against which any controller must improve. The distinction is operational: conformal prediction controls the coverage of a prediction set over candidate labels, whereas the controller we study authorizes (or declines to authorize) a single commitment that the system emits in its own name. The two devices are formally similar but target orthogonal objects.
AI-systems control, escalation, and commitment authorization. Several recent threads frame deployed AI as needing a system-level control surface beyond the model’s verdict, including multi-agent debate frameworks (Du et al. 2023) where verdict aggregation can converge wrongly under social reinforcement. Jülich (2026) argues that responsible AI in high-stakes settings must satisfy a Brouwerinspired assertibility constraint before publicly committing to a claim, treating commitment as a distinct epistemic act. Wang et al. (2026) formalize multi-agent deliberation as a social-choice problem and apply split conformal prediction to convert debate outputs into calibrated act-versus-escalate decisions, intercepting wrong-consensus cases before automated action. Our work complements these directions by adapting them to the LLM-as-judge setting: we make explicit the separation principle—verdict generation and commitment authorization should be implemented by distinct components—and treat N O -C OMMIT as a routed controller state rather than a new fact-verification verdict. The controller we instantiate is intentionally minimal; we present it as a design probe for the two-channel space, not as the final commitment controller.
3
Background
Selective prediction and the abstention lineage. The idea that a classifier should sometimes refuse to answer goes back to Chow’s rule for character recognition (Chow 1957), which formalized abstention as the optimal action whenever no class posterior crosses a confidence threshold. Modern selective classification inherits this framing: a scoring function ranks inputs by reliability, and a coverage– risk curve traces what is gained by ceding low-confidence items (El-Yaniv and Wiener 2010; Geifman and El-Yaniv 2019). Across this lineage the implicit licence to abstain is uncertainty about which directional answer to give. Cherrypick Override breaks that template: the judge is confident, a directional answer is reachable from the inputs, and the reason to withhold is not in the judge’s posterior at all but in the structure of the evidence—supporting and refuting strands coexist on a load-bearing subclaim. A purely confidencedriven abstention rule cannot see this regime, because nothing in the judge’s own signal flags it. Issuing a verdict versus being entitled to commit it. A second, older distinction is more useful here: producing a judgment and being authorized to make that judgment the system’s public output are not the same act. The separation is familiar in the philosophy of assertion (one can hold a belief without being entitled to assert it) and in accountability framings of automated decision-making, where the component that decides is deliberately distinct from the component that commits on the system’s behalf. The separation principle of §4.5 is the LLM-as-judge operationalization of this older distinction: the typed panel still generates a directional verdict, but a separate controller decides whether that verdict is authorized to leave the system, and N O -C OMMIT names the controller-internal state of declining authorization. C ONFLICTING is not I NSUFFICIENT. Selective prediction collapses every non-answer into one bin labelled “ab-
Condition
In scope?
Schema exposes C ONFLICTING as a valid verdict Evidence has supporting and refuting strands on a loadbearing subclaim Gold label is C ONFLICTING (materially mixed) Schema only has S/R/NEI (no C ONFLICTING) Gold label is S or R (accuracy error, not CCO) Evidence is merely insufficient (calibration, not CCO) Deployment policy waives C ONFLICTING (contract waiver)
yes yes yes no no no no
Table 1: Scope conditions for CCO. The diagnosis is contract-based. stention,” but two epistemically distinct reasons to withhold live inside that bin. I NSUFFICIENT is epistemic abstention: the evidence is absent, so no warrant exists for any directional answer. C ONFLICTING is pragmatic noncommitment: the evidence is sufficient to know that no directional answer can safely be authorized. Atomic-claim decomposition for factuality assessment (Min et al. 2023) treats each atomic claim as a separate truth-bearing unit but does not separate epistemic from pragmatic abstention. The judges studied here distinguish the two operationally on AVeriTeC, returning I NSUFFICIENT on 80% of goldI NSUFFICIENT cases and C ONFLICTING on 77% of goldC ONFLICTING cases (§4.3). The selective-prediction literature has not had a reason to mark this boundary, because its single confidence axis cannot represent it; the task contract introduced in §4.1 is what makes the distinction operational, by exposing C ONFLICTING as a first-class verdict that the controller can authorize.
4 4.1
Problem Definition and Diagnostic Protocol
Task contract and scope
We treat fact verification as a per-claim commitmentauthorization problem. We study settings in which the benchmark’s task schema exposes C ONFLICTING as a firstclass verdict and gold = C ONFLICTING identifies a materially mixed claim under that schema. Under this contract, returning S UPPORTS/R EFUTES on a gold-C ONFLICTING claim is an unauthorized directional commitment; we do not claim that mixed-evidence claims are absolutely undecidable, nor that downstream tasks must always avoid a leaning answer (Table 1 bounds the diagnosis). A typed LLM panel Π = {J1 , J2 , J3 } returns a typed verdict ŷ ∈ Y = {S, R, I, C} and confidences pi ∈ [0, 1]; we aggregate by majority vote with a C tie-break and write P p̄ = 31 i pi . Outputs partition into three disjoint categories: S/R are directional verdicts; C and I are non-directional verdicts; N O -C OMMIT is a controller routing state, not a verification verdict (§4.5).
4.2
Cherry-pick Override
Formally, CCO is the event {pred ∈ {S, R} ∧ gold = C}. It is wrong commitment type, not wrong label.
4.3
What CCO is not
CCO is distinct from neighbouring failures (Table 2): direction error and hallucination concern wrong direction or commit without evidence; calibration concerns overconfidence regardless of evidence structure; generic abstention failure concerns withholding on a one-sided claim. CCO is a wrong commitment type under a schema that exposes C ONFLICTING. C ONFLICTING is not I NSUFFICIENT. The two nondirectional verdicts in Y encode different epistemic states. I NSUFFICIENT is epistemic abstention under absence of evidence; C ONFLICTING is pragmatic non-commitment under known conflict. The distinction is operationally real for our panel: on AVeriTeC the panel returns I NSUFFICIENT on 80% of gold-I NSUFFICIENT cases and C ONFLICTING on only 17% of them; conversely it returns C ONFLICTING on 77% of gold-C ONFLICTING cases and I NSUFFICIENT on only 4% (NI =30, NC =150). Conflating the two would mean treating CCO as a special case of generic abstention failure, which is the framing the literature on selective prediction inherits (Jung, Brahman, and Choi 2025) and which our contract explicitly rejects. Failure
Operational definition
gold ∈ {S,R}, pred ∈ {S,R}, pred ̸= gold pred is directional, not entailed by any evidence in E Calibration error overconfidence on any pred, independent of evidence Abstention failure withholds on a one-sided claim it could commit correctly Cherry-pick gold = C, pred ∈ {S, R} under contract Override Direction error Hallucination
Table 2: CCO vs neighbouring failures. CCO shares neither cause nor cure with the other four.
• RecC — fraction of gold-C ONFLICTING claims predicted C ONFLICTING, with denominator NC . Controllers are compared only at matched coverage; paired differences are reported with a paired bootstrap (5000 resamples) on per-case outcomes, without multiplecomparison correction. To test whether a controller’s veto behavior is structurally targeted rather than merely an equalsized random promotion, we add a random Stage-1 null that matches the controller’s action mechanically: if the controller promotes k of the baseline’s directional commits to C ONFLICTING, the null draws 2000 random subsets of the same size k from the baseline’s directional commits and promotes them to C ONFLICTING as well. We report the controller’s empirical one-sided p-value against the resulting distribution on each metric. A small p here is structural selectivity (the controller picked a non-random subset to promote); the test does not certify magnitude dominance. A non-matched alternative that demotes commits to N O -C OMMIT would leave RecC mechanically unchanged and is therefore not the fair null for this comparison.
4.5
Standard LLM-as-judge pipelines treat the judge’s verdict as the system’s commitment, conflating verdict generation with verdict authorization. Under mixed evidence the two come apart: a strong judge can recognise internal conflict yet still output a direction because the prompt or label vocabulary requires one. The separation principle states that these decisions should be implemented by separate components, with an external lightweight controller mediating between the LLM’s typed proposal and the system’s authorized commitment. N O -C OMMIT is a controller-internal routing state, not a verification verdict: it marks the controller declining to authorize a commitment and leaves the downstream choice (escalate to a human, re-retrieve, defer) to the deploying application.
4.6 4.4
A same-denominator diagnostic protocol
We evaluate commitment authorization with five metrics reported on a single denominator N , so that comparisons across controllers cannot be gamed by shifting the evaluated subset. Let NS/R and NC denote the counts of pure-S/R and gold-C ONFLICTING cases. • Cov (commit coverage) — fraction of N committed to S UPPORTS/R EFUTES. • SE (selective error) — fraction of committed cases that are wrong (subset denominator). • CCON — fraction of N that are gold-C ONFLICTING yet committed directionally; the reviewer-proof rate for cross-controller comparison. • CCOC — fraction of the gold-C ONFLICTING subset (NC ) that the system commits directionally; used to compare judges restricted to the mixed-evidence regime. • AccS/R — directional accuracy on the pure-evidence subset, with denominator NS/R .
The separation principle
A minimal two-channel reference probe
This probe is included only to instantiate the separation principle in a transparent way; it is not an optimised controller and should not be read as a solution to CCO. A certificate validator V , run independently of the panel, decomposes the claim into subclaims {sj } and assigns each an evidence state τj ∈ {supports, refutes, mixed, insufficient} (Schlichtkrull, Guo, and Vlachos 2023). Operationally, a subclaim is material when the validator does not assign it the insufficient state; we write material mixed(c, E) := ∃ j: τj = mixed ∧ sj is material. We treat materiality as an operational output of the validator, not as a ground-truth semantic primitive: the decomposition is imperfect (Cases 5 and 6 in Appendix A), so material mixed functions as a structural veto input rather than a settled judgment about the evidence. Probe E (confidence only). Authorize a directional commitment only when p̄ ≥ τ ; otherwise return N O -C OMMIT. Non-directional proposals pass unchanged.
Algorithm 1 Minimal two-channel authorization probe. Require: typed panel proposal ŷ; per-judge confidences {pi }; validator subclaim states {τj }; threshold τ Ensure: authorized commitment ỹ ∈ Y ∪ {N O -C OMMIT} 1: ỹ ← ŷ 2: if ỹ ∈ {S, R} and material mixed({τj }) then 3: ỹ ← C ONFLICTING // Stage 1: structural veto 4: end if 5: if ỹ ∈ {S, R} and p̄ < τ then 6: 7: return N O -C OMMIT 8: end if 9: return ỹ
Table 3: Residual-failure ladder. Each common fix leaves a distinct residual; the ladder is diagnostic, not competitive. Design implications are unpacked in each rung’s subsection. Rung Added channel †
L0
L1
// Stage 2: confidence gate
L2 L3
Probe F (two-channel: structural veto + confidence). Stage 1 vetoes any directional proposal flagged material mixed (downgrade to C ONFLICTING); Stage 2 applies the confidence threshold as in E (Algorithm 1). The probe has two deterministic stages, one scalar threshold, and no learned components. Lemma 1 (Monotonic veto) For any fixed τ , the directional commits of F are a subset of those of E: {c : F (c; τ ) ∈ {S, R}} ⊆ {c : E(c; τ ) ∈ {S, R}}. Proof sketch. Stage 2 is identical in E and F . F adds Stage 1, which downgrades a strict subset of E’s directional proposals before Stage 2 runs. By Lemma 1, F can never widen the directional commit set at fixed τ , so any improvement on AccS/R or RecC must come from selectivity (Stage 1 blocks the right cases), not from larger coverage. Implementation. Panel: Claude Haiku 4.5 + Claude Sonnet 4.5 + GPT-4o-mini, prompted with identical typedverdict templates. Validator: Claude Haiku 4.5 with a fewshot certificate prompt; we audit prompt sensitivity in §6. Sampling is greedy and deterministic per judge; the validator decomposition is sampled once per claim. Implementation notes and prompts are available at https://github.com/ HrxuAlbert/cherry-pick-override.
5
Intervention Ladder: What Common Fixes Do and Do Not Fix
We organize the design space of common CCO mitigations as an intervention ladder and ask, rung by rung, what each fix recovers and what residual failure remains. The ladder is a design-space map, not a leaderboard. Rungs L0–L4 are described without reference to the two-channel probe, which enters only at L5 to make the separation principle operational. Table 3 summarises the residual-failure pattern; the table cells are the sentences each rung’s subsection unpacks.
5.1
Experimental setup
We evaluate on two fact-verification substrates. AVeriTeC is a stratified sample of N = 285 claims from the AVeriTeC shared task (Schlichtkrull, Guo, and Vlachos 2023), whose
L4 L5
Residual failure
3-option judge
by construction directional on C ONFLICTING gold (>84%); NEI means absence, not conflict typed C ONFLICTING first in-contract rung; structurally avoidable residual remains panel majority vote amplifies CCO on AVeriTeC (CI-separated); aggregation is not universally helpful confidence threshold calibrated for direction (ECE 0.07) but cannot operationally separate CCO from correct commits at tested τ validator-only filter classifier collapses AccS/R (0.78→0.39); veto raises SE over L3 two-channel reference magnitude ∆ over L3 modprobe est (CI straddles 0); residual subclaim-mismatch failures
†
L0 is a pre-contract baseline: a 3-option schema does not expose C ONFLICTING and therefore cannot, strictly, measure CCO under our contract (Table 1). We report L0 to show that without the in-contract vocabulary the failure manifests as a near-mechanical directional rate on gold-C ONFLICTING; the diagnostic ladder begins in-contract at L1.
Conflicting/Cherrypicking class (N = 150) supplies naturally occurring mixed evidence. VitaminC-Mixed is a cross-dataset control of N = 250 claims built from VitaminC contrastive pairs (Schuster, Fisch, and Barzilay 2021), where mixed evidence is synthesized by concatenating supporting and refuting sources. The typed panel is Claude Haiku 4.5, Claude Sonnet 4.5, and GPT-4o-mini; the validator is Claude Haiku 4.5 with the certificate prompt. All metrics share a single denominator (Section 4.4); controllers are compared only at matched coverage; the paired bootstrap uses 5000 resamples and the random-veto control uses 2000 seeds. Table 4 reports the ladder on AVeriTeC; Figure 1 visualises the structural-selectivity property of the L5 controller’s veto under a matched-coverage random-veto control. Cross-dataset replication on VitaminC-Mixed. The full per-system VitaminC-Mixed table mirroring Table 4 is provided in Appendix C (Table 8). Two findings transfer: (a) the L0 panel-amplification effect does not replicate (∆ CCOC = 0.000, 95% CI [−0.060, +0.060]; NC = 100), so we scope the amplification claim to AVeriTeC; (b) the L3→L5 operating-point pattern at τ =0.90 matches the AVeriTeC main operating point (L5 SE = 0.388 vs L3 0.416; CCON = 0.084 vs 0.104) and ties in the extreme low-coverage regime at τ =0.95. Paired-bootstrap CIs. On per-case paired differences (5000 resamples, seed 0): (i) L0 panel amplification
Table 4: Intervention ladder on AVeriTeC mixed-evidence claims (N =285, stratified; NS/R =100, NC =150). Metrics defined in Section 4.4; all share denominators across systems. CCON is the same-denominator rate over N (used for cross-system comparison); CCOC is the conditional rate on the gold-C ONFLICTING subset. Paired-bootstrap CIs for the key differences are reported in the paragraph below the table. System
Cov ↑ SE ↓ CCON ↓ CCOC ↓ AccS/R ↑ RecC ↑
L0 — three-option direct judge (pre-contract; reported for illustration)† Haiku 3-opt 0.800 0.632 (0.442) (0.840) 0.840 Panel 3-opt (majority) 0.846 0.606 (0.467) (0.887) 0.950 L1 — typed four-option judge Panel + typed (majority) 0.396 0.310 0.098 0.187 0.780 L2 — alternative panel aggregation Conflict-if-any panel 0.305 0.230 0.053 0.100 0.670 L3 — confidence-threshold selective E τ =0.90 0.288 0.207 0.056 0.107 0.650 L4 — validator-only filtering Validator-veto 0.295 0.226 0.053 0.100 0.650 L5 — two-channel reference probe F τ =0.85 0.281 0.200 0.046 0.087 0.640 F τ =0.90 0.232 0.152 0.032 0.060 0.560
0.000 0.000 0.773 0.893 0.773 0.860 0.860 0.860
†
Parentheses on L0’s CCON and CCOC entries indicate descriptive reporting only: the 3-option schema does not expose C ONFLICTING as a verdict, so these cells fall outside our contract for CCO measurement (Table 1); they quantify the precondition failure that motivates L1.
on AVeriTeC (NC = 150), ∆ CCOC = +0.047, CI [+0.013, +0.080], separated; (ii) L0→L1 vocabulary fix, ∆ CCOC = −0.653, CI [−0.733, −0.573], separated; (iii) validator-as-classifier collapse (NS/R = 100), ∆ AccS/R = −0.190, CI [−0.280, −0.100], separated; (iv) L5 vs L3 at matched coverage ≈ 0.28, ∆ SE = −0.007, CI [−0.084, +0.066] and ∆ CCON = −0.010, CI [−0.035, +0.011] — both straddle 0. The robust L3→L5 finding is structural selectivity (Figure 1), not aggregate magnitude.
5.2
L0 → L1: the in-contract starting point
L0 is a pre-contract baseline: a three-option judge (S/R/NEI) cannot, by construction, return C ONFLICTING, so any C ONFLICTING gold case it sees becomes a forced directional commitment (rate >0.84 on AVeriTeC). We report L0 not to score it as a CCO controller—our contract scopes CCO to schemas that expose C ONFLICTING (Table 1)—but to motivate L1 as the first in-contract rung of the ladder. The single largest reduction in the ladder occurs at this contract entry point (L1 CCON = 0.098 on AVeriTeC; 0.120 on VitaminC-Mixed). A structurally avoidable residual remains, which the remaining rungs L2–L5 are designed to interrogate.
5.3
L2: panel voting (a scoped negative finding)
Under the three-option schema on AVeriTeC, threejudge majority voting amplifies the direction-onconflict rate (panel 0.887 vs single 0.840, +0.047, CI [+0.013, +0.080]); on VitaminC-Mixed the effect is absent (CI [−0.060, +0.060]). To explain the asymmetry, we inspect per-judge vote distributions on the gold-
C ONFLICTING subsets (Table 9). Under 3-opt roughly two-thirds of conflicting cases have all three judges voting directionally on both datasets (AVeriTeC 71%, VitaminC 66%); amplification is dominated by shared directional bias rather than by the majority overriding minority dissent. Under the 4-opt typed schema—the in-contract regime—we then inspect the panel agreement structure within the N =27 AVeriTeC CCO commits themselves1 and find that the mechanism is split nearly evenly between two failure modes: 52% (14/27) are unanimous on the committed direction (shared bias across judges) and 48% (13/27) are 2-vs-1 with the dissenting judge voting non-directionally (majority suppression of a single judge who flagged conflict). Almost half of all CCO is therefore an aggregation artifact in which a single judge produced the safe verdict and was outvoted—evidence that typed-vocabulary alone is insufficient when paired with majority aggregation. An alternative aggregation rule—commit only when no judge votes C ONFLICTING—raises RecC to 0.893 and lowers CCON to 0.053, at the cost of AccS/R (0.670 vs typed direct 0.780) and a higher SE than L3 (Table 4). Aggregation choices expose different safety–coverage trade-offs rather than producing a single dominant intervention.
5.4
L3: confidence-threshold selection
Committing only when p̄ ≥ τ traces a clean risk–coverage curve (E τ =0.90: Cov 0.288, SE 0.207, CCON 0.056). The relevant question for our diagnosis is not whether the curve exists but whether it can disentangle CCO from correct directional commits. It cannot, in any operationally useful sense. The panel is in fact well-calibrated for directional prediction on the pure-S/R subset (ECE = 0.07 on NS/R =100), so CCO is not the residual of pure-S/R miscalibration. CCO commits (n=28) have mean p̄=0.902 versus 0.933 for correct directional commits (n=78 on NS/R =100); the two confidence distributions are statistically distinguishable (Mann–Whitney two-sided p ≈ 0.0002) but the mean separation is 0.031, far below what a usable threshold needs: 86% of CCO commits sit at p̄ ≥ 0.85, 57% at p̄ ≥ 0.90, and 7% (2/28) at p̄ ≥ 0.95. No threshold within the tested range filters CCO without also filtering correct directional commits, because confidence measures decision strength rather than evidence structure. The face-mask example of Section 1 (panel R EFUTES at p̄=0.96) is exactly this high-confidence pass-through.
5.5
L4: validator-only filtering
If the gap left by L3 is insensitivity to structural conflict, the natural fix is a deterministic evidence-state validator. As a primary classifier it collapses pure-S/R accuracy from 0.78 to 0.39 on AVeriTeC: the deterministic rule cannot weight subclaim importance against the main directional question. As a veto channel on top of typed proposals it preserves AccS/R (0.650) and reaches CCON 0.053, but its SE 1 One of the 28 CCO commits used in the L3 confidence analysis below had a per-judge vote count ̸= 3 (a parse failure on one judge) and is omitted here, since the 3-0/2-1 anatomy requires a complete panel.
Random Stage-1 promotion (2000 seeds; AVeriTeC k=16; VitaminC k=10)
VitaminC-Mixed (N = 250) AVeriTeC (N = 285)
F at 45th pctl ( )
F0.200
F at 67th pctl ( )
E0.207
0.125 0.150 0.175 0.200 0.225 0.250
F at 6th pctl ( )
F0.388
E0.416
0.36 0.38 0.40 0.42 0.44 0.46 SE (selective error)
F = L5 two-channel probe
[5, 95] percentile band
MAGNITUDE (F near random median; expected)
F at 0th pctl ( )
F0.046
F0.640
E0.056
0.03
0.04
0.05
0.06
F at 4th pctl ( )
F0.084
E = L3 (confidence-only)
SELECTIVITY (F at distribution extreme; structural)
0.50
0.55
0.60
F at 0th pctl ( )
0.65
F at 6th pctl ( )
F0.630
E0.104
0.08 0.09 0.10 CCON (same-denom CCO rate)
F0.860
E0.650 E0.773
0.78
0.80
0.82
0.84
0.86
F at 0th pctl ( ) E0.660 E0.700
0.56 0.58 0.60 0.62 0.64 0.66 AccS/R (pure-S/R accuracy)
0.70
F0.770
0.72 0.74 0.76 RecC (conflict recall)
Figure 1: Apples-to-apples random Stage-1 control on both datasets. The controller’s Stage-1 promotes k of L3’s directional commits to C ONFLICTING (k=16 on AVeriTeC, k=10 on VitaminC-Mixed); the null draws 2000 random subsets of the same size from L3’s commits and promotes them to C ONFLICTING as well. (An earlier control that demoted commits to N O -C OMMIT cannot change RecC mechanically; this fair control removes that limitation.) Each panel shows the KDE of the null distribution with the [5, 95] percentile band shaded. Columns 1–2 (magnitude axis): on SE and CCON , F sits near the random median on AVeriTeC (p=0.45 and p=0.67) and at p ∈ [0.04, 0.07] on VitaminC; magnitude is not cleanly separated, consistent with the bootstrap CIs in Table 4. Columns 3–4 (direction-preservation axis): on AccS/R and RecC , no random seed matches or exceeds F on AVeriTeC (empirical p < 1/2001); on VitaminC RecC retains p < 1/2001 and AccS/R is p=0.06. These two axes are algebraically coupled under fixed k—each promoted case either decrements AccS/R or increments RecC —so the joint extremity is one selection event seen from two angles rather than two independent tests. The control distinguishes “targeted promotion” from “random promotion”; it does not certify magnitude dominance or claim a solution. (0.226) is higher than L3’s (0.207). The validator catches structural conflicts L3 misses while also firing on cases the judge settles correctly.
5.6
L5: two-channel external authorization
L3 and L4 leave orthogonal residuals—one cannot see structural conflict, the other is too brittle alone—so the top rung uses each channel for the role it is good at: the validator as a structural-evidence veto, confidence as a decisionstrength gate. At matched coverage ≈ 0.28 on AVeriTeC, F τ =0.85 reaches SE 0.200 and CCON 0.046 vs L3’s 0.207 and 0.056; magnitude improvements are modest and their bootstrap CIs straddle zero (above). The robust finding is structural selectivity: under the apples-to-apples random Stage-1 null, no random sample matches L5 on AccS/R (empirical p < 1/2001). Because each of the k promotions either decrements AccS/R (if the promoted case has gold S/R) or increments RecC (if the gold is C ONFLICTING and the panel committed directionally), the two metrics are algebraically coupled under fixed k, and we therefore treat the joint result on both axes as a single selection event expressed in two ways rather than as two independent tests. The veto selects which cases to block in a structurally non-arbitrary way; the null does not certify magnitude dominance, only non-arbitrariness. By Lemma 1, F is at-least-as-restrictive as E at any fixed τ , so selectivity— not extra coverage—is the only available source of improvement. The same operating-point pattern replicates on
VitaminC-Mixed at τ =0.90 (L5 SE 0.388, CCON 0.084 vs L3’s 0.416, 0.104), and the random-veto null again gives empirical p < 1/2001; at τ =0.95 L5 and L3 are tied at nearzero CCON (0.024). Taken together, no single channel L0– L4 simultaneously preserves pure-S/R accuracy and conflict recall at the matched operating points; L5 is the minimal honest top-rung instance, exhibiting structural selectivity but not magnitude dominance.
6
Channel-Orthogonality Tests
Each test below asks a channel-orthogonality question— does adding channel X recover something channel Y cannot, or does either alone suffice?—rather than defending a controller design. Table 5 reports the matched-coverage numbers on AVeriTeC. Can structural evidence replace the verdict? No. Used as a primary classifier, the validator collapses AccS/R from 0.78 to 0.39 (Table 5). A strict evidence-state rule cannot substitute for verdict generation; structural evidence is useful only as a veto channel on top of typed proposals. Does either single channel reach the two-channel point? No. Stage 2 alone (confidence-only) reaches SE 0.207 at Cov 0.288; Stage 1 alone (structural veto on the typed proposal) reaches CCON 0.053 but SE 0.226. Neither subsumes the other: on the N =28 CCO commits the validator’s material mixed flag fires on 46% (13/28) of cases the confidence gate passes, while the confidence
Table 5: Channel-orthogonality diagnostic on AVeriTeC (N =285), at comparable coverage. This is a diagnostic of channel complementarity, not a method ablation proving superiority. Controller
Cov
Two-channel probe (F τ =0.85) − confidence gate (Stage 1 only) − structural veto (Stage 2 only) Validator-as-classifier Panel-margin rule (tuned) Calibrated (degenerate)
0.281 0.295 0.288 0.193 0.281 0.413
SE CCON 0.200 0.226 0.207 0.291 0.250 0.305
0.046 0.053 0.056 0.053 0.053 0.105
AccS/R 0.640 0.650 0.650 0.390 0.600 0.820
gate—calibrated for direction prediction (ECE = 0.07 on pure-S/R)—catches a complementary subset of hesitant-butcommitted cases. The validator’s coverage of CCO is nontrivial but not complete; its false-alarm rate on pure-S/R is 21% (21/100), which bounds the directional-accuracy cost of using it as a veto channel. The two channels carry orthogonal signal at incomplete strength on each axis. Is the probe’s Stage-1 structurally targeted, or matched by random promotion? We use the apples-to-apples random Stage-1 null (Section 4.4): the probe promotes k of the confidence baseline’s directional commits to C ONFLICTING; the null draws 2000 random size-k subsets and promotes them to C ONFLICTING as well. On AVeriTeC, no random subset matches the probe on AccS/R (empirical p < 1/2001); the RecC axis reaches the same p, but the two are algebraically coupled under fixed k (each promotion either lowers AccS/R or raises RecC ), so we report the joint outcome as one selection event expressed on two axes rather than as two independent tests. The probe sits near the random median on SE and CCON (Figure 1). On VitaminC, the result is weaker: RecC remains at p < 1/2001, but AccS/R , SE, and CCON sit at p between 0.04 and 0.07—the probe’s promotions are better than random but not extreme. By Lemma 1, the probe cannot widen the directional commit set at fixed τ , so any gain comes from which commits it promotes, not from how many. The null tests targeted selection; it does not certify magnitude dominance. Do richer single-channel rules help? No. A votedistribution (panel-margin) rule does not improve on the confidence threshold at matched coverage (SE 0.250 vs 0.207): typed panel agreement carries no signal beyond mean confidence on this substrate. Calibrating the controller against a CCO target is degenerate at our sample sizes— every target collapses to a single configuration near the typed-direct baseline—so we report it as a negative result and leave calibration over richer parameter spaces to future work. Is the veto a few-shot prompt artifact? No. Under a strict zero-shot variant of the validator prompt, the material mixed flag agrees with the few-shot version on 86% of AVeriTeC claims, and re-running the two-channel probe with the strict variant produces qualitatively the same operating points (SE 0.216, CCON 0.049 at τ =0.85, versus 0.200 and 0.046); the validator-as-classifier collapse persists
under both prompts. The veto signal is not purely a few-shot artifact. Summary. The diagnostic tests above characterise the channel-orthogonality structure of the design space: structural evidence carries information confidence cannot recover and vice versa; structural evidence cannot replace verdict generation; richer single-channel aggregation does not beat a simple confidence threshold; and the random-veto control shows the two-channel probe’s veto is structurally targeted, not arbitrary. Aggregate magnitude ∆s over the strongest single channel are modest and matched-coverage bootstrap CIs straddle zero; the robust cross-method finding is channel orthogonality, not magnitude dominance.
7 7.1
Scope Conditions and Limitations
Contract boundary
CCO applies when the task schema exposes C ONFLICTING as a valid verdict. We do not claim that all mixedevidence claims are absolutely undecidable, nor that downstream tasks must always avoid a leaning answer; we study the contract above and call its specific failure mode CCO (Table 1). Deployments whose contract explicitly waives the C ONFLICTING option—e.g. forced-choice summaries where a leaning answer is the documented expectation—fall outside the diagnosis. Gold-label dependency; CCO is contract-relative. CCO is defined relative to the benchmark’s task contract: gold = C ONFLICTING marks the schema-authorized non-directional verdict, not an independently validated semantic state of the evidence. We do not separately reannotate either gold standard, and the diagnosis is therefore a contract-relative measurement rather than a claim about the underlying epistemic situation of each individual claim. AVeriTeC gold is expert annotation (Schlichtkrull, Guo, and Vlachos 2023); VitaminC-Mixed gold is constructive, derived by concatenating a supporting and a refuting sentence from each VitaminC contrastive pair (Schuster, Fisch, and Barzilay 2021). The cross-dataset pattern we report therefore holds under two different annotation regimes— peer-reviewed expert labels and benchmark construction— but does not validate either gold standard independently; the qualitative audit (Appendix A) and the N =10 singleannotator blind audit (label distribution: 5 S UPPORT, 2 C ONFLICT, 2 I NSUFFICIENT, 1 R EFUTE; broadly consistent with the panel’s 4-way distribution on the same draws but not powered for an agreement statistic) are illustrative, not re-annotation efforts.
7.2
Evidence and dataset boundary
We evaluate text-based fact verification only, on two substrates: AVeriTeC (stratified N =285) and VitaminC-Mixed (N =250). VitaminC-Mixed synthesizes mixed evidence from contrastive pairs by concatenation; AVeriTeC contains naturally occurring conflicting cases. Broader replication (other fact-verification datasets, retrieval-augmented pipelines, multimodal evidence) is future work. The N=10 qualitative audit (Appendix A) is illustrative, not statistical; we report no rate from it. We do not extend to general
agentic-AI settings or multi-agent deliberation benchmarks where commitment timing is the metric.
7.3
Method boundary
The two-channel probe is a deterministic reference probe, not a learned controller; replacing the deterministic Stage 1 with a calibrator that weights subclaim importance against the main directional question is the explicit subject of follow-up work. We make no formal coverage or risk bound: the structural-selectivity claim is empirical (a random-veto null), not theoretical; distribution-free calibration of the authorization rate is future work. Our panel is restricted to three closed-source model families (Claude Haiku 4.5, Claude Sonnet 4.5, GPT-4o-mini); open-source-only panels, smaller models, and multimodal judges are not tested, and panel-amplification on AVeriTeC may interact with this choice. Matched-coverage bootstrap CIs on the magnitude ∆s over confidence-only selection straddle zero at our sample sizes; the structural-selectivity finding under the applesto-apples random Stage-1 null is the robust cross-method claim, and Appendix A cases 5 and 6 show the honest cost of a deterministic veto. Validator reliability and the price of typed vocabulary. The structural validator is itself imperfect: it fires material mixed on 46% of the AVeriTeC CCO subset (the ceiling on what a Stage-1 veto built solely on this signal can recover) and produces a 21% false-alarm rate on the pure-S/R subset (the cost that any structural-veto controller pays in directional accuracy). Independent of the validator, the typed-vocabulary schema itself carries a cost: on the pure-S/R subset the panel returns C ONFLICTING on 17% of cases (17/100), so adopting CONFLICTING as a verdict is not a free intervention. These imperfections motivate the framing of the probe as a reference instantiation: a learned, evidence-aware controller has room to dominate it on both axes. Honest acknowledgement of finding strength. On the magnitude axis we have nothing defensible to claim: the controller’s matched-coverage ∆s vs the confidence baseline are not separated from zero at our sample sizes. What we have is a selectivity finding: the controller’s Stage-1 promotes a non-random subset of E’s directional commits to C ONFLICTING (strong on AVeriTeC, p < 1/2001 on both direction-preservation metrics; weaker but in the same direction on VitaminC-Mixed, p in the 0.04–0.07 range on AccS/R / SE / CCON , p < 1/2001 on RecC ). The paper’s empirical core is therefore: CCO persists, single-channel fixes leave distinct residuals, and the structural-evidence and confidence channels are non-substitutable. This is a diagnostic contribution, not a method contribution.
7.4
Design implications: arguments, not demonstrations
The implications below are argued from the ladder analysis; we do not demonstrate that adopting them produces measurable system-level benefits, and end-to-end validation in a deployed pipeline (e.g. with explicit escalation pathways and downstream cost measurement) is itself follow-up work.
The separation principle. We argue that verdict generation and commitment authorization may benefit from architectural separation: an external controller authorizes or withholds the commitment based on structural-evidence and confidence signals, and the system has an explicit place to route withheld cases. In practice this suggests a pipeline can keep the judge it already has and add a thin authorization surface in front of the recorded commitment, rather than trying to make the judge itself never over-commit. N O C OMMIT as a routed state. N O -C OMMIT is a routed controller state, not a verification verdict: a routed claim may still receive a directional verdict once escalated with more evidence or sent to a human. Orthogonal channels. A confidence threshold does not distinguish a structurally mixed claim the judge settles decisively, and a deterministic structural veto does not catch a confidently wrong commit it does not flag; we suggest a commitment-control layer accept both, and that a learned layer weight subclaim importance against the main directional question—the failure mode the subclaim-mismatch cases expose. When the trade is rational. External commitment-control layers are most rational where unsafe directional commitment is costlier than escalation; under such deployment profiles the same measurements convert into a cost-sensitive operating-point analysis, reported as a supplementary view in Appendix B rather than as a primary claim.
8
Conclusion
In LLM-judge-based AI systems, the question “what is the verdict?” is not the same as the question “is the system authorized to commit this verdict?” Confusing them produces Cherry-pick Override: a directional verdict committed on mixed evidence when the non-directional verdict C ONFLICTING was the authorized outcome. Walking a same-denominator intervention ladder over common singlechannel fixes, we find that each fix leaves a distinct residual failure—typed vocabulary leaves a structurally avoidable residual, panel voting amplifies CCO on one dataset and is dominated by aggregation-suppressed dissent on the other, confidence is well-calibrated for direction but cannot operationally separate CCO from correct directional commits, and a deterministic validator cannot replace verdict generation. The core lesson is that improving the judge does not answer the authorization question. We argue that mixed-evidence verification may benefit from an explicit commitment-control layer that decides whether a directional verdict is safe to record, route, or withhold; demonstrating end-to-end deployment benefit, designing learned, evidence-aware selective commitment controllers, and integrating routed-state cases with escalation pathways are all explicitly left to follow-up work.
References Angelopoulos, A. N.; and Bates, S. 2021. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511. Bartlett, P. L.; and Wegkamp, M. H. 2008. Classification
with a Reject Option using a Hinge Loss. Journal of Machine Learning Research, 9: 1823–1840. Chow, C. K. 1957. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, EC-6(4): 247–254. Cortes, C.; DeSalvo, G.; and Mohri, M. 2016. Learning with Rejection. In Proceedings of the 27th International Conference on Algorithmic Learning Theory (ALT). Du, Y.; Li, S.; Torralba, A.; Tenenbaum, J. B.; and Mordatch, I. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv preprint arXiv:2305.14325. El-Yaniv, R.; and Wiener, Y. 2010. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11: 1605–1641. Geifman, Y.; and El-Yaniv, R. 2019. SelectiveNet: A deep neural network with an integrated reject option. In International Conference on Machine Learning (ICML). Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML). Hagar, N.; Agustianto, W.; and Diakopoulos, N. 2025. Not Wrong, But Untrue: LLM Overconfidence in DocumentBased Queries. arXiv preprint arXiv:2509.25498. Ji, Z.; Lee, N.; Frieske, R.; Yu, T.; Su, D.; Xu, Y.; Ishii, E.; Bang, Y. J.; Madotto, A.; and Fung, P. 2023. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12). Jülich, M. 2026. Upholding Epistemic Agency: A Brouwerian Assertibility Constraint for Responsible AI. arXiv preprint arXiv:2603.03971. Jung, J.; Brahman, F.; and Choi, Y. 2025. Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement. In Proceedings of the International Conference on Learning Representations (ICLR). ArXiv:2407.18370. Lifshitz, S.; McIlraith, S. A.; and Du, Y. 2025. Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers. arXiv preprint arXiv:2502.20379. Liu, Y.; Iter, D.; Xu, Y.; Wang, S.; Xu, R.; and Zhu, C. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). ArXiv:2303.16634. Min, S.; Krishna, K.; Lyu, X.; Lewis, M.; tau Yih, W.; Koh, P. W.; Iyyer, M.; Zettlemoyer, L.; and Hajishirzi, H. 2023. FactScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). Schlichtkrull, M.; Guo, Z.; and Vlachos, A. 2023. AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). ArXiv:2305.13117.
Schuster, T.; Fisch, A.; and Barzilay, R. 2021. Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). ArXiv:2103.08541. Thorne, J.; Vlachos, A.; Christodoulopoulos, C.; and Mittal, A. 2018. FEVER: A Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). Verga, P.; Hofstätter, S.; Althammer, S.; Su, Y.; Piktus, A.; Arkhangorodsky, A.; Xu, M.; White, N.; and Lewis, P. 2024. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv preprint arXiv:2404.18796. Vovk, V.; Gammerman, A.; and Shafer, G. 2005. Algorithmic Learning in a Random World. Springer. Wang, M. F.; Xie, H.; Wang, G.; Gao, A.; Yang, G.; Li, Z.; Qiu, Q. W.; Han, F.; Qiu, H.; Huang, Y.; Zhu, B.; and Woo, J. O. 2026. From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation. arXiv preprint arXiv:2604.07667. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E. H.; Narang, S.; Chowdhery, A.; and Zhou, D. 2023. SelfConsistency Improves Chain of Thought Reasoning in Language Models. In Proceedings of the International Conference on Learning Representations (ICLR). Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-aJudge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. ArXiv:2306.05685.
A
Qualitative Case Studies
We provide six representative cases to illustrate the failure modes the controller catches and the failure modes it does not. The cases are illustrative, not exhaustive; no statistical rate is derived from the N = 10 qualitative case study. Case 1 — clean Cherry-pick Override. For the claim that US chain migration lets a person bring dozens of relatives, the cited evidence both refutes the direct reading (one cannot directly petition an aunt, uncle, or cousin) and supports an indirect multi-step chain (a naturalized citizen can petition a parent, who can later petition siblings). The panel issues R EFUTES at mean confidence 0.91; gold is C ONFLICTING. This is a clean CCO in the sense of Section 4: the validator’s subclaim decomposition flags material mixed, and the two-channel controller downgrades to C ONFLICTING regardless of confidence. Case 2 — panel amplification on AVeriTeC. For the claim “Waving the British flag will result in arrest for breach of the peace” (AVeriTeC case 148), the cited evidence has two strands: a general rule (you cannot be arrested for waving the British flag) and a Northern-Irelandspecific qualifier (display of flags including the Union flag
has been politically and legally controversial, with the Police Service of Northern Ireland treating loyalist flag erection in mixed-population areas as a public-order matter). One typed judge votes C ONFLICTING, recognising the qualifier; the other two judges default to R EFUTES on the general rule. Majority voting suppresses the dissent and the panel commits R EFUTES at mean confidence 0.89; gold is C ON FLICTING . The case exemplifies the population-level finding that panel CCOC = 0.887 exceeds single-judge 0.840 on AVeriTeC (paired-bootstrap CI [+0.013, +0.080], separated from zero); the effect does not replicate on VitaminC-Mixed under the same configuration (CI [−0.060, +0.060], straddles zero). Case 3 — high-confidence CCO. For “face masks cause hypoxia,” evidence refutes the claim for healthy individuals while a separate strand notes prolonged N-95 use in patients with preexisting lung disease could raise CO2 . The panel issues R EFUTES at mean confidence 0.96; no confidence threshold within the explored range (τ ≤ 0.95) blocks the commit. The validator flags the qualifier subclaim as material mixed, and the controller downgrades to C ONFLICTING. The case is the canonical illustration that confidence and structural evidence carry orthogonal information. Case 4 — good validator veto. For “Shazad Latif is of Pakistani descent,” two near-identical evidence sentences describe the subject as of mixed “South Asian” versus mixed “Pakistani” descent. The panel issues S UPPORTS at confidence 0.87, which the confidence channel alone would authorize; the validator flags material mixed, and the controller downgrades to C ONFLICTING. The auditor’s independent reading agrees that a directional verdict would be misleading. The structural-evidence channel adds signal the confidence channel cannot recover. Case 5 — false validator veto. For the claim that Justice Ruth Bader Ginsburg was known for a grueling fitness regime, the evidence is overwhelmingly supportive (a longterm trainer, twice-weekly sessions, push-ups). The panel issues S UPPORTS at confidence 0.90, in agreement with the auditor’s reading, and the confidence channel would have committed correctly. The validator nonetheless flags a peripheral subclaim as material mixed, and the controller over-blocks to C ONFLICTING. This is the honest cost of a deterministic veto: it cannot weight subclaim importance against the main directional question. Case 6 — subclaim mismatch. For “for a cumulative 29 of Nigeria’s 60 years . . . under military rule,” the evidence cleanly supports the arithmetic (13+16 = 29). The validator tags one year-range subclaim as material mixed, likely because the evidence does not state the cumulative total explicitly, and the controller over-blocks despite the directional verdict being arithmetically correct. Here the validator’s decomposition is upstream of its veto rule—a different failure from Case 5. Cases 5 and 6 (the validator’s failure modes) motivate the follow-up emphasis on learned evidence-aware controllers and richer structured-evidence parsers.
Table 6: Three named cost profiles. Profile A treats CCO as 5× costlier than N O -C OMMIT; Profile C treats it as 12×. Profile
correct-S/R correct-Conf wrong-S/R CCO false-Conf No-Commit
A (accuracy) B (balanced) C (safety-critical)
+1 +1 +1
+1 +1 +1
−3 −5 −8
−5 −8 −12
−2 −2 −3
−1 −1 −1
Table 7: Per-case utility (↑ better) on AVeriTeC (N = 285) and VitaminC-Mixed (N = 250) for the three profiles defined in Table 6. Dataset
System
A
B
C
AVeriTeC typed-direct (L1) −0.158 −0.502 −1.046 validator-only (L4) −0.011 −0.197 −0.583 E τ =0.85 (L3) −0.109 −0.404 −0.881 E τ =0.90 (L3) −0.039 −0.214 −0.526 F τ =0.85 (L5) +0.018 −0.140 −0.488 F τ =0.90 (L5) +0.032 −0.070 −0.340 VitaminC typed-direct (L1) −0.596 −1.196 −2.064 validator-only (L4) −0.460 −0.944 −1.684 E τ =0.85 (L3) −0.572 −1.132 −1.940 E τ =0.90 (L3) −0.524 −1.004 −1.700 F τ =0.85 (L5) −0.436 −0.888 −1.580 F τ =0.90 (L5) −0.428 −0.832 −1.456
B
Cost-Sensitive Operating-Point Analysis
We convert the same-denominator outcome counts in Table 4 into per-case expected utility under three named deployment profiles. The point of the appendix is not to argue that the controller is preferred in general; it is to make the deployment trade-off explicit so a reader can see when the authorization layer is rational and when it is not. Per-case utility. Let an outcome category o ∈ O count one of: CORRECT-S/R (correct directional commit on pureS/R gold), CORRECT-C ONF (correct C ONFLICTING commit on gold-C ONFLICTING), WRONG -S/R (wrong directional commit on pure-S/R gold), CCO (directional commit on gold-C ONFLICTING), FALSE -C ONF (C ONFLICTING commit on pure-S/R gold), N O -C OMMIT. A profile assigns each category a cost wo ; the per-case utility is the average wo over all N cases. Per-system results. Table 7 reports per-case utility for each controller under each profile on AVeriTeC (N = 285) and VitaminC-Mixed (N = 250). F τ =0.90 is the percase utility maximiser on every (dataset × profile) cell at these profile settings. The mechanism is structural rather than driven by the CCO penalty magnitude: Stage 1 converts CCO events into CORRECT-C ONF credit (+1) rather than 0; at cost ratios where WRONG -S/R and CCO are equal, typed direct (L1) becomes competitive on AVeriTeC because it commits more often. Sensitivity to the asymmetry. The asymmetry between WRONG -S/R and CCO costs is the load-bearing assumption of all three profiles in Table 6: a deployment that values the two equally (wWRONG -S/R = wCCO ) collapses the L5 advantage on AVeriTeC, because L1’s higher commit rate
Table 8: Cross-dataset replication on VitaminC-Mixed (N =250). Metric definitions match Table 4. L0 single-3opt and panel-3-opt are reported only on the conflicting subset for CCOC and RecC ; the L0 panel and single rows tie (CCOC = 0.77), showing the AVeriTeC panel-amplification effect does not transfer here. Cov
System
SE CCON CCOC AccS/R RecC
L0 — three-option direct Single 3-opt — — Panel 3-opt — —
— —
0.770 0.770
— 0.000 — 0.000
L1 — typed panel direct (A) Panel + typed 0.536 0.448
0.120
0.300
0.740 0.700
L3 — confidence-only (E) τ =0.85 0.508 0.433 τ =0.90 0.452 0.416 τ =0.95 0.208 0.212
0.120 0.104 0.024
0.300 0.260 0.060
0.720 0.700 0.660 0.700 0.410 0.700
L4 — validator-veto only (D) conflict only 0.476 0.412
0.092
0.230
0.700 0.770
L5 — two-channel reference probe (F) τ =0.85 0.456 0.395 0.092 τ =0.90 0.412 0.388 0.084 τ =0.95 0.204 0.216 0.024
0.230 0.210 0.060
0.690 0.770 0.630 0.770 0.400 0.770
compensates for its higher CCO. The cost-sensitive analysis is therefore deployment-conditional: it identifies a regime in which an authorization layer is rational, not a universal recommendation.
C
Cross-Dataset Replication: VitaminC-Mixed
We mirror Table 4’s ladder layout on the VitaminC-Mixed substrate (N = 250; NS/R = 100, NC = 100). The VitaminC-Mixed pool is built by concatenating the supporting and refuting sentence in a VitaminC contrastive pair to synthesise an evidence set that is mixed by construction. Mixed-evidence claims here therefore have a different distributional character than AVeriTeC’s natural Conflicting/Cherrypicking subset, and this appendix isolates which findings transfer. What transfers. (i) The vocabulary-fix step (L0→L1) reduces conditional CCO from 0.770 to 0.300 — same direction as on AVeriTeC, though residual remains. (ii) Across L1→L3→L5 the operating-point pattern at τ =0.90 matches AVeriTeC: L5 reduces SE and CCON over L3 at comparable coverage. (iii) The random-veto control (Figure 1 on AVeriTeC) re-run on VitaminC again places L5 at empirical p < 1/2001 on AccS/R and RecC at matched coverage (see Section 6). What does not transfer. (i) Panel amplification at L0 (AVeriTeC: +0.047, CI [+0.013, +0.080]) does not replicate on VitaminC (0.000, CI [−0.060, +0.060]). The synthetic construction of VitaminC-Mixed concatenates a single supporting and a single refuting sentence, and the typed panel agrees unanimously on directional commitment at this
level of evidence homogeneity. (ii) At the extreme lowcoverage regime (τ =0.95), L3 and L5 tie on every metric; both reach CCON =0.024 at Cov≈0.20, where confidence alone already filters the remaining mixed-evidence commits. Panel amplification anatomy. Table 9 reports the perjudge vote distribution on the gold-C ONFLICTING subsets of both datasets, which Section 5.3 cites inline. Under the 3opt schema, roughly two-thirds of conflicting cases have all three judges voting directionally on both datasets, so amplification is dominated by shared directional bias rather than majority overriding minority dissent. Under the 4opt typed schema, VitaminC actually has more suppresseddissent cases (22% vs 9%) but also far fewer unanimousdirectional cases, so panel CCO does not amplify. Table 9: Panel amplification anatomy on the goldC ONFLICTING subset of each dataset. “Unanimous directional” is a 3/0 split (all three judges vote S UPPORTS or R EFUTES); “dissent suppressed” is 2 directional + 1 nondirectional, where the panel majority commits despite a minority abstention. Vote pattern (gold = C ONFLICTING) AVeriTeC VitaminC 3-opt panel (NC =150/100) unanimous directional (3/0) dissent suppressed (2 dir + 1 NEI) 4-opt typed panel unanimous directional (3/0) C ONFLICTING dissent suppressed no directional commit (safe)
71% 18%
66% 11%
9% 9% 56%
13% 22% 41%