Alignment by Confident Debate: Preference Learning from Multi-Agent Debate

Published as a preprint.

A LIGNMENT BY C ONFIDENT D EBATE : P REFERENCE L EARNING FROM M ULTI -AGENT D EBATE Nikhil Verma, Dingxian Wang, Mohammad Reza Rezaei, Andrew Rabinovich & Zhao Chen Upwork Inc. Email: nikhilverma@upwork.com

arXiv:2611.09568v1 [cs.AI] 6 Nov 2026

A BSTRACT Post-training language models for reasoning typically rely on gold answers, taskspecific verifiers, or learned reward models, limiting scalability when such signals are scarce. Multi-agent debate offers a self-supervised alternative, but converting debate into reliable training supervision remains challenging: majority voting weights agreeing responses equally, lacks a principled tie-breaking rule, and is illdefined for open-ended tasks without a canonical answer; likelihood-based scoring can conflate fluency with correctness; and homogeneous committees may reinforce correlated errors. We introduce Alignment by Confident Debate (AbCD), which converts debate traces into preference data without gold answer labels. A samplingdiverse committee debates each problem, after which final answers are clustered by task-level semantic equivalence. Each cluster is scored by aggregating confidence on its claimed answer, and completions from the highest-scoring cluster are labeled as preferred over the remaining completions. We use verifiable tasks (mathematical reasoning, multiple-choice quantitative QA, and executable code-generation) as a controlled test bed: their answer keys or executable checkers let us evaluate the quality of debate-derived supervision against ground truth, even though the oracle itself never uses gold labels. Post-training on the resulting preferences outperforms inference-only ensembling and debate, as well as SFT on the same traces, across five model families spanning 1B to 8B parameters and four datasets, improving consensus accuracy, with a maximum gain of 21.5pp . Ablations show that the approach transfers to a larger model and to code generation, while also revealing limitations of heterogeneous committees and agreement-based weighting. On tasks with a discrete answer, majority voting is the standard ensemble and lands close to confidence ensembling, because both aggregate the same extracted strings; that comparison is reported in the appendix. Overall, the results suggest that debate traces can yield useful preference supervision when answer equivalence is combined with confidence-weighted disagreement.

1

I NTRODUCTION

Post-training for reasoning typically relies on external supervision to identify desirable solutions or solution traces. Such supervision can come from labeled demonstrations or gold answers, taskspecific verifiers that evaluate final outcomes, or learned evaluators such as reward models and LLM-based judges. Gold answers and executable verifiers provide direct correctness signals when an answer key or reliable checking procedure exists, as in numerical reasoning and unit-tested code (Cobbe et al., 2021; Yuan et al., 2023; Austin et al., 2021). Reward models and LLM-based judges extend evaluation to less readily verifiable outputs (Ouyang et al., 2022; Zheng et al., 2023; Yuan et al., 2024; Wu et al., 2025). However, these evaluators must themselves be trained, calibrated, or otherwise trusted and may share systematic errors with the policy being optimized. Thus, although post-training objectives are well established, obtaining reliable correctness supervision remains difficult and domain-dependent. When external correctness supervision is unavailable, agreement among the model’s own samples provides a natural candidate signal. Multi-agent debate allows several agents to exchange reasoning over multiple rounds, and can improve answer accuracy relative to a single sampled solution (Du et al., 2023; Liang et al., 2024; Madaan et al., 2023). Subsequent debate-training methods often use 1

Published as a preprint.

these traces for training, for example by labeling the majority answer as preferred and the remaining answers as dispreferred (Samanta et al., 2025; Zhou et al., 2025). This strategy is fragile, however. In homogeneous committees, communication need not improve correctness beyond independent sampling (Zhu et al., 2026); repeated interaction can also induce correlated reasoning and convergence on shared errors (Dai, 2026). On GSM8K, for example, a five-agent committee achieves its largest consensus accuracy after the first exchange, whereas additional rounds increase agreement without improving—and in some cases reducing—correctness (Figure 1). Consensus is therefore an outcome of the interaction and selection process, not by itself evidence that the selected answer is correct. We therefore study how to convert debate traces into preference supervision by separating answer equivalence from answer selection: final answers are clustered by task-level semantic equivalence and the resulting clusters are ranked by aggregate confidence in their claimed answer spans. Majority voting requires a procedure for mapping final answers to equivalent outcomes. Without a canonical answer or reliable semantic-equivalence rule, voting cannot determine when two responses should count as equivalent, particularly for open-ended generation where correct responses may differ in wording (Chen et al., 2023). Even when equivalence is available, majority voting treats all agreeing traces equally and provides no additional signal for resolving ties. In small committees of cloned models, agreement may also reflect correlated errors rather than independent corroboration. Token probabilities provide another signal available during generation, but they measure the likelihood of generated tokens rather than correctness of the full reasoning chain. Length-normalized Figure 1: Committee behavior for Qwenfull-trace likelihood can therefore favor fluent or familiar 2.5-1.5B-it on GSM8K. The committee size reasoning without reliably identifying the correct answer. varies from one to five agents under indeConfidence computed on answer-relevant tokens is a more pendent sampling, after which the number targeted alternative and has been used to improve answer of debate rounds is varied with five agents. selection (Razghandi et al., 2025; Lin & Hooi, 2025). Nev- Additional rounds increase agreement but do not improve consensus accuracy beyond the ertheless, these scores remain imperfect proxies because first exchange and can reduce correctness. language models can be overconfident, and identifying the answer-relevant span may itself depend on the task. Exisiting confidence-based approaches largely use such scores for inference-time selection; we instead use them to construct preference data from debate traces. That gap motivates Alignment by Confident Debate (AbCD), which constructs preference data from debate traces without using gold answer labels for supervision. A sampling-diverse committee generates and exchanges solutions over multiple debate rounds, after which final answers are clustered by task-level semantic equivalence. We score each cluster by aggregate confidence on its claimed answer-span rather than by the likelihood of the full reasoning trace. Completions from the highestscoring cluster are labeled preferred, while completions from the remaining clusters are labeled dispreferred. Agreement rate is recorded as a reliability signal and can optionally be incorporated as a soft weight, but it does not define the winning cluster. On GSM8K, SVAMP, and MathQA, confidence-based cluster selection identifies the correct answer cluster more often than majority voting, particularly on instances with tied vote counts. Post-training on the resulting preferences improves both selected-answer and mean agent accuracy, outperforming inference-only ensembling, inference-only debate, and SFT trained on the same debate traces across the evaluated settings. The same comparison holds for a larger model, for heterogeneous committees, and for execution-based evaluation on code generation. Where an answer is a discrete string, majority voting is the usual ensemble and should track confidence ensembling, so we report the comparison in the appendix N. Outside verifiable tasks a vote is no longer defined in the same way, and AbCD still is, selecting the preferred cluster by confidence in the claimed answer. Ablations on coding and open-ended tasks are reported in Appendix T and U. Contributions. We make the following contributions:

2

Published as a preprint.

1. We propose a confidence-based debate oracle that separates task-level answer equivalence from answer selection: equivalence groups completions into answer clusters, while confidence mass on the claimed-answer span determines the cluster weight. 2. We construct a gold-label-free preference-data pipeline that converts the oracle’s cluster assignments into preferred and dispreferred completions for standard post-training objectives. 3. We characterize the effects of committee diversity and debate depth through controlled experiments. Empirical observations suggest that sampling-diverse committees capture most of the observed benefit.

2

R ELATED WORK

Multi-agent debate at inference. Self-consistency aggregates independently sampled solutions by majority vote when final answers can be mapped to equivalent outcomes (Wang et al., 2022), and universal self-consistency extends this idea to free-form outputs by using an LLM to select a consistent answer among candidates (Chen et al., 2023). Multi-agent debate instead allows agents to exchange reasoning over multiple rounds and can improve answer selection relative to a single sampled solution (Du et al., 2023; Liang et al., 2024; Madaan et al., 2023). However, these gains are not guaranteed: under homogeneous agents, debate preserves expected correctness over time and can underperform independent voting (Choi et al., 2026; Zhu et al., 2026), and repeated interaction can increase similarity among agents’ rationales, amplifying shared errors (Dai, 2026). These findings motivate treating consensus as an outcome of the interaction and selection process rather than as a correctness label. Confidence-based answer selection. Token likelihoods provide an alternative to vote counts, but full-response likelihood is a coarse signal because it scores all generated tokens, including reasoning tokens that may be only weakly tied to final-answer correctness. Confidence computed on answerrelevant tokens provides a more targeted correctness signal (Razghandi et al., 2025; Lin & Hooi, 2025), and related work uses confidence estimates (including verbalized confidence, token-distribution statistics, and calibrated answer-token scores) to select answers at inference time (Taubenfeld et al., 2025; Kang et al., 2026; Kadavath et al., 2022; Tian et al., 2023). Our work differs in using claimed answer-span confidence to construct preferred and dispreferred traces for post-training. An extended discussion of related work, including a finer-grained taxonomy of debate-based and confidence-based methods, is provided in Appendix A. Self-training from unlabeled or weakly labeled traces. Prior methods construct training signals from verifier-approved traces (Zelikman et al., 2022; Gulcehre et al., 2023), pseudo-labels derived from sampled answers (Zuo et al., 2026; Shao et al., 2024), or judgments from another language model (Yuan et al., 2024; Bai et al., 2022). Other work uses debate traces for imitation, preference learning, or multi-agent reinforcement learning (Subramaniam et al., 2025; Srivastava et al., 2025; Samanta et al., 2025; Zhou et al., 2025; Motwani et al., 2024; Liu et al., 2026). These methods differ in how they identify preferred traces and assign training weights. To our knowledge, prior debate-training methods do not construct preference pairs by ranking task-level answer-equivalence clusters using aggregate claimed answer-span confidence. AbCD combines answer-equivalence with span-confidence mass, while treating agreement as a reliability statistic rather than as the definition of the winning answer.

3

M ETHODOLOGY

Let πθ be the base instruction-tuned policy, which produces a reasoning trace for each query x. We instantiate N sampling-diverse agents from πθ per query and run M rounds of multi-agent debate. The final traces are clustered according to task-level answer equivalence. Aggregate confidence on the claimed-answer span selects the winning cluster, whose traces are treated as preferred and the remaining traces as dispreferred. A learner initialized from πθ is then trained using one of several objectives: supervised fine-tuning (SFT), preference optimization (DPO or KTO), or group-relative debate-derived preference reward optimization (GRPO). Figure 2 and Algorithm 1 summarize the pipeline. The post-training objectives are standard; the novelty of AbCD lies in its confidence-based 3

Published as a preprint.

Figure 2: AbCD instantiates N sampling-diverse agents from the base policy πθ and lets them debate for M generations. Final traces are clustered by task-level answer equivalence, and the cluster with the largest aggregate confidence on the claimed-answer span is selected. Traces in the winning cluster are preferred, while the remaining traces are dispreferred (green ≻ red). The resulting preference supervision updates πθ without gold answers in the oracle.

oracle for selecting preferred traces. The following sections detail the committee, confidence score, equivalence-based oracle, and post-training objectives. 3.1

S AMPLING - DIVERSE COMMITTEE

The autoregressive policy (πθ ) generates a trace y = (y1 , . . . , y|y| ) conditioned on an input prompt x ∈ Xtrain :

πθ (y | x) =

|y| Y

πθ (yt | x, y<t )

t=1

A task parser A extracts the claimed answer a = A(y). For each prompt x, we instantiate N agents {πi }N i=1 from the base policy πθ . The agents may share parameters but use heterogeneous decoding configurations, such as different temperatures, role prompts, or sampling seeds. We refer to this as a sampling-diverse committee. At round 1, agents generate traces independently conditional on x: (1)

yi

∼ πi (· | x),

i = 1, . . . , N

Let seri serialize the previous-round traces shown to agent i. For exchange rounds m = 2, . . . , M , agents synchronously revise their traces:    (m) (m−1) yi ∼ πi · | x, seri {yj }j̸=i Thus, the procedure consists of one independent generation followed by M − 1 exchange rounds. The final debate collection is
(M ) Y(x) = y1 , . . . , yN , yi = yi , ai = A(yi ) Traces for which A(yi ) is undefined are excluded from equivalence clustering, assigned to the dispreferred set during preference construction, and counted as incorrect during evaluation. 3.2

A NSWER - CONFIDENCE SCORE

We define canswer as the answer-confidence score for agent i’s generated trace and consider two i span constructions: full-trace confidence ctrace and claimed answer-span i  confidence ci . For each gen (m−1)

erated token, the corresponding log-probability is ℓi,t = log πi yi,t | x, seri({yj All confidence scores below reuse these recorded token log-probabilities. 4

}j̸=i ), yi,<t .

Published as a preprint.

Trace confidence. We define the confidence of the full generated reasoning trace as the geometric mean of its token probabilities:   |yi | X 1 ctrace = exp ℓi,t  (1) i |yi | t=1 Answer-span confidence. Let S(yi ) ⊆ {1, . . . , |yi |} denote the token indices selected by a taskspecific answer-span parser. For a nonempty answer-span, S(yi ) ̸= ∅, we define   X 1 cspan = exp ℓi,t  (2) i |S(yi )| t∈S(yi )

This score is the geometric mean of the token probabilities on the selected claimed answer-span. If S(yi ) = ∅ or cannot be aligned with the generated tokens, then cspan is undefined. i 3.3

E QUIVALENCE CLUSTERS AND THE CONFIDENCE - WEIGHTED ORACLE

For each prompt x, parseable answers are partitioned into equivalence clusters using a task-specific relation ≡ that compares candidate answers with one another. The relation is evaluated without access to the gold answer. Let Ik denote the indices of the agents whose answers belong to cluster k. We define the cluster size and aggregate answer confidence as X nk = |Ik |, wk = canswer i i∈Ik

We select the winning cluster by lexicographic maximization k ⋆ = arg maxk wk , nk , which prioritizes aggregate answer confidence and uses cluster size as a secondary tie-break. The oracle answer is the canonical representative of the selected cluster â(x) = rep(k ⋆ ), where rep(k ⋆ ) returns a canonical string for cluster k ⋆ . If multiple clusters remain tied after both criteria, the prompt is excluded from oracle construction. For comparison, majority voting selects the cluster with the largest support, kmaj = arg maxk nk . We construct preferred and dispreferred trace sets relative to â(x): G − (x) = {yi ∈ Y(x) : ai ̸≡ â(x)}

G + (x) = {yi ∈ Y(x) : ai ≡ â(x)},

(3)

−

Unparseable traces are excluded from clustering and assigned to G during preference construction. Preference objectives use only prompts for which both G + (x) and G − (x) are nonempty. Unanimous debates may still contribute their preferred traces to SFT and GRPO. We also measure the support for the selected cluster by |G + (x)| α(x) = N The statistic α(x) can optionally be used as a soft weight for the loss contribution of a preference example. It measures support for the confidence-selected winner and does not determine which cluster wins. 3.4

P OST- TRAINING

For each prompt with a defined oracle, we retain the oracle answer and its winner support in 
Doracle = x, â(x), α(x) x∈Xtrain For preference-based objectives, we additionally retain prompts for which both G + (x) and G − (x) are nonempty: 
Dpref = x, G + (x), G − (x), α(x) x∈Xtrain The preferred and dispreferred sets provide the supervision for the corresponding post-training objective. SFT maximizes the likelihood of traces in G + (x). DPO forms preference pairs (y + , y − ) ∈ G + (x) × G − (x), while KTO uses the corresponding unpaired preferred and dispreferred traces. The 5

Published as a preprint.

(a) # Agents

(b) # Debate rounds

(c) Agreement-rate

(d) Correctness status

Figure 3: Committee behavior for Qwen-2.5-1.5B-it on GSM8K. Subfigures show accuracy by committee size (a), accuracy by debate rounds (b), agreement-rate across rounds (c), and correctness across rounds (d). contribution of each preference example can be optionally weighted by α(x). For GRPO, the oracle answer â(x) is retained as a fixed pseudo-target. Each newly sampled response y receives the reward rx (y) = 1[A(y) ≡ â(x)] . This reward measures agreement with the frozen debate oracle rather than correctness against a gold answer. The reward or the corresponding loss contribution can be optionally scaled by α(x).

4

E XPERIMENTS AND R ESULTS

We evaluate AbCD in three stages: committee configuration, oracle selection, and post-training. We first fix the committee and debate configuration through inference-only experiments, then compare full-trace and answer-span confidence on identical transcripts, and finally lock the winning configuration for all post-training experiments. This separates decisions about generating debate traces from decisions about labeling and using those traces for training. 4.1

E XPERIMENTAL SETUP

Models. We post-train five instruction-tuned models: Qwen-2.5-1.5B-it, Llama-3.2-1B-it, Phi-4-mini3.8B-it, Gemma-3-4B-it, and Llama-3.1-8B-it (Qwen et al., 2025; Grattafiori et al., 2024; Abouelenin et al., 2025; Team et al., 2025). Unless otherwise noted, post-training uses a single iteration from the instruction-tuned checkpoint with QLoRA (4-bit base model, rank r=32). Datasets. We evaluate mathematical reasoning on GSM8K (Cobbe et al., 2021) and SVAMP (Patel et al., 2021), quantitative question answering on MathQA (Amini et al., 2019), and code generation on MBPP (Austin et al., 2021). GSM8K and SVAMP use boxed numerical answers, MathQA uses answer-choice letters, and MBPP uses executable Python programs; the corresponding dataset and task-equivalence details are provided in Appendix C. Gold answers are used only for evaluation and designated ablations, never for debate, oracle construction, or post-training. Debate protocol. Each prompt is answered by a sampling-diverse committee of N =5 agents for M =2 rounds: independent generation followed by one exchange round. The agents use temperatures {0.01, 0.25, 0.5, 0.75, 1.0}. Unless otherwise noted, decoding uses top-p=0.9 and at most 256 new tokens. Detailed inference and training configurations are given in Appendix G. Metrics. We report pre-debate agent accuracy, the mean accuracy of the independently sampled responses; post-debate agent accuracy, the mean accuracy of the N responses after the final round; and consensus accuracy, the accuracy of the answer cluster selected by the confidence-based oracle. Accuracies are computed against gold labels on the full test set, with missing or unparseable outputs counted as incorrect. 4.2

C OMMITTEE C ONFIGURATION

We first characterize committee behavior using inference-only experiments to determine the default configuration for debating. Varying committee size from N =1 to N =9 with independent sampling at temperature 0.5 shows that consensus accuracy improves up to N =5, with diminishing returns thereafter (Figure 3a). The first exchange round provides an additional gain, while further rounds increase agreement (Figure 3c) without improving consensus accuracy and can reduce correctness (Figure 3d). We observe the same trend across models and datasets; additional results are provided in Appendix 6

Published as a preprint.

Table 1: Consensus accuracy (%) for inference-only baselines and post-training objectives. Ensemble denotes inference-only independent sampling, and Debate denotes inference-only multi-round debate. SFT, DPO, KTO, and GRPO are trained on the same debate traces, with preferred completions selected by the answer-span confidence oracle. The best post-training result for each model–dataset pair is in bold. The final column reports the absolute improvement of the best post-training method over inference-only debate, in percentage points. Model

Baselines

Dataset

Post-training

Best ∆

Ensemble

Debate

SFT

DPO

KTO

GRPO

Llama-3.2-1B-it

SVAMP GSM8K MathQA

18.3 14.8 7.2

19.0 15.6 5.0

25.2±2.12 15.6±1.13 6.3±1.56

40.5±1.70 27.0±3.96 9.2±1.41

36.2±4.03 21.6±4.24 8.3±1.27

27.2±1.20 14.8±3.11 6.7±1.56

21.5pp 11.4pp 4.2pp

Qwen-2.5-1.5B-it

SVAMP GSM8K MathQA

77.0 44.5 21.6

79.7 66.0 45.2

78.7±0.49 66.3±0.71 46.0±2.26

79.7±0.92 71.4±0.28 53.0±1.13

78.0±2.83 69.2±1.13 53.7±1.27

80.3±2.33 68.9±0.14 51.1±1.56

0.6pp 5.4pp 8.5pp

Phi-4-mini-3.8B-it

SVAMP GSM8K MathQA

88.2 88.6 36.6

89.7 87.6 41.6

89.9±0.21 87.9±1.27 42.2±2.83

90.0±0.42 87.9±1.14 59.5±5.52

89.4±0.49 88.3±0.99 50.8±5.66

89.4±0.49 87.9±0.14 43.1±2.97

0.3pp 0.7pp 17.9pp

Gemma-3-4B-it

SVAMP GSM8K MathQA

80.7 65.2 15.8

75.7 60.6 16.6

73.5±1.70 51.8±0.85 11.8±0.57

81.4±3.75 72.7±1.27 21.5±2.12

80.4±1.91 68.8±2.26 17.3±0.99

80.2±2.62 62.9±0.14 16.3±1.56

5.7pp 12.1pp 4.9pp

E. This divergence shows that agreement alone is not a reliable proxy for correctness, motivating an oracle that uses more than vote count. We therefore lock N =5 and M =2 as the default. Among the diversity strategies evaluated, temperature diversity outperforms homogeneous clones, role diversity, and seed diversity (Appendix Table 8); we therefore use temperature-diverse committees by default. 4.3

T RACE C ONFIDENCE VS . S PAN C ONFIDENCE

Before constructing the preference dataset, we determine which portion of each response should be used to compute confidence. For agent i, we compare full-trace confidence ctrace with claimed i answer-span confidence cspan , computed from the same debate transcripts to isolate the effect of i the scored tokens. Figure 4 and Table 9 report AUROC for distinguishing correct from incorrect final answers before and after the first exchange. Across 12 model–dataset pairs, full-trace AUROC decreases by an average of 0.07 after the first exchange, whereas answer-span AUROC increases by an average of 0.04 (details in Appendix F). This pattern is consistent with full-trace likelihood capturing properties of intermediate reasoning tokens whose fluency need not reflect final-answer correctness. Although the trend is not uniform across models, the aggregate comparison motivates using the claimed answer-span; we therefore set canswer := cspan in all subsequent experiments. i i 4.4

P REFERENCE T RAINING WITH THE S PAN O RACLE

We use the answer-span oracle to construct preference data for post-training, comparing inference-only independent sampling (Ensemble), inference-only multi-round debate (Debate), SFT, and three preference-based objectives (DPO, KTO, GRPO). Results are averaged over two training runs with the same seed and reported with the corresponding standard deviation. All post-training methods use the same debate traces and base model; implementation details are in Appendix G. For SFT, the training target is G + , the oracle-selected completions, not gold solutions. For preference-based objectives, completions Figure 4: Change in AUROC from the initial-response round to the first exin G + are preferred over those in competing clusters. change round for full-trace and claimed answer-span confidence. Each bar is Overall effectiveness. Table 1 shows that across all 12 one model–dataset pair across GSM8K, model–dataset pairs, the best preference-based objective im- SVAMP, and MathQA. Positive values proves over inference-only debate. DPO achieves the best indicate improved discrimination after post-training result on 9 pairs, KTO on 2, and GRPO on 1. debate.

7

Published as a preprint.

Table 2: Consensus accuracy (%) for Llama-3.1-8B-it on GSM8K and MBPP. The final column is the absolute improvement of the best post-training method over inference-only debate, in percentage points. Model Llama-3.1-8B-it

Baselines

Dataset GSM8K MBPP

Post-training

Ensemble

Debate

SFT

DPO

KTO

GRPO

85.8 46.0

82.6 45.7

75.6 45.7

87.6 49.1

83.6 47.9

83.8 47.2

Best ∆ 5.0pp 3.4pp

The largest gain occurs for Llama-3.2-1B-it on SVAMP, where DPO raises consensus accuracy from 19.0% to 40.5% (+21.5pp ); Phi-4-mini-3.8B-it on MathQA shows a comparable gain, from 41.6% to 59.5% (+17.9pp ). Gains are smallest in near-ceiling settings, such as SVAMP (89.7% → 90.0%) and GSM8K (87.6% → 88.3%) for Phi-4-mini-3.8B-it, consistent with limited room for improvement when baseline accuracy is already high. Value of dispreferred traces. SFT and preference-based objectives use the same oracle-generated traces, but SFT imitates only the oracle-selected completions, whereas DPO, KTO, and GRPO also use completions from competing clusters. SFT improves over inference-only debate in 7 of 12 settings, while at least one preference-based objective improves over debate in all the settings. This gap suggests that incorporating competing-cluster completions provides training signal beyond the oracle-selected completions used by SFT. The best post-training method also improves mean post-debate agent accuracy in all 12 model–dataset pairs (Appendix H), indicating that gains extend beyond consensus selection to individual-agent accuracy. Appendix N compare majority-vote labels performance on the same traces.

5

A BLATIONS

We assess the reliability of the framework around three questions: whether the gains transfer to a larger policy, whether the framework extends beyond numerical and multiple-choice reasoning, and which committee, oracle, and loss-design choices are necessary. Additional implementation details are in Appendix I. Larger model. Table 2 evaluates Llama-3.1-8B-it on GSM8K. DPO improves consensus accuracy from 82.6% to 87.6% (+5.0pp ), confirming that the method scales beyond the 1-4B models in Table 1. The gain is smaller than the largest improvements among smaller models, consistent with the higher inference-only baseline and reduced headroom. Code generation. We test whether the framework extends to code, where answer equivalence is determined by program-output matching on public inputs rather than a boxed numerical answer. Llama-3.1-8B-it is evaluated on MBPP (Austin et al., 2021) (374 training, 200 test problems). DPO improves polled pass rate of all agents from 45.7% to 49.1% (+3.4pp ), showing that the confidence oracle generalizes to execution-based settings. Held-out unit tests are used only for evaluation. Check Appendix T for details. Oracle selection protocol We compare four labeling rules on GSM8K with Qwen-2.5-1.5B-it: gold labels, majority voting, full-trace confidence, and claimed answer-span confidence. Table 3 reports DPO accuracy for cloned and temperature-diverse committees (3-seed averages). The span oracle achieves the highest post-training accuracy in both settings (72.4% and 70.4%). Majority voting under-performs span-confidence selection for both committee types. For the cloned committee, gold supervision also falls below the span oracle (68.6% vs. 72.4%), whereas for the temperaturediverse committee it remains below the span oracle (69.0% 8

Table 3: GSM8K consensus accuracy (%) after DPO by training oracle. Debate is the inference-only baseline. Trace and Span use full-response and claimed answer-span confidence. Gold labels serve as a supervised bound. Oracle

Cloned

Temp.-diverse

Debate Gold Majority Trace Span

65.0 68.6 69.4 67.2 72.4

65.3±1.30 69.0±0.69 68.1±1.5 69.9±0.90 70.4±1.11

Published as a preprint.

Table 4: Comparison of majority voting (MV) and aggregate answer-span confidence before posttraining on GSM8K, across four models. Disagree is the fraction of examples on which the two rules select different answer clusters. The final two columns report selection accuracy on disagreement and count-tie cases, respectively. Model

Span

MV

Disagree

Disagree: Span/MV

Ties

Ties: Span/MV

Llama-3.2.1B-it Qwen-2.5-1.5B-it Phi-4-mini-3.8B-it Gemma-3-4B-it

15.6 66.0 88.8 60.6

14.6 65.0 88.4 60.6

3.2 5.8 1.2 0.0

31.2 / 0.0 31.0 / 13.8 33.3/ 0.0 0.0 / 0.0

5.8 12.8 1.8 0.4

24.1 / 6.9 31.2 / 23.4 33.3 / 11.1 50.0 / 50.0

Average

57.5

57.2

2.6

31.4 / 7.8

5.2

29.8 / 18.3

Table 5: GSM8K consensus/mean-agent accuracy (%). Homogeneous committee (Hom.) uses five sampling-diverse copies; Heterogeneous committee (Het.) uses one member each from L1, Q1, Q2, P4, and G4. ∆ is the Debate→DPO change in percentage points. Model

Type

Ens.

Debate

DPO

∆

L1 Q1 Q2 P4 G4

Hom. Hom. Hom. Hom. Hom.

14.8/4.2 44.5/24.2 2.0/9.0 88.6/76.5 65.2/48.6

15.6/5.4 66.0/44.5 32.4/17.4 87.6/82.2 60.6/45.3

27.0/13.2 71.4/60.8 60.8/45.4 87.9/82.6 72.7/66.0

+11.4/+7.8 +5.4/+16.3 +28.4/+28.0 +0.3/+0.4 +12.1/+20.7

All

Het.

75.0/31.5

79.2/36.8

87.0/68.2

+7.8/+31.4

vs. 70.4%) but above majority voting (68.1%). Table 4 compares majority voting (MV) and span-confidence selection on identical, pre-training transcripts. Across four models on GSM8K, the two rules disagree on at most 5.8% of examples, with span confidence achieving slightly higher average oracle accuracy (57.5% vs. 57.2%). On disagreement cases, span confidence selects the correct cluster more often than MV (31.4% vs. 7.8%), with a similar advantage on count-tie cases (29.8% vs. 18.3%). Check Appendix I.3 for details on more datasets. Heterogeneous committee. The default committee uses sampling-diverse copies of one policy. We test a heterogeneous committee with one model from each of five families: Llama-3.2-1B-it (L1), Qwen-2.5-1.5B-it (Q1), Qwen3-1.7B-it (Q2), Phi-4-mini-3.8B-it (P4), and Gemma-3-4B-it (G4), all at temperature 0.5 with M =2 on GSM8K. DPO updates every member on the same span-oracle preference pairs. Table 5 shows that heterogeneous debate reaches 79.2% consensus, rising to 87.0% after DPO (+7.8pp ), while mean agent accuracy increases by 31.4pp . Per-agent accuracy and more post-training approaches with heterogeneous committee are described in Appendix I.4. The gap between consensus and mean-agent accuracy (18.8pp ) is far larger than in any homogeneous committee, indicating that the shared preference partition does not benefit all members equally. This exposes a calibration limitation: token-probability confidence may not be comparable across architectures, so cross-model calibration remains an assumption of this experiment rather than an established property. With per-model Platt calibration maps fit on gold correctness, heterogeneous debate consensus gives more than 8pp improvement, while member accuracies stay uneven (Appendix P). Agreement weighting. We compare three DPO variants with the span oracle fixed: unweighted DPO (α(x) = 1), supportweighted DPO (per-example loss scaled by α(x) = |G + (x)|/N ), and agreement-filtered DPO (retain only examples with ≥ 3 of 5 agents supporting the preferred cluster). Table 6 shows that weighting does not improve final accuracy: unweighted DPO reaches 70.4%, compared with 70.0% for support-weighted and 68.5% for agreement-filtered. We therefore recommend to retain unweighted DPO as the default and treat α(x) as a diagnostic quantity, not as part of the training objective. 9

Table 6: GSM8K consensus accuracy (%) for Qwen-2.5-1.5Bit after DPO with different uses of winner support. Means ± std over three seeds. Method

Accuracy

Debate Unweighted DPO Weighted DPO Agreement-filtered

67.1±1.56 70.4±1.11 70.0±1.99 68.5±3.25

Published as a preprint.

6

C ONCLUSION

Alignment by Confident Debate (AbCD) constructs preference data from multi-agent debate traces. The framework clusters final answers by task-level equivalence and ranks the clusters using aggregate confidence on their claimed answer, converting debate disagreement into supervision compatible with standard post-training objectives. Across mathematical reasoning, question-answering, and coding benchmarks, our oracle selection identifies correct answer cluster more accurately than majority voting overall, with a larger advantage on disagreement and count-tie instances. Preference training on the resulting pairs improves over inference-only ensembling, inference-only debate, and SFT trained on the same traces, with the largest gains occurring when the inference-only committee has substantial headroom. However, heterogeneous committees expose a limitation of aggregating raw token-probability confidence across models, motivating future work on cross-model confidence calibration. These results indicate that combining task-level answer equivalence with confidenceweighted disagreement can turn multi-agent debate into a practical self-training signal, provided that outputs are parseable, a suitable equivalence procedure is available, and confidence is comparable across the committee.

AI USE STATEMENT In this work, we used generative AI tools as research and writing assistants to provide feedback on the presentation and interpretation of experimental results, check consistency between reported numerical results and tables, and improve the clarity, grammar, and organization of the manuscript. We also used generative AI tools for limited assistance with LaTeX formatting and presentation of tables and figures. Generative AI tools were not used to generate datasets, run experiments, produce experimental results, or implement the proposed method or training pipeline. The research methodology, experimental design, analysis, and scientific conclusions were determined by the authors, with AI assistance used only as a supporting tool. All AI-assisted content was reviewed and verified by the authors. In particular, numerical values, statistical results, experimental comparisons, mathematical formulations, and claims were checked against the underlying experimental outputs and implementation. The authors made the final decisions regarding the methodology, interpretation, and presentation of the work and take full responsibility for the final manuscript and all claims and artifacts contained therein.

R EFERENCES Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2357–2367, 2019. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation. arXiv preprint arXiv:2311.17311, 2023. 10

Published as a preprint.

Hyeong Kyu Choi, Jerry Zhu, and Sharon Li. Debate or vote: Which yields better decisions in multi-agent large language models? Advances in Neural Information Processing Systems, 38: 101732–101764, 2026. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Chengxiao Dai. When consensus is not correctness: Diversity collapse and manufactured overconfidence in multi-agent llm debate. In KDD workshop on Evaluation and Trustworthiness of Agentic AI, 2026. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. In International Conference on Learning Representations, volume 2026, pp. 127924–127945, 2026. Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899, 2018. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty. Advances in neural information processing systems, 38:19720–19745, 2026. Ali Keramati, Justin Cheok, Jacob Horne, and Mark Warschauer. Early-token confidence predicts reasoning quality in multi-agent llm debate. In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), pp. 653–667, 2026. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multiagent debate. In Proceedings of the 2024 conference on empirical methods in natural language processing, pp. 17889–17904, 2024. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, pp. 39578–39601, 2024. Zijie Lin and Bryan Hooi. Enhancing multi-agent debate system performance via confidence expression. arXiv preprint arXiv:2509.14034, 2025. Shuo Liu, Zeyu Liang, Xueguang Lyu, and Christopher Amato. Llm collaboration with multiagent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 32150–32158, 2026. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems, 36:46534–46594, 2023. 11

Published as a preprint.

Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip HS Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. Malt: Improving reasoning with multi-agent llm training. arXiv preprint arXiv:2412.01928, 2024. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– 27744, 2022. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 2080–2094, 2021. John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74, 1999. Dan Qiao, Binbin Chen, Fengyu Cai, Jianlong Chen, Wenhao Li, Fuxin Jiang, Zuzhi Chen, Hongyuan Zha, Tieying Zhang, and Baoxiang Wang. Epistemic gain, aleatoric cost: Uncertainty decomposition in multi-agent debate for math reasoning. arXiv preprint arXiv:2603.01221, 2026. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. Ali Razghandi, Seyed Mohammad Hadi Hosseini, and Mahdieh Soleymani Baghshah. Cer: Confidence enhanced reasoning in llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7918–7938, 2025. Ankur Samanta, Akshayaa Magesh, Runzhe Wu, Ayush Jain, Youliang Yu, Daniel Jiang, Boris Vidolov, Paul Sajda, Yonathan Efroni, and Kaveh Hassani. Self-improvement of language models by post-training on multi-agent debate. arXiv preprint arXiv:2509.15172, 2025. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Gaurav Srivastava, Zhenyu Bi, Meng Lu, and Xuan Wang. Debate, train, evolve: Self-evolution of language model reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 32752–32798, 2025. Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. In International Conference on Learning Representations, volume 2025, pp. 10840–10862, 2025. Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in llms. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 20090–20111, 2025. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433–5442, 2023. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. 12

Published as a preprint.

Yue Wu, Zhiqing Sun, Rina Hughes, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. In International Conference on Learning Representations, volume 2025, pp. 91558–91582, 2025. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022. Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, et al. A careful examination of large language model performance on grade school arithmetic. Advances in Neural Information Processing Systems, 37: 46819–46836, 2024. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023. Xiaofeng Zhou, He-Yan Huang, and Lizi Liao. Debate, reflect, and distill: Multi-agent feedback with tree-structured preference optimization for efficient language model enhancement. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 9122–9137, 2025. Xiaochen Zhu, Caiqi Zhang, Yizhou Chi, Tom Stafford, Nigel Collier, and Andreas Vlachos. Demystifying multi-agent debate: The role of confidence and diversity. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 33909–33930, 2026. Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Xinwei Long, Ermo Hua, Biqing Qi, et al. Ttrl: Test-time reinforcement learning. Advances in Neural Information Processing Systems, 38:131459–131483, 2026.

13

Published as a preprint.

A

L ITERATURE REVIEW

Multi-agent debate at inference. Self-consistency aggregates independent chains by majority vote when answers can be checked for equivalence (Wang et al., 2022). Universal self-consistency relaxes that requirement by asking a model which free-form samples are consistent (Chen et al., 2023). Multi-agent debate instead lets agents read one another across multiple rounds, and the group’s answer can be stronger than any single chain (Du et al., 2023; Liang et al., 2024). That gain is not automatic. Under homogeneous agents and unweighted belief updates, debate preserves expected correctness over time, so the protocol can underperform a vote over independent samples (Choi et al., 2026; Zhu et al., 2026). Over rounds, repeated interaction can increase similarity among agents’ rationales, a failure mode termed representational collapse, so agreement can manufacture confidence while errors stay shared (Dai, 2026). The remaining uncertainty mixes epistemic disagreement with aleatoric decoding noise, and longer debates can let the latter flip a correct answer (Qiao et al., 2026). These results diagnose test-time debate. AbCD takes it as a constraint on the training oracle: consensus is an outcome of the protocol, so it cannot be the label. Confidence-weighted selection. A natural alternative to vote count is the model’s own likelihood. Full-response likelihood is a coarse signal because it scores all generated tokens, including reasoning tokens that may be only weakly tied to final-answer correctness. CISC weights independent samples by verbalized confidence, P (true), or sequence probability (Taubenfeld et al., 2025); self-certainty uses the peakedness of the token distribution (Kang et al., 2026). Restricting the score to answerrelevant tokens such as numbers, entities, or choice letters ranks correctness better than scoring every token (Razghandi et al., 2025). In debate, length-normalized answer-token confidence, after calibration, can improve the committee’s answer selection (Lin & Hooi, 2025), and diversity-aware initialization combined with calibrated confidence communication can allow homogeneous debate to outperform majority vote (Zhu et al., 2026). On open-ended debate without a reference answer, early-generation log-probabilities vary more across traces than the rest of the sequence and better predict judged quality (Keramati et al., 2026). Token probabilities are also a well-studied uncertainty signal more broadly (Kadavath et al., 2022; Tian et al., 2023). All of this work uses confidence to select an answer at inference time. AbCD uses span-level confidence to construct preferred and dispreferred traces for post-training. Self-training without gold labels. STaR and ReST fine-tune on traces that pass a verifier (Zelikman et al., 2022; Gulcehre et al., 2023). TTRL and related methods treat the majority among independent samples as a pseudo-label for GRPO or DPO (Zuo et al., 2026; Shao et al., 2024). Self-rewarding models and RLAIF replace the verifier with another language model (Yuan et al., 2024; Bai et al., 2022). Process reward models need step-level annotation (Lightman et al., 2024). These methods still rely on an external correctness signal from a checker, a judge, or an independent majority. AbCD stays in this family but derives its signal from a debating committee and from span-level likelihood, not from a gold answer, a judge, or vote count alone. Debate as post-training data. Multi-agent interaction is increasingly used as a training corpus, not only as a test-time ensemble. One line of work imitates debate roles or consensus traces using SFT or GRPO (Subramaniam et al., 2025; Srivastava et al., 2025). Another turns majority versus minority traces, or debate-derived preference trees, into preference pairs (Samanta et al., 2025; Zhou et al., 2025). A third jointly trains specialized agents in generate–verify–refine loops, student–teacher setups, or cooperative multi-agent reinforcement learning (MARL), typically against gold labels, a teacher model, or a shared task reward (Motwani et al., 2024; Liu et al., 2026). AI-safety debate is a different setting: two agents argue for a human judge (Irving et al., 2018). What these methods share is how they name and weight the data. The winner is determined by vote count, a gold answer, or a teacher; every retained pair is treated equally; and the committee is typically composed of homogeneous clones, with diversity obtained only by training distinct roles rather than by scoring a diverse sample of a single policy. To our knowledge, prior debate-training methods do not construct preference pairs by ranking task-level answer-equivalence clusters using aggregate claimed answerspan confidence, or treat the degree of disagreement within the committee as a reliability weight rather than as the label. Unlike prior debate-based training methods that use vote count, a teacher, or a gold answer to assign labels, AbCD constructs preference data by combining task-level answer equivalence, span-confidence mass, and disagreement-based reliability weighting. 14

Published as a preprint.

B

A LGORITHM

Algorithm 1 outlines AbCD. For each prompt, a diverse committee debates for M rounds; final answers are clustered by equivalence and ranked by confidence mass on the claimed answer-span. The winning cluster yields preferred completions, the rest dispreferred, and the agreement rate can be used as a soft reliability weight. We collect SFT and preference datasets from these traces and post-train the base policy with standard objectives. Algorithm 1 AbCD: Alignment by Confident Debate Require: Training prompts Xtrain , base policy πθ , parser A, span extractor S, equivalence relation ≡, committee size N , and number of debate rounds M 1: DSFT ← ∅ 2: Dpref ← ∅ 3: Doracle ← ∅ 4: for each x ∈ Xtrain do 5: Instantiate sampling-diverse agents {πi }N i=1 from πθ (1) 6: yi ∼ πi (· | x) for i = 1, . . . , N 7: for m = 2, . . ., M do   (m) (m−1) 8: yi ∼ πi · | x, seri {yj }j̸=i for all i 9: end for (M ) 10: Y(x) ← {yi }N i=1 11: ai ← A(yi ) for i = 1, . . . , N 12: Compute canswer for each parseable trace i 13: Cluster parseable answers by ≡ into {Ik } 14: if no parseable cluster exists then 15: continue 16: end if P 17: wk ← i∈Ik canswer and nk ← |Ik | i 18: k ⋆ ← arg maxk (wk , nk ) 19: â(x) ← rep(k ⋆ ) 20: G + (x) ← {yi : ai ≡ â(x)} 21: G − (x) ← {yi : ai ̸≡ â(x)}, including unparseable traces 22: α(x) ← |G + (x)|/N 23: Doracle ← Doracle ∪ {(x, â(x), α(x))} 24: DSFT ← DSFT ∪ {(x, G + (x))} 25: if G + (x) ̸= ∅ and G − (x) ̸= ∅ then 26: Dpref ← Dpref ∪ {(x, G + (x), G − (x), α(x))} 27: end if 28: end for 29: Post-train πθ using SFT, DPO, KTO, or GRPO on the corresponding dataset

C

DATASETS

We evaluate AbCD on four publicly available benchmarks spanning mathematical word problems, multiple-choice quantitative question answering, and Python code generation. GSM8K, SVAMP, and MathQA evaluate mathematical or quantitative reasoning; MBPP evaluates code generation with executable unit tests. Gold answers and full unit-test outcomes are used only for evaluation and reporting accuracy; they are not provided to the debate oracle or used as training labels. GSM8K. GSM8K contains 8,500 grade-school mathematical word problems with natural-language solutions and numeric final answers (Cobbe et al., 2021). It evaluates multi-step arithmetic reasoning in natural language. For AbCD, we parse and cluster the boxed numerical final answers produced by committee members. SVAMP. SVAMP contains arithmetic word problems designed to test robustness to variations in linguistic form and problem structure (Patel et al., 2021). Its controlled perturbations reduce the 15

Published as a preprint.

Table 7: How agent answers are clustered and scored for the training label. Group

Datasets

What is aggregated

Confidence

Discrete math Executable code

GSM8K, SVAMP, MathQA MBPP

Span confidence Public tests passed

Checklist text

RaR

The same extracted answer Programs with the same public outputs Identical satisfied-criteria patterns

Weighted hit score, penalties subtracted

usefulness of superficial shortcuts and emphasize the underlying arithmetic relations. As in GSM8K, committee members produce boxed numerical final answers. MathQA. MathQA is a large-scale quantitative reasoning benchmark containing natural-language questions paired with executable mathematical programs and multiple-choice answer options (Amini et al., 2019). We use it as a multiple-choice quantitative QA setting, where agents produce a final answer in the prescribed answer format and the oracle clusters answers by selected option equivalence. MBPP. MBPP contains approximately 1,000 crowd-sourced Python problems intended to be solvable by entry-level programmers (Austin et al., 2021). Each example provides a natural-language specification, a reference function, a public test list, and an optional challenge list. A public test is an assertion of the form assert f (x) = y. We parse each assertion into the arguments x and the expected return value y, execute the candidate agent response on x, and mark the case passed when the returned value matches y. Programs that emit the same sequence of public outputs, including the same exception type on a failing case, form one equivalence class. The execution oracle ranks a class by how many of these public cases its members pass. Challenge assertions are omitted from this trace, so they do not affect clustering or the training label. RaR. The main paper we use tasks on which a final answer is a number, a choice letter, or a program, so two traces can be tested for equivalence and then counted. RaR-Medicine (Gunjal et al., 2026) does not have that kind of answer. We use the medicine split of Rubrics as Rewards (Gunjal et al., 2026), released as ScaleAI/RaR-Medicine on Huggingface. Each example is a clinical question, a reference answer, and a checklist. A checklist is an ordered list of pass/fail criteria produced by the autorubric pipeline of Gunjal et al. (2026). We used this dataset in appendix U. Task-level equivalence. The relation ≡ in Section 3.3 is computed entirely from the committee’s own outputs: it compares parsed candidate answers with one another and never receives the reference answer. A trace for which the parser A(y) is undefined is excluded from clustering. On GSM8K and SVAMP, y ≡ y ′ when the numbers parsed from the final boxed spans denote the same value, so two derivations of the same quantity form one cluster regardless of wording. On MathQA, y ≡ y ′ when those spans name the same choice letter. On MBPP, y ≡ y ′ when both completions parse as programs, execute on the public tests, and return the same outputs; the public tests are not part of the generation prompt, and the hidden challenge tests are used only to score the selected program at evaluation. In all four tasks, ≡ is a relation on task outcomes, not on the text of the reasoning trace. But in RaR we used a Judge model to find the equivalence of free text generated by agents. Training label formation. The training label is a cluster of agent answers. The rule that decides which answers belong together, and the score that chooses the winning cluster, depend on the dataset. GSM8K and SVAMP are grade-school word problems with one numerical answer. MathQA is also a math problem, but the answer is a choice letter. On these three datasets every agent produces a short discrete answer, so confidence ensembling is an alternative to a majority vote. The same construction extends to domains without a single gold string. MBPP is execution. Programs that produce the same public outputs form one cluster, a vote is well-defined after that execution, and confidence is the number of public tests passed. RaR is free text. A judge marks the checklist, including penalty items with negative weight, and confidence is the weighted score of the marked items. Voting remains a baseline after the same clustering: on code it counts programs with the same public outputs, and on rubrics it counts copies of the same checklist. Table 7 lists the three groups. 16

Published as a preprint.

D

P ROMPT T EMPLATES

All agents in a committee share the same base prompt for a given domain, with minor task-specific variants for math, multiple-choice QA, and coding. For math and QA tasks, the initial prompt instructs the model to provide a bullet-point summary of its step-by-step reasoning and to emit a final answer in the form answer . For coding (MBPP), the prompt requests a single Python function in a fenced code block with no additional text. In each debate round, agents receive the other agents’ previous responses and are asked to update their reasoning or code accordingly, while preserving the same final-answer format. Math (GSM8K, SVAMP) Initial Prompt for Math problems Solve the following math problem. {question} Provide a bullet point summary of your step-by-step reasoning . Your final answer should be a single numerical number, in the form \boxed{answer}, at the end of your response.

Multiple-choice QA (MathQA) Initial Prompt for Multiple-Choice QA Answer the following multiple choice question as accurately as possible. {question} Provide a bullet point summary of your step-by-step reasoning . Your final answer should be a single choice letter in the form \boxed{answer}, at the end of your response.

For Coding (MBPP) Initial Prompt for Coding problems Write Python for this coding problem. Output code only. {question} Reply with ONLY a ‘‘‘python fenced block. No other text. Inside it, write the complete function. No tests, no print(), no extra drivers.

Debate exchange rounds (all domains). In each debate round, agents receive the other agents’ previous responses and are asked to update their solution while preserving the domain-specific final-answer format. The exchange prompt is: Debate Exchange round prompt Here are solutions from other agents: {other_agent_solutions}

17

Published as a preprint.

[Using each response as additional advice, give an updated bullet point summary of your step-by-step reasoning to the question: {question}] OR [Using each response as additional advice, update your Python solution to the question: {question}] [Make sure your final answer is in the form \boxed{answer}, at the end of your response.] OR [Make sure your final answer is a single choice letter in the form \boxed{answer}, at the end of your response.] OR [Reply with ONLY a ‘‘‘python fenced block. No tests, no print (), no extra drivers.]

E

C OMMITTEE C ONFIGURATION D ETAILS

Unless otherwise noted, all agents decode with top-p = 0.9 and generate at most 256 tokens per completion. Figure 3a reports GSM8K accuracy as a function of committee size N and number of debate rounds M for cloned Qwen-2.5-1.5B-it committees at a shared temperature of 0.5. We observe the same qualitative trends for Llama-3.1-8B-it across committee sizes and debate rounds (Figure 5 and 6e).

Figure 5: Committee behavior for Llama-3.1-8B-it model on GSM8K. The consensus accuracy by committee size beyond 5 agents shows diminishing returns. We additionally fix the committee size (5) and compare the effect of additional exchange rounds across model families on GSM8K (Figure 6). We observe similar behavior across other datasets (Figure 7). Across these settings, the first exchange round often improves consensus accuracy, whereas further rounds increase agreement without a corresponding improvement in correctness and can subsequently reduce consensus accuracy. This pattern is consistent with debate-induced correlation: later rounds make agents’ answers more alike without necessarily adding independent evidence for the correct answer. We further stratify oracle precision by agreement-rate. For a five-agent committee, an agreement rate of 2/5 means that two agents produce parseable final answers in the same answer-equivalence cluster; the remaining agents either support other clusters or produce unparseable answers. Precision is the fraction of oracle-selected answers that are correct within each agreement-rate bin, using reference answers only for evaluation. Across the evaluated models, later exchange rounds tend to reduce precision at several agreement levels, including partially split committees, despite increasing the frequency of high-agreement outcomes. To isolate the effect of committee diversity, we compare four configurations: (i) clones, which share temperature 0.01; (ii) temperature-diverse agents, which use temperatures {0.01, 0.25, 0.5, 0.75, 1.0} across five agents; (iii) role-diverse agents, which use distinct system prompts at the same temperature 0.01 and (iv) seed-diverse agents, which use distinct sampling seeds at a shared temperature 18

Published as a preprint.

(a) Llama-3.2-1B-it

(b) Qwen2.5-1.5B-it

(c) Phi-4-mini-3.8B-it

(d) Gemma-3-4B-it

(e) Llama-3.1-8B-it

Figure 6: Effect of additional debate rounds on GSM8K across model families. Left: Consensus accuracy at a fixed committee size as the number of exchange rounds increases. Right: Oracle precision stratified by agreement rate, where k/5 denotes that k of five agents produce parseable answers in the selected answer-equivalence cluster. The first exchange round often improves consensus accuracy, while subsequent rounds can increase agreement without improving correctness and can reduce oracle precision at several agreement levels. Reference answers are used only for evaluation.

(a) SVAMP

(b) MathQA

Figure 7: Effect of additional debate rounds on Qwen-2.5-1.5B-it across datasets. Left: Consensus accuracy at a fixed committee size as the number of exchange rounds increases. Right: Oracle precision stratified by agreement rate, where k/5 denotes that k of five agents produce parseable answers in the selected answer-equivalence cluster. Reference answers are used only for evaluation.

0.01. Table 8 reports the corresponding inference-only results. Across the evaluated models, the temperature-diverse committee achieves either the highest or second-highest consensus accuracy. We therefore use it as the default committee configuration for the experiments in Section 4.

Role prompt templates. These system prompts are used for the role-diverse committee for committee configuration (Section 4.2).

19

Published as a preprint.

System Prompts for Role-Diverse Committee mechanism: You are the MECHANISM reasoner. Derive the answer from first principles. Identify governing quantities, write governing relations, and chain causal steps before computing a final result. pattern: You are the PATTERN reasoner. Match this problem to prototypical cases. You recognize (standard templates, common problem types) and adapt the known solution pattern to this instance. arbiter: You are the ARBITER. Weigh competing arguments fairly and synthesize the most defensible conclusion. verifier: You are the VERIFIER. Check peer reasoning for logical and arithmetic errors; correct mistakes explicitly. solver: You are the SOLVER. Produce a clear step-by-step solution.

Table 8: Inference-only GSM8K accuracy (%) at N =5, M =2. Initial response accuracy is singlesample accuracy before any ensemble or debate; Avg. agent performance is average per-agent accuracy after M rounds of debate; Consensus correctness is aggregated response accuracy. Committee

Initial response accuracy

Avg. agent performance

Consensus correctness

Llama-3.2-1B-it Clones Temperature-diverse Role-diverse Seed-diverse

3.2 4.4 5.6 3.2

Clones Temperature-diverse Role-diverse Seed-diverse

24.6 23.2 21.0 23.8

Clones Temperature-diverse Role-diverse Seed-diverse

77.2 76.7 65.8 77.2

Clones Temperature-diverse Role-diverse Seed-diverse

47.8 48.1 14.3 47.7

Clones Temperature-diverse Role-diverse Seed-diverse

68.6 66.0 40.9 69.3

4.6 7.4 7.3 4.3

6.4 17.0 20.6 6.6

Qwen-2.5-1.5B-it 48.7 44.5 44.6 47.3

56.4 66.0 63.6 56.6

Phi-4-mini-3.8B-it 81.3 80.9 76.8 80.0

84.8 87.6 86.6 83.0

Gemma-3-4B-it 44.6 42.6 4.6 44.6

51.4 57.8 18.2 51.8

Llama-3.1-8B-it 66.4 61.6 27.2 67.0

20

74.0 81.0 65.4 75.2

Published as a preprint.

F

C ONFIDENCE C OMPARISON

This section provides the full numerical results underlying Figure 4, which compares full-trace confidence with claimed answer-span confidence as predictors of correctness. For each agent and each sample across datasets, we compute AUROC using that agent’s own confidence scores against its correctness on the gold label (evaluation only; gold labels are not used in training). In the table, “CoT” denotes full-trace confidence ctrace , and “Box” denotes claimed answer-span confidence cspan . R1 and R2 refer to before and after one round of exchange, respectively, and ∆ = R2 − R1. Table 9 reports per-model, per-dataset AUROC for four model families across GSM8K, SVAMP, and MathQA. Across most settings, span confidence either starts higher or improves after debate, whereas full-trace confidence generally degrades. The average ∆ is negative for CoT (−0.07) and positive for Box (+0.04), consistent with our claim that answer-span confidence provides a more stable signal for ranking correctness in multi-agent debate. Table 9: Full-trace (CoT) versus claimed-answer-span (Box) AUROC before and after one debate round. ∆ = R2 − R1.

Model

Dataset

CoT R1

CoT R2

∆ CoT

Box R1

Box R2

∆ Box

Llama-3.2-1B-it

GSM8K SVAMP MathQA

0.60 0.55 0.64

0.50 0.56 0.54

-0.10 0.01 -0.10

0.63 0.59 0.63

0.72 0.77 0.65

0.10 0.18 0.02

Qwen-2.5-1.5B-it

GSM8K SVAMP MathQA

0.74 0.66 0.75

0.66 0.57 0.68

-0.08 -0.09 -0.07

0.71 0.68 0.73

0.80 0.76 0.77

0.09 0.08 0.04

Phi-4-mini-3.8B-it

GSM8K SVAMP MathQA

0.66 0.66 0.65

0.46 0.48 0.52

-0.19 -0.18 -0.13

0.56 0.66 0.62

0.52 0.68 0.63

-0.04 0.01 0.01

Gemma-3-4B-it

GSM8K SVAMP MathQA

0.46 0.42 0.75

0.45 0.41 0.83

-0.01 -0.01 0.09

0.64 0.68 0.36

0.57 0.54 0.52

-0.06 -0.14 0.17

Average

G

-0.07

0.04

T RAINING D ETAILS

This section summarizes the training and generation configuration for all reported results. All runs use 4-bit QLoRA on the attention projection modules {q, k, v, o}proj , with rank r=32, α=64, dropout 0.1, and a single post-training iteration from the instruction-tuned checkpoint. Unless otherwise noted, experiments use seed 10; ablations in Section 5 average over three seeds. Gold labels and unit-test outcomes are never provided to the oracle or to the standard training objectives; they are used only for evaluation. Preference construction. DPO pairs are formed as the Cartesian product G + (x) × G − (x) after discarding prompts with an empty preferred or dispreferred set. Unanimous debates (where G − (x) = ∅) may still contribute G + (x) to the SFT dataset. Preference datasets do not include the previousround peer context; debate itself uses peer context. Objective-specific settings. DPO and KTO are trained for 3 epochs with per-device batch size 8, learning rate 1 × 10−5 , β=0.1, and gradient accumulation 4, unless a specific run configuration overrides batch size or accumulation. GRPO uses a frozen match reward rx (y) = 1[A(y) ≡ â(x)], group size 4 (num generations), and the same LoRA recipe. SFT uses 3 epoch, batch size 8, and learning rate 1 × 10−5 . Table 10 summarizes the broader hyperparameter ranges explored across objectives. 21

Published as a preprint.

Generation and debate configuration. Unless otherwise specified, generation uses temperature 0.5, top-p 0.9, and up to 256 new tokens (Table 11). Multi-agent debate uses 5 agents and 2 rounds by default, with batch debate size 32–256 and synchronous updates (Table 12). Table 13 lists the base models and dataset splits. GSM8K and MathQA use 1500/500 train/test splits; SVAMP and MBPP use the respective subset of available test sets for evaluation only. Table 10: Training parameters. We report the hyperparameter ranges explored across objectives; the main results use r=32, α=64, and objective-specific learning rates and batch sizes given in the text. Parameter

SFT

Learning rate Weight decay Batch size Epochs Gradient accumulation steps Optimizer LoRA rank (r) LoRA α LoRA dropout LoRA target modules Entropy coefficient Beta (regularization) Number of generations per step Maximum sequence length Mixed precision Quantization

GRPO −5

DPO

KTO

1 × 10 1 × 10 1 × 10−5 −2 −2 1 × 10 1 × 10 1 × 10−2 8 8 8 3 3 3 4 4 4 AdamW AdamW AdamW 32 32 32 64 64 64 0.1 0.1 0.1 q proj, k proj, v proj, o proj 0.01 – – – 0.1 0.1 4 – – Model-dependent (2048–4096) bf16 4-bit (BitsAndBytesConfig)

1 × 10 1 × 10−2 8 3 4 AdamW 32 64 0.1 – – –

−5

−5

Table 11: Generation parameters used for all debate and evaluation runs unless otherwise noted. Parameter

Value

Temperature Top-p sampling Maximum new tokens Do sample Pad token ID

0.5 0.9 256 (GSM8K, SVAMP, MATHQA); 1024 (MBPP) True EOS token ID

Table 12: Multi-agent debate configuration. Default settings are N =5 agents and M =2 rounds; sweeps in Section 4.2 vary these. “Majority vote” indicates that majority consensus is computed for diagnostics and baselines, not as the training label.

H

Parameter

Value

Number of agents Debate rounds Batch debate size Majority vote computed Include debate context

5 2 32–256 True (diagnostics only) True

AVERAGE AGENT P ERFORMANCE A FTER P OST-T RAINING

Section 4 reports consensus accuracy for baselines and for models post-trained on two-round debate traces. In addition to consensus, we evaluate how post-training affects the average quality of individual agents within the committee. Table 14 reports mean per-agent accuracy after M =2 debate rounds, averaged over agents and test samples. 22

Published as a preprint.

Table 13: Base models and dataset splits. Parameter

Value

Base models Training datasets Test datasets Training size Test size Use quantization Use adapter mode Use scheduler

Qwen2.5-1.5B-it, Llama-3.2-1B-it, Phi-4-mini-3.8B-it, Gemma-3-4B-it, Llama-3.1-8B-it GSM8K, SVAMP, MathQA, MBPP GSM8K, MathQA, SVAMP, MBPP 1500 samples (GSM8K, MathQA); 700 samples (SVAMP); 374 samples (MBPP) 500 samples (GSM8K, MathQA); 300 samples (SVAMP); 200 samples (MBPP) 4-bit (BitsAndBytesConfig) True (LoRA) True

Columns are organized as follows: “Ensemble” and “Debate” are inference-only baselines (no post-training); “SFT” fine-tunes on the preferred completions G + (x); “DPO” and “KTO” use the oracle-derived preference labels, while “GRPO” uses the oracle-derived reward as described in Section 4. For each model and dataset, we highlight the best post-training objective in bold. Across most settings, preference-based objectives improve mean agent accuracy over SFT on the same traces, with DPO providing the largest gains in most cases. Table 14: Mean per-agent accuracy (%) after M =2 debate rounds. “Ensemble” and “Debate” are inference-only baselines; “SFT”, “DPO”, “KTO”, and “GRPO” denote post-training with the corresponding objective on the same debate traces. Column definitions match Table 1. Best posttraining objective per row is highlighted in bold. Model

Baselines

Dataset

Post-training

Ensemble

Debate

SFT

DPO

KTO

GRPO

Llama-3.2-1B-it

SVAMP GSM8K MathQA

4.5 4.2 1.8

5.1 5.4 1.7

8.6±0.85 5.8±0.21 3.8±1.98

18.3±1.48 13.2±2.26 3.5±0.49

15.3±2.69 9.85±2.47 2.6±0.42

10.2±0.14 5.4±1.41 2.3±0.58

Qwen-2.5-1.5B-it

SVAMP GSM8K MathQA

50.9 24.2 8.7

60.9 44.5 23.2

63.6±0.64 45.3±0.64 23.4±1.20

70.6±1.91 60.8±1.56 39.8±2.90

64.5±3.75 54.1±6.08 36.1±2.12

59.4±0.14 54.7±0.35 35.1±1.08

Phi-4-mini-3.8B-it

SVAMP GSM8K MathQA

88.2 76.5 18.2

87.4 82.2 18.2

88.5±0.00 81.4±0.49 29.8±1.91

88.7±0.21 82.6±2.05 46.8±4.88

88.1±0.28 82.0±0.99 37.6±2.83

87.8±0.42 83.2±1.63 31.0±2.83

Gemma-3-4B-it

SVAMP GSM8K MathQA

68.9 48.6 9.4

61.3 45.3 8.7

55.1±3.96 32.9±1.77 6.5±0.21

73.3±6.51 66.0±2.62 12.9±1.91

69.0±1.84 60.4±3.82 10.1±0.21

67.1±3.89 49.8±1.77 8.8±0.57

H.1

H OW P OST- TRAINING O BJECTIVES U SE D EBATE T RACES

Table 14 shows that DPO achieves the highest mean post-debate agent accuracy in 11 of 12 model– dataset pairs. The objectives differ in how they use the structure produced by the debate oracle, providing a possible explanation for this empirical pattern. Preference data. For each prompt x, the oracle partitions the final agent completions from the same M -round debate into preferred and dispreferred sets: G − (x) = {yi : ai ̸≡ â(x)}

G + (x) = {yi : ai ≡ â(x)},

where ai = A(yi ) is the parsed answer and â(x) is the span-confidence winning answer (Section 3). Unanimous debates have G − (x) = ∅ and are excluded from preference learning. For the remaining examples, the data contain same-x winner–loser comparisons among traces that participated in the same debate. The oracle therefore supplies the chosen/rejected structure required by DPO, with y + ∈ G + (x) and y − ∈ G − (x). The objective explicitly contrasts competing traces for the same question. 23

Published as a preprint.

   + πθ (y − |x)  θ (y |x) LDPO (x, y + , y − ) = − log σ β log ππref − log (y + |x) πref (y − |x) SFT: winner-only supervision. SFT maximizes log πϕ (y + |x) for y + ∈ G + (x) and does not include G − (x) in its objective. Thus, SFT provides no explicit contrast between competing traces and can provide smaller gains than preference-based objectives. KTO: independent desirability labels. KTO assigns desirable or undesirable labels to individual completions rather than explicitly contrasting competing completions for the same prompt. It therefore uses the oracle labels without directly encoding the pairwise relation between traces in G + (x) and G − (x). KTO nevertheless improves mean agent accuracy in many settings, indicating that the oracle-derived labels remain useful without explicit pairwise comparisons. GRPO: reward-based use of the oracle. GRPO does not directly optimize on the oracle-labeled debate completions. Instead, it samples new completions and assigns them the oracle-derived reward rx (y) = α(x) 1[A(y) ≡ â(x)] where α(x) is a reliability weight. Because GRPO computes relative advantages within each newly sampled group, groups whose members receive identical rewards provide no within-group ranking signal under the standard formulation. Thus, unlike DPO, GRPO does not directly contrast the oracle-selected and competing debate traces during optimization.

I

A BLATION DETAILS

I.1

MBPP EVALUATION

Official train split of 374 problems; 200 test problems subsampled from the official test set. Generation uses the task description only. The training oracle then clusters parseable programs by their public-test outputs and scores a cluster by how many public cases its members pass. Challenge assertions are withheld from that oracle. The reported correctness check executes every public assertion together with any challenge assertion, and it passes only when all of them succeed. The challenge list may be empty, in which case the check repeats the public assertions but remains all-or-nothing, whereas the oracle still counts how many public cases passed. Table 2 reports the polled overall public and hidden test cases passed for test-set by all the agents of the committee. I.2

O RACLE -P ROTOCOL RUNS

Other than the results shared in Table 3, we also evaluated two aggregation rules for the answer-span confidence oracle: confidence sum and confidence mean. All runs use Qwen-2.5-1.5B-it on a GSM8K splits. The committee uses N =5 agents and M =2 debate rounds, followed by one DPO post-training iteration. Table 15 reports accuracy after the initial response, after the exchange round, and after aggregation by the committee. Table 15: Oracle-protocol ablation on GSM8K using Qwen-2.5-1.5B-it, N =5 agents, and M =2 debate rounds. “Initial” denotes accuracy (%) before debate, “Exchange” denotes accuracy (%) after exchange, and “Consensus” denotes the accuracy (%) of the aggregated committee answer. Aggregation

Method

Initial

Exchange

Consensus

Ensemble

24.2

-

48.6

Sum

Debate DPO

24.0 35.4

44.5 59.7

66.0 71.6

Mean

Debate DPO

23.8 38.9

44.0 60.6

63.2 67.6

24

Published as a preprint.

(a) Agreement-rate count

(b) Correctness per agreement-rate

Figure 8: Comparison of confidence-sum and confidence-mean aggregation for answer-equivalence clusters on GSM8K with Qwen-2.5-1.5B-it and five agents. The agreement rate is the fraction of parseable agent answers assigned to the selected cluster. Left: frequency with which each aggregation rule selects an answer at each agreement rate. Right: correctness of the selected answer within each agreement-rate bin, measured against GSM8K reference answers for evaluation only. Confidence-sum aggregation retains the total confidence mass contributed by the committee, whereas confidence-mean aggregation normalizes away cluster size. Across agreement-rate bins, the sum rule selects more accurate oracle answers than the mean rule. For each prompt x, the oracle parses the agents’ final answers, clusters them by task-level equivalence, and assigns each cluster Ck a confidence mass X wk = ci , i∈Ck

where ci is the answer-span confidence of agent i. The selected cluster is k ⋆ = arg max(wk , nk ),

nk = |Ck |,

k

where nk provides a deterministic tie-break based on cluster size. Why we use confidence sum in Sec 4. We use confidence sum rather than mean because a cluster represents the committee’s accumulated evidence for an answer, not the confidence of a typical individual member. The sum therefore acts as a confidence-weighted vote: X wk = ci . i∈Ck

It allows several moderately confident agents to outweigh a single highly confident outlier, while still allowing two highly confident agents to outweigh several weakly confident agents. By contrast, mean confidence normalizes away cluster size. For example, a singleton with confidence 0.95 would defeat a four-agent cluster whose members each have confidence 0.80, because 0.95 > 0.80, despite the latter cluster containing substantially more independent agreement. Mean confidence can therefore treat a singleton as equivalent to a crowd and make cluster size contribute only indirectly through the tie-break. It is also sensitive to miscalibration: an overconfident incorrect answer can dominate under mean aggregation, whereas sum aggregation requires its confidence to outweigh the accumulated confidence of the competing cluster. Figure 8 visualizes the corresponding accuracy trends for the two aggregation rules. Table 15 alongside motivate the use of confidence mass as the default aggregation rule in AbCD. Both the debate and post-trained checkpoint consensus accuracy are better in aggregation by sum compared to mean. I.3

M AJORITY VOTING VS . S PAN -C ONFIDENCE S ELECTION

We compare majority voting (MV) with span-confidence selection on identical debate transcripts before any post-training. For each model–dataset pair, both rules operate on the same sampled 25

Published as a preprint.

Table 16: Comparison of majority voting (MV) and aggregate claimed-answer-span confidence on identical transcripts before post-training. Span and MV report overall selection accuracy (%). Disagree is the percentage of examples for which the two rules select different answer-equivalence clusters. Disagree: Span/MV reports the accuracy of the respective selected cluster on disagreement cases. Ties is the percentage of count-tie cases, with Ties: Span/MV reporting the corresponding selection accuracy. Gold labels are used only for evaluation. Model

Dataset

Span

MV

Disagree

Disagree: Span/MV

Ties

Ties: Span/MV

Llama-3.2-1B-it

SVAMP GSM8K MathQA

29.3 15.6 5.0

28.7 14.6 5.2

3.3 3.2 0.6

40.0 / 20.0 31.2 / 0.0 0.0 / 33.3

4.3 5.8 0.8

46.2 / 30.8 24.1 / 6.9 0.0 / 25.0

Qwen-2.5-1.5B-it

SVAMP GSM8K MathQA

79.7 66.0 45.2

79.7 65.0 42.2

3.0 5.8 8.8

22.2 / 22.2 31.0 / 13.8 47.7 / 13.6

7.0 12.8 12.8

33.3 / 33.3 31.2 / 23.4 43.8 / 25.0

Phi-4-mini-3.8B-it

SVAMP GSM8K MathQA

89.7 88.8 41.6

89.7 88.4 42.6

0.7 1.2 1.8

0.0 / 0.0 33.3 / 0.0 22.2 / 77.8

1.7 1.8 2.2

20.0 / 20.0 33.3 / 11.1 27.3 / 63.6

Gemma-3-4B-it

SVAMP GSM8K MathQA

75.7 60.6 16.6

76.0 60.6 16.6

0.3 0.0 0.0

0.0 / 100.0 0.0 / 0.0 0.0 / 0.0

0.3 0.4 0.0

0.0 / 100.0 50.0 / 50.0 0.0 / 0.0

SVAMP GSM8K MathQA

68.6 57.5 27.1

68.5 57.2 26.7

1.8 2.6 2.8

27.3 / 22.7 31.4 / 7.8 41.1 / 25.0

3.3 5.2 3.95

32.5 / 35.0 29.8 / 18.3 39.2 / 30.4

Average

responses and answer-equivalence clusters; gold labels are used only to evaluate the selected answer. Span denotes the accuracy of the cluster selected by aggregate claimed-answer-span confidence, while MV denotes the accuracy of the cluster selected by majority vote. Disagree is the fraction of examples for which the two rules select different clusters. For disagreement cases, we report the accuracy of the cluster selected by each rule. We additionally report the corresponding accuracy on count-tie cases. On GSM8K, the two rules disagree on only 2.6% of examples on average across the four models, and their overall selection accuracies are similar (57.5% for span confidence and 57.2% for MV). The difference is larger on cases where the rules disagree: the span-selected cluster is correct on 31.4% of such cases, compared with 7.8% for MV. Span confidence also achieves higher accuracy on count-tie cases (29.8% vs. 18.3%). Across SVAMP and MathQA, the same analysis shows that the relative advantage is dataset- and model-dependent. Overall, Span confidence is most useful precisely when majority voting is ambiguous or the two rules disagree. I.4

H ETEROGENEOUS C OMMITTEE

To examine whether debate benefits from architectural and scale diversity, we form a heterogeneous committee from five instruction-tuned models: L1 (Llama-3.2-1B-it), Q1 (Qwen-2.5-1.5B-it), Q2 (Qwen3-1.7B-it), P4 (Phi-4-mini-3.8B-it), and G4 (Gemma-3-4B-it). All agents use temperature 0.5, and the committee debates for M =2 rounds with N =5 agents on GSM8K. The ensemble, debate, and post-training results are reported in Table 17. DPO updates all five models using the shared preferred and dispreferred sets G + (x) and G − (x) constructed from answer-span confidence. In addition to consensus accuracy, we report the average accuracy of individual agents before and after debate and after one iteration with SFT, DPO, KTO and GRPO. The smaller models (L1, Q1, and Q2) generally benefit more from interaction with the larger committee members, whereas the gains for the larger models (P4 and G4) are less. This pattern suggests that heterogeneous debate can transfer useful information to weaker agents, although it does not uniformly improve every member.

J

L IMITATIONS OF THE E XPERIMENTAL P ROTOCOL

Our evaluation has several limitations that bound the conclusions supported by the experiments. Most main-grid cells use two post-training runs, while oracle and weighting ablations use three seeds 26

Published as a preprint.

Table 17: Accuracy (%) of a heterogeneous committee on GSM8K. The committee consists of L1 (Llama-3.2-1B-it), Q1 (Qwen-2.5-1.5B-it), Q2 (Qwen3-1.7B-it), P4 (Phi-4-mini-3.8B-it), and G4 (Gemma-3-4B-it). “Consensus” denotes accuracy of aggregate solution. “Avg. Agent” denotes mean per-agent performance. The ∆ row reports the per-agent change from Ensemble to DPO. Accuracy

Method Consensus

Avg. Agent

L1

Q1

Q2

P4

G4

Ensemble

78.2

30.9

3.6

26.0

1.4

76.8

46.8

Debate

79.2

36.8

16.6

57.6

3.2

71.6

35.2

SFT

79.8

37.4

17.8

54.2

4.2

75.6

35.4

DPO

87.0

68.2

71.0

75.8

41.8

81.2

71.4

KTO

86.6

59.2

52.0

71.6

22.6

83.6

66.0

GRPO

83.8

48.5

28.6

70.4

7.2

79.8

56.4

∆ (DPO − Ensemble)

8.8

37.3

67.4

49.8

40.4

4.4

24.6

where explicitly reported; these repetitions provide limited evidence about variance and statistical significance. We evaluate on subsets of the available training and test splits rather than the full datasets, limiting the precision with which the reported improvements characterize performance on the complete benchmarks. Post-training is restricted to a single iteration from each instruction-tuned checkpoint with QLoRA, so the experiments do not establish whether the observed gains persist under longer or repeated self-training. The maximum generation length of 256 tokens can truncate reasoning or final-answer spans, with truncation effects particularly visible for Llama-3.2-1B-it on GSM8K. Although the oracle uses no gold labels for supervision, it requires task-specific answer parsing and an equivalence relation, making the approach less directly applicable to outputs for which correctness cannot be parsed or reliably compared. The reported code-generation extension uses execution-based equivalence and therefore does not establish applicability to arbitrary open-ended generation without an executable correctness criterion. Answer-span confidence is computed from token probabilities and is not calibrated against correctness; the method therefore assumes that these probabilities provide a sufficiently informative within-committee ranking signal. Confidence scores are not recalibrated across heterogeneous architectures, and the heterogeneous-committee results show that raw token probabilities need not be directly comparable across models. The confidence oracle is evaluated against gold answers only retrospectively to measure selection quality; these labels are never available during oracle construction or post-training, so the experiments establish empirical utility rather than a guarantee that the oracle identifies the correct cluster. Finally, the experiments cover five instruction-tuned model families and four primary datasets. Near-ceiling GSM8K accuracy. Phi-4-mini-3.8B-it reaches 88.6% on GSM8K and 88.2% on SVAMP under independent sampling, and the best gain over debate is 0.7pp and 0.3pp (Table 1). On MathQA the same checkpoint moves from 41.6% after debate to 59.5% after DPO. The small GSM8K movement is the behavior of a model with little remaining headroom on that benchmark due to overfitting on the test set. Earlier work (Zhang et al., 2024) find that earlier Phi releases score higher on GSM8K than on GSM1k, a held-out set matched to GSM8K in difficulty, which is consistent with partial familiarity with GSM8K-style items. MathQA is the dataset on which this checkpoint’s base accuracy leaves room to observe the oracle.

K

L ARGER DATASET SPLITS

Section 4 trains on subsets of the available splits. Appendix G states the sizes: 1,500 training and 500 test examples for GSM8K and MathQA. The limitation is recorded in Appendix J: subset evaluation does not by itself characterize the full benchmark. This section repeats the locked protocol on larger splits of the same two datasets, for Qwen-2.5-1.5B-it. GSM8K uses the official training and test sets, 7,473 and 1,319 examples. MathQA uses 7,015 training and 2,985 test examples. The committee is unchanged: N =5, M =2, temperatures {0.01, 0.25, 0.5, 0.75, 1.0}, top-p=0.9, and at most 256 new 27

Published as a preprint.

tokens. Preferred and dispreferred traces are still assigned by claimed answer-span confidence. Gold answers are used only to compute the accuracies in Table 18. Table 18: Consensus accuracy (%) for Qwen-2.5-1.5B-it on the larger GSM8K and MathQA splits. Ensemble is inference-only independent sampling and Debate is inference-only debate, under the same span-confidence selection used in Table 1. Post-training is one QLoRA iteration on debate traces from the larger training split. The final column is the absolute improvement of the best post-training entry over inference-only debate, in percentage points. Each entry is a single run. Model

Baselines

Dataset

Qwen-2.5-1.5B-it

Post-training

Best ∆

Ensemble

Debate

SFT

DPO

KTO

46.6 20.1

65.6 45.1

65.4 44.0

72.1 54.7

72.9 45.8

GSM8K MathQA

7.3pp 9.6pp

On GSM8K, inference-only debate reaches 65.6% and the best post-training objective, KTO, reaches 72.9% (+7.3pp ). On MathQA, inference-only debate reaches 45.1% and DPO reaches 54.7% (+9.6pp ). Both gains point in the same direction as the subset results in Table 1, where the corresponding best improvements are +5.4pp on GSM8K and +8.5pp on MathQA. On the GSM8K subset the best entry is DPO; on this larger GSM8K split it is KTO, and DPO remains close (72.1%). On the larger MathQA split KTO falls to 45.8%, next to the debate baseline, whereas KTO is the best subset entry in Table 1. SFT stays within one percentage point of inference-only debate on both larger splits. It does show that moving from the 1,500-example subset to these larger splits does not remove the improvement of preference post-training over inference-only debate.

L

R EPEATED POST- TRAINING ITERATIONS

This section continues the iterations. At iteration 0 the policy is the instruction-tuned checkpoint. Each later iteration rebuilds the debate traces with the current policy, rebuilds G + (x) and G − (x) with the span-confidence oracle, and post-trains with DPO objective, same committee, and token limit as Section 4.1. We report three test-set quantities defined in Section 4.1: pre-debate agent accuracy (the mean accuracy of the independent samples), post-debate agent accuracy (the mean accuracy of the five final-round traces), and consensus accuracy (the accuracy of the span-selected cluster). Table 19: Test accuracy (%) across post-training iterations for Llama-3.2-1B-it and Qwen-2.5-1.5B-it. It-0 is the instruction-tuned checkpoint. Each later iteration rebuilds debate traces from the updated policy and applies one further post-training step with the span-confidence oracle. Pre-debate is mean agent accuracy before the exchange round; final-round average is mean agent accuracy after M =2; consensus is the accuracy of the span-selected cluster. Gold answers are used only for evaluation. Model

Pre-debate agent

Dataset

Post-debate agent

Consensus

It-0

It-1

It-2

It-3

It-4

It-0

It-1

It-2

It-3

It-4

It-0

It-1

It-2

It-3

It-4

Llama-3.2-1B-it

GSM8K MathQA

3.6 1.4

5.3 2.2

5.5 2.0

4.6 3.0

5.6 2.4

5.2 1.9

9.3 2.8

8.4 2.4

8.3 3.0

9.1 3.5

15.0 5.6

23.6 7.4

21.6 7.4

20.8 9.8

22.4 10.2

Qwen-2.5-1.5B-it

GSM8K MathQA

24.0 8.5

39.8 13.7

39.8 13.9

39.8 13.2

39.8 12.7

44.5 23.2

61.9 34.0

60.7 36.5

60.5 35.4

60.0 32.5

66.0 45.2

70.4 52.8

70.4 55.0

70.0 54.8

69.8 49.8

For Qwen-2.5-1.5B-it on GSM8K, consensus accuracy moves from 66.0% to 70.4% at iteration 1 and is 69.8% at iteration 4. Pre-debate agent accuracy on that pair moves from 24.0% to 39.8% at iteration 1 and does not move further. For Llama-3.2-1B-it on GSM8K, consensus accuracy peaks at iteration 1 (23.6%) and is lower at iterations 2 and 3. Llama-3.2-1B-it on MathQA is the exception among these four rows: consensus accuracy continues from 7.4% at iteration 1 to 10.2% at iteration 4, still on a low base. Qwen-2.5-1.5B-it on MathQA peaks at iteration 2 (55.0%) and falls to 49.8% at iteration 4, which is below its iteration-1 consensus accuracy. Repeated self-training therefore does not produce a consistent further gain after the first iteration on these splits. The single-iteration protocol in the main experiments is the setting in which the measured improvement appears. 28

Published as a preprint.

M

L ONGER GENERATION BUDGET

The main experiments generate at most 256 new tokens per agent turn. Appendix J notes that this cap can truncate the reasoning trace or the claimed answer span.A missing span makes cspan undefined, i removes the trace from equivalence clustering, and counts it as incorrect at evaluation. Part of what post-training changes may therefore be whether the model emits a parseable answer inside the cap. This section raises the cap to 512 new tokens and repeats the protocol for Qwen-2.5-1.5B-it, the model used for the committee sweeps in Section 4.2 with results noted in Table 20. The 256-token rows are the corresponding rows of Table 1. The 512-token GSM8K row is a single run. Table 20: Consensus accuracy (%) for Qwen-2.5-1.5B-it under two generation caps. The 256-token block includes the standard deviation over the two runs in Table 1. The 512-token block is a single run. The final column is the absolute improvement of the best post-training entry over inference-only debate, in percentage points. Model

Baselines

Dataset Ensemble

Debate

Post-training SFT

DPO

Best ∆

KTO

GRPO

69.2±1.13 53.7±1.27

68.9±0.14 51.1±1.56

5.4pp 8.5pp

75.4 63.2

70.6 59.2

4.6pp 5.4pp

256 new tokens Qwen-2.5-1.5B-it

GSM8K MathQA

44.5 21.6

66.0 45.2

66.3±0.71 46.0±2.26

71.4±0.28 53.0±1.13 512 new tokens

GSM8K MathQA

49.5 53.0

70.8 57.8

71.0 60.2

72.2 62.6

On GSM8K, raising the cap from 256 to 512 tokens raises inference-only debate from 66.0% to 70.8%, and raises independent-sampling consensus from 44.5% to 49.5%. The 256-token debate baseline is therefore partly a property of the generation budget, not only of the committee. Preference post-training still improves on the longer budget: KTO reaches 75.4%, which is +4.6pp over the 512-token debate baseline, compared with +5.4pp for the best objective under the 256-token cap. SFT remains within 0.2pp of debate at both caps on this dataset (66.3% and 71.0%). The longer budget changes which preference objective is best in the point estimate, from DPO at 256 tokens to KTO at 512 tokens.

N

C OMPARISON WITH MAJORITY- VOTE DEBATE POST- TRAINING

The closest post-training use of debate traces labels a completion by vote count. Multi-Agent Consensus Alignment treats traces whose final answer matches the majority answer as preferred and the dissenting traces as dispreferred, then applies SFT, DPO, KTO, or a majority-vote reward under GRPO (Samanta et al., 2025). AbCD uses the same preference objectives and the same requirement that a prompt contribute to DPO or KTO only when both a preferred and a dispreferred set exist. The difference is the rule that fills those sets. MACA uses kmaj = arg maxk nk . AbCD uses k ⋆ = arg maxk (wk , nk ), where wk is the sum of claimed answer-Span-confidences in the cluster (Section 3.3). Agreement rate does not choose k ⋆ . Table 21 compares majority-vote and span-confidence oracle pipelines on GSM8K and MathQA using Qwen-2.5-1.5B-it and Gemma-3-4B-it. For each model–dataset setting, both pipelines use the same source debate transcripts and the same objective-specific training hyperparameters. The only change from row to row is whether G + (x) is the majority cluster or the span-confidence cluster. Both labeling rules raise the best post-training consensus over the shared debate baseline. Span-confidence achieves higher best post-training consensus accuracy in all four model-dataset settings. For Qwen2.5-1.5B-it, the best scores increase from 70.8 to 71.4 on GSM8K and from 47.4 to 53.7 on MathQA. For Gemma-3-4B-it, they increase from 66.8 to 72.7 and from 20.2 to 21.5, respectively. The advantage is not uniform across training objectives. For example, majority-labeled KTO outperforms span-labeled KTO on Qwen GSM8K, while majority-labeled SFT outperforms span-labeled SFT on Gemma GSM8K. These results support span-confidence as a competitive oracle pipeline, while showing that its benefits depend on the model, dataset, and post-training objective. 29

Published as a preprint.

Table 21: Matched comparison of majority-vote labels and span-confidence labels. Both rows in a pair are trained on the same debate traces, with the same objective and the same QLoRA setup. Consensus is the accuracy (%) of the cluster selected at test time by the oracle named in the row. Gold answers are used only for evaluation. Model

Baselines

Dataset

Post-training

Ensemble

Debate

SFT

GSM8K MathQA

44.5 21.6

66.0 45.2

66.3±0.71 46.0±2.26

GSM8K MathQA

44.5 21.6

65.0 42.2

63.8 42.2

GSM8K MathQA

65.2 15.8

60.6 16.6

51.8±0.85 11.8±0.57

GSM8K MathQA

65.0 15.8

60.6 16.6

57.8 11.8

DPO

Best ∆

KTO

GRPO

69.2±1.13 53.7±1.27

68.9±0.14 51.1±1.56

5.4pp 8.5pp

70.8 45.6

67.0 44.4

5.8pp 5.2pp

68.8±2.26 17.3±0.99

62.9±0.14 16.3±1.56

12.1pp 4.9pp

66.4 20.2

63.2 14.8

6.2pp 3.6pp

Span-confidence Qwen-2.5-1.5B-it

71.4±0.28 53.0±1.13 Majority voting 68.6 47.4

Span-confidence Gemma-3-4B-it

72.7±1.27 21.5±2.12 Majority voting 66.8 19.0

Table 22: Factorial evaluation of Qwen-2.5-1.5B-it on GSM8K test (N =5). Rows are the training oracle. Columns are the test-time consensus, recomputed on the post-training debate. Entries are accuracy over 500 questions, with the correct count in parentheses. Span-confidence sums boxedanswer probabilities inside an answer cluster. Method

Training oracle

Majority

Span-confidence

Gap

DPO KTO GRPO

Span-confidence Span-confidence Span-confidence

71.2 (356) 68.4 (342) 68.6 (343)

71.6 (358) 70.0 (350) 69.0 (345)

+2 +8 +2

DPO KTO GRPO

Majority voting Majority voting Majority voting

68.6 (343) 70.8 (354) 67.0 (335)

70.0 (350) 71.6 (358) 68.6 (343)

+7 +4 +8

Factorial Evaluation. We further did the comparison in a factorial format on GSM8K with a temperature-diverse Qwen-2.5-1.5B-it committee. Each of DPO, KTO, and GRPO is trained once on majority-vote labels and once on span-confidence labels, and both rules are recomputed on the same post-training test debate. Accuracy in Table 22 is the number of correct answers over the expected 500 test questions, with a missing debate counted as incorrect. Span-confidence at test time is the more accurate readout in every cell. On one of the runs (quoted in Table 22) the best accuracy is 71.6%, reached by DPO trained on Span-confidence.

O

C OMPARISON WITH INDEPENDENT- SAMPLE POST- TRAINING

Section 4.4 fits SFT, DPO, KTO, and GRPO on traces collected after one independent-sample generation followed by one exchange round. In this section we replace that collection with only independent-samples of the same policy and keeps every other choice fixed. The model is Qwen-2.51.5B-it, the dataset is GSM8K, and both pipelines use N =5 temperature-diverse agents, and same training setup. The independent-sample pipeline stops after the initial generation. Test consensus is the accuracy of the span-selected cluster. Gold answers are used only for evaluation. Table 23 shows a higher consensus accuracy for every objective when the training traces include the exchange round. The largest gap is SFT, 66.3 against 47. On the best objective the gap is 7.0pp, from independent-sample DPO at 64.4% to exchange-round DPO at 71.4%. Independent-sample DPO remains 1.6pp below the inference-only debate consensus of 66.0%, so the exchange round is the 30

Published as a preprint.

Table 23: GSM8K consensus accuracy (%) for Qwen-2.5-1.5B-it. Ensemble and Debate are the instruction-tuned checkpoint before either training run. The one-exchange row is the GSM8K entry of Table 1. The independent-sample row is a single run. The gap row subtracts that run from the one-exchange mean. Training traces Independent samples One exchange round Gap

Ensemble

Debate

SFT

DPO

KTO

GRPO

44.5 44.5 –

– 66.0 –

47.0 66.3±0.71 +19.3

64.4 71.4±0.28 +7.0

61.2 69.2±1.13 +8.0

56.4 68.9±0.14 +12.5

step that puts post-training above that baseline on this model and dataset. The comparison is one model–dataset cell, and the independent-sample entries are a single run.

P

C ONFIDENCE CALIBRATION

Section 5 shows that token-probability confidence is not comparable across architectures. Answerspan confidence cspan in Eq. equation 2 is a length-normalized geometric mean of token probabilities. i It is an approximation of probability of correctness, and models with different tokenizers do Pnot place it on a common scale. In the heterogeneous committee the oracle sums those scores, wk = i∈Ik cspan . i A member whose raw scores sit higher can then dominate â(x). This section fits a separate map for each architecture and applies it before that sum, both for the inference-only oracle and for the preference partition used by DPO. We fit a Platt map (Platt et al., 1999) for each model m, σ(z) = 1 + e−z

c̃ = σ(am cspan + bm ),

−1

.

The coefficients (am , bm ) are an unpenalized logistic regression of gold correctness on the raw span score. The regression is fit on the heterogeneous committee itself: five members, M =2, one independent generation and one exchange, on GSM8K. The official training set is shuffled with the paper data seed. The paper training prefix is excluded. The next 2,000 training questions are the calibration pool. The uncalibrated committee uses the paper cap of 256 new tokens. Under that cap, Qwen3-1.7B-it (Q2) often ends before a parseable answer span, so its final-round trace contributes neither a cluster member nor a span score. For the calibration pool, the cap is raised to 512 new tokens. Every member then has room to finish the reasoning and emit the claimed answer before cspan is computed and before the Platt map is fit or applied. The score in the regression is the i final-round span confidence, after the exchange. Unparseable traces are dropped. The larger of the correct and incorrect classes is downsampled to the smaller class, and 80% of each class is used to fit (am , bm ). The remaining 20% is not used for the coefficients. Gold labels on this pool supervise only the logistic P regression. They are not preference targets. Because the fitted map is the weight inside wk = i c̃i , those labels do determine which cluster becomes G + (x). The same protocol is run once for each of the five architectures. At test time, and again inside the DPO run, each member uses its own (am , bm ). Table 24 compares the uncalibrated heterogeneous committee with a committee that uses the fitted maps. On consensus accuracy, Platt-Debate reaches 88.0%, against 79.2% for Raw-Debate (+8.8pp ). Platt-DPO reaches 89.6%, against 87.0% for Raw-DPO (+2.6pp ). The larger share of the consensus change is already present before post-training. DPO then adds 1.6pp of consensus accuracy on the calibrated committee, compared with 7.8pp on the raw committee, where the inference-only baseline is lower. The map rescales span scores after the traces have been generated, so the consensus column is the quantity it changes: a different weighting can select a different cluster from the same five answers. Read against its own debate baseline, Platt-DPO raises mean agent accuracy from 61.6% to 75.7%. Across the two DPO rows, L1 is the exception: Raw-DPO reaches 71.0% and Platt-DPO reaches 68.0%. The calibrated committee still leaves a wide gap between consensus and the weaker members. At Platt-Debate, consensus accuracy is 88.0% while L1 is at 27.6% and P4 is at 83.2%. Putting the five scores through separate logistic maps improves the selected answer on test split, and the members remain uneven. 31

Published as a preprint.

Table 24: GSM8K accuracy (%) for the heterogeneous committee, one member each from L1, Q1, Q2, P4, and G4. Ensemble, Raw-Debate, and Raw-DPO are the uncalibrated committee reported in Table 17. Platt-Debate and Platt-DPO apply the per-model map before the span-confidence sum. The map is fit on the 2,000 training questions; those questions are disjoint from the paper training set and from the test split. Consensus is the accuracy of the selected cluster. The other columns are per-member accuracy and their mean. The best entry in each column is bold. Gold answers are used to fit the maps and to compute accuracy. Accuracy

Method

Q

Consensus

Avg. Agent

L1

Q1

Q2

P4

G4

Ensemble

78.2

Raw-Debate Raw-DPO

79.2 87.0

30.9

3.6

26.0

1.4

76.8

46.8

36.8 68.2

16.6 71.0

57.6 75.8

3.2 41.8

71.6 81.2

35.2 71.4

Platt-Debate Platt-DPO

88.0 89.6

61.6 75.7

27.6 68.0

70.2 80.8

47.8 58.2

83.2 85.0

79.0 86.4

Z ERO -S HOT S INGLE -AGENT R EFERENCE P ERFORMANCE

To contextualize the performance of the local debate agents, we evaluate larger and closed-source models in a zero-shot, single-agent setting. All runs use temperature 0 and a maximum completion budget of 256 tokens, matching the generation budget per response used by the local debate agents. Table 25 separates responses into three categories: correct, when the extracted answer matches the reference answer; wrong, when an answer is extracted but does not match; and unparseable, when no valid final answer can be extracted. Unparseable responses are counted as incorrect when computing accuracy. This breakdown distinguishes answer errors from failures to produce a parseable final answer within the prescribed budget. The results characterize performance under this constrained generation protocol, rather than performance with unrestricted generation. Table 25: Zero-shot single-agent performance of models on the evaluation splits at temperature 0 with a maximum completion budget of 256 tokens. Values are percentages of evaluated examples. Accuracy is the percentage of correct responses; wrong and unparseable responses are both counted as incorrect. Model

R

Dataset

Incorrect (%)

Correct (%)

Unparseable

Wrong

GPT-4o

SVAMP GSM8K MathQA

1.7 17.8 51.4

5.0 4.0 4.6

93.3 78.2 44.0

Qwen-3.8-27B

SVAMP GSM8K MathQA

55.3 87.0 81.0

13.7 8.6 0.2

31.0 14.4 18.8

J UDGE S UPERVISION

We also studied the impact of a closed-source Judge model used for supervision for the oracle selection. We used GPT-4o model as judge for the multi-agent tempertaure diverse committee of five Qwen2.5-1.5B-it models, debating for two rounds. GPT-4o does not debate. After the second round it reads the question and the parsed final answer, and returns a probability that the answer is correct. It does not see the gold label or the other agents. Equivalent answers form a cluster, and the cluster score is the sum of those probabilities. A failed judge call contributes zero. The winning cluster supplies the DPO labels: its members are preferred, and the other parsed answers are dispreferred. Majority vote and the gold cluster are stored on the same generations. The gold cluster is the correct answer when any debater produced it. 32

Published as a preprint.

Table 26: Test accuracy of the selected answer before training. Each entry is correct questions out of 500. The gold cluster is the best answer already present in the debate. Dataset

Majority

GPT-4o judge

Gold cluster

GSM8K MathQA

65.6 41.0

71.6 47.2

72.8 49.6

Table 27: GSM8K test accuracy before and after DPO. DPO trains on the GPT-4o split for 3 epochs. The after-DPO majority column is a vote over that trained committee, not a model trained on majority labels. Selection

Debate

After DPO

Majority GPT-4o judge Gold cluster

65.6 71.6 72.8

71.2 78.2 78.6

On GSM8K the judge matches the majority cluster on 451 of 500 questions. On the other 49, the judge is correct and majority is wrong 31 times, and the reverse happens once. On MathQA the two rules match on 348 questions. The judge is the only correct rule 35 times, and majority 4 times. The remaining gap to the gold cluster is 6 GSM8K questions and 12 MathQA questions.

S

T EMPERATURE AND CONFIDENCE ACROSS MODELS

The same five temperatures, 0.01, 0.25, 0.5, 0.75, and 1.0, are used for every model in the pre-training debates. Full-trace confidence is the exponentiated mean token log-probability of the final reply. Boxed confidence is the same quantity on the tokens inside the final \boxed{} span. Table 28 reports both scores on the training questions stored for math model–dataset pairs. The arrow is temperature 0.01 to temperature 1.0. Trace uses every stored training reply. Boxed uses the replies that contain a boxed answer. Table 28: Training-split confidence from temperature 0.01 to 1.0. Model Llama-3.2-1B Qwen-2.5-1.5B Phi-4-mini Gemma-3-4B Llama-3.2-1B Qwen-2.5-1.5B Phi-4-mini Gemma-3-4B

Dataset

Trace mean

Trace median

Boxed mean

Boxed median

GSM8K

0.926 → 0.807 0.864 → 0.694 0.902 → 0.798 0.880 → 0.857

0.931 → 0.819 0.873 → 0.718 0.938 → 0.802 0.882 → 0.857

0.989 → 0.966 0.989 → 0.968 0.999 → 0.996 1.000 → 1.000

1.000 → 0.999 0.999 → 0.999 1.000 → 1.000 1.000 → 1.000

SVAMP

0.924 → 0.787 0.836 → 0.659 0.850 → 0.741 0.835 → 0.803

0.931 → 0.804 0.844 → 0.678 0.854 → 0.747 0.825 → 0.799

0.993 → 0.984 0.985 → 0.974 0.997 → 0.997 1.000 → 1.000

1.000 → 1.000 0.999 → 0.999 1.000 → 1.000 1.000 → 1.000

Full-trace confidence falls with temperature in every row. The drop in the mean runs from 0.023 (Gemma-3-4B, GSM8K) to 0.177 (Qwen-2.5-1.5B, SVAMP). On the debates that record whether each reply is correct, an incorrect reply at temperature 0.01 still outscores a correct reply at temperature 1.0: Qwen-2.5-1.5B on GSM8K (0.858 against 0.722) and SVAMP (0.826 against 0.674), Llama-3.2-1B on GSM8K (0.926 against 0.790), SVAMP (0.924 against 0.774), and Llama-3.1-8B on GSM8K (0.944 against 0.902). Raw trace scores from different temperatures are therefore not on a common scale. On GSM8K and SVAMP the boxed median stays at 0.999 or 1.000 for every model. The boxed mean moves by at most 0.023. Agents can be compared on that score without a per-temperature calibration. In each of those rows the trace mean falls by more than the boxed mean. The flat boxed scale is a property of the short numerical span. Table 29 expands the Qwen-2.5-1.5B-it GSM8K training debate across all five temperatures. 33

Published as a preprint.

Table 29: Qwen-2.5-1.5B-it on the 1,500 GSM8K training questions. Trace uses all 1,500 replies at each temperature. Temperature 0.01

0.25

0.50

0.75

1.0

Boxed mean Boxed median

0.989 0.999

0.989 0.999

0.984 0.999

0.982 0.999

0.968 0.999

Trace mean Trace median

0.864 0.873

0.855 0.865

0.834 0.844

0.790 0.801

0.694 0.718

Figure 9: Pass rate of the winning cluster on MBPP. Majority voting (blue) is a plurality over publicoutput clusters. Execution confidence (green) keeps the cluster with the most public cases passed. Hidden success-rate is the all-or-nothing check over public and challenge assertions. Public pass-rate pools the visible cases of the selected program. Debate is the shared pre-training trace. DPO is three epochs on that selector’s label.

From 0.01 to 1.0 the trace mean drops by 0.170. The colder reply has the higher trace score on 95.6% of the 1,500 questions, and the median gap is 0.152. Correct replies alone fall from 0.869 to 0.722. The boxed median gap on the 660 questions where both endpoints box is 0.0001.

T

A NALYSIS OF EXECUTION CONFIDENCE

Following the MBPP setup above in Appendix C, each candidate program is scored on the public assertions. Programs that emit the same sequence of public outputs form one cluster. Majority voting selects the largest cluster. Execution confidence selects the cluster whose members pass the most public cases, summing those passes across the agents that share the cluster. The reported correctness check is separate from that selection. It executes every public assertion together with any challenge assertion, and it passes only when all of them succeed (Hidden success-rate). The challenge list may be empty, in which case the check repeats the public assertions but remains all-or-nothing, whereas the oracle still counts how many public cases passed. A non-empty challenge assertion can reject a program that passed every public case. The hidden check is stricter than the public pass-rate even when the two sets of assertions coincide. We train Llama-3.1-8B with DPO for three epochs on shared five-agent debate. One run uses the majority cluster as the training label. The other uses the execution-confidence cluster. Figure 9 reports the hidden-check pass rate and the public-case pass rate of the winning cluster on the 200-question test set, before training and after DPO. The debate bars are the two selectors applied to the same traces. The DPO bars are each trained model scored by the selector it was trained on. On the shared debate, execution confidence passes the hidden check on 42.0% of questions (84/200), against 39.0% (78/200) for the vote, and passes 53.3% of public cases, against 48.6%. After DPO 34

Published as a preprint.

Table 30: RaR-Medicine test split success rate and hit rate. Each cell is the mean fraction of checklist items satisfied.

Oracle Majority Voting Hit Confidence

Full success rate

Rubric hit rate

Debate

DPO

Debate

DPO

4.2 5.6

3.2 6.0

30.5 41.9

29.3 42.8

the same ordering holds, and the public-case gap widens. The confidence-trained model, scored by execution confidence, reaches 43.5% hidden (87/200) and 55.1% public. The vote-trained model, scored by majority voting, reaches 40.0% hidden (80/200) and 49.1% public. Training on the execution-confidence label therefore improves the cluster that the oracle keeps, on both the public count that defines the label and the stricter check used for evaluation. In this execution setting, selecting and training on how many public tests a cluster passes is the stronger rule.

U

TASKS WITHOUT A DIRECT ANSWER - EQUIVALENCE RELATION

The main experiments use tasks on which a final answer is a number, a choice letter, or a program, so two traces can be tested for equivalence and then counted. RaR-Medicine (Gunjal et al., 2026) does not have that kind of answer. This subsection records the dataset, the checklist, the judge that turns a free-form reply of Qwen-2.5-1.5B-it into a checklist pattern, and the labeling rule that replaces a vote over those patterns. Post-training is otherwise the same pipeline as in the main experiments: a sampling-diverse committee, followed by DPO on the preferred cluster. Dataset. We use the medicine split of Rubrics as Rewards (Gunjal et al., 2026), released as ScaleAI/RaR-Medicine on Huggingface. Each example is a clinical question, a reference answer, and a checklist. The reference answer is not a training target and is not shown to the committee. The sweep uses 1500 training questions and 500 test questions, shuffled with the same data seed as the other datasets. Agents are asked only to answer the question completely and generate and open-ended reply. Checklist. A checklist is an ordered list of pass/fail criteria produced by the autorubric pipeline of Gunjal et al. (2026). Each item has a title, a description of what the response must do, and an integer weight from the release. An item contributes one hit when the judge marks it satisfied, and zero otherwise. For a question with K criteria, a response is summarized by a bit string b ∈ {0, 1}K . PK The hit count is c(b) = k=1 bk , and the hit rate (confidence) is c(b)/K. Judgment. Equivalence is not computed from the text of the answer. A frozen judge (GPT-4o at temperature 0) reads the question, the full assistant response, and the criteria in order, and returns JSON of the form {"satisfied": [true, false, ...]} with one boolean per criterion. That boolean vector is the bit string b. Two responses are the same cluster exactly when their bit strings match, so a cluster is one way of meeting the checklist, not a paraphrase class. The same judge is used to score the committee during debate and to score a sample at evaluation time. The reference answer is not an input to this judgment. Evaluation. The primary number is the weighted hit rate on the 500 test questions. For each question the judge scores the selected answer: satisfied positive criteria add their weight, satisfied negative criteria subtract, and the total is divided by the sum of the positive weights. A missing or unparseable answer counts as zero. Debate is the answer selected from the round-2 committee, before training. Each DPO entry is the policy trained on that row’s debate labels, then judged on its own test generations under the same selection rule. Table 30 reports that comparison. The rubric success rate is the fraction of test questions on which every positive criterion is satisfied and every negative criterion is avoided. A method can raise the average hit rate while still rarely clearing the whole checklist, so that rate is reported separately and is not the selection rule used for training. 35

Published as a preprint.

Table 31: Vote score and hit-count score on one question with K=10 criteria. The vote selects the repeated partial checklist. The hit-count score selects the checklist that meets more criteria. Cluster

Agents n

Criteria met c

Score

Vote (n) Vote (n)

3 1

2 8

3 1

Hit count (n · c) Hit count (n · c)

3 1

2 8

6 8

Why a vote is the wrong label. Inside a cluster every member has the same bit string, so every member has the same hit count c. A vote assigns the cluster the score n, the number of agents that landed on it. The rubric oracle assigns each member the weight c and, under the same sum reduction used in the main experiments, assigns the cluster the score n · c. The vote therefore keeps only how often a pattern was produced. The hit count also keeps how much of the checklist that pattern covers. A pattern that meets more criteria can outrank a pattern that more agents happened to repeat. Table 31 is the definition, not a measured outcome: three agents share a checklist with 2 of 10 items satisfied, and one agent meets 8 of 10. The vote selects the first cluster. The hit-count score selects the second, because 1 · 8 > 3 · 2. On these questions the patterns are usually all different, so a vote is a tie among singletons and the tie break is string order of the bit strings, which is unrelated to how many criteria are met. The preferred cluster under AbCD is the one with the largest hit-count (confidence) score. Completions whose judged bit string matches that cluster are the preferred side of the preference pair. The others are the rejected side. DPO then use those pairs exactly as on the verifiable tasks.

V

Q UALITATIVE ANALYSIS

The quantitative results report accuracies aggregated over prompts. This section shows the same quantities on individual transcripts, so that the oracle rule in Section 3.3 can be checked against the committee that produced it. Each block is one selected prompt from an evaluation run. The gold answer is printed only to interpret the transcript. It is not an input to debate, to equivalence clustering, or to post-training. A trace is unparseable when S(yi ) = ∅, and it is placed in G − (x) rather than in a cluster, as in Section 3.3. For Qwen-2.5-1.5B-it model on GSM8K dataset, every prompt on which majority voting and span confidence select different clusters has two clusters sharing the largest nk . The blocks in Sections V.2 and V.3 are prompts of that kind. V.1

F ULL RESPONSES

The following blocks record question asked to the committee members and their full responses. Round 1 is the independent answer. Round 2 is the answer written after the other agents’ round-1 answers have been shown. One prompt ends on the gold answer. The other ends on a shared incorrect answer after agents who had the gold answer adopt a calculation already present in the committee. Convergence onto the gold answer. Katelyn sees 50 fairies, 25 more join, and 30 leave, so 45 remain. The independent answers are 55, 45, 15, 85, and 45. After the exchange every agent answers 45. Agent 5’s second trace says the revision follows the solutions given by the other agents. Question asked to committee Run. model: Qwen2.5-1.5B-it dataset / split: GSM8K / test example id: 392 seed: 10 committee: N=5, M=2, temperatures {0.01, 0.25, 0.5, 0.75, 1.0} Question shown to the agents. While playing with her friends in their school playground,

36

Published as a preprint.

Katelyn saw 50 fairies flying above the nearby forest. After about twenty minutes, one of her friends saw half as many fairies as Katelyn saw come from the east and join the fairies that were there. Ten minutes later, 30 fairies flew away. How many fairies are remaining? Gold answer (evaluation only). 45 Response: agent 1, round 1 To solve this problem, we need to follow these steps: 1. Determine how many fairies joined from the east after twenty minutes. • Katelyn initially saw 50 fairies. • Her friend saw half as many fairies as Katelyn did, so they saw 50 = 25 2 fairies from the east. 2. Determine how many fairies remained after 30 fairies flew away. • Ten minutes later, 30 fairies flew away. So, the total number of fairies remaining is: 50 − 25 + 30 = 55 Therefore, the final answer is 55 . Response: agent 2, round 1 To solve this problem, let’s break it down into steps: 1. Initial Count: Katelyn initially sees 50 fairies. 2. Friends’ Observation: • Her friend sees half as many fairies as Katelyn did. • Calculation: 50 = 25 2 fairies. 3. Fairies Joining: • Thirty more fairies fly away after ten minutes. • Calculation: 50 + 25 − 30 = 45. So, the total number of fairies remaining is 45. Final Answer: 45 . Response: agent 3, round 1 Let’s break down the problem step by step: 1. Initial Number of Fairies: Katelyn initially saw 50 fairies. 2. Fairies Joining from East:

37

Published as a preprint.

One friend saw half as many fairies as Katelyn did, which is 50 = 25. 2 So, these 25 fairies joined from the east. The total now becomes: • Katelyn: 50 • Friends: 25 3. Fairies Leaving: Ten minutes later, 30 fairies flew away. This leaves us with: • Katelyn: 50 − 30 = 20 • Friends: 25 − 30 = −5 Since we cannot have negative fairies, this means all 25 fairies left. 4. Final Calculation: Total remaining fairies: 20 + (−5) = 15. Therefore, the final answer is 15 .

Response: agent 4, round 1 To solve this problem, let’s break it down into steps: 1. Initial Count: At first, Katelyn sees 50 fairies. 2. Friends’ Observation: Her friend sees half as many fairies as she does. Therefore, 50 = 25 2 fairies. 3. Fairy Migration: Ten minutes after seeing her friend, 30 fairies fly away. 4. Final Calculation: The total number of fairies initially seen is 50. Her friend then sees 25 more. In addition to these, 30 fairies leave after ten minutes. Therefore, 50 + 25 − 30 = 85. This gives us 85, but since only 30 were actually left behind (the 25 seen plus the others minus those that flew away), ... The final answer is: 85 .

Response: agent 5, round 1 1. Initial Count: Initially, Katelyn sees 50 fairies flying above the forest. 2. Friends’ Observation: • Her friend sees half as many fairies as Katelyn did. • This means she observes: 50 = 25 fairies. 2 3. Fairies Joining: Ten minutes after joining, 30 more fairies fly away. 4. Final Calculation:

38

Published as a preprint.

• Total fairies before these events: 50 + 25 = 75. • Fairies joined plus those who stayed minus those that left: 75 − 30 = 45. Therefore, the number of remaining fairies is 45 . Response: agent 1, round 2 1. Initial Count: Katelyn initially sees 50 fairies. 2. Friends’ Observation: Her friend sees half as many fairies as Katelyn did, which is 50 = 25 2 fairies. 3. Fairy Migration: Ten minutes after seeing her friend, 30 fairies fly away. 4. Final Calculation: Add the initial count of 50 fairies, the 25 fairies observed by her friend, and subtract the 30 fairies that flew away: 50 + 25 − 30 = 45. Therefore, the final answer is 45 . Response: agent 2, round 2 1. Initial Count: Katelyn initially sees 50 fairies. 2. Friends’ Observation: Her friend sees half as many fairies as Katelyn did. This means she sees 50 = 25 2 fairies. 3. Fairies Joining: These 25 fairies join the 50 fairies already present. 4. Total Before Departure: 50 + 25 = 75 fairies. 5. Departure: Ten minutes later, 30 fairies fly away. 6. Remaining Fairies: 75 − 30 = 45. Therefore, the final answer is 45 . Response: agent 3, round 2 To find out how many fairies are remaining, follow these steps: 1. Initial Count: Katelyn initially sees 50 fairies. 2. Friends’ Observation: Her friend sees half as many fairies as Katelyn did. This means she observes: 50 = 25 fairies. 2 3. Fairy Migration: Ten minutes after joining, 30 fairies fly away.

39

Published as a preprint.

4. Final Calculation: • Total fairies before these events: 50 + 25 = 75. • Fairies joined plus those who stayed minus those that left: 75 − 30 = 45. Therefore, the number of remaining fairies is 45 .

Response: agent 4, round 2 • Initial count: Katelyn sees 50 fairies. • Friend observation: Katelyn’s friend sees half as many fairies as Katelyn did, which is 25 fairies. These 25 fairies join from the east. • Remaining fairies after the friend joins: Add 25 (friend’s fairies) to 50 (initial Katelyn’s fairies), giving a total of 75 fairies. • Fairies leaving: Ten minutes later, 30 fairies fly away. • Final count: Subtract 30 from the current total of 75 fairies, resulting in 45 fairies remaining. Thus, the final answer is 45 .

Response: agent 5, round 2 Here are the revised step-by-step solutions provided by other agents: 1. Katelyn initially sees 50 fairies. 2. Her friend sees half as many fairies as Katelyn, which is 50 = 25 2 fairies. 3. These 25 fairies join with the 50 already present, totaling 50 + 25 = 75 fairies. 4. After ten minutes, 30 fairies fly away. 5. Adding the newly arrived fairies to those still in place: 75 − 30 = 45. The final answer is 45 .

Convergence onto a shared incorrect answer. Violetta has 20 dollars and needs five crayons at 2 dollars each, so the change is 10 dollars. Two independent traces reach the decoding limit before a boxed answer. Of the three that finish, one answers 0 and two answer 10. The answer 0 comes from buying ten crayons and spending the whole 20 dollars. After the exchange, four agents answer 0, including both agents whose independent answer was 10. One agent moves from 0 to 10. 40

Published as a preprint.

Question asked to committee Run. model: Qwen2.5-1.5B-it dataset / split: GSM8K / test example id: 237 seed: 10 committee: N=5, M=2, temperatures {0.01, 0.25, 0.5, 0.75, 1.0} Question shown to the agents. Violetta wants to buy new crayons. She needs them in 5 different colors and prepared \$20 for this purchase. One crayon costs \$2. How much change will she get? Gold answer (evaluation only). 10 Response: agent 1, round 1 To solve this problem, let’s break it down into steps: 1. Determine the total cost of crayons Violetta can afford with her $20: • Each crayon costs $2. • Violetta has $20. 2. Calculate how many crayons she can buy: To find out how many crayons she can buy, divide the total amount of money ($20) by the price per crayon ($2). Number of crayons =

$20 = 10 crayons $2 per crayon

3. Calculate the total cost of these crayons: Since each crayon costs $2 and she buys 10 crayons, Total cost = 10 crayons × $2 = $20 4. Calculate the change she gets back from her initial $20: Subtract the total cost of the crayons from the amount she initially had. Change = $2 Response: agent 2, round 1 To solve this problem, let’s break it down into steps: 1. Determine the total cost of crayons Violetta can afford: • Violetta has $20. • Each crayon costs $2. 2. Calculate how many crayons she can buy with her money: To find out how many crayons she can buy, divide her total amount ($20) by the price per crayon ($2). Number of crayons =

41

$20 = 10 $2

Published as a preprint.

3. Verify that she can afford all five colors: • Since she can buy 10 crayons and there are only 5 different colors, she can indeed afford all the colors she wants. 4. Calculate the total cost of these crayons: Multiply the number of crayons (10) by the cost per crayon ($2). Total cost = 10 × $2 = $20 5. Determine the change she will receive: Subtract the total cost from the amount she initially had ($20). Response: agent 3, round 1 1. Determine how many crayons Violetta can buy with her $20 if each crayon costs $2. To find out how many crayons she can afford, divide her total money ($20) by the cost per crayon ($2). $20 = 10 crayons $2 2. Calculate the total amount spent on the crayons. Since each crayon costs $2 and Violetta buys 10 crayons, the total cost is: Number of crayons =

Total cost = 10 × $2 = $20 3. Subtract the total cost from the initial amount Violetta has to determine her remaining balance. If Violetta started with $20 and spends all $20 on crayons, she will have no change left. Change = $20 − ($20) Change = $0 Therefore, Violetta will get 0 as her change.

Response: agent 4, round 1 To determine how much change Violetta will receive after her purchase, follow these steps: 1. Calculate the total cost of the crayons Violetta buys: • Each crayon costs $2. • Violetta is buying 5 different colors. 2 dollars × 5 = 10 dollars So, the total amount spent on crayons is $10. 2. Subtract the total amount spent from the initial amount of money Violetta has: $20 − $10 = $10 Thus, Violetta will have $10 left as change after making her purchase. Final answer: 10

42

Published as a preprint.

Response: agent 5, round 1 1. Calculate the total cost of buying one crayon: • One crayon costs $2. 2. Determine the total cost if Violetta buys five crayons: • Total = Number of crayons × Cost per crayon. • Total = 5 crayons × $2/crayon. • Total = $10. 3. Subtract the total cost from the amount Violetta initially prepared: • Amount Violetta prepared = $20. • Change received = Initial amount - Total cost. • Change received = $20 - $10. 4. Therefore, the amount Violetta gets as change is: 10 Response: agent 1, round 2 • Calculate the total cost of crayons Violetta can afford with her $20 if each crayon costs $2. – Divide her total money ($20) by the cost per crayon ($2): Number of crayons =

$20 = 10 crayons $2

• Verify that she can afford all five colors. – Since she can buy 10 crayons and there are only 5 different colors, she can indeed afford all the colors she wants. • Calculate the total amount spent on the crayons. – Multiply the number of crayons (10) by the cost per crayon ($2): Total cost = 10 × $2 = $20 • Determine the change she will receive. – Subtract the total cost from the initial amount Violetta has: Change = $20 − $20 = $0 Final Answer: Violetta will get 0 as her change. Response: agent 2, round 2 • Calculate the total cost of crayons Violetta can afford with her $20. – Each crayon costs $2. – Violetta has $20. • Determine how many crayons she can buy. – Divide the total amount of money ($20) by the price per crayon ($2). Number of crayons = • Calculate the total cost of these crayons.

43

$20 = 10 crayons $2 per crayon

Published as a preprint.

– Since each crayon costs $2 and Violetta buys 10 crayons, Total cost = 10 crayons × $2 = $20 • Calculate the change she will receive. – Subtract the total cost from the amount Violetta initially has. Change = $20 − $20 = $0 Therefore, Violetta will get 0 as her change. Response: agent 3, round 2 1. Calculate the total cost of one crayon: Each crayon costs $2. 2. Determine the total cost for 5 different colors: 5 crayons × $2/crayon = $10 3. Subtract the total cost from the amount Violetta initially prepared: $20 − $10 = $10 Final Answer: 10 Response: agent 4, round 2 Step-by-Step Reasoning Summary 1. Determine the total cost of crayons Violetta can afford with her $20: • Each crayon costs $2. • Violetta has $20. • The total number of crayons Violetta can buy is calculated as: Number of crayons =

$20 = 10 crayons $2 per crayon

2. Calculate the total cost of these crayons: Since each crayon costs $2 and Violetta buys 10 crayons, Total cost = 10 crayons × $2 = $20 3. Calculate the change she gets back from her initial $20: Subtract the total cost of the crayons from the amount she initially had. Change = $20 − $20 = $0 Final Answer 0 Response: agent 5, round 2 • Determine the total cost of crayons Violetta can afford: Divide the total money by the price per crayon. Number of crayons =

44

$20 = 10 crayons $2 per crayon

Published as a preprint.

• Calculate the total cost: Multiply the number of crayons by the price per crayon. Total cost = 10 crayons × $2 = $20 • Calculate the change: Subtract the total cost from the initial amount Violetta had. Change = $20 − $20 = $0 The answer is 0 .

V.2

C OUNT TIE RESOLVED BY SPAN CONFIDENCE

Majority voting selects arg maxk nk . When two or more clusters share that maximum, vote count does not name a winner, and the span-confidence oracle uses wk , with cluster size only as a secondary key. The prompt below is one on which the largest nk is shared and the two rules select different clusters. Table 4 summarizes how often this happens; the block below is one such example. Question asked to committee Run. model: Qwen2.5-1.5B-it dataset / split: GSM8K / test example id: 19 seed: 10 committee: N=5, M=2, temperatures {0.01, 0.25, 0.5, 0.75, 1.0} Question shown to the agents. In Mr. Roper’s class of 30 students, 20% of the class are football players. Out of the remaining class, 25% of the students are cheerleaders or part of band. These 3 groups of students will need to leave early today to travel to an away game. How many students are leaving early? Gold answer (evaluation only). 12 Count tie response analysis Final-round traces. agent 1 2 3 4 5

temp 0.01 0.25 0.50 0.75 1.00

parsed answer a_i 24 12 12 6 6

c_iˆspan 0.9906 0.9996 0.9987 0.2426 0.8191

parseable yes yes yes yes yes

Equivalence clusters. cluster A B C

representative 24 12 6

n_k 1 2 2

w_k 0.9906 1.9983 1.0617

matches gold no yes no

Selection. majority cluster: C (B and C share n_k=2; the string tie-break selects 6) span-confidence cluster k*: B tie broken by w_k: yes

45

Published as a preprint.

alpha(x): 0.40 winning cluster correct: yes

V.3

M AJORITY VOTING AND SPAN CONFIDENCE DISAGREE

On most prompts the two rules select the same cluster (Table 16). The block below is a prompt on which they do not. The two prompts below are one case in which the span-selected cluster is correct and one case in which the majority-selected cluster is correct. Question asked to committee Run. model: Qwen2.5-1.5B-it dataset / split: GSM8K / test example id: 308 seed: 10 committee: N=5, M=2, temperatures {0.01, 0.25, 0.5, 0.75, 1.0} Question shown to the agents. Jack decides to visit a museum 150 miles from home. He drives 75 mph there and back. He spends 6 hours at the museum. How long is he gone from home? Gold answer (evaluation only). 10 Disagreement: span-selected cluster is correct Final-round traces. agent 1 2 3 4 5

temp 0.01 0.25 0.50 0.75 1.00

parsed answer a_i 480 10 8 8 10

c_iˆspan 0.9963 0.9992 0.9992 0.1139 0.9993

parseable yes yes yes yes yes

Equivalence clusters. cluster A B C

representative 480 10 8

n_k 1 2 2

w_k 0.9963 1.9985 1.1131

matches gold no yes no

Selection. majority cluster / correct?: C (8) / no (B and C share n_k=2; the string tie-break selects 8) span-confidence cluster / correct?: B (10) / yes alpha(x): 0.40 Question asked to committee Run. model: Qwen2.5-1.5B-it dataset / split: GSM8K / test example id: 328 seed: 10 committee: N=5, M=2, temperatures {0.01, 0.25, 0.5, 0.75, 1.0}

46

Published as a preprint.

Question shown to the agents. Janet had 22 green pens and 10 yellow pens. Then she bought 6 bags of blue pens and 2 bags of red pens. There were 9 pens in each bag of blue and 6 pens in each bag of red. How many pens does Janet have now? Gold answer (evaluation only). 98 Disagreement: majority-selected cluster is correct Final-round traces. agent 1 2 3 4 5

temp 0.01 0.25 0.50 0.75 1.00

parsed answer a_i 98 98 100 100 unparseable

c_iˆspan 0.9988 0.9986 0.9996 0.9998 na

parseable yes yes yes yes no

Equivalence clusters. cluster A B

representative 98 100

n_k 2 2

w_k 1.9975 1.9995

matches gold yes no

Selection. majority cluster / correct?: A (98) / yes (A and B share n_k=2; the string tie-break selects 98) span-confidence cluster / correct?: B (100) / no alpha(x): 0.40

V.4

L ATER ROUNDS RAISE AGREEMENT WITHOUT RAISING CORRECTNESS

Section 4.2 fixes M =2 because further exchange rounds increase agreement while consensus accuracy stops improving and can fall. The block below follows one prompt across rounds. The rows below are the parsed answer and cspan after the independent generation and after each exchange, together i with the selected cluster and whether it matches the gold answer. Question asked to committee Run. model: Qwen2.5-1.5B-it dataset / split: GSM8K / test example id: 324 seed: 10 committee: N=5 clones, temperature 0.5 rounds recorded: 1 (independent), 2, 3, 4, 5 Question shown to the agents. Christina is planning a birthday party and needs .75 gift bags per invited guest, because 1/4 of attendees don’t show up. She invited 16 friends. Gift bags are $2 each. How much will she spend? Gold answer (evaluation only). 24

47

Published as a preprint.

Agreement rises, selected answer stays incorrect Committee by round. round 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5

agent 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

parsed answer a_i unparseable 24 18 24 18 18 18 18 24 18 18 24 18 18 18 18 18 18 18 18 18 18 18 18 18

c_iˆspan na 0.9994 0.9995 0.9981 0.9995 0.9996 0.9996 0.9992 0.9997 0.9975 0.9991 0.9990 0.9998 0.9998 0.9998 0.9994 0.9998 0.9997 0.9998 0.9998 0.9997 0.9998 0.9998 0.9999 0.9998

parseable no yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes yes

Selected cluster by round. The selected answer is the span-confidence cluster. round n_max w_of_selected alpha(x) consensus 1 2 1.9990 0.40 18 2 4 3.9959 0.80 18 3 4 3.9986 0.80 18 4 5 4.9985 1.00 18 5 5 4.9990 1.00 18 V.5

correct no no no no no

U NANIMOUS INCORRECT DEBATE

Preference objectives (DPO & KTO) use a prompt only when both G + (x) and G − (x) are nonempty (Section 3.4). A debate in which every parseable agent names the same incorrect answer has G − (x) = ∅ once unparseable traces are set aside, or a G − (x) that contains only unparseable traces. Such a prompt does not supply a competing-cluster comparison. The block documents one prompt of this kind. It is the case in which agreement is high and the selected answer is wrong, which is the regime where α(x) is large and correctness is not. Question asked to committee Run. model: Qwen2.5-1.5B-it dataset / split: GSM8K / test example id: 153 seed: 10 committee: N=5, M=2, temperatures {0.01, 0.25, 0.5, 0.75, 1.0} Question shown to the agents. Sandy wants to lose as much weight as Joey does but needs

48

Published as a preprint.

4 weeks to lose the same amount of weight that Joey loses in a single week. If Joey loses 8 pounds in 4 weeks, how many weeks will it take Sandy to lose the same amount of weight? Gold answer (evaluation only). 16

Unanimous incorrect committee Final-round traces. agent 1 2 3 4 5

temp 0.01 0.25 0.50 0.75 1.00

parsed answer a_i 4 4 4 4 4

c_iˆspan 0.9988 0.9985 0.9988 0.9978 0.9991

parseable yes yes yes yes yes

Preference partition. â(x) is the span-confidence answer. Every parseable trace names it, so there is no competing cluster. hat a(x): 4 |G+(x)|: 5 |G-(x)|: 0 unparseable traces in G-(x): 0 included in D_pref: no alpha(x): 1.00 selected answer correct: no

V.6

T HE SAME PROMPT BEFORE AND AFTER POST- TRAINING

Consensus accuracy after post-training is still computed by running the committee and applying the oracle. The block below uses one test prompt under the instruction-tuned checkpoint and under the post-trained checkpoint, with the same N , M , temperatures, and decoding limit. Record pre-debate agent answers as well as final-round answers, matching the three metrics in Section 4.1. Question asked to committee Run. model: Qwen2.5-1.5B-it dataset / split: GSM8K / test example id: 430 seed: 10 objective: DPO checkpoint: instruction-tuned vs. one QLoRA iteration committee: N=5, M=2, temperatures {0.01, 0.25, 0.5, 0.75, 1.0} Question shown to the agents. Tara bought 8 packs of 5 canvas bags for $4 each. She painted them and sold them at a craft fair for $8 each. How much profit did she earn on her bags? Gold answer (evaluation only). 160

49

Published as a preprint.

Before and after post-training Instruction-tuned checkpoint. round agent temp parsed answer a_i 1 1 0.01 32 1 2 0.25 32 1 3 0.50 608 1 4 0.75 8 1 5 1.00 32 2 1 0.01 32 2 2 0.25 32 2 3 0.50 32 2 4 0.75 32 2 5 1.00 608 selected cluster: 32 consensus correct: no mean agent correct (final round): 0/5

c_iˆspan 0.9981 0.9979 0.9975 0.9965 0.9980 0.9996 0.9991 0.9998 0.9998 0.9992

Post-trained checkpoint. round agent temp parsed answer a_i 1 1 0.01 32 1 2 0.25 288 1 3 0.50 32 1 4 0.75 288 1 5 1.00 160 2 1 0.01 160 2 2 0.25 160 2 3 0.50 160 2 4 0.75 unparseable 2 5 1.00 32 selected cluster: 160 consensus correct: yes mean agent correct (final round): 3/5

V.7

c_iˆspan 0.9983 0.9964 0.9993 0.9981 0.9973 0.9995 0.9997 0.9998 na 0.9991

H ETEROGENEOUS COMMITTEE

In the heterogeneous committee of Section 5, span confidence is the geometric mean of token probabilities from different tokenizers and different scales. Those scores are summed as if they were on one scale. The first prompt is one on which the winning cluster is carried by one model’s confidence mass. The second prompt is one on which a lower-confidence model has the correct answer. Per-agent accuracies for this committee are in Table 17. Question asked to committee Run. dataset / split: GSM8K / test example id: 288 seed: 10 members: L1=Llama-3.2-1B, Q1=Qwen2.5-1.5B, Q2=Qwen-3-1.7B, P4=Phi-4-mini, G4=Gemma-3-4B temperature 0.5, M=2 Question shown to the agents. Jason was told he could earn $3.00 for doing his laundry, $1.50 for cleaning his room, $0.75 for taking the trash to the curb each week and $0.50 for emptying the dishwasher. In a two week period, Jason emptied the

50

Published as a preprint.

dishwasher 6 times, did his laundry once, took the trash out twice and cleaned his room once. How much money did Jason earn? Gold answer (evaluation only). 9 Heterogeneous scores are not on one scale Final-round traces. member L1 Q1 Q2 P4 G4

parsed answer a_i 8.00 unparseable unparseable 9.00 unparseable

c_iˆspan 0.9773 na na 0.9056 na

correct no no no yes no

Clusters and selection. Both parseable answers have nk = 1. The winning mass is L1’s score alone, and it exceeds P4’s score on the gold answer. cluster A B

members L1 P4

representative 8.00 9.00

n_k 1 1

w_k 0.9773 0.9056

matches gold no yes

selected by span confidence: A member with the largest c_iˆspan: L1 that member correct: no Question asked to committee Run. dataset / split: GSM8K / test example id: 106 seed: 10 members: L1=Llama-3.2-1B, Q1=Qwen3-1.7B, Q2=Qwen2.5-1.5B, P4=Phi-4-mini, G4=Gemma-3-4B temperature 0.5, M=2 Question shown to the agents. Jackson is planting tulips. He can fit 6 red tulips in a row and 8 blue tulips in a row. If Jackson buys 36 red tulips and 24 blue tulips, how many rows of flowers will he plant? Gold answer (evaluation only). 9 Lower-confidence member holds the correct answer Final-round traces. member L1 Q1 Q2 P4 G4

parsed answer a_i 3 unparseable 3 9 unparseable

c_iˆspan 0.9996 na 0.9903 0.9967 na

51

correct no no no yes no

Published as a preprint.

Clusters and selection. P4 has the gold answer and a lower span confidence than L1. The wrong cluster wins because the two wrong scores sum to more than P4’s score. cluster members representative n_k A Q2, L1 3 2 B P4 9 1 selected by span confidence: A member with the largest c_iˆspan: L1 that member correct: no V.8

w_k 1.9899 0.9967

matches gold no yes

MBPP EXAMPLE

The following example shows a MBPP task trace generated using Llama-3.1-8B model committee. The committee must compute sum of positive integers down to the last positive term. There are three public tests given unit-testing the program. Before DPO, three agents agree on a program that starts at n − 2 and passes none of these tests. Execution confidence keeps the single program that starts at n and passes all three. After DPO on the respective oracle chosen the final responses are shown below. A majority of three again passes 0/3 tests, while the confidence selection passes 3/3. The vote follows the larger cluster. The execution-confidence oracle follows the cluster that passes the most public tests. Execution confidence versus majority voting on MBPP Run. model: Llama-3.1-8B dataset / split: MBPP / test example id: 162 seed: 10 objective: DPO on execution confidence (public cases passed) checkpoint: instruction-tuned debate vs. one DPO iteration committee: N=5, M=2 temperatures {0.01, 0.25, 0.5, 0.75, 1.0} Question shown to the agents. Write a function to calculate the sum of the positive integers of n+(n-2)+(n-4)... (until n-x =< 0). Implement the function sum_series. Public tests (oracle and reported pass rate). sum_series(6) == 12 sum_series(10) == 30 sum_series(9) == 25 Before DPO. Majority, 3 agents, public 0/3. def sum_series(n): total = 0 x = 2 while n - x >= 0: total += n - x x += 2 return total Before DPO. Execution confidence, 1 agent, public 3/3. def sum_series(n): total = 0 x = 0 while n - x >= 0: if n - x > 0:

52

Published as a preprint.

total += n - x x += 2 return total After DPO. Majority, 3 agents, public 0/3. def sum_series(n): total = 0 x = 2 while n - x >= 0: total += n - x x += 2 n -= 2 return total After DPO. Execution confidence, 1 agent, public 3/3. def sum_series(n): total = 0 x = 0 while n - x >= 0: total += n - x x += 2 return total

53

Related documents

• Not Your Diary, Just Your Topic: Grading the Wrong-User Control#1598914
• CoMoFlow: Controllable Protein Conformational Ensemble Generation via Explicit Collective-Mode Modeling#1598916
• CID: Measuring Cognitive Work Through Semantic State Change#1598917
• Multimodal Training with Knowledge Traction for Video Anomaly Detection#1598918
• PreForceVision: Physics-Guided Optical Estimation of Bolt Preload from Bolt-Head Deformation#1598919