Where Is the Truth When AI Agents Disagree? Dependence-Aware Belief Propagation for Unsupervised Truth Discovery (Extended Version)

Where Is the Truth When AI Agents Disagree? Dependence-Aware Belief Propagation for Unsupervised Truth Discovery [Extended Version] Laure Berti-Equille

arXiv:2611.04805v1 [cs.AI] 3 Nov 2026

IRD, ESPACE-DEV Montpellier, France laure.berti@ird.fr Abstract— LLM agents often give conflicting answers to the same question, and many deployed agents are variants of one base model, so their errors are correlated. We study truth discovery among such agents: inferring the true answer and each agent’s reliability without labels. Classical truth-discovery and crowdsourcing estimators reward an agreeing bloc and can fall below majority vote. We propose BP-TD, which adds peragent decorrelation weights to mean-field variational inference on a one-coin Dawid–Skene model, so that agents sharing a base model count together as about one vote. Its first iteration is exactly a weighted vote; later iterations estimate agent reliability. With 13 local agents on MMLU, BP-TD beats majority vote without any labels (0.716 vs. 0.624 on 160 questions; 0.707 vs. 0.674 on 10,000) and ranks agents by their true accuracy, which classical estimators invert. The gain depends on the pool: over 2,852 sub-pools of four benchmarks, decorrelation helps when the largest correlated bloc is weaker than the other agents and hurts when it is the strongest, and we found no label-free statistic that tells the two cases apart. Labels change the picture quickly: with 20 labelled questions, selecting the single best agent is more accurate than every aggregator we tested on MMLU, ARC and TruthfulQA. Index Terms—truth discovery, LLM agents, source dependence, variational inference, data quality, ensembling

I. I NTRODUCTION Ask the same factual question to several large language model (LLM) agents and you will often get several different answers: on a multiple-choice item, one model answers A, another B, a third abstains. As LLM agents are wired together into pipelines, debates, and mixtures, this disagreement is no longer a curiosity but an operational problem—a system must commit to one answer, ideally the true one, and ideally without paying for ground-truth labels at every step. Deciding which answer to trust is exactly truth discovery (TD): a dataquality problem with two decades of theory for conflicting Web sources and crowd workers [1], [2], [3], [4]. Whether that theory transfers to AI agents, and when it does, has not been characterized. The gap. Classical TD rests on a structural assumption: sources are conditionally independent given the truth, so that agreement is evidence of correctness and reliability can be estimated from how often a source sides with the emerging consensus. This assumption is routinely violated by AI © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Accepted at the 2026 IEEE International Conference on Big Data (BigData 2026).

agents for a reason specific to them: deployed agents are frequently variants of the same base model—prompt personas, self-consistency samples, or retrieval wrappers around one LLM—so their errors are strongly correlated. Reliabilityby-agreement then rewards a correlated bloc that may be confidently and unanimously wrong, and the standard TD machinery inverts: it can rank the agreeing-but-wrong majority as most reliable. Our approach. We propose BP-TD, which places a peragent decorrelation weight wa inside mean-field variational inference on a one-coin Dawid–Skene model. Agents that share a base model, or that agree with each other far more than chance, share one vote. The weights come either from provenance or from answer co-occurrence, with no labels. The first iteration of BP-TD is exactly a wa -weighted plurality vote; later iterations add per-agent reliability estimates. We test the method on 13 local agents and four benchmarks, against classical, dependence-aware and concurrent LLM-aggregation methods, and we vary the agent pool to find out when decorrelation helps and when it hurts. Contributions. C1. Why classical TD fails on LLM agents. Reliability estimators that reward agreement rank a correlated bloc of weak agents as the most reliable and fall below majority vote. The same implementations work on independent agents, so the failure comes from the dependence, not from the estimators. C2. A decorrelation method and what each part does. BPTD discounts dependence groups found from provenance or from answer co-occurrence. We show that its first iteration is a weighted vote, separate the effect of the weights from that of iteration and of the threshold choice, and give label-free settings of its one hyperparameter. C3. When agent truth discovery helps. Decorrelation helps when the largest correlated bloc is weaker than the other agents and hurts when it is the strongest; this holds across thousands of sub-pools of four benchmarks and in a controlled stress test. The gain on MMLU survives at 10,000 questions. Twenty labels are worth more than any aggregation we tested, so BP-TD is useful when labels cannot be had and the pool’s largest bloc is not its strongest.

The code, the cached agent answers with provenance, and every script behind the numbers below are released. II. R ELATED W ORK Classical truth discovery. A large body of work jointly estimates source trust and claim belief by iterating between the two. Iterative fixed-point methods include TruthFinder [1] and the Investment and Sums/Hubs–Authorities schemes [6], while CRH and CATD [3] cast the problem as joint optimization with confidence-aware weighting; earlier fixed-point corroboration [18], correlation-aware fusion [31], and knowledge fusion over machine-extracted triples [19] broaden the family (surveyed in [21], [22], [23]), and controlled evaluations find no estimator dominates across source regimes [20], [24]. These methods are unsupervised and effective when sources err independently, but they provide no account of why a method should be expected to help on a given workload—the regime question we make central. Crowdsourcing aggregation. Latent-variable models treat each source as a noisy annotator and infer a confusion structure: Dawid–Skene estimates per-annotator error rates by EM [4], and LTM extends this with a generative Bayesian treatment of truth and reliability [5]; further variants estimate annotator ability and item difficulty (GLAD [27]), infer confusability by minimax entropy [28], or aggregate by message passing—order-optimal in [26], variational/BP in [29]— with a uniform benchmark of seventeen such methods [25]. Belief propagation over the Dawid–Skene model is itself established and provably optimal under independent annotators [13]; our contribution is the dependence-aware decorrelation layer that keeps inference valid when annotators—here, agents—are correlated, which that line does not address. Our messagepassing update reduces to a one-coin Dawid–Skene model when agent weights are uniform; the contribution is not the latent-variable machinery but the dependence-aware weighting layered on top of it. Source dependence and copying. A key assumption of the above is conditional independence of sources, which copying violates. Dong et al. [2] detect copying among Web sources from shared false values and down-weight copied evidence, with global detection of complex copying networks across many sources [30]; in crowdsourcing, spectral and coupledconfusion methods handle correlated annotators beyond independent Dawid–Skene [12]. Large-scale evidence confirms the phenomenon directly for LLMs: higher-capability and sameprovider models make correlated errors even across architectures [14]. Our setting is the agent analogue: agents that share a base model exhibit correlated errors by construction. We adapt this intuition with family- and agreement-based discounting so that a correlated bloc counts as roughly one effective vote, and we show that a shared-error signal—the direct copy-detection analogue—degrades when the dependent bloc is the majority. LLM ensembling, self-consistency, and debate. Recent work aggregates multiple LLM samples or agents— self-consistency [8], multi-agent debate [10], mixture-ofagents style ensembling [9], learned pairwise-ranking fu-

sion [32], sampling-and-voting at scale [33], juries of diverse judges [34], and LLM-as-a-judge evaluation [35], often leaning on self-reported model confidence [11]—generally assuming a cooperative crowd whose members add complementary signal. Reference-free aggregation of LLM judges has been studied directly [15], and recent analyses question the diversity assumption: mixing distinct models can underperform ensembling a single strong one [16], and correlated judges cap panel gains regardless of the aggregation rule [17]. Concurrent work aggregates dependent LLM judges with second-order answer statistics (ISP and OW [36]), with Ising models of judge dependence [37], or by separating quality from shared confounders (CARE [38]). The Ising aggregator is defined for binary labels only, so we use a mean-field surrogate. What is new here. BP-TD builds on one-coin Dawid–Skene inference [4], [29], discounting of dependent sources [2], [31], and grouping of correlated voters (our ClusterVote baseline). What differs with BP-TD is the dependence signal it uses: provenance or raw answer co-occurrence, never the current truth estimate, which we show is circular when the dependent bloc is the majority. It differs from cluster voting in keeping soft per-agent weights inside a reliability model rather than collapsing each group to one hard vote; on MMLU, with identical groups, this is worth +0.043 (Sec. VI-B). The main empirical finding—decorrelation helps when the largest correlated bloc is weaker than the other agents and hurts when it is the strongest—is not specific to BP-TD: any rule that discounts a group of agreeing agents gives up that group’s votes, which costs accuracy exactly when the group is right. Signed graphs. Balance theory and signed bipartite graph neural networks [7] exploit the sign of relations for representation learning. We use signed agreement for the opposite purpose: not to embed agents, but to detect correlated blocs and decorrelate their votes during inference, keeping the aggregator unsupervised and label-blind. III. P ROBLEM F ORMULATION We are given a set of agents A = {a1 , . . . , am } (here, LLM-based answerers) responding to a set of questions Q = {q1 , . . . , qn }, where each question q has a known, finite answer set Vq of size |Vq |. For each (a, q) pair, agent a either asserts a single value yaq ∈ Vq or abstains; let Aq ⊆ A denote the agents that answer q. We seek, with no gold labels at inference time, (i) the latent truth t⋆q ∈ Vq for every question, and (ii) a scalar reliability ra ∈ [0, 1] for every agent, used as a relative ranking score, not a calibrated probability of correctness (BP-TD recovers the ranking, not the magnitude). We represent the observations as a tripartite agent–question– value graph (Fig. 1): nodes are agents, questions, and candidate values, and an edge (a, q, v) exists iff yaq = v. This makes the inference structure explicit—belief flows from agents to questions (which value is likely true) and back from questions to agents (how reliable each agent is). The dependence problem. Classical truth discovery and the Dawid–Skene model [4] assume agents are conditionally independent given the truth, so that agreement is evidence

Agents

Claims

a1

correlated bloc wa ↓: ≈1 vote

a2

Values

v1A wa

q1 v1B

a3 q2

v2A

a4 v2B a5

Fig. 1. The agent–question–value graph and the two messages of BP-TD. An edge links an agent to the value it asserts on a question. Truth update (agents→questions, Eq. 1): each agent sends its weighted vote wa E log ra to the value it chose. Reliability update (questions→agents, Eqs. 2–3): each agent receives the current belief in the values it asserted, which updates its Beta reliability. Dashed box: a dependence group (agents on one base model); each member’s vote is scaled by wa , so the group counts as about one vote.

of correctness. This assumption is violated for LLM ensembles: agents that share a base model, a prompt template, or a retrieval corpus produce correlated errors, and a single such bloc can manufacture a spurious majority. Our goal is therefore aggregation that down-weights dependent agents so a correlated bloc contributes roughly one effective vote. IV. M ETHOD : BP-TD BP-TD combines (i) variational message passing on a onecoin Dawid–Skene factor graph with (ii) a per-agent decorrelation weight wa that encodes how much an agent’s vote should count given its dependence on others. On terminology: the inference is mean-field variational message passing (coordinate ascent on the evidence lower bound), not sum-product belief propagation. We keep the name BP-TD because the updates are messages passed along the edges of the agent–question– value graph of Fig. 1, and loopy BP and mean-field updates share this structure; the estimator itself is the mean-field one below. A. Variational message passing We adopt a one-coin observation model: with probability ra agent a emits the true value, and otherwise emits a value drawn uniformly from the |Vq |−1 wrong alternatives. Placing a Beta(α0 , β0 ) prior on each ra , mean-field variational inference factorizes the posterior over the question beliefs bq (·) and the reliabilities ra , and alternates two closed-form updates:  P log bq (v) ∝ a∈Aq wa 1[yaq =v] E log ra
 + 1[yaq ̸=v] E log(1−ra ) − log(|Vq |−1) , (1) P αa = α0 + q bq (yaq ), (2)
P βa = β0 + q 1 − bq (yaq ) . (3)

Intuition. The truth update (Eq. 1) is a weighted vote: each answering agent adds wa E log ra to its asserted value and a smaller wa E log(1−ra ) to the |Vq |−1 competitors, so reliable agents push their answer harder while wa shrinks the influence of dependent agents. The reliability update (Eqs. 2–3) is conjugate Beta accounting: αa accumulates the belief mass P on a’s chosen value ( q bq (yaq )) and βa the complementary mass, so E[ra ] = αa /(αa + βa ) rises for agents that agree with confident consensus. The two updates form a soft, EMstyle loop; setting wa ≡ 1 removes decorrelation and recovers one-coin Dawid–Skene / SSTF-VI. One pass is a weighted vote. At initialization every agent has the same prior reliability, so E log ra and E log(1−ra ) are the same P constants for all a. Eq. (1) then gives log bq (v) = cq + κ a∈Aq wa 1[yaq =v] with κ = E log r − E log(1−r) + log(|Vq |−1) > 0, so the first truth update ranks values exactly as the wa -weighted plurality vote. A single pass (T =1) therefore predicts the weighted vote, and with wa ≡ 1 it predicts majority vote. Reliability estimation affects the predicted answers only from the second iteration on. We report both the one-pass weighted vote and the iterated estimator, and state the iteration count of every BP-TD row. Why wa tempers the likelihood. The weights are not part of the generative model; they enter Eq. (1) as exponents on each agent’s likelihood term. This is a composite (power) likelihood: when k agents are near-copies, the product of their k likelihood terms counts the same evidence k times, and raising each to the power 1/k restores roughly one copy’s worth of evidence. The update for bq stays a closed-form coordinate-ascent step for this tempered objective, and the reliability update (Eqs. 2–3) is unchanged. A generative model of dependence (e.g. a latent bloc variable) would replace this heuristic correction; we leave that to future work. B. Dependence-aware decorrelation The decorrelation weight wa ∈ (0, 1] is the core contribution: it discounts an agent in proportion to the size of the dependent bloc it belongs to, so that a k-agent bloc contributes P w a∈bloc a ≈ 1 effective vote rather than k. We compare three ways to define the blocs, ordered by how much external information they require: • family (structural). Agents sharing a base-model family form one cluster; wa = 1/|family(a)|. Requires only provenance (which base model produced each agent), no observed answers. Cheapest, label-free, fixed. • cooccur (empirical, truth-free). Discount each agent by its number of above-threshold neighbors in the signed cooccurrence graph—how often two agents give the same answer, regardless of whether it is correct: wa = 1/ 1 +
|{b : agree(a, b) ≥ θ}| , so an agent that agrees with k−1 others counts as ≈1/k of a vote. Uses the answer matrix but not any estimate of truth. Adapts to behavior without trusting the current belief. • shared error (truth-aware). Following copy detection on shared false values [2], cluster agents that agree on the same likely-wrong answers, where “wrong” is read

off the current beliefs bq ; recomputed each iteration. Most principled in principle—correlated mistakes are the strongest dependence signal. Why shared_error fails when family/cooccur succeed. shared_error defines dependence relative to the inferred truth: two agents are dependent if they share errors. But when the dependent bloc is the majority, it defines the belief bq —the bloc’s answers look correct, so its members appear to share no errors and escape discounting. The measure is thus circular precisely in the regime it is meant to fix: it trusts the consensus to decide who is correlated, but the correlation is what corrupts the consensus. By contrast, family (provenance) and cooccur (raw agreement) never consult the bloc-defined truth, so they detect and discount a dominant bloc regardless of whether it is currently winning the vote—which is why they are robust (Sec. VI). The signed-agreement view underlying cooccur connects to balance theory on signed graphs [7], used here to decorrelate rather than to embed. C. Algorithm and complexity P Algorithm 1 summarizes BP-TD. Let |E| = q |Aq | ≤ nm be the number of asserted answers. Each iteration costs O(|E|· |V¯ |) for the truth update (Eq. 1) plus O(|E|) for the reliability update (Eqs. 2–3), and the dependence step costs O(m2 ) for cooccur/shared_error (pairwise agent agreement) or O(m) for family. Per iteration this is O(n m |V¯ | + m2 ), linear in the data and quadratic only in the (small) agent count; total cost is that times the number of iterations T . Algorithm 1 BP-TD: dependence-aware variational truth discovery 1: Input: answers {yaq }, value sets {Vq }, prior (α0 , β0 ),

dependence mode ∈ {family, cooccur, shared_error}, iterations T , optional seed set S 2: Init: E[ra ] ← α0 /(α0 + β0 ); bq ← uniform on Vq ▷ clamp bq for q ∈ S to seeds 3: Compute wa ▷ family/cooccur precomputed once; shared_error recomputed in the loop 4: for τ = 1 to T do 5: for all questions q ∈ / S do 6: update bq (v) via Eq. (1) and normalize 7: end for 8: for all agents a do 9: update αa , βa via Eq. (2); E[ra ] ← αa /(αa + βa ) 10: end for 11: smooth E[ra ] over the signed agreement graph (weight λ) 12: if mode = shared_error then 13: recompute wa from current bq 14: end if 15: end for 16: Output: t⋆q = arg maxv bq (v) and reliabilities E[ra ]

D. Variants One-pass runs a single truth+reliability update (T =1), the cheapest setting, and is exactly the wa -weighted vote shown above. Full loopy iterates Eqs. (1)–(2) to convergence, which is the default and the variant behind every “BP-TD” row unless

a row is marked one-pass. Two further steps run in each iteraold tion. The truth messages are damped (bq ← 0.7 bnew q +0.3 bq ). The reliability means are then smoothed over the signed agreement graph W (agreement minus disagreement rate P per agent P pair): r̄a ← 12 + (1−λ)(r̄a − 12 ) + λ b Wab (r̄b − 12 )/ b |Wab | with λ = 0.4, so an agent that agrees with reliable agents and disagrees with unreliable ones gains reliability. Iteration stops when the largest belief change is below 10−4 (at most 100 iterations); the mean iteration count per dataset and variant is given with each result table (7–21 for most rows; 93 on SciQ when θ=0.2 is selected, where beliefs converge slowly near unanimity). Removing the smoothing (λ = 0) changes fullset MMLU accuracy by less than 0.013 (Sec. VI-B). Semisupervised seeding clamps a fraction S of questions to known answers, which propagates through the reliability update to calibrate the rest; we evaluate only on Q \ S. V. E XPERIMENTAL S ETUP We evaluate BP-TD under a deliberately conservative, leakage-free protocol and report every operating point— including those unfavorable to our method. a) Agent pool: The pool has 13 answering agents built on 5 base models, all served locally via Ollama (no external API calls). Table I lists every agent with its base model and accuracy. The pool combines (i) six direct-prompting agents (qwen2.5:14b, qwen2.5:7b, llama3.2, mistral, phi3.5, gemma2; the two qwen2.5 sizes share one base model); (ii) two self-consistency agents (five sampled chains, majority over chains); (iii) three persona agents; and (iv) two retrieval-augmented (RAG) agents. Terminology: an agent is one answering configuration; its base model (or family) is the underlying LLM; a dependence group is a set of agents that BP-TD discounts together, defined either by base model (family) or by answer agreement (cooccur). Three further configurations were run and are excluded because they never produced a parseable answer: two deepseek-r1 reasoning agents returned only truncated <think> traces at the harvest token budget, and an external CLI agent (OpenClaw) had no headless command configured. OpenClaw is kept in the released answer files as an all-abstain row; it casts no vote and does not change any result. The pool is intentionally correlated: the mean signed pairwise agreement (agree minus disagree per co-answered question, ∈ [−1, 1]) is 0.33–0.35 on MMLU and TruthfulQA (raw agree-rate ≈0.66). It is also unbalanced in a way that matters for the results: on MMLU the largest dependence group (6 llama3.2 agents) is the least accurate (0.43–0.49), while the second group (4 qwen2.5 agents) is the most accurate (0.76–0.83). Some agents are exact or near copies: the three qwen2.5:7b agents have the same accuracy on MMLU, TruthfulQA and ARC. Sec. VI-C varies this composition. b) Datasets: Four closed-book multiple-choice benchmarks, all answered by the same pool. MMLU (160 questions, stride-sampled across subjects) has high disagreement among agents. SciQ (160) is grade-school science on which almost every agent is right. TruthfulQA-MC1 (100) is adversarial

TABLE I AGENT POOL : BASE MODEL , AGENT TYPE , AND ACCURACY PER DATASET ( MEASURED ON GOLD , ON ANSWERED QUESTIONS ). O PEN C LAW NEVER ANSWERS . Agent

Base model

Type

MMLU

SciQ

TQA

ARC

direct:qwen2.5:14b direct:qwen2.5:7b rag:qwen2.5:7b sc:qwen2.5:7b sc:llama3.2:latest rag:llama3.2:latest direct:llama3.2:latest persona:cautious persona:expert persona:skeptic direct:mistral:7b direct:phi3.5:latest direct:gemma2:2b openclaw

qwen2.5 qwen2.5 qwen2.5 qwen2.5 llama3.2 llama3.2 llama3.2 llama3.2 llama3.2 llama3.2 mistral phi3.5 gemma2 openclaw

direct direct RAG self-cons. self-cons. RAG direct persona persona persona direct direct direct CLI

0.825 0.762 0.762 0.756 0.487 0.472 0.465 0.459 0.444 0.434 0.631 0.625 0.613 —

0.988 0.981 0.975 0.981 0.950 0.969 0.950 0.950 0.956 0.925 0.944 0.981 0.931 —

0.810 0.740 0.740 0.740 0.520 0.520 0.531 0.580 0.540 0.580 0.560 0.500 0.440 —

0.919 0.906 0.906 0.906 0.744 0.731 0.725 0.713 0.694 0.694 0.762 0.850 0.694 —

by construction: popular misconceptions are offered as wrong answers. ARC-Challenge (160) is grade-school reasoning with moderate disagreement. For scale we use two further MMLU harvests of 5,000 and 10,000 questions answered by the same 13 agents served with vLLM in half precision (Sec. VI-E). The SciQ and TruthfulQA harvests cover their full question sets only at this size; we did not harvest them at larger scale. A BoolQ pilot is reported in Appendix B; it is not a valid closed-book test because 11 of the 13 agents did not see the passage the task requires. c) Baselines: Majority vote (MV); the classical TD and crowdsourcing estimators Dawid–Skene, TruthFinder, Investment, Sums, CRH, CATD, LTM, WeightedMV and SSTFIV/VI [3], [1], [4], [6], [5]; the dependence-aware ACCU and DEPEN [2], [30], correlation fusion (PochampallyCorr) [31], a mean-field Ising aggregator over the signed agreement graph (IsingMF), and ClusterVote (agents grouped by agreement ≥ θ, each group casting one vote decided by its internal majority); and the concurrent LLM aggregators CARESVD [38] and ISP / OW-I [36]. For LTM, our singletruth implementation sets each agent’s false-positive rate to (1−sensitivity)/(|Vq |−1), which makes its update identical to one-coin Dawid–Skene; its predictions coincide with Dawid– Skene on every dataset, so we report one row. CARE is run from the authors’ released code: each (question, option) pair becomes a binary item scored 1 if the agent chose it, and the option with the highest CARE score is predicted. Its structurelearning step needs an invertible judge-correlation matrix; our pools contain agents with identical answers, which makes the matrix singular, and the published solver then returns no solution on most splits (341 of 520 solves on MMLU, all on the other datasets). We therefore report CARE with a small ridge (10−3 I) added to the correlation matrix, and give the unmodified results in Appendix A. ISP and OW-I have no released code; we implement them from the paper’s equations, estimating the second-order statistics from answer co-occurrence. Two single-agent references bound the value of labels: the best agent selected on the validation half (ties broken at random) and, in Sec. VI-F, the best agent selected on k labelled questions.

d) Protocol and test of record: Each dataset is split 40 times at random into a validation half and a test half. Methods with a hyperparameter select it on the validation half (grids in Appendix A) and are scored on the test half; label-free methods are scored on the test half directly. We report the mean test accuracy over the splits. One test is used for every comparison: the Nadeau–Bengio corrected resampled-t over the splits, which accounts for the overlap between splits, with Holm correction across all comparisons in a table. A method marked “val.-tuned” uses the labels of the validation half to pick its hyperparameter and is therefore not label-free; every BP-TD variant marked “no labels” fixes θ in advance (0.5) or sets it with the label-free Otsu rule (Sec. VI-B), or does not use θ (family). BP-TD is label-blind: its predictions are identical when the gold labels are shuffled or erased. e) Reproducibility and data availability: We release the BP-TD implementation, all baselines, the harvested agent answers with base-model provenance, the agent prompts and persona texts, and the split seeds on GitHub (https://github. com/LaureBerti/BPTD) and as an archived artifact on Zenodo (DOI: 10.5281/zenodo.21294747, all versions; the version used in this paper is v2.0.0, 10.5281/zenodo.23916537), released under the MIT license. Every number above is recomputable on CPU from the bundled cached answers, with no model re-querying; we redistribute only the derived agent answers from the public MMLU, TruthfulQA, SciQ and ARC benchmarks, not the source corpora. VI. R ESULTS We first compare all methods on the four benchmarks (Sec. VI-A), then take BP-TD apart to see where its gain comes from (Sec. VI-B), vary the agent pool (Sec. VI-C), and finally look at reliability recovery, scale, labels, and the regime question (Secs. VI-D–VI-G). A. Main comparison Table II gives every method on every benchmark. MMLU is the only benchmark on which any aggregator significantly beats majority vote, and only decorrelated BP-TD does so: every BP-TD variant with decorrelation reaches 0.713–0.751 against 0.624 for majority vote, including the three variants that use no labels (0.713–0.716). The classical estimators and the concurrent aggregators do not beat majority vote; Dawid– Skene, Investment and SSTF-VI are significantly below it. Correlation fusion (PochampallyCorr, 0.743) is numerically close to BP-TD with θ tuned (0.751); the difference is not significant (p=1.0 after Holm), and correlation fusion’s splitto-split spread is twice BP-TD’s (SD 0.085 vs. 0.046). We therefore do not claim that BP-TD beats correlation fusion; we claim that it beats majority vote and the classical estimators, and that it does so without labels. On SciQ every method is at or near 0.97: almost all agents are right, so there is nothing to correct. On TruthfulQA no aggregator beats majority vote, decorrelated BP-TD is below it (not significantly), and Dawid–Skene, Investment, ISP and OW-I are significantly below it. On ARC, BP-TD family is

TABLE II ACCURACY ON THE TEST HALVES , MEAN OVER 40 SPLITS . + /− : SIGNIFICANTLY ABOVE / BELOW MAJORITY VOTE (NADEAU –B ENGIO CORRECTED RESAMPLED -t, H OLM - CORRECTED WITHIN THE DATASET ). [n]: MEAN NUMBER OF ITERATIONS OF LOOPY BP-TD; ROWS MARKED ONE - PASS USE ONE ITERATION . CARE, ISP AND OW-I WERE RUN ON THE SAME SPLITS IN A SEPARATE PASS (H OLM FAMILY OF 7). B OLD : BEST AGGREGATOR PER COLUMN ( THE BEST- AGENT ROW USES LABELS AND IS EXCLUDED ). Method

MMLU (160)

SciQ (160)

TruthfulQA (100)

ARC (160)

0.624

0.969

0.650

0.847

0.751+ [12] 0.715+ 0.716+ 0.713+ 0.716+ [11] 0.717+ 0.513− [18]

0.969 [92] 0.969 0.969 0.961 0.976 [9] 0.976 0.969 [7]

0.629 [14] 0.622 0.615 0.617 0.586 [11] 0.599 0.566 [14]

0.842 [18] 0.843 0.832 0.843 0.879 [11] 0.877 0.849 [21]

ClusterVote, θ val.-tuned ClusterVote, θ=0.5 PochampallyCorr ACCU DEPEN IsingMF (surrogate)

0.698 0.672 0.743 0.530 0.570 0.620

0.969 0.969 0.962 0.969 0.969 0.969

0.674 0.653 0.645 0.562 0.556 0.593

0.839 0.826 0.870 0.850 0.866 0.851

CARE-SVD (ridge, γ val.-tuned) ISP OW-I

0.580 0.639 0.587

0.959 0.969 0.969

0.683 0.526− 0.553−

0.859 0.853 0.852

Dawid–Skene (= LTM here) CRH CATD TruthFinder Investment Sums SSTF-IV SSTF-VI WeightedMV

0.500− 0.558 0.493 0.525 0.506− 0.619 0.624 0.504− 0.599

0.969 0.969 0.965 0.969 0.969 0.969 0.969 0.969 0.969

0.551− 0.570 0.555 0.554 0.554− 0.590 0.650 0.555 0.580

0.840 0.849 0.844 0.858 0.817 0.848 0.847 0.839 0.851

Best agent, val.-selected (uses labels)

0.823+

0.976

0.801+

0.905

Majority vote (MV) BP-TD cooccur, θ val.-tuned one-pass = weighted vote BP-TD cooccur, θ=0.5 (no labels) BP-TD cooccur, Otsu θ (no labels) BP-TD family (no labels) one-pass = weighted vote BP-TD none (wa ≡1)

the best label-free aggregator (0.879 vs. 0.847), but the gain is not significant. On all four benchmarks the best agent selected with the validation labels is the most accurate method (0.823, 0.976, 0.801, 0.905). B. Where the gain comes from Weights, not iteration. One pass of BP-TD is exactly a wa -weighted vote (Sec. IV), and on every benchmark the one-pass and loopy cooccur rows at θ=0.5 differ by at most 0.007 (p > 0.75); for family weights the largest difference is 0.013 (TruthfulQA). Iteration adds accuracy only with θ tuned on MMLU (0.751 vs. 0.715, not significant). The reason is visible in the weights: at the value usually selected (θ=0.7) the strongest agent, qwen2.5:14b, leaves its group and keeps a full vote, and iterating then raises the reliability of the strong qwen2.5 agents. Without decorrelation, iteration hurts: BPTD none falls from 0.624 (one pass, identical to majority vote) to 0.513 after 18 iterations, because the reliability update rewards the large, weak llama3.2 bloc. The signed reliability smoothing changes full-set MMLU accuracy by less than 0.013 in either direction. Soft weights versus hard groups. At θ=0.5 the cooccurrence groups on MMLU are exactly the base-model families, recovered from the answers alone. With these identical groups, the soft weighted vote beats ClusterVote, which

reduces each group to one vote decided by its internal majority: 0.716 vs. 0.672 (+0.043, p=0.017). The soft weights keep a split inside a group as fractional votes and break ties between groups; hard grouping throws both away. On the other benchmarks the difference is not significant. The threshold θ. Fig. 2a shows full-set accuracy as a function of θ. On MMLU the gain is flat at +0.10 for θ ∈ [0.4, 0.6], peaks at +0.16 for θ=0.7, and turns negative at θ=0.9, where only near-identical agents are grouped and the weak bloc again outvotes the rest. The label-free Otsu rule, which splits the pairwise agreement values into two classes, chooses θ=0.61 on MMLU and lands on the plateau. On TruthfulQA most thresholds are below majority vote, and on ARC and SciQ the curve is flat. A fixed θ=0.5 or the Otsu rule loses about 0.035 on MMLU relative to tuning θ on 80 labels, but tuning on very few labels is worse than either (Sec. VI-F). C. When decorrelation helps: the agent pool Our pool has a large, weak llama3.2 group and a smaller, strong qwen2.5 group (Table I), which is the case that decorrelation favours. To see how much the result depends on this, we evaluated the label-free methods on 713 sub-pools per benchmark: 378 random sub-pools of 5, 7, 9 and 11 agents, and 335 sub-pools built so that the four qwen2.5 agents form the largest group (all four plus up to three llama3.2 agents

0.0 −0.1 −0.2

largest bloc weaker largest bloc strongest

0.3

0.1

MMLU ARC 0.2

TruthfulQA SciQ 0.4

0.6

0.2 0.1 0.0 −0.1 −0.2

(c) label budget accuracy gain over MV

(b) gain by pool composition

gain over MV

accuracy gain over MV

(a) θ sensitivity (⋆: Otsu θ)

0.3

MMLU ARC TruthfulQA

0.2 0.1 0.0 −0.1

0.8

MMLU

ARC

TruthfulQA

SciQ

2

5

10

20

40

80

labelled questions k

co-occurrence threshold θ

Fig. 2. (a) Gain of loopy BP-TD cooccur over majority vote as a function of θ (full sets); stars mark the label-free Otsu θ. (b) Gain of BP-TD family over majority vote across sub-pools. Dark: random sub-pools whose largest base-model group is not the most accurate group; light: sub-pools whose largest group is the most accurate (random and constructed). (c) Gain of the best agent selected on k labelled questions (solid) against label-free BP-TD family (dashed, k=0); mean over 40 draws.

D. Reliability recovery BP-TD estimates a reliability for each agent, and on MMLU these estimates rank agents by their true accuracy. On the full 160 questions the Spearman correlation is ρ=+0.94 (bootstrap over agents, 95% CI [0.75, 1.0]; Fig. 3), whereas Dawid– Skene, CRH and agreement frequency give −0.62, −0.62 and −0.58: they rank the weak llama3.2 bloc first. On the 40 test halves of 80 questions the correlation is 0.86 ± 0.09 for BP-TD cooccur (θ=0.5; minimum 0.63) and −0.65 ± 0.05 for Dawid–Skene. On ARC, BP-TD family also recovers the ranking (0.91 ± 0.05). On SciQ and TruthfulQA the estimates

1.0 estimated reliability

and any subset of the other three). For each sub-pool we record whether its largest group is also its most accurate one; this uses the labels only to describe the pool. Fig. 2b gives the answer. When the largest group is weaker than the rest, BP-TD family beats majority vote on MMLU in 98% of random sub-pools (median +0.10) and on ARC in 87% (median +0.04). When the largest group is the most accurate, it loses on MMLU in 84% of random and 87% of constructed sub-pools (median −0.03 and −0.05) and on ARC in 77% and 94%. On TruthfulQA it loses in most sub-pools of either kind, and on SciQ the gain stays near zero. On MMLU, BP-TD cooccur (θ=0.5 or Otsu) gives the same picture; on ARC it gains little in either kind of sub-pool. The gain is therefore not a property of the method alone; it is a property of the method and the pool. A controlled version gives the same picture. We added k noisy copies of the strongest agent (qwen2.5:14b, 10% of answers replaced by random wrong options) to the full pool. On MMLU, majority vote rises from 0.612 to 0.808 at k=10, because the copies are mostly right; BP-TD family stays at 0.71–0.72, because it discounts the copies as one group; undiscounted BP-TD none reaches 0.825. TruthfulQA and ARC behave alike. Decorrelation is the right correction against a weak dominant bloc and the wrong one against a strong dominant bloc.

0.9 0.8

BP-TD (cooccur) Dawid--Skene

0.7 0.6 0.5 0.5 0.6 0.7 0.8 true agent accuracy (measured on gold)

Fig. 3. Estimated vs. true per-agent reliability on MMLU. BP-TDcooccur recovers the ranking (Spearman ρ=+0.94); Dawid–Skene inverts it (ρ=−0.62). Dashed lines are least-squares fits.

vary as much between splits as their mean, so no ranking can be recovered. On the MMLU harvests of 5,000 and 10,000 questions, where the half-precision agents are closer to one another in accuracy, the correlation is lower but still positive (0.75 and 0.68), and every baseline remains negative. Reliability estimates are what iteration adds to the weighted vote, and they are the output that matters when the goal is to choose or audit agents rather than to answer questions. E. Scale Table III repeats the comparison on all 5,000 and 10,000 questions of the larger MMLU harvests (10 splits; each test half has 2,500 or 5,000 questions). The gain over majority vote is smaller than on the 160-question sample but significant: +0.074 with θ tuned, and +0.032–0.036 for BP-TD family with no labels. BP-TD none stays below majority vote, and the best agent selected on the validation half is again the most accurate method. Runtime grows close to linearly with the number of answers: one pass takes 16, 433 and 940 ms on 160, 5,000 and 10,000 questions, and loopy BP-TD needs about 11 iterations at scale. On synthetic data up to n m=160,000 the fitted runtime exponent is 0.89 (95% CI [0.79, 1.04]; Fig. 4),

Method

5,000 questions

10,000 questions

Majority vote BP-TD cooccur, θ val.-tuned one-pass = weighted vote BP-TD cooccur, θ=0.5 BP-TD cooccur, Otsu θ BP-TD family BP-TD none ClusterVote, θ val.-tuned Best agent, val.-selected

0.681 0.755+ (+0.074) 0.755+ (+0.073) 0.710 (+0.029) 0.737 (+0.056) 0.718+ (+0.036) 0.637− (-0.044) 0.740+ (+0.059) 0.790+ (+0.109)

0.674 0.748+ (+0.073) 0.744+ (+0.069) 0.702+ (+0.028) 0.715 (+0.040) 0.707+ (+0.032) 0.633− (-0.041) 0.729+ (+0.055) 0.787+ (+0.113)

BP-TD gain over majority vote (Δ acc)

TABLE III MMLU AT SCALE ( V LLM, HALF PRECISION ): ACCURACY ON THE TEST HALVES , MEAN OVER 10 SPLITS , GAIN OVER MAJORITY VOTE IN BRACKETS . M ARKS AS IN TABLE II.

0.3

MMLU (favorable) SciQ (near-consensus) TruthfulQA (adversarial)

0.2 0.1 0.0 −0.1 −0.2

0.0

0.1

0.2

0.3

0.4

0.5

Agent disagreement (1 − majority share)

Fig. 5. BP-TD’s accuracy gain over majority vote vs. per-question agent disagreement (1−majority share), stratified into quartiles within each dataset (single held-out split). The gain rises with disagreement on MMLU, is nonpositive throughout on TruthfulQA, and is flat on SciQ.

(a) Log-log runtime scaling with bootstrap CI data

runtime (s)

102

power-law fit 95% CI

101 100 10−1

104 105 103 problem size nm = nquestions × magents

(b) Agent-axis sweep exposes the O(m 2) term runtime (s)

quadratic fit data

0.5 m 2 share at m = 128: 0.13 (O(m 2) non-negligible)

0.0 0

20 40 60 80 100 120 m = number of agents (fixed n = 1000)

140

Fig. 4. Scalability (synthetic). (a) log–log runtime vs. n · m; (b) agent-axis sweep (n=1000) exposing the O(m2 ) term.

and with n fixed the cost has the predicted quadratic term in the number of agents. F. Labels Label budget. Fig. 2c compares label-free BP-TD with the best agent selected on k labelled questions (40 random draws; ties broken at random). The best agent overtakes BP-TD family at k=5 on MMLU and k=20 on ARC; on TruthfulQA it overtakes majority vote at k=5; on SciQ all differences stay below 0.015. Using the same few labels to tune θ is a poor use of them: on MMLU, tuning on 2, 5 or 10 labels gives 0.600, 0.637 and 0.669, below the 0.713 of the label-free Otsu rule; tuning beats the label-free choice only from about 40 labels. Seeding. Clamping 1–20% of the questions to their gold answers changes the accuracy on the remaining questions by at most 0.001 on MMLU and SciQ, 0.01 on ARC and 0.02 on TruthfulQA (40 draws; SD up to 0.03). With 13 agents each

seed adds one observation per agent to the reliability counts, which is too little to move them. Do the classical estimators work at all? On synthetic data with 13 independent agents of accuracy 0.30–0.92, Dawid– Skene (and LTM) beat majority vote (0.992 vs. 0.978 on 160 questions, 0.992 vs. 0.973 on 1,000; 20 seeds), as does BPTD none. Their collapse on MMLU is therefore caused by the data: Dawid–Skene’s answers agree on 98.8% of questions with the plurality of the six llama3.2 agents, which is right only 47% of the time. On the same independent agents, BP-TD cooccur loses a little (0.983), the price of discounting agents that agree only because they are right. G. When does agent truth discovery work? The sub-pool study gives the practical rule: decorrelation helps when the agents that agree most are not the agents that are most often right. Two further analyses refine it. Disagreement. Within MMLU, the gain over majority vote grows with per-question disagreement (+0.00, +0.00, +0.125, +0.275 across quartiles on one held-out split; Fig. 5); within TruthfulQA it is never positive; within SciQ it is flat. MMLU and TruthfulQA have nearly the same mean signed agreement (0.33 and 0.35), so how much agents agree does not decide the outcome; whether their shared answer points toward or away from the truth does. A wrong bloc. On the ARC answers we appended k synthetic agents that all give the same wrong answer (Fig. 6). Majority vote and BP-TD none collapse once the bloc is the majority. BP-TD family is unaffected when the bloc shares a base model (≈0.86 at every k); when the bloc hides its provenance, BP-TD cooccur still recovers the truth (≈0.72). The truth-aware shared_error weights fail throughout (≈0.46), because the bloc defines the current truth estimate and so appears to share no errors. This is the mirror image of the strong-bloc experiment of Sec. VI-C: the same discount that rescues the answer from a wrong bloc throws away a right one. No label-free diagnostic. A practitioner needs to know which case applies without labels. We tested two candidate statistics, chosen before seeing the results, on the 2,852 subpools: the agreement between the largest group’s plurality

accuracy (mean ± SD over 20 injections)

Controlled correlated-bloc stress test on real ARC answers (k = 0 is the unperturbed near-consensus dataset)

(a) Shared provenance (family + cooccur can detect the bloc)

1.0

(b) Distinct provenance (only cooccur can detect the bloc)

0.8 0.6 0.4 0.2 0.0 0

5 10 15 20 injected correlated-wrong bloc size k (14 real agents) MV

cooccur

0

family

5 10 15 20 injected correlated-wrong bloc size k (14 real agents) shared-err

none

Fig. 6. Controlled correlated-bloc stress test on the real ARC answers: we append k synthetic agents sharing one wrong answer per question and sweep k at fixed hyperparameters, under (a) shared and (b) distinct bloc provenance (↑ better). Curves: majority vote (MV) and four BP-TD variants (cooccur, family, shared-err, none). Mean ± SD over 20 injections.

answer and the plurality answer of the other agents, and the fraction of questions on which BP-TD changes majority vote’s answer. Neither separates sub-pools where BP-TD helps from those where it hurts: the AUC is 0.53–0.67 per benchmark for the first and 0.48–0.85 for the second, and 0.53 and 0.51 when the benchmarks are pooled. Telling the two cases apart remains the main open problem; where it cannot be done, spending a few labels on choosing an agent is the safer option. VII. D ISCUSSION AND L IMITATIONS What the gain means in practice. Consider a pipeline that answers 10,000 MMLU-style questions with these 13 agents and no labels. BP-TD family answers about 320 more questions correctly than majority vote (0.707 vs. 0.674) and reports which agents are reliable; its one-pass form (0.702) runs in under a second on CPU. If the pool’s largest group is its strongest, the same choice costs about 300–500 answers per 10,000 (median −0.03 to −0.05 on MMLU sub-pools). If 20 questions can be labelled, choosing the best agent is better still: on the 160-question MMLU sample it reaches 0.805, against 0.711 for BP-TD family. BP-TD is therefore a tool for the label-free case, for pools in which redundant variants of a weak model would otherwise outvote stronger, independent models, and for auditing agent reliability. Limitations. (i) Benchmarks and scale. The regime results rest on four benchmarks, of which only MMLU was harvested at scale; SciQ and TruthfulQA were not. All agents are local open models of up to 14B parameters; larger API-served models were not tested and may be less redundant. (ii) Pool composition. The headline gain depends on a pool whose largest group is weak (Sec. VI-C). (iii) No diagnostic. We found no label-free way to tell whether decorrelation will help. (iv) Dependence model. The weights are a heuristic tempering of the likelihood, not a generative model of dependence,

and the truth-aware shared_error weights fail when the dependent bloc is the majority. (v) Contamination. The base models have likely seen these benchmarks. Every aggregator receives the same answers, so contamination does not favour one aggregator on a given answer matrix, but it can change the answer matrix itself: if models of one family memorized the same items, their errors are more shared, which is the variable our regime analysis depends on. The TruthfulQA result in particular may partly reflect contamination rather than the benchmark’s adversarial design alone. VIII. C ONCLUSION Truth discovery among LLM agents fails in its classical form because agents built on one base model agree for reasons unrelated to the truth. Discounting such agents, as BP-TD does with provenance or co-occurrence weights inside a Dawid– Skene model, recovers a significant gain over majority vote on MMLU without labels, at 160 and at 10,000 questions, and recovers the ranking of agents by accuracy where classical estimators invert it. Its first iteration is a weighted vote; iteration contributes the reliability estimates. Across thousands of sub-pools, the gain appears when the largest correlated group is weaker than the other agents and reverses when it is the strongest, and we found no label-free statistic that predicts which case holds. With twenty labelled questions, choosing the best single agent is better than every aggregator we tested. Two directions follow: a label-free test for the direction of shared errors, and a rule for switching from aggregation to agent selection as labels arrive. ACKNOWLEDGMENT The author used DeepL as a writing-assistance tool for translating and enhancing the text and for LATEX proofreading, and Claude for automating the experiments. All scientific

contributions—experimental design, algorithm development, formalization, data collection and analysis, result interpretation, and the claims made in the paper—are entirely the author’s own work. The AI assistant did not generate any experimental results, derive any mathematical content, or contribute to the conceptual framing of the research. A PPENDIX A H YPERPARAMETER GRIDS AND CARE DETAILS Tuned methods select their hyperparameter on the validation half by accuracy and freeze it before scoring the test half. Grids: BP-TD cooccur θ ∈ {0.2, . . . , 0.8} (step 0.1); ClusterVote θ ∈ {0.3, . . . , 0.7}; TruthFinder γ ∈ {0.1, 0.3, 0.5, 1.0}; CARE-SVD γ ∈ {0.1, 0.2, 0.25, 0.5, 0.75, 1, 2, 3, 5, 7, 10} (the authors’ grid). The thresholds most often selected for BP-TD cooccur are 0.7 on MMLU, 0.2 on SciQ, 0.5 on TruthfulQA and 0.8 on ARC. Label-free BP-TD uses θ=0.5 or the Otsu threshold of the off-diagonal signed agreements (mean 0.56 over MMLU test halves). Tuning the iteration count of Dawid– Skene and LTM, CATD’s α and TruthFinder’s γ on the validation half does not lift them above majority vote on MMLU (Dawid–Skene 0.501, CATD 0.493, LTM 0.500, TruthFinder 0.525). CARE without the ridge. With the published code unchanged, the structure solve fails on 341 of 520 solves on MMLU and on all 520 on SciQ, TruthfulQA and ARC, and CARE then returns majority vote. Its accuracy is 0.598 (γ=1) and 0.607 (γ tuned) on MMLU, and equal to majority vote on the other three benchmarks. On synthetic independent agents the ridge version is slightly below majority vote (0.959 vs. 0.978), which suggests that the one-indicator-per-option encoding we use for multiple choice is not where CARE is strongest. A PPENDIX B B OOL Q PILOT Before the main harvest we ran the same pool on 128 BoolQ questions. BoolQ asks a yes/no question about a passage, but only the two RAG agents received a passage; the other 11 agents answered from memory. The pilot therefore does not test closed-book aggregation, and we do not use it in the regime analysis. Under the protocol of Sec. V (40 splits), no aggregator differs significantly from majority vote (0.682): ClusterVote 0.722, BP-TD cooccur 0.706, BP-TD family 0.681, Dawid–Skene 0.654, PochampallyCorr 0.640. The best agent selected on the validation half reaches 0.826, significantly above majority vote. R EFERENCES [1] X. Yin, J. Han, and P. S. Yu, “Truth discovery with multiple conflicting information providers on the web,” IEEE TKDE, vol. 20, no. 6, pp. 796– 808, 2008. doi:10.1109/TKDE.2007.190745 [2] X. L. Dong, L. Berti-Equille, and D. Srivastava, “Integrating conflicting data: the role of source dependence,” PVLDB, vol. 2, no. 1, pp. 550–561, 2009. doi:10.14778/1687627.1687690 [3] Q. Li, Y. Li, J. Gao, B. Zhao, W. Fan, and J. Han, “Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation,” SIGMOD, pp. 1187–1198, 2014. doi:10.1145/2588555.2610509

[4] A. P. Dawid and A. M. Skene, “Maximum likelihood estimation of observer error-rates using the EM algorithm,” Applied Statistics, vol. 28, no. 1, pp. 20–28, 1979. doi:10.2307/2346806 [5] B. Zhao, B. I. P. Rubinstein, J. Gemmell, and J. Han, “A Bayesian approach to discovering truth from conflicting sources for data integration,” PVLDB, vol. 5, no. 6, pp. 550–561, 2012. doi:10.14778/2168651.2168656 [6] J. Pasternack and D. Roth, “Knowing what to believe (when you already know something),” COLING, pp. 877–885, 2010. https://aclanthology.org/ C10-1099/ [7] J. Huang, H. Shen, Q. Cao, S. Tao, and X. Cheng, “Signed bipartite graph neural networks,” CIKM, pp. 740–749, 2021. doi:10.1145/3459637.3482392 [8] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” ICLR, 2023. doi:10.48550/arXiv.2203.11171 [9] J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Zou, “Mixtureof-agents enhances large language model capabilities,” ICLR, 2025. doi:10.48550/arXiv.2406.04692 [10] Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, “Improving factuality and reasoning in language models through multiagent debate,” ICML, pp. 11733–11763, 2024. doi:10.48550/arXiv.2305.14325 [11] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan, “Language models (mostly) know what they know,” arXiv:2207.05221, 2022. doi:10.48550/arXiv.2207.05221 [12] Y. Zhang, X. Chen, D. Zhou, and M. I. Jordan, “Spectral methods meet EM: A provably optimal algorithm for crowdsourcing,” NeurIPS, pp. 1260–1268, 2014. https://proceedings.neurips.cc/paper files/paper/ 2014/hash/700c82b01b20a5e3e160dab2e511c48a-Abstract.html [13] J. Ok, S. Oh, J. Shin, and Y. Yi, “Optimality of belief propagation for crowdsourced classification,” ICML, PMLR vol. 48, pp. 535–544, 2016. https://proceedings.mlr.press/v48/ok16.html [14] E. Kim, A. Garg, K. Peng, and N. Garg, “Correlated errors in large language models,” ICML, pp. 30038–30066, 2025. doi:10.48550/arXiv.2506.07962 [15] G. Sun, A. Kagrecha, P. Manakul, P. Woodland, and M. Gales, “SkillAggregation: Reference-free LLM-dependent aggregation,” ACL, pp. 15532– 15548, 2025. doi:10.18653/v1/2025.acl-long.756 [16] W. Li, Y. Lin, M. Xia, and C. Jin, “Rethinking mixture-of-agents: Is mixing different large language models beneficial?” arXiv:2502.00674, 2025. doi:10.48550/arXiv.2502.00674 [17] G. Kohli, “Nine judges, two effective votes: Correlated errors undermine LLM evaluation panels,” arXiv:2605.29800, 2026. doi:10.48550/arXiv.2605.29800 [18] A. Galland, S. Abiteboul, A. Marian, and P. Senellart, “Corroborating information from disagreeing views,” WSDM, 2010, pp. 131–140. doi:10.1145/1718487.1718504 [19] X. L. Dong, E. Gabrilovich, G. Heitz, W. Horn, K. Murphy, S. Sun, and W. Zhang, “From data fusion to knowledge fusion,” PVLDB, vol. 7, no. 10, pp. 881–892, 2014. doi:10.14778/2732951.2732962 [20] D. A. Waguih and L. Berti-Equille, “Truth discovery algorithms: An experimental evaluation,” QCRI Tech. Rep., arXiv:1409.6428, 2014. https://arxiv.org/abs/1409.6428 [21] Y. Li, J. Gao, C. Meng, Q. Li, L. Su, B. Zhao, W. Fan, and J. Han, “A survey on truth discovery,” ACM SIGKDD Explorations, vol. 17, no. 2, pp. 1–16, 2016. doi:10.1145/2897350.2897352 [22] L. Berti-Equille and J. Borge-Holthoefer, Veracity of Data: From Truth Discovery Computation Algorithms to Models of Misinformation Dynamics. Synthesis Lectures on Data Management, Morgan & Claypool, 2015. doi:10.2200/S00676ED1V01Y201509DTM042 [23] L. Berti-Equille and M. L. Ba, “Veracity of big data: Challenges of crossmodal truth discovery,” ACM J. Data Inf. Qual., vol. 7, no. 3, pp. 1–3, 2016. doi:10.1145/2935753 [24] L. Berti-Equille, “Data veracity estimation with ensembling truth discovery methods,” in Proc. IEEE Int. Conf. Big Data, pp. 2628–2636, 2015. doi:10.1109/BigData.2015.7364062 [25] Y. Zheng, G. Li, Y. Li, C. Shan, and R. Cheng, “Truth inference in crowdsourcing: Is the problem solved?” PVLDB, vol. 10, no. 5, pp. 541– 552, 2017. doi:10.14778/3055540.3055547 [26] D. R. Karger, S. Oh, and D. Shah, “Iterative learning for reliable crowdsourcing systems,” NeurIPS, pp. 1953–1961, 2011. https://papers. nips.cc/paper/4396-iterative-learning-for-reliable-crowdsourcing-systems [27] J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan, “Whose vote should count more: Optimal integration of labels

from labelers of unknown expertise,” NeurIPS, pp. 2035– 2043, 2009. https://papers.nips.cc/paper files/paper/2009/hash/ f899139df5e1059396431415e770c6dd-Abstract.html [28] D. Zhou, J. C. Platt, S. Basu, and Y. Mao, “Learning from the wisdom of crowds by minimax entropy,” NeurIPS, pp. 2204– 2212, 2012. https://papers.nips.cc/paper files/paper/2012/hash/ 46489c17893dfdcf028883202cefd6d1-Abstract.html [29] Q. Liu, J. Peng, and A. Ihler, “Variational inference for crowdsourcing,” NeurIPS, pp. 701–709, 2012. https://papers.nips.cc/paper files/ paper/2012/hash/cd00692c3bfe59267d5ecfac5310286c-Abstract.html [30] X. L. Dong, L. Berti-Equille, Y. Hu, and D. Srivastava, “Global detection of complex copying relationships between sources,” PVLDB, vol. 3, no. 1– 2, pp. 1358–1369, 2010. doi:10.14778/1920841.1921008 [31] R. Pochampally, A. Das Sarma, X. L. Dong, A. Meliou, and D. Srivastava, “Fusing data with correlations,” SIGMOD, pp. 433–444, 2014. doi:10.1145/2588555.2593674 [32] D. Jiang, X. Ren, and B. Y. Lin, “LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion,” ACL, pp. 14165–14178, 2023. doi:10.18653/v1/2023.acl-long.792 [33] J. Li, Q. Zhang, Y. Yu, Q. Fu, and D. Ye, “More agents is all you need,” TMLR, 2024. https://arxiv.org/abs/2402.05120 [34] P. Verga, S. Hofstätter, S. Althammer, Y. Su, A. Piktus, A. Arkhangorodsky, M. Xu, N. White, and P. Lewis, “Replacing judges with juries: Evaluating LLM generations with a panel of diverse models,” arXiv:2404.18796, 2024. https://arxiv.org/abs/2404.18796 [35] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” NeurIPS Datasets and Benchmarks Track, vol. 36, 2023. https://proceedings.neurips.cc/paper files/paper/2023/ hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets and Benchmarks.html [36] R. Ai, Y. Pan, D. Simchi-Levi, M. Tambe, and H. Xu, “Beyond majority voting: LLM aggregation by leveraging higher-order information,” ICML, 2026. https://arxiv.org/abs/2510.01499 [37] K. Balasubramanian, A. Podkopaev, and S. P. Kasiviswanathan, “Dependence-aware label aggregation for LLM-as-a-judge via Ising models,” arXiv:2601.22336, 2026. https://arxiv.org/abs/2601.22336 [38] J. Zhao, C. Shin, T.-H. Huang, S. S. S. Namburi GNVV, and F. Sala, “CARE: Confounder-aware aggregation for reliable LLM evaluation,” ICML, 2026. https://arxiv.org/abs/2603.00039

Related documents

• VST: Verifiable Structured Transport for Auditable Agent-to-Agent Alpha Discovery#975129
• Provenance-Aware Belief Updating in Human-AI Feedback Loops#1297360
• Consensus and Factual Dynamics in Large Populations of Interacting Language Models#1144178
• Truthmakers#763661
• CiteGuard-RAG: A Validation-Centered AI System for Evidence-Grounded Question Answering#1109870