One Human, N Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence Cesare Zavattari, Alessandro Tommasi, and Giuseppe Prencipe (
)
arXiv:2607.28317v1 [cs.AI] 30 Jul 2026
Dipartimento di Informatica, Università di Pisa, Pisa, Italy
Abstract. A single human must audit N LLM agents under a budget of B ≪ N audits per round, guided by self-reported confidence that may be adversarially miscalibrated and by correlated errors. We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold δ ∗ past which confidence-ranked auditing is worse than random. Two a-priori expectations reverse: δ ∗ rises as the budget shrinks, and cross-family correlation is not low—shared difficulty dominates lineage. Five open-weight LLMs show operationally useless (near-constant) confidence, point estimates at or beyond the flip though CIs straddle it; a proprietary model is informative and lands below it. We give a quantitative criterion for vacuous oversight, and replaying policies on recorded traces confirms the ordering. Keywords: human oversight · LLM agents · audit budget · calibration · correlated errors
1
Introduction
Deployments increasingly run not one LLM agent but a fleet: N persistent agents built from a few base models, emitting outputs faster than any human can check. Regulation nonetheless mandates effective human oversight of high-risk systems (EU AI Act, Art. 14 [6]), while one supervisor can audit B ≪ N outputs per round. Auditing “the least confident outputs” trusts a signal supplied by the auditees themselves—and though LLMs mostly know what they know [13], their verbalized confidence is badly calibrated [15, 21, 24]. Throughout, agents are honest-but-miscalibrated ; strategic behavior is the province of AI Control [8] and future work. We formalize budgeted noisy inspection of a fleet with two coupled imperfections: self-reported confidence whose informativeness degrades with an adversarial-miscalibration strength δ, and errors correlated through a two-level Gaussian copula. We ask when confidence-ranked auditing flips to worse than random (threshold δ ∗ ); how much correlation-aware posterior transfer helps; and when oversight is vacuous—no feasible policy beats no oversight by more than a fraction τ , a quantitative reading of “rubber-stamping” [7]. To the best of our knowledge, no prior work jointly models (i) a single human supervisor allocating (ii) a limited audit budget across (iii) a fleet of persistent
2
C. Zavattari et al.
LLM agents whose (iv) self-reported confidence is subject to adversarial miscalibration and whose (v) error correlation is partially predictable from fleet composition (capability tier and lineage); nor does prior work connect such allocation to a quantitative criterion for vacuous human oversight. We measured the premises before trusting the phase diagram, and two apriori expectations reversed: δ ∗ rises as the budget shrinks (§4), and shared item difficulty, not lineage, dominates cross-family correlation (§5). Contributions: the model with its located flip threshold δ ∗ (B/N ) and vacuity criterion (§3– 4); locked measurements of six LLMs spanning the spectrum; a trace-grounded replay confirming the ordering, all pre-specified contrasts Holm-significant (§5). Illustrative material for the paper is collected in the Appendix.
2
Related Work
Scoping: deferral allocates instances; we allocate audits. Selective prediction decides, per instance, whether the model predicts or abstains [20]; learning-to-defer instead routes the instance to a model or a human [18, 19]; multi-expert L2D routes instances to experts [22, 16, 17, 1]. They allocate instances to experts; we allocate audits to persistent agents: the human never replaces an output, only spends a budget to catch errors. Five features separate this from generic budgeted selection: (1) the units are persistent agents, not data points; (2) their group structure is given by the fleet’s bill of materials, not estimated; (3) error correlation is partially predictable from fleet composition (chiefly capability tier; §5); (4) an audit on one agent updates beliefs about its siblings; and (5) the priority signal is supplied by the audited agent itself, not estimated by the allocator—which—once adversarially miscalibrated—is exactly what makes confidence-ranked auditing flip to worse-than-random (§4). The problem touches five literatures. Budgeted inspection: sequential search goes back to Pandora’s box [23], acceptance sampling [4], and skip-lot schemes (ISO 2859-3 [12]) even modulate inspection by supplier self-reports, without a formal treatment of adversarial miscalibration or cross-unit correlation. Budgeted query allocation in ML: active learning and crowd labeling allocate queries under budgets [11, 3], batch multi-fidelity methods decorrelate batches [14], and correlated bandits exploit known arm correlation [9, 10]—the closest mechanism to ours, but they maximize reward from signals the learner observes, whereas we detect errors where the priority signal is supplied by the auditee. AI control protocols audit untrusted models assuming strategic subversion [8], leaving the honest-but-miscalibrated fleet with a human allocator open. On confidence, LLMs mostly know what they know [13], yet their verbalized estimates are systematically overconfident [15, 21, 24], worse in agentic settings [25, 26]; we consume these as premises and measure their allocation-level consequence. Finally, socio-legal oversight mandates are criticized as rubber-stamping [7], with fleet-level accountability gaps [6, 2]; for vacuous oversight we found no explicit quantitative criterion—H3 supplies one.
One Human, N Agents
3
3
Model
The lineage is acceptance sampling [4] extended with priority signals that are self-reported, adversarially miscalibrated, and correlated across units. Fleet and errors. N persistent agents are partitioned into F base-model families. In round t, agent i errs with indicator Ei ∈ {0, 1}, Pr(Ei =1) = ei ∼ U (0.05, 0.30) fixed per agent; correlation p enters through a two-level Gaussian √ √ copula Zi = ρ0 G0 + ρf − ρ0 Gfam(i) + 1 − ρf Ui , with Ei = 1{Φ(Zi ) < ei } and G0 , Gf , Ui ∼ N (0, 1): ρ0 a fleet-wide shared factor (e.g. item difficulty), ρf ≥ ρ0 the within-family latent correlation. The shocks are drawn once per replication and evolve as stationary AR(1) with persistence φ (default 1): at temperature 0, same-family co-failure is a stable difficulty profile, not per-round noise—persistence makes an audit on one agent evidence about its siblings. Marginals are unaffected. ρ is quoted in latent units; estimating ρ̂ inverts the copula map given measured marginals (caveat in §6). Confidence. Correct outputs report confidence c ∼ Beta(6, 2); errors draw from the mixture (1 − δ) Beta(3, 3) + δ Beta(10, 1.5): δ is the fraction of errors whose confidence mimics that of correct outputs. Audit and risk. Per round the supervisor audits At , |At | ≤ B; an audit P reveals Ei flipped with probability ε. The metric is residual risk R(π) = E[ i Ei 1{i ∈ / caught(At )}], the expected undetected errors per round. Oversight is vacuous at level τ if maxπ (Rnone − Rπ )/Rnone < τ over the deployable policies considered. Policies. random; conf_ranked (B lowest, ties random); round_robin; oracle_conf (confidence at δ=0: upper reference, not deployable); diversity_bayes: per-agent Beta posteriors, audits update same-family siblings with weight w (robust over {ρ, ρ2 , ρ/2}), batched-greedy posterior UCB—a knowledge-gradient-style instantiation [3, 9], not an optimality claim.
4
Synthetic Phase Diagram (E1)
We sweep N ∈ {20, 100}, F ∈ {1, 4}, B/N ∈ {0.02, . . . , 0.2}, δ ∈ {0, . . . , 1}, ρ ∈ {0, . . . , 0.8}, ε ∈ {0, . . . , 0.2}; 200 replications of T =50 rounds per cell (heatmaps in the supplementary, Figs. S1–S2; degenerate B=0 cells excluded). H1 — the flip, with reversed direction. Past a threshold δ ∗ (B/N ), confidenceranked auditing is dominated by random. We expected a priori that δ ∗ decreases as the budget shrinks; the experiment shows the opposite: δ ∗ increases as B/N shrinks—tight budgets are protected by their own scarcity (they audit only the extreme low-confidence tail, which stays informative), generous budgets flip first (they dip into the poisoned mid-ranks). Located values: δ ∗ = 0.93, 0.89, 0.84, 0.75 at B/N = 0.02, 0.05, 0.1, 0.2; relatively insensitive to ρ, ε, N, F in our sweeps
4
C. Zavattari et al.
(overconfident-error mass piles at the high-confidence end, so the extreme tail degrades last). On the measured fleet’s matched-marginals background δ ∗ (0.1) ≈ 0.83 (Fig. S1); the flip heatmap is in the supplementary (Fig. S1). H2 — correlation-aware transfer. The advantage of diversity_bayes over random grows with ρ (0.34 at ρ=0 to 3.47 risk units at ρ=0.8; δ=0.5, B/N =0.1). The advantage is robust to the correlation model (a Beta log-odds shock reproduces the trend) and to the transfer weight (w ∈ {ρ, ρ2 , ρ/2}: 3.47/3.46/3.47), accumulates over rounds, and collapses without persistence (0.29 at φ=0 vs. 3.47 at φ=1)—full sweeps in the supplementary. It needs a not-too-noisy verifier or strong correlation; the ε crossover is quantified in §6(a).1 H3 — vacuous oversight. With Rnone = 17.5, 10 of 44 cells are vacuous at τ =0.1 (heatmap: Fig. S2, supplementary): at B/N =0.02, oversight in this model is rubber-stamping almost regardless of δ. Correlation rescues high-δ cells (a companion (δ, ρ) sweep at B/N =0.05 leaves 6 vacuous): with useless confidence, transfer is the only signal.
5
Measuring the Premises (E2) and Replaying Policies (E3)
Protocol. Six models—Qwen3 {0.6B, 4B, 8B} (one family), Mistral-7B-Instructv0.3, Phi-4-mini-instruct, and hosted gpt-4o-mini as a cross-provider robustness point—answer the same fixed 500-item manifests of GSM8K and HotpotQA (distractor) at temperature 0 under one locked single-completion prompt eliciting an answer and a 0–100 confidence. Errors are exact-match failures (GSM8K numeric; HotpotQA EM, F1 logged); two temperature-0.7 runs per open-weight model give same-checkpoint anchors (per-model values and parse failures ≤ 3.2%: Table S1, supplementary). Miscalibration is real and extreme (H4). All five open-weight models pile verbalized confidence at ≈1.0 regardless of correctness (ECE 0.43–0.94, supplementary); their confidence is near-constant (Var(c) ≤ 0.016), so AUROC ≈ 0.5: real small/mid LLMs give operationally useless (near-constant) verbalized confidence under our locked elicitation protocol, so confidence-ranked auditing ≈ random by construction. Because global AUROC is the wrong sufficient statistic for audit value—what matters is the error composition of the audited tail—we place models by δprec : the error precision of the lowest-confidence B/N tail, inverted through the synthetic δ → precision map at matched ê. The six models span the spectrum (Table S1): the five open-weight models have error rates ê ∈ [0.43, 0.94] and AUROC 0.50–0.52, with point estimates at or beyond the flip but CIs straddling δ ∗ (inversion sensitivity at high ê, Fig. S1); the proprietary model has informative confidence (AUROC 0.59/0.56, positive tail-precision lift) and is correctly placed below it (δprec = 0.68/0.75 vs. δ ∗ ≈ 0.83; Fig. S1). 1
Clustering pushes errors beyond any fixed budget: E[(nerr − B)+ ] rises 7.59→9.31 as ρ=0→0.8 (N =100, B=10).
One Human, N Agents
5
gsm8k (fleet 0=0.52 [0.43,0.60], matched e) budget B/N
0.2 0.1 0.05 0.0
model (colour) qwen3-0.6b mistral-7b qwen3-4b phi-4-mini qwen3-8b gpt-4o-mini marker (CI verdict) * below flip (CI < ) straddling (CI * ) beyond flip (CI > * ) * band
0.2 0.4 0.6 0.8 miscalibration (measured: precision-based prec)
1.0
Fig. S1. Measured fleet on the matched-marginals background (GSM8K; fleet ρ0 , permodel ê); δprec : the allocation-faithful statistic (§5). Colour = model; marker = CI verdict vs. δ ∗ (circle below, square straddling, triangle beyond). Point estimates put 4/5 open-weight models beyond the flip at B/N =0.1; their CIs straddle δ ∗ (long left tails = flatness of the δ →precision map at low δ, propagated by the inversion). gpt4o-mini (black) sits clearly below the flip (CIs exclude δ ∗ at B/N ≥ 0.1). Replicates on HotpotQA (Fig. S3; closest call B/N =0.05, δprec 0.870 vs. δ ∗ =0.887).
Correlation: shared difficulty dominates lineage. We expected a priori low crossfamily correlation; measured inter-family latent correlation is 0.33–0.85 (indicator units 0.14–0.63), overlapping intra-family, and gpt-4o-mini correlates at 0.27– 0.88 with the open-weight five: the shared factor spans providers. A difficultycontrolled decomposition (partial correlation given leave-out item difficulty, item bootstrap) finds a lineage excess ∆ρ = −0.009 [-0.064, 0.047] (GSM8K) and 0.023 [-0.035, 0.086] (HotpotQA)—both CIs include zero: given item difficulty, lineage adds no measurable correlation, while a fleet-wide residual remains; in item-response-theory terms [5], shared difficulty explains co-failure better than ancestry. The two-level copula (§3) with measured ρ0 = 0.52/0.70 (CIs in Fig. S1) and ρf ≈ ρ0 is the amended model. H2’s mechanism generalizes—transfer pays when persistent errors are correlated, fleet-wide; the lineage claim softens to “correlation is partially predictable from fleet composition.” Within Qwen3, correlation grades by size adjacency (4B↔8B latent 0.72/0.93 vs. 0.6B↔{4B,8B} 0.39–0.61): “family” is not binary. E3: replay on real traces. We replay the policies over a 15-agent fleet (5 models × 3 instances); each round is one benchmark item, so within-round correlation is as measured. Budgets are B ∈ {1, 2, 3} audits on N =15 (realized fractions 6.7/13.3/20%). The oracle uses perfect error knowledge (1 − E): an upper bound, not a deployable policy. The E1 ordering survives (replay figures: Figs. S4–S5, supplementary): oracle < diversity-Bayes < conf-ranked ≈ random on both benchmarks (at B=2, GSM8K: 10.58 < 10.75 < 10.85 ≈ 10.89). All four pre-specified paired contrasts are significant on both benchmarks (Holm-adjusted p: random−div ≤ 0.002/≤ 0.002, conf−div ≤ 0.002/≤ 0.002, at the 2000-sample bootstrap floor), with effect sizes 0.133 and 0.095 risk units on GSM8K (modest, a floor effect at the fleet’s extreme ê) and larger 0.322 / 0.312 on HotpotQA.
6
C. Zavattari et al.
The structure arbiter confirms the a-priori prediction: with ∆ρ ≈ 0, family ≈ tier ≈ pooled, capturing all the value.
6
Limitations
(a) The transfer advantage requires a not-too-noisy verifier or strong correlation: at ρ=0.5 diversity-Bayes cedes to conf-ranked at ε=0.2, but resists at ρ=0.8 (crossover values: supplementary). (b) It requires persistent error profiles (collapses at φ=0), so fleets under frequent fine-tuning have φ < 1 and an attenuated advantage. (c) ρ̂ is measured on aligned items—an upper bound for fleets whose agents process different inputs—and is reported in latent units conditional on the Gaussian copula, extrapolated at high ê (so indicator-unit values are given alongside); the decomposition says which part (shared difficulty vs. lineage residual) transfers. (d) δ̂ comes from one locked elicitation protocol (single-completion confidence) and the direct-answer regime (no CoT); others—separate turn, logits, sampling variance—may recover variance, and with CoT error rates drop and confidence may change: a lower-capability operating point, not anomalous. (e) At very high δ a flip is near-definitional, so the contribution is locating δ ∗ (B/N ) and its reversed direction; relatedly, “AUROC = 0.5 is the flip boundary” holds because measured confidence is near-constant (ranking = jitter ⇒ conf-ranked ≡ random), not because the synthetic flip sits at 0.5 (there it maps to AUROC ≈ 0.34). (f) E3 uses a small fleet (N =15) and is preliminary. (g) gpt-4o-mini is a single proprietary robustness point, not a sample of hosted models, and lacks same-checkpoint anchors (hosted temperature-0 is not deterministic); parse failures are concentrated in one model, with a declared, non-differential imputation policy (rate and policy: supplementary).
7
Conclusion
Trusting self-reported confidence has a failure mode: past δ ∗ (B/N ) —which rises as budgets shrink—confidence-ranked auditing is worse than random, where the five measured open-weight models’ point estimates sit. Correlation, driven by fleet-wide shared difficulty not lineage, is both a threat (clustering errors beyond any budget) and the only resource when confidence is useless; pooled transfer needs no lineage map. The vacuity criterion gives a quantitative, model-level proxy for “effective human oversight”: below it, no studied policy beats rubberstamping. Future work : a multi-provider survey, the strategic case (AI Control), sequential within-round auditing, and richer elicitation. Disclosure of Interests. The authors have no competing interests to declare.
One Human, N Agents
7
References 1. Alves, J.V., Leitão, D., Jesus, S., Sampaio, M.O.P., Liébana, J., Saleiro, P., Figueiredo, M.A.T., Bizarro, P.: Cost-sensitive learning to defer to multiple experts with workload constraints. Transactions on Machine Learning Research (2024). https://doi.org/10.48550/arXiv.2403.06906 2. Butt, T.A., Iqbal, M., Iqbal, R.: Governing what the EU AI Act excludes: Accountability for autonomous AI agents in smart city critical infrastructure. arXiv preprint arXiv:2605.01091 (2026). https://doi.org/10.48550/arXiv.2605.01091 3. Chen, X., Lin, Q., Zhou, D.: Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 28, pp. 64–72. PMLR, Atlanta, Georgia, USA (17–19 Jun 2013), https://proceedings.mlr.press/v28/chen13f.html 4. Dodge, H.F., Romig, H.G.: Sampling Inspection Tables: Single and Double Sampling. Wiley, 2nd edn. (1959) 5. Embretson, S.E., Reise, S.P.: Item Response Theory for Psychologists. Lawrence Erlbaum Associates (2000) 6. European Parliament and Council: Regulation (EU) 2024/1689 (AI Act), article 14: Human oversight (2024) 7. Green, B.: The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review 45, 105681 (2022). https://doi.org/10.1016/j.clsr.2022.105681 8. Greenblatt, R., Shlegeris, B., Sachan, K., Roger, F.: AI control: Improving safety despite intentional subversion. In: Proc. 41st International Conference on Machine Learning (ICML). PMLR, vol. 235 (2024). https://doi.org/10.48550/arXiv.2312.06942 9. Gupta, S., Chaudhari, S., Joshi, G., Yağan, O.: Multi-armed bandits with correlated arms. IEEE Transactions on Information Theory 67(10), 6711–6732 (2021). https://doi.org/10.1109/TIT.2021.3081508 10. Gupta, S., Joshi, G., Yağan, O.: Correlated multi-armed bandits with a latent random source. In: Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 3572–3576 (2020). https://doi.org/10.48550/arXiv.1808.05904 11. Hacohen, G., Dekel, A., Weinshall, D.: Active learning on a budget: Opposite strategies suit high and low budgets. In: Proc. 39th International Conference on Machine Learning (ICML). PMLR, vol. 162, pp. 8175–8195 (2022). https://doi.org/10.48550/arXiv.2202.02794 12. International Organization for Standardization: ISO 2859-3: Sampling procedures for inspection by attributes — part 3: Skip-lot sampling procedures (2005) 13. Kadavath, S., Conerly, T., Askell, A., et al.: Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 (2022). https://doi.org/10.48550/arXiv.2207.05221 14. Li, S., Phillips, J.M., Yu, X., Kirby, R.M., Zhe, S.: Batch multi-fidelity active learning with budget constraints. In: Advances in Neural Information Processing Systems 35 (NeurIPS) (2022). https://doi.org/10.48550/arXiv.2210.12704 15. Lin, S., Hilton, J., Evans, O.: Teaching models to express their uncertainty in words (2022). https://doi.org/10.48550/arXiv.2205.14334, https://arxiv.org/abs/2205.14334
8
C. Zavattari et al.
16. Mao, A., Mohri, C., Mohri, M., Zhong, Y.: Two-stage learning to defer with multiple experts. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS ’23, Curran Associates Inc., Red Hook, NY, USA (2023). https://doi.org/10.52202/075280-0159 17. Mao, A., Mohri, M., Zhong, Y.: Principled approaches for learning to defer with multiple experts. arXiv preprint arXiv:2310.14774 (2023). https://doi.org/10.48550/arXiv.2310.14774 18. Mozannar, H., Sontag, D.: Consistent estimators for learning to defer to an expert. In: Proc. 37th International Conference on Machine Learning (ICML). PMLR, vol. 119, pp. 7076–7087 (2020). https://doi.org/10.48550/arXiv.2006.01862 19. Palomba, F., Pugnana, A., Alvarez, J.M., Ruggieri, S.: A causal framework for evaluating deferring systems. In: Proc. 28th International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, vol. 258 (2025). https://doi.org/10.48550/arXiv.2405.18902 20. Pugnana, A., Ruggieri, S.: A model-agnostic heuristics for selective classification. In: Proc. 37th AAAI Conference on Artificial Intelligence. vol. 37, pp. 9461–9469 (2023). https://doi.org/10.1609/aaai.v37i8.26133 21. Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., Manning, C.D.: Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In: Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 5433–5442 (2023). https://doi.org/10.48550/arXiv.2305.14975 22. Verma, R., Barrejón, D., Nalisnick, E.: Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In: Proc. 26th International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, vol. 206, pp. 11415–11434 (2023). https://doi.org/10.48550/arXiv.2210.16955 23. Weitzman, M.L.: Optimal search for the best alternative. Econometrica 47(3), 641–654 (1979). https://doi.org/10.2307/1910412 24. Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., Hooi, B.: Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In: Proc. 12th International Conference on Learning Representations (ICLR) (2024). https://doi.org/10.48550/arXiv.2306.13063 25. Xuan, W., Zeng, Q., Qi, H., Xiao, Y., Wang, J., Yokoya, N.: The confidence dichotomy: Analyzing and mitigating miscalibration in tool-use agents. arXiv preprint arXiv:2601.07264 (2026). https://doi.org/10.48550/arXiv.2601.07264 26. Zhang, J., Xiong, C., Wu, C.S.: Agentic confidence calibration. arXiv preprint arXiv:2601.15778 (2026). https://doi.org/10.48550/arXiv.2601.15778
One Human, N Agents
9
Appendix This appendix collects illustrative material for the main paper. Every loadbearing number is reported in the main text; the figures and tables here only illustrate. Figures and tables are numbered S1, S2, . . . ; references to the main paper are given by description.
8
Synthetic phase diagrams (E1)
budget B/N
conf helps 0 random helps
H1 flip: Rrand Rconf ( = 0.5, = 0.05) 5.0
0.2
2.5
0.1
0.0
0.05
2.5
0.02
* (B/N)
0.0
0.2
5.0 0.4 0.6 miscalibration
0.8
1.0
Fig. S2. H1 flip (synthetic; N =100, F =4; theory background, ei ∼ U (0.05, 0.30)): Rrandom −Rconf over (δ, B/N ); the black curve is the flip threshold δ ∗ (B/N ), monotone in B/N . The located values, the reversed direction, and the matched-fleet placement are stated in full in the main paper (§4).
H3 vacuity: best reduction (hatched < = 0.1) 0.5
(Rnone minR )/Rnone
budget B/N
0.2
0.4
0.1
0.3
0.05
0.2
0.02
0.1 0.0
0.2
0.4 0.6 miscalibration
0.8
1.0
0.0
Fig. S3. H3 vacuity (synthetic; same background as Fig. S2): best risk reduction over the deployable policies considered, (Rnone − minπ Rπ )/Rnone ; hatched cells are vacuous (< τ =0.1). 10 of 44 cells are vacuous; at B/N =0.02 mandated oversight is rubberstamping almost regardless of δ (main paper, §4, H3).
10
9
C. Zavattari et al.
Per-model measurements (E2)
Table S1. Per-model measurements: error rate ê, AUROC of confidence for correctness, precision-based miscalibration δprec at B/N =0.1, and expected calibration error (ECE). 500 items per benchmark, temperature 0, locked single-completion elicitation. GSM8K
10
Model
ê
Qwen3-0.6B Qwen3-4B Qwen3-8B Mistral-7B-Instr. Phi-4-mini gpt-4o-mini
0.94 0.76 0.73 0.92 0.79 0.64
HotpotQA
AUROC δprec ECE 0.50 0.50 0.50 0.51 0.51 0.59
0.84 0.88 0.86 0.81 0.87 0.68
0.94 0.76 0.73 0.92 0.78 0.63
ê
AUROC δprec ECE
0.77 0.46 0.43 0.52 0.60 0.40
0.50 0.51 0.50 0.52 0.50 0.56
0.83 0.86 0.81 0.78 0.86 0.75
0.76 0.46 0.43 0.51 0.59 0.38
Measured fleet on HotpotQA
hotpotqa (fleet 0=0.70 [0.66,0.75], matched e) budget B/N
0.2 0.1 0.05 0.0
model (colour) qwen3-0.6b mistral-7b qwen3-4b phi-4-mini qwen3-8b gpt-4o-mini marker (CI verdict) below flip (CI < * ) straddling (CI * ) beyond flip (CI > * ) * band
0.2 0.4 0.6 0.8 miscalibration (measured: precision-based prec)
1.0
Fig. S4. HotpotQA counterpart of the GSM8K money diagram (main paper, Fig. 1; same axes, markers, and legend). 2/5 open-weight models are beyond the flip at B/N =0.1; gpt-4o-mini sits below it, with its closest call at B/N =0.05 (δprec 0.870 vs. δ ∗ =0.887).
11
E3 replay on real traces
The E3 replay (main paper, §5) carries all its numbers inline—the ordering, the four Holm-adjusted contrasts, and the effect sizes; these two panels illustrate it on both benchmarks.
One Human, N Agents
11
gsm8k (N=15, ê=0.83, =0.05) residual risk R
= random div sig. (paired)
12 n.s.
11 *** random conf-ranked diversity-Bayes (pooled) oracle (perfect targeting)
10 1
2
div-Bayes [family] div-Bayes [tier] no oversight
***
3
audits per round B
residual risk R
Fig. S5. E3 replay on GSM8K traces (15 agents, ε=0.05): residual risk vs. budget; stars mark the pre-specified random−diversity contrast; oracle = perfect error knowledge (reference); diversity-Bayes under family (dashed) / tier (dotted) / pooled coincides (∆ρ ≈ 0).
8.5 8.0 7.5 7.0 6.5 6.0
hotpotqa (N=15, ê=0.56, =0.05) = random div sig. (paired)
***
random conf-ranked diversity-Bayes (pooled) oracle (perfect targeting)
1
***
2
div-Bayes [family] div-Bayes [tier] no oversight
audits per round B
***
3
Fig. S6. HotpotQA counterpart of Fig. S5 (same axes and policies). The ordering oracle < diversity-Bayes < conf-ranked ≈ random holds, and family/tier/pooled groupings coincide. The contrast effect sizes (0.322 / 0.312 risk units) are larger than on GSM8K, as reported in the main text (§5).
12
H2 in the synthetic model; robustness to the copula choice
E1 grid cells with N =20, B/N =0.02 realize an integer budget B=0 (all policies coincide); they are excluded from every figure and number, and the exclusion is asserted in the figure code. The H2 advantage accumulates over rounds (2.73 at T =10, 3.47 at T =50, 3.72 at T =200; ρ=0.8) and depends critically on persistence: 0.29 at φ=0 (perround shocks) vs. 3.47 at φ=1—the transfer mechanism is learning persistent family error profiles from audits. (The persistence collapse is also reported inline in the main paper, §4.)
12
C. Zavattari et al.
= 0.5, B/N = 0.1, = 0.05, N = 100, F = 4 induced indicator correlation at e = 0.15
residual risk R (undetected errors / round)
0.0
0.1
random conf-ranked diversity-Bayes
17 16
0.2
0.3
0.4
0.5 0.6
round-robin oracle-conf
15 14 13 12 11 10
0.0
0.1
0.2 0.3 0.4 0.5 0.6 latent (copula) correlation
0.7
0.8
Fig. S7. H2 (synthetic, amended model): residual risk vs. latent ρ at δ=0.5, B/N =0.1, ε=0.05. diversity-Bayes pulls away from random as ρ grows; conf-ranked and the oracle drift slightly upward because clustered errors exceed the fixed budget more often (main paper, §4).
One Human, N Agents
13
H2 robust to correlation model ( = 0.5, B/N = 0.1, = 0.05)
diversity-Bayes advantage Rrand Rdiv
3.5
Gaussian copula (primary) Beta family-shock (variant)
3.0 2.5 2.0 1.5 1.0 0.5 0.0 0.0
0.1 0.2 0.3 0.4 0.5 error-indicator correlation (at e = 0.175)
Fig. S8. Sensitivity: a marginal-preserving Beta log-odds family-shock error model reproduces the H2 trend (advantage up to 2.94 at the strongest shock) and the H1 thresholds, so neither finding is an artifact of the Gaussian copula. Transfer-weight choice w ∈ {ρ, ρ2 , ρ/2} (3.47/3.46/3.47 at ρ=0.8) changes convergence speed, not conclusions.
Reliability diagrams
1.0
accuracy
0.8
gsm8k
1.0
perfect qwen3-0.6b (ECE=0.94) qwen3-4b (ECE=0.76) qwen3-8b (ECE=0.73) mistral-7b (ECE=0.92) phi-4-mini (ECE=0.78) gpt-4o-mini (ECE=0.63)
0.6
0.8
accuracy
13
0.4 0.2 0.0 0.0
hotpotqa perfect qwen3-0.6b (ECE=0.76) qwen3-4b (ECE=0.46) qwen3-8b (ECE=0.43) mistral-7b (ECE=0.51) phi-4-mini (ECE=0.59) gpt-4o-mini (ECE=0.38)
0.6 0.4 0.2
0.2
0.4 0.6 0.8 verbalized confidence
1.0
0.0 0.0
0.2
0.4 0.6 0.8 verbalized confidence
1.0
Fig. S9. Reliability diagrams for the six models. Open-weight confidence piles at ≈1.0 with accuracy far below the diagonal (ECE 0.43–0.94); gpt-4o-mini (black) is the only model whose confidence variation is error-correlated.
14
C. Zavattari et al.
14
Confidence parse failures and imputation
Confidence parse failures (no numeric confidence in the completion) are at most 3.2% per trace and concentrate in Qwen3-0.6B; they are zero for four of the six models. Two consumers handle them differently, by design: analysis statistics (AUROC, Var(c), δprec ) drop unparsed records, while the E3 replay imputes maximum confidence (1.0), so conf-ranked audits those outputs last. The asymmetry is declared, affects 0.6% of replay cells, and is non-differential across the conclusions: the AUROC ≈ 0.5 finding is shared by four models with zero parse failures.
15
Limitation support values
Quantifying values for the boundary statements in the main paper’s Limitations (§6); each limitation states its boundary in full, with the supporting numbers here. (a) Verifier-noise crossover at ε=0.2: at ρ=0.5 diversity-Bayes 14.27 vs. confranked 14.04 (cedes); at ρ=0.8, 13.56 vs. 14.26 (resists). (g) Parse-failure rate ≤ 3.2%, concentrated in Qwen3-0.6B (details and imputation policy in §14).