Agora: Enhancing LLM Agent Reasoning Via Auction-Based Task Allocation Kaiji Zhou Ales Leonardis University of Birmingham University of Birmingham [email protected] [email protected]
arXiv:2607.09600v1 [cs.AI] 10 Jul 2026
Abstract
a system capable of dynamic competence discovery—routing every reasoning step to the agent best suited to solve it. However, implementing this fine-grained orchestration faces two hurdles: structural alignment and trustworthy valuation. First, coarse-grained routing (at the query level) fails to exploit the subtask structures generated by planners. Second, and more critically, establishing a reliable auction is plagued by overconfidence (Huang et al., 2025). Agents often hallucinate certainty, claiming high confidence on incorrect answers. Without a reliable measure of an agent’s true probability of success, dynamic allocation risks assigning critical logic nodes to overconfident but incompetent agents, causing the reasoning chain to collapse. To address these challenges, we propose Agora, a framework that reformulates task allocation via an incentive-compatible auction mechanism. Specifically, Agora operates in two phases: a Planner decomposes the query into atomic units, and an Auction treats these units as tradeable items. Agents compete to solve them by submitting “bids” derived from their execution cost and calibrated confidence. Crucially, by employing a hierarchical calibration strategy—combining static baselines with online adaptation—Agora filters out hallucinated certainty. This ensures the auction is driven by genuine competence, allowing the system to adaptively “learn to trust” the right experts for each specific step. In summary, our contributions are as follows: • Auction-Based Reasoning Framework: We propose Agora, a framework that leverages an auction mechanism to dynamically route reasoning steps, enabling specialized agents to collaborate efficiently on complex tasks. • Competence-Driven Calibration: We introduce a strategy combining embedding-based binning with online refinement to standardize confidence estimation, ensuring that allocation is based on re-
Enhancing the reasoning capabilities of large language model (LLM) agents requires effective orchestration of diverse expert models and tools. However, existing frameworks typically call APIs based on coarse-grained matching between tasks and the functions of expert models or tools, while overlooking critical factors such as performance variability and cost efficiency among functionally similar alternatives. To address this, we propose Agora, a framework that introduces an incentive-compatible auction mechanism for dynamically allocating tasks to expert models and tools. By treating reasoning steps as tradeable items, Agora enables agents to bid based on their rectified competence— ensuring that critical logic is routed to the most capable solver rather than the most overconfident one. Evaluations across five benchmarks show that Agora improves over matched singlemodel, routing, and cascade baselines under comparable candidate pools, while exposing a controllable cost-quality trade-off through a single auction parameter.
1
Yue Feng* University of Birmingham [email protected]
Introduction
Advancing the reasoning capabilities of Large Language Models (LLMs) requires moving beyond monolithic execution. While strategies like Chainof-Thought (CoT) (Wei et al., 2023) provide a structural blueprint by decomposing queries into atomic steps, executing these complex chains often exceeds the reliable scope of any single generalist model. A critical bottleneck restricts current reasoning systems: the mismatch between task difficulty and model capability. Current paradigms often default to static assignment, routing all steps to a fixed agent, ignoring that specialized “expert” models can often outperform generalist giants on specific sub-problems (e.g., retrieval or code execution) (Dubois et al., 2024). Therefore, we need * Corresponding author.
1
liability rather than hallucinated certainty. • Empirical Validation: Extensive evaluations show that Agora improves matched-pool routing on text, scientific-code, and multimodal benchmarks, while making cost-quality behavior directly tunable.
individual tokens (fine-grained). They lack the semantic granularity to decompose complex problems into distinct sub-tasks. Our framework addresses this gap by auctioning task units, ensuring that each logic step is routed to the optimal expert.
2
Related Work
2.1
Model Selection and Specialized Reasoning
In an auction-based system, reliable routing hinges on accurate self-valuation. Early “Ask-forcalibration” methods prompted models to verbalize certainty (Tian et al., 2023), and follow-up studies refined this via prompting strategies and aggregation (Xiong et al., 2024; Yang et al., 2024). Post-hoc methods like QA-Calibration improved reliability via embedding-based grouping (Manggala et al., 2025), yet they typically rely on static calibration sets. Recent advancements leverage continuous semantic spaces for greater precision. Kernel Language Entropy (KLE) calculates uncertainty via semantic similarity kernels (Nikitin et al., 2024), while Semantic Nearest Neighbor Entropy (SNNE) aggregates pairwise similarities to handle longform generation (Nguyen et al., 2025a). Unlike prior static methods, our framework incorporates these continuous metrics into an online refinement loop. This allows agents to dynamically rectify their “bids,” effectively filtering out hallucinated certainty to maintain market fairness.
2.2
Enhancing reasoning capabilities relies on coordinating diverse expertise. The prohibitive cost of generalist models has spurred research on dynamic model selection to access specialized capabilities efficiently. Early heuristic cascades, such as FrugalGPT, sequentially query models to cut costs but suffer from serial latency (Chen et al., 2023). To address this, recent work employs learned routers. Supervised classifiers like HybridLLM (Ding et al., 2024) and Adaptive-Solver (Zhou et al., 2024) optimize accuracy under budgets. Advanced approaches utilize contextual bandits (RouteLLM (Ong et al., 2025)) or structural modeling (MetaLLM (Nguyen et al., 2025b)) to adapt to query complexity. Building on this, TensorOpera Router achieves significant throughput gains by projecting queries into a learned latent space to map inputs to experts (Stripelis et al., 2024). Parallel research pushes towards finer granularities. Techniques like Confidence-Token routing (Chuang et al., 2025) and Mixture-of-Depths (MoD) (Raposo et al., 2024) optimize compute at the token level by skipping unnecessary operations. Parallel to system-level routing, architectural Mixture-of-Experts (MoE) methods have gained prominence for scaling model capacity. Architectures like Switch Transformers (Fedus et al., 2022) and Mixtral (Jiang et al., 2024) route tokens to specific internal expert layers to optimize computational efficiency. Recent innovations like DeepSeek-MoE (Dai et al., 2024) further refine this by employing fine-grained expert segmentation. However, while these methods share the philosophy of specialization, they require end-to-end pre-training. In contrast, our framework focuses on auction-based orchestration, enabling diverse agents (e.g., proprietary planners combined with open-weight executors) to collaborate on reasoning tasks without architectural modifications. Crucially, existing systems operate at extremes: either routing the entire query (coarse-grained) or
2.3
Calibration for Reliable Valuation
Mechanism-Driven Reasoning
Breaking complex problems into steps is essential for reasoning, exemplified by Chain-of-Thought (Wei et al., 2023) and Least-to-Most prompting (Zhou et al., 2023). To relax single-model constraints, multi-agent frameworks like HuggingGPT (Shen et al., 2023), Chameleon (Lu et al., 2023), and Gradientsys (Song et al., 2025) dispatch sub-tasks to diverse specialists. Recent orchestration work further studies adaptive centralized controllers, including reinforcement-learned evolving orchestrators (Dang et al., 2025), zero-supervision inference-time MAS design (Ke et al., 2025), and holistic function-calling orchestration (Ke et al., 2026). These methods optimize the structure or sequence of multi-agent collaboration, whereas Agora focuses on a complementary mechanismdesign layer: given a planner and a candidate pool, it allocates each task unit by calibrated competence minus cost, without requiring end-to-end training or access to the internals of proprietary agents. Thus, Agora is not a jointly optimized swarm in 2
Figure 1: Overview of Agora’s auction-based reasoning framework. Given a complex query x, a planner decomposes it into a graph of dependencies, which are grouped into task units. Each unit is dynamically allocated to the optimal agent via a confidence-calibrated auction, which filters out hallucinated certainty by weighing verified competence against execution cost. Finally, the unit outputs are synthesized into the final answer Y .
which planning and allocation are co-trained; it is a plug-and-play allocation layer that can sit on top of such planners or agent systems. To ground collaborative reasoning in economic principles, we draw on Mechanism Design. Duetting et al. demonstrated that incentive-compatible auctions can align LLM agents with global system goals (Duetting et al., 2024). We apply this guarantee to resource allocation: by treating subtasks as auctionable items, we combine multi-agent decomposition with auction-theoretic optimality. This perspective also connects to work on LLMs as rational agents. Recent studies show that agent debate can improve reasoning by simulating social dynamics (Du et al., 2023). Game-theoretic analyses further find that LLMs can act rationally in bargaining and other games (Fan et al., 2023), supporting their use as bidders in our setting.
3
is driven by verified competence rather than raw, often hallucinated, certainty. The Auction module allocates resources by maximizing a utility function of the final calibrated confidence. Selected agents Execute the subtasks, and a composer synthesizes the Final Answer. Finally, the Refinement module closes the loop by using an evaluator to generate feedback, continuously updating the S ′ parameters to reduce calibration error over time. 3.1
Task Planning and Unit Extraction
The LLM Planner (f1 ) first interprets the input x to generate a directed task graph G = (V, E), where nodes represent atomic steps with defined skill requirements and data dependencies (see Appendix A for detailed prompts). Subsequently, the Task-unit Extractor optimizes this graph by merging tightly coupled nodes into coarser executable task units T . Candidate groups are identified via coupling signals—specifically shared I/O, prompt similarity, or explicit control dependencies. To ensure feasibility, a merge is finalized only if it satisfies a decision gate: (i) the resulting unit must be supported by agent capabilities and (ii) maintain structural validity within the graph.
Methodology
Our framework follows a closed-loop pipeline: Planning → Calibration → Auction → Execution → Refinement (see Fig. 1). Given a user request x, the Planning module decomposes it into a directed graph of atomic task units. In the Calibration phase, potential agents estimate their success probabilities using a nested mechanism: a pre-trained calibrator S for base accuracy, and a dynamic posthoc calibrator S ′ that adapts to distribution shifts. Crucially, this ensures that the subsequent auction
3.2
Confidence Calibration
We define the calibrated confidence p̂ij for agent aj on task ti using a hierarchical composition of static and dynamic transformations. 3
out Here, cin j and cj represent the agent’s unit execution costs (e.g., token prices for LLMs), while lj denotes its throughput (tokens/sec). These are normalized against a domain-specific reference price ceiling (Cref ) and reference throughput (Lref ). The weights wp and wl govern the trade-off between budget and speed.
Static Calibration (S). To mitigate intrinsic model overconfidence and ensure generalization across diverse domains, we employ a static calibrator S trained on a broad-spectrum corpus of diverse public benchmarks (including math, coding, and multimodal datasets; details in Appendix C). Instead of fitting a single global scaler, S applies a group-specific scaling followed by histogram binning: p̂0 = S(praw ) = Sbin σ(wgT · ϕ(praw ) + bg ) (1) where praw is the raw confidence, and parameters (wg , bg ) are specific to task group g, determined by embedding clustering (KD-tree) on the heterogeneous training data. This allows the system to retrieve appropriate calibration parameters even for unseen task types based on semantic similarity.
Winner Selection. The final bid is derived by penalizing the transformed confidence with the weighted cost: bij = (p̂ij )γ − β · Cnorm,j
The parameter β ≥ 0 acts as the global Cost Sensitivity. A higher β biases the system toward costefficiency, selecting cheaper or faster agents unless a premium model offers a substantial confidence gain. The winning agent is identified via j ∗ = arg maxj bij .
Dynamic Calibration (S ′ ). To handle distribution shifts (e.g., specific scientific workflows not present in the static corpus), we introduce an online dynamic calibrator S ′ . It refines the static estimate p̂0 via a time-variant transformation:
3.4
The parameters θt = {αt , βt } are updated online via gradient descent to minimize the negative loglikelihood of recent auction outcomes, continuously adapting to the test-time distribution. Auction and Execution
Bid Construction. For each task unit ti and candidate agent aj , we compute a scalar bid bij based on two components. First, we apply a concave power-law transformation to the calibrated confidence p̂ij : v(p̂ij ) = (p̂ij )γ ,
with γ ∈ (0, 1].
Refinement Loop
To handle distribution shifts in long-horizon deployment, the final stage optionally closes the loop by updating the dynamic calibrator S ′ . This module is designed to be adaptive: it activates when a feedback mechanism (e.g., unit tests for code or a lightweight verifier) provides a binary correctness label ylabel ∈ {0, 1}. In our experiments, to establish a theoretical upper bound, we utilize ground truth labels as the feedback signal. This label is used to update the parameters θt of S ′ via gradient descent:
p̂final = S ′ (p̂0 ; θt ) = σ (αt · logit(p̂0 ) + βt ) (2)
3.3
(5)
θt+1 ← θt − η∇θ LBCE (S ′ (p̂0 ; θt ), ylabel )
(6)
This online update allows the system to progressively align with the test distribution when reliable feedback is available.
(3)
This transformation compresses the highconfidence regime, making the mechanism less sensitive to minor fluctuations in high-probability estimates (e.g., distinguishing 0.99 from 0.95 becomes less critical than 0.60 from 0.50). Second, we calculate a Normalized Cost Cnorm,j ∈ [0, 1], defined as a weighted sum of monetary expense and latency: ! out cin + c j j Cnorm,j = wp · min , 1 Cref (4) Lref + wl · min , 1 , lj
4
Experiments
4.1
Datasets
MuSiQue-Ans: We use MuSiQue-Ans (Trivedi et al., 2022), a multi-hop QA benchmark constructed from connected single-hop questions. It provides explicit question decompositions and a training split for calibration. We report Exact Match (EM) and token-level F1 on a fixed 500example development subset. MMLU-Pro: To evaluate robustness, we use MMLU-Pro (Wang et al., 2024b). Due to computational constraints, we evaluate on a fixed stratified random subset of 2,000 examples shared by all compared methods, reporting the overall accuracy. 4
Table 1: Overview of baseline methods and their compatibility with text-based versus multimodal inference tasks (✓: Supported; ×: Not supported/evaluated). Type
Method
Text
Vision
Static
Single Backend Random Router
✓ ✓
✓ ✓
Heuristic
1NN Router (Text Embed)
✓
×
Cascade
Consistency Cascade FrugalGPT
✓ ✓
✓ ✓
Learned
Adaptive-Solver Hybrid LLM (Best-Route)
✓ ✓
× ×
Table 2: Candidate backend pools used for each benchmark. Model roles describe the matched pool used for all methods on that benchmark, not cross-benchmark rankings. Backend 1
Backend 2
MuSiQue-Ans MMLU-Pro SciCode SPIQA MathVision
Mistral-Small-3.2-24B Mistral-Small-3.2-24B xai/grok-4-1 xai/grok-4-1-fast-vision xai/grok-4-1-fast-vision
Qwen3-14B Qwen3-14B openai/gpt-5-mini Qwen3-VL-Thinking Qwen3-VL-Thinking
mize routing. Adaptive-Solver (Zhou et al., 2024) utilizes a CodeBERT classifier to predict model success and skip larger models for simpler queries. Hybrid LLM (Ding et al., 2024) (Best-Route) explicitly models query difficulty and agent expertise to maximize expected accuracy under specific budget constraints.
SciCode: This benchmark simulates scientific workflows (Tian et al., 2024). We report Pass Rates on the full test split. Following official recommendations, we utilize the with_background setting to strictly test knowledge retrieval. SPIQA: For multimodal context, we use the Test-A split of SPIQA (Pramanick et al., 2025), reporting Retrieval Accuracy and L3 Score. MathVision: We assess visual mathematical reasoning using the testmini split of MathVision (Wang et al., 2024a), reporting Pass@1. 4.2
Benchmark
4.3
Implementation
Model Selection. To test the framework’s adaptability while keeping comparisons matched within each benchmark, we use the benchmark-specific candidate pools in Table 2. All routing baselines and Agora variants on a benchmark draw from the same pool. System Configuration. To demonstrate the controllability of Agora, we define three operating modes based on the cost sensitivity β: (1) QualityFirst (β = 0.001): Prioritizes accuracy, using cost only as a tie-breaker. This is our default setting for matched-pool accuracy comparisons. (2) Balanced (β = 0.1): Seeks a trade-off between performance and budget. (3) Cost-Efficient (β = 0.25): Aggressively optimizes for savings, comparable to frugal baselines. In our main results (Table 3), we report the Quality-First performance to compare matched candidate pools, while the trade-off characteristics are analyzed in Sec. 5.4. The complete defaults are summarized in Appendix B. All three modes reuse the same calibrated competence scores; only the cost penalty changes. Thus the sensitivity curve measures allocation behavior under a fixed router rather than retuning for each budget. For replay-based comparisons, we use cached candidate outputs and report bootstrap 95% confidence intervals where appropriate; benchmark-specific calibrator settings are listed in Appendix B.
Baselines
We compare our proposed framework against a diverse set of baselines, ranging from static singlemodel executions to advanced routing and cascading strategies. Since not all baselines support multimodal inputs, we categorize them based on their compatibility with text-based and visual inference tasks (see Table 1). Static and Heuristic Baselines: To establish performance bounds, we evaluate single-backend executions using the benchmark-specific candidate pool in Table 2. We also include a Random Router as a stochastic lower bound. For retrieval-based routing, we employ a 1NN Router that selects agents based on text similarity for language tasks. Cascading Strategies: To address cost-efficiency, we evaluate sequential frameworks. Consistency Cascade (Wang et al., 2023) starts with a smaller model and escalates to a stronger one only if the consistency of sampled outputs falls below a threshold. FrugalGPT (Chen et al., 2023) queries a chain of models ordered by cost, stopping early if the answer is deemed reliable; we adapt this for visual tasks by chaining vision-language models. Learned Routers: Finally, we compare against representative learned predictors trained to opti-
Planning Configuration. While Agora can employ an LLM planner, we use planner-free task units whenever a benchmark already provides a reliable structure. For MuSiQue-Ans, we use the 5
Table 3: Text-based Reasoning Results. Columns use benchmark-standard metrics: MuSiQue EM/F1, MMLU-Pro accuracy, and SciCode Sub/Main pass rates. Vanilla (Strong/Weak) refers to single-backend execution from the benchmark-specific pool in Table 2. For MMLU-Pro, Agora (S/W) denotes the Strong or Weak model as the LLM planner; MuSiQue and SciCode are planner-free. Agora results use the Quality-First configuration (β = 0.001).
dataset-provided question decomposition; for SciCode, we use the official sub-problems; and for SPIQA, we use its natural retrieval–reasoning structure. Only MMLU-Pro uses an LLM planner to decompose queries into explicit reasoning steps. This separation lets us evaluate the auction mechanism independently from planning quality on the benchmarks with intrinsic decompositions. It also keeps the MuSiQue and SciCode protocols aligned: all compared routers receive the same units, so differences mainly reflect allocation rather than decomposition quality.
Methods
Baseline Implementation Details. To ensure a rigorous comparison, we categorize baselines into training-free strategies and learned routers. For Learned Baselines (Adaptive-Solver, Hybrid LLM, 1NN), we strictly isolated training data to avoid leakage where those baselines are evaluated: (1) For MuSiQue-Ans, the 200-example official training subset is used for calibration and kept disjoint from the development evaluation subset; (2) For MMLU-Pro, we utilized the remaining examples excluding our stratified test subset; (3) For SciCode, we used the official validation split. For Cascading Strategies, we implemented inferencetime variants without parameter updates: Consistency Cascade generates k = 3 samples and escalates if agreement is low, while FrugalGPT employs an LLM-based verifier (confidence threshold ≥ 0.7) to substitute the original trained scoring function. On MuSiQue, these cascade baselines operate at the original-query level, while Agora and replayable routers use the dataset-provided decomposition. Detailed hyperparameters (e.g., consistency thresholds) are provided in Appendix F. 4.4
MuSiQue
MMLU-Pro
SciCode
EM
F1
Acc.
Sub
Main
Vanilla (Strong) Vanilla (Weak) Random Router Consist. Cascade 1NN Router Adaptive-Solver Hybrid LLM FrugalGPT
33.6 25.2 39.2 26.4 42.4 41.0 41.0 26.0
44.1 35.5 50.4 37.3 54.1 52.6 52.6 36.5
68.1 64.4 66.3 67.0 68.3 69.3 64.5 66.7
44.2 39.2 42.3 45.4 43.0 43.2 43.2 44.4
13.6 12.4 12.6 14.1 12.3 13.6 11.2 13.6
Agora (S) Agora (W)
43.0
54.3
71.9 69.1
44.4
13.8
Table 4: Multimodal Results. Comparison on SPIQA and MathVision (MV). For SPIQA, Ret denotes Retrieval Accuracy; Avg, ≥0.6, and ≥0.8 refer to the GPT4o-evaluated L3 Reasoning Scores. SPIQA
Methods
Math Vision
Ret
Avg
≥0.6
≥0.8
Vanilla (Grok) Vanilla (Qwen) Random Router Consist. Cascade FrugalGPT
85.4 67.3 72.0 72.6 75.2
60.5 55.2 57.6 55.5 61.7
65.6 59.2 62.2 60.0 64.4
43.4 48.2 44.4 45.6 45.1
50.3 53.3 50.0 51.4 54.6
Agora
84.9
65.0
72.4
56.9
55.3
not significantly exceed it. Given the high variance observed in this benchmark, we report the mean pass rate over three runs. We attribute this plateau to distribution shift: the calibrator is trained on competitive programming (e.g., Codeforces) rather than SciCode’s scientific workflows, which weakens expert discrimination. Still, Agora avoids the collapse seen in learned routers. This pattern is consistent with our claim: Agora helps most when the candidate agents have separable strengths, and it degrades to competitive routing when the calibrated signals are less domain-aligned. The planner-free MuSiQue and SciCode settings make this interpretation conservative because they remove planner quality as an explanation for the routing gains. Multimodal Reasoning. Table 4 validates the framework’s generality in visual domains. On SPIQA, Agora achieves
Results
We analyze performance across text-based and multimodal benchmarks (Tables 3 and 4). Text-based Reasoning. Agora improves or remains competitive on text reasoning tasks under matched candidate pools. On MuSiQue-Ans, Agora reaches 43.0 EM / 54.3 F1 (F1 95% CI: [50.6, 58.2]), slightly above the replayed 1NN router (42.4/54.1) and ahead of single-backend and cascade baselines. On MMLU-Pro, Agora reaches 71.9%, improving over the best single-backend baseline (68.1%) and confirming the auction’s ability to identify subtle capability differences. On SciCode, Agora matches the best single-backend baseline on sub-problem pass rate (44.4%) but does 6
Table 6: MMLU-Pro architectural ablation on the same 2,000-example stratified evaluation split. Configuration Vanilla (single model) + Task planning + Planning + auction
Accuracy
∆
68.1 70.0 71.9
– +1.9 +3.8
Table 7: Effect of online refinement on MMLU-Pro accuracy. The static-only row makes the refinement gain directly auditable. Calibration Variant
MMLU-Pro Acc.
Static calibrator only (S) Static + online refinement (S + S ′ )
Figure 2: Reliability diagrams comparing raw (red) versus calibrated (green) confidence scores. Metrics show ECE and MCE before and after calibration.
by routing harder steps to stronger agents based on calibrated confidence. Appendix E reports a smallsample math stress test with the same qualitative planning-then-auction pattern; we treat it only as supporting evidence for this trend, not as part of the main benchmark suite.
Table 5: Calibration ablation. MuSiQue reports F1, SPIQA reports L3≥0.8 accuracy, and other columns report accuracy/pass rate; parentheses show absolute change over Vanilla. MuSiQue F1
MMLU-Pro Acc.
SciCode Sub
SPIQA (L3≥0.8)
Math Vision
Vanilla
44.1
68.1
44.2
48.2
53.3
Auction ((No Calibration))
51.4
67.5
42.3
46.9
49.3
(+7.4)
(-0.6)
(-1.9)
(-1.3)
(-4.0)
54.3
71.9
44.4
56.9
55.3
(+10.2)
(+2.9)
(+0.2)
(+8.7)
(+2.0)
Configuration
Auction (With Calibration)
Impact of Online Refinement (S ′ ). Beyond architectural changes, we analyze the benefit of the dynamic calibrator (S ′ ). While static calibration (S) suffices for smaller datasets, our main result for MMLU-Pro (Table 3) incorporates online refinement to handle long-horizon streams. Compared to a static-only baseline, adding S ′ yields an accuracy improvement of +1.2% (reaching 71.9% vs. 70.7%). This gain confirms that the online update rule (Eq. 2) effectively mitigates distribution shifts during large-scale inference. Since this refinement uses correctness feedback, we treat S ′ as an upper-bound analysis unless a benchmark provides reliable automatic feedback such as unit tests, extractive verification, or tool-based validators. The deployment default remains the static calibrator when such feedback is unavailable. This distinction keeps the main framework usable in black-box settings while isolating the extra value of feedback when it exists.
the strongest result among the matched multimodal routing baselines with a 65.0% Average L3 score, surpassing FrugalGPT by over 3 percentage points. Notably, on the strict metric (≥0.8), Agora achieves 56.9%, a substantial improvement over the best single model (Qwen: 48.2%). This shows that the auction combines Grok’s retrieval strength (84.9%) with Qwen’s visual reasoning strength into a system stronger than either model. Because SPIQA uses a fixed retrieval–reasoning structure, this gain provides additional planning-independent evidence that the auction exploits functional complementarity.
5
Discussions
5.1
Ablation Study
70.7 71.9
5.2
Impact of System Architecture. As shown in Table 6, both decomposition and dynamic routing drive gains on MMLU-Pro under the same evaluation split. Task Planning reduces problem complexity for the strong model, and the Auction Mechanism adds a further +1.9% on MMLU-Pro
Calibration Efficacy Analysis
A central premise of our auction mechanism is that an agent’s bid must accurately reflect its probability of success. We evaluate this from two perspectives: intrinsic reliability (ECE metrics) and extrinsic impact (downstream accuracy). 7
Figure 4: Impact of cost sensitivity β on MathVision. Increasing β shifts allocation toward the cost-efficient agent (Grok), reducing normalized cost at the expense of marginal accuracy drops.
Figure 3: Mechanism Analysis on SPIQA. (a) Functional complementarity: Grok excels in retrieval, Qwen in reasoning. (b) Dynamic orchestration: The system strategically shifts logic-intensive sub-tasks to Qwen.
while Qwen3-VL-Thinking is stronger at strict reasoning (48.2% vs. 43.4%). Figure 3 shows that Agora sends 81.1% of retrieval subtasks to Grok and 32% of harder reasoning subtasks to Qwen.
5.2.1 Intrinsic Reliability Figure 2 visualizes the reliability diagrams for six representative agents. The curves consistently deviate below the diagonal (y = x), indicating that when models predict high confidence (e.g., > 0.8), their empirical accuracy is substantially lower. For instance, grok-4-1-fast-reasoning and qwen3-14b show high Expected Calibration Errors (ECE) of 0.193 and 0.357, respectively, rendering their raw logits unreliable for bidding. Applying our hierarchical calibrator (green curves) effectively linearizes these probabilities. ECE drops significantly across the board—most notably, grok-4-1 improves by an order of magnitude (0.222 → 0.023). Crucially, this also holds for Qwen3-VL (ECE 0.182 → 0.100), enabling comparable bids across modalities.
5.4
The cost sensitivity β provides a direct deployment knob. On MathVision, increasing β raises Grok usage from 21.5% to 78.5% while accuracy moves from 55.26% to 51.06% (Figure 4); the balanced setting (β = 0.1) nearly doubles Grok usage with only a ≈ 0.9-point drop. These points make explicit how much accuracy is traded for each increase in cheap-agent usage. The auction layer adds ∆Calls = |T |×|A| short bid calls. In our two-agent settings with 2–3 units per query, this is about 4–6 calls, small relative to reasoning-model executor calls. Since bid prompts ask only for scalar competence estimates, they can run in parallel before solving and do not scale with the length of final reasoning traces. This overhead is therefore separate from the execution cost controlled by β. The extra token budget is bounded by the number of task units and candidate agents, rather than by answer length. For larger candidate pools, the same interface supports a top-k pre-selection stage or grouping of nearduplicate agents, which limits bid traffic without changing the auction rule.
5.2.2
Extrinsic Impact: Preventing the Winner’s Curse Calibration is an economic necessity. Without it, overconfident agents can win bids; Table 5 verifies this effect. Calibration improves the auction across modalities: it raises MuSiQue from 51.4 to 54.3 F1, MMLU-Pro from 67.5% to 71.9%, and SPIQA strict accuracy from 46.9% to 56.9%. On MathVision, it turns a 4.0% uncalibrated loss into a +2.0% gain, underscoring its role as a guardrail against hallucinated certainty. The negative uncalibrated results are important because they show that auction structure alone is insufficient; reliable bid scores are the mechanism that prevents systematic over-allocation to overconfident agents. This confirms that reliable bid scores, not auction structure alone, determine useful winners. 5.3
Cost-Quality Trade-off and Overhead
6
Conclusion
Agora casts step-level reasoning allocation as a calibrated auction among candidate agents. Across five benchmarks, it improves or stays competitive with matched-pool routing baselines, especially when agents have complementary strengths, and exposes a clear cost-quality control without retraining. By keeping modules separate, Agora can use dataset-provided steps or planner outputs and admit new backends through the same bid rule.
Mechanism of Complementarity
SPIQA gains arise from functional orthogonality: Grok is stronger at retrieval (85.4% vs. 67.3%), 8
Limitations
Paul Duetting, Vahab Mirrokni, Renato Paes Leme, Haifeng Xu, and Song Zuo. 2024. Mechanism design for large language models. Preprint, arXiv:2310.10826.
Agora has three practical boundaries. Calibration Generalization. The auction depends on calibrated competence estimates. Under distribution shift, a deployment should fall back to the strongest agent or require a larger bid margin before assigning work to cheaper agents.
Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao He. 2023. Can large language models serve as rational players in game theory? a systematic analysis. Preprint, arXiv:2312.05488. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Preprint, arXiv:2101.03961.
Dependency on Planning. Poor decompositions or tightly coupled steps can violate the independentunit assumption. In such cases, adjacent steps should be merged and routed as blocks.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1–55.
Model Composition. Gains require meaningful agent complementarity. If one model dominates, or candidates are nearly identical, candidate preselection or single-model execution is preferable.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, MarieAnne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, and 7 others. 2024. Mixtral of experts. Preprint, arXiv:2401.04088.
References Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. Preprint, arXiv:2305.05176. Yu-Neng Chuang, Prathusha Kameswara Sarma, Parikshit Gopalan, John Boccio, Sara Bolouki, Xia Hu, and Helen Zhou. 2025. Learning to route llms with confidence tokens. Preprint, arXiv:2410.13284.
Zixuan Ke, Yifei Ming, Austin Xu, Ryan Chin, XuanPhi Nguyen, Prathyusha Jwalapuram, Jiayu Wang, Semih Yavuz, Caiming Xiong, and Shafiq Joty. 2026. Mas-orchestra: Understanding and improving multiagent reasoning through holistic orchestration and controlled benchmarks. Preprint, arXiv:2601.14652.
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. Preprint, arXiv:2401.06066.
Zixuan Ke, Austin Xu, Yifei Ming, Xuan-Phi Nguyen, Ryan Chin, Caiming Xiong, and Shafiq Joty. 2025. Mas-zero: Designing multi-agent systems with zero supervision. Preprint, arXiv:2505.14996.
Yufan Dang, Chen Qian, Xueheng Luo, Jingru Fan, Zihao Xie, Ruijie Shi, Weize Chen, Cheng Yang, Xiaoyin Che, Ye Tian, Xuantang Xiong, Lei Han, Zhiyuan Liu, and Maosong Sun. 2025. Multi-agent collaboration via evolving orchestration. Preprint, arXiv:2505.19591.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, KaiWei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. Preprint, arXiv:2304.09842.
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid llm: Cost-efficient and quality-aware query routing. Preprint, arXiv:2404.14618.
Putra Manggala, Atalanti Mastakouri, Elke Kirschbaum, Shiva Prasad Kasiviswanathan, and Aaditya Ramdas. 2025. Qa-calibration of language model confidence scores. Preprint, arXiv:2410.06615.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. Preprint, arXiv:2305.14325.
Dang Nguyen, Ali Payani, and Baharan Mirzasoleiman. 2025a. Beyond semantic entropy: Boosting llm uncertainty quantification with pairwise semantic similarity. Preprint, arXiv:2506.00245.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2024. Alpacafarm: A simulation framework for methods that learn from human feedback. Preprint, arXiv:2305.14387.
Quang H. Nguyen, Thinh Dao, Duy C. Hoang, Juliette Decugis, Saurav Manchanda, Nitesh V. Chawla, and Khoa D. Doan. 2025b. Metallm: A high-performant and cost-efficient dynamic framework for wrapping llms. Preprint, arXiv:2407.10834.
9
Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. Kernel language entropy: Finegrained uncertainty quantification for llms from semantic similarities. Preprint, arXiv:2405.20003.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2025. Routellm: Learning to route llms with preference data. Preprint, arXiv:2406.18665.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024b. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Preprint, arXiv:2406.01574.
Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. 2025. Spiqa: A dataset for multimodal question answering on scientific papers. Preprint, arXiv:2407.09413.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. Preprint, arXiv:2404.02258.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. Preprint, arXiv:2306.13063.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Preprint, arXiv:2303.17580.
Daniel Yang, Yao-Hung Hubert Tsai, and Makoto Yamada. 2024. On verbalized confidence scores for llms. Preprint, arXiv:2412.14737.
Xinyuan Song, Zeyu Wang, Siyi Wu, Tianyu Shi, and Lynn Ai. 2025. Gradientsys: A multi-agent llm scheduler with react orchestration. Preprint, arXiv:2507.06520.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. Preprint, arXiv:2205.10625.
Dimitris Stripelis, Zijian Hu, Jipeng Zhang, Zhaozhuo Xu, Alay Dilipbhai Shah, Han Jin, Yuhang Yao, Salman Avestimehr, and Chaoyang He. 2024. Tensoropera router: A multi-model router for efficient llm inference. Preprint, arXiv:2408.12320.
Jianpeng Zhou, Wanjun Zhong, Yanlin Wang, and Jiahai Wang. 2024. Adaptive-solver framework for dynamic strategy selection in large language model reasoning. Preprint, arXiv:2310.01446.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. Preprint, arXiv:2305.14975.
A
Prompt Templates
To ensure reproducibility, we provide the exact system prompts used in the Agora pipeline. These templates are stored as plain text within the codebase.
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, and 11 others. 2024. Scicode: A research coding benchmark curated by scientists. Preprint, arXiv:2407.13168.
A.1
Planner (Plan Graph Generation)
The planner is implemented in aucteam/planner_agent/. The prompt is assembled by concatenating the SYSTEM_PRIMER, few-shot exemplars, and OUTPUT_INSTRUCTIONS.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.
Rationale for Plan Constraints. We explicitly instruct the planner to limit output to a maximum of 4 steps. Empirical observation suggests that while longer plans offer finer granularity, they significantly increase the probability of cascading failures. The 4-step limit serves as a heuristic regularization.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. 2024a. Measuring multimodal mathematical reasoning with math-vision dataset. Preprint, arXiv:2402.14804.
10
System Primer.
A.3
You are an expert research planner. Break complex requests into modular steps with explicit data/control dependencies. Represent every plan as a single JSON object that conforms to this schema:
Before the auction, agents self-report a scalar confidence praw . Rate your confidence (0-1) that you can correctly implement this function:
- Root object: * steps (list): ordered execution steps. * edges (list): dependencies between IDs. * notes (str): clarifications.
{step_description_prompt} # truncated to 800 chars Function signature: {function_header}
- Step object template: {"id": "T1", "description": "Outcome", "skills": ["capability"], "reads": ["art_id"], "writes": ["art_id"]} * id: unique string starting with 'T'. * description: what to accomplish. * skills/reads/writes: always lists.
Respond with ONLY a number between 0 and 1 (e.g., 0.85).
B
- Edge object template: {"from": "T1", "to": "T2", "type": "data", "why": "Rationale"} * type: "data" or "control".
Hyperparameters and Defaults
Table 8: Default hyperparameters used in the auction and experiment configuration. Dataset-specific deviations are stated in the corresponding experiment scripts.
Ensure the JSON is syntactically valid.
Output Instructions. Respond with exactly one JSON object, followed by <END_JSON> on its own line. Do not include Markdown fences. CRITICAL - Plan Requirements: 1. MAXIMUM 4 STEPS. 2. Keep step descriptions HIGH-LEVEL. 3. Do NOT embed: - Specific numerical calculations - Concrete formulas or equations - Intermediate computation results 4. Describe the GOAL, not the METHOD.
A.2
Self-Reported Confidence Prompt
Parameter
Default / Values
Confidence exponent γ Quality-first β Balanced β Cost-efficient β Max planner steps / units Consistency cascade samples k Consistency threshold τ FrugalGPT verifier threshold MuSiQue local model loading MuSiQue client cache MuSiQue decoding MuSiQue calibration subset MuSiQue calibrator bins MuSiQue evaluation subset Bootstrap intervals
0.5 0.001 0.1 0.25 4 3 0.6 0.7 4-bit NF4 quantization persistent; per-call fallback for OOM greedy, seed 9552 200 train examples 2 500 dev examples 1,000 resamples
Executor (Task-Unit Execution)
Each task unit is executed with a structured prompt that isolates the unit’s context.
C
Calibration Implementation Details
Our calibration framework is designed to ensure robustness across diverse task types. We explicitly distinguish the training data sources for the static and dynamic components to balance generalization with adaptability.
Task-Unit Prompt Template. Overall task: {task_description} Executing unit {unit_id} Steps in this unit: - {node_id}: {step_description} - ...
Static Calibrator (S): Broad-Spectrum Training. To enable the static calibrator to adapt to unseen task types (e.g., domain-specific scientific problems) and minimize the risk of distribution shift, we trained it on a comprehensive, largescale corpus aggregated from diverse public benchmarks. Instead of relying on simple binning which fails to capture the complexity of broad-domain data, we utilized the ground-truth labels from these datasets to train the Hierarchical Scaling (HSQAB) parameters. Crucially, we prioritized using validation or test splits from source benchmarks where possible to minimize overlap with the pretraining corpora of large language models. The training corpus is categorized by modality:
Skills: {skills} ======================================= INPUTS FROM PRIOR STEPS: ======================================= [{artifact_name}]: {artifact_content} # truncated to 8000 chars per artifact ======================================= Instructions: - If final unit: - This is the FINAL step. - {final_answer_hint_1} - Else: - Focus only on this unit. - Provide intermediate findings. - Do not give the final answer yet. - Use INPUTS FROM PRIOR STEPS. Context: {optional_context}
11
Table 9: Supplementary AIME results: matched-pool routing and architecture ablation.
• Text-based Reasoning Tasks: We aggregated datasets covering mathematics, coding, and commonsense reasoning, including: ACEReason (Math), AGIEval, MuSiQueAns (Training split), ARC-Challenge/Easy (Train/Val/Test), Codeforces, CSQA, GSM8K, HumanEval, MMLU (Subset), and TACO. • Multimodal Tasks: We utilized visual reasoning datasets including MathVerse (Test), SPIQA (Validation), ChatQA, and Geometry3K.
F.2
Supplementary AIME 2025 Results
AIME 2025 provides a compact math stress test. The benchmark has 30 examples, so one additional correct answer changes accuracy by 3.33 percentage points; we therefore use it only as a smallsample check of the planning-then-auction trend.
F.1
Training Data Construction
Configuration of Cascading Baselines
Consistency Cascade: This method operates without training. We configured it to first query the cost-efficient model. We generate k = 3 reasoning paths; if the self-consistency agreement (vote ratio) is below τ = 0.6, the system escalates to the strong model. FrugalGPT: We adapt FrugalGPT (Chen et al., 2023) by replacing its trained accept/reject scorer with an LLM-as-a-Judge confidence check. The weak response is accepted when the judge score is ≥ 0.7; otherwise, the query cascades to the strong model. This keeps the baseline training-free and matched across benchmarks.
Task-Unit Extraction. For tightly coupled plans, we merge nodes before bidding when they satisfy one of three heuristic gates: data reuse within a bandwidth of 2, prompt similarity above a Jaccard threshold of 0.6, or an explicit control dependency. Overlapping candidate groups are unioned, then accepted only when the merged block preserves executable inputs/outputs and its estimated routing savings exceed the merge threshold.
Baseline Implementation Details
Acc.
• SciCode: Following standard protocol, we utilized the provided validation split (featuring distinct problems from the test set) for training.
Algorithm Details
F
89.5 Vanilla 89.5 71.7 + Planning 90.0 76.7 + Planning + auction 93.3 89.5 86.7 86.7 93.3 90.0
• MMLU-Pro: The original dataset contains 12k+ examples. We reserved our stratified 2,000-sample test set and used the remaining ∼10k examples for training learned routers.
Dynamic Calibrator (S ′ ): Historical Adaptation. While the static calibrator targets broad generalization, the dynamic calibrator S ′ is trained on the historical interaction data accumulated during the current inference session. This allows the system to correct residual errors and adapt to the specific distribution of the target task in real-time.
E
Acc. Configuration
Vanilla (Strong) Vanilla (Weak) Random Router Consist. Cascade Adaptive-Solver FrugalGPT Agora (S) Agora (W)
training split, stratified by hop count and disjoint from the development evaluation subset.
This extensive training ensures that S provides a robust baseline probability estimate, enabling the system to generalize even to benchmarks not seen during calibration (e.g., scientific workflows).
D
Method
For baselines requiring supervision (1NN Router, Adaptive-Solver, Hybrid LLM), we constructed training sets distinct from the evaluation splits: • MuSiQue-Ans: We used a fixed 200-example calibration/training subset from the official 12