Maestro Order: A Model-Agnostic Orchestration Harness On Organizing Intelligence for Reliable Problem Solving Hidayet Aksu
arXiv:2606.23983v1 [cs.CR] 22 Jun 2026
Abstract A single forward pass of a capable model is a fast, fluent, and unreliable problem-solver: it is right often enough to be useful and wrong often enough to be dangerous; in language models, such confident errors are known as hallucinations. We present Maestro Order, a model-agnostic orchestration harness that turns unreliable solvers into reliable problem-solving systems by composing them according to four structural primitives (decompose, ensemble, verify, and recurse) and a budget-aware controller that decides where to spend compute. The harness treats any model as a black-box base solver behind a uniform interface, layers a verifier ensemble whose discrimination is measured online, and allocates verification and voting to the stages with the highest marginal reliability per unit cost. We give the architecture, the message and state schema, the controller algorithm, and the engineering that makes it deterministic, observable, and fault-tolerant. We then specify an evaluation methodology (reliability at fixed cost, coverage, calibration, and ablations) and report results from a faithful Monte Carlo simulation of the harness over a parameterized solver/verifier model. The simulation reproduces the predicted laws quantitatively: verification amplifies reliability geometrically (e.g. 0.55 → 0.98 with two gates, → 0.999 with four), voting helps only above chance and is limited by shared errors, and a budget-aware controller reaches a target reliability at a small fraction of the cost of voting alone by selecting the cheapest mechanism for each regime. We close with failure modes (verifier gaming, correlated errors, and decomposition error compounding) and concrete guidance: build robust checkers, diversify solvers, and let the controller put compute where the information is. Maestro Order is the framework layer of a two-part program (Theory: Odds Law → Framework: Maestro Order), built on the laws of the companion Odds Law report.
Keywords LLM orchestration, agents, verification, self-consistency, test-time compute, reliability, ensembles, scaffolding, evaluation, hallucination mitigation, long-horizon agents
1
Introduction
Modern models answer hard questions in one shot with impressive fluency and uneven reliability. For deployment, where a wrong answer has a cost, the relevant question is not “how good is the model on average?” but “how do we build a system whose answers we can trust at a known cost?” The same question recurs across substrates: a research group made of fallible people, a compiler made trustworthy by its test suite, a distributed job made correct by re-execution. The answer, in every case, is organization: many unreliable attempts, arranged so their errors are caught and outvoted. Two failure patterns make this urgent for language models: ∗ Code and reproduction scripts: https://github.com/hidayetaksu/maestro-order
hallucination, the confident generation of incorrect or unsupported content, and the compounding of small per-step error rates that makes long-running, multi-step jobs fail almost surely without correction. This paper is the applied counterpart to a companion theory report, Odds Law [1], which shows that a small algebra of four combinators (sequential decomposition, parallel ensembling, verification gating, and recursion) generates the organizations that solve problems reliably, and that reliability flows through them by explicit laws (most importantly, a verification gate multiplies the odds of correctness by the verifier’s likelihood ratio). Here we turn those laws into a running system: a harness that wraps any model as a base solver and composes solvers and verifiers under a controller that spends compute where it buys the most reliability. Why a harness, not a better model. Three properties make orchestration attractive and complementary to improving the base model. (i) Generate–verify asymmetry: for many tasks, checking a candidate (run the tests, recompute the sum, search for a counterexample) is far cheaper and more reliable than producing it; this is exactly the regime where verification gating pays off. (ii) Test-time compute is elastic: a harness can trade more calls for more reliability on the instances that need it, and abstain or escalate on the rest. (iii) Model-agnosticism: the same scaffolding improves whatever model sits behind the interface, and survives model upgrades. Contributions. • A model-agnostic harness architecture (§4–§6) that instantiates the four combinators as composable components (planner, solver pool, verifier ensemble, aggregator, controller, blackboard, and tool sandbox) behind a uniform solver interface. • A budget-aware controller (§7) that allocates voting and verification by greedy marginal log-odds per unit cost, the operational form of the theory’s water-filling optimum (spend where the return is highest until all options return the same), with online estimation of verifier discrimination Λ. • An engineering account (§11) of determinism, idempotency, concurrency, cost accounting, and tracing required to make reliability measurable and reproducible. • An evaluation methodology and simulation study (§12–§13) that reproduces the predicted reliability laws and quantifies the cost of voting vs. verification, the diversity floor, and the value of abstention. • A catalog of failure modes and mitigations (§15): verifier gaming (Goodhart), correlated solver errors, and decomposition error compounding. • An open-source release of the harness code, simulation, and reproduction scripts at https://github.com/hidayetaksu/ maestro-order.
2
Background and Related Work
Eliciting and aggregating reasoning. Chain-of-thought prompting elicits intermediate steps [2]; self-consistency samples many chains and takes the majority answer, the voting combinator in practice [3]. Tree-of-thoughts searches over partial solutions with look-ahead [4], a decomposition-plus-search organization.
Table 1: The four combinators and their harness realization. The reliability law column summarizes the companion theory [1].
Acting, checking, and refining. ReAct interleaves reasoning with tool actions [5]; Reflexion [6] and Self-Refine [7] iterate generate– critique–revise loops; multi-agent debate has models cross-examine each other [8]. These are instances of verification and refinement with the model (or a tool) as verifier.
Combinator
Harness component
Reliability law
Sequential Parallel Verify Recurse
Planner / decomposer Voter / aggregator Verifier ensemble + gate Recursive dispatch
Î errors compound ( 𝜌𝑖 ) amplify if 𝑝 > 12 odds × Λ per gate union bound over nodes
problem 𝑥
Controller (budget, policy)
Verifiers and process supervision. Training explicit verifiers improves math word-problem reliability [9]; process-level reward models that check each step outperform outcome-only checks [10]; “LLM-as-judge” uses a model to score candidates [11]. Our harness treats any such checker as a gate and measures its discrimination Λ online.
Planner / Decomposer
Tools, retrieval, and code execution. Grounding answers in retrieval [12], tool use [13], and program execution / unit tests [14] supplies robust verifiers whose false-acceptance is bounded independently of how the candidate was produced, which is precisely the property the theory shows is needed under optimization pressure. Retrieval grounding is also a primary defense against hallucination [15]. Reliable systems from unreliable parts. The organizing idea predates models: von Neumann’s reliable automata from unreliable components [16], Condorcet’s jury theorem [17], and divide-andconquer in distributed systems [18]. Our harness is a contemporary, model-agnostic realization, and our companion paper [1] supplies the reliability calculus it optimizes. How this work differs. Most of the techniques above are studied and deployed in isolation: self-consistency or a verifier or refinement, each used as a separate technique. Our contribution is to treat them as instances of one algebra and to put a controller in charge of choosing among them per instance under an explicit budget, guided by the reliability laws. Two consequences distinguish the approach. First, the harness measures verifier discrimination Λ online and credits each mechanism only with the amplification its measured quality supports, rather than fixing a pipeline in advance. Second, by reporting reliability at a measured cost and selecting the cheapest configuration that meets a target, it makes the cost–reliability tradeoff an explicit, tunable quantity instead of an implicit side effect of a prompt template. The aim is not a new trick but a principled way to compose the existing ones.
3
From Algebra to Architecture
The harness is a direct compilation of the four combinators into runtime components. Table 1 gives the correspondence; the rest of the paper elaborates each row and the controller that composes them. Three design principles follow from the laws. First, check more than you decompose: since sequential decomposition can only lose reliability, every decomposition boundary should be paired with a verifier that restores it. Second, prefer verification to voting when
Blackboard (state, traces)
Solver pool (roles, temps)
refine/escalate
Verifier ensemble Aggregator / Gate
Tool sandbox (exec, search)
answer / ⊥
Figure 1: Harness architecture. The controller allocates a compute budget across decomposition, voting, and verification; the blackboard holds shared state and traces; the tool sandbox supplies robust verifiers (execution, tests, search). Arrows show data and control flow; the refine/escalate edge closes the verify loop.
a discriminating checker exists: a single gate with Λ = 6 outperforms dozens of votes for a near-threshold solver. Third, spend at the margin: allocate the next unit of compute to whichever stage currently offers the largest reliability gain per call. The controller (§7) implements all three.
4
Harness Architecture
Figure 1 shows the components. Everything is built on one abstraction. The solver interface. A base solver exposes solve(task, ctx) -> {answer, score, trace, cost}. Any model, tool, or subharness implements it, so components compose uniformly and the controller can treat a whole subtree as a single solver. score is a self-reported confidence (used only as a weak prior); trace feeds the blackboard; cost is the realized call/token count. The blackboard record. All components read and write a single structured record, which is also the unit of tracing and replay: Planner / decomposer. Given a task, the planner optionally emits a typed plan: a DAG of subtasks with dependencies and a recombination step. Decomposition is applied only when the planner’s confidence in a clean split is high, since unguarded decomposition
Maestro Order: A Model-Agnostic Orchestration Harness
Solve { task: {id, type, input, deps[]} budget: {max_calls, target_rel, lambda} cands: [{answer, score, trace, cost}] verdicts:[{verifier_id, accept, score, est_beta, est_alpha}] state: {logodds, calls, coverage} result: {answer | ABSTAIN, confidence} }
Figure 2: Blackboard record (simplified). The controller reads state.logodds and per-mechanism marginal estimates to ˆ 𝛼ˆ used to decide the next action; verdicts carry the online 𝛽, compute b Λ. compounds error (Table 1); each subtask boundary is annotated with the verifier to apply on recombination. Solver pool. A configurable set of solver roles (for example, different prompts, temperatures, tools, or distinct models) provides the diversity that voting needs. The pool composition is the main control on error correlation 𝛾 (§13); pools of near-identical solvers vote poorly.
never as a silent accept. This keeps the false-acceptance rate 𝛼 bounded by the verifier’s genuine error rather than by infrastructure flakiness, which is what makes the discrimination Λ = 𝛽/𝛼 meaningful and stable. A harness that failed open would see its Λ collapse under load exactly when reliability matters most. I3: Monotone confidence. Adding work never decreases the system’s information about an instance: a new vote or gate can only sharpen the posterior, and the controller never discards evidence. This guarantees that spending more budget weakly improves reliability (the refinement order of the theory), so the cost–reliability curve is monotone and the controller’s greedy ascent is well-defined. I4: Budget conservation and accountability. Every call is attributed to a stage and counted against the budget before it is made; the controller cannot overspend, and every reported reliability number has an associated, measured cost. Reliability claims without cost are meaningless, so the two are always reported together. I5: Determinism up to declared randomness. Given fixed seeds and backend versions, a solve is replayable bit-for-bit from its trace. Randomness enters only where declared (sampling temperatures, vote seeds), so experiments are reproducible and regressions are diagnosable. These five invariants are the contract every component signs; the mechanisms below are implementations that honor it.
Verifier ensemble. One or more verifiers, each a checker returning accept/reject with a score: deterministic tools (unit tests, typecheckers, math re-evaluation, retrieval corroboration) and modelbased critics. Each verifier’s completeness 𝛽 and false-acceptance 𝛼 are estimated online (§7); the gate uses the calibrated log-likelihood ratio as its acceptance score.
The harness exposes the combinators as mechanisms the controller can deploy per instance.
Aggregator / gate. Combines candidates: plurality or scoreweighted voting, verified selection (return the highest-scoring accepted candidate), or all-accept gating for amplification. Emits an answer or ⊥ (abstain) with a calibrated confidence.
Self-consistency voting. Sample 𝑛 diverse candidates and return the plurality answer; reliability rises with 𝑛 when per-sample correctness exceeds 12 and errors are diverse (§13). Cheap, model-only, but bounded by the correlation floor.
Blackboard and tool sandbox. A shared store holds intermediate results, verifier verdicts, and a full execution trace for observability and replay. The tool sandbox executes untrusted code and runs searches in isolation, supplying the robust verifiers the theory privileges.
Verified selection and gating. Run candidates through the verifier ensemble; either select the best accepted candidate or, for amplification, require all 𝑘 gates to accept, regenerating otherwise (Algorithm 1). Each accepted gate multiplies the odds of correctness by its Λ.
5
Design Rationale and System Invariants
A reliability harness is only trustworthy if its guarantees hold regardless of what the wrapped model does. We therefore design around a small set of invariants that the components must preserve and that the companion theory [1] shows are sufficient for the reliability laws to apply. I1: Confidence is calibrated log-odds. Every commitment carries a confidence that is, after calibration, the true log-odds of correctness. This single invariant makes the rest composable: gates add to it (the odds law), abstention thresholds it (Chow’s rule), and the controller maximizes it per unit cost. Components that cannot expose a calibrated score are wrapped by a calibration map fit on held-out data; uncalibrated raw scores are never used for decisions, only as weak priors. I2: Fail closed. Any ambiguity, such as a tool error, a timeout, a malformed candidate, or a verifier crash, is treated as a rejection,
6
Reliability Mechanisms
Iterative refinement. On rejection, return the verifier’s critique to the solver and resample, a generate–critique–revise loop. Refinement raises the per-attempt success probability 𝑝, which compounds with gating. Debate and cross-examination. Independent solvers critique each other’s candidates; disagreement is a cheap signal that routes an instance toward more verification or abstention. Abstention and escalation. When the accumulated log-odds score falls below a calibrated threshold, the harness abstains or escalates to a stronger (costlier) sub-organization: selective answering on the risk–coverage frontier (§13).
7
The Budget-Aware Controller
The controller decides, per instance, how to spend a compute budget 𝜅 across the mechanisms. By the companion theory the reliabilitymaximizing allocation equalizes the marginal log-odds gain per
1: input: generator 𝑔, verifiers 𝑣 1 ..𝑣 𝑘 , budget 𝑇
2: for 𝑡 = 1 to 𝑇 do
𝑎 ← 𝑔.solve(𝑥) ⊲ sample a candidate if ∀𝑖 : 𝑣𝑖 .accept(𝑥, 𝑎) then Î 5: return 𝑎 with odds 𝑖 Λ𝑖 applied 6: end if 7: end for 8: return ⊥ ⊲ abstain after 𝑇 rejections 3: 4:
Algorithm 1: VerifyGate: amplify a candidate by all-accept gating 1: input: task 𝑥, budget 𝜅, mechanism set M 2: 𝜎 ← single base solve; estimate log-odds ℓ
3: while cost(𝜎) < 𝜅 and not converged do 4: 5:
call
for 𝑚 ∈ M do ⊲ vote+𝑛, add gate, refine, decompose c (𝑚)/Δcost(𝑚) ⊲ est. marginal log-odds per 𝑟ˆ𝑚 ← Δℓ
6:
end for 𝑚★ ← arg max𝑚 𝑟ˆ𝑚 8: if 𝑟ˆ𝑚★ ≤ 𝜆 then break 9: end if 10: 𝜎 ← apply(𝑚★, 𝜎); update ℓ 11: end while 12: return aggregate(𝜎) or ⊥ if ℓ < 𝜏 7:
⊲ saturated
Algorithm 2: Controller: greedy marginal allocation
unit cost across stages; the controller approximates this by greedy selection (Algorithm 2). Online estimation of verifier discrimination. The controller maintains running estimates of each verifier’s 𝛽 and 𝛼 from instances where ground truth is later revealed (tool oracles, held-out checks, ˆ 𝛼ˆ and hence Δℓ c = log b or human spot-checks), yielding b Λ = 𝛽/ Λ for the gate mechanism. Verifiers whose estimated Λ drifts toward 1 are demoted, an explicit guard against the gaming failure mode (§15). Voting’s marginal gain is estimated from the current vote margin via a Chernoff exponent. Calibration. Raw model and judge scores are poorly calibrated, so the controller fits a monotone calibration map (e.g. isotonic/Platt) from scores to empirical correctness, and uses calibrated log-odds both as the gate score and as the abstention threshold (Chow’s rule: commit only when the calibrated posterior clears a cost-based threshold). Calibration quality is reported as expected calibration error (ECE) in §13.
8
Engineering Robust Verifiers
Because verification is the high-leverage mechanism (§13) and because gameable verifiers are the dominant failure mode (§15), the verifier ensemble deserves the most engineering attention. The goal is high discrimination Λ = 𝛽/𝛼 that is robust: it must hold even when candidates are produced by a process optimizing for acceptance.
Prefer ground-truth checks. The most robust verifiers reduce to executing a specification rather than judging plausibility. For code, run the candidate against unit tests, property-based tests, type-checkers, and sanitizers in the sandbox; for mathematics, re-evaluate numerically, check units and boundary cases, or machine-check a proof; for factual claims, corroborate against retrieved sources and require citation support. The defining property is that their false-acceptance rate is bounded by the specification, not by how the candidate was generated, so 𝛼 does not rise under optimization pressure and Λ stays high. Make model-based judges as robust as possible. When only a model can check (open-ended reasoning, style, subtle correctness), several techniques raise and stabilize Λ: ask the judge to find a counterexample or a concrete flaw rather than to rate quality (refutation is harder to game than approval); require the judge to show its check (a verification chain-of-thought) and verify that; use a judge that is independent of the generator (a different model, prompt, or seed) to keep verifier and generator errors conditionally independent; and combine several judges so that a single exploitable blind spot does not dominate. Each independent, informative judge contributes additively to the log-odds (the odds law), so a panel of diverse weak judges can rival a single strong one, subject to the same correlation floor that limits voting. Measure Λ, do not assume it. The controller treats every verifier’s (𝛽, 𝛼) as quantities to be estimated, not declared. Wherever ground truth eventually becomes available (a tool oracle, a downstream outcome, a held-out labeled stream, or human spot-checks), the ˆ 𝛼ˆ and hence b Λ. A verifier is harness updates running estimates 𝛽, only credited with the amplification its measured discrimination supports; nominal accuracy on a benign distribution is never trusted on its own. Detect and resist drift. Under sustained optimization, a verifier’s effective 𝛼 tends to rise as generators discover what it wrongly accepts: Goodhart’s law in action. The harness watches b Λ over time and, when it drifts toward 1, reduces that verifier’s weight, rotates in fresh or held-out checkers, and raises the abstention threshold for affected instances. Disagreement among diverse verifiers is itself a cheap early-warning signal: a sudden rise in unanimous acceptance with falling downstream correctness is the signature of a gamed gate. Tune the operating point. A verifier exposes a threshold trading completeness 𝛽 against false-acceptance 𝛼 (its ROC curve). For amplification the harness prefers a high-𝛽/low-𝛼 operating point even at the cost of more regenerations, because Λ, not throughput, governs the reachable reliability; for inexpensive screening it may accept lower 𝛽. The controller selects the operating point per instance from the marginal-rate calculus.
9
Decomposition and Planning in Practice
Decomposition is the riskiest combinator: by the serial law it can only lose reliability unless each boundary is restored by verification. The planner is therefore conservative by design.
Maestro Order: A Model-Agnostic Orchestration Harness
Decompose only when the split is clean and checkable. The planner proposes a decomposition only when it is confident the subproblems are (i) genuinely independent or cleanly ordered, and (ii) each individually verifiable. A split into subproblems that cannot be checked inherits the worst of both worlds (more stages that can fail, and no way to catch the failures), and the planner declines it in favor of a monolithic solve plus verification. This is the operational reading of the recursion master theorem: depth is affordable precisely when every level is gated. Typed plans and verified recombination. A plan is a typed DAG: each node declares its input and output types, its dependencies, and the verifier to apply to its result and to the recombination. Type compatibility is checked statically before any model call, catching a class of decomposition errors for free. The recombination step is itself a gated solver, so a correct assembly of correct parts is verified rather than assumed. Interleaving planning and acting. For tasks where the right decomposition is not knowable up front, the harness interleaves planning with acting: solve a subproblem, observe the result and its verdict on the blackboard, and let the planner revise the remaining plan. This keeps decomposition adaptive while preserving the invariant that every committed intermediate has passed a gate, so errors do not silently propagate down the tree. Budgeting across the tree. The controller allocates the global budget across plan nodes by the same marginal-rate rule, spending more verification on nodes whose errors are most likely to corrupt the final answer (those with high fan-out or on the critical path) and less on cheap and easily checked leaves. The result is the verifier-saturated, marginal-rate-equalizing organization the theory predicts, realized at the granularity of a plan.
10
Anatomy of a Solve
To make the controller concrete, Table 2 shows the controller’s expected progress with a weak base solver (𝑝 0 = 0.55) and a robust verifier (Λ = 6, so each accepted gate adds log 6 = 1.79 to the logodds). The controller starts with one base solve, then repeatedly adds the highest-rate mechanism (here, another verification gate), updating the running log-odds and cost, until the calibrated logodds clears the target threshold or the marginal rate drops below 𝜆. The reliabilities and call counts are the measured simulation values of §13; the log-odds column is their logit. The trace exhibits the law operationally: the benefit per gate is constant in log-odds (+1.79) but the cost per gate grows (each gate lowers acceptance probability, so more regenerations are needed), which is why the marginal rate falls and the controller eventually stops. A target of 0.97 is met at step 2 (7.4 calls) and 0.999 at step 4 (17.3 calls). Voting alone from this near-threshold base would need several hundred calls for the same targets (Fig. 4b).
11
Table 2: A controller trace (simulation). Each gate multiplies the odds by Λ=6 (+1.79 log-odds); reliability saturates while cost grows linearly. The controller stops at the first row that meets the target, or when the next gate’s marginal rate falls below 𝜆. Step
Action
Δℓ
ℓ
𝑃 (corr)
calls
0 1 2 3 4
base solve + verify gate + verify gate + verify gate + verify gate
— 1.79 1.79 1.79 1.79
0.20 1.99 3.78 5.58 7.37
0.550 0.880 0.978 0.996 0.999
1.0 3.8 7.4 11.7 17.3
stop: the next gate would offer Δℓ/Δcalls ≈ 0.25 < 𝜆
fix seeds per attempt so an experiment is bit-reproducible given the same backends. Idempotency and caching. Solver and verifier calls are keyed by (input, config) and cached, so retries and refinement loops do not double-bill and so ablations reuse work. Concurrency. Independent candidates and verifiers run in parallel; the controller bounds fan-out by the remaining budget and merges results on the blackboard. Failure handling. Tool/model errors are treated as rejections (fail closed), never as silent acceptances; timeouts count toward cost and trigger escalation or abstention rather than a guess. Cost accounting and observability. Every call’s token and latency cost is attributed to a stage, so the reliability–cost curves of §13 are measured, not estimated, and per-stage marginal rates are visible to the controller and to operators. Scheduling and backpressure. The controller issues independent candidate and verifier calls concurrently up to a fan-out bounded by the remaining budget and backend rate limits; results stream onto the blackboard and the controller re-plans as each arrives. Under load it applies backpressure by lowering fan-out and raising the marginal-rate threshold 𝜆, gracefully degrading from a highreliability profile to a cheaper one rather than violating latency budgets. Sandboxing and security. Tool verifiers (code execution, shell, search) run in an isolated sandbox with no access to secrets or the host, because robust verification often means executing untrusted candidate code. Sandbox failures are rejections (invariant I2), and resource limits (CPU, memory, wall-clock) are enforced per call so a pathological candidate cannot exhaust the budget. Regression testing the harness itself. Because solves are replayable, a fixed suite of recorded instances doubles as a regression test: a change to a prompt, verifier, or controller policy is evaluated by rerunning the suite and comparing reliability-at-cost and calibration, catching silent degradations before they reach production.
Implementation Concerns
Reliability claims are only as trustworthy as the plumbing beneath them. Determinism and reproducibility. Each solve records model id, decoding parameters, seed, prompt hash, and tool versions; runs are replayable from the trace. Stochastic steps (sampling, voting)
12
Evaluation Methodology
We evaluate the harness along four axes. Metrics. (i) Reliability at fixed cost: correctness rate on committed answers at a capped expected number of base calls. (ii) Coverage and the risk–coverage curve (and its area, AURC) for selective answering.
(iii) Calibration (ECE) of the harness confidence. (iv) Cost (expected base calls) and latency. Ablations. We vary the vote count 𝑛, the number of gates 𝑘, decomposition depth, verifier quality Λ, and solver diversity (correlation 𝛾), isolating each combinator’s contribution. Statistical methodology. Each reported point is an average over many i.i.d. trials with fixed seeds; we report it with a binomial confidence interval and treat differences smaller than the interval as noise. Because the harness is deterministic up to declared randomness (invariant I5), any reported number is reproducible from the recorded seeds and backend versions, and ablations reuse cached calls so that only the varied factor changes. Protocol for real models. The harness is designed to be evaluated on standard suites (grade-school and competition math, code synthesis with execution-based verifiers, and open-domain QA with retrieval corroboration) using standard metrics: pass@𝑘 accuracy, execution pass rate, and exact match, with the deterministic-replay machinery of §11. Simulation study (this paper). To exercise the controller and validate that the implementation obeys the predicted laws without conflating harness behavior with any particular model’s idiosyncrasies, we report a faithful Monte Carlo simulation of the harness over a parameterized solver/verifier model: a generator emits a correct candidate with probability 𝑝; each verifier accepts a correct candidate with probability 𝛽 and a wrong one with probability 𝛼 (Λ = 𝛽/𝛼); correlated solver pools are produced by a Gaussian-copula latent factor with pairwise correlation 𝛾. We emphasize that the numbers below are simulation outputs, not measurements of a deployed model; they test the harness logic and the theory, and set expectations for the real-model protocol above. Each point uses between 2 × 104 and 4 × 104 trials; repeated runs differ by Monte Carlo noise of roughly ±0.004 in reliability and ±0.1 in expected calls, which explains small discrepancies between tables. Every number, figure, and table in this paper can be regenerated exactly with the released code (§18).
13
Results
Verification amplifies as the odds law predicts. Figure 4a overlays simulated gate reliability on the theoretical prediction 𝑟𝑘 = 𝑜 0 Λ𝑘 /(1 + 𝑜 0 Λ𝑘 ) for a strong (Λ = 8) and a weak (Λ = 2) verifier, starting from 𝑝 0 = 0.55. The simulation tracks theory to within Monte Carlo noise: three strong gates take the base from 0.55 to 0.998, and because 83 = 29 the weak verifier needs exactly three times as many gates for the same odds gain. Across all gate counts and both verifiers the simulated reliabilities agree with the closedform odds law to within ±0.004 (the binomial standard error at our trial counts), and the controller trace of Table 2 reproduces the same values, evidence that the implementation faithfully realizes the algebra rather than merely being inspired by it. This agreement is the point of the simulation: it validates the harness logic and isolates it from any particular model’s quirks before a real-model evaluation introduces them. Verification dominates voting on the cost frontier. Figure 4b plots reliability against expected base calls. A verifier with Λ = 6 reaches 0.98 within eight calls and 0.997 within twelve; majority voting
Figure 3: Controller trajectories (simulation): reliability versus cumulative base calls for three strategies on a weak base (𝑝 0 =0.55). Verification and the hybrid strategy reach high reliability at an order of magnitude lower cost than voting; the controller follows whichever trajectory has the steepest current slope.
on a 𝑝 = 0.62 solver needs about 90 calls to pass 0.99, and a nearthreshold solver barely improves at all. This is the practical form of the Θ(log 𝛿1 ) scaling, with very different constants. The controller follows the steepest trajectory. Figure 3 plots reliability against cumulative cost for verify-only, vote-only, and a hybrid that votes over three candidates before gating. Verify-only reaches 0.999 at about 17 calls and the hybrid at about 23; vote-only is still below 0.98 at 60 calls. The controller’s greedy policy (Algorithm 2) is a walk up whichever trajectory is locally steepest, and the figure shows why it would choose gates from the start here: from 𝑝 0 = 0.55, the first gate offers about 0.64 log-odds per call, while extra votes offer about 0.05 (a 3-vote lifts 0.55 only to 0.575). Voting becomes the better first move only when the vote margin is large or the verifier is weak; the controller detects which regime it is in from the measured marginal rates instead of committing to a fixed pipeline. Diversity sets a hard ceiling on voting. Figure 4c shows majority reliability versus committee size for error correlations 𝛾 ∈ {0, 0.05, 0.2} under the Gaussian-copula model. Independent solvers approach 1; at 𝛾 = 0.05 reliability plateaus near 0.85 and at 𝛾 = 0.2 near 0.71, no matter how many correlated members are added. The plateaus match the shared-cause floor of the companion theory [1]: with 𝑝 = 0.6, the predicted asymptotic limits (the probability that the shared factor keeps the committee above chance) are 0.87 for 𝛾 = 0.05 (the 𝑛 = 255 committee is still approaching it) and 0.71 for 𝛾 = 0.2. Diversifying the solver pool (varied prompts, temperatures, models) is therefore a primary reliability decision, not a detail. Abstention buys reliability cheaply. Figure 4d is the risk–coverage curve from a calibrated harness score: committing only on the most-confident 20% of instances drops the error rate from 0.45 (no abstention) to under 0.1. Selective answering plus escalation is the cheapest reliability available when full coverage is not required. The hybrid controller wins. Table 3 ablates the mechanisms from a weak base (𝑝 0 = 0.55, Λ = 6). Voting alone is nearly useless this
Maestro Order: A Model-Agnostic Orchestration Harness
Table 4: Deployment profiles (simulation, 𝑝 0 =0.55, Λ=6).
(a) amplification vs. theory
(b) cost–reliability
Profile
Mechanisms
Cheap Balanced High-reliability
single solve vote 5 + verify 𝑘=2 verify 𝑘=4 (+abstain)
0.90 0.97 0.99 0.999
Table 3: Ablation (simulation): reliability, coverage, and expected base calls for harness configurations on a weak base (𝑝 0 =0.55, verifier Λ=6, 𝑇 =20). Hybrid configurations dominate at the high-reliability end.
single shot vote n=5 verify k=2 verify k=4 vote5 + verify k=2 vote5 + verify k=4
1.0 16.0 17.3
Config
Verify calls
Voting calls
𝑘=1 𝑘=2 𝑘=3 𝑘=4
3.8 7.4 11.7 17.3
∼27 ∼60 ∼90 ∼160
(d) risk–coverage
Figure 4: Simulation results (Monte Carlo over the parameterized solver/verifier model; not measurements of a deployed model). (a) gate reliability matches the odds law; (b) verification reaches a target at far lower cost than voting; (c) error correlation caps voting gains; (d) abstention trades coverage for reliability.
Configuration
Calls
0.55 0.98 0.999
Table 5: Cost-to-target (simulation). Verification reaches each target with slowly growing incremental cost; the voting-only column is the approximate cost for a 𝑝=0.62 solver (normal approximation) and is roughly an order of magnitude larger. Target reliability
(c) diversity floor
Reliability
Reliability
Coverage
Calls
0.549 0.591 0.978 0.999 0.980 0.999
1.000 1.000 1.000 0.999 1.000 1.000
1.0 5.0 7.4 17.4 16.0 28.8
close to threshold (0.55 → 0.59); two gates reach 0.978 at 7.4 calls; four gates reach 0.999. Combining a small vote with gating gives the best reliability-per-call at the high end. The controller’s job is to pick the point on this surface that meets the target at least cost: here, “verify 𝑘=2” for a 0.97 target and “𝑘=4” for 0.999.
for the same target but does not change the geometric character of the amplification. The practical implication: because the required depth is divided by log Λ, a modest improvement in the verifier multiplies through the entire frontier, and often helps more than a large increase in vote count or a slightly better base model. This is the empirical form of the theory’s recommendation to invest in robust checkers. Calibration. The selective-answering results (Fig. 4d) depend on the harness confidence being calibrated: the abstention threshold is only meaningful if a reported 0.9 confidence corresponds to a 0.9 empirical correctness rate. After fitting the monotone calibration map of §7, the harness confidence is well-calibrated by construction in the simulation, so the risk–coverage curve is the true achievable frontier; on real models, calibration error (ECE) would be reported alongside it, and miscalibration would manifest as a gap between the nominal threshold and the realized risk. Calibration is thus not a cosmetic metric but the precondition that makes abstention and gating trustworthy.
13.2
Deployment profiles and cost-to-target
Sensitivity to verifier quality and base reliability
Operators rarely tune individual mechanisms; they pick a profile and a target. Table 4 distills the ablation into three profiles, and Table 5 inverts the relationship: given a reliability target, the cheapest configuration the controller selects and its expected cost. The pattern is clear: each order-of-magnitude reduction in error costs only a few more calls under verification, because cost grows with log 𝛿1 while voting grows far faster.
The two parameters that matter most are the verifier discrimination Λ and the base reliability 𝑝 0 , and their effects are exactly those the theory prescribes. The number of gates to reach a target scales as 1/log Λ, which is why the Λ = 8 curve in Figure 4a needs one third of the gates of the Λ = 2 curve (log 8 = 3 log 2). A higher base 𝑝 0 shifts the whole cost–reliability frontier left by the additive term 𝑝 log 1−𝑝0 0 in the log-odds, so a stronger base solver needs fewer gates
Summary. The harness reproduces every qualitative prediction of the theory and quantifies the trade-offs an operator faces: verification is the high-leverage mechanism when a discriminating checker exists, voting is a cheap supplement bounded by diversity, and abstention converts confidence into reliability. The controller’s marginal-rate policy reduces all of these choices to two settings: a target reliability and a budget.
13.1
14
Case Studies
The same harness specializes to very different tasks by changing only the solver pool and the verifier ensemble; the controller and invariants are unchanged. We sketch three configurations to make the mapping concrete. Grade-school and competition mathematics. The base solver samples chain-of-thought solutions at moderate temperature; diversity comes from multiple seeds and two prompt styles. The verifier ensemble is deliberately robust: a deterministic arithmetic re-evaluator recomputes the final expression, a units and sanity checker rejects out-of-range answers, and, for problems with a checkable form, substitutes the candidate back into the problem’s constraints. Because these checks reduce to executing the problem’s own definition, their false-acceptance is low and stable, so Λ is high and the controller leans heavily on gating, falling back to self-consistency voting only when no closed-form check applies. The expected behavior is the amplification curve of Figure 4a: a few gates take a near-threshold solver to high reliability at modest cost. Program synthesis. Here verification is nearly ideal. The base solver proposes a program; the verifier executes it in the sandbox against unit tests, a type-checker, and property-based fuzzing, with failures counting as rejections (invariant I2). Execution-based checking has very high discrimination: a program that passes a strong test suite is very likely correct, and one that fails is certainly rejected. The controller’s dominant mechanism is therefore a single high-Λ gate plus refinement: on rejection, the failing test output is fed back as a critique and the solver revises. Decomposition appears naturally for multi-function tasks, with each function gated by its own tests before integration; this is the recursion master theorem realized as test-driven assembly. This is the regime where orchestration most clearly beats single-shot generation. Open-domain question answering. Verification is hardest here because correctness is not mechanically checkable. The harness leans on grounding: a retrieval step supplies sources, the solver must answer with citations, and the verifier checks that each claim is supported by a retrieved passage (a corroboration check) and cross-examines with an independent judge model prompted to find unsupported assertions. No single check is decisive, so the controller combines a modest verifier panel with self-consistency over diverse retrievals, and, crucially, sets a conservative abstention threshold, declining or escalating when corroboration is weak. The operating point sits on the risk–coverage frontier of Figure 4d: the system answers confidently where sources agree and abstains where they do not, which is the responsible behavior for factual tasks. Adaptive allocation versus fixed pipelines. A fixed pipeline, such as “always sample five and verify twice,” spends the same budget on every instance, over-paying on easy cases and under-serving hard ones. The controller instead reads each instance’s running log-odds and stops early when the target is already met, redirecting budget to the instances that need it. Because difficulty is heavy-tailed (most instances are easy, a few are hard), this adaptivity is where much of the practical cost saving comes from: the average cost is dominated by the easy majority, on which the controller commits quickly, while the hard tail receives the deep verification it requires. The
fixed pipeline is the special case the controller reduces to when every instance happens to have the same difficulty, which is rare in practice.
15
Discussion: Failure Modes and Guidance
Verifier gaming (Goodhart). When solvers are optimized against a fixed checker, the checker’s effective false-acceptance rises and its discrimination collapses toward Λ = 1, and amplification silently fails. (This is Goodhart’s law: a proxy measure stops tracking what it was meant to measure once it is optimized directly.) Mitigations: prefer robust verifiers whose soundness is independent of the generator (execution, proofs, types, corroborated retrieval); monitor each Λ and demote drifters; rotate or ensemble diverse verifier’s online b verifiers; and hold out a ground-truth stream for recalibration. Correlated errors. A pool of near-identical solvers inherits shared blind spots; voting then plateaus far below 1 (Fig. 4c). Mitigation: engineer diversity (distinct prompts, temperatures, tools, and ideally distinct models) and measure 𝛾 directly from disagreement rates. Decomposition error compounding. Aggressive decomposition multiplies per-stage error and can underperform a monolithic solve. Mitigation: decompose only when a clean split is confident, and gate every recombination boundary so each level is restored before it propagates; this is the recursion master theorem in practice [1]. Cost and latency. Reliability is bought with calls; the controller’s budget cap and abstention keep total cost bounded, and caching plus parallel fan-out keep latency manageable. The right operating point is application-specific and is exactly the marginal-rate threshold 𝜆. When not to use a harness. Orchestration is not free, and not every task warrants it. For tasks where the base solver is already reliable enough, where no informative verifier exists (Λ ≈ 1), or where latency dominates and a single fast answer is preferable to a slow reliable one, the cheap profile (single solve) is correct and the overhead of voting and gating is waste. The controller makes this an explicit decision: if the estimated marginal rate of every mechanism is below 𝜆 from the start, it commits the single solve. The harness adds value exactly in the regime the theory identifies: weak base solvers with discriminating, robust checkers. Interaction with model improvements. A better base model raises 𝑝 0 and shifts the whole cost–reliability curve left: the same target is reached with fewer gates and votes, and the controller spends less by default. Crucially, orchestration and model quality are complements, not substitutes: a stronger model with a robust verifier reaches extreme reliability more cheaply than either alone. The harness therefore remains valuable as models improve, simply operating at a cheaper point. The exception is verification quality: if generation improves faster than checking, the verifier becomes the limiting factor, which reinforces the paper’s central recommendation to invest there. Hallucinations and long-running jobs. Two deployment problems map directly onto the harness. The first is hallucination: a generative model’s confident production of incorrect or unsupported content [15]. This is exactly the failure that calibrated confidence targets: a hallucinated candidate either fails corroboration and test
Maestro Order: A Model-Agnostic Orchestration Harness
gates and is rejected, or fails to accumulate enough log-odds and becomes an abstention or an escalation. Confident errors are thereby converted into checked refusals. The second is the long-running, multi-step job: because per-step error compounds, even a step that is 99% reliable fails almost surely over hundreds of steps. Verifying every boundary bounds end-to-end error at logarithmic overhead (the recursion master theorem of the companion theory), and idempotent, replayable state lets such jobs checkpoint, suspend, and resume, which is what long-horizon automation requires. Safety. Fail-closed verification (treating errors and timeouts as rejections), calibrated abstention, and full traceability make the harness conservative by construction: when it is unsure, it declines or escalates rather than emitting a confident guess. The full execution trace also makes every answer auditable after the fact (which candidates were generated, which verifiers accepted, and why the controller stopped), itself a requirement for high-stakes deployment.
16
Limitations and Threats to Validity
The headline results are simulations of a parameterized model, not measurements of a deployed system; they validate the harness logic and the reliability laws but do not, by themselves, establish absolute numbers on any benchmark. Real models violate the simulation’s clean assumptions: verifier errors are not perfectly conditionally independent, solver correctness is instance-dependent rather than a single 𝑝, and 𝛽, 𝛼, 𝛾 vary across the input distribution. The companion theory [1] bounds the cost of these violations (correlated verifiers lose effective Λ; correlated solvers can hit a floor when shared causes push the committee below chance), and the harness estimates the relevant quantities online, but a full real-model evaluation under the §12 protocol is necessary to claim deployment numbers and is the natural next step. Finally, our controller is greedy; the constrained optimum it approximates can in principle be solved more globally when marginal-rate estimates are reliable.
17
Conclusion
Generalization. Although we describe the harness in terms of language models, nothing in the architecture or the controller depends on that choice. A base solver is any process behind the solve interface (a model, a heuristic, a search procedure, a tool, a human, or another harness), and a verifier is anything that returns a calibrated accept/reject. The same controller would organize a pool of human reviewers with an automated checker, or a mix of models and classical algorithms, identically. This is the practical payoff of grounding the system in an algebra over abstract solvers: the harness is a general-purpose organizer of unreliable problem-solving, of which the language-model case is one instance. We presented a model-agnostic harness that organizes unreliable solvers into reliable problem-solving systems by compiling four structural combinators into runtime components and a budgetaware controller that spends compute where it buys the most reliability. A faithful simulation reproduces the predicted laws: geometric amplification by verification, threshold-and-floor behavior of voting, and the value of abstention. It also shows that a controllerdriven organization reaches reliability targets at a small fraction of
Table 6: Reproduction map: each published artifact, the subcommand of reproduce.py that regenerates it, and the PCG64 seed. Artifact
Subcommand
Seed
Fig. 4(a–d), Table 3 Table 2 (trace, stop rate) Fig. 3 (trajectories) Closed-form numbers in the text
figures trace traj predictions
20260607 7 11 —
the cost of voting alone, automatically selecting the cheapest mechanism for the regime at hand. The practical message mirrors the theory: build robust checkers, diversify solvers, verify every boundary, and let the controller allocate at the margin. Reliability is not a property of a model; it is a property of how models are organized.
18
Artifact and Reproducibility Statement
All experimental results in this paper are Monte Carlo simulations of the parameterized solver/verifier model of §12; no proprietary models, datasets, or services are required to reproduce them. The full source code is publicly available at https://github.com/hidayetaksu/ maestro-order, together with pinned dependencies, reference outputs, and a README with step-by-step instructions. One-command reproduction. With Python 3.10+ and the pinned packages from requirements.txt (NumPy 2.2.6 is the version that matters for bit-exact randomness), python reproduce.py all regenerates every artifact in two to four minutes on a single CPU core, writing to ./output. Table 6 maps each artifact to its subcommand and seed. The four panels of Figure 4 and the ablation table share one pseudo-random stream in a fixed order, so the figures subcommand runs them together; the exact pattern of random draws inside the simulation primitives is documented in the code and is part of the artifact. Verification. Reproduced outputs can be checked against the published artifacts by direct comparison: the regenerated ablation table in output/ is byte-identical to the published copy in figures/, and JSON summaries of every data series (including the measured maximum deviation between simulation and the closed-form odds law, 0.0019 ≤ 0.004) are provided as reference outputs. Derived quantities quoted in the text (the voting cost-to-target estimates of Table 5, the diversity plateaus of §13, and the marginal rates in the trajectory analysis) are computed in closed form by the predictions subcommand, with no random input. The paper itself builds from the repository root with pdflatex and the ACM acmart class; figure and table sources are read from the figures/ directory that the reproduction script can regenerate.
References [1] Hidayet Aksu. 2026. Odds Law: The Decomposition Algebra—On How Intelligence Organizes Itself to Solve Difficult Problems Reliably. arXiv:2606.15712. https://arxiv.org/abs/2606.15712 [2] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Proc. NeurIPS.
[3] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In Proc. ICLR. [4] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Proc. NeurIPS. [5] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In Proc. ICLR. [6] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. In Proc. NeurIPS. [7] Aman Madaan, Niket Tandon, Prakhar Gupta, et al. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In Proc. NeurIPS. [8] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325. [9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168.
[10] Hunter Lightman, Vineet Kosaraju, Yura Burda, et al. 2023. Let’s Verify Step by Step. arXiv:2305.20050. [11] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Proc. NeurIPS Datasets and Benchmarks. [12] Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Proc. NeurIPS. [13] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, et al. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Proc. NeurIPS. [14] Mark Chen, Jerry Tworek, Heewoo Jun, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374. [15] Ziwei Ji, Nayeon Lee, Rita Frieske, et al. 2023. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys 55, 12, Article 248. [16] John von Neumann. 1956. Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components. In Automata Studies. Princeton Univ. Press, 43–98. [17] Marquis de Condorcet. 1785. Essai sur l’application de l’analyse à la probabilité des décisions. Paris. [18] Jeffrey Dean and Sanjay Ghemawat. 2004. MapReduce: Simplified Data Processing on Large Clusters. In Proc. OSDI. USENIX, 137–150.