ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision

arXiv:2608.02444v1 [cs.AI] 3 Aug 2026

ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision Wei-Jung Huang

Bonan Shen

Independent Researcher United States william.wj.huang@gmail.com

Independent Researcher United States shenbonan2@gmail.com

Abstract—LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete. A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation. Early tasks can omit important parts of a benchmark, running cheaper tasks first can distort the observed sample, and a rule that decides only easy pairs can appear accurate while leaving many comparisons unresolved. We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance. For each partial run, it records whether the tested agent system is better by the required amount, is not better by that amount, needs more evidence, or should abstain. We evaluate ParEvalLayer by replaying completed public benchmark data as if each evaluation had stopped earlier. At each point, ParEvalLayer applies the policy using only the outcomes observed so far; if it reaches one of the two comparison judgments, we check whether that judgment matches the completed data for the same system pair. With the main comparison rule, three of the public benchmarks reach the same decision as the completed evaluation after observing only 15% to 25% of task outcomes. Other benchmarks require more task outcomes. This variation shows why a partial score alone is not enough: reports should also state the decision rule and how many comparisons remain without a decision. Index Terms—LLM agents, agent evaluation, partial evaluation, early stopping, benchmark reporting

I. I NTRODUCTION Agent benchmark evaluations often end as a single score, but producing that score can be expensive. A full run may require wall-clock time, API budget, sandbox resources, and human inspection when agent runs fail or need debugging. The practical question often arrives before the full run finishes: from the task outcomes seen so far, can the evaluation report a comparison between two agent systems, or should it keep running tasks? We use agent system to mean the whole evaluated setup that produces task outcomes: the model, prompts, tools, scaffolding, retry logic, and the benchmark verifier. The comparison studied here asks whether one such system scores higher than another by more than a required improvement amount. A partial score is tempting because it looks like the final report in miniature. It is often not enough to decide whether one agent system is better than another by the amount specified in the evaluation policy. The tasks observed early can miss repositories, difficulty splits, application groups, or domains that would change the completed conclusion. Running cheaper

tasks first can reduce cost while showing only a narrow slice of the benchmark. A rule that decides only easy pairs can appear accurate while leaving many comparisons unresolved. What the report needs is therefore not another partial score, but a record that says whether the task outcomes observed so far are enough to support a comparison conclusion. As a concrete example, suppose a team compares a new system with a previous one on 100 shared tasks and requires a 5 percentage point gain before reporting the new system as better. After 25 observed tasks, the partial score may show a large gain. That number is still hard to interpret if all 25 tasks came from the same repository family, if harder domains have not appeared, or if the rule would abstain on most close comparisons. A decision record makes those conditions explicit: it can decide that the tested agent system is better by the required amount, decide that it is not, request more outcomes, or state that the current budget is not enough. ParEvalLayer turns that example into a decision layer for an evaluation service. It reads paired task outcomes for the two agent systems, together with a comparison policy chosen in advance. That policy fixes the required improvement amount, task order, coverage requirement, decision-error target, allowed unresolved rate, and task or resource limit. For the current partial run, the layer either decides that the tested agent system is better by the required amount, decides that it is not, requests more task outcomes, or abstains because the budget ended without enough evidence. The corresponding actions are promote, reject, continue, and abstain. We evaluate ParEvalLayer by replaying completed public benchmark data as if each evaluation had stopped earlier; we call this completed-record replay. For each pair of agent systems, the completed public record defines whether the tested agent system is better by the required amount on the shared tasks. At each budget, we reveal only the task outcomes observed by that point, apply the policy chosen in advance, and compare each promote or reject action with the conclusion from the completed data. Continue and abstain remain unresolved. We use this replay target because released benchmark reports usually expose a fixed task-outcome record rather than a live stopping process. ParEvalLayer is aimed at in-progress comparisons rather than benchmark compression. Recent task-reduction work asks whether selected subsets can preserve agent rankings across

systems [1]. ParEvalLayer asks a different question: for the the Terminal-Bench wall-clock check also leaves too partial outcomes already visible in a two-system comparison, is many comparisons unresolved at 95% budget. there enough evidence to report under a required improvement II. R ELATED W ORK amount, coverage rule, and task or resource limit? The benchmark set is chosen to cover different partialPrior work addresses how to reduce evaluation cost, stanevaluation risks. SWE-bench Lite and Verified test software- dardize execution, or control risk when stopping or deferring, repair leaderboards with large shared task sets [2], [3]. but it does not specify what a partial benchmark report should AppWorld and OSWorld-Verified test app-use and desktop- contain. Task-subset methods ask whether fewer tasks preserve control agents with different task-group structures [4], [5]. scores or rankings [1]. Infrastructure such as HAL standardizes tau-bench adds domain labels and cost fields for a cheap-first how tasks are run and logged [9]. Sequential and abstention ordering check [6], [7]. Terminal-Bench adds wall-clock timing methods study when a procedure can stop, defer, or control for terminal workflows [8]. The main task-count simulations risk [10]–[14]. ParEvalLayer connects these concerns at the use SWE-bench, AppWorld, OSWorld-Verified, and tau-bench reporting layer: given the task outcomes currently visible for because they expose benchmark-provided task groups. Terminal- two agent systems, what action should the partial benchmark Bench is used separately to ask whether elapsed-time budgets report contain? reduce unresolved comparisons enough to report. The replay results separate partial runs that are ready to report A. Score-Efficient Evaluation from cases that remain unresolved. At the 0 pp threshold under Score-efficient evaluation asks how many examples, queries, the primary policy, ParEvalLayer reaches the same conclusions or benchmark tasks are needed to estimate a score reliably. as completed evaluations after 15% to 25% of task outcomes HELM standardizes broad language-model evaluation scenarios on AppWorld, OSWorld-Verified, and tau-bench. SWE-bench and metrics [15]. MT-Bench and Chatbot Arena study LLMVerified requires 90% of task outcomes, SWE-bench Lite as-a-judge and pairwise arena evaluation for chat models [16]. remains unresolved by 95% when the policy only asks whether Cer-Eval and FAQ study test-data sufficiency and statistical one system scores higher, and the Terminal-Bench wall-clock guarantees for LLM evaluation [17], [18]. Finite-population subset leaves too many comparisons unresolved even at a 95% sampling tools also apply when benchmark tasks are treated elapsed-time budget. The empirical lesson is not that every as a fixed population [19], [20]. benchmark can be shortened by the same fraction. It is that Within agent evaluation, Efficient Benchmarking of AI partial evaluation needs a decision record: the action must be Agents studies whether mid-difficulty task subsets can preserve tied to the required improvement amount, task order, coverage leaderboard rankings under scaffold and temporal shifts [1]. rule, task or resource limit, and unresolved-comparison target. That objective is related but not the same as ours. A rankThis paper makes four contributions: preserving subset can make a leaderboard cheaper; it does 1) We introduce ParEvalLayer for partial LLM-agent evalu- not tell an evaluation service whether an in-progress twoations. It reads task outcomes from existing evaluation system comparison can stop, continue, or abstain under a pipelines and records whether the tested agent system required improvement amount, coverage rule, and unresolvedis better by the required amount, is not, needs more comparison target. These methods inform our uncertainty checks, but a partial outcomes, or should abstain. benchmark report needs more than a score estimate. It must also 2) We evaluate these partial decisions with completed-record state the improvement threshold, required task groups, decisionreplay: replay completed public benchmark data as if runs error target, and allowed unresolved rate. ParEvalLayer puts had stopped earlier, then check whether each judgment those choices in the decision record, so a reader can distinguish that the tested agent system is or is not better matches a supported partial decision from a partial score that is missing the conclusion from the completed data. For a given required evidence. budget grid, the minimum sufficient budget is the smallest tested budget that meets stated decision-error, task-group B. Agent Evaluation Infrastructure coverage, and unresolved-comparison targets. 3) We make partial decisions traceable to the rule that Agent-evaluation infrastructure runs tasks and records the produced them. Each record stores the action, required evidence from which benchmark reports are made. SWE-bench, improvement amount, task-selection policy, coverage AppWorld, OSWorld, and tau-bench provide the public taskstatus, and bootstrap tail probabilities, so aggregate level records used here [2], [4]–[7]. AI Agents That Matter budget claims can be traced back to the underlying policy. argues that agent evaluation should account for accuracy, cost, 4) We show that early reporting is possible in some public downstream usefulness, overfitting, and reproducibility [21]. A records, but not uniformly. At the 0 pp threshold under recent disclosure audit of LLM-agent benchmark papers makes the paper’s primary policy, AppWorld, OSWorld-Verified, a related point at the paper level: many benchmark reports and tau-bench reach decisions at 15% to 25% task budget. do not expose enough information about harnesses, inference SWE-bench Verified requires 90%, SWE-bench Lite cost, failure handling, and scoring to support independent leaves too many comparisons unresolved by 95%, and interpretation [22]. ParEvalLayer addresses a different point in

the same reporting chain: it records the policy and unresolved the required amount; reject reports that it is not better by that status behind a partial comparison. amount; continue requests more task outcomes; and abstain The Holistic Agent Leaderboard (HAL) is the closest states that the budget ended without enough evidence for either infrastructure comparison [9]. HAL standardizes evaluation final action. In simulations from completed public records, across virtual machines, models, scaffolds, and benchmarks, promote and reject are checked against the completed-record and reports the cost and implementation pitfalls of large-scale decision, while continue and abstain remain unresolved. agent evaluation. HAL-like infrastructure can produce task outcomes; ParEvalLayer starts from those outcomes and decides A. System Interface ParEvalLayer keeps the input contract small: one row per task whether the current partial record supports reporting, needs outcome, with the system identifier, task identifier, outcome, more tasks, or ends in abstention. Recent harnesses and benchmark suites also broaden what optional task group, optional resource fields, and provenance. agent evaluation records can contain. General Agent Evaluation, Once both systems have an outcome for the same task, the A2Perf, ProSoftArena, TAM Bench, Terminal-Bench, and layer forms the paired difference and applies the decision policy AndroidWorld add common interfaces, resource measures, chosen before the run reaches the current task or resource limit. reliability-oriented outcomes, professional software tasks, task The decision record reports the action, number of observed coverage, or software-environment traces [8], [23]–[27]. These tasks, coverage status, margin, and uncertainty values used to systems improve what an evaluation pipeline can log; ParEval- produce that action. This contract keeps task execution and stopping reports Layer asks when a partial record produced from such logs is enough to report a comparison. We use Terminal-Bench separate. HAL or a similar execution system can manage virtual separately for a wall-clock resource check because the primary machines, logs, retries, raw-trajectory scoring, and execution task-count analyses require benchmark-provided task groups. cost; ParEvalLayer starts from the task outcomes those systems Trace-level cost systems such as ClawTrace are complemen- write. Using the same task-outcome representation for inditary because they explain trajectory cost rather than deciding vidual decisions and aggregate summaries keeps each budget claim tied to the decision records that produced it. whether a partial comparison is ready to report [28]. At each simulated partial run, the layer reads visible task C. Stopping, Abstention, and Risk outcomes up to a task or resource limit and writes a decision Sequential testing, confidence sequences, abstention methods, record with the action, coverage status, partial margin, and and conformal risk control provide tools for stopping under uncertainty check. Completed-record fields are used only for uncertainty, deferring predictions, or controlling risk when after-the-fact replay checks, not as inputs to the stopping rule. predictions are withheld [10]–[14]. They do not by themselves specify what a partial benchmark report should contain. B. Decision Target Each comparison is paired at the task level. Whenever both ParEvalLayer uses these ideas to answer a reporting question: which action should be reported for a partial two-system outcomes are available, the same task compares the two systems. Public leaderboard records do not always provide exactly the comparison, and which rule produced it? ParEvalLayer uses abstention as a benchmark-reporting same task set for every entry, so the comparison is defined on action rather than a prediction outcome. For each completed the shared usable overlap for each ordered pair. For an ordered comparison, A is the tested agent system and public record, the simulation hides later task outcomes and asks whether the outcomes observed so far lead to the same B is the comparison agent system. Let TA,B = {t1 , . . . , tNA,B } conclusion as the completed evaluation under thresholds, be the tasks with usable outcomes for both systems after sourcecoverage rules, and abstention targets chosen before those specific filtering. Tables report median pair overlap when NA,B later outcomes are revealed. If not, the decision record states varies across pairs. For readability, we write T and N below when the ordered pair is clear. continue or abstain instead of forcing promote or reject. Each task outcome is a normalized score, Yi,A , Yi,B ∈ [0, 1]. III. PAR E VAL L AYER Binary outcomes are the common case. When a public record ParEvalLayer answers a narrow reporting question: do the contains repeated runs for the same system and task, we average task outcomes observed so far justify a comparison decision those runs before forming pairs. The paired task difference is now? Figure 1 shows the contract. An evaluation service Di = Yi,A − Yi,B , (1) writes task-level outcomes; ParEvalLayer reads those outcomes, applies the comparison policy chosen in advance, and writes a and the paired improvement on the completed shared task decision record for the current budget. record is N The contract has four parts: input and output fields, the 1 X ∆ = Di . (2) full completed-record decision used in replay, the actions and N i=1 aggregate error rates, and the coverage rule and bootstrap check used by the primary policy. We use budget for the task or resource limit at which a partial The decision record has four actions. Promote reports that run is evaluated. In the task-count analyses, b is a fraction of the tested agent system is better than the baseline by more than the completed shared task set; in the resource check, the same

From task outcomes to a decision record Evaluation

Outcome rows

ParEvalLayer

Decision record

Decision action

Run tasks Check outputs

task_id, system_id outcome, group cost/time

threshold, order coverage, limit

action coverage status reason

promote, reject continue abstain

How replay tests the layer Completed data

Hide later outcomes

Check judgment

All outcomes Completed conclusion

Reveal partial run At each budget

Compare with completed conclusion

Fig. 1. ParEvalLayer reads task outcome rows and a comparison policy chosen in advance, then writes a decision record for the current budget.

notation refers to a fraction of elapsed time. For a subset Sb observed by budget b, the partial paired improvement is X bb = 1 ∆ Di . (3) |Sb |

the reject action but the completed-record decision is promote. Abstentions remain in the denominator for the class assigned by the completed record, but they count as neither correct decisions nor decision errors. Otherwise a rule could decide only easy i:ti ∈Sb cases and appear accurate while leaving most comparisons Before a partial comparison, the improvement threshold δ unanswered. For aggregate rates, one evaluation case is an ordered pair is chosen in advance. On the available overlap, the completedrecord decision is promote when ∆full > δ and reject otherwise. p = (A, B), a task order r, and a budget b. We compute A 0 percentage point (0 pp) threshold asks only whether false-promotion rates over cases where the completed-record A’s paired success rate is strictly higher than B’s. Positive decision is reject, and false-rejection rates over cases where it thresholds, such as 5 pp or 10 pp, require a larger margin is promote. Coverage failure and abstention rates are averaged before promotion. The completed-record decision is defined on over all cases. the shared public overlap for the ordered pair; tasks missing We read directed pairs as decision instances, not independent from either entry are not part of that comparison. systems. The two directions of a pair share outcomes, but they This replay target matches what many benchmark reports answer different one-sided questions. release: a fixed task-outcome record with a stated comparison Aggregate summaries are derived from per-comparison rule. decision records. A reader can inspect the decision threshold, The ordered comparison matters. A versus B asks whether task policy, missing task groups, and abstentions behind each the tested system is better than the comparison system by more partial action rather than only a plotted budget curve. than the threshold. B versus A asks the reverse question. The main analysis therefore keeps ordered comparisons, and we separately test what changes when only one direction is kept D. Task-Group Coverage from each unordered pair. Task-group coverage prevents one specific mistake. A partial C. Actions and Decision Records run can appear decisive inside a narrow slice while missing At a given budget, the policy receives the observed subset Sb . groups that would change the decision from the completed Before the configured budget limit, it can continue evaluation. record. The groups come from public metadata: repository At the limit, it can promote A, return reject, or abstain. A for SWE-bench, split by difficulty for AppWorld, application promote action means the observed tasks support the conclusion group for OSWorld-Verified, and domain for tau-bench. In the that the partial paired difference exceeds δ. A reject action equations, we call these groups strata. The coverage rule asks a simple question before any final means the observed tasks support the conclusion that the partial action: has the partial run seen enough tasks from every required paired difference does not exceed δ. Continue and abstain state group for this budget? which task groups or evidence conditions are still missing. We measure false promotions and false rejections against the For each ordered pair, coverage is computed on the shared completed-record decision. A false promotion occurs when the task set TA,B . Let qi be the stratum for task ti , ng the number partial policy promotes A but the completed-record decision is of shared tasks in stratum g, b ∈ (0, 1] the budget fraction, reject; a false rejection occurs when the partial policy returns and kb = ⌈bN ⌉. Strata absent from a pair’s shared overlap are

Input: visible paired outcomes Sb , threshold δ, required groups, budget status, tail cutoff a. 1. If task-group coverage fails, return CONTINUE if budget remains, otherwise ABSTAIN. b b and bootstrap tails Rpromote , Rreject . 2. Compute ∆ 3. If budget remains, return CONTINUE. At the budget limit, return PROMOTE if b b > δ and Rpromote ≤ a, REJECT if ∆ b b ≤ δ and Rreject ≤ a, otherwise ∆ ABSTAIN.

Fig. 2. Decision routine used for each partial comparison.

excluded for that pair. The primary required count for stratum g is rg (b) = max{1, min{ng , ⌊kb ng /N ⌋}}. (4) An observed subset satisfies task-group coverage when it contains at least rg (b) tasks from every required group. The rule has two roles: every nonempty group must appear at least once when possible, and larger groups receive proportionally larger required counts. For example, if N = 100, b = 0.20, and three groups contain 50, 30, and P 20 shared tasks, the required counts are 10, 6, and 4. If g rg (b) > kb , the target is impossible at that budget and the rule cannot issue a final decision. A coverage failure occurs when a policy returns promote or reject before satisfying the required counts. The rule deliberately favors a simple reporting check over an optimal task-allocation procedure: it makes missing-slice failures visible before a partial comparison is reported.

A budget is sufficient only if it avoids three misleading outcomes: wrong decisions, final decisions before required task groups are represented, and a low error rate caused by leaving many comparisons unresolved. For budget b, threshold δ, decision-error target α, coverage-failure target γ, and abstention target η, we require FPRcond (b) ≤ α,

(7)

FRRcond (b) ≤ α,

(8)

coverage failure(b) ≤ γ,

(9)

abstain(b) ≤ η.

(10)

Unless otherwise stated, α = 0.05, γ = 0.05, and η = 0.25. These defaults define the paper’s primary reporting policy. The 5% decision-error targets require false promotions and false rejections to be rare, and the 5% coverage target requires promote or reject actions before required groups appear to be rare. FPRcond is computed over completed-record negative cases and counts false promotions; FRRcond is computed over completed-record positive cases and counts false rejections. Abstentions remain in the relevant class denominator and are tracked separately, so these are class-conditional disagreement rates over all cases rather than selective risks among only decided cases. The 25% abstention target prevents a rule from looking successful by deciding only easy comparisons. Sufficient budgets are therefore reported relative to the stated policy. Different evaluation settings can choose different targets E. Decision Rule and Sufficiency and thresholds before the run and report those choices in the The primary policy applies the coverage rule before it condecision record. We also report budgets that meet the two siders the score margin. The bootstrap rule checks uncertainty decision-error targets and the coverage target but still abstain over the observed paired task differences. We use it because too often to be useful as a decision procedure. the layer sees paired task outcomes and does not require a parametric model for their observed differences. Section V-C IV. E VALUATION P ROTOCOL checks whether paired-test, task-selection, and budget-grid A. Public Records variants change the reporting conclusion. Figure 2 summarizes the routine used for both partial-run ParEvalLayer needs task-level outcomes for matched tasks, decisions and batch simulations. The routine first asks whether not only aggregate leaderboard scores. We therefore use public required task groups are represented. Only then does it test records that expose outcomes for multiple agent systems, whether the observed paired margin is far enough from the enough shared tasks to form directed pairs, and benchmarkthreshold to support a final action. provided groups when coverage is part of the reporting rule. Once coverage holds for Sb , the policy estimates uncertainty Table I summarizes the task-count sources. Pair counts are by resampling the observed paired differences {Di : ti ∈ Sb }. ordered comparisons, with unordered counts in parentheses. Each resample draws |Sb | tasks with replacement and computes Positive counts are ordered comparisons where the completed b ∗ . With bootstrap tail cutoff a, define a bootstrap mean ∆ record promotes the tested system on the shared available b ∗ overlap. b ∗b ≤ δ), Rpromote = Pr(∆ (5) The main task-count analyses use SWE-bench Lite, SWE∗ ∗ bench Verified, AppWorld, OSWorld-Verified, and tau-bench b Rreject = Pr(∆b > δ). (6) because they provide task outcomes and task groups suitable The tail cutoff is a reporting rule for the current budget cap. The for the coverage rule. Terminal-Bench is evaluated separately b b > δ and Rpromote ≤ a. It returns because its public files provide wall-clock timings for usable policy promotes when ∆ b the reject action when ∆b ≤ δ and Rreject ≤ a. Otherwise it task cells, but not the benchmark-provided task groups needed continues if budget remains and abstains when the configured for the primary coverage rule. Section V-C asks whether budget is exhausted. Unless otherwise stated, a = 0.05. The stopping by elapsed time reduces unresolved comparisons bootstrap tail is the empirical uncertainty check used inside enough to report. The tau-bench grid has only 12 directed the replay policy; repeated live looks at the same run would pairs, so we read it as a cost-ordering check rather than a require a separate time-uniform sequential boundary. broad estimate for all conversational-agent benchmarks.

TABLE I P UBLIC TASK - LEVEL RECORDS USED FOR COMPLETED - RECORD REPLAY OF PARTIAL - EVALUATION DECISIONS . Source

Pairs

Task units

Pos. 0 pp

Pos. 5 pp

Pos. 10 pp

Groups

Cost

Role in analysis

SWE-bench Lite

552 (276)

275

199

154

repository

no

SWE-bench Verified

156 (78)

77

60

54

repository

no

AppWorld leaderboard OSWorld-Verified

306 (153) 56 (28)

153 28

128 27

108 21

split by difficulty application

no no

tau-bench public grid

12 (6)

300 median overlap 496 median overlap 585 tasks 359 median overlap 278 domain-task items

6

5

2

domain

yes

software-repair task-count software-repair high-budget case app-use task-count desktop-control task-count cost-ordering case

The source-specific choices are filtering, grouping, repeatedrun handling, and cost availability. For SWE-bench, we keep near-complete leaderboard entries, defined as entries with outcomes for at least 95% of the split; these records do not include per-task cost, so they support task-count budgets only. AppWorld contributes released task-level success labels and split-by-difficulty groups. OSWorld-Verified contributes application-grouped task outcomes for eight released configurations; when two runs are available for the same configuration and task, we average the task score before pairing. This treats the released record as fixed, while run-level variation is discussed in Section VI. tau-bench contributes domain-task outcomes and cost fields, so it supports the cheap-first ordering check. We exclude banking-knowledge trajectories because retrieval configuration differs across the selected entries. B. Policies and Budget Settings

headline sufficiency claims always come from the 5 percentage point sweep. In replay, task execution is already represented as normalized task outcomes. Partial-run and aggregate analyses use the same task rows, and completed-record fields used for after-the-fact checks remain outside the stopping rule. The stopping rule therefore sees only the outcomes available at the current task or resource budget. V. R ESULTS The results separate three cases that a partial score would conflate: partial runs whose observed outcomes already support the same conclusion as the completed evaluation, partial runs that leave too many comparisons unresolved, and checks showing that changing the budget measure or uncertainty rule does not remove the need for a decision record. We call a budget sufficient only when the decision-error, coverage, and abstention targets in Section III-E all hold; “none by 95%” means no tested budget met all targets. The intervals in Table II describe directed-pair variability in unresolvedcomparison rates at the listed 0 pp budget; they are not additional decision criteria.

The replay simulation fixes the reporting rule before any partial-run outcomes are computed. The rule includes the improvement threshold, source-specific task groups, taskselection policy, decision rule, budget grid, and sufficiency targets. The completed data define the comparison decision. At each budget, we reveal only the outcomes observed by that point, apply ParEvalLayer, and check promote or reject actions A. Replay Finds Early Decisions in Some Benchmarks against the decision from the completed data. Figure 3 and Table II show where completed-record replay The primary policy asks whether the observed task set covers finds early decisions. At 0 pp, AppWorld reaches the reporting the required task groups and is decisive enough to report. targets at 15% task budget, OSWorld-Verified at 20%, and It first checks source-specific task-group coverage and then tau-bench at 25%. In these benchmarks, ParEvalLayer reaches applies the bootstrap rule to decide whether the observed margin the same conclusion as the completed evaluation using only supports promote or reject. The comparison policies remove 15% to 25% of task outcomes under the stated policy. one safeguard at a time. Forced uniform and forced groupThe early-decision pattern does not hold for every benchmark. aware policies ignore uncertainty. Paired-test rules replace the SWE-bench Verified reaches the targets only at 90% task bootstrap tail rule. Neyman allocation uses group variances budget, and SWE-bench Lite remains insufficient at 95% under from the completed record, and Serfling finite-population the primary repository coverage rule. The decision record makes bounds use a conservative finite-population inequality. Together, this distinction explicit: some partial evaluations are ready to these variants separate the effects of task ordering, uncertainty report, while others remain unresolved at the tested budget. estimation, and prior variance information. The limiting factor is often abstention rather than wrong For SWE-bench, AppWorld, and OSWorld-Verified, we decisions. SWE-bench Lite meets the decision-error and sample 500 task orders per ordered comparison and 200 coverage targets at 25% task budget at 0 pp, but 55.92% of bootstrap samples per policy evaluation. For tau-bench, we comparisons remain unresolved. Even at 95%, 27.25% remain sample 2,000 task orders and 500 bootstrap samples. The unresolved, above the 25% target. SWE-bench Verified shows main search uses budgets from 5% to 95% in 5 percentage the same issue at early budgets: at 25%, the primary policy point increments. We keep a small round-budget check, but the meets the error and coverage targets but leaves 93.64% of

When do partial runs meet the unresolved-comparison target? 0 pp threshold, 5 pp budget grid. Green band: at most 25% unresolved; open markers: first budgets meeting all targets.

Unresolved comparisons (%)

100

80

60 no sufficient budget by 95%

40

20% 25%

90%

25% target

SWE-bench Lite SWE-bench Verified tau-bench OSWorld-Verified AppWorld

20

15% unresolved target

0 5

15

25

50

75

95

Task budget (%) Fig. 3. Unresolved comparisons at the 0 pp threshold across the 5 percentage point budget grid. Lower is better; budgets above 25% unresolved comparisons are insufficient under the primary policy. TABLE II M INIMUM SUFFICIENT TASK BUDGETS UNDER THE PRIMARY POLICY.

Source

0 pp

5 pp

10 pp

0 pp unresolved interval

Result

AppWorld

15%

15%

15%

20.60% [17.17, 23.86]

OSWorld-Verified tau-bench SWE-bench Verified

20% 25% 90%

20% 30% 90%

30% 35% 90%

23.61% [16.19, 31.03] 23.35% [7.25, 43.84] 24.22% [20.36, 28.33]

none by 95%

55%

35%

27.25% [24.35, 30.29]

15% sufficient at all thresholds. 20% to 30% sufficient. 25% to 35% sufficient. 90% sufficient at all thresholds. No 0 pp budget by 95%.

SWE-bench Lite

comparisons unresolved. The abstention target is reached only at 90%. The threshold and task-selection policy are part of the reporting claim. SWE-bench Lite has no sufficient budget at 0 pp, but reaches 55% at 5 pp and 35% at 10 pp. AppWorld remains sufficient at 15% across the three thresholds, while taubench and OSWorld-Verified move as the threshold changes. Because tau-bench contributes 12 directed pairs, its sufficient budgets are evidence that the policy catches a cost-ordering failure mode in this grid, not a precise benchmark-wide rate estimate. On AppWorld, the coverage-aware bootstrap policy reaches sufficiency at 15%, but split-difficulty forced evaluation needs 50% task budget and uniform forced evaluation fails the coverage target at 25%. A budget number without the threshold and task-selection policy is therefore not an interpretable reporting claim. B. Partial Scores Hide Reporting Conditions A score-only partial report can hide whether the observed tasks are enough to stop and report the comparison. At 25%

task budget, uniform forced evaluation has 99.96% coverage failure on both SWE-bench Lite and AppWorld, so a partial score from that subset can look precise while missing required task groups. Task order can create a different failure. In tau-bench, costaware forced evaluation at 25% task budget uses only 11.51% of tested-system cost on average, but it has 100% coverage failure. It also produces wrong pairwise decisions at all three thresholds, including false rejections in 4 of 5 positive 5 pp cases and 2 of 2 positive 10 pp cases. Saving cost is not enough if the observed tasks no longer support the comparison. These examples show why unresolved cases must be reported separately from decision errors. Without that field, a partial score or an accuracy summary over only decided cases can make an incomplete comparison look ready to report. C. Resource and Decision-Rule Checks The remaining checks ask whether the main pattern depends on measuring budget by task count or on the bootstrap uncertainty rule. Terminal-Bench lets us replace task count

with elapsed time. We use public Terminal-Bench 2.0 result The reported budgets are meaningful only through the policy files [8]. After requiring at least 80 shared tasks per pair, the recorded with the decision. The 25% unresolved-comparison usable subset contains 72 directed comparisons over 89 shared target is one such input: stricter targets would demand later terminal tasks. Wall-clock stage timing is available for all budgets and looser targets would allow earlier reporting. usable task cells, but native USD cost and token fields are too Thresholds, task groups, coverage requirements, orientation sparse for a cross-system cost analysis. choices, bootstrap cutoffs, and abstention targets all affect Elapsed-time budgets do not remove the reporting problem in whether a partial evaluation is ready to report, so a task fraction this subset. At the 0 pp threshold, no budget measured by task is meaningful only with those choices. The bootstrap tail fraction, tested-system elapsed time, or pair elapsed time meets rule supports the budget comparisons in this paper; repeated the primary targets by 95%. At 95%, wrong final decisions live looks at the same run would require a sequential test are rare: false-promotion rates are 0.44% to 0.62%, and false- or confidence-sequence design validated for the benchmark’s rejection rates are 0.00%. The limiting factor is unresolved sampling and dependence structure. comparisons, which remain at 36.10% to 36.31%, above the The uncertainty intervals should be read descriptively 25% target. Reports based on resource fractions still need an because ordered pairs share systems, tasks, and outcomes. explicit decision rule and must state how many comparisons Clustered or hierarchical uncertainty estimates would be may remain unresolved. stronger for future records with enough independent systems The decision-rule checks point in the same direction. Re- and repeated runs. laxing the bootstrap cutoff moves some minimum budgets, The released metadata supports coverage checks at recogas expected, but it does not turn high-budget benchmarks nizable benchmark slices. Our rule uses repository, difficulty into early-reporting cases. On AppWorld, the sufficient budget split, application group, and domain labels to report whether a moves from 25% with cutoff a = 0.02, to 15% with a = 0.05, partial run skipped an important group before a comparison is to 10% with a = 0.10. SWE-bench Verified remains a high- reported. Finer behavioral strata or learned task clusters require budget case across these cutoffs. pre-evaluation definitions and separate validation. The main pattern is not driven by the number of bootstrap Resource claims differ by source. OSWorld-Verified supports samples. When the primary coverage-aware bootstrap rule is task-count checks for the released verified configurations, rerun with 100, 200, and 500 bootstrap samples, the widest Terminal-Bench supports wall-clock checks, and tau-bench 5th-to-95th percentile span for unresolved comparisons across supports cleaner cost fields than the other sources. Token, checked seeds, benchmarks, thresholds, and budgets is 1.61 compute, or USD reporting decisions would require more pp. The largest observed conditional decision-error rate is complete resource logs across systems and tasks. 1.40%, below the 5% target. Paired-normal tests give the same sufficient-budget pattern for AppWorld, tau-bench, and SWEVII. C ONCLUSION bench Verified at all three thresholds; on SWE-bench Lite, they still find no sufficient 0 pp budget by 95%. Partial agent evaluations should report decisions, not only The budget grid and comparison orientation affect how the scores. ParEvalLayer makes that reporting contract explicit: results should be read. The 5 percentage point budget grid it reads paired task outcomes observed so far, applies a avoids a misleading SWE-bench Lite summary because, under comparison policy chosen in advance, and records whether the primary coverage rule, SWE-bench Lite still misses the the tested agent system is better by the required amount, limit on unresolved comparisons by 95%. Directed comparisons is not, needs more outcomes, or should abstain. Completed preserve the one-sided decision question. If SWE-bench Lite is public records let us test whether those partial-run judgments collapsed to one orientation, the 0 pp case contains 275 positive agree with the conclusions from completed data while keeping decisions and only 1 negative decision, which no longer tests unresolved comparisons visible. both directions symmetrically. The replay results show that the budget needed before the policy can decide varies sharply across benchmarks and policies. VI. L IMITATIONS At the 0 pp threshold, AppWorld, OSWorld-Verified, and These experiments support claims about partial decisions tau-bench can reach the same conclusion as the completed in replayed public benchmark records, where later outcomes evaluation from early partial runs, while SWE-bench Verified can be hidden after the fact. They do not by themselves requires much later budgets and SWE-bench Lite and the establish a live stopping guarantee. A live benchmark service Terminal-Bench wall-clock subset remain limited by unresolved would also need task orders and stopping rules specified before comparisons. This variation is the main empirical finding: execution, complete resource logs, explicit retry and verifier- partial evidence can be enough to report in some records, failure policies, and enough repeated attempts to separate task but only when the report states the decision rule and how variation from run variation. When public sources provide many comparisons remain unresolved. A benchmark service repeated runs for the same system and task, we average them can support this practice by recording the decision rule, fixing before forming task pairs, so the analysis treats the released task order before execution, and keeping complete resource record as fixed. logs.

R EFERENCES [1] F. Ndzomga, “Efficient benchmarking of AI agents,” arXiv preprint arXiv:2603.23749, 2026. [2] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” in International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum? id=VTF8yNQM66 [3] SWE-bench Team, “SWE-bench leaderboards,” 2026, accessed 12 June 2026. [Online]. Available: https://www.swebench.com/ [4] H. Trivedi, T. Khot, M. Hartmann, R. Manku, V. Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian, “AppWorld: A controllable world of apps and people for benchmarking interactive coding agents,” arXiv preprint arXiv:2407.18901, 2024. [5] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu, “OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” arXiv preprint arXiv:2404.07972, 2024. [6] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ -bench: A benchmark for tool-agent-user interaction in real-world domains,” arXiv preprint arXiv:2406.12045, 2024. [7] V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan, “τ 2 -Bench: Evaluating conversational agents in a dual-control environment,” arXiv preprint arXiv:2506.07982, 2025. [8] M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, J. Jitsev, D. Lu, O. M. Mastromichalakis, Z. Xu, Z. Chen, Y. Liu, R. Zhang, L. L. Chen, A. Kashyap, J.-L. Uslu, J. Li, J. Wu, M. Yan, S. Bian, V. Sharma, K. Sun, S. Dillmann, A. Anand, A. Lanpouthakoun, B. Koopah, C. Hu, E. Guha, G. H. S. Dreiman, J. Zhu, K. Krauth, L. Zhong, N. Muennighoff, R. Amanfu, S. Tan, S. Pimpalgaonkar, T. Aggarwal, X. Lin, X. Lan, X. Zhao, Y. Liang, Y. Wang, Z. Wang, C. Zhou, D. Heineman, H. Liu, H. Trivedi, J. Yang, J. Lin, M. Shetty, M. Yang, N. Omi, N. Raoof, S. Li, T. Y. Zhuo, W. Lin, Y. Dai, Y. Wang, W. Chai, S. Zhou, D. Wahdany, Z. She, J. Hu, Z. Dong, Y. Zhu, S. Cui, A. Saiyed, A. Kolbeinsson, J. Hu, C. M. Rytting, R. Marten, Y. Wang, A. Dimakis, A. Konwinski, and L. Schmidt, “Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces,” arXiv preprint arXiv:2601.11868, 2026. [9] S. Kapoor, B. Stroebl, P. Kirgis, N. Nadgir, Z. S. Siegel, B. Wei, T. Xue, Z. Chen, F. Chen, S. Utpala, F. Ndzomga, D. Oruganty, S. Luskin, K. Liu, B. Yu, A. Arora, D. Hahm, H. Trivedi, H. Sun, J. Lee, T. Jin, Y. Mai, Y. Zhou, Y. Zhu, R. Bommasani, D. Kang, D. Song, P. Henderson, Y. Su, P. Liang, and A. Narayanan, “Holistic Agent Leaderboard: The missing infrastructure for AI agent evaluation,” arXiv preprint arXiv:2510.11977, 2025. [10] A. Wald, “Sequential tests of statistical hypotheses,” The Annals of Mathematical Statistics, vol. 16, no. 2, pp. 117–186, 1945. [11] S. R. Howard, A. Ramdas, J. McAuliffe, and J. Sekhon, “Time-uniform, nonparametric, nonasymptotic confidence sequences,” The Annals of Statistics, vol. 49, no. 2, pp. 1055–1080, 2021. [12] Y. Geifman and R. El-Yaniv, “Selectivenet: A deep neural network with an integrated reject option,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 97. PMLR, 2019, pp. 2151–2159. [13] H. Mozannar and D. Sontag, “Consistent estimators for learning to defer to an expert,” in Proceedings of the 37th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 7076–7087. [14] A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster, “Conformal risk control,” arXiv preprint arXiv:2208.02814, 2022. [15] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda, “Holistic evaluation of language models,” Transactions on Machine Learning Research, 2023.

[16] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” in Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2023. [17] G. Wang, Z. Chen, B. Li, and H. Xu, “Cer-Eval: Certifiable and cost-efficient evaluation framework for LLMs,” arXiv preprint arXiv:2505.03814, 2025. [18] S. Wu, Y. Nair, and E. J. Candès, “Efficient evaluation of LLM performance with statistical guarantees,” arXiv preprint arXiv:2601.20251, 2026. [19] R. J. Serfling, “Probability inequalities for the sum in sampling without replacement,” The Annals of Statistics, vol. 2, no. 1, pp. 39–48, 1974. [20] J. Neyman, “On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection,” Journal of the Royal Statistical Society, vol. 97, no. 4, pp. 558–625, 1934. [21] S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan, “AI agents that matter,” arXiv preprint arXiv:2407.01502, 2024. [22] M. N. Moghadasi and F. Ghaderi, “What twelve LLM agent benchmark papers disclose about themselves: A pilot audit and an open scoring schema,” arXiv preprint arXiv:2605.21404, 2026. [23] E. Bandel, A. Yehudai, L. Eden, Y. Sagron, Y. Perlitz, E. Venezian, N. Razinkov, N. Ergas, S. Shachor Ifergan, S. Shlomov, M. Jacovi, L. Choshen, L. Ein-Dor, Y. Katz, and M. Shmueli-Scheuer, “General agent evaluation,” arXiv preprint arXiv:2602.22953, 2026. [24] I. Uchendu, J. Jabbour, K. Van den Berghe, J. Runevic, M. Stewart, J. Ma, S. Krishnan, I. Gur, A. Huang, C. Bishop, P. Bailey, W. Jiang, E. M. Songhori, S. Guadarrama, J. Tan, J. K. Terry, A. Faust, and V. J. Reddi, “A2Perf: Real-world autonomous agents benchmark,” arXiv preprint arXiv:2503.03056, 2025. [25] J. Ai, Y. Feng, F. Zhang, J. Sun, Z. Li, C. Li, Y. Chang, W. Wu, R. Wang, M. Zhai, and K. Zhang, “ProSoftArena: Benchmarking hierarchical capabilities of multimodal agents in professional software environments,” arXiv preprint arXiv:2601.02399, 2025. [26] H. Jia, Y. Qian, H. Tong, X. Wu, L. Chen, and F. Wei, “Towards adaptive ML benchmarks: Web-agent-driven construction, domain expansion, and metric optimization,” arXiv preprint arXiv:2509.09321, 2025. [27] C. Rawles, S. Clinckemaillie, Y. Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell-Ajala, D. Toyama, R. Berry, D. Tyamagundlu, T. Lillicrap, and O. Riva, “AndroidWorld: A dynamic benchmarking environment for autonomous agents,” arXiv preprint arXiv:2405.14573, 2024. [28] B. Yuan, Y. Su, R. Song, S. Yang, and J. Qin, “ClawTrace: Cost-aware tracing for LLM agent skill distillation,” arXiv preprint arXiv:2604.23853, 2026.

Related documents

• TokTier: Exact Stateful Tokenization for Agentic LLM Serving#422218
• Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents#471273
• Improving Constraint Models with LLM Agents#441885
• GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation#300000
• BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure#673419