APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows

APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows

Zelin Wan

Arash Nourian

Xiaoxiao Li Nihar Nandan Postman, Inc.

Kamalakannan Nandagopal

arXiv:2608.29128v1 [cs.AI] 29 Aug 2026

{zelin.wan, arash.nourian, xiaoxiao.li, nihar.nandan, kamalakannan}@postman.com

Abstract Tool-using agents are commonly evaluated by a single bit: whether an end-toend workflow completed. This metric fails to distinguish failures that matter in production, such as expired credentials, malformed payload fields, unverified writes, or correct execution followed by incorrect final delivery. We introduce APIFlow-Bench, a fully auditable benchmark for long-horizon, dependent RESTAPI workflows that decomposes performance into seven engineering capabilities and requires agents to produce answers supported by the actual call path. In place of mining tasks, we procedurally generate synthetic API worlds forward, subtask by subtask. Each subtask is admitted only when a zero-LLM self-test triad verifies that its grader is load-bearing and an oracle establishes solvability (pass@10 ≥ 3). A subsequent adversarial audit identified and fixed six grader exploits. Evaluation is deterministic and provenance-sensitive: a state check traces a mock-minted canary through the API data flow to the response from which the answer is required to originate, while a typed answer card is verified field by field. We release all answer keys and 44,362 unredacted execution transcripts since successful completion depends on evidence available only through the correct call path. Across 19 frontier and open-weight models evaluated under one neutral scaffold, we find the following. (1) Longer dependency chains degrade success, from 93% on individual subtasks to 74% on clean 20-subtask chains and 61% when including the 8% of chain trials that a model-consensus screen flags as passed by no model. (2) Reliability separates models more than best-case capability does, with best-of-five performance spanning seven points but all-five-of-five reliability spanning 44 points above an approximately seven-point noise floor. (3) The standard independent-error account of compounding failure does not fit the data: observed pass rates on 20-subtask chains are 33 percentage points higher than the product of the corresponding subtask-level pass rates. This gap is not explained by agents simply failing late in execution. On the clean slice, 77% of failing runs drove the world to the correct final state and failed only at final delivery.

1

Introduction

In enterprise systems, consequential work is often performed through API calls embedded in business logic. An agent operating in such environments needs to authenticate, inspect state, interpret failures, repair malformed requests, retry safely, preserve dependencies across calls, and return an output that a downstream system can trust. Existing state-based benchmarks, such as τ -bench [Yao et al., 2025] and AppWorld [Trivedi et al., 2024], appropriately evaluate whether an agent reaches a target world state. However, a single workflow-level pass/fail outcome fails to distinguish an agent that successfully recovers from an authentication error from one that follows an error-free trajectory. In addition, such an outcome conflates an agent that correctly executes the workflow with an agent that reaches the correct backend state but fails to return the required result.

We introduce APIFlow-Bench, a benchmark for long, dependent REST-API workflows designed to expose this failure surface. APIFlow-Bench evaluates seven capabilities separately: authentication, discovery, schema repair, multistep execution, error recovery, pagination, and statefulness. Each task begins as a realistic service ticket: an initial failing API call, a truthful error signal, and a request to complete a job. Tasks can be composed into cumulative workflows of up to 20 dependent subtasks within one evolving synthetic API world. Generated tasks provide privacy, scale, controlled variation, and reproducibility, but they introduce a central validity question: are the tasks solvable, correctly graded, and resistant to shortcuts? We therefore treat benchmark validation as a first-class contribution. A generated subtask enters the bank only after passing a zero-LLM self-test triad designed to expose grader failures and an oracle solvability gate evaluated in the same harness used for models. Each assembled chain also passes a two-sided golden replay. Finally, an independent adversarial reviewer attempted to exploit the completed grader. The audit identified six grader exploits, all of which we fixed. APIFlow-Bench uses deterministic, provenance-gated evaluation. A run has to satisfy two conditions. First, a state check verifies that a mock-minted canary reaches the final workspace through the API data flow required by the task. Second, a typed answer card is checked field by field. Contributions. 1. APIFlow-Bench: a frozen, fully public benchmark comprising 467 tasks across 13 generated REST API worlds, 241 individual subtasks and 226 cumulative chains of up to 20 subtasks. It evaluates seven capability axes using a seven-tool action space over five typed entity kinds. We release a content-hash-pinned manifest and 44,362 unredacted execution transcripts (§3).1 2. Provenance-gated grading: a deterministic two-surface evaluator that requires both correct backend state and correct final delivery. The state check traces a mock-minted canary in the final workspace to the API response from which it originates. A typed answer card is then compared field by field (§5). 3. A validation stack for generated benchmarks: a zero-LLM grader self-test triad, an oracle solvability criterion of pass@10 ≥ 3, two-sided golden replay for assembled chains, and an adversarial audit of the completed grader that revealed six exploits missed by automated validation (§4). 4. An empirical anatomy of long-horizon API-agent failure: across 19 models under a common scaffold, we find sharp degradation with chain length, a large reliability gap that best-of-k masks, systematic departures from independent-error predictions, and a predominance of final-delivery failures after otherwise correct state execution (§8). 5. A model-consensus screen for cells no model passes: a practical diagnostic for generated benchmarks based on world×length cells unsolved by a broad model panel. The screen flags 18 of 226 chain cells, or 8.0% of chain trials. For each flagged cell a reference solution passes the assembled grader (the tasks are solvable), while the cause of the zero pass rate is left open. We report filtered and unfiltered results and document where this heuristic fails (§7).

2

Related Work

Function-call and API-use evaluation: Function-calling benchmarks evaluate whether a model selects appropriate tools and constructs valid arguments. API-Bank [Li et al., 2023], ToolLLM/ToolBench [Qin et al., 2024], Gorilla [Patil et al., 2024], and the Berkeley Function Calling Leaderboard (BFCL) [Patil et al., 2025] have established rigorous settings for function selection, argument construction, and, in some cases, serial, parallel, multi-turn, or multi-step calls. APIFlow-Bench addresses a different unit of analysis: the completion of a dependent REST workflow where correct calls may be insufficient. It measures how call-level competence composes into reliable, stateful workflow execution. 1

Harness and frozen bank: github.com/postmanlabs/APIFlow-Bench; raw transcripts: github.com/ postmanlabs/apiflow-bench-transcripts; leaderboard: postman.com/ai/apiflow-leaderboard.

2

Stateful interactive environments: A complementary line of work evaluates agents in environments with persistent state and grades task outcomes instead of matching an action transcript. AppWorld [Trivedi et al., 2024] evaluates interactive coding agents in a controllable world of applications and people, including task success and collateral effects. τ -bench [Yao et al., 2025] evaluates customer-service agents that interact with simulated users and domain APIs under policy constraints, comparing final database states to annotated goal states. It also introduced passk for repeated-trial reliability. ToolSandbox [Lu et al., 2024] evaluates stateful, conversational tool use, including implicit state dependencies and dynamically assessed intermediate and final milestones. Broader agent benchmarks such as AgentBench [Liu et al., 2024], WebArena [Zhou et al., 2024], OSWorld [Xie et al., 2024], SWE-bench [Jimenez et al., 2024], and Terminal-Bench [Merrill et al., 2026] evaluate analogous capabilities in web, desktop, software-engineering, and terminal settings. We share the central premise of this literature: final world state is a necessary component of faithful agent evaluation. Table 1 (Appendix C) tabulates the coverage comparison. APIFlow-Bench differs in granularity and experimental control. It retains state-based evaluation while separately exposing the seven capabilities introduced in Section 1. Long-horizon reliability: Long-horizon tasks reveal failure modes that are often hidden by short, independently sampled instances. τ -bench’s passk evaluates the probability that all k independent executions succeed, not whether a single execution does. APIFlow-Bench adopts repeated-trial evaluation to distinguish reach from reliability: best-of-five performance measures whether a model can succeed, while all-five-of-five measures whether it always succeeds under the same task conditions. Our controlled chain construction permits a more specific test of horizon effects. We grow dependent subtasks forward within the same synthetic API world, so chain length changes without intentionally changing the tool surface, task distribution, or labeling rule. This design lets us compare observed long-chain success with predictions derived from subtask-level rates. Benchmark integrity and generated tasks: Public benchmarks can become unreliable measurements through training-data contamination, benchmark saturation, underspecified task validity, and evaluators that introduce uncontrolled judgment error. BIG-Bench [Srivastava et al., 2023] publishes a canary intended to help model developers exclude benchmark data from training corpora. Deterministic evaluators also avoid known risks of LLM-as-a-judge scoring, including position, verbosity, and self-preference biases documented in recent judge-evaluation work [Zheng et al., 2023]. These concerns motivate our content-hash-pinned manifest, public artifacts, deterministic ranking evaluator, and use of any LLM verification only for non-ranking labels. Generated interactive benchmarks additionally require evidence that tasks are solvable, graders are sound, and evaluation shortcuts are unavailable. APIFlow-Bench supplies this evidence through the four-safeguard validation stack of §4. We further use model-panel consensus as a diagnostic (i.e., non-conclusive) screen for cells no model passes: a world×length cell unsolved by all models in a broad panel receives manual investigation. This screen identified 18 cells that no model passes in our bank, while our analysis also documents a further such cell that it misses.

3

APIFlow-Bench: Interface, Tasks, and Composition

The action space is a fixed contract of seven polymorphic tools. An agent acts through exactly seven tools (read, write, edit, search, execute, clarify, report_blocked), each polymorphic over five typed EntityRef kinds (request, response, variable, spec, handle). Hence, protocol differences are parameters, not new tools, and the measured surface stays identical: the REST bank, GraphQL adapter, and MCP adapter share those seven. clarify and report_blocked let the agent stop. An abstaining trial forfeits the ranked metric but is recorded separately, since a benchmark that fails to tell a calibrated stop from a confabulated answer rewards guessing (§4). Tasks are failure-first tickets. Each task opens in the state developers are in when they ask for help, with something broken, the offending call inline, and the error carrying a hint. In the stn-12 ticket (Figure 1a), a nightly pull fails with a 401 on an expired token. The hint names the refresh route, and the task requests the live reading: its value and its reference identifier. Across the seven axes the same structure repeats. Table 2 (Appendix D) lists the axes. Each appears at each difficulty level and is scored separately, so a model’s profile is a vector. 3

(a) A failure-first task

(b) A neutral seven-tool scaffold

My nightly pull 401s on the readings API. This is the broken call:

agent = Inspect AI react() loop; no planner, summariser or retry logic.

GET /v3/stations/stn-12/readings Authorization: Bearer tk_live_OLD {"error":"token_expired", "hint": "refresh via POST /v3/oauth/token"} Refresh auth and get the live reading for stn-12: the value and the ref.

read write edit search prompt execute clarify report_blocked polymorphic over five entity kinds: request response variable spec handle

axis AUTHENTICATION · difficulty HARD canary GUID in the task.yaml header

deterministic mock REST backend expiring tokens · partial writes 429 with Retry-After · paging deprecated /v1 decoys it mints the canary value only on the correct path

(c) Grading is G1 ∧ G2 G1 · state—the work happened final state

The earned canary is in the final workspace, and a data-flow check ties it back to the response it had to come from.

∧ G2 · answer—the user was served A typed answer card, compared field by field (numeric / currency / string / set), never by substring.

PASS—both surfaces, or nothing

Figure 1: One APIFlow-Bench task, end to end: the stn-12 ticket, this paper’s running example. (a) A failure-first task; (b) seven polymorphic tools over five typed entity kinds against a deterministic local mock, under Inspect AI’s stock react() loop; (c) a pass requires both surfaces: the state check verifies the backend state produced by the calls; the answer card verifies the answer returned to the user.

Worlds, subtasks, solo and chain tasks. The unit of generation is a world: one procedurally generated mock-API environment carrying a multi-segment storyline. Each storyline position is a segment; the task frozen from segment i is its subtask, and each subtask also ships as a standalone solo task. A cumulative chain-1tok task starts from the initial world; its instruction is one compressed goal centered on segment k, and reaching it means re-earning whatever earlier state the terminal grading gates on; planted identifiers make earlier segments load-bearing where a coupling exists (coupling is sparse by design; §4). Efficient runs finish well under one call per nominal segment (median 14 among clean passing chain-20 runs). Grading gates the terminal answer card, the terminal state check, and each dependency-critical earlier state check (§5). In the frozen 1.0 bank, 467 tasks span 13 worlds: 241 solo, 226 chain: eleven full 20-subtask worlds plus two truncated at lengths 5 and 14 (§4); each of those two also ships one built-but-unverified solo beyond its truncation (11×20 + 5 + 14 = 239 gated segments +2 = 241 solos). Solo-versus-chain contrasts therefore hold the world, tooling, and difficulty labels fixed, while carried state and the terminal segment’s identity vary with k. We therefore read pooled trends, not individual lengths (§8).

4

Building a Verifiable Generated Bank

Each task is synthetic. No user traffic is mined or replayed. Synthesis adds difficulty control and a regenerable private split, while a hand-authored bank inherits an author’s judgment that a task is fair and solvable. A generated bank has to demonstrate that its tasks are fair and solvable. Forward growth through a shared world. Initialized once as a state S0 , a world is propagated through the chain (Figure 5a, Appendix B). Subtask i is proposed against Si−1 , the agent workspace plus the mock’s accumulated side effects, taken from a run that passed. On acceptance it emits Si . A random seed fixes the axis, difficulty, and coupling sequences, so that a world regenerates deterministically. A coupling ledger forces cross-subtask dependencies (an early subtask mints a server-generated identifier a later one can consume), which stops a long chain degenerating into k independent short tasks. Gate 1: certify the grader is load-bearing prior to trusting it. A validator that passes regardless of its input provides no signal. Prior to a subtask reaching any model, a zero-LLM triad runs against its validator: (1) a blank exam (the agent does nothing) needs to FAIL; (2) the answer key (the reference solution) needs to PASS; and (3) a sabotaged exam (each check’s evidence, corrupted in place) needs to FAIL. The first two conditions are necessary but weak. Many uninformative graders satisfy them. The third is what certifies each assertion load-bearing. Gate 2: an oracle is required to solve it, in the scoring harness. Each subtask is then attempted ten times in parallel by an oracle, in the same harness that later scores the leaderboard. Since the generator-tier model passed at a rate that left no headroom, we selected the oracle from a lower capability tier. A subtask is admitted only at pass@10 ≥ 3. Otherwise, a reviser model that shares the generator’s family (a fact §9 treats as part of the entanglement) repairs it in place using the failed oracle transcripts as evidence, then the subtask is torn down and re-proposed. Shipped defaults allow three repair rounds and a single re-proposal. If it fails to clear, the chain is truncated at the last solid 4

subtask instead of being padded with an uncertified subtask or made easier. Two of thirteen worlds terminate this way, at five and fourteen subtasks. We ship them short, and hence the headline slice of §8 is the eleven full-length worlds. Scope notes: the gate is applied to each subtask (assembled chain-1tok tasks are not oracle-solved end to end, a gap §7 shows to matter). The oracle’s generous message budget means it certifies solvability without ensuring difficulty parity. Gate 3: two-sided golden replay. Once assembled, a chain is replayed deterministically. The forward replay of its reference solution is required to PASS. A negative replay, applied wherever a later subtask depends on an earlier one’s result, is required to FAIL: the terminal segment’s reference alone, run from the initial world state S0 with segments 1, . . . , k−1 skipped. Static checks run alongside. A seed-workspace value equal to an answer-card value (the leak predicate) blocks publication outright. Of the 467 shipped tasks, 465 are replay-verified. The two exceptions are documented in the release manifest. Gates need to be discoverable. The cheapest way to make a generated task look hard is to hide an undocumented requirement, a mandatory header no error message ever reveals. We forbid it on incentive grounds. Against a hidden trap, a reckless model guesses and sometimes gets lucky, while a calibrated model correctly concludes it is blocked, calls report_blocked, and is marked wrong. The trap inverts the thing being measured. Each gate on the success path needs to be learnable from the served spec or a truthful error hint. Harder tasks demand more inference, not more guesswork. Under this rule and Gate 2, stacking obstacles failed to manufacture difficulty (discoverable ones are satisfied by inference. Undiscoverable ones drove the oracle pass rate to zero). This implies that a low pass rate alone fails to distinguish “hard” from “flagged” (§7). What restores headroom is length (§8).

5

Grading: Certify the Labor, Then the Delivery

A substring check fails in both directions, rejecting 48,210 for 48210 and accepting a lucky coincidence, so APIFlow-Bench does not grade text. A run passes only if two independent surfaces hold. G1 (state): the required calls are executed. Nearly all tasks have a canary value that is distinctive, hard-coded in the mock backend, reachable only along the correct path, and different on each decoy: the deprecated /v1 route serves other numbers, the summary endpoint a rounded one. On the stn-12 ticket the canary is the live reading, served only on the /v3 route under the refreshed token. G1 requires that this value appear in the agent’s final workspace and that a data-flow check tie it to the response it needed to come from. Possessing it is evidence of having made the real call. (28 of the 241 solo tasks ship with an empty state signature (the task-specific list of state requirements the state check verifies) and are gated by the answer card and structural checks alone. Chains inherit their segments’ checks.) G2 (answer): the final answer matches the reference. The agent declares its final answer as an answer card, compared field by field with a typed comparator (numeric, currency, string, set). Format is normalized, leaving a correct answer intact under punctuation differences. Structure remains strict, failing a run that buries the number in prose or drops the reference identifier, even when the state check passes. Checkpoints (such as whether the agent applied backoff after an HTTP 429 response) are diagnostics that do not determine a pass. A parallel LLM verifier reviews units, decimals, and provenance. The verifier can only overturn a pass to a fail and is advisory in this release: the published board is the deterministic grader’s alone. Provenance gating enables release of answer keys and transcripts. Since a pass is gated on evidence an agent can only earn through the real call path, a leaked expected answer alone fails to let a model pass. Expected answers, segment-level reference solutions (239 of 241 solo tasks, chains compose their segments’ references), and all 44,362 transcripts ship unredacted. Gating removes the guessing exploit within a run. It fails to prevent training on released solutions, so the release shares one contamination horizon (§9). Provenance gating delivers results and grading that third parties can check (the generation gates’ internal logs remain unreleased). 5

Adversarial audit of the grader. An independent adversarial reviewer attempted to exploit the completed grader by constructing a correct solution it wrongly rejects and a wrong answer it wrongly accepts, each confirmed against the live mock. Six attacks succeeded (Table 4, Appendix G), each involving the grader scoring something other than the work the model earned. Each fix is re-verified by running the exploit (now required to fail) and the correct solution (required to pass). The audit was one reviewer’s single pass, and its attempt coverage is uninstrumented. The six exploits touch four of the seven axes. No exploit was found on the remaining three axes. This does not establish that they are exploit-free. All six exploits were implementation errors instead of flaws in the grading design, indicating that design-level gates alone fail to validate an implementation.

6

Experimental Setup

We evaluate 19 frontier and open-weight models on the frozen 1.0 bank: 467 tasks × 5 epochs for each model, temperature 1.0, under Inspect AI’s stock react() loop [UK AI Security Institute, 2024] with no custom planner, summarizer, or retry logic. The panel spans providers, price tiers, and openversus closed-weight models. The panel is a slate, not a census: API access and serving stability are the binding constraints on membership, and no model is dropped for its score. (One further frontier model is run and left unranked, because an API-layer safety filter declined 87.8% of its runs. The model passed 90.9% where allowed.) Of the 44,365-cell grid, 44,362 trials ran and 44,343 completed without harness error. We report results based on the completed trials. Since five epochs of a given chain are five draws from the same world, all confidence intervals are 90% cluster bootstraps over worlds (2,000 resamples), keeping each world’s epochs together. Treating the headline slice’s 55 runs of each model as independent would understate the interval by roughly half. At a true n of eleven worlds the intervals are wide and all overlap (Figure 3a). This implies that the defensible statements are about pooled levels and the well-powered full-bank statistics of §8.4.

7

Screening the Bank for Cells No Model Passes

We screen the bank with the panel of §6 prior to reading model results. A low pass rate alone fails to separate hard from flagged (§4). On a wide panel, that has a constructive converse: if each model fails a world×length cell across all attempts, the cell is consistent both with genuine difficulty and with an instruction that omits a requirement the grader checks. The 19 models span roughly seven independent training pipelines, and a cell’s 95 trials are 19 clustered draws (§6). With models as units, zero passes is consistent with a model-level probability of passing the cell at least once in five epochs of up to 14.6% (one-sided 95% exact binomial, 11.4% at the paper’s usual 90%). Treating the roughly seven independent pipelines as the unit widens that bound to 34.8% at 95%. Hence, the screen nominates and does not prove. Positives require root-cause analysis. Across the 226 chain cells, 18 have zero passes across all 19 models (1,710 of 21,455 chain trials, 8.0%), falling in two families (v56-w08 at lengths 13–18 and 20, v56-w10 at 10 and 12–20) plus a single cell of a third (v56-w04 at 11). For each of these cells the assembled grader checks mid-chain steps that the compressed one-goal instruction does not state, yet the assembled chain’s forward golden replay passes: a reference solution exists and clears the grader, so the tasks are solvable. Whether the zero pass rate reflects this gap between the compressed instruction and the assembled grader or the capability ceiling of the current models, we do not adjudicate here (see §9). No gate could catch it, since Gate 2 runs at the subtask level, prior to assembly, and golden replay composes segment-level references, each of which passes. The screen, running after assembly, catches the all-zero cells the unit-level pipeline structurally misses. Three caveats bound the method. First, 17 of the 18 flagged cells sit in the two families where signatures gate mid-chain segments. The screen fired where enforcement was strictest, implying that a correctly built mid-chain-gating chain may also score low. Only w04@11 is flagged in isolation. Second, the threshold has known escapes: w16@19 (documented same pattern) and w08@19 (inside that family’s length span but unflagged) each pass 1 of 95. Hence, we read those cells as near-zero-pass. w08@19 and nine further 1–3-pass cells remain unadjudicated. Several show the dropped-requirement signature at solo-implied rates far above their chain rate, and one solo task is a zero-pass cell, which makes the solo-clean assumption approximate. Third, since 6

pass rate

100%

abstained correct end state, wrong answer card

closed-weight open-weight

state incomplete, answer card also wrong failed the state check

all 11 worlds: 409 failed runs

80%

169

222

7

29 pp spread

60%

11

9 clean worlds (defective families excluded): 219 failed runs

169 7 0

40% 5

solo

10

15

20

32 11

50

100

150

200

250

300

350

400

failed runs on the 20-subtask slice

dependent subtasks in the chain

(a) One line per model, five task sets. (b) Failures on the 20-subtask slice. Figure 2: (a) Chain length collapses the level: 93% pooled on solo, 74% on clean 20-subtask chains (61% with the flagged cells). Read the right-edge fan-out with care: chain-20 rests on eleven worlds and every interval overlaps every other (Figure 3a); Appendix E tabulates every point. (b) Failures with the screened families of §7 included (top) and excluded (bottom): on clean worlds the dominant category is final-delivery failure after otherwise correct state execution.

we performed root-cause analysis only on the screen’s positives, we make no specificity claim. A controlled-injection study measuring precision and recall is the natural next step. We leave both consequences unrepaired, since the frozen bank needs to stay reproducible. Cells no model passes compress aggregate pass rates but do not reorder models. All models fail the flagged cells. Pooled chain pass reads 77.0% with the flagged cells, 83.7% without. The headline slice moves from 43.6–72.7% to 53.3–88.9% (nine worlds), gpt-5.5 leading, deepseek-v4-flash tied second. Axis-level slices should not be read raw. Two of the three AUTHENTICATION worlds at length 20 are flagged (0.0%). The survivor passes 17.9%. Read raw, the slice suggests a 6% pass rate for authentication at length 20, although most of that figure reflects the flagged cells instead of task difficulty.

8

Results

8.1

Chain Length Collapses the Level and Takes the Bank off the Ceiling

Chain length is the only parameter that increases difficulty without reducing the discoverability of task information (§4). Pooled over the panel, pass rates fall from 92.9% on solo tasks to 74.4% on the nine clean 20-subtask chains, 60.9% with the screened cells included, at a fixed world set, tool surface, and difficulty labeling (Figure 2a). On solo tasks the bank is at the ceiling: 18 of 19 models land between 88% and 97%, only gpt-5.4-mini below at 77%. The spread of point estimates grows from 19.8 points (solo) to 29.1 (chain-20, 8.3 to 21.8 excluding the one low outlier). However, chain-20 rests on eleven worlds against solo’s 241 tasks, and on a log-odds scale, which undoes the ceiling’s compression, the spread does not widen at all (2.1 solo, 1.2 chain-20). At this bank size, model separation is unresolvable here: all cluster-bootstrap intervals overlap each other (§6). The level collapse is the robust chain-length effect. Models separate with headroom in reliability (§8.4) and failure anatomy (§8.3, §8.5). Each accounting leaves closed- and open-weight models interleaved at all lengths and the descent non-monotone in k (Figure 7, Appendix E). 8.2

What the Headline Slice Supports

Figure 3a gives the headline slice, scoped to the 19-model launch panel of the July 22, 2026 release (the public board has since grown). In this slice, we observed: (1) the point estimates order gpt-5.5 first at 72.7%, deepseek-v4-flash (a cheap open-weight model) tied with claude-opus-4-8 for second, and glm-5p2 fourth with qwen3p7-plus; and (2) all intervals overlap. The launch blog mislabels qwen3p7-plus as open-weight. Qwen’s “-plus” tier is API-only, and both qwen models are classified 7

closed-weight

open-weight ● gpt-5.5 claude-sonnet-4-6 kimi-k2p6 ● claude-sonnet-5 claude-opus-4-8 kimi-k2p7-code ● deepseek-v4-pro ● deepseek-v4-flash ● glm-5p2 ● qwen3p6-plus gpt-5.6-terra kimi-k2p5 ● claude-haiku-4-5 minimax-m2p7 ● qwen3p7-plus gpt-oss-120b ● gpt-oss-20b ● gpt-5-mini gpt-5.4-mini

gpt-5.5 deepseek-v4-flash ● claude-opus-4-8 qwen3p7-plus glm-5p2 ● qwen3p6-plus gpt-5.6-terra deepseek-v4-pro ● claude-haiku-4-5 kimi-k2p7-code ● claude-sonnet-4-6 claude-sonnet-5 minimax-m2p7 ● kimi-k2p6 ● kimi-k2p5 ● gpt-oss-20b ● gpt-oss-120b ● gpt-5-mini gpt-5.4-mini

40%

60%

pass5 (all five) pass1 (mean) pass@5 (any)

40%

80%

60%

80%

100%

share of 467 tasks (5 epochs)

pass rate, 20-subtask chains (90% CI)

(a) 20-subtask slice, 90% cluster-bootstrap CIs. (b) Reliability over five epochs, all 467 tasks. Figure 3: (a) The headline slice: 1,045 ranked runs; every interval overlaps every other, so the figure supports an ordering of point estimates but does not establish separation between models. (b) Hollow dot pass@5, small dot pass1 , filled dot pass5 (cells with fewer than five completed epochs excluded): best-of-five compresses the panel into seven points; all-five-of-five spreads it across 44, a consistency ranking.

closed here. Price sharpens the interleaving (Figure 6): below $0.01 a trial the frontier is entirely open-weight, and gpt-5.5’s +3.6-point edge costs ∼22× more (pinned list prices). 8.3

Long-Horizon Failures Concentrate at Final Delivery After Correct State Execution

Of the 20-subtask slice’s 409 failing runs, 190 sit on the two screened families. Roughly half of a full-slice failure decomposition would therefore reflect these screened cells, not model behavior. Figure 2b decomposes both. The analysis below uses the clean subset (n=219 of 855). Abstention is nearly absent. Only seven of 219 failures (3.2%) invoked clarify or report_blocked, defensible on clean tasks, where each gate is discoverable by construction (§4). The other 212 submitted an answer and failed at least one check: 180 failed the typed answer card, and 169 of those (77% of all clean-slice failures) failed only the answer card, with each state check green. In these runs the agent drove the world to the correct final state, then delivered something the typed comparator rejected: a wrong value, missing field, or undeclared card (the released transcripts support the finer split). The category is world-concentrated (Figure 8). Two worlds hold 167 of the 169: 90 in w05, where the cell passes 4.2%, and 77 in w01, 17.9%. Hence, the lost delivery is a property of specific world–task shapes, not a uniform tax. These runs also persist to the end. The median failing run on this slice makes 15 tool calls (16 among the answer-card-only failures, maximum 168): they perform the work and lose the thread at the delivery. For calibration, the flagged cells add one further observation. Across this slice’s 190 failing runs on the flagged tasks, none abstained. Eight of all 1,710 flagged-cell trials bank-wide abstained, six of them from a single small model. Facing a flagged task, models nearly always submit a wrong answer and do not abstain. 8.4

The Leaderboard Ranking Primarily Reflects Consistency

Five epochs for each task separate two things a single pass rate conflates: whether a model can do a task, and whether it does it each time. Figure 3b reports, for each model over all 467 tasks, pass@5 (any epoch passed), pass1 (the mean), and pass5 (all five passed), the reliability statistic of Yao et al. [2025]. We exclude tasks with fewer than five completed epochs (466–467 for each model, 449 for qwen3p7-plus). In Figure 3b, we observed: (1) best-of-five compresses the whole panel into seven points (85.4– 92.3%), i.e., after five attempts nearly all tasks are within reach of even the weakest model; and (2) all-five-of-five spreads the same panel across 44 points (42.2–86.1%), double the 20.5-point pass1 spread. A binomial null (19 models at the panel-mean pass5 over 467 tasks, max-minus-min across 2,000 draws) puts the noise floor near seven points. World-level clustering raises it, and the observed pass5 spread dwarfs either. The leaderboard is therefore mostly a consistency ranking. Consistency is where small models pay: gpt-5.4-mini drops 26.3 points from pass1 to pass5 and gpt-5-mini 14.6, against 1.8–3.8 for the frontier tier. The instructive middle case is deepseek-v4-flash: the highest pass@5 on the panel (92.3%) with a mid-pack consistency drop, a frontier-class ceiling where the 8

remaining gap is reliability. A best-of-n wrapper masks nearly the whole capability gap at n=5, while an agent wired to side-effecting APIs does not get free retries. Hence, pass5 is the operationally relevant statistic, and a single-epoch benchmark misses it. 8.5

Long-Horizon Failure Is Not the Product of Independent Steps

Since solo and chain tasks freeze from the same accepted segments, each chain has a natural null model. If segment i failed independently at its solo pass rate pi (estimated for each model and world, so between-model and between-world heterogeneity does not manufacture the gap), chain-k pass Q would be i≤k pi . For gpt-5.5, a 94.9% mean solo rate predicts 0.94920 ≈35%. We observe 72.7%. Figure 4 (Appendix A) plots that null against observed chain pass rates on the nine full-length worlds outside the two screened families (flagged cells w04@11 and w16@19 remain as dips). At k=2 the pooled curves agree (84.3% vs. 84.4%, though large world-level residuals cancel underneath). By k=20 they have diverged by 33 percentage points (74.4% observed vs. 41.4% predicted). Two mechanisms plausibly produce the gap: the first is a grading artifact, the second genuine competence. First, chain grading is selective: a chain-1tok task is graded on the final subtask’s answer and state plus dependency-critical earlier checks (§3). A chain run need not re-earn each intermediate diagnostic. Second, within-session competence is strongly correlated: a model that handles a world’s early segments tends to handle its later ones, so that segment-level failures rarely multiply. They concentrate on segments that interact badly with accumulated state, and on the final delivery (§8.3). Hence, the independent-step account of “compounding error” is inconsistent with the data for these agents under chain grading. Length does not tax a chain multiplicatively. Length samples more of the failure surface in a given task, raising the chance of meeting the segment or delivery that breaks the run. The difficulty comes from exposure instead of hidden information (§8.1). Each gate stays discoverable.

9

Conclusion, Limitations, & Future Work

Key findings. Three key findings: the level collapse (93% to 74% clean); reliability separation (7 vs. 44 points); long-horizon failure that is correlated rather than per-step (partly a selective-grading effect). A model-consensus screen flags the cells no model passes, and every aggregate is reported with and without them. Artifacts are public (Apache-2.0, Appendix H). Small n and entanglement. The headline slice is 1,045 runs but eleven sampling units, so Figure 3a’s intervals are wide, all overlap, and the top reads as a tie. The second is entanglement. One frontier family generated all worlds, a sibling served as solvability oracle, a same-family reviser repaired gate failures (§4), and that family’s models are ranked. Mitigations: deterministic validators and golden replay decide pass/fail; the LLM verifier is non-ranking; the gate filters for solvable, not oracle-easy. A generator-family advantage remains possible: we ran no cross-family re-gating control and retained no gate yield statistics. The headline leader comes from another family. Scope. Deterministic mocks trade realism for replayability and difficulty control. A high score means surviving modeled failure modes under the seven-tool contract, not running arbitrary production systems. The 1.0 bank is REST only (GraphQL and MCP adapters ship without task banks); the panel is the 19-model slate of §6. Contamination. All scores predate the public release, so are uncontaminated by it. Later runs can be contaminated (each task carries a GUID canary [Srivastava et al., 2023] and a private held-out split is the intended anchor). Future work: more worlds and longer chains, a cross-family re-gating control, and, for the cells no model passes (§7), methods that preserve requirement discoverability under compression or grading that gates only what the served instruction states.

9

References Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), Vienna, Austria, May 2024. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. API-Bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3102–3116, Singapore, December 2023. doi: 10.18653/v1/2023.emnlp-main.187. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=zAdUB0aCTQ. Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities, 2024. Preprint, not peer-reviewed. Mike A. Merrill et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. Preprint, not peer-reviewed. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. In Advances in Neural Information Processing Systems (NeurIPS), pages 126544–126565, Vancouver, BC, Canada, December 2024. doi: 10.52202/079017-4020. Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), pages 48371–48392, Vancouver, Canada, July 2025. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In International Conference on Learning Representations (ICLR), Vienna, Austria, May 2024. Aarohi Srivastava et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research (TMLR), 2023. Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 16022–16076, Bangkok, Thailand, August 2024. doi: 10.18653/v1/2024.acl-long.850. Best Resource Paper Award. UK AI Security Institute. Inspect AI: Framework for large language model evaluations, 2024. URL https://github.com/UKGovernmentBEIS/inspect_ai. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, pages 52040–52094, Vancouver, BC, Canada, December 2024. doi: 10.52202/079017-1650. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations (ICLR), Singapore, April 2025. arXiv:2406.12045. 10

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, pages 46595–46623, New Orleans, LA, USA, December 2023. doi: 10.52202/075280-2020. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations (ICLR), Vienna, Austria, May 2024.

A

The Independent-Step Null Model

pass rate, pooled panel

100%

80%

60%

+33 pp observed

40%

Figure 4: If subtasks failedQindependently at their solo rates, chain-k pass would be i pi (dashed); the observed rate (solid; nine full-length worlds outside the screened families) sits far above it, and the gap grows with k.

independence prediction ∏pi i

2

5

10

15

20

chain length k

B

Overview of the Generation Pipeline (a) One world, grown forward coupling ledger: subtask 1 plants an id, subtask 3 must consume it

S0

subtask 1

S1

pagination

subtask 2 schema

S2

subtask 3 auth

···

subtask 20 statefulness

S20

Si = the agent workspace plus the mock’s accumulated side effects, carried forward from a run that passed.

(b) Inside one cell: two gates clear before any model is scored redesign the validator once write the mock overrides and the validator

propose a subtask seeded from Si−1

self-test triad blank ✗ key ✓ sabotage ✗

solvability gate the oracle must reach pass@10 ≥ 3

accept save Si , settle any coupling

repair in place (≤ 3), re-propose (≤ 1), else truncate the chain here

(c) Freezing an accepted chain into the shipped bank cut the accepted chain at every k: subtasks 1, . . . , k freeze into one chain-1tok task each segment also freezes on its own into a solo task

golden replay forward replay must PASS; sabotaged replay must FAIL

frozen release 467 tasks over 13 worlds, sha256-pinned in registry.json

Figure 5: The generation pipeline. (a) A world is grown forward: each subtask is seeded from the verified end state of the last, and a coupling ledger forces cross-subtask dependencies that block a prefix-only shortcut. (b) Two gates are cleared before models are scored (i.e., a zero-LLM self-test triad that certifies the grader is load-bearing, and an oracle solvability gate run in the scoring harness itself). (c) The accepted chain is cut at every k, replayed forward and sabotaged, and pinned by content hash.

11

C

Prior-Benchmark Coverage Matrix

Table 1: Where prior benchmarks land on the API-engineering taxonomy. ✓ scored as a first-class axis, • exercised but not separately graded, ✗ not covered. Marks are our reading of each benchmark’s published task taxonomy, not a quality judgment. Each of these benchmarks covers part of the picture well, and AppWorld in particular grades pagination and end state. However, the decomposition itself is rare, and two of the failure modes practitioners encounter most often (i.e., auth refresh and schema repair) are the least targeted. stateful end state / auth pagination schema chain side effect refresh to end repair

Benchmark

D

AppWorld [Trivedi et al., 2024] τ -bench [Yao et al., 2025] BFCL v3 [Patil et al., 2025] SWE-bench [Jimenez et al., 2024]

✓ ✓ ✓ ✓

✓ ✓ • ✓

✗ ✗ ✗ ✗

✓ ✗ ✗ ✗

• ✗ ✗ ✗

APIFlow-Bench (this work)

✓

✓

✓

✓

✓

The Seven Capability Axes

Table 2: The seven capability axes, each generated at three creation-time difficulty levels and composed forward into chains. Pass rates pool the 19 models and all task lengths on the frozen 1.0 bank, where n is trials. The third column excludes the 18 flagged cells of §7. Chain tasks carry a single generation-time axis annotation (the chain’s dominant capability, labeled at assembly). Hence, each trial contributes to exactly one row.

E

Axis

What it tests

Pass Pass (excl. flagged)

MULTISTEP SCHEMA STATEFULNESS PAGINATION ERROR RECOVERY AUTHENTICATION DISCOVERY

carry state across calls repair a rejected body confirm the side effect walk to the last page back off and retry refresh a token mid-session pick among look-alikes

91.2 88.9 88.4 87.8 84.6 80.7 76.1

92.6 92.0 90.8 89.6 89.3 86.1 80.5

n 6,457 5,604 7,215 4,840 7,217 6,078 6,932

Per-Model Results on Every Task Set 75.0% 70.0% deepseek-v4-flash

claude-opus-4-8 qwen3p7-plus

gpt-5.5

pass rate

65.0% claude-haiku-4-5

60.0% 55.0% 50.0%

gpt-oss-20b

gpt-5-mini

45.0%

gpt-5.4-mini

40.0% 10−2

closed-weight open-weight Pareto frontier

10−1

mean cost per trial, 20-subtask chains (USD, log)

Figure 6: Cost against 20-subtask pass rate. The frontier below $0.01 per trial is entirely open-weight. Prices come from the release-pinned table of July 19, 2026: a list-price comparison, not a serving-efficiency claim.

Table 3 reports each model on all five task sets cut from the same worlds, together with the 90% cluster-bootstrap interval on the headline slice and the mean cost of each trial there. Cost spans two orders of magnitude at comparable accuracy, which is a second reason not to read the top of the 20-subtask column as a ranking. 12

Table 3: All 19 models, frozen 1.0 bank, five epochs, temperature 1.0. • marks open-weight models. “All” pools each task in the bank. “Solo” is the 241 single-subtask tasks. Chain-k is the cumulative k-subtask slice. The interval is a 90% cluster bootstrap over worlds on the chain-20 slice. The $/trial column is the mean cost on that same chain-20 slice. Model

All Solo Chain-5 Chain-10 Chain-15 Chain-20 (90% CI) $/trial

gpt-5.5 claude-opus-4-8 deepseek-v4-flash • glm-5p2 • qwen3p7-plus claude-haiku-4-5 deepseek-v4-pro • gpt-5.6-terra qwen3p6-plus claude-sonnet-4-6 kimi-k2p7-code • claude-sonnet-5 kimi-k2p6 • minimax-m2p7 • gpt-oss-20b • kimi-k2p5 • gpt-oss-120b • gpt-5-mini gpt-5.4-mini

87.8 88.6 88.4 87.1 86.2 86.0 86.7 85.4 87.3 89.0 87.6 87.8 87.9 85.5 80.9 85.9 82.0 80.6 68.5

94.9 96.0 95.6 93.9 94.0 92.9 94.6 92.4 95.1 95.7 95.4 96.5 95.2 93.0 88.2 94.4 90.2 89.6 76.7

92.3 92.3 98.5 92.3 90.8 96.9 92.3 90.8 86.2 86.2 95.4 84.6 100.0 92.3 86.2 92.3 86.2 83.1 67.7

75.0 68.3 75.0 75.0 73.3 75.0 75.0 71.7 75.0 75.0 73.3 73.3 75.0 75.0 73.3 68.3 61.7 58.3 45.0

72.7 72.7 70.9 74.5 68.5 58.2 65.5 65.5 69.1 69.1 72.7 72.7 70.9 61.8 61.8 61.8 69.1 56.4 47.3

72.7 [45.5, 90.9] 69.1 [45.5, 90.9] 69.1 [47.3, 89.1] 65.5 [40.0, 85.5] 65.5 [40.0, 85.5] 63.6 [36.4, 81.8] 63.6 [36.4, 81.8] 63.6 [36.4, 81.8] 63.6 [40.0, 85.5] 61.8 [36.4, 83.6] 61.8 [36.4, 81.8] 60.0 [36.4, 81.8] 58.2 [34.5, 81.8] 58.2 [34.5, 80.0] 56.4 [30.9, 81.8] 56.4 [30.9, 81.8] 52.7 [27.3, 78.2] 50.9 [29.1, 72.7] 43.6 [23.6, 63.6]

0.170 0.086 0.008 0.079 0.039 0.022 0.106 0.043 0.035 0.048 0.022 0.070 0.087 0.010 0.004 0.017 0.007 0.006 0.010

panel spread

20.5 19.8

32.3

30.0

27.3

29.1

—

5

10

15

20

100%

pooled pass rate

90% 80% 70% 60% 50% 40% 1

chain length k (subtasks)

Figure 7: Pass rate pooled over the 19-model panel against each chain length present in the bank. The descent is real but not monotone: worlds contribute unevenly across lengths, per-length samples are small, and the flagged cells of §7 drag specific lengths far down (per-cell zero; pooled minima near 50%). The trend is more informative than individual per-length points.

13

The 20-Subtask Slice by World 100

v56-w08 (authentication) v56-w10 (authentication) v56-w05 (error recovery) v56-w01 (authentication) v56-w16 (multi-step workflow) v56-w14 (pagination) v56-w04 (error recovery) v56-w11 (multi-step workflow) v56-w13 (multi-step workflow) v56-w03 (multi-step workflow) v56-w02 (statefulness)

80 60 40 20

pass rate over 5 epochs

F

cla g ud pt-5 .5 de e-o p ep se us-4 ek 8 -v 4fla sh g qw lm-5 p e cla n3p 2 7ud pl eus h a de ep iku4se 5 ek -v gp 4-pr t-5 o qw .6-te rra cla en3 p6 ud -p elu so s n ki m neti-k 46 2 cla p7c ud e- ode so nn etki 5 m mi-k in 2p im 6 ax -m 2 gp t-o p7 ss -2 0b ki m i gp -k2 p5 t-o ss -1 2 0 gp t-5 b gp -mi ni t-5 .4 -m in i

0

Figure 8: The 20-subtask slice, with a cell for each (world, model) across five epochs, rows sorted hardest-first and labeled with each world’s dominant axis. Variation across worlds (rows) exceeds variation across models (columns). The two rows at zero are cells no model passes for which a reference solution nonetheless clears the assembled grader (§7); whether they are hard or a compression artifact is left open. This implies that an all-zero row must be diagnosed before it is interpreted.

G

The Adversarial Grader Audit in Full

Table 4: The six exploits an independent red-team pass found in the finished grader, and their fixes. All numbers reported in this work are obtained after these fixes. Axis

Class

The exploit

The fix

SCHEMA

false neg.

accept the real success code ([200,201])

SCHEMA

false pos.

STATEFULNESS

false neg.

PAGINATION

false pos.

MULTISTEP

false pos.

MULTISTEP

false pos.

a creation POST returns 201 while the check pinned 200, rejecting the minimal correct solve the graded value was echoable (present in a payload the agent could copy without fetching it) a currency comparator demanded a USD unit the agent had no source for, rejecting correct numbers the total was also served on a summary endpoint, making the task passable without paging the final canary was reachable by guessing a single URL, skipping the chain a value was readable before the publish step that was supposed to produce it

H

remove the echo; pin a distinct canary the mock serves only on the real path grade numerically; stop forcing a hallucinated unit remove the leak; a terminal-page marker forces the full walk gate the value behind stateful mock logic and a non-guessable id gate the read on the publish; add chain-level state checks

Datasheet, Licensing, and Reproduction

Composition and collection. The bank is 467 tasks in 13 procedurally generated worlds (i.e., 241 solo subtasks and 226 cumulative chains). The bank involves no human subjects, no user traffic, and no personally identifiable information. Each world, mock backend, instruction, and expected answer is synthesized by the pipeline of §4. Task instructions are English. Artifact URLs. Harness and frozen bank: github.com/postmanlabs/APIFlow-Bench; transcripts: github.com/postmanlabs/apiflow-bench-transcripts; leaderboard: postman. com/ai/apiflow-leaderboard. Licensing and distribution. The harness, bank, scoring code, and leaderboard builder are released under Apache-2.0 at github.com/postmanlabs/APIFlow-Bench. All 44,362 raw trial transcripts are released at github.com/postmanlabs/apiflow-bench-transcripts, and model-level bundles are attached to the repository’s releases. The authors bear responsibility for the artifacts and confirm the license permits the stated use. 14

Reproduction without API keys. scripts/bank_sha256.py verifies the bank byte for byte against the content hash pinned in registry.json. scripts/golden_replay.py replays each world’s reference trajectory forward and in its sabotaged form. Both run against the local mocks with no network access and no credentials. Reproducing the leaderboard requires provider API keys. On the published panel the full 467 × 5 grid cost between roughly $8 and $730 in total for each model under the release-pinned price table. These totals differ in scale from the cost column of Table 3. This is because the cost column reports the mean cost of each trial on the chain-20 slice. Maintenance and versioning. Each release is pinned in registry.json by bank path, content hash, epoch count, and price-table version. Hence, running 1.0 in a later year reproduces 1.0 as the repository grows. The pinned hash covers the annotation-stripped bank that ships publicly. Golden replay re-verifies equivalence to the evaluated bank after the strip. The flagged cells are listed in the repository, not silently repaired in a frozen bank (§7). A refreshed bank with a private held-out split is planned to anchor rankings once the public bank has aged. Safe use. Task code (validator.py, mock_overrides.py) executes in the harness process. Hence, third-party task banks should be treated like other executed code.

15

Related documents

• APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows#793951
• APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows#789932
• Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines#1105012
• Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models#1130381
• Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models#1108791