Outcome Monitors: Recovery Affordances for Silent Tool Failures

Outcome Monitors: Recovery Affordances for Silent Tool Failures Sugam Panthi, Rabab Abdelfattah

arXiv:2608.19303v1 [cs.AI] 19 Aug 2026

The University of Southern Mississippi sugam.panthi@usm.edu, rabab.abdelfattah@usm.edu

Abstract When a tool call times out, the agent sees the failure and can route around it. A cached error page or negative price can instead arrive in the expected format and be consumed as fact. We introduce Outcome Monitors, which detect violations of outcome contracts mined from task-disjoint traces or derived from public schemas. On a violation, the monitor preserves the result and issues a nonbinding receipt naming the violated property and public recovery tools. In frozen, prespecified evaluations with injected failures, Outcome Monitors raise ToolMaze completion from 10.9% to 28.1% across four models in two provider families and replicate in a third. In τ -bench retail, completion improves by 14.0 and 12.0 points on two tiers. In separate ToolMaze controls, removing the recovery-tool list eliminates the measured gain and restoring it recovers the effect; diagnostic detail and timing produce no detectable differences. Gains concentrate where the fault blocks completion. On a suite transcribed from a published incident taxonomy, detection outside the mined vocabulary falls to 46%, though delivery continues and completion is unchanged. Recovery tools are the active receipt content in these controls; extending detection beyond the contract vocabulary remains open.

1

Introduction

Language agents frequently combine reasoning with external tool calls (Yao et al. 2023; Schick et al. 2023). This lets them perform more tasks, but it also makes them rely on information they cannot directly observe. Timeouts are easy to detect. Contradictory, stale, or physically impossible results might instead arrive in the expected format and be taken at face value: in one documented production incident, a cached HTTP error page consumed as content became a confident fabricated analysis, and 70% of that runtime’s silent failures were first caught by a human reading output (Wu 2026). ToolMaze demonstrates that agents struggle to recover from such implicit failures (Zhu et al. 2026). Silent failure creates two distinct reasoning problems. The system must first recognize that an observation is inconsistent and localize the violated property. Then, it must select a recovery action that advances the task. Traditional agent loops rely on the language model to infer both from the raw observation. At the opposite extreme, a runtime guard prevents actions or selects repairs, obscuring whether improved

behavior stems from better reasoning or the guard’s intervention. We introduce a deterministic detector that identifies when a tool result violates a mined or public invariant and signals the violation to the acting agent. We name these invariants outcome contracts, as they specify expected relationships between tool calls and their results, mined from nominal executions or derived from public schemas. On a violation the detector appends an advisory receipt; it never restricts an action, executes a repair, reveals a benchmark fault label, or consults the evaluator. Recent contract and gate systems attach checks before execution and enforce them (Liu et al. 2026; Reddy, Challaram, and Basu 2026; Bhardwaj 2026). Our monitor instead checks a returned result and does not enforce a response. ToolMaze contracts use task-disjoint cross-fitting, and injection changes only what the model observes; evaluators therefore score the agent’s chosen recovery rather than a repair encoded by the runtime. The resulting interface separates detection from recovery: the monitor identifies a violated outcome property and lists public recovery tools, while the language agent chooses what to do. We call these tools the signal’s recovery affordances. Three findings support the interface: 1. Detector-triggered receipts raise completion across three provider families. We find paired completion gains with Outcome Monitors on a frozen, task-disjoint ToolMaze confirmation. The effect holds for four models spanning DeepSeek and Qwen. A separately frozen MiniMax M3 replication extends it to a third provider family. 2. Benefits replicate in τ -bench and concentrate where failures block completion. We evaluate 50 τ -bench retail tasks, each subject to both fault types. Categorical violations improve sharply with no paired losses, while conservation violations show no measured effect. Across all studies, the gains concentrate where the fault blocks baseline completion. Delivery is not universally beneficial: held-out AppWorld shows no measured net effect, and clean ToolMaze controls contain both rescues and harms. 3. Recovery tools—not additional diagnostic detail— drive the detectable ToolMaze gain. Neither the baseline’s standing fault-aware prompt nor an extended rea-

RAW INTERFACE lookup_inventory

available = -3

detect · localize · recover silent failure

left to the agent

value consumed as fact

DETECT AND ADVISE lookup_inventory

raw result kept

contract violated mined or public invariant ADVISORY RECEIPT witness expected ≥ 0, observed -3 recovery query_warehouse active content get_stock_by_sku (Section 5) verify · retry · switch

action set intact

Figure 1: A raw interface leaves detection, localization, and recovery with the agent. Detect and advise instead flags a contract violation and appends an advisory receipt carrying a diagnostic witness and a list of available recovery tools. It neither removes actions nor selects a repair. Controls (Section 5) identify the recovery-tool list, not the property-specific witness, as the active content in our tested sample. soning budget removes the effect. A generic caution matches the localized witness, and changing placement or salience shows no detectable advantage. A warning stripped of recovery tools performs at baseline, while restoring the list restores the gain. A schema-only detector recovers much of the fault-enriched gain but fires more often; the learned contracts improve detection selectivity. We measure the remaining boundary with an incidentderived study. Outside the mined vocabulary, detection degrades but delivery does not.

2

Related Work

Tool-using language agents. ReAct interleaves naturallanguage reasoning with environment actions, while Toolformer trains models to decide when and how to invoke APIs (Yao et al. 2023; Schick et al. 2023). AgentBench and AppWorld broaden evaluation to interactive multi-step environments (Liu et al. 2024; Trivedi et al. 2024), subsequent work scales tool repertoires (Qin et al. 2024), and SWEagent shows agent-computer interface design affects task success (Yang et al. 2024); tool-hallucination work targets the agent’s own faulty call (Zhang et al. 2024; Xu et al. 2025).

These lines primarily evaluate planning and tool selection under an assumed interface. We instead modify what the interface communicates after a semantically suspect result. Feedback and self-correction. Reflexion stores linguistic feedback across trials (Shinn et al. 2023), CRITIC uses external tools to critique and revise outputs (Gou et al. 2024), and Self-Refine iterates with the same model as generator and critic (Madaan et al. 2023); self-correction without external grounding can degrade outputs (Huang et al. 2024). Reinforced Agent places a separate LLM reviewer before execution and iteratively revises provisional tool calls (Ta, Zhu, and Shayandeh 2026); we compare against a mechanismfaithful port in our experiments. Process supervision trains a learned verifier to score each reasoning step (Lightman et al. 2024); our step-level receipts are instead deterministic, interface-derived checks requiring no trained verifier and no extra model call. PALADIN instead trains on failure-recovery trajectories and retrieves recovery exemplars at inference time (Vuddanti et al. 2025); ours is training-free and targets implicit inconsistencies in returned results. AgentPropBench reports little rank association between rejection and recovery across nine models (Gurram 2026), supporting the distinction between detecting a bad interaction and recovering from it. Runtime monitoring and verification. ToolEmu uses an LM-emulated sandbox to surface consequential agent risks (Ruan et al. 2024). ToolMaze directly perturbs tool environments to study dynamic replanning (Zhu et al. 2026). τ -bench evaluates agents in stateful retail and airline domains (Yao et al. 2024). Runtime shields enforce safety constraints during execution (Alshiekh et al. 2018). The closest recent systems make the enforcement contrast concrete: ToolGate admits a tool call only when Hoare-style pre/postconditions verify (Liu et al. 2026); Reason Less, Verify More rejects policy-violating writes before execution (Reddy, Challaram, and Basu 2026); Agent Behavioral Contracts enforce session-level governance rules (Bhardwaj 2026); and further systems supply rule languages, transactional compensation, or state-machine postconditions with recovery owned by the runtime (Wang, Poskitt, and Sun 2026; Chang and Geng 2025; Guo et al. 2026). ToolSafe trains a guardrail model to flag unsafe invocations before execution (Mou et al. 2026). Our monitor acts after a result arrives, makes no enforcement decision, requires no trained verifier, and leaves recovery with the acting agent. Post-trajectory false-success detectors identify unsupported completion claims (Advani 2026); our checker instead emits a per-call signal while the same trajectory can still recover. Design by contract and runtime assertions. Checking a computed result against a declared expectation is a longstanding software-engineering discipline: design by contract specifies pre- and postconditions a component must honor (Meyer 1992), higher-order contract systems attribute blame on violation (Findler and Felleisen 2002), and runtime verification monitors execution against formal properties (Leucker and Schallhart 2009). We port the interfacechecking idea to language-agent observations and explicitly

choose an advisory propagation policy, so the model, not the runtime, decides how to recover. Learned invariants and specification mining. Because hand-written contracts are costly, a parallel line of work infers likely specifications from observed executions. Daikon detects likely invariants from dynamic traces (Ernst et al. 2007), and specification-mining methods generalize this to temporal and API-usage properties (Ammons, Bodík, and Larus 2002). We adopt the same non-oracle stance: our contracts are mined from clean agent traces or derived from public response schemas rather than authored against a task solution. Unlike program-analysis settings, the mined invariant is not used to certify code but to explain a suspect observation to a downstream agent at inference time. Contract2Tool likewise infers preconditions and effects from documentation, schemas, and traces, but deploys them for causal tool filtering (Babu and Iyer 2026); we cross-fit invariants over returned outcomes and expose violations as advisory observations. Agent safety and guardrails. Recent work proposes guardrail frameworks that filter or constrain agent actions before or after execution (Dong et al. 2024; Inan et al. 2023). A related threat is adversarial payloads carried in tool results themselves (Zhan et al. 2024); our threat model is instead benign-but-inconsistent results; adversarial tools are out of scope. Although nonbinding, false-positive context can still induce the agent to take a harmful action. Our experiments measure the advisory alternatives— generic caution, always-verify, and a reviewer-loop port— under matched scaffolds; hard enforcement remains a design contrast (Table 1).

3

Outcome Monitors

The system has two roles: a detector that identifies when a tool result violates a learned or public invariant, and an advisory signal that communicates the violation to the acting agent. The mined invariants (outcome contracts) define the detector; the receipt encodes the signal. These components serve different roles: contracts determine when a result is suspect, while public recovery relations determine which alternative tools the receipt exposes. The controls isolate the receipt’s active content; they do not make contract-based detection dispensable.

Interface At step t, an agent selects tool call at from action set At and receives result yt . A contract checker C maps (at , yt ) to an ordered set Vt of violation records, each a machine-readable property code optionally with a path, expected/observed values, and human-readable detail. If Vt is empty, the interface returns yt unchanged. Otherwise a versioned encoder Es maps (τ, Vt , Rt ) to receipt ρt , where Rt contains only recovery affordances available from the public tool interface. At is unchanged. The agent may verify, retry, switch tools, ignore the receipt, or take any other advertised action (Figure 1). Algorithm 1 formalizes the per-call runtime: deterministic given the call, result, and learned contract, and it never in-

Algorithm 1 CheckAndReceipt: per-call contract evaluation. Require: tool τ , arguments a, result y, registry C, visible tools T , encoder Es Ensure: augmented result; action set A unchanged 1: V ← C.Check(τ, a, y) 2: if V = ∅ then 3: return y {consistent} 4: end if 5: R ← {τ } ∪ {s ∈ Subs(τ ) : s ∈ T } 6: ρ ← Es (τ, V, R) {versioned serialization} 7: return {tool_result : y, outcome_contract : ρ}

Algorithm 2 LearnContracts: invariant mining with acceptance rules. Require: nominal traces D, K disjoint task folds D1 , . . . , DK Ensure: per-fold contract registry C1 , . . . , CK 1: for k = 1 to K do 2: Dtrain ← D \ Dk 3: for each tool Tτ observed in Dtrain do S 4: Freq ← result field sets; Fall ← result field sets 5: types[f ] ← union of kinds observed for f 6: echo: accept f iff f is in ≥ 2 call/result pairs and a[f ] = y[f ] in all 7: positivity: accept f ∈ Freq iff numeric in ≥ 3 samples and always > 0 8: domain: accept f ∈ Freq iff name matches categorical lexicon, ≥ 4 string samples, ≤ 8 distinct normalized values 2 9: affine: accept (f, g) ∈ Freq iff ≥ 4 samples with ≥ 3 distinct values fit one y[g] = α y[f ] + β (10−8 tolerance) 10: order: accept (f, g) among three fixed date pairs iff ≥ 2 parsed samples and f ≤ g in all 11: end for 12: Ck ← registry over all accepted tool specs 13: end for

vokes tools, scores candidate actions, or waits for task-level failure. The frozen confirmation uses encoder Ev1 ; later controls use Ev2 with an explicit witness structure. Algorithm 1 shows the ToolMaze envelope; in AppWorld and τ -bench, whose tool results are plain text, the receipt is appended as text (exact envelopes in the supplement).

Detector Construction The abstraction permits different non-oracle sources. In ToolMaze, five-fold cross-fitting keeps every evaluated workflow out of the traces used to build its registry. We mine those registries from nominal, unperturbed (P0) trajectories using Algorithm 2. Acceptance is conservative. An invariant must hold on every training sample and meet its class-specific support minimum (the categorical lexicon appears in the supplement). At runtime, the checker evaluates each admitted class and collects its violations in Vt ; affine relations use a 10−5 tolerance versus the tighter mining-time fit. Recovery relations are constructed separately from detection. We build the static mapping Subs(τ ) programmatically

Strategy

Trigger

Grounding

Localized witness

Restricts actions

Extra tool reads

Generic caution Self-critique Always verify Hard guard

learned detector event model’s own judgment every advertised read monitor-detected event

external invariant internal none (fixed policy) external rule

no uncertain only after comparison yes

no no no yes

no optional yes no

Outcome Monitor

learned detector event

external invariant

yes

no

no

Table 1: Conceptual comparison of recovery interfaces. Shaded cells mark where a strategy differs from the Outcome Monitor row, which serves as the reference; the shading locates design differences and does not rank them. The table characterizes mechanisms, not measured superiority.

from two public, task-agnostic ToolMaze sources: per-tool substitute fields and the entry tool of each sibling path in the alternatives file. The mapping therefore uses no task descriptions, solutions, gold sequences, perturbations, or outcomes. The indexing code and full mapping ship with the artifact. At runtime, we intersect Subs(τ ) with the workflow’s advertised tools to form Rt , so a receipt never names a tool the agent cannot call. A task-disjoint audit of the detector on the experiments’ difficult C3/C4 workflow classes flags 79.6% of implicit faults at a 1.39% clean false-positive rate, with a learned contract available for 94.2% of fault outcomes; rates are per observed tool outcome, not per episode. An earlier schema-only gate reached 55% recall at an 18% falsepositive rate. Section 5 measures both detectors end-to-end, so the audit characterizes detection precision rather than a completion ceiling (full audit in the supplement). For AppWorld, contracts derive from released standard API response schemas: eligible read results are validated recursively, and a receipt fires on a missing required field or a scalar type mismatch. For τ -bench retail, contracts are mined from the public database (orders.json) without reading task descriptions, gold sequences, or reward functions. The checker validates categorical domains (order status, transaction type), positivity constraints (item prices), and affine conservation relations (net payment equals item total for non-cancelled orders). These are richer than schema-only contracts but still non-oracle. In all three environments, the injector changes only the text observed by the model; environment state is never mutated, so evaluators score the true state.

does not usually reduce recovery to a single choice, not that every contract is correct or every action useful.

4

Experimental Design

We ask three questions. RQ1: Does detector-triggered advisory feedback improve full task completion under implicit tool failures? RQ2: How does it compare to an inference-time reviewer alternative? RQ3: Does the effect persist across environments and controls, and where does it fail? Task identities, settings, and artifact hashes are frozen before the corresponding outcomes are opened, except where noted; studies frozen or amended after related outcomes were seen are labeled retrospective.

ToolMaze Confirmation (RQ1)

Construction cost. Construction is deterministic and local with zero model calls: the five cross-fitted ToolMaze registries (15,041 invariants from 400 nominal workflows), the AppWorld schema contracts, and the τ -bench database contracts each build in under two seconds (supplement).

ToolMaze crosses workflow topology (C1–C4) with explicit/implicit and transient/permanent tool perturbations (Zhu et al. 2026). From the difficult C3/C4 implicit-failure region we sample 80 unique workflow IDs, exactly 20 each from C3/P3, C3/P4, C4/P3, and C4/P4 (transient and persistent implicit faults), each carrying one fault mode. For each model, baseline and advisory episodes are randomized and interleaved, giving 160 episodes per model. We evaluate DeepSeek V4 Flash and V4 Pro via the direct DeepSeek API and Qwen 3.7 Plus and 3.7 Max via OpenRouter, all with identical prompts, temperature zero, and extended reasoning disabled. The primary endpoint within a model is paired full workflow completion, tested with the two-sided exact McNemar/binomial test. Because all four models see the same 80 workflows, joint inference clusters by workflow. We use an exact task-level sign-flip test and a 200,000-resample task-cluster bootstrap. A replication applies the unchanged 80-workflow method to MiniMax M3 through an exact MiniMax-hosted FP8 route on OpenRouter. It is analyzed independently and is not pooled with the four-model primary.

Information and Non-Oracle Boundary

Method Comparison (RQ2)

The runtime is prohibited from reading the task solution, perturbation label, evaluator, gold API sequence, or hidden recovery path. Contract construction uses only public tool metadata and disjoint clean traces. We audit the remaining risk of a de facto oracle by reporting the fraction of receipt events with at least two admissible public recovery actions. This test is necessary but not sufficient: it shows the interface

A retrospective study compares three arms on 80 additional ToolMaze workflows: contemporaneous baseline, localized receipt, and a mechanism-faithful Reinforced Agent port (Ta, Zhu, and Shayandeh 2026). The port preserves separate preexecution review, generic v1 feedback, and up to five review loops, but substitutes the matched DeepSeek tier for the source paper’s GPT-4o base and reviewer; it is therefore

not an exact reproduction. Both DeepSeek tiers run all three arms.

Cross-Environment Studies (RQ3) AppWorld. AppWorld provides state-based tests that recognize alternative valid solutions (Trivedi et al. 2024). We deterministically select one variant from each of the 19 generator families in the released development split, and both DeepSeek tiers run fault baseline, fault advisory, and clean advisory, with fault type alternating between a missing field and a scalar type flip. A held-out study then selects 32 generator families from test_normal and runs the same three conditions on DeepSeek V4 Flash. The two samples are analyzed and reported separately. τ -bench retail. τ -bench evaluates tool-calling agents in a stateful retail environment with a simulated user and statebased reward (Yao et al. 2024). A factorial samples 50 of 115 retail test tasks with seed 20260718 and applies both persistent read faults to every task: categorical status-out-ofset and affine broken-conservation. Each task runs faulted baseline and advisory under both faults plus clean advisory, yielding 250 episodes per tier at temperature zero for each of DeepSeek V4 Flash and V4 Pro on the same 50 tasks. The prespecified primary endpoint is paired completion within each fault class, and the secondary aggregate is clustered by the 50 shared tasks. An earlier exploratory study confounded fault class with task identity and suggested a harmful conservation effect; the same-task design removes that confound. Incident-derived faults. A two-tier study reruns the 80 confirmation workflows unperturbed and injects one of eight fault types. We derive those types from the five failure classes of a published taxonomy of production silent failures (Wu 2026), under a documented independence protocol: the fault author saw only the taxonomy and fixture shapes, not the contract classes. Each fault lands on the tool the benchmark’s own fault corrupts, and per-cell assignment is frozen among the types expressible on that tool’s nominal output. Every injection’s delivered result and field-level diff are logged. The prespecified primary is intention-to-treat paired completion, and detection is a receipt on the injected tool. Because a post-run audit found imprecise placement, an earlier run was invalidated; the amended protocol was frozen after that run’s outcomes were known and is disclosed as such (supplement).

Controls Five control studies probe the receipt’s active components, all using the Ev2 encoder and paired workflow-clustered analyses. Three run on 57 reserved workflows: a specificity control (generic warning vs. localized receipt), an alwaysverify control (deterministic substitute after each advertised read, frozen after specificity outcomes and therefore retrospective), and a clean-P0 control (no fault, with the manifest frozen before any P0 outcome on these workflows was opened). Two more run on 57 fresh workflows, with interpretations registered before execution: a four-arm matchedtrigger placebo varying only when a fixed generic warning attaches, and a three-arm stripped-envelope rerun that re-

Family

Model

Base Monitor

DeepSeek V4 Flash DeepSeek V4 Pro Qwen 3.7 Plus Qwen 3.7 Max Primary aggregate MiniMax M3 replication

∆ W/L

14/80 13/80 3/80 5/80

27/80 +16.25 23/80 +12.50 15/80 +15.00 25/80 +25.00

15/2 14/4 12/0 21/1

35/320

90/320 +17.19 62/7

5/80

20/80 +18.75 17/2

Table 2: Frozen paired ToolMaze completion. The fourmodel primary aggregate clusters shared tasks; MiniMax M3 is a separately frozen third-family replication and is not pooled. ∆ is in percentage points; W/L counts discordant pairs. All models run with extended reasoning disabled. Primary task-cluster sign-flip p < .00001; 95% bootstrap interval [11.25, 23.44].

moves the recovery-tools list, because a post-run audit found the always-warn envelope leaks affordances.

5

Results

Prespecified, frozen tests are identified as such below; retrospective and post hoc analyses are labeled where they appear. Inferential statistics for the remaining control and boundary analyses are descriptive.

Detector-Triggered Feedback Replicates Across Three Families The four-model primary is positive throughout. Table 2 shows a positive, individually significant effect for every model, and all 16 prespecified model-by-stratum point estimates are nonnegative. Baseline succeeds on 35/320 modelworkflow pairs (10.9%) and the advisory on 90/320 (28.1%): a clustered mean effect of +17.2 percentage points (95% bootstrap interval [11.25, 23.44], sign-flip p < .00001). The task remains hard under the advisory, and 230/320 episodes still fail. The separate third-family replication agrees. MiniMax M3 improves from 5/80 to 20/80 completions, +18.75 points with 17 paired wins and two losses (exact p = .00073). All four prespecified stratum estimates are nonnegative. All advisory wins contain at least one detector event. Of 484 events, 443 (91.5%) retain at least two admissible recovery actions. After a receipt, the agent switches to a listed substitute in 48.6% of events, calls another tool in 33.5%, retries in 14.0%, and makes no further tool call in 3.9%. The seven aggregate losses also contain receipts, so localization does not guarantee replanning. A reasoning ablation with 8,192-token thinking budgets shows effects persist on both tiers (95% interval [10.00, 25.62], combined sign-flip p < .0001): neither standing fault awareness nor extended reasoning substitutes for the delivered signal. Both arms receive the same unmodified, fault-aware system prompt, which already instructs the model to inspect unexpected tool outputs; no system instruc-

baseline

advisory

same

RECOVERY TOOLS ADDED

ToolMaze implicit (4 models)

+17.2

receipt − stripped receipt − baseline

τ -bench status

+28.0

RECOVERY TOOLS UNCHANGED

0.0

ToolMaze clean P0 CLEAN AppWorld type flip DEV

+16.7

τ -bench conservation

0.0

AppWorld missing field DEV

0.0 0

25

50

75

100

task completion (%)

Figure 2: Paired baseline and advisory completion by study, sorted by baseline rate. Open circles mark baseline and filled circles mark advisory; concentric markers indicate identical rates. Development rows are labeled dev and are not frozen tests. Rows are not pooled; counts appear in the supplement. tion mentions contracts or receipts. The intervention is entirely in-band in the returned tool result.

Benefits Replicate in a Stateful Retail Environment A same-task factorial establishes a second positive environment. In the τ -bench factorial on DeepSeek V4 Flash, aggregate completion rises from 41/100 to 55/100: +14.0 points (95% bootstrap interval [3.0, 25.0], taskclustered sign-flip p = .028). Categorical status violations improve from 6/50 to 20/50, +28.0 points with 14/0 W/L (exact p = .00012). Broken-conservation violations hold at 35/50 in both arms, an estimate of 0 points, though the 9/9 W/L split shows pairs moving in both directions (p = 1.0). A matched DeepSeek V4 Pro run on the same 50 tasks confirms the pattern: +12.0 points overall (p = .023), categorical +18.0 points (p = .004), conservation +6.0 points (p = .607). The conservation null is consistent with any effect below the ≈23 points single-tier minimum detectable effect at 80% power; per-tier breakdowns and restriction analyses appear in the supplement. Clean advisory emits one spurious receipt across 100 episodes on the two tiers.

Recovery Tools Carry the Detectable Receipt Gain The recovery-tool list carries the detectable gain. Five matched contrasts vary one part of the receipt on the same 114 workflow pairs (Figure 3). In the registered three-arm rerun, the stripped receipt performs at baseline; restoring the recovery-tool list adds +11.4 points (p = .028). Witness detail and deferred timing produce no detectable differences, though both nulls are power-bounded (≈18 points MDE). The always-warn comparison is confounded because it supplies recovery tools at every step. Targeting the same content with the detector reduces warning volume from ≈600 per tier to 80–91 without a detected completion loss (per-tier estimates in the supplement). Schema-only detector ablation. A contemporaneous three-arm study reruns the 80 confirmation cells on

+11.4 +12.3

stripped − baseline localized − generic on-time − deferred

+0.9 −2.6 +0.9 −10

0

10

20

completion difference (points), 95% interval

Figure 3: Contrast estimates with 95% bootstrap intervals on 114 paired workflows (57 per tier). Filled markers clear zero; open markers do not. The three null contrasts have an ≈18 point minimum detectable effect at 80% power, bounding rather than establishing equivalence.

DeepSeek V4 Flash. The learned arm replicates the confirmation (13/80 to 25/80, +15.0 points, p = .0075). The schema-only arm recovers much of the gain (21/80, +10.0 points, 11/3 W/L, p = .057); the head-to-head difference is +5.0 points (9/5 W/L, p = .42). Detection behavior differs sharply: the schema arm fires 166 receipts to the learned arm’s 111, with only 12.0% listing multiple recovery actions versus 91.9%. A companion clean-P0 study finds no measured completion harm from the schema arm’s extra false triggers (p = .375). Clean behavior. On clean P0 workflows, advisory and baseline each complete 74/114 pairs, an estimate of 0 points (95% interval [−5.26, 5.26]). The advisory emits 16 falsetrigger receipts in 14 treated pairs, with five paired rescues and five paired harms (p = 1.0), so clean traffic is not harmfree.

Receipts Outperform the Evaluated Reviewer Port Receipts outperform the reviewer port on these workflows, at a fraction of its cost. In the retrospective 80-workflow comparison, full monitor receipts improve over baseline by +15.0 points (95% bootstrap interval [6.875, 23.75], sign-flip p = .0011) and over a mechanism-faithful Reinforced Agent port by +11.25 points ([3.125, 19.375], p = .0144); the reviewer improves over baseline by +3.75 points ([−2.5, 10.0], p = .349). The receipt adds $0.13 in all-cache-miss list price (+7% of baseline cost) with no additional model call; the reviewer adds $9.24 (+498%) through up to five additional calls. The port is not an exact reproduction, so a calibrated reviewer could be more accurate (per-tier counts in the supplement).

Where Benefits and Detection Reach Their Limits Gains concentrate where faults block completion. Figure 2 synthesizes the frozen studies alongside two development rows. When faulted-baseline completion is at most 16% (ToolMaze 10.9%, τ -bench categorical 12.0%), net gains are +15.0 to +28.0 points. When most baselines complete despite the fault (conservation 70.0%, held-out AppWorld

78.1%), net effects are +6.3 points or less with rescues offset by harms. Conditional rescue rates are 21.8–34.6% in seven of eight rows; τ -bench Flash conservation is the outlier (60%, 9/15). This synthesis is post hoc and descriptive, not a deployment criterion. Held-out AppWorld: delivery without net gain. In the 19-family development characterization, type-flip faults produce same-sign gains on both DeepSeek tiers, missing-field completion is unchanged, and the schema checker emits zero receipts across 38 clean episodes. In a held-out 32family test_normal study on DeepSeek V4 Flash, baseline and advisory each complete 25/32 families (p = 1.0). The checker fires in all 32 faulted advisory episodes, but 23 pairs complete despite the fault, leaving little room for rescue. Per-fault directions and development details appear in the supplement. Incident-derived faults expose the detection boundary. The suite transcribes fault content from real production incidents, authored blind to the contract vocabulary. The unchanged detector flags 46.1% and 45.3% of injected faults on Flash and Pro. Recall is 83% (25/30) for violations expressible over structured values but 22% (6/27) for corruption inside plausible strings. Completion is null in both tiers (p = 1.0), where faulted baselines already complete 67.5% and 65.0% of episodes. Yet corrupted values propagate in roughly one in five trackable episodes, and about two thirds of those episodes still pass. The contracts’ vocabulary therefore bounds detection, while task completion can miss downstream corruption. Organic traffic evaluates detection, not recovery. On 53,078 recorded StableToolBench responses from 427 public tools, with no agent or counterfactual, the unchanged detector fires on 0.80%, concentrated in 54 tools, versus 1.39% on synthetic clean workflows (Guo et al. 2024). Flags include HTTP errors inside transport-success responses and empty payloads missing learned fields; not every flag is adjudicated as a true failure. This measures detection, not recovery.

6

Discussion and Limitations

Why nonbinding matters. Because the receipt is nonbinding, a false positive can only add context and salience; every action stays available. That does not eliminate harm: the paired losses show an agent can overreact to correct but irrelevant information, so preserving the action space is not a formal safety shield. Deployment value depends on fault prevalence. Every positive study uses a fault-enriched sample, so we bound the deployment mixture by fault prevalence π. At the frozen point estimates the advisory is net-nonnegative at every prevalence; at the unfavorable interval endpoints of both studies a positive net effect requires π ≳ 0.32 (supplement). At low prevalence the case rests on the clean effect staying near its observed zero. A deployment should therefore audit both its clean receipt rate and fault prevalence before relying on the observed net benefit.

What the signal contributes. The controls point to recovery tools as the active content in these ToolMaze samples; null contrasts for witness detail, timing, and salience remain power-bounded. The schema ablation suggests that learned contracts buy selectivity and better recovery lists rather than a clearly larger completion effect. Detector–injector alignment. We designed the τ -bench and AppWorld faults so that their contract sources could express them, so those studies test whether the signal is actionable, not whether the detector finds arbitrary faults. ToolMaze draws on the benchmark’s own fault taxonomy and on generic Daikon-lineage properties, but the miner and the injector may still be aligned. The incident-derived suite addresses that: on faults authored blind to the vocabulary, detection falls to ≈46%, high on violations a value can express and low inside plausible strings. No study here measures recovery from failures that occur naturally. Limitations. No episode here runs against a live deployment. What we inject is placement and prevalence rather than fault content: the incident suite’s eight types come from a published taxonomy of production silent failures (Wu 2026), authored blind to the contract vocabulary, and the unchanged detector also runs on 53,078 recorded public-API responses. Recovery from organic failures remains open. ToolMaze spans five models from three provider families, but τ -bench covers one family and the signal decomposition only two DeepSeek tiers. Contracts inherit their sources’ blind spots and false positives: schema contracts cover fewer modes, and AppWorld’s development sample leans on two domains. Completion is a blunt measure too. Clean controls record paired harms even where the aggregate effect is zero, and an episode that propagates a corrupted value can still pass. Nothing here establishes safety under adversarial tools (Zhan et al. 2024), privacy failures, or irreversible actions. Reproducibility. The supplement and anonymous artifact report the full 4,683-episode study inventory together with sanitized manifests, aggregate analyses, code, hashes, and invalidation reports. Row-level provider and protected benchmark observations are excluded from the review package.

7

Conclusion

Outcome Monitors detect contract violations and expose recovery tools without restricting the agent. Gains replicate across three provider families and two environments but concentrate where faults block completion. The results frame silent-failure recovery as an observability problem: viable actions may already exist, but the interface must make them legible. Detection beyond the contract vocabulary remains open.

References Advani, L. 2026. From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents. Accepted at the ICML 2026 Workshop on Failure Modes in Agentic AI, arXiv:2606.09863.

Alshiekh, M.; Bloem, R.; Ehlers, R.; Könighofer, B.; Niekum, S.; and Topcu, U. 2018. Safe Reinforcement Learning via Shielding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. Ammons, G.; Bodík, R.; and Larus, J. R. 2002. Mining Specifications. In Proceedings of the 29th ACM SIGPLANSIGACT Symposium on Principles of Programming Languages (POPL), 4–16. Babu, R. S.; and Iyer, L. G. 2026. Contract2Tool: Learning Preconditions and Effects for Reliable Tool-Augmented LLM Agents. arXiv:2606.07904. Bhardwaj, V. P. 2026. Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agents. arXiv:2602.22302. Chang, E. Y.; and Geng, L. 2025. SagaLLM: Context Management, Validation, and Transaction Guarantees for MultiAgent LLM Planning. Proceedings of the VLDB Endowment, 18: 4874–4886. Dong, Y.; Mu, R.; Zhang, Y.; Sun, S.; Zhang, T.; Wu, C.; Jin, G.; Qi, Y.; Hu, J.; Meng, J.; Bensalem, S.; and Huang, X. 2024. Safeguarding Large Language Models: A Survey. arXiv:2406.02622. Ernst, M. D.; Perkins, J. H.; Guo, P. J.; McCamant, S.; Pacheco, C.; Tschantz, M. S.; and Xiao, C. 2007. The Daikon System for Dynamic Detection of Likely Invariants. Science of Computer Programming, 69(1–3): 35–45. Findler, R. B.; and Felleisen, M. 2002. Contracts for HigherOrder Functions. In Proceedings of the ACM SIGPLAN International Conference on Functional Programming (ICFP), 48–59. Gou, Z.; Shao, Z.; Gong, Y.; Shen, Y.; Yang, Y.; Duan, N.; and Chen, W. 2024. CRITIC: Large Language Models Can SelfCorrect with Tool-Interactive Critiquing. In International Conference on Learning Representations. Guo, L.; Liu, W.; Heng, Y. W.; Chen, T.-H. P.; and Wang, Y. 2026. Agent-SAMA: State-Aware Mobile Assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 29459–29467. Guo, Z.; Cheng, S.; Wang, H.; Liang, S.; Qin, Y.; Li, P.; Liu, Z.; Sun, M.; and Liu, Y. 2024. StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models. In Findings of the Association for Computational Linguistics (ACL), 11143–11156. Gurram, B. 2026. Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents. arXiv:2604.16706. Huang, J.; Chen, X.; Mishra, S.; Zheng, H. S.; Yu, A. W.; Song, X.; and Zhou, D. 2024. Large Language Models Cannot Self-Correct Reasoning Yet. In International Conference on Learning Representations. Inan, H.; Upasani, K.; Chi, J.; Rungta, R.; Iyer, K.; Mao, Y.; Tontchev, M.; Hu, Q.; Fuller, B.; Testuggine, D.; and Khabsa, M. 2023. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674. Leucker, M.; and Schallhart, C. 2009. A Brief Account of Runtime Verification. Journal of Logic and Algebraic Programming, 78(5): 293–303.

Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2024. Let’s Verify Step by Step. In International Conference on Learning Representations. Liu, X.; Yu, H.; Zhang, H.; Xu, Y.; Lei, X.; Lai, H.; Gu, Y.; Ding, H.; Men, K.; Yang, K.; Zhang, S.; Deng, X.; Zeng, A.; Du, Z.; Zhang, C.; Shen, S.; Zhang, T.; Su, Y.; Sun, H.; Huang, M.; Dong, Y.; and Tang, J. 2024. AgentBench: Evaluating LLMs as Agents. In International Conference on Learning Representations. Liu, Y.; Peng, X.; Cao, J.; Wang, X.; Deng, S.; Chen, J.; Yin, J.; and Zhang, X. 2026. ToolGate: Contract-Grounded and Verified Tool Execution for LLMs. In Findings of the Association for Computational Linguistics: ACL 2026, 9653–9684. Association for Computational Linguistics. Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In Advances in Neural Information Processing Systems, volume 36. Meyer, B. 1992. Applying “Design by Contract”. Computer, 25(10): 40–51. Mou, Y.; Xue, Z.; Li, L.; Liu, P.; Zhang, S.; Ye, W.; and Shao, J. 2026. ToolSafe: Enhancing Tool Invocation Safety of LLM-based Agents via Proactive Step-level Guardrail and Feedback. In Findings of the Association for Computational Linguistics: ACL 2026, 37125–37153. Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; Zhao, S.; Hong, L.; Tian, R.; Xie, R.; Zhou, J.; Gerber, M.; Li, D.; Liu, Z.; and Sun, M. 2024. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs. In International Conference on Learning Representations. Reddy, V.; Challaram, S. R.; and Basu, A. 2026. Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents. Accepted at the KDD 2026 Workshop on Evaluation and Trustworthiness of Agentic AI, arXiv:2607.07405. Ruan, Y.; Dong, H.; Wang, A.; Pitis, S.; Zhou, Y.; Ba, J.; Dubois, Y.; Maddison, C. J.; and Hashimoto, T. 2024. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. In International Conference on Learning Representations. Schick, T.; Dwivedi-Yu, J.; Dessì, R.; Raileanu, R.; Lomeli, M.; Hambro, E.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems, volume 36. Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 36. Ta, A.; Zhu, J.; and Shayandeh, S. 2026. Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents. In Proceedings of the Fifth Workshop on Generation, Evaluation

and Metrics, 136–147. Association for Computational Linguistics. Trivedi, H.; Khot, T.; Hartmann, M.; Manku, R.; Dong, V.; Li, E.; Gupta, S.; Sabharwal, A.; and Balasubramanian, N. 2024. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 16022–16076. Vuddanti, S. V.; Shah, A.; Chittiprolu, S. K.; Song, T.; Dev, S.; Zhu, K.; and Chaudhary, M. 2025. PALADIN: Self-Correcting Language Model Agents to Cure ToolFailure Cases. Accepted oral paper at the non-archival AAAI 2026 Workshop on LLM-based Multi-Agent Systems, arXiv:2509.25238. Wang, H.; Poskitt, C. M.; and Sun, J. 2026. AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents. In Proceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE). Wu, W. 2026. When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime. arXiv:2606.14589. Xu, H.; Zhu, Z.; Pan, L.; Wang, Z.; Zhu, S.; Ma, D.; Cao, R.; Chen, L.; and Yu, K. 2025. Reducing Tool Hallucination via Reliability Alignment. In Proceedings of the 42nd International Conference on Machine Learning (ICML). Yang, J.; Jimenez, C. E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K.; and Press, O. 2024. SWE-agent: AgentComputer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems, volume 37. Yao, S.; Shinn, N.; Razavi, P.; and Narasimhan, K. 2024. τ -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045. Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations. Zhan, Q.; Liang, Z.; Ying, Z.; and Kang, D. 2024. InjecAgent: Benchmarking Indirect Prompt Injections in ToolIntegrated Large Language Model Agents. In Findings of the Association for Computational Linguistics (ACL). Zhang, Y.; Chen, J.; Wang, J.; Liu, Y.; Yang, C.; Shi, C.; Zhu, X.; Lin, Z.; Wan, H.; Yang, Y.; Sakai, T.; Feng, T.; and Yamana, H. 2024. ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 11388–11422. Zhu, D.; Ma, X.; Shen, Y.; Li, X.; Zhao, Y.; Wang, S.; Yan, L.; and Yin, D. 2026. When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents. arXiv:2606.05806.

Supplementary Material: Outcome Monitors Outcome Monitors: Recovery Affordances for Silent Tool Failures This supplement accompanies the main paper and provides additional tables, analysis details, and reproducibility information. The main paper is self-contained; this material supports deeper inspection.

A

Reasoning Ablation

D

Reviewer-Port Comparison Counts

Strategy

Flash

Pro

List cost

Baseline Localized receipt Reinforced Agent port

16/80 27/80 19/80

13/80 26/80 16/80

$1.853 $1.984 $11.089

Table 6: Matched ToolMaze comparison on 80 separately hashed workflows (main paper, Results). Counts are completed workflows per tier; cost is the two-tier all-cache-miss list price. The port substitutes matched DeepSeek tiers for the source paper’s GPT-4o and is not an exact reproduction.

E Model

Think

Base

Contract

∆

W/L

V4 Flash V4 Flash V4 Pro V4 Pro

off 8k off 8k

14/80 13/80 13/80 14/80

27/80 25/80 23/80 30/80

+16.25 +15.00 +12.50 +20.00

15/2 15/3 14/4 16/0

27/160

55/160

+17.50

31/3

Think-on combined

Table 3: Contract effects persist in every configuration. Reasoning ablation on DeepSeek V4 with an 8,192-token thinking budget. Combined thinking-on task-cluster sign-flip p < .0001; 95% bootstrap interval [10.00, 25.62].

B

Paired Outcome Decomposition

Environment

Model

Rescue Harm Both pass Both fail

ToolMaze DS Flash ToolMaze DS Pro ToolMaze Qw Plus ToolMaze Qw Max ToolMaze repl. MiniMax M3 AppWorld dev DS Flash AppWorld dev DS Pro AppWorld test DS Flash

15 14 12 21 17 3 2 2

2 4 0 1 2 1 1 2

12 9 3 4 3 13 14 23

51 53 65 54 58 2 2 5

Table 4: Paired outcome decomposition. Rescue means advisory-only success; harm means baseline-only success. AppWorld development and held-out test samples remain separate. MiniMax M3 is a separately frozen replication and is not pooled with the four-model primary.

C

Post-Receipt Recovery Behavior First action after receipt

Events

Share

Listed recovery substitute Different tool Retry receipted tool No further tool call

235 162 68 19

48.6% 33.5% 14.0% 3.9%

Table 5: Descriptive first-action distribution after 484 receipt events in the four frozen ToolMaze confirmations. The categories characterize retained recovery behavior; they do not identify an effect of receipt wording.

Detector Development and Audit

ToolMaze detector Schema-only gate Learned C2 gate C3/C4 cross-fit audit

Impl. recall

Clean FPR

Coverage

55.0% 80.0% 79.6%

18.0% 4.0% 1.39%

— — 94.2%

Table 7: The initial schema-only gate failed its pre-model audit. Outcome-level detector development and task-disjoint audit. Coverage is the share of implicit fault outcomes with a learned contract available; recall given coverage is 84.5% in the C3/C4 audit. The audit rate is per observed outcome rather than per episode, so repeated opportunities accumulate. In the separate clean-P0 control they reach 16 false contract events in 14/114 treated episodes, with five paired harms and five paired rescues.

F

Schema-Only Detector Ablation

Contrast

Completion

∆ W/L

p

Learned vs. baseline 13/80 → 25/80 +15.0 15/3 .0075 Schema-only vs. baseline 13/80 → 21/80 +10.0 11/3 .0574 Learned vs. schema-only 21/80 → 25/80 +5.0 9/5 .4240

Table 8: Frozen three-arm schema ablation (protocol OUTCOME_CONTRACT_SCHEMA_ABLATION_80_V1, manifest hash 592d5a43) on the exact 80 frozen confirmation cells, DeepSeek V4 Flash, all three arms contemporaneous, temperature 0. The learned arm contemporaneously replicates the frozen confirmation effect. Mechanism: the learned arm emits 111 receipt events on 67 cells (91.9% listing ≥ 2 admissible recovery actions); the schema-only arm emits 166 events on 65 cells (12.0%), because the public registry carries no recovery-start relations. Every cell is faulted, so this sample does not exercise the schema detector’s 18% clean false-positive rate.

G

Incident-Derived Fault Suite

Derivation and independence protocol. Eight fault types were transcribed from the five-class production-incident taxonomy of Wu (2026, arXiv:2606.14589; 22 public postmortems). The fault author was shown only the taxonomy section of that paper, two P0 fixture files, and the transformation API contract, never the contract classes, learned

registries, or this paper. They returned deterministic transformation algorithms with per-fault incident provenance. The returned specification was implemented verbatim (two internal example/algorithm inconsistencies were resolved in favor of the algorithm text and logged). Per cell, the fault is injected on the first eligible result from the exact tool(s) the benchmark’s own P3/P4 fault corrupts. Assignment is frozen among the fault types that change the target tool’s nominal fixture output, recomputed by the manifest validator from hashed fixtures. Every injection records the delivered result and a field-level before/after diff.

Class Fault type C B E C C D B A

Endpoint

Flash

Table 9: Incident-derived fault suite, protocol OUTCOME_ CONTRACT_INCIDENT_FAULT_80_V2 (Flash manifest f4e9924a, Pro manifest d39bd825), 160 episodes per tier on the 80 frozen confirmation workflows with bit-identical per-task fault assignment across tiers. Contamination is a string-match lower bound: a distinctive injected value (≥ 8 characters) appearing in the trace demonstrates propagation into a later real tool call or the final answer, because the engine logs only pre-injection results. Non-injections are attributed: target tool never called (3 episodes per arm on Flash; 5 baseline and 4 advisory episodes on Pro) or live target result ineligible for the assigned transform (1 episode per arm, both tiers).

8/8 8/10 7/10 4/10 3/10 3/10 1/9 0/8

Cross-Environment Counts

Env.

Fault class

Base

Adv.

∆ Status

ToolMaze implicit (4 models) 35/320 90/320 +17.2 frozen τ -bench status (Flash) 6/50 20/50 +28.0 frozen τ -bench conserv. (Flash) 35/50 35/50 0.0 frozen AppWorld dev type flip 12/18 15/18 +16.7 dev AppWorld dev missing field 17/20 17/20 0.0 dev

Table 11: Underlying counts for the main paper’s descriptive synthesis. Rows are not pooled. The figure’s 95% intervals are task-cluster bootstrap intervals from the same run ledgers: τ -bench status [16.0, 40.0], conservation [−16.0, 16.0]; AppWorld type flip [0.0, 33.3], missing field [−20.0, 20.0]; ToolMaze uses the frozen confirmation interval [11.25, 23.44]. The τ -bench intervals resample the 50 tasks shared by both fault classes.

Pro

Primary ITT (base → adv.) 54/80 → 54/80 52/80 → 52/80 paired W/L, exact p 3/3, p = 1.0 4/4, p = 1.0 Injected episodes 152/160 149/160 Same-tool detection 35/76 (46.1%) 34/75 (45.3%) Coincidental-receipt episodes 11 10 Contaminated / trackable (base) 12/61 15/61 of which passed 7 10

10/10 8/10 7/10 3/9 3/10 3/10 1/9 0/8

Table 10: Same-tool detection by fault type (denominators are changed injections). Classes follow Wu (2026): A environment quirks, B design-assumption mismatches, C error swallowing/dilution, D chained fabrication, E operational omission. The gradient separates value-level violations (caught by domain, positivity, and echo invariants) from content-level corruption inside plausible strings (missed).

H Invalidated first run (disclosure). A first run of this suite used step-count placement. A post-run audit then found that only 58% of its injections hit the benchmark-targeted tool, that its detection count included receipts on non-injected tools, and that delivered content was not recorded. The run was therefore invalidated, and its numbers are not reported. The amended protocol (tool-level targeting, same-tool detection, delivered-content audit trail, eligibility-aware assignment) was frozen after that run’s outcomes were known.

Det. (Flash) Det. (Pro)

vacuous_exec_swallow positional_field_shift stale_declared_state cause_stripped_http_payload error_page_as_payload context_pollution_fabrication ref_id_substitution env_threshold_truncation

I

AppWorld Studies

The development characterization and held-out confirmation are distinct studies. The former covers 19 generator families on both DeepSeek tiers and is used for fault-class analysis. The latter freezes 32 families from test_normal and evaluates DeepSeek V4 Flash only. Its manifest listed both DeepSeek tiers as planned, but no Pro episodes were executed or analyzed. Sample

Base

Adv. W/L Clean

Dev, Flash 14/19 16/19 Dev, Pro 15/19 16/19 Held-out test, Flash 25/32 25/32

Inj.

3/1 15/19 100% 2/1 16/19 100% 2/2 26/32 100%

Table 12: The held-out test_normal study has zero net effect (McNemar p = 1.0) despite fault injection in all 64 faulted episodes. AppWorld studies are reported separately. Injection is the fraction of faulted episodes receiving the protocol-specified corruption.

Fault-Type Breakdowns Sample

Fault

8/10 9/10 6/9 6/9 12/16 13/16

Specificity (localized vs. generic) Always-verify Clean-P0 Placebo timing (true vs. placebo) Salience (stripped vs. baseline) τ -bench conservation

∆

Model Base Contract

Dev Missing field Flash Dev Missing field Pro Dev Type flip Flash Dev Type flip Pro Held-out Missing field Flash Held-out Type flip Flash

8/10 0 9/10 0 8/9 +22.2 7/9 +11.1 14/16 +12.5 11/16 −12.5

Table 13: The development by-fault direction does not replicate in the held-out sample. AppWorld completion by fault assignment, with development and held-out samples kept separate.

J

τ -bench Detailed Results

Fault class

Base

Contract

∆

W/L

Categorical status Conservation

6/50 35/50

20/50 35/50

+28.0 0.0

14/0 9/9

Aggregate

41/100

55/100

+14.0

23/9

Table 14: Frozen same-task τ -bench factorial. The categorical primary comparison has exact paired p = .00012. The secondary aggregate clusters both fault contrasts by task (p = .028; 95% bootstrap interval [3.0, 25.0]). An earlier 115-task-per-tier exploratory run assigned fault class by task-index parity. Its harmful conservation estimate is retained as audit history but does not persist in the deconfounded factorial design.

Control Study Counts and Power Analysis

Control Always verify Clean P0 advisory

Flash

Pro

Joint ∆ [95% CI]

11→12 38→40

16→18 36→34

+2.63 [−2.63,7.89] 0.00 [−5.26,5.26]

Table 15: Separately hashed ToolMaze controls, each with 57 workflows per tier. Counts are baseline→treatment completions; joint ∆ and its interval are in percentage points; intervals cluster the shared tasks. Always verify is retrospective and has only 19 verification reads. The specificity control’s joint counts (generic caution 43/114, localized 40/114, ∆ = −2.63 [−11.40,6.14]) appear in the main text. The control studies (specificity, always-verify, clean-P0, τ -bench conservation) all report null results, so what matters is how much each null can rule out. The minimum detectable effect (MDE) at 80% power quantifies that.

114 114 114 114 114 50

−2.63 pp 18 pp +2.63 pp 19 pp 0.0 pp 17 pp +0.88 pp 18 pp +0.88 pp 18 pp 0.0 pp 23 pp

Table 16: Simulation-based MDE (sign-flip test, 10,000 simulations per delta, α = .05, seed 42). All controls are underpowered to detect effects below ≈ 17–23 percentage points; null results cannot be interpreted as equivalence. The timing and salience contrasts of the decomposition studies below share the specificity design (114 pairs, sign-flip) and therefore its ≈18 percentage points MDE.

L

The frozen factorial samples 50 retail test tasks and applies both persistent get_order_details faults to every task, so each task contributes two baseline/advisory pairs plus one clean-advisory episode. The 250-episode DeepSeek V4 Flash run completes without errors: all 200 faulted episodes fire the intended corruption, and none of the 50 clean episodes emits a receipt.

K

n pairs Observed ∆ MDE

Control study

Warning-Signal Decomposition Controls

Matched-trigger placebo. Protocol OUTCOME_ CONTRACT_MATCHED_PLACEBO_57_V1 samples 57 fresh C3/C4 workflows from the task IDs appearing in no prior manifest (strata c3/P3 14, c3/P4 15, c4/P3 14, c4/P4 14) and runs four arms per tier (228 episodes per tier). All warned arms deliver the byte-identical generic caution of the specificity control inside the same envelope; the arms differ only in when the warning attaches. The matched placebo suppresses the warning at each detected violation, forwards the violating result raw, and delivers one queued identical warning at the nearest following detector-consistent result, the only online-implementable position match. Of the suppressed violations, 75 of 77 (Flash) and 80 of 82 (Pro) had their warning delivered, and any undelivered pending warning is recorded. Arm

Flash

Pro

Baseline True trigger Matched placebo Always warn

12/57 16/57 17/57 20/57

14/57 17/57 15/57 26/57

Table 17: Matched-trigger placebo completions. Registered primary (true trigger minus matched placebo, two-tier workflow clusters): +0.88 percentage points, sign-flip p = 1.0, 95% bootstrap interval [−4.39, 6.14]; per-tier W/L 3/4 (Flash, p = 1.0) and 3/1 (Pro, p = .625). Secondary armminus-baseline estimates: true trigger +7.0 (p = .344) and +5.3 (p = .453); matched placebo +8.8 (p = .180) and +1.8 (p = 1.0); always warn +14.0 (p = .039) and +21.1 (p = .004) for Flash and Pro, respectively. Always-warn confound (audit disclosure). The registered envelope attaches admissible_recovery_tools to every warned result. A post-run audit found that no other channel exposes substitute names (0 of 270 tool descriptions) and that on Pro nine of the fourteen always-warn paired wins first call an envelope-surfaced substitute. Excluding those wins moves the always-warn-versus-baseline p from .004 to .45 (Flash: .039 to .065). The exclusion was not registered and is diagnostic only; the stripped-envelope study tests the

hypothesis prospectively. The detector is additionally silent in 9 of 57 workflows, where only a constant arm delivers any warning, so part of the always-warn margin is recall coverage rather than timing. Stripped-envelope study. Protocol OUTCOME_ CONTRACT_STRIPPED_ALWAYS_WARN_57_V1 reruns the same 57 cells contemporaneously with three arms per tier: baseline, the full always-warn envelope, and the identical envelope with the admissible_recovery_tools key absent. The registered primary is always-warn-stripped minus baseline; the registered secondaries are full minus stripped (the leakage estimate) and full minus baseline (the contemporaneous replication). Interpretations for each outcome were frozen before execution. Arm

Flash

Pro

Baseline Always warn (full) Always warn (stripped)

17/57 19/57 14/57

12/57 24/57 16/57

Table 18: Stripped-envelope study completions. Two-tier joint contrasts: stripped minus baseline +0.88 percentage points (p = 1.0, [−6.14, 7.89]); full minus stripped +11.40 (p = .028, [2.63, 20.18]); full minus baseline +12.28 (p = .018, [3.51, 21.93]). Per-tier full minus stripped: Flash +8.8 (9/4, p = .267), Pro +14.0 (10/2, p = .039). Pro reproduces its frozen always-warn estimate (12 → 24, 13/1, p = .0018); Flash does not (17 → 19, +3.5, p = .774), its contemporaneous baseline sitting five completions above the frozen value with a stable always-warn arm. Warned arms fire on every tool result (617/599 full and 550/540 stripped events for Flash/Pro, against 80–91 for the triggered arm of the placebo study). Envelope integrity is verified per event in both ledgers: the full arm carries the affordance key in every event and the stripped arm in none. Both tiers complete with zero infrastructure retries.

M

N

This section reproduces the exact texts the models received. Unicode em dashes and one emoji glyph in the original source strings are rendered below as - and [emoji]; everything else is verbatim.

ToolMaze agent system prompt

Baseline and advisory arms use the identical, unmodified fault-aware system prompt from the pinned ToolMaze checkout (evaluation/agents/openai_agent.py), shown in Figure 4. No arm receives any instruction mentioning contracts, receipts, or the outcome_contract field: the receipt is purely in-band.

ToolMaze receipt envelope

On a detected violation the interface replaces the tool message content with the JSON serialization of {"tool_result": <raw result>, "outcome_contract": <receipt>}; the raw result is always preserved. A real receipt from the frozen confirmation ledger (schema v1; task with the corrupted stock quote): Receipt, schema v1 (frozen confirmation) { "status": "inconsistent", "violations": [ {"code": "nonpositive_value", "detail": "nominally positive field price_usd is not positive"}, {"code": "unseen_category", "detail": "field currency has unseen categorical value"} ], "admissible_recovery_tools": [ "get_stock_yahoo_finance", "get_stock_alpha_vantage" ]

Per-Corruption-Type Detector Recall

The cross-fitted audit covers nine corruption types across the C3/C4 region, and recall varies substantially by fault class. Corruption type Logic violation Physics violation Temporal contradiction Missing required fields Tool leakage Internal contradiction Fact contradiction Irrelevant response Statistical inconsistency

n faults

Recall

Coverage

408 463 393 328 347 195 596 323 354

91.2% 85.3% 82.4% 80.2% 79.8% 77.9% 76.2% 70.3% 70.1%

97.5% 93.5% 95.9% 95.7% 87.3% 94.9% 92.3% 96.6% 94.9%

Table 19: Per-corruption-type detector recall (C3/C4 crossfitted audit, sorted by recall). Coverage is the share of fault outcomes with a learned contract available; recall given coverage exceeds 73% for all types.

Prompts and Receipt Texts

}

The frozen confirmation uses the compact schema v1 above. The ToolMaze control studies use receipt schema v2 (ledger field receipt_schema_version value tool_property_path_ expected_observed_first_v2), which carries the paper’s five-tuple witness. A real v2 receipt:

ToolMaze agent system prompt (P1–P4, identical in baseline and advisory arms) You are an autonomous AI assistant that executes tool-based workflows. Tools may occasionally fail or return unexpected data. TASK COMPLETION - The user query may describe multiple sub-tasks. You must complete ALL of them - do NOT skip any sub-tasks. - Every fact, data retrieval, or state change must come from a tool call. Never substitute a tool result with internal knowledge, and never report a step as completed unless you actually called the corresponding tool and received its result in this conversation. - Sub-tasks may be accomplishable through different tool combinations. FAULT HANDLING - When evaluating tool outputs, consider whether the returned data is coherent and usable for the next step - fields should be present, values should be semantically reasonable, and types should match what the workflow requires. - If you determine that a required step cannot be accomplished by any available tool or approach, stop the workflow and clearly report which step failed and why, rather than proceeding with incomplete results. FINAL ANSWER FORMAT - Be CONCISE and DIRECT. No emojis, markdown formatting (**, [emoji], etc.), or bullet points. - If SUCCESSFUL: State only essential facts and requested results (e.g., confirmation codes). Omit internal technical details. - If HALTED: State clearly that the task was aborted and the reason (e.g., “Task aborted: the pricing tool returned corrupted data after retry and no other alternative approaches exist”). Do not apologize excessively.

Figure 4: Verbatim ToolMaze system prompt. Receipt, schema v2 (control studies) { "status": "inconsistent", "violation": { "tool": "get_weatherapi_alert_card_native", "property": "learned_type_mismatch", "path": "celsius_now", "expected": ["integer"], "observed": "string" }, "admissible_recovery_tools": [ "get_weatherapi_alert_card_native", "get_visualcrossing_forecast_chip_native" ]

Generic-caution text (specificity control). The generic arm shares the detector, envelope, timing, and recovery affordances, and replaces only the violation witness with one frozen string: Generic-caution receipt (specificity control) { "status": "inconsistent", "violation": { "property": "generic_caution", "message": "This result may be unreliable; verify it before relying on it." }, "admissible_recovery_tools": ["..."] }

}

The decomposition controls (Section L) reuse this envelope. The warned arms of the matched-trigger placebo and the full always-warn arm are byte-identical to it; the stripped arm removes only the affordance key: Stripped always-warn envelope (decomposition study) { "status": "inconsistent", "violation": { "property": "generic_caution", "message": "This result may be unreliable; verify it before relying on it." } }

AppWorld and τ -bench advisory text. Both ports append a plain-text block to the observed result. The τ -bench template (AppWorld differs only in naming its source “its public response contract” and referring to “documented APIs”): τ -bench advisory template (appended to the raw result) [OUTCOME CONTRACT ADVISORY] The observed result from ‘<function>‘ is inconsistent with a known invariant of the retail database. Violation: <code> at ‘<path>‘; expected <expected>, observed <observed>. This advisory is nonbinding. Treat the result as unreliable and verify or recover using the available documented tools and task context before relying on it. All actions remain available.

Fired invariant classes. Eight violation classes fire across the frozen ToolMaze ledgers, each shown with a real fired instance: • nonpositive_value: nominally price_usd is not positive.

positive

field

• unseen_category: field currency has unseen categorical value. • learned_type_mismatch: field rate has unexpected type. • learned_echo_mismatch: field flight_number differs from call argument. • missing_learned_field: display_currency.

missing

nominal

field

• unexpected_field: field absent from nominal traces: error. • explicit_error: error-bearing field(s): error. • affine_relation_broken: fields cart_subtotal and item_count violate a nominal relation.

O

Organic Tool Traffic

Every paired-completion study in the main paper injects faults, because that comparison requires a known fault. This section drops the agent and the counterfactual and asks a narrower question that organic data can answer: how often does the unmodified detector fire on real recorded tool responses? Corpus and protocol. We use the response cache released with StableToolBench (Guo et al., 2024; arXiv:2403.07714), which records real RapidAPI responses collected for ToolBench (Qin et al., 2024); both are Apache-2.0. We keep responses the cache marks successful whose payload is a JSON object, and we keep tools with at least 45 such responses. That leaves 427 tools and 53,078 responses. Per tool we mine contracts from 45 responses and hold out 20, repeating over five seeds. The detector, the miner, and the invariant classes are unchanged from the main paper’s method section. Nothing is injected and no fault is authored.

Violation class unexpected_field missing_learned_field explicit_error learned_type_mismatch unseen_category learned_echo_mismatch nonpositive_value affine_relation_broken

Firings

Share

364 78 69 53 12 6 6 2

61.7% 13.2% 11.7% 9.0% 2.0% 1.0% 1.0% 0.3%

Table 20: The detector fires on 0.80% of organic responses, and firings concentrate in 54 of 427 tools. Violation classes summed over five seeds on held-out real RapidAPI responses; 6,860 responses scored per seed, 41 to 65 flagged. Fieldshape classes dominate; the value and relation classes that carry the main paper’s effects fire rarely on this corpus. Result. The detector flags 0.80% of held-out responses (41 to 65 of 6,860 per seed, range 0.60% to 0.95%). For comparison, the clean-P0 control in the main paper puts the false-positive rate at 1.39% on synthetic clean workflows, so the synthetic estimate is not optimistic relative to this corpus. Firings are concentrated rather than diffuse: 54 of 427 tools account for all of them. What the firings look like. We do not adjudicate every flag, and field-shape deviations on a public API are often benign. Two recurring patterns are worth showing. The first is an error delivered inside a response the transport layer reports as successful, which is the failure mode the paper is about: airportflights → {"Error": "invalid ident, please provide a valid ident. You can use the airport search to determine the right one."}

The second is an expected payload that is simply absent, as when coins_get_markets returns {} where the mined contract expects a data field. Scope. This is a detection measurement, not an effectiveness one. There is no agent, no receipt, and no counterfactual, so nothing here speaks to recovery or completion in deployment. The responses are cached and filtered by StableToolBench rather than collected live. What the section establishes is narrower. Contracts mine cleanly from organic traffic. The detector’s firing rate there is consistent with the synthetic clean-P0 estimate. Silent failures of the kind the paper targets are present in real tool traffic, without anyone injecting them.

P

Construction Cost Detail

ToolMaze: 400 unique nominal workflows produce 1,800 workflow-fold inputs across five cross-fitted registries. The build materializes 15,551 candidate invariants and accepts 15,041 (96.7%). Invariant classes include per-field type constraints, echo fields (a[f ] = y[f ]), positivity (y[f ] > 0), categorical domains, affine relations, and temporal orderings. AppWorld: 11 released API-documentation files yield 473 candidate read contracts, of which 149 (31.5%) are accepted after recursive schema validation.

τ -bench: the public orders.json database yields categorical (status, transaction_type), range (item price > 0), and affine (net payment conservation) contracts. All builds are deterministic with zero model or API calls. Historical human authoring and inspection time was not contemporaneously recorded and is reported as unknown.

Q

Reproducibility Details

Models and access. DeepSeek V4 Flash and V4 Pro via the direct DeepSeek API. Qwen 3.7 Plus and 3.7 Max via OpenRouter. The separate MiniMax M3 replication uses OpenRouter pinned to the MiniMax-hosted FP8 route with fallbacks disabled. All ToolMaze runs use temperature zero and extended reasoning disabled. Token usage and cost. The ToolMaze confirmation uses 24.81 million input tokens and 0.906 million output tokens; its all-cache-miss list-price upper bound is $15.46. The AppWorld development study uses 0.539/0.741 million cache-miss input tokens, 7.56/7.40 million cache-hit input tokens, and 0.332/0.328 million output tokens for Flash/Pro, at $1.204 total list price. Its held-out test_normal study uses 0.724 million cache-miss input tokens, 9.754 million cache-hit input tokens, and 0.383 million output tokens, at $0.345 total list price. The method comparison costs $14.926. The always-verify and clean-P0 controls cost $2.730 and $2.372, respectively. The MiniMax M3 replication costs $2.18. The frozen τ -bench factorial costs $0.43. The matched-trigger placebo study costs $5.73 and its stripped-envelope companion $4.36. Artifact mapping. The anonymous artifact’s MANIFEST.md and ARTIFACT_INDEX.md map each paper result to its sanitized manifest, aggregate report, and analysis code. Row-level provider responses and protected benchmark observations are intentionally excluded. Selected study status and analysis units. Study

Status

Unit and aggregate

ToolMaze primary MiniMax replication Reasoning ablation Specificity / clean

frozen 80 shared tasks; four-model aggregate frozen separately 80 paired tasks; per-run analysis separate hashes 80 shared tasks; paper aggregate separate hashes 57 shared workflows; per-run joint analyses Matched placebo frozen 57 fresh workflows; two-tier joint Stripped always- frozen same 57 workflows; three contemporawarn neous arms AppWorld dev development 19 families; joint task analysis AppWorld held-out frozen, Flash run 32 families; Flash analysis τ -bench factorial frozen 50 tasks; two fault contrasts per task

Table 21: Selected study status and analysis units. The artifact’s STUDY_INVENTORY.csv lists all 22 studies and 4,683 episodes. Separately hashed studies are never pooled. The held-out AppWorld manifest planned two tiers, but only Flash was executed. MiniMax is never pooled into the primary ToolMaze aggregate; the τ -bench secondary aggregate clusters both contrasts by task.

Aggregate regeneration. paper_aggregate_analysis.py validates all four frozen ToolMaze ledgers in the research workspace, reconstructs the shared-task effect and bootstrap interval, and independently reproduces the 484-event recovery-action counts. The review artifact packages the resulting aggregate, analysis code, and independent headline-value and cross-study validators, but not row-level provider responses. The aggregate explicitly excludes the retired substitute-availability post-hoc because its classification requires the unavailable external task catalog. For AppWorld, protected API-observation ledgers remain ignored; the packaged Flash aggregate and report contain only family-level counts, injection rates, usage, and hashes. The τ -bench analysis validates the frozen 50×5 factorial grid and records task-clustered inference separately from the prespecified fault-stratified tests. Finally, submission.tex is a flattened, bibliography-inlined source generated from the modular manuscript; the build checks that it has the same page count as main.pdf.

Related documents

• Silence Is Endorsement: Verification-Status Laundering in LLM Agent Pipelines#1033981
• When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning#1117932
• From idea to creation : : exploring designers’ real-time cognitive and emotional experiences throughout the design process#1106700
• Recovery beyond the individual: Toward a place-based paradigm for recovery science.#312728
• Information recovery problems#712130