Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair Xueping Gao
Jianwei Yang
Qiang Yang
[email protected] Alibaba Cloud Hangzhou, China
[email protected] Alibaba Cloud Hangzhou, China
[email protected] Alibaba Cloud Hangzhou, China
arXiv:2607.24604v1 [cs.CL] 27 Jul 2026
Abstract Generate–test–revise loops are common in coding agents, but repetition alone provides no reliability guarantee. We study the gap between finding a correct patch and retaining, verifying, and submitting it. A sealed five-seed study over 30 HumanEval repairs produces 900 three-revision trajectories. Under forced revision, current correctness with current traces falls from 0.820 after one revision to 0.673 after two, although ever-correct rises to 0.847. Two common-state studies use 2,430 branches from identical frozen programs to remove post-treatment risk-set bias. In a prespecified 14B replication, stale traces harm 34/135 correct starts versus 4/135 with current traces, a 22.2-point increase (task-cluster 95% CI [8.9, 37.0], exact Holm 𝑝 = 0.0337). A prospective 540-rollout policy eliminates observed correct-start harm but reduces wrong-start repair and fails its joint criterion. Repository experiments over 24 bugs and four coder stacks expose floor effects and component heterogeneity without Holm-significant effects. We therefore separate admission, preservation, grounded certification, competence, and liveness. We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation that binds verifier evidence to exact code states, preserves verified checkpoints, and emits auditable admission receipts. The implementation is an executable specification and conformance artifact, not evidence of improved repair competence or calibrated verifier dependence.
CCS Concepts
only when its accepted evidence has sufficiently low risk. The distinction matters because an increasing ever-correct rate can coexist with a decreasing current or final correctness rate. Prior work establishes important pieces of this problem. Intrinsic self-correction may fail or regress without reliable external feedback [9, 11]; error localization and revision ability are distinct [25]; and weak verifiers limit self-correction [28]. FeedbackEval measures how feedback types affect code repair [5]. Work on evaluator self-preference and correlated model errors cautions that a second model is not automatically an independent judge [12, 18]. These findings do not yet provide a joint account of state transitions, evidence provenance, verifier selective risk, verifier dependence, and stopping in one code-agent protocol. This paper asks five research questions:
RQ1 Is correctness an absorbing state under repeated revision? RQ2 From the same wrong or correct program, how do evidence content and state alignment change repair and continuation harm? RQ3 Can verifier quality or independence be inferred from modelfamily diversity? RQ4 Which orchestration guards directly block the observed failure classes, and what do they cost? RQ5 How do contracts and revision allowance affect repository completion?
• Software and its engineering → Software testing and debugging; • Computing methodologies → Artificial intelligence.
Keywords coding agents, agent reliability, code repair, verification, stopping policies, LLM evaluation
1
Introduction
Coding agents increasingly operate as loops: propose code, execute tests, consume feedback, revise, and stop. Iterative refinement can improve the probability that a correct solution is produced [5, 15, 22]. This search-centric view, however, obscures a deployment question: after a correct patch appears, what prevents the next revision from destroying it, a stale observation from reviving an old bug, or a correlated verifier from confirming the same error? We call this distinction proposal search versus completion reliability. Search asks whether any revision in a trajectory is correct. Completion reliability asks whether the orchestrator retains a correct state, bases revisions on evidence about that state, and stops
Our contributions are: (1) A trajectory-level reliability decomposition that jointly reports correctness discovery and retention, wrong-to-correct and correct-to-wrong transitions, verifier risk–coverage, conditional false-accept dependence, and sound completion. (2) Controlled evidence studies. Forced revision shows correctness loss; paired common-state interventions at two coder capacities estimate state-alignment contrasts from identical selected states. (3) An executable contract and reference implementation. We derive testable obligations for state-bound evidence, typed actions, checkpoint preservation, fresh completion certification, and risk-aware stopping. A reference implementation instantiates the mechanically enforceable subset as an admission layer around existing coding agents and verifiers. (4) Repository-agent factorials. Over 24 bugs, four coder stacks test a contract; two Devstral factorials separate components and depth. All use sealed adjudication and task-cluster inference.
AgenticDev 2026, October 12, 2026, Munich, Germany
2 Related Work 2.1 Self-Correction and Feedback-Driven Repair Huang et al. find that intrinsic self-correction does not reliably improve reasoning when no external signal is available [9]. Kamoi et al.’s survey similarly identifies reliable external feedback as a central success condition [11]. Tyen et al. show that models often fail to locate errors even when they can correct an error whose location is supplied [25]; Zhang et al. separate refinement ability from the strength of the verifier that decides when to refine [28]. In code, FeedbackEval evaluates feedback types and reports diminishing gains over repeated repair [5]. We do not claim the new observation that feedback quality matters. We instead measure the bidirectional transition process and intervene on whether evidence is attached to the program state it describes. Self-Refine and Reflexion exemplify inference-time refinement with generated feedback or memory [15, 22]. Their motivating metric is typically task success. Our focus is the reliability contract required after a proposal exists: preservation, provenance, verification, and stopping.
2.2
Agentic Software Engineering Systems
Repository-scale systems make the orchestration question concrete. SWE-agent shows that an agent-computer interface can materially change coding performance [27], while Agentless reports that a simpler localization–repair–validation pipeline can outperform more elaborate open-source agents on SWE-bench Lite [26]. SWE-bench, meanwhile, demonstrates the long-context and multi-file demands of real issue resolution [10]. These systems primarily compare endtask success and cost. We study a narrower but complementary systems property: whether the loop preserves a valid state, consumes evidence about that state, and applies a calibrated completion gate. Function-level interventions identify these mechanisms under controlled states; a separate repository-agent factorial tests their bundled system consequence in a real Git/search/edit/test loop. Recent process-level studies are closer to our question. Bouzenia and Pradel characterize feedback integration in repair-agent trajectories [1]; the “lucky pass” AgentLens identifies regression cycles [21], while a separate production-assessed AgentLens reviews verification and recovery over whole trajectories [19]; and PAIR-Bench controls hint depth while measuring preservation [13]. FixedBench shows action bias on already-correct tasks [7], while SpecBench exposes visible–held-out gaps [29]. EviACT coordinates retrieval, compile, and test-driven evidence guards [16]; Cambronero et al. combine abstention and validation [2]; and Mnemosyne separates proposals from transactional admission [3]. We do not claim evidence guardrails or transactional admission as novel. Our narrower contribution is to hold code state fixed while intervening on evidence provenance, connect transitions to selective verifier risk and dependence, and test policy consequences prospectively. Adjacent infrastructure enforces related but different boundaries. In-toto records and verifies software-supply-chain steps [24], SLSA provenance describes where, when, and how an artifact was produced [23], and OPA separates general policy decisions from their enforcement [17]. Our reference implementation adopts provenance and admission ideas but applies them inside an agent-repair transaction: verifier evidence is bound to the candidate code state
Gao et al.
on which it was produced, and admission preserves or recertifies that state. It neither provides supply-chain attestation nor repairs an inadequate oracle.
2.3
Code Correctness and Executable Evidence
HumanEval introduced execution-based functional correctness for generated programs [4]. EvalPlus expands its tests and demonstrates that a small visible suite can overestimate correctness [14]. We use disjoint visible, challenge, and hidden input sets derived from HumanEval+, while explicitly treating challenge and hidden tests as construction-dependent rather than independent oracles.
2.4
Verifier Risk and Dependence
LLM evaluators may recognize and favor their own generations [18]. More broadly, errors remain correlated across model architectures and providers [12]. We therefore distinguish two quantities: selective risk at a specified coverage and conditional error dependence among incorrect candidates. Risk–coverage follows selective prediction, where abstention is part of evaluation rather than silently discarded [6]. Our contribution is to connect these quantities to agent completion decisions and to show that dependence changes with the acceptance operating point.
3
Reliability Decomposition
Let 𝑌𝑡 ∈ {0, 1} denote the executable correctness of the code after revision 𝑡, and let 𝑒𝑡 denote the evidence supplied for the next revision. For a fixed evidence condition 𝑒, define 𝛼𝑒 = 𝑃 (𝑌𝑡 +1 = 1 | 𝑌𝑡 = 0, 𝑒),
(1)
𝛽𝑒 = 𝑃 (𝑌𝑡 +1 = 0 | 𝑌𝑡 = 1, 𝑒).
(2)
If 𝑞𝑡 = 𝑃 (𝑌𝑡 = 1), then 𝑞𝑡 +1 − 𝑞𝑡 = (1 − 𝑞𝑡 )𝛼𝑒 − 𝑞𝑡 𝛽𝑒 .
(3)
Repair ability alone is therefore insufficient: when the current state is likely correct, a nonzero regression rate can make another revision harmful in expectation. If 𝛼𝑒 + 𝛽𝑒 > 0, continuation has negative expected correctness change when 𝑞𝑡 > 𝛼𝑒 /(𝛼𝑒 + 𝛽𝑒 ). This transition view makes the paper’s three factors precise without pretending that all are separately identifiable. Let 𝑒 0 be a matched nonactionable control. We operationalize the effective evidence response on wrong and correct states as 𝑅𝑊 (𝑒) = 𝛼𝑒 − 𝛼𝑒0 ,
𝑅𝐶 (𝑒) = 𝛽𝑒0 − 𝛽𝑒 .
(4)
These contrasts combine two latent quantities: how much taskrelevant information the evidence carries and how responsively the coder maps that information into an edit. A single-coder evidence intervention identifies their joint effective action, not “informativeness” and “responsiveness” separately. Zero response can mean uninformative evidence or an unresponsive reviser; negative response can mean misleading or state-misaligned evidence, overreaction, or both. This identification limit is important: the three-factor thesis is an interface decomposition, not a fitted multiplicative law. Evidence also has provenance. Write 𝑒 = (ℎ𝑠 , ℎ𝑞 , 𝑥) for sourcestate hash, test-suite hash, and payload. The evidence is statealigned only if ℎ𝑠 = ℎ(𝑆𝑡 ). Payload 𝑥 may contain true input/output facts while the envelope is invalid for the current state. The
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
common-state study varies this alignment while holding the starting code fixed. For verifier 𝑣 with threshold chosen to attain coverage 𝑐, we report Cov𝑣 (𝑐) = 𝑃 (𝐴𝑣 = 1),
(5)
Risk𝑣 (𝑐) = 𝑃 (𝑌 = 0 | 𝐴𝑣 = 1),
(6)
where 𝐴𝑣 denotes acceptance. To avoid conflating shared prevalence with shared errors, dependence is computed only among incorrect candidates. For verifiers 𝑖, 𝑗 we report excess joint false acceptance, 𝐷𝑖 𝑗 (𝑐) = 𝑃 (𝐴𝑖 𝐴 𝑗 | 𝑌 = 0) − 𝑃 (𝐴𝑖 | 𝑌 = 0)𝑃 (𝐴 𝑗 | 𝑌 = 0),
(7)
and the corresponding 𝜙 coefficient. Both are operating-point conditional: changing coverage changes which errors are accepted. For an ensemble that accepts only when both verifiers accept, let 𝑓𝑖 = 𝑃 (𝐴𝑖 = 1 | 𝑌 = 0). Its conditional false-accept rate is 𝑃 (𝐴𝑖 𝐴 𝑗 = 1 | 𝑌 = 0) = 𝑓𝑖 𝑓 𝑗 + 𝐷𝑖 𝑗 .
Every revision is judged by native self-declaration, fresh-context Qwen2.5-7B, same-family Qwen2.5-14B, cross-family DeepSeekCoder-6.7B [8], and an executable challenge verifier. This produces 13,500 verdicts. Replaying four stopping rules for each verifier produces 18,000 paired outcomes: verifier-only, visible gate, challenge gate, and a strict gate requiring challenge acceptance plus evidence sourced from the immediately preceding state. The three model verifiers use an order-swapped, single-token A/B forced choice; the acceptance score is the mean ACCEPT probability under both label orders. Native thresholds are fixed at 0.5. Retrospective coverage matching ranks scores on the frozen revision distribution and accepts the top fraction; it compares verifiers but does not estimate deployment calibration. A stopping replay selects the first revision satisfying its gate; if none does, it rejects after revision 3. Because generation is frozen before replay, every policy sees the same candidates and executable outcomes. “Mean rounds” includes rejected trajectories, which consume the full three-revision budget.
(8)
Thus the familiar product-of-errors argument is valid only when 𝐷𝑖 𝑗 = 0. Positive dependence is an additive reliability penalty even if both marginal verifiers are individually strong; low dependence is not useful when either marginal risk is poor. Finally, sound completion is 𝑃 (𝐴 = 1, 𝑌 = 1) over all trajectories. Unlike risk alone, it exposes a policy that obtains zero observed error by rejecting nearly everything. We report coverage, risk, sound completion, and mean selected revision together rather than choosing a universal scalar utility.
4 Study Design 4.1 Tasks and Executable Adjudication We select 30 Python function-repair tasks from the HumanEval portion of FeedbackEval [5]. Initial wrong candidates are stratified by source (existing benchmark candidate, LLM-generated candidate, and rule-based mutation) and frozen by hash before the confirmatory runs. We do not redistribute the source benchmark data; the artifact records task identifiers, candidate indices, sources, and hashes. Correctness requires passing visible, challenge, and hidden tests. Challenge and hidden inputs are disjoint partitions of HumanEval+ [14]; they share the same benchmark construction process. Generation sees only visible evidence. Challenge and hidden suites remain sealed until all trajectory and branch matrices pass structural validation. Evaluation is cached by (suite, task, code SHA-256).
4.2
AgenticDev 2026, October 12, 2026, Munich, Germany
Trajectory Study
The coder is Qwen2.5-7B-Instruct [20], sampled with temperature 0.3 and top-𝑝 0.9. Each trajectory performs three forced revisions. We cross 30 tasks, six evidence conditions, and five seeds, yielding 900 trajectories and 2,700 revisions. The evidence conditions are current binary outcome, current execution trace, stale binary outcome, stale trace, unavailable evidence, and a flipped-binary stress condition. Forced revision intentionally estimates what happens when an orchestrator continues; it is not presented as the behavior of every adaptive agent.
4.3
Common-State Intervention
Transition rates in a free-running trajectory have a post-treatment risk-set problem: evidence affects whether a revision reaches a correct state, so comparing later regressions conditions on an evidencedependent event. We address this with common-state intervention. For each task we freeze one wrong and one correct code state, selected before assigning the new evidence branches. Twenty-seven of 30 tasks contain both states and enter this study; the estimand therefore concerns tasks with an observed repairable state. From every frozen wrong state we branch current binary, current trace, unavailable, and length-matched nonactionable evidence. From every correct state we branch the same conditions plus a stale trace generated from the paired wrong state. The prompt permits returning unchanged code. Crossing 27 tasks, nine state– evidence branches, and five common-random-number seeds gives 1,215 single-step branches for Qwen2.5-7B. Before inspecting replication outcomes, we freeze an exact Qwen2.5-14B repeat with the same states, evidence, prompts, sampling parameters, seeds, and adjudication, adding 1,215 branches. The two capacities are analyzed separately rather than pooled. The frozen wrong state is the original visible-failing candidate. The frozen correct state is selected mechanically as the lexicographically first v0.4.3 revision passing all three suites; no failure case is hand-picked. The matched control contains no task, test, PASS/FAIL, or numerical semantics and exactly matches the target report in Unicode character count. It does not exactly match tokenizer length, so prompt-token distributions remain an audited nuisance variable. Within each task–state–seed block, evidence branches share the sampling seed; invalid or unchanged outputs remain in the intention-to-treat denominator.
4.4
Inference and Audit
The task, not the revision, is the independent sampling unit. Confirmatory common-state contrasts average seeds within each task and then weight tasks equally. Within each coder, we use 10,000 task-cluster bootstrap draws, 100,000 paired sign flips, and one Holm correction over six preregistered contrasts. Because five-seed
AgenticDev 2026, October 12, 2026, Munich, Germany
Gao et al.
Table 1: Evidence interventions. Current evidence is generated from the code being revised; stale evidence is generated from a different revision or the frozen wrong state. Condition
Payload
Source state
Role
Current binary Current trace Stale binary/trace Unavailable Flipped binary Matched control Stale wrong trace
Visible PASS/FAIL Outcome, pass count, up to 3 failures Same formats UNAVAILABLE Negated visible PASS/FAIL Neutral text, character-length matched Failure trace
Current code Current code Previous revision None Current code None Frozen wrong code
Minimal aligned signal Actionable aligned signal Free-running freshness stress No-evidence baseline Misleading-signal stress Report-length/attention control State-misalignment intervention
Table 2: Confirmatory study inventory. Component Tasks Seeds Coder capacities Evidence conditions Revisions per branch Trajectories/branches Candidate revisions Verifier verdicts Policy outcomes
Trajectory study
Common-state study
30 5 1 6 3 900 2,700 13,500 18,000
27 5 2 4 wrong / 5 correct 1 2,430 2,430 — —
task effects lie on a discrete grid, an exact dynamic-programming sign-flip sensitivity removes Monte Carlo error. A secondary paired analysis subtracts 7B task-level effects from 14B effects and applies a separate six-test Holm family; it was specified before inspecting 14B outcomes but is not a randomized model-family comparison. Trajectory confidence intervals also resample complete task clusters, preserving all seeds, conditions, revisions, and policies. Invalid generations remain in the intention-to-treat denominator. All protocol, data-partition, trajectory, verdict, and analysis artifacts carry SHA-256 lineage hashes. Generation processes can read only the visible bundle. Challenge and hidden bundles are opened only after trajectory IDs, condition matrices, hashes, parser outputs, and lineage fingerprints validate exactly. Hidden adjudication then recomputes the complete verifier and policy matrices. This ordering prevents a partial run or adaptive policy choice from leaking hidden outcomes back into generation.
4.5
Prospective Paired Online Commit-Admission Comparison
Replay can show whether a guard would intercept recorded outputs, but not how enforcing it changes subsequent model behavior. We therefore run a sealed paired online comparison on the 27 commonstate tasks: two start states, five seeds, and baseline/guarded arms yield 540 rollouts. Both arms use Qwen2.5-7B, the same typed action prompt, visible diagnostics, challenge executions, seed schedule, and three-step budget. Baseline commits every parseable patch and ignores challenge verdicts for admission. Guarded execution binds evidence to the current state, admits a patch only after the challenge gate, and retains a last-known-good checkpoint after invalid or rejected proposals. Hidden outcomes are opened only
after the complete paired matrix validates. The preregistered endpoints are unsafe-completion superiority and sound-completion non-inferiority at a −5-point margin; inference uses paired taskcluster bootstrap intervals.
4.6
Repository-Agent Factorial
To test the contract beyond single-function repair, we freeze 24 historical bugs from nine Python projects, with at most three tasks per project. Eligibility requires direct-parent bug/fix endpoints, stable three-repeat signatures across the frozen suites, no ignored oracle, and a gold production patch of at most five files and 200 changed lines. The primary-only manifest excludes pilot tasks, is frozen before model output, and permits no outcome-driven replacement. Each task has a sanitized agent mirror containing only the buggy tree and a separate evaluator mirror containing sealed tests but no gold source object. A hash-locked local Qwen2.5-14B snapshot operates through typed search, view, edit, shell, test, submit, and escalate actions in a real Git worktree, sampled at temperature 0.3, top-𝑝 0.9, and at most 1,024 new tokens per action. We cross current versus one-step-stale diagnostic evidence with contract OFF versus ON and repeat every cell over three preregistered seeds, yielding 24×2×2×3 = 288 intention-to-treat rollouts. Each rollout is limited to 15 actions, three source revisions, six visible tests, and a shared wall-clock budget. Contract ON bundles evidence–state binding, a private commit gate, rejection with rollback, last-known-good checkpoints, and completion certification. Contract OFF executes and records the same private gate but does not use its verdict to block a revision. Diagnostic, gate, fail-to-pass, pass-to-pass, and full-suite assets are content-hashed; generation can access only the diagnostic interface. Every proposed patch, including rejected proposals, is preserved and later replayed in an evaluator-owned worktree. Before unsealing, a 24-task by four-suite baseline audit found residue only from terminal full-suite runs; the frozen evaluator orders failto-pass, pass-to-pass, then full, and hard-resets and cleans before each candidate. Final resolution requires all three suites to pass. An orchestrator-accepted completion is either an explicit agent submission or, in the ON arm, a checkpoint accepted by the frozen completion rule. Unsafe and sound completion are accepted-andincorrect and accepted-and-correct, respectively. The task is the inferential unit: seeds are averaged within task and all 24 tasks receive equal weight. The co-primary intersection– union decision requires the one-sided 95% upper bound on the Contract ON-minus-OFF unsafe-completion effect to be below zero
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
and the lower bound on sound completion to exceed the preregistered absolute −5-point margin. This margin was frozen before outcomes; when baseline sound completion later falls below 5%, its non-inferiority decision is necessarily weak evidence about liveness. We use 200,000 task-cluster bootstrap draws; task sign flips, project-equal estimates, and leave-one-project-out checks are sensitivities. To probe stack generality under the 14B completion floor, we froze three separate 288-rollout replications before their outcomes: Qwen2.5-Coder-32B-Instruct-AWQ, Qwen3-Coder-Next-FP8, and Devstral Small 2 24B. Each preserves the task matrix, co-primary estimands, budgets, completion rule, and task-equal inference, and is analyzed separately. Model, tokenizer, adapter, and runtime changes remain bundled ecological contrasts, not identified capacity or family effects. Outcome-blind runtime or adapter faults were repaired before restarting empty main roots; discarded attempts remain receipted and no scientific design changed. With Devstral fixed, a 576-rollout 23 factorial randomizes state binding, proposal admission/rollback, and completion certification. Task-equal main effects enter one six-test Holm family over unsafe and sound completion. A separate 576-rollout equal-budget factorial crosses Contract OFF/ON with revision limits 1, 3, 5, and 8 while holding action, test, token, and wall budgets fixed; its six-test family covers depth, contract, and interaction effects. Full stack, runtime, pilot, and sensitivity specifications are in the reproduction artifact. All reported experiments use the frozen research harness described above. The reference implementation was developed afterward as an executable realization of the mechanically enforceable contract obligations and was not used to generate the reported empirical results.
5 Results 5.1 RQ1: Correctness Is Non-Absorbing Figure 1b separates current correctness from cumulative discovery. With current traces, 82.0% of trajectories are correct after revision 1. Current correctness then falls to 67.3% at revision 2 and recovers only to 69.3% at revision 3. In contrast, ever-correct increases from 82.0% to 84.7% and 85.3%. At revision 3, 16.0% of trajectories have produced a correct patch and subsequently lost it. Current binary evidence has lower loss (8.0%) but still does not make correctness absorbing. The shape is evidence-specific rather than a generic depth penalty. Current-binary correctness is 0.700, 0.700, 0.720 across revisions; stale-trace correctness is 0.687, 0.713, 0.660; and unavailable evidence is 0.687, 0.687, 0.707. Consequently, neither “more rounds help” nor “more rounds hurt” is supported as a context-free rule. Loop depth changes the number of transition opportunities; the transition kernel determines their sign. This result is deliberately scoped: it establishes non-absorption under the tested forced-revision protocol. It does not show that an adaptive loop must regress. It shows why an adaptive loop needs a preservation and stopping rule.
AgenticDev 2026, October 12, 2026, Munich, Germany
Table 3: Common-state repair and continuation harm. Each cell is event count/135; starts and assignments are identical across coders.
5.2
Start
Evidence
7B event
14B event
Wrong Wrong Wrong Wrong
Current binary Current trace Unavailable Matched control
105/135 113/135 97/135 105/135
115/135 120/135 126/135 119/135
Correct Correct Correct Correct Correct
Current binary Current trace Unavailable Stale wrong trace Matched control
0/135 5/135 0/135 12/135 2/135
4/135 4/135 10/135 34/135 17/135
RQ2: Evidence Helps Repair but Can Harm Retention
Table 3 and Figure 2 report assignment-based common-state outcomes. For 7B, current traces repair 113/135 wrong starts (0.837), compared with 105/135 (0.778) for both binary evidence and the matched control, and 97/135 (0.719) when evidence is unavailable. Trace minus control is +5.9 points with task-cluster 95% CI [−2.2, 17.0]. From correct starts, stale traces harm 12/135 versus 5/135 under current traces and 2/135 under control. All six 7B contrasts have Holm-adjusted 𝑝 = 1.0. The 14B replication changes the transition kernel. Unavailable evidence repairs 126/135 wrong starts (0.933), more than trace (120/135), binary (115/135), or control (119/135). Thus additional task feedback is not monotonically useful to the stronger coder. More importantly, stale traces harm 34/135 correct starts (0.252), versus 4/135 (0.030) under current traces. The +22.2-point contrast has CI [8.9, 37.0] and remains significant across the six-test family (exact Holm 𝑝 = 0.0337). Stale minus control is +12.6 points, CI [0.7, 25.9], but its sign-flip test does not survive Holm (𝑝 = 0.3934). The control’s 0.126 harm shows that unsolicited long context and editing pressure explain part, but not all, of the stale-current gap. The paired coder interaction for stale minus current harm is +17.0 points, CI [0.7, 33.4], but is only suggestive after its separate Holm correction (𝑝 = 0.3615). We therefore do not claim a general capacity effect. The result does establish that evidence effects cannot be treated as properties of payload alone: under identical tasks and state/evidence assignments, revision behavior changes materially with the coder. Secondary outcomes separate editing from useful revision. For 14B correct starts, stale traces change code in 0.815 of branches, compared with 0.474 under current traces and 0.778 under control; the respective harm rates are 0.252, 0.030, and 0.126. Exact-code churn is therefore neither repair nor semantic harm, but it exposes a correct state to risk. Prompt-token means are 498.4 for stale traces and 451.7 for control under both coders, so character matching reduces but does not eliminate attention-budget differences.
5.3
Stale Evidence Resurrects Fixed Bugs
Average uncertainty does not erase constructive counterexamples. For 7B, all correct-state harms occur in 4/27 tasks (Figure 3); stale
AgenticDev 2026, October 12, 2026, Munich, Germany
Gao et al.
(b) Correctness is not monotone 0.85
0.25 0.20 0.15 0.10 Qwen 7B Qwen 14B DeepSeek 6.7B
0.05
0.60
0.65
0.70 0.75 0.80 Selective coverage
0.85
0.90
Correct at current revision
Risk among accepted revisions
(a) Verifier risk-coverage
Current binary Current trace Stale trace Unavailable
0.80 0.75 0.70 0.65 0.60
1
2 Forced revisions
3
Figure 1: Left: retrospective selective risk at matched coverage. Cross-family diversity does not ensure verifier quality. Right: current correctness across forced revisions is non-monotone even while cumulative ever-correct increases. traces regress HumanEval/10 and /89 in all five seeds. In /89, a stale report from code using Caesar shift 12 changes the fixed shift 4 back to 12 in every seed. The report contains valid input/expected-output facts, but falsely asserts that they describe the current code. For 14B, stale harm is broader: 34 branches across 12/27 task clusters. HumanEval/10 and /46 regress in 5/5 stale seeds while their matched controls harm 0/5 and 1/5, respectively. Conversely, /89 no longer regresses, and /50 and /120 fail under both stale and control prompts. This redistribution is important: stale evidence is a reproducible failure mechanism, not a deterministic property of a task, and responsiveness determines whether the mechanism fires. This yields a sharper mechanism than “more information helps”: utility depends jointly on payload, provenance, and the coder’s response. Candidate-source diagnostics also shift across coders and were not randomized, so they are descriptive rather than subgroup effects. The clustering and redistribution explain why task-level, not branch-level, uncertainty is essential.
5.4
RQ3: Verifier Quality Is Not Independence
At native thresholds, Qwen-7B covers 0.757 of revisions at risk 0.095; Qwen-14B covers 0.711 at risk 0.040; DeepSeek covers 0.991 at risk 0.298. Coverage confounds direct comparison, so Figure 1a imposes retrospective matched coverage. At 70%, risks are 0.050, 0.041, and 0.240, respectively. Qwen-14B minus DeepSeek is −19.9 points with task-bootstrap CI [−26.4, −9.5] when each bootstrap draw refits the cutoff. This compares the frozen distribution; deployment requires held-out calibration. This ordering is not an artifact of choosing 70% coverage: Qwen14B minus DeepSeek risk differences remain negative from 60% through 90% coverage, ranging from −20.5 to −4.0 points, with all five task-bootstrap intervals below zero. Repeated code states do affect the magnitude. Collapsing 2,700 revisions to 617 unique task– code hashes raises 70%-coverage risks to 0.400, 0.387, and 0.472; Qwen-14B still has lower empirical AURC (0.225 versus 0.402) and 8.5 points lower risk than DeepSeek. We therefore interpret 19.9
Table 4: Verifier quality and conditional dependence. Risk is at 70% coverage; 𝜙 is conditional on incorrect revisions. Verifier/pair
Risk
Conditional 𝜙
Qwen-7B Qwen-14B DeepSeek-6.7B
0.050 0.041 0.240
— — —
— — —
0.641 −0.072 0.021
Qwen-7B / Qwen-14B Qwen-7B / DeepSeek Qwen-14B / DeepSeek
points as the revision-distribution estimate, not an intrinsic model constant. Quality also does not imply independent errors. Among hiddenincorrect revisions at 70% coverage, Qwen-7B/Qwen-14B have 𝜙 = 0.641 with task-cluster CI [0.096, 0.935] and excess joint false acceptance 0.060 [0.001, 0.133]. Qwen-14B/DeepSeek have 𝜙 = 0.021 [−0.231, 0.272] at the same coverage, but DeepSeek’s risk is much higher. At 75% coverage, all three scored pairs have positive excessjoint intervals. Dependence is thus an operating-point property, not a static label attached to a pair of model families.
5.5
RQ4: Bundled Admission Changes Retention and Repair
Table 5 evaluates stopping and verification on the same 900 trajectories. Self-only stopping accepts 99.8% but has 29.5% risk and 70.3% sound completion. A visible gate prevents 248 false completions and introduces none, reducing risk to 2.4% while increasing sound completion to 77.8%. The challenge gate prevents all 265 self-only false completions in this sample and attains 77.8% sound completion at 77.8% coverage. Its zero observed errors do not imply zero population risk: the conservative one-sided 95% zero-event bound over 30 task clusters is 9.5%.
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
100%
Repair rate
5
78%
84%
0
HumanEval/10
(a) Frozen wrong state 85%
80%
Qwen2.5-14B
60%
0
0
0
5
0
HumanEval/113
0
0
0
1
0
HumanEval/137
0
5 (5 invalid)
0
1
2
e
ne
le
l tro Con
20%
Binary
Continuation harm
3
2
1
0
y
ar Bin
0% Trace
None
20% 15%
12.6% 7.4%
8.9%
3.0% 3.7%3.0%
5% 0%
Binary
None
Sta
Table 5: Stopping-policy replay over 900 fixed trajectories. CIs are task-clustered; paired changes are reported in text.
1.5%
0%
Trace
No
Figure 3: 7B correct-to-incorrect outcomes are clustered, not uniform. HumanEval/10 and /89 show stale-evidence bug resurrection in 5/5 seeds; current-trace failures on /137 are invalid action-schema outputs.
25.2%
10%
c Tra
Control
(b) Frozen correct state
25%
0%
0
HumanEval/89
40%
30%
5
88% 78%
72%
0
4
93%
89%
0
Harm seeds
Qwen2.5-7B
AgenticDev 2026, October 12, 2026, Munich, Germany
Stale
Control
Figure 2: Assignment outcomes from identical frozen states. Evidence ordering changes across coder capacities; 14B stale evidence produces the largest continuation harm. Inference is reported in text.
Task-cluster uncertainty is wide enough to matter. Self-only risk has a 95% interval of [0.154, 0.445], visible-gated risk [0, 0.077], and visible-gated sound completion [0.642, 0.900]. The zero-event task bound is 1 − 0.051/30 = 0.095; treating 1,877 challenge-passing revisions as independent would instead give 0.0016 and would be anti-conservative for cross-task generalization. The empirical result supports executable gating in this domain, not a universal safety guarantee. Strict freshness is not free. Qwen-14B plus a current-evidence challenge has no observed false completion but covers only 38.4%, loses 311 baseline sound completions, and requires 2.27 mean revisions. Reliability evaluation must expose this abstention/cost trade-off. The prospective paired comparison estimates the bundled policy’s orchestrator-level effect. From correct starts, baseline continuation harms 13/135 branches and finishes correct on 122/135; guarded execution has no observed harm and retains correctness on 135/135. The −9.6-point harm contrast has a task-bootstrap 95% CI of [−20.7, 0.0] and sign-flip 𝑝 = 0.25, because all baseline harms occur in only 3/27 task clusters. From wrong starts, repair falls from 100/135 to 93/135. Across both start states, unsafe completion changes from 1/270 to 0/270 (effect −0.4 points, CI [−1.1, 0.0]),
Policy
Cov.
Risk
Sound
Rounds
Self only Self + visible Self + challenge Qwen-14B only Qwen-14B + challenge Challenge executable Qwen-14B + fresh challenge
0.998 0.797 0.778 0.794 0.769 0.788 0.384
0.295 0.024 0.000 0.046 0.000 0.000 0.000
0.703 0.778 0.778 0.758 0.769 0.788 0.384
1.03 1.55 1.56 1.52 1.55 1.52 2.27
which is too sparse to demonstrate superiority. Sound completion changes from 219/270 to 211/270 (effect −3.0 points, CI [−8.5, 1.1]), so the preregistered −5-point non-inferiority criterion is not met. The bundled guard therefore shifts observed errors while reducing both revision plasticity and completion coverage; these data do not identify any component’s individual effect.
5.6
RQ5: Repository Contracts Expose Completion-Gate Dependence
The sealed repository factorial exhibits a severe completion floor. Contract OFF produces five orchestrator-accepted completions: current evidence yields 3/72 (two unsafe and one sound), and stale evidence yields 2/72 (both unsafe). Contract ON produces 0/72 accepted completions in each evidence arm. Thus the contract removes every observed unsafe completion, but it also removes the only observed sound completion. Only one rollout discovers, admits, retains, and submits a fully resolved revision; 276/288 rollouts (95.8%) exhaust the 15-action budget. The study therefore tests completion control under a weak repair process, not repository-level repair-rate improvement. Figure 4 reports the frozen task-equal analysis. The contract effect on unsafe completion is −2.78 points (task-cluster 95% CI [−8.33, 0.00]), with a one-sided upper bound of exactly zero, missing strict superiority. The sound-completion effect is −0.69 points (95% CI [−2.08, 0.00]), whose lower bound passes the −5-point margin.
AgenticDev 2026, October 12, 2026, Munich, Germany
Gao et al.
Co-primary repository-agent effects
-2.78 pp [-8.33, +0.00]
Unsafe completion
-0.69 pp [-2.08, +0.00]
Sound completion sound NI margin (-5 pp)
−10.0
−7.5
−5.0 −2.5 0.0 2.5 Contract effect (ON minus OFF), percentage points
5.0
Figure 4: Prespecified repository-agent contract effects (ON minus OFF). Bars are task-cluster bootstrap 95% intervals over 24 task-level effects; the three seeds per cell are averaged before inference. Unsafe-completion superiority requires its one-sided upper bound below zero; sound-completion non-inferiority requires its lower bound above −5 points. Only the latter, floorlimited criterion is met, so the joint IUT fails. Because OFF-arm sound completion is only 1/144, this pass is weak evidence about liveness; the joint decision is negative. Zero ON-arm unsafe events are not a population guarantee. Secondary analyses support an abstention interpretation: the accepted-completion effect is −3.47 points and remains non-positive in every leave-one-project-out analysis, but only five events determine it. Ninety of 288 rollouts contain invalid actions, confirming interface friction. Full gate, action, cost, and sensitivity results are in the artifact. The contract suppresses unsupported completion; it does not create revision competence or liveness. Independent capable-coder replication. The complete 288-rollout replication yielded accepted/unsafe/sound counts of 2/0/2, 5/3/2, 1/0/1, and 4/3/1 in the current/OFF, current/ON, stale/OFF, and stale/ON cells, respectively (72 rollouts per cell). The contract effect on unsafe completion was +4.17 points (task-cluster 95% CI [0.00, 11.11]; one-sided upper bound 9.72) and the effect on sound completion was 0.00 points (95% CI [−2.08, 2.08]; one-sided lower bound −1.39). Unsafe-completion superiority was therefore not established, sound-completion non-inferiority at the prespecified −5point margin was established, and the joint intersection–union decision failed. The six unsafe completions were concentrated in two tasks (two-sided task sign-flip 𝑝 = 0.50), although the unsafe effect remained positive in every leave-one-project-out analysis. Mechanistically, the gate false-passed 7/237 incorrect proposal commits and all 3 incorrect explicit submissions; the terminal-checkpoint rule certified four checkpoints, three of them incorrect. Thus a completion gate can propagate its own errors when it does not provide sufficiently independent correctness evidence. Newer-generation temporal replication. A separately frozen Qwen3-Coder-Next-FP8 matrix completed 288 rollouts without infrastructure failure. Accepted/unsafe/sound counts were 0/0/0, 2/0/2, 0/0/0, and 4/0/4 for current/OFF, current/ON, stale/OFF, and stale/ON. The empirical unsafe-completion effect was zero (task-cluster 95% CI [0, 0]), which neither establishes population safety nor strict superiority. The sound-completion effect was +4.17 points (95% CI [0.00, 11.11]; one-sided lower bound 0.00),
passing non-inferiority but not the joint decision; only two tasks support it (sign-flip 𝑝 = 0.50). Contract ON converted five retained checkpoints into sound automatic completions, but 282/288 rollouts exhausted the action budget and 202/288 encountered invalid actions. One proposal-gate and one terminal false pass produced no unsafe completion. This is stack-level heterogeneity, not an identified model-family or capacity effect. Cross-family Devstral replication. A separately frozen Devstral Small 2 24B matrix completed 288 rollouts without infrastructure failure. Accepted/unsafe/sound counts were 0/0/0, 2/0/2, 0/0/0, and 2/0/2 for current/OFF, current/ON, stale/OFF, and stale/ON. The contract effect on unsafe completion was exactly zero (taskcluster 95% CI [0, 0]), so strict superiority was not established. Its sound-completion effect was +2.78 points (95% CI [0.00, 7.64]; onesided lower bound 0.00), passing non-inferiority but not the joint decision; two tasks support the estimate (two-sided sign-flip 𝑝 = 0.50). All eight discovered resolutions were admitted and retained, but only Contract ON certified four as sound completions. The frozen gate made no observed error over 307 proposal, submission, and terminal decisions. Nevertheless, 284/288 rollouts exhausted the action budget, 148/288 encountered invalid actions, and 230/288 triggered the typed-test deadline. This is a cross-family stack replication, not an identified architecture or capacity effect. Across these matrices, non-inferiority passes are floor-limited because OFF-arm sound-completion rates are 0–2.1%; they do not establish liveness. Randomized contract-component ablation. A separately frozen 2× 2 × 2 ablation crosses state binding (B), proposal admission/rollback (A), and completion certification (C) on the same Devstral checkpoint, 24 tasks, and three seeds (576 rollouts). No main effect survives the preregistered six-test Holm family. Certification has the largest descriptive sound-completion effect: +6.25 points (taskcluster 95% CI [+0.69, +13.19]; task sign-flip 𝑝 = 0.125; Holm 𝑝 = 0.753). Its unsafe effect is +0.69 points ([0.00, +2.08]; Holm 𝑝 = 1.0), supported by one task. State binding changes unsafe completion by −0.69 points ([−2.08, 0.00]; Holm 𝑝 = 1.0); proposal
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
admission changes it by 0.00 points. Only four tasks support a nonzero certification effect on sound completion, so the experiment separates policy mechanics but does not establish a general component benefit. Here 𝐶 = 0 removes the checkpoint completion path, so certification’s descriptive completion contrast partly reflects endpoint mechanics rather than an isolated semantic verification benefit. Equal-budget loop-depth factorial. A frozen Devstral 4 × 2 depth– contract factorial completes 576 rollouts under equal 15-action budgets: unsafe completion is 0/576 and sound completion is 25/576. The fitted cap-1-to-cap-8 sound effect is −0.23 points (95% CI [−1.77, +1.09]; Holm 𝑝 = 1.0); caps rarely bind, so nominal depth shows no detectable benefit in this sparse-proposal process.
6
Evidence-Bound Typed Loop Contract
The observed failures motivate an auditable orchestration specification, not a universally validated remedy or a new prompting recipe. Its four interfaces separate obligations that the negative policy and factorial decisions show must be evaluated rather than assumed. State-bound evidence. Every observation is carried in an envelope 𝐸 = (ℎ𝑐𝑜𝑑𝑒 , ℎ𝑠𝑢𝑖𝑡𝑒 , 𝑒𝑥𝑒𝑐𝑢𝑡𝑖𝑜𝑛_𝑖𝑑, 𝑝𝑎𝑦𝑙𝑜𝑎𝑑). The orchestrator may use 𝐸 to revise only when ℎ𝑐𝑜𝑑𝑒 equals the current state hash. A mismatch triggers refresh, Keep, or Escalate; it never directly triggers a patch. Typed revision actions. Free-form text conflates “no edit is needed” with “the attempted patch was unparsable.” We use 𝑅𝑒𝑣𝑖𝑠𝑖𝑜𝑛𝐴𝑐𝑡𝑖𝑜𝑛 = Keep | Patch(𝑐𝑜𝑑𝑒) | Escalate(𝑟𝑒𝑎𝑠𝑜𝑛). Only a parsed, executable Patch may replace the current state. In the 7B common-state study, all five current-trace harms on HumanEval/137 are responses containing only STATUS: DONE without code. A fail-closed parser that maps this action to Keep, rather than treating it as an invalid patch, would retain the correct checkpoint in these recorded branches. Last-known-good checkpoint. The ledger records code and evidence hashes for the last state that passed the configured gate. Invalid actions never overwrite it. This makes correctness absorbing with respect to parser and protocol failures even when model behavior is not. Risk- and dependence-aware stopping. Verifier selection uses risk at the intended coverage and conditional dependence, not model diversity as a proxy. Our matched-coverage estimates are retrospective; deployment additionally requires held-out calibration and drift monitoring. A completion decision records its operating point so residual risk remains auditable.
6.1
Operational Semantics
The contract turns a permissive text loop into a small state machine. At step 𝑡, the orchestrator stores current state 𝑆𝑡 , last-known-good state 𝐿𝑡 (possibly empty), and an append-only evidence ledger. It executes the following transaction: (1) run a named suite on 𝑆𝑡 ; append evidence bound to the state and suite;
AgenticDev 2026, October 12, 2026, Munich, Germany
(2) reject or refresh 𝐸𝑡 if either hash does not match the requested state and suite; (3) request one typed action; Keep leaves state unchanged, Escalate abstains, and only a parseable Patch creates candidate 𝑆𝑡′ ; (4) evaluate 𝑆𝑡′ with the configured commit gate; on failure retain 𝑆𝑡 or roll back to 𝐿𝑡 , never overwrite 𝐿𝑡 ; (5) stop only if the completion policy accepts the committed state at its declared risk–coverage operating point; otherwise continue within budget. Two safety properties follow mechanically. First, a state-hash mismatch cannot trigger a commit. Second, parser or action-schema failure cannot erase a checkpoint. Neither property claims that a hash-matched patch is semantically correct; that residual risk belongs to the executable and learned verifiers.
6.2
StateSeal Reference Implementation and Conformance
StateSeal, the reference implementation, wraps existing agents and project-defined verifiers. An agent proposes in an isolated worktree; the admission broker recomputes state identity, runs the verifier outside the agent process, preserves a checkpoint, and recertifies it before issuing a digest-bearing receipt. It records coverage and delivery impact but does not learn risk or dependence. At the frozen artifact snapshot, all 35 deterministic failureinjection cases pass, as do managed Go, Python, and Node.js fixtures (3/3). This is conformance evidence, not an effectiveness estimate, competence gain, verifier-independence result, or validation of the study. The StateSeal artifact snapshot provides source, commands, and the case catalog. Deterministic component replay remains useful for diagnosis. State binding intercepts all 12 and 34 stale-condition harms for 7B and 14B; typed action plus state binding reduces total harm from 19 → 2 and 69 → 33. A challenge checkpoint blocks all 19 7B harms but leaves two 14B regressions that pass visible and challenge tests while failing hidden tests. All replayed repairs are retained only because stale evidence is assigned to correct starts. The prospective loss of seven repairs confirms that replay does not estimate end-to-end utility.
7
Discussion
Controlled counterexamples. Search progress does not ensure retention; valid reports become harmful under state mismatch; and model-family diversity ensures neither low verifier risk nor independent errors. Iteration cannot substitute for evidence, state management, or verification. What they do not refute. Loops remain useful when expected repair exceeds continuation harm. We critique unconditional continuation and unqualified completion, not adaptive search: depth samples transitions, while evidence and orchestration determine whether value accumulates. Why the three factors are not interchangeable. Better feedback cannot help an unresponsive coder; responsiveness can amplify stale evidence; and proposal quality cannot offset correlated false
AgenticDev 2026, October 12, 2026, Munich, Germany
Gao et al.
Table 6: Contract obligations, reference mechanisms, conformance cases, and residual risks. Obligation
Reference mechanism
Conformance evidence
Residual risk
State-bound evidence
Bind candidate, tree, policy, verifier, and context identities Broker alone advances verified state and issues decisions Retain and exactly restore the last verified candidate
Reject replay, mutation, and wrongtree evidence (SB001–SB007) Self-report cannot authorize admission (SB021, SB027) Regressions and malformed actions cannot overwrite it (SB008–SB010, SB014, SB016) Failed recertification cannot complete (SB012, SB020, SB031) False pass/rejection, abstention, and coverage collapse are explicit (SB028–SB035)
Aligned evidence may be wrong; host and verifier remain trusted Process separation does not imply error independence Only oracle-covered properties are preserved
Independent admission Checkpoint preservation
Fresh certification Auditable stopping
Re-execute checks on the selected completion state Typed outcomes, hash-linked events, digestbearing receipt, coverage and liveness fields
acceptance. Equation 8 requires both low marginal false acceptance and low dependence. Safety, plasticity, liveness, and interface fit are separate. The prospective policy prevents observed regression but rejects useful repairs; repository contracts similarly trade unsupported completion against sparse sound completion. Neither randomized factorial yields a Holm-significant effect. Admission, preservation, certification, action-language fit, and depth remain distinct obligations. Implications for agent evaluation. Pass@1 or best-of-𝑘 misses state loss, incorrect acceptance, and low risk achieved by abstention. Iterative-agent evaluations should additionally report current versus ever-correct, common-state transition rates, coverage–risk, sound completion, cost, and conditional verifier overlap. Our 24 selected repository bugs test mechanisms, not population prevalence. Mechanism, policy, and calibration. State binding, transition guards, checkpoint restoration, and fresh recertification can be enforced mechanically. Verifier adequacy, error dependence, acceptable residual risk, and the safety–liveness trade-off remain empirical policy questions. The implementation makes failure modes falsifiable obligations; it does not improve agent or oracle competence.
8
Threats to Validity
External validity. The trajectory and common-state studies use public HumanEval tasks that may occur in model training data, and 27 tasks are selected for 7B-observed repairability. Pairing strengthens assignment contrasts but not absolute-rate generalization. The repository study covers 24 bugs from nine Python projects, four coder stacks, and one scaffold. Stack factors vary jointly, interface friction is material, and tasks within projects may be correlated; these are mechanism tests, not random samples of agents, languages, projects, or bugs. Construct validity. Challenge and hidden inputs are disjoint but share HumanEval+ construction, and no finite repository suite is a complete specification. One-step staleness is an intervention, not its natural prevalence; exact-code change is not semantic churn. Verifier family and capacity are confounded. Self acceptance is declarative, whereas repository acceptance also includes the ON arm’s frozen checkpoint rule.
Flakiness, drift, and suite incompleteness remain Auditability is not calibration, liveness, or attestation
Internal and statistical validity. Forced revision is not a natural adaptive policy. Common states address post-treatment risk-set bias, but all six 7B and five of six 14B contrasts are nonsignificant after Holm correction; no secondary cross-coder interaction survives correction. The original repository contract identifies only a joint effect; component randomization remains event-sparse, and the depth caps rarely bind. The four stack matrices have only 5, 12, 6, and 4 accepted completions; under this floor, the absolute −5point non-inferiority margin weakly constrains liveness. Task-equal inference has project-equal and leave-one-project-out checks, but only nine projects support project-level generalization. Reproducibility. Prompts, private suites, and full lineage are retained internally; licensing and evaluator privacy prevent rawtrajectory adjudication during review. The post-study implementation artifact tests mechanism conformance, not contract effectiveness, verifier adequacy or independence, or trust in the host, runner, Git, and OS. Strict admission can reject useful repairs; its UI is outside the trusted boundary.
9
Conclusion
Iteration supplies proposals, not reliability. Evaluation must separate state alignment, admission, preservation, grounded completion, competence, liveness, and interface fit. The typed contract makes these obligations explicit; its reference implementation enforces state binding, checkpoints, fresh certification, and auditable decisions. Verifier quality, dependence calibration, and general reliability remain empirical. Availability and disclosures. The numerical package supports reconstruction and hash auditing; private suites and licensed source are withheld. The StateSeal implementation artifact is at https: //anonymous.4open.science/r/agentic-reliability-artifact-8622/. AIassisted code and prose were author-verified.
References [1] Islem Bouzenia and Michael Pradel. 2025. Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories. In 40th IEEE/ACM International Conference on Automated Software Engineering. IEEE, 2846–2857. doi:10.1109/ASE63991.2025.00234 [2] José Pablo Cambronero, Michele Tufano, Sherry Shi, Renyao Wei, Grant Uy, Runxiang Cheng, Chin-Jung Liu, Shiying Pan, Satish Chandra, and Patrick Rondon. 2026. Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice. Association for Computing Machinery, 118–129. doi:10.1145/3786583.3786858
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
[3] Edward Y. Chang, Longling Geng, and Emily J. Chang. 2026. Mnemosyne: Agentic Transaction Processing for Validating and Repairing AI-Generated Workflows. arXiv preprint arXiv:2607.00269 (2026). doi:10.48550/arXiv.2607.00269 [4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). doi:10. 48550/arXiv.2107.03374 [5] Dekun Dai, MingWei Liu, Anji Li, Jialun Cao, Yanlin Wang, Chong Wang, Xin Peng, and Zibin Zheng. 2025. FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks. arXiv preprint arXiv:2504.06939 (2025). doi:10.48550/arXiv.2504.06939 [6] Yonatan Geifman and Ran El-Yaniv. 2019. SelectiveNet: A Deep Neural Network with an Integrated Reject Option. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97). PMLR, 2151–2159. https://proceedings.mlr.press/v97/geifman19a.html [7] Thibaud Gloaguen, Niels Mündler, Mark Niklas Mueller, Veselin Raychev, and Martin Vechev. 2026. Coding Agents Don’t Know When to Act. In Second Workshop on Agents in the Wild: Safety, Security, and Beyond. https://openreview.net/ forum?id=CVg4Honh4Y [8] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming—The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024). doi:10.48550/arXiv.2401.14196 [9] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large Language Models Cannot SelfCorrect Reasoning Yet. In International Conference on Learning Representations. https://openreview.net/forum?id=IkmD3fKBPQ [10] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66 [11] Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of SelfCorrection of LLMs. Transactions of the Association for Computational Linguistics 12 (2024), 1417–1440. doi:10.1162/tacl_a_00713 [12] Elliot Myunghoon Kim, Avi Garg, Kenny Peng, and Nikhil Garg. 2025. Correlated Errors in Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, 30038–30066. https://proceedings.mlr.press/v267/kim25e.html [13] Cuong Chi Le, Aashish Yadavally, Minh Le-Anh, and Tien N. Nguyen. 2026. Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback. arXiv preprint arXiv:2607.01360 (2026). doi:10.48550/arXiv.2607.01360 [14] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems, Vol. 36. Neural Information Processing Systems Foundation, Inc., 21558–21572. doi:10.52202/075280-0943 [15] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In Advances in Neural Information Processing Systems, Vol. 36. Neural Information Processing Systems Foundation, Inc., 46534–46594. doi:10.52202/075280-2019 [16] Qianru Meng, Xiao Zhang, Zhaochun Ren, and Joost Visser. 2026. EviACT: An Evidence-to-Action Framework for Agentic Program Repair. arXiv preprint arXiv:2605.27238 (2026). doi:10.48550/arXiv.2605.27238 [17] Open Policy Agent Contributors. 2026. Open Policy Agent Documentation. Accessed 2026-07-24. https://www.openpolicyagent.org/docs [18] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM Evaluators Recognize and Favor Their Own Generations. In Advances in Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, Inc., 68772–68802. doi:10.52202/079017-2197 [19] Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, and Sergey Nikolenko. 2026. AgentLens: ProductionAssessed Trajectory Reviews for Coding Agent Evaluation. arXiv preprint arXiv:2607.06624 (2026). doi:10.48550/arXiv.2607.06624 [20] Qwen Team. 2024. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115 (2024). doi:10.48550/arXiv.2412.15115 [21] Priyam Sahoo, Gaurav Mittal, Xiaomin Li, Shengjie Ma, Benjamin Steenhoek, Pingping Lin, and Yu Hu. 2026. AgentLens: Revealing the Lucky Pass Problem in SWE-Agent Evaluation. arXiv preprint arXiv:2605.12925 (2026). doi:10.48550/ arXiv.2605.12925 [22] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 36. Neural Information Processing Systems Foundation, Inc., 8634–8652. doi:10.52202/075280-0377 [23] Supply-chain Levels for Software Artifacts. 2026. SLSA v1.2 Provenance. Accessed 2026-07-24. https://slsa.dev/spec/v1.2/provenance
AgenticDev 2026, October 12, 2026, Munich, Germany
[24] Santiago Torres-Arias, Hammad Afzali, Trishank Karthik Kuppusamy, Reza Curtmola, and Justin Cappos. 2019. in-toto: Providing Farm-to-Table Guarantees for Bits and Bytes. In 28th USENIX Security Symposium. USENIX Association, 1393– 1410. https://www.usenix.org/conference/usenixsecurity19/presentation/torresarias [25] Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen, and Tony Mak. 2024. LLMs Cannot Find Reasoning Errors, but Can Correct Them Given the Error Location. In Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 13894–13908. doi:10.18653/v1/2024. findings-acl.826 [26] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents. Proceedings of the ACM on Software Engineering 2, FSE (2025), 801–824. doi:10.1145/3715754 [27] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, Inc., 50528–50652. doi:10.52202/079017-1601 [28] Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. 2024. Small Language Models Need Strong Verifiers to Self-Correct Reasoning. In Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 15637–15653. doi:10.18653/v1/2024.findings-acl.924 [29] Bingchen Zhao, Dhruv Srikanth, Yuxiang Wu, and Zhengyao Jiang. 2026. SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents. arXiv preprint arXiv:2605.21384 (2026). doi:10.48550/arXiv.2605.21384