arXiv CS open access

Safety Must Survive Self-Improvement: Why Failures Persist and How Agents Recover

Yunbei Zhang et al. · arxiv_cs

arXiv CS · Papers

cryptography cybersecurity privacy security

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Safety Must Survive Self-Improvement: Why Failures Persist and How Agents Recover Yunbei Zhang1,†,* Kaiqu Liang4

arXiv:2610.01073v1 [cs.SE] 1 Oct 2026

Janet Wang1,†

Zijian Jin5

Saiyue Lyu2

Yingqiang Ge3

Chandan K. Reddy6

Jihun Hamm1

2 Tulane University University of British Columbia 3 4 Rutgers University Princeton University 5 6 New York University Virginia Tech †

Equal contribution.

Project Page: https://RSI-Safety.github.io/

ABSTRACT

Recursive self-improvement (RSI) allows agents to carry useful changes across generations. Maintaining safety across these generations involves both preventing unsafe behavior from persisting and enabling recovery when failures occur. We study these challenges through a controlled testbed of stateful authorization tasks, where fixed LLM editors optimize executable agent components and independent traces record their effects. Paired interventions separate which revisions pass validation, which program continues running, and which program the editor revises next. After a new authorization dependency invalidates previously tested optimizations, historical scores preserve the same unsafe programs in 22 of 48 framework histories despite a correct alternative in every affected archive. Refreshing scores restores correctness on the original suite, with residual failures on independently composed tests. Failures also persist under an unchanged contract when all proposals are rejected and the failed incumbent remains active. Starting from shared failures, editing the initial implementation instead of the failed one improves recovery, although the advantage varies across editors. Full validation and validated rollback end with fully correct programs in the core trajectory study while retaining over 43% deployment savings. Preserving agent safety requires checking what will run under current conditions and choosing which implementation to edit next.

1 Introduction Recursive self-improvement (RSI) aims to turn an agent’s experience and feedback into lasting gains that shape subsequent improvement (Schmidhuber, 2007, Duan et al., 2026). These changes can operate through memory, tools, and workflows as well as model parameters (Gao et al., 2025, Fang et al., 2025). The Self-Taught Optimizer (STOP) recursively improves its code-improvement procedure, the Darwin Gödel Machine (DGM) evolves coding agents, and AlphaEvolve searches for increasingly effective algorithms (Zelikman et al., 2024, Zhang et al., ∗

Correspondence to: yzhang111@tulane.edu.

Zhang et al. · Safety Must Survive Self-Improvement

2025a, Novikov et al., 2025). Selected implementations can serve later requests and supply the starting point for further revisions. Agents therefore inherit both useful capabilities and the assumptions behind them, making safety across generations a central concern. Agent-safety benchmarks document risky tool use and unauthorized actions (Ruan et al., 2024, Zhang et al., 2024, Debenedetti et al., 2024, Wei et al., 2026), while studies of agent evolution identify insecure tool reuse, memory degradation, and behavioral drift (Shao et al., 2026, Han et al., 2026, Lin et al., 2026). Work on open-ended systems also examines how constraints can be preserved as behavior changes (Ecoffet et al., 2020, Sheth et al., 2025). We focus on what happens once a failure becomes observable: whether the failed implementation continues to run, and how the system proceeds toward recovery. Detecting a failure, producing a correct replacement, and selecting that replacement for execution are distinct outcomes. A correct final program also does not establish that execution remained safe during the intervening generations. (a) Failure entry under a new authorization dependency New dependency

Check every request

Cache the decision

Upstream Local version revoked unchanged

Passes initial suite

Unauthorized action

(b) Failure persistence through selection Stale archive scores

Keep after rejection

Old score still high

Archive

Unsafe program selected

Repeated execution

Correct repair

Current + proposals fail

in same archive

Failed current retained

Which code supplies the next edit?

Run passing code

Current or

Founder

Refresh archive scores

Validate fallback. Pause if none passes.

Protect execution during repair

LLM editor

Revised code

Editing source changes recovery

Figure 1. Safety depends on what runs and what supplies the next edit. (a) New authorization dependencies make previously tested reuse unsafe. (b) Historical scores and keepafter-rejection can preserve unsafe execution. (c) Current validation protects deployment, while editing source changes recovery. Red denotes current code, blue the founder (initial correct program), and green a currently passing fallback. Execution pauses if no fallback passes.

Consider an agent that checks permission before every protected action. An efficiency optimization caches this decision and refreshes it when the token’s version changes, passing the full initial suite at lower cost. If the service later introduces delegated permission, upstream revocation can invalidate authorization without changing that local version. The optimized agent continues acting, while the initial program remains correct because it checks effective permission on every request (Fig. 1a). New tests expose the failure, yet historical scores can still favor the optimized program over a correct repair in the archive. Under an

Zhang et al. · Safety Must Survive Self-Improvement

unchanged contract, a different rule can preserve failure: when the current program and all proposals fail validation, keep-after-rejection leaves the failed current program active. Both mechanisms show why evaluation alone does not determine whether unsafe execution ends (Fig. 1b). We investigate these mechanisms in a controlled testbed covering session authorization, filesystem containment, tool approval, and structured user consent. Fixed LLM editors repeatedly optimize executable agent components while selected programs and their histories evolve. Seeded event streams interleave requests with revocation, scope changes, and other state updates. Independent traces distinguish unauthorized effects from completion of permitted work, so blanket refusal cannot count as successful repair. Following every generation reveals failure entry, persistence, and recovery. Three matched comparisons separate repair availability from the decisions that govern its use (Fig. 1c). First, we compare historical and current readouts of the same archive. A separate refresh intervention updates stored scores, feedback, and selection metadata before search resumes. Second, we hold the incumbent, proposals, and test outcomes fixed while varying retention after rejection. Third, we start from the same failed programs and vary the editing source while holding validation and retention fixed. Using the founder as the editing source does not automatically deploy it. Current validation checks execution eligibility, while validated rollback selects a historical program passing current checks or pauses if none passes. These controls separate generating a repair from protecting execution during repair. Full initial validation does not prevent later failure under new authorization dependencies. In searches using upstream framework components, historical scores retain the same unsafe programs in 22 of 48 histories despite a correct alternative in every affected archive. Refreshing scores restores correctness on the original suite, while independently composed tests expose remaining coverage gaps. Under an unchanged contract, matched comparisons confirm that keeping the incumbent after rejection can preserve a detected failure. Editing the initial implementation improves recovery from shared failures relative to editing the failed version, with the advantage varying across editors. In the core trajectory study, full validation and validated rollback each achieve fully correct endpoints with mean deployment service-cost savings above 43%, excluding validation expenditure. Thus, execution and editing decisions jointly determine whether useful improvements preserve safety across generations. Our contributions are threefold: ∙ We construct a controlled testbed with four stateful authorization families, reproducible

event streams, and independent effect traces for studying safety across agent generations. ∙ We identify two mechanisms of failure persistence, historical eligibility and keep-afterrejection, through matched comparisons that separate evaluation, repair availability, and deployment. ∙ We isolate the effect of editing source on recovery from shared failures and evaluate validation and fallback interventions that improve safety while preserving useful optimization in these tasks.

Zhang et al. · Safety Must Survive Self-Improvement

2 Safety Across Generations RSI turns experience and feedback into persistent changes to an AI system that can modify how later improvements are generated, evaluated, or retained (Duan et al., 2026). We instantiate this loop through repeated editing of executable agent components, holding the LLM editor fixed while the selected program and its history evolve. Each generation inherits an implementation and produces the context from which later revisions are made. Our target is the authorization safety of the resulting actions throughout this process. A safety contract specifies permitted effects for each request and environment state. An unsafe program causes an unauthorized effect. The objective is efficiency gains with full correctness, including authorized effects, expected outputs, and completion of permitted work. Effects are recorded independently of the program. Improvement loop. At generation 𝑔, 𝑤𝑔−1 is the current program (incumbent) and 𝑤0 is the initially correct founder. The editor uses a parent 𝑠𝑔 , the editing objective, and (2) available feedback to produce two children 𝑢(1) 𝑔 and 𝑢𝑔 . Inheritance sets 𝑠𝑔 = 𝑤𝑔−1 , whereas founder editing sets 𝑠𝑔 = 𝑤0 . Let Pass𝑔 (𝑤) be one if program 𝑤 passes the required tests at generation 𝑔 and zero otherwise. The validation gate produces the eligible set { } (2) 𝑔 = 𝑤 ∈ {𝑤𝑔−1 , 𝑢(1) 𝑔 , 𝑢𝑔 } ∶ Pass𝑔 (𝑤) = 1 . Retention and recovery. The program selected for subsequent execution is 𝑤𝑔 . With select applying the condition’s ranking and tie rules, the keep rule is { select(𝑔 ), 𝑔 ≠ ∅, 𝑤𝑔 = (1) 𝑤𝑔−1 , 𝑔 = ∅. If no program passes, 𝑤𝑔 = 𝑤𝑔−1 despite Pass𝑔 (𝑤𝑔−1 ) = 0. A detected authorization failure can therefore remain active. Alternative fallback rules restore the founder, select a historical program passing current checks (validated rollback), or pause execution until one passes. We study how retention and editing source affect persistence and recovery across the program sequence, or lineage. An archive stores programs and evaluations, extending the selection pool. Historical eligibility reuses recorded results, while current validation determines eligibility using the current tests and contract. A certificate denotes a recorded suite pass, not a formal proof. Parent eligibility governs editing sources and deployment eligibility governs execution. A repair can thus be available in the archive without being selected for either role. Outcomes over time. Safety rate is the percentage of evaluated selections with no unauthorized effect. For initially failed roots, safety restored measures current safety, not whether safety was restored at any earlier generation. Safe throughout requires no violation in any generation so far. Unknown outcomes are not counted as safe. Full recovery additionally requires full correctness after a violation, including completion of permitted work. We use recovery in this sense.

3 Experimental Evaluation

Zhang et al. · Safety Must Survive Self-Improvement

Table 1. Matched comparisons separate failure entry, persistence, and recovery. Each study isolates one decision. The last row tests utility under safety controls. Question

What the comparison holds fixed

Evidence

Does initial validation suffice?

New contract, frozen founder control

Fig. 2a,b

Why is a repair not selected?

Same archive, different eligibility

Fig. 2c,d

Does rejection end unsafe use?

Same incumbent, proposals, and tests Fig. 3

What affects recovery?

Same failures, different editing source Fig. 4

Can safe improvement be useful? Same workloads, correctness and cost Fig. 5

3.1 Setup Our stateful tasks cover session authorization, filesystem containment, tool approval, and structured user consent. Event streams interleave requests with changes in permission or environment state. Public tests determine eligibility and feedback, audit tests support rollback, and hidden tests measure outcomes. The core public, audit, and hidden suites contain different instances of known test categories. The new-dependency studies introduce authorization mechanisms absent from all initial suites. Fixed editors receive efficiency or maintenance objectives under the same full contract, requesting lower service cost or clearer implementation, respectively. Paired branches share starting programs and workloads. Statistical comparisons resample starting blocks rather than generations. Table 1 maps the controlled comparisons, and Appendix A.1 specifies dataset construction and splits. 3.2 New Authorization Dependencies We first ask whether a program can become unsafe after passing every initial condition. Qwen3 Coder, Claude Sonnet 4.5 (Sonnet), and GPT-5.6 Sol (GPT) optimize two stateful authorization tasks under full initial validation, producing 72 histories. After four generations, a previously absent dependency makes permission depend on an upstream grant or resource classification without changing the meaning of token-local version queries. Neither mechanism appears in any public, audit, or hidden suite before this reveal. All strategies receive the new contract and expanded tests simultaneously. They then continue from the same checkpoint under different editing, retention, and archive-eligibility rules. The dependencies, reveal time, and strategies were fixed before the pilot, with additional seeds run under the same protocol (Appendix B). The new dependency exposes unsafe behavior in 22 of 24 GPT checkpoints despite their initial validation, while every frozen founder remains fully correct. Sonnet checkpoints instead tend to miss permitted work, and Qwen’s remain correct. The frozen founder faces the same changed environment without inheriting the optimization and still completes the task correctly. The new condition therefore breaks an assumption used by the optimized program while leaving a correct implementation available. Ordinary incumbent editing repairs the unsafe checkpoints promptly, whereas historical archive selection preserves them throughout the continuation (Fig. 2a,b). Thus the change exposes a repairable failure, and the selection rule determines whether that failure persists.

Zhang et al. · Safety Must Survive Self-Improvement Historical archive

Inherited · keep

(b) New contract: full correctness Full correctness (%)

Safety rate (%)

(a) New contract: safety 100

100 75 50

69.4 4

Current deployment checks

100

44.4

Generation (change after 4)

Current readout

100

100 75

54.2

50 60

Score refresh

(d) Frameworks: independent events Safety rate (%)

Safety rate (%)

Generation (change after 4)

Historical readout

100

93.8

100

87.5 75

50 20

Edit iteration

100

Edit iteration

Figure 2. Current eligibility restores safety after authorization changes. (a,b) In 72 histories, generation-4 checkpoints are evaluated under the new contract. Inherited editing and deployment checks overlap at full safety from generation 5. (c,d) In 48 framework histories, readouts select from the same archives. Score refresh resumes search from iteration 20 to 60, while readouts continue to 100. Independent compositions expose residual failures. Each row shares its own legend. Table 2. Parent revalidation does not replace deployment validation. Endpoint counts use 72 histories at generation 10. Unsafe-state counts use 432 post-change parent or deployed states on the original suite. Independent streams compose events outside the original templates. Parent checks

Deploy checks

Original suite

Independent

Unsafe states

Unsafe ↓

Fully correct ↑

Unsafe ↓

Fully correct ↑

Parent

Deployed

Historical

132

Current

Historical

132

Historical

Current

We vary whether the editing parent and the deployed program must pass the current tests. Revalidating only the editing parent removes unsafe editing sources but leaves unsafe deployed states unchanged. Deployment revalidation instead selects safe programs on the original suite, even with an unsafe editing parent (Table 2). At the first post-change selection, paired deployment rules share identical parents and proposals, so the improvement comes from choosing what runs. Independent event compositions preserve the advantage while exposing residual coverage errors. Some programs that most recently passed the old suite fail after the change, while frozen founders remain correct. A fallback must therefore be checked against the current contract even if it passed earlier tests.

Zhang et al. · Safety Must Survive Self-Improvement

Table 3. Score refresh improves safety and useful completion under matched continuation. Matched and long branches share the first 20 edits. Only the 60-edit comparison holds continuation budget fixed. Casbin uses its own held-out policy-engine evaluation and is a separate cohort. Independent compositions are reported for the two-mechanism cohorts. Cohort

Matched, 60 edits Long, 100 edits Casbin, 40 edits

Selection

Histories

Original suite

Independent

Unsafe ↓ Fully correct ↑ Unsafe ↓ Fully correct ↑

Historical scores

Score refresh

Historical readout

Current readout

Historical readout

n/a

Current readout

n/a

3.3 Persistence through Historical Scores We next test whether an available repair is selected within existing search components. The study uses OpenEvolve’s controller and database (Sharma, 2025), together with unchanged DGM (Zhang et al., 2025a) functions for parent sampling, archive updates, and original scores. Our authorization evaluator is integrated with these pinned upstream components, preserving their archive and selection behavior. GPT and Sonnet optimize both authorization mechanisms across 48 histories, with the new dependency revealed after eight fully validated iterations. A readout chooses a program from an archive. Historical and current readouts rank the same stored programs using recorded evaluations or current tests, isolating selection without changing search. A separate score refresh intervention updates stored scores, feedback, and selection metadata before continuing evolution. Matched continuations compare refresh with historical scores under the same editing budget. The readout comparison tests whether selection overlooks a correct program already in the archive. Continuing the search with or without refreshed scores then tests whether updating the evaluations improves subsequent selections under the same editing budget. Historical readout retains the same 22 unsafe programs through iteration 100, all in GPT histories, although every affected archive contains a correct alternative. Current readout selects fully correct programs on the original suite, and score refresh keeps the matched continuation correct throughout (Fig. 2c and Table 3). A recorded DGM delegation case explains the mechanism: a local-version cache remains highly ranked after upstream revocation, while correct later revisions are already available. Refresh replaces the failed program with an existing repair before the editor generates any new code. Later edits also receive feedback from the updated tests. Independent compositions, kept out of prompts and selection, confirm gains in safety and useful completion while revealing errors missed by the original suite (Fig. 2d). These residual failures show that refreshing eligibility and expanding test coverage serve complementary purposes. The Casbin1 policy-engine task also exhibits historical persistence with Sonnet, 1

Python implementation: https://github.com/apache/casbin-pycasbin.

Zhang et al. · Safety Must Survive Self-Improvement

(b) Same proposals, different retention

(a) Fixed pools: checking versus fallback

Safety rate (%)

Keep incumbent

Founder fallback

100 98 96 94 3

All Safety restored (%) 100

Editor

rejected Keep

96.8

GPT

100

Sonnet

100

Public probes per candidate

Fallback

Each editor processes 42 failed roots.

Figure 3. Rejection alone does not restore safety. (a) Exact subset evaluation freezes 432 candidate pools and costs. Unknown outcomes do not count as safe. At full coverage, keep achieves 96.8% safety, retaining 14 unsafe incumbents from three Qwen lineages. Founder fallback achieves 100% safety and correctness. (b) Both rules see the same incumbent, two proposals, and tests at 42 failed roots per editor. The rows show only blocks in which the incumbent and both proposals are rejected. Only fallback changes which program executes.

while current readout restores correctness. When a separate contract change invalidates every historical program, pausing protects execution until a passing repair is generated. These controls show how to end unsafe execution in both cases: select a currently valid alternative when one is available, and pause while all available programs fail. Appendix C details execution scope, independent tests, and restoration. 3.4 Persistence after Rejection Current validation can detect a failure without ending its execution. Under the keep rule, an empty eligible set returns the incumbent (Equation 1). We isolate this decision under an unchanged contract by giving keep and founder fallback the same failed incumbent, two GPT or Sonnet proposals, and test outcomes. Whenever all candidates fail, keep leaves the unsafe incumbent active, whereas fallback restores the correct founder (Fig. 3b). Because this comparison precedes any divergence in subsequent edits, the deployment difference is attributable to fallback alone. A complementary analysis freezes 432 candidate pools, costs, and outcomes while varying validation coverage. At full coverage, the remaining safety gap comes entirely from pools without a passing program: keep retains the failed incumbent, while founder fallback remains correct (Fig. 3a). These unsafe pools come from a small number of Qwen lineages. The identical-proposal comparison shows the same keep-versus-fallback difference with both GPT and Sonnet. Complete-history audits likewise confirm repeated retention of unchanged failed code after all candidates are rejected, most often with Qwen. Thus broader detection and better candidate generation leave a separate obligation to specify safe behavior when no proposal passes. The code-identity audit and recorded histories appear in Appendices D.2 and F. 3.5 Editing Source and Recovery Retention also shapes the next repair attempt by supplying its editing context. We compare current-code and founder editing from the same 42 naturally failed programs produced by

Zhang et al. · Safety Must Survive Self-Improvement (a) Original-editor endpoints

(b) Feedback and editing source

All (n=42) Devstral (n=14) GLM (n=16)

Qwen (n=12) 0

Founder Safety restored (%)

Inherited

66.7 97.6 100 (both) 87.5 100 91.7 50

100

Inherited · routine Founder · routine

97.6 90.5 66.7 52.4

100 75 50 25

Safety restored (%)

Inherited · keep

Founder · keep

100 90 80 1

3 4 5 Repair generation

Inherited · fallback

(d) Sonnet 4.5 100 (all)

Safety restored (%)

Inherited · full Founder · full

3 4 5 Repair generation

100

97.6

90 80 1

3 4 5 Repair generation

Figure 4. Editing source changes safety restoration from the same failures. Safety restored is the percentage of initially failed roots whose current selection has no unauthorized effect. (a) Endpoints are paired within each editor row, with root counts shown. (b) Routine feedback uses 12 probes, while full feedback uses 24 probes and up to two failures. The gate and keep rule are fixed. (c,d) Each new editor processes all 42 roots. Founder fallback edits current code and restores the correct founder. Values describe the current selection, not whether it was safe earlier.

Devstral 2, GLM 4.7 Flash, and Qwen3 Coder (the original editors), holding full validation, full feedback, and the keep rule fixed. Both conditions leave the incumbent in competition, so founder editing changes the source of proposals without automatically deploying the founder. After six generations, current-code editing leaves 14 unsafe endpoints and founder editing leaves one unsafe endpoint (Fig. 4a). In this original-editor cohort, safe selections are also fully correct. This paired difference shows that recovery depends on which implementation the editor revises, even when both branches receive the same failure information. The effect varies across editors and feedback conditions. Devstral repairs its roots under either source, GLM shows a smaller source difference, and the largest residual gap occurs with Qwen. Each comparison uses the same failed roots within an editor group. The paired comparison does not establish that fuller feedback alone improves recovery (Fig. 4a,b). Evaluating GPT and Sonnet on all shared roots substantially reduces endpoint failures, yet founder fallback still protects execution during repair (panels c,d). These cohorts start from failed programs and measure how often they are repaired. The number of roots therefore does not measure how often each editor produces a failure in the first place. The comparison separates the editor’s ability to produce a repair from the controller’s responsibility to choose valid behavior during repair. Recorded traces connect recovery to the missing authorization condition. In one consent case, the same stream contains permitted publish requests followed by requests after

Zhang et al. · Safety Must Survive Self-Improvement Efficiency · inherited

Efficiency · founder

(b) Useful optimization remains

(a) Consent Full validation

Full validation + rollback

90.3

Fully correct (%)

Safety rate (%)

100 75

58.3

33.3

25 0

Maintenance · inherited

Full validation

E-F

M-F 80 60

Rollback

Founder (g0)

100

Guard

M-I 0

E-I 40

Deployment savings (%)

Generation

Figure 5. Safety can recover while useful optimization remains. (a) Safety rate among 72 matched consent lineages, measured as selected programs with no unauthorized effect on the hidden suite. Full public validation begins at generation 4. Inherited and founder editing end at 33.3% and 90.3% safety. (b) Mean founder-relative deployment savings and full correctness across 288 blocks. E/M denote efficiency/maintenance, I/F inherited/founder editing, and RB rollback. The star is the founder. Savings exclude validation expenditure. Safety alone does not imply full correctness.

revocation. The failed program executes both, reusing consent whose scope and expiry still appear valid. Checking that the saved permission epoch matches the current epoch prevents the later unauthorized actions while preserving the earlier permitted work. Appendix F.4 aligns the source change with the same requests before and after repair. The distinction between eventual recovery and protection throughout improvement is also visible in a separate correct-start cohort: every Sonnet branch ends correctly despite unsafe intermediate execution. Appendix D reports histories and outcomes by editor. 3.6 Safety and Utility We finally ask whether preserving safety still permits useful optimization. The core study follows correct founders for 12 generations across four task families, the same three original editors, two version-query costs, and 12 seeds, yielding 288 matched blocks. Eight conditions vary editing objective, source, validation, and retention. Routine validation expands to the full public suite at generation 4, while preservation controls use full validation from the start, validated rollback, or an execution guard. We measure safety through independent hidden effects and utility through full correctness and founder-relative service-cost savings. Founders already complete permitted tasks correctly, so useful improvement means reducing the service cost of that work. This separates efficiency gains from simply skipping actions or checks. Consent shows the largest persistence contrast. Safety initially falls under both inherited and founder editing, then recovers much more strongly with founder editing after validation expands (Fig. 5a). Both conditions encounter failures in similar numbers of lineages, but fewer remain unsafe with founder editing. Founder editing also has higher final safety for each original editor. Unlike the shared-root experiment, these trajectories begin with correct programs and allow each condition to generate its own failures. They therefore describe

Zhang et al. · Safety Must Survive Self-Improvement

how often errors enter and remain in the selected histories. The shared-root comparison then isolates recovery from exactly the same failed program, resolving an ambiguity that the natural trajectories alone cannot. Approval shows a smaller effect, while session and filesystem failures are rare. The task and editor breakdowns and paired estimates appear in Appendix E. Validated rollback and full validation finish with fully correct endpoints while retaining mean service-cost savings above 43% (Fig. 5b). The founder at zero savings and full correctness makes the benefit of continued optimization explicit. An execution guard also prevents unauthorized effects, but evolved programs can still miss permitted work, explaining why safety and full correctness must be measured separately. Validation itself consumes resources, so the deployment savings should be read together with how often each selected program is used. Repeated use amortizes the cost of checking, while short deployments can spend more on validation than they save in execution. The savings at each generation and the total cost including validation in Appendix E quantify this distinction. Preserving current eligibility can therefore support useful improvement while protecting the actions carried out between edits.

4 Related Work Recursive self-improvement. RSI studies how an agent’s acquired capabilities can feed into its own subsequent improvement (Schmidhuber, 2007, Duan et al., 2026). Surveys organize agent evolution across model, memory, tool, and architectural changes (Gao et al., 2025, Fang et al., 2025). STOP improves optimization procedures, DGM retains an evolving archive of agents, and Gödel Agent revises its own logic (Zelikman et al., 2024, Zhang et al., 2025a, Yin et al., 2025). Related work automates agent design, workflow search, and reusable skill acquisition (Hu et al., 2025, Zhang et al., 2025b, Wang et al., 2023, Zheng et al., 2025). These directions make persistent reuse central to capability growth. Our study examines the safety of that reuse across generations by separating historical evaluation, current execution eligibility, and the implementation used to generate subsequent changes. Matched comparisons establish when an improvement loop preserves a detected failure despite having a valid alternative. Agent safety. Agent-SafetyBench and ToolEmu evaluate harmful actions in tool environments (Zhang et al., 2024, Ruan et al., 2024). AgentDojo evaluates prompt-injection attacks and defenses (Debenedetti et al., 2024). Other studies examine risks in reusable skills, agent memory, and the transition from a safe model to an acting system (Hao et al., 2026, Chen et al., 2024, Wei et al., 2026). Privilege controls and guard agents constrain what actions may execute (Shi et al., 2025, Xiang et al., 2025), while misevolution and lifecycle studies analyze how risks emerge or amplify during agent adaptation (Shao et al., 2026, Han et al., 2026, Lin et al., 2026). We extend this perspective to the decisions that keep behavior active across generations. Independent effect traces distinguish authorization from task success, and matched retention and editing-source comparisons explain why detecting a failure need not end unsafe action. Additional comparisons to interactive benchmarks, testing, and adaptation methods appear in Appendix H.

Zhang et al. · Safety Must Survive Self-Improvement

5 Discussion A program’s authorization must be assessed under its current contract. A historical pass remains a record of the conditions that were tested, but a new dependency can change which actions are permitted. Regression tests can expose that change without deciding what executes next. Our readout comparisons show the consequence directly: an archive may contain a correct repair while its stored ranking continues to favor an invalid program. A controller can record which contract each evaluation used and recheck deployment candidates when a relevant dependency changes. Execution and further improvement also require different decisions. A controller can protect the next action through a currently passing fallback while the editor continues searching for a better implementation. It can independently choose the implementation used to generate that next revision. The shared-root results show why this matters: identical failures and feedback can lead to different recovery trajectories depending on the editing source. Stronger editors shorten some repair processes, but protection is still needed while repair is underway. Revalidation identifies programs that still pass, fallback determines what runs after rejection, and the editing source determines where the next repair attempt starts. Safety evaluation should follow selected behavior throughout that loop. Final correctness describes the eventual outcome, while intermediate actions reveal whether reaching it involved unauthorized execution. The same distinction applies to utility: a cheaper program is useful only if it completes permitted work, and the cost of validating it must be included when assessing deployment benefits. Tests with new combinations of events still expose failures in programs that pass the current suite, so revalidation remains limited by test coverage. An evaluation should therefore record which programs run, whether their actions remain authorized, and which conditions were tested at each generation, alongside the final task score.

6 Conclusion We study agent safety across generations of recursive self-improvement and distinguish detecting a failure, making a repair available, and ending unsafe execution. Optimizations can pass the full initial suite yet become unsafe under a new authorization dependency. Historical scores may preserve these failures despite correct alternatives in the archive, while keeping the incumbent after rejection provides another route to persistence under an unchanged contract. These findings place safety responsibilities in selection and retention as well as in validation. Recovery also depends on the implementation supplied to the next edit. Our controlled testbed separates these decisions through paired histories and independent effect traces. Current deployment validation, refreshed archive scores, and validated fallback improve safety while preserving useful efficiency gains in the studied tasks. Safety must be preserved throughout improvement, even when no revision is accepted. Limitations. We study controlled authorization tasks with fixed LLM editors, initially correct founders, and externally specified contracts. Our conclusions assume correct validators within the tested conditions. Larger deployments, longer histories, and jointly evolving editors remain directions for further study. Additional scope and limitations appear in

Zhang et al. · Safety Must Survive Self-Improvement

Appendix I.

References Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 𝜏 2 -bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. URL https://arxiv.org/abs/2506.07982. Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=aOIJ2gVRWW. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. AgentPoison: Red-teaming LLM agents via poisoning memory or knowledge bases. Advances in Neural Information Processing Systems, 37:130185–130213, 2024. Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. Advances in Neural Information Processing Systems, 37:82895–82920, 2024. Chad DeChant. Episodic memory in AI agents poses risks that should be studied and mitigated. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 321–332, 2025. doi: 10.1109/SaTML64287.2025.00024. Yi Duan, Ying Liu, Zirui Tang, Haodong Chen, Jun Zhou, Yumou Liu, Bangrui Xu, Yukai Wu, Sidi Chen, Yuhan Zhou, Haoyu Wang, Xiaoyou Yu, Shaokun Han, Xuzhou Zhu, Le Zhou, Bolin Lu, Wei Zhou, Jiachen Liu, Nuozhou Fang, Jiaxin Tian, Ruoyu Chen, Yuxuan Li, Kai Zuo, Kaiyan Zhang, Qianyu Yang, Zijie Wang, Jiantao Qiu, Conghui He, Guoliang Li, Bowen Zhou, Zhiyuan Liu, Zhoufutu Wen, Jihua Kang, Xuanhe Zhou, and Fan Wu. The last AI built by humans: Toward genuine recursive self-improvement. arXiv preprint arXiv:2609.11873, 2026. URL https://arxiv. org/abs/2609.11873v2. Adrien Ecoffet, Jeff Clune, and Joel Lehman. Open questions in creating safe open-ended AI: Tensions between control and creativity. In Artificial Life Conference Proceedings 32, pages 27–35. MIT Press, 2020. Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 2021. doi: 10.1007/s11229-021-03141-4. Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu, Siwei Liu, Zihao Li, et al. A comprehensive survey of self-evolving AI agents: A new paradigm bridging foundation models and lifelong agentic systems. arXiv preprint arXiv:2508.07407, 2025. Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In International Conference on Machine Learning (ICML), pages 13481–13544. PMLR, 2024. Huanang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Qihan Ren, Cheng Qian, Zhenhailong

Zhang et al. · Safety Must Survive Self-Improvement

Wang, Minda Hu, Huazheng Wang, Qingyun Wu, Heng Ji, and Mengdi Wang. A survey of selfevolving agents: On path to artificial super intelligence. arXiv preprint arXiv:2507.21046, 2025. Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. RedCode: Risky code execution and generation benchmark for code agents. Advances in Neural Information Processing Systems, 37:106190–106236, 2024. Siwei Han, Kaiwen Xiong, Jiaqi Liu, Xinyu Ye, Yaofeng Su, Wenbo Duan, Xinyuan Liu, Cihang Xie, Mohit Bansal, Mingyu Ding, Linjun Zhang, and Huaxiu Yao. Alignment tipping process: How self-evolution pushes LLM agents off the rails. arXiv preprint arXiv:2510.04860, 2026. URL https://arxiv.org/abs/2510.04860. Haochang Hao, Dehai Min, Zhifang Zhang, Yunbei Zhang, Miao Xu, Yingqiang Ge, and Lu Cheng. POISE: Position-aware undetectable skill injection on LLM agents. arXiv preprint arXiv:2606.07943, 2026. Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Safe LoRA: The silver lining of reducing safety risks when fine-tuning large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=t9U3LW7JVX. Wenyue Hua, Xianjun Yang, Zelong Li, Cheng Wei, and Yongfeng Zhang. TrustAgent: Towards safe and trustworthy LLM-based agents through agent constitution. arXiv preprint arXiv:2402.01586, 2024. URL https://arxiv.org/abs/2402.01586v1. Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-Zero: Self-evolving reasoning LLM from zero data. arXiv preprint arXiv:2508.05004, 2025. Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive LLMs that persist through safety training. arXiv preprint arXiv:2401.05566, 2024. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=VTF8yNQM66. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling declarative language model calls into selfimproving pipelines. In International Conference on Learning Representations (ICLR), 2024. Juyong Lee, Dongyoon Hahm, June Suk Choi, W Bradley Knox, and Kimin Lee. MobileSafetyBench: Evaluating safety of autonomous agents in mobile device control. arXiv preprint arXiv:2410.17520, 2024. Junjie Li, Xi Xiao, Yunbei Zhang, Chen Liu, Lin Zhao, Xiaoying Liao, Yingrui Ji, Janet Wang, Yingqiang Ge, Weijie Xu, Xi Fang, Xiang Xu, Tianchen Zhao, Youngeun Kim, Jihun Hamm, Tianyang Wang, and Chandan Reddy. Agent harness engineering: A survey, 2026. URL https://openreview.net/ pdf?id=eONq7FdiHa. Ruixiao Lin, Xinhao Deng, Qingming Li, Jianan Ma, Yunhao Feng, Yuqi Qing, Zhenyuan Li, Yechao Zhang, Shiwen Cui, Changhua Meng, et al. Safety in self-evolving LLM agent systems: Threats, amplification, and case studies. arXiv preprint arXiv:2606.23075, 2026.

Zhang et al. · Safety Must Survive Self-Improvement

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Shuaicheng Niu, Guohao Chen, Yaofo Chen, Zhiquan Wen, Jinwu Hu, Zeshuai Deng, Deyu Chen, Shuhai Zhang, Renjie Chen, Zihao Lian, et al. A survey on self-improving test-time intelligence: Feedback-driven adapting, learning, and scaling at inference. arXiv preprint arXiv:2609.01679, 2026. Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations (ICLR), 2022. Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. CWEval: Outcome-driven evaluation on functionality and security of LLM code generation. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pages 33–40. IEEE, 2025. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946, 2024a. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview. net/forum?id=hTEGyKf0dZ. Jiahao Qiu, Xuan Qi, Tongcheng Zhang, Xinzhe Juan, Jiacheng Guo, Yifu Lu, Yimin Wang, Zixin Yao, Qihan Ren, Xun Jiang, et al. Alita: Generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self-evolution. arXiv preprint arXiv:2505.20286, 2025. Maxime Robeyns, Martin Szummer, and Laurence Aitchison. SICA a self-improving coding agent. In ICLR 2025 Workshop on Scaling Self-Improving Foundation Models, 2025. URL https://openreview. net/forum?id=rShJCyLsOr. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the risks of LM agents with an LMemulated sandbox. In International Conference on Learning Representations (ICLR), 2024. Jürgen Schmidhuber. Gödel Machines: Fully self-referential optimal universal self-improvers. In Artificial General Intelligence, pages 199–226. Springer, 2007. doi: 10.1007/978-3-540-68677-4_7. Shuai Shao, Qihan Ren, Dongrui Liu, Chen Qian, Boyi Wei, Dadi Guo, Jingyi Yang, Xinhao Song, Linfeng Zhang, Weinan Zhang, et al. Your agent may misevolve: Emergent risks in self-evolving LLM agents. In International Conference on Learning Representations, 2026. Asankhaya Sharma. OpenEvolve: an open-source evolutionary coding agent. GitHub, 2025. URL https://github.com/algorithmicsuperintelligence/openevolve. Ivaxi Sheth, Jan Wehner, Sahar Abdelnabi, Ruta Binkyte, and Mario Fritz. Safety is essential for responsible open-ended systems. arXiv preprint arXiv:2502.04512, 2025. August Shi, Alex Gyori, Suleman Mahmood, Peiyuan Zhao, and Darko Marinov. Evaluating testsuite reduction in real software evolution. In Proceedings of the 27th ACM SIGSOFT International

Zhang et al. · Safety Must Survive Self-Improvement

Symposium on Software Testing and Analysis, ISSTA ’18, pages 84–94. ACM, July 2018. doi: 10.1145/ 3213846.3213875. Tianneng Shi, Jingxuan He, Zhun Wang, Hongwei Li, Linyu Wu, Wenbo Guo, and Dawn Song. Progent: Programmable privilege control for LLM agents. arXiv preprint arXiv:2504.11703, 2025. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems (NeurIPS), 2022. Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel, Esin Durmus, Spandana Gella, Karolina Stańczak, and Siva Reddy. SafeArena: Evaluating the safety of autonomous web agents. arXiv preprint arXiv:2503.04957, 2025. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. Yingxu Wang, Siwei Liu, Jinyuan Fang, and Zaiqiao Meng. EvoAgentX: An automated framework for evolving agentic workflows. arXiv preprint arXiv:2507.03616, 2025a. Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025b. Bowen Wei, Yunbei Zhang, Jinhao Pan, Kai Mei, Xiao Wang, Jihun Hamm, Ziwei Zhu, and Yingqiang Ge. ClawSafety: “safe” LLMs, unsafe agents. arXiv preprint arXiv:2604.01438, 2026. Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, and Bo Li. GuardAgent: Safeguard LLM agents by a guard agent via knowledge-enabled reasoning. In International Conference on Learning Representations (ICLR), 2025. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 𝜏-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. URL https://arxiv.org/abs/2406.12045. Xunjian Yin, Xinyi Wang, Liangming Pan, Li Lin, Xiaojun Wan, and William Yang Wang. Gödel Agent: A self-referential agent framework for recursively self-improvement. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 27890–27913, 2025. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. TextGrad: Automatic “differentiation” via text. arXiv preprint arXiv:2406.07496, 2024. Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (STOP): Recursively self-improving code generation. arXiv preprint arXiv:2310.02304, 2024. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10471–10506. Association for Computational Linguistics, 2024. Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Gödel Machine: Openended evolution of self-improving agents. arXiv preprint arXiv:2505.22954, 2025a.

Zhang et al. · Safety Must Survive Self-Improvement

Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. In The Thirteenth International Conference on Learning Representations, 2025b. URL https://openreview.net/forum?id=z5uVAKwmjf. Yunbei Zhang, Akshay Mehra, Shuaicheng Niu, and Jihun Hamm. DPCore: Dynamic prompt coreset for continual test-time adaptation. In International Conference on Machine Learning, pages 75757– 75778. PMLR, 2025c. Yunbei Zhang, Yingqiang Ge, Weijie Xu, Yuhui Xu, Jihun Hamm, and Chandan K Reddy. Visual exclusivity attacks: Automatic multimodal red teaming via agentic planning. arXiv preprint arXiv:2603.20198, 2026a. Yunbei Zhang, Kai Mei, Ming Liu, Janet Wang, Dimitris N Metaxas, Xiao Wang, Jihun Hamm, and Yingqiang Ge. Agents in the wild: Safety, society, and the illusion of sociality on Moltbook. arXiv preprint arXiv:2602.13284, 2026b. Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, and Chandan K Reddy. STOP comparing LLM agents without disclosing the harness. arXiv preprint arXiv:2605.23950, 2026c. Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. Agent-SafetyBench: Evaluating the safety of LLM agents. arXiv preprint arXiv:2412.14470, 2024. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In AAAI Conference on Artificial Intelligence (AAAI), 2024. Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03335, 2025. Arman Zharmagambetov, Chuan Guo, Ivan Evtimov, Maya Pavlova, Ruslan Salakhutdinov, and Kamalika Chaudhuri. AgentDAM: Privacy leakage evaluation for autonomous web agents. arXiv preprint arXiv:2503.09780, 2025. Boyuan Zheng, Michael Y Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, et al. SkillWeaver: Web agents can self-improve by discovering and honing skills. arXiv preprint arXiv:2504.07079, 2025. Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, and Jun Wang. Memento: Fine-tuning LLM agents without finetuning LLMs. arXiv preprint arXiv:2508.16153, 2025. URL https://arxiv.org/abs/2508.16153. Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. GPTSwarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=uTC9AFXIhg.

Zhang et al. · Safety Must Survive Self-Improvement

Appendix The appendix is organized around the controls needed to interpret the main findings. Section A explains what a dataset instance contains, how its behavior is scored, and how paired statistics are computed. Sections B and C distinguish changed contracts from historical selection, including independent evaluation. Sections D and E examine recovery, editor differences, useful optimization, and alternative selection rules. Section F connects these aggregate results to four recorded cases, including a request-level visualization of failure and repair. Native-workflow transfer and extended related work follow in Sections G and H. Section I discusses scope and limitations.

A Methods and Statistics A.1 Dataset Construction Instance and generation. A dataset instance is an ordered event stream, not a standalone prompt. Setup and update events change the environment, while requests ask the evolving program to perform protected actions. Seeded templates specify eight categories per family: basic behavior, repetition, denial, scope, updates, rebinding, boundaries, and ordering. Each category contains three probes. Generators instantiate identifiers, scopes, and categoryspecific state changes so the same contract can be tested across reproducible workloads. Consent shares the approval event templates but uses a distinct service contract for asking and reusing user agreement. Splits and matched reuse. Each block fixes a task family, editor, version-query cost, and seed, then generates 24 public, 24 audit, and 24 hidden streams with separate split seeds. Public tests govern eligibility and feedback, audit tests support rollback, and hidden outcomes are measured without entering either decision. Conditions and generations reuse the same streams within a block. The 288 blocks give 20,736 stream assignments indexed by block, split, and probe (Table 4). This count includes streams assigned to different blocks even when they share an event sequence. It does not count distinct authorization problems. Repeated program executions do not create additional dataset instances or independent statistical samples. Oracle and observable outcomes. The environment computes permission from its current state and records each protected effect independently of the program’s return value. Full correctness requires the expected outputs and effects, including completion of permitted requests. A program cannot erase an unauthorized effect by suppressing its output, nor obtain full correctness by refusing every action. Records connect each request to service calls, effects, costs, and program identity. Fig. 13 illustrates one saved consent stream before and after repair, with the corresponding source change given alongside it. These records make it possible to distinguish proposing a repair, deploying it, and preserving safe behavior during subsequent edits. Coverage extensions. The core public, audit, and hidden suites are separate seeded instances of known categories. They measure generalization within the designed templates. To test new mechanisms, delegation and resource-policy extensions each expand the initial

Zhang et al. · Safety Must Survive Self-Improvement

Table 4. The constructed dataset pairs stateful authorization with observable actions. Each block has 24 public, 24 audit, and 24 hidden streams. Every split covers eight categories with three probes each. Blocks cross three editors, two query costs, and 12 seeds per family. Request ranges count protected-action requests within a stream, excluding setup and update events. Assignments are reused across conditions and generations. Task family

Core obligation

Requests/stream Blocks Assignments

Session

Current permission

2–33

5,184

Filesystem

Canonical containment

1–33

5,184

Tool approval Valid scope and uses

2–25

5,184

User consent

2–25

5,184

288

20,736

Applicable agreement

Total

24-stream suite to 36 streams when four additional categories are revealed, with public and hidden instances generated separately. Another 64 independently composed streams per mechanism, 128 in total, test deeper dependencies and interleaved requests. The framework studies reuse these authorization tasks within upstream search components. The Casbin task uses a real policy engine with constructed request streams, including 36 public and 60 held-out streams after the change. The native dialogue study separately uses released airline and retail tasks (Appendix G). These extensions and external tasks are distinct from the core corpus above. A.2 Contracts and Execution Session authorization. Requests identify a tenant, session, and resource. A payload may be returned only when permission is current and the session is not revoked. State changes grant or remove permission, revoke a session, or restore it. Reusing authorization for another scope or after revocation violates the contract. Checking costs ten units and a protected read costs one. Filesystem containment. Requests specify a path and content. The program must resolve the path and write only when its canonical target lies inside the authorized root. Symbolic-link targets can change between requests. Correctness requires the requested target and content as well as containment. Resolution costs eight units, containment testing three, and writing one. Evaluations use disposable filesystem fixtures with no access to arbitrary external files. Tool approval. A token authorizes a tool and resource, an amount within a limit, a finite number of uses, and an expiration time. State changes can mint, cancel, renew, or rebind authorization and advance logical time. Execution consumes one use and changes the global version. The program returns the execution result when permitted and otherwise produces no effect. Approval costs 12 units and execution one. Structured user consent. The request identifies a simulated user, tool, resource, and amount. Willingness is private to the simulator. Asking presents these details and returns agreement, scope, version, remaining uses, and expiration. Agreement permits reuse for the

Zhang et al. · Safety Must Survive Self-Improvement

same action while these conditions remain valid. Refusal, revocation, changed scope, and exhausted uses invalidate execution. The action itself enforces consent only in the guard condition. Asking costs 12 units and execution one. The task measures structured consent semantics, with no natural-language persuasion or human participants. Tests and workloads. Eight categories cover basic allowed behavior, repeated requests, denial, scope changes, state updates, authorization rebinding, boundary values, and operation ordering. Three instances per category give 24 streams. The routine gate uses basic, repetition, denial, and scope categories. Full validation uses every category in the current public suite. The ordinary gate switches to full validation at generation 4, while the full-validation conditions use it from generation 1. Public, audit, and hidden suites use separate seeds. Within a block, workloads remain fixed across conditions and generations. Hidden outcomes never enter selection or feedback. Founders and sampling. Each model, family, and query-cost combination contains 12 seeds. Even-indexed seeds use a redundant correct founder and odd-indexed seeds a simple correct founder. Paired conditions share the same founder, with the execution guard applied only in its intervention. Workload construction, condition order, and tie-breaking use seeds derived from the block identifier. Model access and output limits. Table 5 records the model identifiers and sampling parameters. The core trajectories and original-editor recovery use output limits of 6,144 tokens for Devstral and GLM and 4,096 for Qwen. Qwen uses 6,144 tokens in the changedauthorization study and 8,192 in the prompt, selection, price, and tie ablations. GPT and Sonnet use 8,192 tokens in the new-editor recovery, correct-start, changed-authorization, framework, and ablation studies. In the native-dialogue study, workflow edits allow 12,288 tokens for GPT and 6,144 for GLM. Dialogue actor and user requests are capped at 4,096 tokens, and the GLM consent-review call at 3,000. The framework bridge forwards requests through the same Bedrock client and caps output at 8,192 tokens. Experiment seeds do not control provider-internal token sampling. Table 5. Model identifiers and explicit sampling settings. All requests use Amazon Bedrock Converse in us-west-2. A dash means that the parameter was omitted from the request, not set to zero. Identifiers are the routed strings recorded in the experiment configurations. Editor

Routed model ID

Temp. Top-𝑝

Devstral 2

mistral.devstral-2-123b

0.7

0.95

GLM 4.7 Flash zai.glm-4.7-flash

0.7

0.95

Qwen3 Coder qwen.qwen3-coder-30b-a3b-v1:0

0.7

0.95

Sonnet

us.anthropic.claude-sonnet-4-5-20250929-v1:0

–

GPT

us.openai.gpt-5.6-sol

–

Edit instructions. Efficiency requests lower weighted service and interaction cost while preserving all behavior and the complete contract. Maintenance instructions request clarity and maintainability under the same requirement and ask the editor not to optimize service or interaction cost. Both receive the public contract, parent code, and routine-test feedback. Neither requests weaker authorization.

Zhang et al. · Safety Must Survive Self-Improvement

Execution and accounting. Proposed programs run in restricted Python in separate processes with execution limits. They access only the task’s documented queries, checks, and protected operations. Internal simulator state and auxiliary methods are inaccessible. Suite success requires expected outputs and zero unauthorized effects on every test stream. API failures and incomplete evaluations remain recorded. All 288 blocks complete their scheduled 12 generations. Across 2,304 lineages, we observe 29,952 selected states including the founders. All final safety and correctness outcomes are observed. The results use this complete cohort throughout. Earlier pilot experiments are not pooled into its counts or paired tests. A.3 Cost Accounting Let 𝐶𝑏,𝑔 be the routine service cost of the selected program in block 𝑏 at generation 𝑔, with 𝐶𝑏,0 its founder cost. Final deployment savings in Table 13 average 1 − 𝐶𝑏,12 /𝐶𝑏,0 over the 288 blocks, giving equal weight to each lineage. This differs from a ratio of aggregate costs. Guard operations are included in deployment cost, while model generation and validation are accounted for separately. Let 𝑉𝑏,𝑔 denote the service cost of validating the incumbent and children, including audit and rollback checks. If each selected generation serves 𝐵 routine workload batches, the mean total cost ratio to founder deployment over 𝑛 matched blocks is 𝑛

𝑅(𝐵) =

𝐵 ∑𝑔=1 𝐶𝑏,𝑔 + ∑𝑔=1 𝑉𝑏,𝑔 1 ∑ . 𝑛 𝑏=1 12𝐵𝐶𝑏,0

(2)

Validation is charged as fresh execution of the required suites even when the experimental runner caches an identical evaluation. Hidden evaluation is measurement overhead and is excluded. These are task-service units rather than wall-clock latency or API expenditure. Fig. 11 reports the deployment-volume sensitivity. A.4 Statistical Inference Each main task-family contrast pairs 72 blocks by model, query cost, and seed. We resample blocks within each model, retain equal model weights, and use 10,000 bootstrap resamples for percentile 95% intervals. Two-sided exact paired tests condition on discordant endpoint outcomes. Holm adjustment applies jointly to the 16 comparisons of efficiency with inheritance against maintenance with inheritance, founder editing, full validation, and validated rollback across four families. Fig. 6 includes the complete declared family, including nonsignificant contrasts. Generations and proposed programs are not independent samples. For consent, founder editing improves endpoint safety over inherited efficiency by 56.9 percentage points (95% CI 45.8 to 68.1, Holm-adjusted 𝑝 < 0.001). The respective ever-unsafe counts are 63 and 61 among 72 lineages per condition. Thus similar numbers of lineages encounter a failure, but more regain safety with founder editing. The 42-root recovery intervals resample within original-editor strata. The editing-source contrast under full feedback reduces unsafe endpoints by 31.0 percentage points (95% CI 23.8 to 38.1), with 13 improved roots, no worsening, and Holm-adjusted 𝑝 = 0.000488.

Zhang et al. · Safety Must Survive Self-Improvement

Under current editing, fuller feedback gives seven improved roots and one worsening (exact 𝑝 = 0.0703). The 42-root study’s two principal paired contrasts compare editing source and feedback. The two new editors are evaluated separately on all roots. All new-editor endpoint contrasts against current editing with keep have exact paired 𝑝 = 1, reflecting the small final differences. Table 11 gives the paired endpoints and selected states. The eight exploratory ever-unsafe contrasts from recovery and prevention form a separate Holm family. Framework branches likewise remain paired by their shared initial history, and the independent-composition comparison is reported separately from the original-suite comparison family. (a) Consent

(b) Approval Holm p

Holm p

Maintenance

0.002

Maintenance

1.000

Founder editing

<0.001

Founder editing

0.074

Full validation

<0.001

Full validation

0.002

Rollback

<0.001

Rollback

0.002

Safety advantage (pp)

(d) Filesystem Holm p

Holm p

Maintenance

1.000

Maintenance

1.000

Founder editing

1.000

Founder editing

1.000

Full validation

1.000

Full validation

1.000

Rollback

1.000

Rollback

1.000

Safety advantage (pp)

Figure 6. Consent has the largest paired safety advantage. All 16 declared comparisons show the named comparator’s safety rate minus that of efficiency with inheritance, in percentage points. Each uses 72 matched blocks. Whiskers are percentile 95% intervals from 10,000 bootstrap resamples within model strata. Printed values are two-sided exact paired 𝑝 values with Holm adjustment over all 16 contrasts. Intervals and adjusted tests are different inferential summaries. Zero marks equal safety rates. Maintenance is gray, founder editing blue, and preservation controls green, with distinct markers.

B Changed Authorization B.1 Conditions and Policies The initial delegation API checks a token’s direct authorization. The extension makes effective permission depend on upstream grants, so revoking an ancestor can invalidate a descendant without changing its local version. The resource-policy extension adds a resource classification that can exceed the token’s clearance without changing that token.

Zhang et al. · Safety Must Survive Self-Improvement

Table 6. Deployment validation removes unsafe archive selections. Each row contains the same 72 paired blocks. Endpoints are at generation 10. Unsafe and paused generations count repeated selected states across generations 5 to 10, with 432 states per row. Full correctness requires both authorized effects and completion of permitted work. Policy

Unsafe endpoints

Correct

Unsafe generations

Paused

Incumbent / keep

Founder / keep

Incumbent / stop

Incumbent / rollback

Archive / historical

132

Archive / parent current

132

Archive / deployment current

Archive / both current

Both extensions were fixed before the corresponding evolution runs and absent from all initial public, audit, and hidden streams. They are revealed with their documentation and expanded tests after generation 4. The founders call the authoritative permission interface and pass before and after the extension. The 72 histories cross three editors, two mechanisms, and 12 seeds. Eight strategies share the first four generations and continue through generation 10. Incumbent editing uses keep, stop, or validated rollback after rejection. Founder editing regenerates candidates from the original implementation. Four archive strategies independently vary historical or current eligibility for parent selection and deployment. Historical eligibility reuses the entry’s stored evaluation, whereas current eligibility evaluates it against the expanded public suite. Table 6 reports all eight policies, including cases where rapid editing makes the immediate intervention difference small. The experiment began with four seeds per editor and mechanism, then expanded by eight seeds after the pilot was inspected. GPT’s unsafe frozen checkpoints occur in seven of eight pilot cases and 15 of 16 expansion cases. Sonnet has no unsafe frozen checkpoint in either batch but misses permitted work, and Qwen remains correct. Current deployment validation gives correct endpoints in both batches. The combined 12-seed cohort is complete, while the expansion decision was made after observing the pilot. B.2 Eligibility and Coverage At the first post-change selection, deployment-policy pairs use identical parents and proposals in every block. Revalidating deployment improves 22 paired unsafe endpoints with no worsening on the original suite (exact 𝑝 = 4.77 × 10−7 , Holm-adjusted 𝑝 = 9.54 × 10−7 ). The stratified safety difference is 30.6 percentage points (95% CI 26.4 to 33.3), and full correctness improves by 55.6 points (CI 48.6 to 62.5). Later proposed programs can diverge across policies. Independent replay composes deeper dependencies, multiple tokens, and interleaved resources beyond the original templates. These outcomes are excluded from search and

Zhang et al. · Safety Must Survive Self-Improvement

Table 7. A certificate from the old contract may be invalid after the change. The boundary comparison has 72 histories. The bottom rows describe the sole all-candidates-rejected pool among 432 replayed pools. Program or replay rule

Unsafe

Fully correct

Other error

Latest old-certified program at change

Actual incumbent at change

Frozen founder at change

Keep at rejected pool

Old-certificate fallback at rejected pool

Revalidated history at rejected pool

feedback. The four archive-policy results are in Table 2. Some programs that pass the expanded suite still fail these new event combinations. Updating scores therefore leaves a separate need to test combinations the suite does not cover. A passing new proposal is already available in 123 of the 132 unsafe historical selections, confirming that repair availability alone does not ensure deployment. Current deployment validation retains 36.4% mean founder-relative service savings while raising mean validation probes from 439.0 to 540.5 per history, a 23.1% increase. Authoritative checks cost 12, local version queries one, global version queries two, and protected execution one. B.3 Old Certificates We define the old-certified program as the most recent archive entry that passed the complete old public suite before the contract changed, breaking within-generation ties by code hash. This choice uses only pre-change metadata. Replaying the selected versions under the new contract leaves 20 unsafe, 37 fully correct, and 15 otherwise incorrect (Table 7). Some programs that passed the old suite become unsafe under the new contract, while others remain correct. The replay also examines the 432 recorded keep-policy candidate pools. Only one has no currently passing candidate. Keep and old-certificate fallback choose the same program in that event, which misses permitted work, whereas current revalidation selects a correct historical version. This replay changes only which program is selected from each recorded pool. It does not run evolution under the old-certificate fallback rule, because the alternative selections do not generate the next proposals.

C Framework Studies C.1 Execution Scope The framework study keeps the editing model fixed and evolves an authorization component. OpenEvolve uses its upstream controller, database, workers, prompt construction, and edit parsing. The DGM integration executes its unchanged parent-sampling, archive-update, and original-score functions with task-specific metadata and an external editor. Our adapter

Zhang et al. · Safety Must Survive Self-Improvement

Table 8. Search systems reuse programs through different selection rules. Execution scope is distinct from source inspection. The paragraphs below give pinned versions and the inspected behavior when evaluations change. System

Editing source

Retained information

Evidence used here

OpenEvolve

Performance and diversity

Programs and metrics

Native controller

DGM

Score and offspring weights

Functioning revisions

Three upstream functions

STOP

Current target

Target and prior improver

Source inspection

SICA

Score confidence bound

Code and benchmark results

Source inspection

AlphaEvolve

Evolutionary database

Programs, scores, outputs

Technical report

supplies authorization evaluations and the deployment readout. Casbin supplies a real policyenforcement implementation for a custom stateful task. These are separate execution scopes. Table 8 identifies these scopes and comparisons with STOP, SICA (Robeyns et al., 2025), and AlphaEvolve. Changed evaluations. OpenEvolve refresh is our intervention. The three transferred DGM functions contain no evaluator-change refresh. STOP checks candidate utility and can restore a prior improver on failure, with no bulk historical refresh in the inspected loop. SICA selects the latest iteration meeting the best mean score’s lower confidence bound from recorded results. The inspected selector does not refresh these results for a changed contract. AlphaEvolve’s report describes evaluation of new candidates but leaves archive refresh under changed conditions unspecified. Pinned sources. OpenEvolve is pinned to commit 411fb59. DGM is pinned to commit a565fd2. The transferred parent-sampling, archive-update, and original-score functions are checked for abstract-syntax-tree equality with upstream. STOP’s outer improvement loop checks each candidate’s utility and retains the target on an unsuccessful attempt. Its exception branch restores a prior improver, a different object from the target algorithm. SICA’s select_base_agent uses recorded benchmark statistics and copies the selected agent as the next starting point. AlphaEvolve’s technical report, Sections 2.2 to 2.5, describes program sampling, candidate evaluation, and an archive based on islands and MAP-Elites. We inspect STOP and SICA source but do not execute their full systems in this study. AlphaEvolve is a report-level comparison. Cohorts and units. The original 48 histories cross two frameworks, two editors, two mechanisms, and six seeds, with 20 edits each. Delegation makes authorization depend on upstream grants. Resource policy makes it depend on a resource’s current classification. The new dependency appears after iteration 8. All 384 pre-change selected states pass the initial evaluation. The long cohort continues these same 48 archives to iteration 100. The refresh cohort branches these archives at iteration 20 and runs to iteration 60. Thus the long and refresh cohorts share their first 20 edits. They are paired branches, not additional independent starting histories. The Casbin cohort crosses two frameworks, two editors, and six seeds, giving 24 histories with 40 edits each. All 192 pre-change selected states pass their initial tests. The implementation uses pycasbin 2.6.1 and its Enforcer, with a separate graph oracle for the custom task. The recovery cohort contains 24 DGM-derived histories and runs six generations

Zhang et al. · Safety Must Survive Self-Improvement

with two proposals per generation. Across these cohorts, 192 recorded branches contain 7,992 scheduled observations, all observed. The original 48 histories also supply the first 20 iterations of their continuations. Readout and refresh. Historical readout ranks the retained programs using the evaluation stored with each program. Current readout checks the same retained programs against the current public suite before ranking eligible programs. These readouts do not change parents, proposals, or database contents. They measure different choices from the same search history. The refresh intervention at iteration 20 updates retained programs’ public scores and feedback, rebuilds score-dependent selection metadata, and then resumes evolution. It preserves the program set and existing island and cell membership. OpenEvolve’s elite and best-program indexes and DGM’s failure metadata reflect the new scores. These updates act jointly on ranking and feedback. OpenEvolve checkpoints do not serialize the Python random-number state, so the continuation restarts with the configured seed. DGM continuation restores its recorded random-number state. Validation and independent streams. All initial categories are public from the first iteration. New dependencies and their expanded tests are revealed together after iteration 8. Independent evaluation initially uses different instances of the same stream templates. A further generator and oracle, frozen before this framework campaign, provide 64 streams for each of delegation and resource policy, with deeper dependencies, multiple tokens, and resource interleavings. These outcomes never enter model prompts, ranking, or feedback. This replay applies to the two-mechanism cohorts. Casbin uses its own held-out evaluation under the policy-engine interface. Every frozen founder and single-check reference used for these controls passes its corresponding evaluation. C.2 Independent Evaluation Fig. 7 separates framework, editor, and task on the original suite. The reversal between delegation/resource policy and Casbin shows why editor capability alone does not determine these failures. Table 9 gives the independent-composition results by cell, including the three unsafe refresh endpoints and the additional failures that miss permitted work. At the matched iteration-60 endpoints, score refresh reduces unsafe selections from 25 to three and raises full correctness from six to 43 on independent compositions. The paired safety contrast contains 23 improvements and one worsening (exact 𝑝 = 2.98 × 10−6 ). One OpenEvolve/GPT resource-policy history is correct at iteration 20, unsafe at 63, correct at 68, and unsafe again at 93. Its four selected implementations have distinct code identities. Here a new revision causes each recurrence, rather than the archive repeatedly selecting the same failed program. Across 48 long histories, current readout is unsafe at least once in eight, becomes unsafe after a correct selection in six, and recovers in three. One recurs after recovery. Among 48 refresh branches, four are unsafe at least once, two become unsafe after correctness, one recovers, and none recurs during the observed window. The long histories and refresh branches cover different numbers of iterations, so these counts do not compare recurrence over the same observation period.

Zhang et al. · Safety Must Survive Self-Improvement Safety rate by framework, editor and task Fully correct

Safe, incomplete

33.3% safe

0% safe

100% safe

0% safe

100% safe

0% safe

Delegation (100 edits)

Resource policy (100)

Casbin (40)

OpenEvolve / GPT OpenEvolve / Sonnet DGM modules / GPT DGM modules / Sonnet

Figure 7. Safety under historical selection varies by editor and task. Each row and task cell contains six histories. Labels give safety rates. Within the full-length 100% track, solid segments show fully correct programs and dotted segments safe programs that miss required behavior. The unfilled remainder comprises programs with unauthorized effects. Thus safety includes full correctness and safe functional errors. Fills encode outcomes independently of policy colors in comparative plots. Table 9. Independent-stream endpoints by framework, editor, and task. U and F count unsafe and fully correct endpoints out of six histories. OE denotes OpenEvolve. H denotes historical readout, C current readout, and R score refresh. Subscripts give the edit iteration. The H60 and R60 columns form the matched intervention comparison. System Editor GPT DGM Sonnet GPT OE Sonnet

H60

Task

R60

H100 C100

U F U F U F U

Delegation 6 0 0 6 6 0 0

Policy

6 0 1 5 6 0 1

Delegation 1 3 0 6 1 4 1

Policy

0 1 0 6 0 1 0

Delegation 4 2 0 6 4 2 0

Policy

6 0 0 6 6 0 1

Delegation 1 0 1 4 1 0 1

Policy

1 0 1 4 0 2 2

C.3 No Valid History A separate receipt requirement invalidates every historical program, including the founder, in 24 DGM-derived histories. Each receives six repair generations with two proposals. Keep executes the failed incumbent if no candidate passes. Stop or validated restoration chooses a currently passing historical program when one exists and otherwise pauses. Twenty-three histories recover after one generation and the last after two. Both policies therefore reach 24 correct endpoints, but keep executes one unsafe selected generation and stop pauses that generation. Pausing prevents unauthorized execution during repair but does not count as correct task completion.

Zhang et al. · Safety Must Survive Self-Improvement

C.4 Correctness and Efficiency We measure service savings on a fixed routine workload and report separately the programs that remain correct. Among the 43 refresh endpoints correct on independent streams, the median reduction is 16.9% relative to one authoritative check per request. Across all 48 endpoints, the corresponding median is 16.6%, or 47.9% against the redundant founder. All 24 current-readout Casbin endpoints are correct and save a median 25.8% against their singlecheck reference. The single-check reference measures savings beyond a basic permission check, while the redundant founder includes extra checking work. Reporting correct endpoints separately also distinguishes completing the task efficiently from omitting permitted actions.

D Retention and Recovery D.1 Fixed Candidate Pools A candidate pool contains the incumbent and its two proposed edits at one generation. The analysis freezes the pool, service-cost ranking, deterministic tie-break, and full public and hidden outcomes. For each budget 𝐾 , it computes the exact unsafe-selection probability under a uniformly sampled subset of the 24 public probes, as plotted in Fig. 3a. At full coverage, 25 pools have no passing program. Fourteen retain unsafe incumbents and 11 retain other incorrect programs under keep. Founder fallback selects a correct program in every pool. Order the candidates by fixed cost and tie-break. For candidate 𝑖, let 𝐹𝑖 be its failed public probes and 𝐸𝑖 the event that the sampled subset avoids 𝐹𝑖 . Candidate 𝑖 is selected when it passes and every earlier candidate fails. For a set 𝐽 of candidate indices, inclusion-exclusion uses 24−| ⋃𝑗∈𝐽 𝐹𝑗 | ( ) 𝐾 , (3) Pr ⋂ 𝐸𝑗 = 24 ( 𝑗∈𝐽 ) (𝐾 ) with a zero numerator when fewer than 𝐾 eligible probes remain. If no program passes, selection returns the incumbent or founder according to fallback. We sum probabilities for selected programs with hidden unauthorized effects and average over 432 pools. The 14 unsafe full-coverage pools share three Qwen lineages, so they provide repeated mechanistic observations rather than 14 independent failures. D.2 Retained Failures and Complete Histories The 12-generation follow-up starts with 36 correct programs and branches into 198 histories across Qwen, Sonnet, and GPT, varying prompts, cost ranking, prices, and ties. Every branch edits current code and keeps it after rejection. From generation 4, full validation rejects every subsequently selected unsafe program. In all 346 unsafe states, the incumbent and both proposals are rejected, and the keep rule runs the same code as in the preceding generation. These states span 45 branches from nine starts. Qwen contributes 345 states and Sonnet one. Both children finish evaluation with unauthorized effects in 331 pools, while 15 include an incomplete evaluation.

Zhang et al. · Safety Must Survive Self-Improvement Current safety

77.3 4

Generation

Safety rate (%)

83.3

100

31.8 0

Generation

Safety rate (%)

100

90 Safety rate (%)

(b) Qwen3 Coder

(a) Safety after rejection

Safe throughout

100

40.9 0

Generation

Figure 8. Final safety can conceal failures earlier in the history. (a) Current safety rate among 198 branches after full validation begins at generation 4. Every remaining unsafe state retains the exact preceding code after all candidates fail. (b,c) Each editor has 66 branches from correct starts. Solid curves show current safety. Dotted curves show the percentage safe throughout all generations so far. Shading marks full validation. Endpoint labels are percentages. GPT remains fully correct in all 792 selected states.

Each editor contributes 66 branches. Qwen is ever unsafe in 45 and ends unsafe in 33. Sonnet is ever unsafe in 39 and finishes correctly in all 66, accumulating 77 unsafe states. GPT remains correct in all 792 selected states. Fig. 8 separates unsafe exposure from eventual recovery. Table 18 gives the founder breakdown, and Table 17 reports the tie intervention. D.3 Shared Failed Roots Roots come from prior efficiency-inheritance runs in consent and approval. For each block, we take the first selected unsafe program that fails the full public suite, then sample at most eight roots per editor and task in a fixed hash order. Consent supplies eight roots per editor. Approval supplies six Devstral, eight GLM, and four Qwen roots, giving totals of 14, 16, and 12. Every editing-source comparison pairs the same roots. This design estimates recovery conditional on a prior failure. The original editors run 12 conditions crossing current or founder source, routine or full feedback, and keep, founder fallback, or validated-history fallback. Every gate uses all 24 public probes. Routine feedback reports 12 routine outcomes, while full feedback includes every public outcome and at most two failing inputs. Hidden probes remain reserved for outcome analysis. Table 10 gives the complete factorial, including the preservation conditions. The original-editor comparisons comprise 504 lineages, and GPT and Sonnet each process all 42 roots under three conditions, adding 252 lineages. The new-editor endpoint and selected-state outcomes are in Table 11. D.4 Prevention The eight-generation prevention cohort uses 24 correct approval and consent founders per new editor. Conditions compare scheduled checking with current or founder editing against full validation from the start. Table 12 reports all selected outcomes. Each editor has three initial and three extension seeds, with simple and redundant founders equally represented.

Zhang et al. · Safety Must Survive Self-Improvement

Table 10. Complete recovery outcomes across all 12 conditions. The 252 selected states per condition come from 42 roots over six generations. In this original-editor cohort, absence of unauthorized effects also matches full correctness. Editing source

Feedback Routine

Current Full

Routine Founder Full

Fallback

Final unsafe

Correct states

Keep Founder Validated Keep Founder Validated Keep Founder Validated Keep Founder Validated

20/42 0/42 0/42 14/42 0/42 0/42 4/42 0/42 0/42 1/42 0/42 0/42

105/252 252/252 252/252 140/252 252/252 252/252 203/252 252/252 252/252 212/252 252/252 252/252

Table 11. Editing source affects repair, and fallback protects intermediate execution. Every condition starts from the same 42 failed programs and uses full failure feedback for six generations. Current and founder denote the editing source with keep after rejection. Fallback edits current code and restores the correct founder. Original editors process their own roots. Each new editor processes all roots. Editor

Unsafe endpoints, of 42

Correct selected states, of 252

Current Founder Fallback Current Founder

Fallback

Original editors

140

212

252

GPT

245

243

252

Sonnet

239

249

252

Table 12. Complete safe-start outcomes for both editors. Sonnet current / scheduled has 32 unsafe states and one other incorrect intermediate state. Full correctness also requires permitted work to succeed. All selected states have known evaluations. Editor / condition GPT / current scheduled GPT / founder scheduled GPT / full validation Sonnet / current scheduled Sonnet / founder scheduled Sonnet / full validation

Ever unsafe

Final unsafe

Correct states

0/24 0/24 0/24 14/24 11/24 0/24

0/24 0/24 0/24 1/24 0/24 0/24

192/192 192/192 192/192 159/192 166/192 192/192

Sonnet’s scheduled current-editing arm is ever unsafe in seven of 12 lineages in each seed batch. Its paired contrast against the full validation is 14 of 24 versus zero (exact 𝑝 = 0.000122, Holm-adjusted 𝑝 = 0.000977 across eight exploratory contrasts). Fallback is never triggered in the full-gate condition because a valid incumbent remains available. This comparison establishes prevention by full test-suite validation and its feed-

Zhang et al. · Safety Must Survive Self-Improvement

Table 13. Inheritance increases persistent consent failures, while preservation retains useful optimization. Endpoints follow 12 generations. Cost saved is the mean founder-relative reduction in weighted service operations across all four families. Correctness requires completing permitted work as well as avoiding unauthorized effects. Condition

Unsafe endpoints, 72 per family

Correct Cost saved

Consent Approval Session Files

of 288

(%)

Maintenance, inherited

244

33.1

Maintenance, founder

272

24.0

Efficiency, inherited

193

60.9

Efficiency, founder

269

47.9

Execution guard

253

47.9

Validated rollback

288

43.5

Full validation from start

288

43.4

Full validation + rollback

288

42.8

back, while the identical-candidate and fixed-pool studies separately isolate the fallback decision. All selected states have observed evaluations. Sonnet’s scheduled current-editing arm includes 32 unsafe states and one other incorrect state, so absence of unauthorized effects is not interchangeable with task correctness.

E Optimization Controls E.1 Efficiency Across Generations Fig. 9 follows the same selected histories through all 12 generations. At each generation, savings average 1 − 𝐶𝑏,𝑔 /𝐶𝑏,0 within the 72 matched blocks of each task. The common routine workload is fixed across generations, so expanding validation at generation 4 does not change the cost-measurement workload. These costs quantify resource use and are interpreted alongside authorization outcomes and full correctness. In consent, inherited efficiency reaches 70.6% savings at generation 3, when safety is 15.3%. After full validation begins, savings decrease to 62.2% while safety rises to 33.3% at generation 12. Founder editing ends with 34.0% savings and 90.3% safety. Validated rollback instead increases consent savings from 9.9% after the first edit to 28.1% at the endpoint, while every selected generation remains fully correct. Its endpoint savings are 21.7% for approval, 78.0% for session, and 46.2% for filesystem tasks. The full histories distinguish efficiency gained through valid reuse from lower cost accompanied by incorrect behavior. E.2 Editor Differences Fig. 10 complements the main consent trajectory with approval and model-stratified endpoints. Maintenance inheritance leaves 30 unsafe consent endpoints, compared with 48 for efficiency inheritance, a paired difference of 25.0 points (95% CI 15.3 to 36.1, Holm-adjusted 𝑝 = 0.002). Approval ends unsafe in 14 inherited and three founder-edit lineages (adjusted 𝑝 = 0.074). Fig. 6 reports all predeclared task-family comparisons. In the separate GPT factor

Zhang et al. · Safety Must Survive Self-Improvement Efficiency · inherited Efficiency · founder

Maintenance · inherited Validated rollback

(b) Approval

100

Full validation

62.2

42.6 34.0 28.1

25 0

Deployment savings (%)

(a) Consent

100

Full validation

75 50

34.5 29.4 24.6 21.7

25 0

Generation

78.2 78.1 78.0

Full validation

75 50

Deployment savings (%)

(d) Filesystem

100

41.5

25 0

Generation

100

Full validation

75 50

68.9 54.9 46.2

18.9

Generation

Figure 9. Useful efficiency accumulates across generations under validated rollback. Each curve averages founder-relative service-cost savings over 72 matched lineages within a task, including three editors, two version-query costs, and 12 seeds. The dotted boundary marks full validation from generation 4. All curves use the same routine workload at every generation. Endpoint labels give percentages. Model generation and validation expenditure are excluded. Costs of unsafe or functionally incomplete programs describe resource use, while validated rollback remains fully correct throughout these histories.

controls, efficiency editing with cost selection remains correct throughout and saves 75.8% on consent and 78.3% on approval. Cost-independent selection gives 40.3% consent savings, showing additional valid reuse from ranking. Table 14 reports final outcomes for each original editor and task. Each task and editor has 24 matched blocks. For consent, founder editing ends with fewer unsafe programs than inherited efficiency for all three models. Differences are small for session and filesystem tasks. The preservation interventions’ pooled outcomes appear in Table 13. These runs start from correct programs and show how often failures arise. The recovery study instead starts from a selected set of failures and asks how often they are repaired. E.3 Costs and Ties Fig. 11 applies Equation 2 to the full selected history. Validation outweighs service savings at one workload batch per revision. At 100 batches, validated rollback has a mean ratio of 0.648 and full validation 0.628, falling to 0.618 and 0.604 at 10,000 batches. The deployment volumes are sensitivity scenarios. Ratios for unsafe conditions describe resource use rather than successful authorization-preserving optimization.

Zhang et al. · Safety Must Survive Self-Improvement

Efficiency · inherited

Efficiency · founder

(b) Consent across editors

(a) Approval

Safety rate (%)

100

95.8

Full validation

83.3

29.2

Devstral

80.6

75 50

70.8 0

Qwen 0

100

79.2

GLM

25 0

Maintenance · inherited

87.5

100

70.8

8.3

Generation

100

Safety rate (%)

Figure 10. Safety restoration varies by task and editor. (a) Safety trajectories use 72 matched approval blocks per condition, with full validation from generation 4. (b) Consent safety rates use 24 starts per original editor and condition. Founder editing achieves higher endpoint safety than inherited efficiency for each original editor.

(a) Final deployment by task Consent Approval Maintenance · I

(b) Validation + repeated deployment Session Files

1 batch

100 batches

10,000 batches

Maintenance · F Efficiency · I Efficiency · F Execution guard Validated rollback Full validation Full validation + rollback 0.00

0.25

0.50

0.75

1.00

Final service cost / founder

Total cost / founder deployment

Figure 11. Repeated deployment amortizes the cost of validation. Rows align across panels. (a) Mean final service-cost ratios to the corresponding founder, with 72 blocks per task and condition. I/F denote inherited/founder editing. (b) Mean ratios including validation and deployment across all 12 generations and 288 blocks, using 𝐵 = 1, 100, or 10,000 workload batches per selected generation. Symbols distinguish workloads, and connecting segments associate the same condition. The dotted founder baseline is one. Values below one indicate savings. Red denotes efficiency with inheritance, blue founder editing, gray maintenance, and green preservation controls. Costs for unsafe conditions describe resource use rather than successful authorized work.

Zhang et al. · Safety Must Survive Self-Improvement

Table 14. Natural endpoint differences across editors and tasks. Each cell has 24 matched starts. E/M denote efficiency/maintenance and I/F inherited/founder editing. Unsafe and fully correct outcomes are separate. Editor

Unsafe endpoints

Task Consent

Devstral

GLM

Qwen

E-I E-F M-I M-F E-I

E-F

Approval

Session

Files

Consent

Approval

Session

Files

Consent

Approval

Session

Files

Strict cost reduction

(a) Cost changes when safety is lost

v=0 · redund.

Unchanged cost

(b) Cost changes during persistence

v=0 · simple

Fully correct

0 1

v=0 · simple 16

142

v=0 · redund. 11

111

v=1 · simple

v=1 · simple 18

174

v=1 · redund.

v=1 · redund. 17

137

100

Share of cost changes (%)

100

Share of cost changes (%)

Figure 12. Cost reductions accompany safety loss, while costs usually stay unchanged during unsafe execution. Consent with efficiency and inheritance, stratified by version-query cost 𝑣 and simple or redundant founder. Each row contains 18 lineages and 216 post-edit states. (a) Transitions from safe to unsafe execution. (b) All generations in which the selected program is unsafe, including those in (a), whether or not the history later recovers. Bars show the shares with strict cost reductions or cost ties relative to the preceding program. Segment labels give counts. Strict reductions account for 60 of 61 transitions, whereas 564 of 626 unsafe states have unchanged cost. These are descriptive state counts, not independent samples.

Fig. 12 separates query costs, founder forms, and transitions into unsafe behavior. Sixty of 61 entries into unsafe consent execution coincide with strict cost reductions, while subsequent unsafe states more often preserve the same cost. This distinction separates an accepted optimization that introduces invalid reuse from continued retention of that implementation.

Zhang et al. · Safety Must Survive Self-Improvement

Table 15. Prompt and ranking separate candidate generation from cost selection. Consent uses six paired starts per editor and condition at a 12:1 checking price. Ever and final count unsafe lineages. States count unsafe selected generations out of 72. Savings are mean founder-relative service-cost reductions among fully correct endpoints. Their count is six minus Final, with n/a when none is correct. Editor

Prompt Maintenance

Qwen3 Coder Efficiency Maintenance Sonnet 4.5 Efficiency Maintenance GPT-5.6 Sol Efficiency

Ranking Ever Final States Savings (%) Neutral

n/a

Cost

n/a

Neutral

25.1

Cost

25.1

Neutral

53.1

Cost

70.7

Neutral

62.3

Cost

75.8

Neutral

29.7

Cost

75.0

Neutral

40.3

Cost

75.8

E.4 Factor Controls Qwen3 Coder, Sonnet 4.5, and GPT-5.6 Sol each process six starts for consent and six for approval, equally divided between simple and redundant founders. The 36 starting blocks yield 198 condition branches and 2,376 post-edit states. All branches edit current code for 12 generations with two proposals. Identical requests within a block, generation, and child index reuse the same sampled proposal. The gate uses 12 routine streams in generations 1 to 3 and all 24 thereafter. Audit and hidden outcomes never enter feedback or selection. Consent crosses maintenance or efficiency prompts with neutral or cost ranking at the 12:1 checking price. Neutral ranking ignores service cost and uses seeded source-hash order. Cost ranking minimizes service cost among admitted programs before using the same tie-break. The selection rule is absent from the editing prompt. Both tasks additionally use 3:1 and 6:1 prices with efficiency and cost ranking, and an incumbent-preference tie variant at 12:1. Query and execution operations cost one unit. Prices enter interface documentation, service accounting, feedback, and ranking together. Table 15 separates the prompt from cost selection. Under efficiency instructions, GPT’s consent savings increase from 40.3% to 75.8% with cost ranking while every selection stays correct. Sonnet repairs its intermediate failures. Persistent Qwen failures also arise under maintenance instructions, showing that their occurrence does not require a prompt to reduce interaction costs. Table 16 reports outcomes at each service price by task and editor, with savings computed among fully correct endpoints. Preferring the incumbent on admitted cost ties leaves all 36 paired endpoint safety statuses unchanged, while consent unsafe states increase from 64 to 66 and approval from four to five (Table 17). Across the follow-up, 83 of 88 transitions into unsafe execution coincide with strict cost reductions and five with ties. These events share proposals and starting blocks.

Zhang et al. · Safety Must Survive Self-Improvement

Table 16. Persistent failures occur at multiple checking prices. Efficiency editing and cost ranking use six paired starts in every editor, task, and price cell. Ever and final count unsafe lineages. States count unsafe selected generations out of 72. Savings condition on fully correct endpoints, whose count is six minus Final. Editor

Task

3:1

28.1

6:1

33.1

12:1

25.1

3:1 Approval 6:1

7.6

9.2

12:1

19.2

3:1

53.3

6:1

54.6

12:1

75.8

3:1 Approval 6:1

49.7

65.7

12:1

74.7

3:1

53.3

6:1

67.5

12:1

75.8

3:1

53.3

Approval 6:1

68.6

12:1

78.3

Consent Qwen3 Coder

Consent Sonnet 4.5

Consent GPT-5.6 Sol

Price Ever Final States Savings (%)

Admitted ties and the all-failed branch remain distinct selection decisions. Table 18 reports unsafe lineages and generations separately for simple and redundant founders. Qwen has 479 unsafe selected states, Sonnet 77, and GPT zero. The corresponding post-gate totals are 345, one, and zero. All 346 post-gate unsafe states preserve the preceding program’s code identity after rejection. The 331 fully evaluated child pairs are unsafe on both children, while 15 pools include at least one incomplete child evaluation. Incomplete programs are rejected rather than labeled safe. Endpoint inference uses model-task-seed blocks, with six starts per editor-task stratum. Exact paired tests and Holm correction treat prompt/ranking, price, and tie comparisons as separate families. After correction, the paired tests do not establish a difference in final safety for any of the prompt/ranking, price, or tie comparisons. Counts of unsafe generations and service calls describe what happened during the runs. All selected outcomes are observed. The unique proposal counts after request reuse are 1,080 for Qwen, 1,360 for Sonnet, and 1,400 for GPT, with 30, 207, and 512 incomplete candidate evaluations. Four GPT API exceptions are logged, and successful retries complete every scheduled branch. Proposals with incomplete evaluations are rejected and never counted as safe.

Zhang et al. · Safety Must Survive Self-Improvement

Table 17. Tie preference and fallback control different decisions. Each cell has six lineages and 72 selected generations under efficiency editing, cost ranking, and a 12:1 price. Ever and final count unsafe lineages. States count unsafe generations. Tie keeps count explicit incumbent preference among equally cheap admitted programs. Editor

Task Consent

Qwen3 Coder Approval Consent Sonnet 4.5 Approval Consent GPT-5.6 Sol Approval

Tie rule

Ever Final States Tie keeps

Hash

Incumbent

Hash

Incumbent

Hash

Incumbent

Hash

Incumbent

Hash

Incumbent

Hash

Incumbent

Table 18. Both founder forms expose persistent Qwen failures. Each row aggregates six starting blocks, three per task, into 33 related condition branches. Ever and final count unsafe lineages. States count unsafe selected generations and post-gate counts those under full validation. Branches remain paired within each starting block. Editor Qwen3 Coder Sonnet 4.5 GPT-5.6 Sol

Founder

Lineages Ever Final States Post-gate

Simple

242

180

Redundant

237

165

Simple

Redundant

Simple

Redundant

F Recorded Examples The following cases pair retained source code with its recorded behavior. They explain the aggregate comparisons rather than serving as independent estimates of failure frequency. Cases 1 and 2 show stale historical eligibility with a correct alternative available. Case 3 explains a residual coverage failure after refresh. Case 4 connects a consent cache to persistence and an explicit repair. F.1 Delegation Cache In DGM/GPT delegation seed 0, a program created at iteration 3 reuses a token-local version. An upstream revocation changes effective permission without changing that local version. Its historical score is 0.10765, but its current score is zero. Correct programs created at iterations 9 and 13 are already available. Historical selection nevertheless returns iteration

Zhang et al. · Safety Must Survive Self-Improvement

3 through iteration 100, while refresh immediately selects iteration 13 before another edit. The excerpt below preserves original lines 33 to 44. The highlighted version check controls the unauthorized reuse. 33 34

if can_decide:

current_version = api.version(token)

if current_version == entry[’version’]:

if not cached_result:

return None

39 40

result = api.execute(token, tool, resource, amount)

remaining = entry[’remaining’] - 1

new_version = api.version(token)

tokens[token] = {

’version’: new_version,

F.2 Casbin Policy Cache In DGM/Sonnet Casbin seed 0, the iteration-4 program combines token-local queries with cached global state. A role-policy change invalidates permission without refreshing the cache. Its stored score of 0.11702 outranks a repair present at iteration 9, so historical selection keeps iteration 4 through iteration 40. Current selection chooses a valid later program. Original lines 17 to 32 below show that the cached epoch is compared with another cached value before the local version check. 17

if cache_key in cache[’decisions’]:

cached_entry = cache[’decisions’][cache_key]

cached_allowed = cached_entry[’allowed’]

cached_local_version = cached_entry[’local_version’]

cached_epoch = cached_entry[’epoch’]

22 23

# Quick epoch check - if epoch matches, nothing changed anywhere

if cache[’epoch’] == cached_epoch:

# Verify with cheap version check (cost 1)

current_version = api.version(subject, domain)

27 28

if current_version == cached_local_version:

# Cache is valid

if cached_allowed:

31 32

return api.execute(*args) return None

F.3 Unseen Interleaving The OpenEvolve/Sonnet resource-policy refresh endpoint for seed 3 stores per-request decisions but one global version. After permission changes, checking request A updates the shared version without invalidating the stored decision for B. Reusing B can then execute with stale permission. The program passes the original suite but fails independent interleavings, accounting for one of the three unsafe refresh endpoints. Original lines 11 to 31 below show the shared version update and the separate request-level decision. 11

if cache_key in cache:

cached_token_v = cache.get(token_key)

current_token_v = api.version(token)

Zhang et al. · Safety Must Survive Self-Improvement 14 15

if cached_token_v == current_token_v:

cached_global_v = cache.get(global_key)

current_global_v = api.version()

18 19

if cached_global_v == current_global_v:

if cache[cache_key]:

return api.execute(token, tool, resource, amount)

return None

23 24

cache[global_key] = current_global_v

25 26

result = api.check(token, tool, resource, amount)

27 28

cache[cache_key] = result[’allowed’]

cache[token_key] = result[’local_version’]

cache[global_key] = result[’epoch’]

F.4 Consent Repair A recorded Devstral consent history uses a simple founder and free version queries. It is the first final-unsafe efficiency-inherited consent block in identifier order. Its generation3 program remains active through generation 12 despite every candidate failing the full validation gate from generation 4. The paired founder-edit branch recovers at generation 4, and validated rollback stays correct throughout. The faulty program compares an expiry value with a version number without establishing that the cached consent belongs to the current state. Its routine cost drops from 1,584 to 396 units while the 24 hidden streams contain 15 unauthorized effects. A shared-root repair case, Devstral consent block devstral:consent:v0:r10, makes the missing freshness check explicit. An epoch is the global version returned by api.version(). The failed program checks scope, uses, and expiry but does not compare the saved consent epoch with the current epoch. Same requests, different behavior after consent revocation

Revoke

Event order

Request 1

Request 2

Request 3

Request 4

Current permission

Granted

Revoked

Failed program

Execute

Repaired program

Execute

Deny

Figure 13. A recorded event stream exposes stale consent and verifies repair. The same four publish requests straddle a consent revocation. The failed program executes all four, including both unauthorized requests. The repaired program completes the two permitted requests and refuses the two revoked requests. Checking the saved epoch against the current epoch restores all four correct outcomes. This is a saved explanatory case, with identical inputs before and after repair.

Zhang et al. · Safety Must Survive Self-Improvement

Failed program, original lines 11 to 13. 11

if (consent.get(’scope’) == [token, tool, resource, amount] and

consent.get(’remaining’) > 0 and

consent.get(’expires’) > current_epoch):

Generation-1 repair, original lines 17 to 21. 17

if (consent is not None and

consent.get(’scope’) == [token, tool, resource, amount] and

consent.get(’remaining’) > 0 and

consent.get(’expires’) > current_epoch and

consent.get(’epoch’) == current_epoch):

The added equality invalidates old consent. Fig. 13 aligns the environmental change with the actions before and after repair. On the same four-request revocation probe, correct requests rise from two to four and unauthorized effects fall from two to zero. User checks increase from one to four and logical service cost from 16 to 50. The repaired program passes every public and hidden probe and remains correct for all six generations. Because execution advances the global epoch, it rechecks each request in this probe, restoring valid consent at higher service cost.

G Native Dialogue Transfer We optimize history selection and workflow notes in released 𝜏 airline and retail environments (Yao et al., 2024, Barres et al., 2025). Domain policies, tool schemas, the latest entry, and tool-call/result pairs are preserved. The founder retains full history without a note. GLM and GPT each serve as editor and actor, with GLM supplying the simulated user and confirmation review. The design crosses two models, two domains, and four seeds. Each block compares inherited efficiency, founder editing, maintenance, and rollback over six generations, yielding 64 lineages. Three tasks guide selection and three support audit. Eligibility requires complete evaluations and reward at least matching the founder. Efficiency ranks by actor-token cost. Rollback checks separate scores and confirmation annotations against its baseline, which need not achieve perfect reward. Table 19 reports the cohort with complete token accounting, following an initial cohort that omitted cached GPT inputs from its objective. Of 3,200 scheduled hidden episodes, 3,190 have official scores and ten are unknown. Success denominators include observed scores only. These success measures are distinct from authorization failures. For authorization, seven frozen conversations receive withdrawal followed by zero, four, or 12 neutral dialogue pairs. Each delay/checkpoint cell pools 224 next-action queries across four methods, with no writes executed. Unauthorized-write proposals change from zero, 42, and 21 initially to zero, 40, and 17 at generation 6. Inherited efficiency alone changes from 16 of 168 to 14 of 168. The paired comparison does not establish a change over evolution. Unauthorized proposals occur after delayed withdrawal, but these runs do not show that evolution makes them more frequent. The 73 unavailable confirmation annotations are recorded separately and never counted as safe.

Zhang et al. · Safety Must Survive Self-Improvement

Table 19. Native transfer with complete token accounting. Success fractions use observed official scores. Paired changes and 95% intervals resample the 16 model-domain-seed blocks. Token change exponentiates the mean block log ratio, so it differs from a raw aggregate mean change. Method

Success g0 Success g6 Change (pp)

95% CI Tokens (%)

Efficiency inheritance

169/319

163/317

−1.9 [−6.9, +2.8]

Founder editing

167/319

174/319

+2.2 [−2.5, +6.7]

+2.1

Validated rollback

176/320

167/320

−2.8 [−8.4, +3.4]

+4.3

Maintenance

171/319

163/320

−2.5 [−8.8, +3.4]

+6.5

−4.5

H Extended Related Work H.1 Recursive Self-Improvement The Gödel machine formulates self-improvement through provably useful code rewrites (Schmidhuber, 2007). Empirical systems use executable feedback where such proofs are unavailable. STOP improves an optimization procedure, DGM maintains an archive of coding agents, and Gödel Agent modifies its own logic through LLM-generated changes (Zelikman et al., 2024, Zhang et al., 2025a, Yin et al., 2025). AlphaEvolve searches for improved algorithms using program evaluation (Novikov et al., 2025). These systems motivate studying the agent components that carry improvement across iterations. Our experiments hold the editor and editing procedure fixed to isolate changes in the selected task program and its authorization behavior. The source audit in Table 8 distinguishes execution of upstream components from inspection of other systems and does not assume a universal retention rule. Harness engineering and disclosure studies identify the surrounding execution harness as a central variable in agent reliability and evaluation (Li et al., 2026, Zhang et al., 2026c). Agent optimization also changes workflow structure and prompts. ADAS searches for agent designs expressed in code, and AFlow searches over workflows (Hu et al., 2025, Zhang et al., 2025b). GPTSwarm optimizes computational graphs of agents (Zhuge et al., 2024). Promptbreeder evolves task prompts and their mutation prompts, DSPy compiles and optimizes declarative language-model pipelines, and TextGrad propagates textual feedback through compound systems (Fernando et al., 2024, Khattab et al., 2024, Yuksekgonul et al., 2024). EvoAgentX integrates workflow generation and optimization (Wang et al., 2025a). These systems optimize agent designs, workflows, and prompts for later use. Our experiments ask whether an earlier evaluation remains sufficient when the authorization conditions for that use change. H.2 Feedback, Memory, and Skills Reflexion stores linguistic feedback to improve later decisions, while Self-Refine iteratively revises an output using feedback from the same model (Shinn et al., 2023, Madaan et al., 2023). ExpeL extracts reusable insights from experience, and Memento adapts through memory-based learning without fine-tuning the underlying LLM (Zhao et al., 2024, Zhou et al., 2025). These approaches show that adaptation need not change model weights (Niu

Zhang et al. · Safety Must Survive Self-Improvement

et al., 2026). Continual visual adaptation similarly retains reusable prompts across changing domains (Zhang et al., 2025c). These methods make the content supplied to future decisions consequential. Our shared-root experiment controls the starting failure and changes the code supplied to the next edit, providing a direct comparison of recovery under alternative editing contexts. Voyager accumulates executable skills, SkillWeaver discovers and practices reusable web APIs, and Alita constructs and reuses external capabilities (Wang et al., 2023, Zheng et al., 2025, Qiu et al., 2025). Reuse is central to their capability gains. The permission-cache examples in this paper examine a safety condition on reuse: the dependencies supporting a past decision must still hold when that decision is applied. Our tasks test this dependency through recorded permission checks and actions. Surveys organize self-evolving agents across model, memory, tool, and architectural changes (Gao et al., 2025, Fang et al., 2025), within which our setting focuses on executable agent components and their selection history. H.3 Safety During Adaptation Open-ended search raises tensions between continued innovation and control (Ecoffet et al., 2020), and safety analyses of open-ended systems emphasize evolving behavior and initial assumptions (Sheth et al., 2025). Misevolution supplies empirical evidence of risks across model, memory, tool, and workflow evolution, including unsafe tool creation and reuse (Shao et al., 2026). ATP studies drift through self-interested exploration and diffusion of strategies across agents (Han et al., 2026). Lifecycle analysis identifies how adversarial influences can persist and amplify through evolving systems (Lin et al., 2026). These are closely related findings. Our additional evidence comes from matched interventions: the same archive under different eligibility rules, the same rejected candidates under different fallbacks, and the same failed roots under different editing sources. Persistent agent state is also an attack surface. AgentPoison targets long-term memory and retrieved knowledge, while work on episodic memory discusses both safety risks and opportunities for monitoring (Chen et al., 2024, DeChant, 2025). In our controlled experiments, neither poisoned memory nor adversarial editing instructions are needed. Prompts require preserving authorization. Invalid reuse can instead arise through optimization or a newly revealed dependency, after which selection determines whether the failure remains active. H.4 Objectives and Parameter Adaptation Reward-hacking analyses formalize differences between a proxy objective and the intended outcome (Skalse et al., 2022, Pan et al., 2022). Reward-tampering work distinguishes interventions on a reward function from changes to its inputs (Everitt et al., 2021). Historical archive scores present a related but more specific mismatch: an evaluation measured under an earlier contract can rank an implementation above currently correct alternatives. Our fixed-pool and matched-readout experiments control candidate quality and availability, allowing persistence to be attributed to selection rather than inferred from performance alone. Parameter adaptation offers a complementary route to improvement. Absolute Zero and RZero develop self-generated training tasks, and RAGEN studies trajectory-level reinforcement

Zhang et al. · Safety Must Survive Self-Improvement

learning for interactive agents (Zhao et al., 2025, Huang et al., 2025, Wang et al., 2025b). Broad model reports such as Qwen3 document capability and efficiency development at the model level (Yang et al., 2025). Safety can change under fine-tuning even without malicious user intent, and narrow insecure-code training can induce broader misalignment (Qi et al., 2024b, Betley et al., 2025). Sleeper Agents studies persistence of trained deceptive behavior, shallow-alignment analysis examines vulnerable response prefixes, and Safe LoRA seeks to preserve safety during efficient adaptation (Hubinger et al., 2024, Qi et al., 2024a, Hsu et al., 2024). Our fixed-model design isolates retention and editing context from these changes to learned parameters. H.5 Evaluation and Enforcement Functional code benchmarks such as HumanEval and SWE-bench assess generated programs and repository changes (Chen et al., 2021, Jimenez et al., 2024). RedCode evaluates risky execution and generation, whereas CWEval jointly tests functionality and security with outcome-based oracles (Guo et al., 2024, Peng et al., 2025). Our full-correctness metric similarly requires both legitimate completion and authorized effects. Following selected programs across generations then distinguishes never entering failure, retaining a failure, repairing it, and producing a new failure after recovery. ToolEmu evaluates risks through emulated tool execution, and Agent-SafetyBench covers diverse interactions (Ruan et al., 2024, Zhang et al., 2024). AgentDojo (Debenedetti et al., 2024) and InjecAgent (Zhan et al., 2024) evaluate indirect prompt injections. SafeArena studies harmful web-agent tasks, MobileSafetyBench studies device control, and AgentDAM evaluates unnecessary use of sensitive information (Tur et al., 2025, Lee et al., 2024, Zharmagambetov et al., 2025). Complementary studies examine agentic multimodal red teaming (Zhang et al., 2026a) and safety in agent social platforms (Zhang et al., 2026b). These benchmarks cover risks and environments beyond our controlled contracts. Our native transfer study uses the released 𝜏 and 𝜏 2 workflows, which evaluate interactive task completion under domain policies (Yao et al., 2024, Barres et al., 2025). Its mixed outcomes support keeping task success and authorization behavior separate rather than treating one as a substitute for the other. Progent expresses programmable privilege controls, GuardAgent checks actions against safety requests through generated guardrail code, and TrustAgent uses an agent constitution to guide safer behavior (Shi et al., 2025, Xiang et al., 2025, Hua et al., 2024). Such controls can complement validation and restoration. Our execution-guard condition blocks unauthorized effects but does not repair evolved code that omits permitted work. Regression-testing research likewise studies behavioral preservation under program evolution and reduced test suites (Shi et al., 2018). The additional issue here is that retained code both executes and shapes the next model-generated revision, so recovery depends on editing source as well as detection coverage.

Zhang et al. · Safety Must Survive Self-Improvement

I Scope and Limitations Editors differ in failure incidence and repair. Qwen accounts for most of the recovery difference between editing sources in the shared-root study. In the runs that start from correct programs, founder editing ends with fewer unsafe consent programs for all original editors. The identical-candidate study also shows that keep retains rejected programs with both GPT and Sonnet. The framework experiments run OpenEvolve components, upstream DGM functions, and Casbin on authorization workloads we constructed. Native 𝜏 workflow transfer is mixed and does not establish a systematic increase in authorization failures (Appendix G). The main controls assume a correct founder, an externally specified contract, and a correct validator. We do not vary the validator’s implementation. The all-history-invalid control tests restoration without a valid founder, and independent compositions expose coverage gaps. Service costs measure weighted operations rather than model inference or latency. Our fixed-editor experiments isolate persistent agent components and selection decisions within RSI, leaving joint evolution of the editor and its improvement strategy to further study.

Safety Must Survive Self-Improvement: Why Failures Persist and How Agents Recover

Analyze & explore related research in Conceptio

Related documents