Repo2Skill-Evo: Repository Skills Go Stale in Silence

Repo2Skill-Evo: Repository Skills Go Stale in Silence

1

ByteDance, 2 Peking University, 3 Beijing Jiaotong University

arXiv:2608.21964v1 [cs.AI] 22 Aug 2026

Full author list in Contributions

Abstract Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, which scripts to run, and which conventions the current release expects. Agent skills externalize this knowledge into reusable units, and prior work shows that they can improve agent performance. What remains unclear is whether that improvement is durable. The same version specificity that makes a skill useful also makes it fragile: after a release, it may become stale without raising any explicit signal, while continuing to provide obsolete guidance. Externalizing knowledge into a skill can therefore make its decay invisible. We study whether agents can keep this externalized knowledge current. Repo2Skill-Evo casts each release transition as a skill-maintenance task: given a V1 skill set and the official V1 → V2 patch, an agent must update obsolete skill content while preserving guidance that remains valid. Across 57 real-world repositories and 105 selected release transitions, every evaluated transition invalidates part of the V1 skill set. Yet six frontier agents reach only 29.9%–69.7% avg@3 macro F1 under a patch-grounded removal metric that balances stale-content recall against over-editing precision. Across runs, two opposing errors dominate: incomplete coverage of affected files in the skill set leaves stale content untouched, while overbroad editing is associated with higher recall but lower precision. Repository skills go stale in silence, and even frontier agents cannot reliably maintain them.

1

Introduction

Large language model (LLM) agents increasingly act through tools and interactive environments [19, 28, 36], and repository-level software engineering has become one of their most demanding settings. To complete realistic repository tasks, an agent must inspect code, read documentation, edit files, and run scripts in an evolving codebase [11, 35]. Success therefore depends not only on the underlying model, but also on repository-specific procedural knowledge: which APIs to call, which scripts to run, how modules are configured, and which conventions the current release expects. Extracting this knowledge from scratch for every task is costly and unreliable, since it requires cross-file retrieval and structural understanding that even strong agents do not consistently achieve [15, 18]. Agent skills offer a compact alternative. They externalize reusable procedural knowledge into on-demand units that agents can retrieve and follow [8, 10, 16]. Prior work shows that skills can improve downstream agent performance [8, 16, 41], and repository-local skills are beginning to appear beyond benchmark settings [1, 27]. Skills are thus becoming persistent agent-facing resources across tasks and releases. We take this utility as our starting point and ask: once a repository evolves, can agents keep its skills from silently going stale? A repository skill is useful precisely because it encodes version-specific knowledge, but

1

Repository Skills Go Stale in Silence Skills look valid, load successfully, yet mislead agents after a repository evolves. 3) Release

2) Use Skills help agents succeed on repository tasks

Repository evolves: V1 to V2 release patch

4) Go Stale (in silence) Skills remain loadable and retrievable, but guidance becomes obsolete

1) Distill Extract procedural knowledge from repository V1 APIs Scripts Defaults Conventions ...

5) Maintain

Lifecycle perspective: Repository skills are versioned knowledge assets. They must be maintained as repositories evolve.

Evaluation: balance two risks

Agents update skills: remove stale content, preserve valid content V1 skill set

+

Skills look valid...

...Yet may mislead agents.

No explicit failure signal for skills...

V1 to V2 release patch

Missed stale lines lead to low recall

VS

Unsupported deletions lead to low precision

V2 skill set

Even frontier LLMs struggle:

29.9% - 69.7% avg@3 macro F1

Figure 1 Lifecycle of repository skills and the Repo2Skill-Evo maintenance setting. Repo2Skill distills repository-specific procedural knowledge from a V1 repository into a traceable skill set. After the repository evolves from V1 to V2 , previously valid skill content may become silently stale: it remains loadable and retrievable while continuing to provide guidance that no longer matches the current repository state. Given the fixed V1 skill set and the official release patch, a maintenance agent produces the updated skill set SV2 . Repo2Skill-Evo evaluates whether the agent removes stale content while preserving guidance that remains valid.

that same specificity ties its correctness to a particular repository state. After a release, a skill distilled from the previous version may remain loadable and retrievable while continuing to provide guidance that no longer matches the current repository state. Because no explicit signal marks this mismatch, the outdated guidance may continue to be retrieved and followed. We refer to this release-driven failure mode as silent staleness. Externalizing version-specific knowledge into a skill preserves its form while rendering its decay invisible. Figure 1 summarizes this lifecycle and the resulting maintenance setting. We therefore treat repository-grounded skills as versioned knowledge assets and center our study on their maintenance. Although related to code migration, the maintained object is the agent-facing guidance derived from a repository, not the repository code itself. Because a release may invalidate only part of a skill package, maintenance requires removing or revising stale content while preserving what remains valid. To study this problem at scale, Repo2Skill-Evo combines three components. First, Repo2Skill uses staged distillation and expert refinement to produce a fixed V1 skill set. Second, given this skill set and the official V1 → V2 release patch, a maintenance agent must remove or revise obsolete skill content while preserving guidance that remains valid. Third, a strictly patch-grounded headline metric rewards the removal or revision of patch-verified obsolete V1 skill lines while penalizing edits to other V1 skill lines, complemented by a rubric-based NL Judge that provides a five-dimensional assessment of the resulting V2 skill set [12, 20, 40]. A focused utility study further motivates the maintenance problem. Across ten repositories randomly sampled from the corpus, skills yield the largest absolute gains on tasks with the lowest baseline utility. Within this sample, the pattern suggests that maintenance may matter most where agents have limited repository knowledge without external support.

2

Our main evaluation across 57 repositories and 105 selected release transitions yields two findings. First, agents do not reliably maintain repository skills across releases. Each selected transition includes a V1 skill set with patch-verified stale content (median: 92 lines), yet six frontier agents achieve only 29.9%–69.7% avg@3 macro F1 . Second, failures reflect complementary localization and edit-selection bottlenecks. Incomplete coverage of affected skill files leaves stale content untouched, whereas broader edits are associated with higher recall but lower patch-grounded precision. Oracle skill-file localization on the 20 hardest transitions improves performance, but substantial residual errors remain, indicating that localization alone does not resolve within-file evidence tracing and edit selection. This paper makes three contributions: • We characterize silent staleness, a release-driven temporal failure mode of repository-grounded skills, and frame these skills as versioned knowledge assets. • We introduce Repo2Skill-Evo, a patch-grounded maintenance benchmark covering 57 repositories and 105 official release transitions, with fixed V1 skill sets and patch-verified obsolete-line annotations. • We evaluate six frontier agents and identify two key maintenance bottlenecks: incomplete localization of affected skill files and imperfect edit selection.

2

Related Work

2.1

Repository-Level Agents and Software Evolution

Repository-level software engineering is a central setting for evaluating LLM agents. SWE-bench casts real GitHub issues as end-to-end repair tasks that require understanding a codebase and producing test-passing patches [11], while SWE-agent shows that the agent–computer interface shapes how agents navigate, edit, and test repositories [35]. Other systems study issue resolution, localization, and repair scaffolding [34, 39]. RepoBench emphasizes cross-file context [18], and RepoMirage shows that broad repository access does not necessarily yield structural understanding [15]. A parallel line studies software evolution directly, including library migration, API refactoring, and migration detection [2, 9, 13, 31]. Recent agent benchmarks extend from static repair to release-note-derived evolution tasks [32], continuous-integration loops [4], and chained release-level upgrades [14]. These works center on source code or downstream client usage. Repo2Skill-Evo instead maintains the agent-facing knowledge that describes how to use a repository. Because this knowledge is a separate artifact, its maintenance does not reduce to code migration.

2.2

Agent Skill Construction and Utility

Agent skills package reusable procedural knowledge into modular units that agents retrieve when relevant. Surveys organize this area around the skill lifecycle, spanning representation, acquisition, retrieval, and evolution [10, 42]. Skills can be generated from source corpora [41] and improved through execution and verifier feedback [21, 37]. Utility benchmarks report heterogeneous effects: curated skills can improve performance, whereas self-generated or mismatched skills do not reliably help [8, 16, 41]. Most of this work evaluates skill construction or use within a fixed environment snapshot. Repository-local skills also appear outside benchmarks. SKILL.md anchors an emerging cross-agent format [1], and PyTorch ships skills for dispatch-macro migration, Metal kernel implementation, AOTInductor debugging, and docstring maintenance [27]. These examples establish repository skills as practical agent-facing artifacts, but do not establish that they remain valid after their repositories advance to new releases.

2.3

Agent Skill Lifecycle and Maintenance

Software engineering has long recognized that natural-language artifacts can drift out of sync with code, as in code-comment inconsistency [33]. A repository skill serves as an agent-facing analogue carrying direct operational consequences. It is not merely reviewed by human maintainers but can also be retrieved and

3

executed by autonomous agents. Runtime feedback can support skill repair after an observed execution failure [23], but silent staleness may provide no error log to react to. Recent work addresses distinct parts of the skill lifecycle. SkillGuard detects environment-contract violations and uses them to localize repair [5]. SkillOps and Library Drift tackle concerns surrounding skill libraries. SkillOps mitigates technical debt via typed contracts and graph-structured maintenance [26], while Library Drift investigates retrieval degradation stemming from unregulated expansion within self-evolving skill banks [38]. SkillHone focuses on iterative revision by preserving decision histories and evaluation evidence across sessions [17]. An empirical study complements these systems by characterizing how registry and personal-use skills are reused, adapted, and maintained in practice [6]. Repo2Skill-Evo isolates release-conditioned maintenance. The release change is provided as an official patch, and the challenge is to map its implications onto a fixed repository-grounded skill set while preserving unaffected guidance. Unlike skill internalization [22], the maintained object remains an external artifact whose validity is tied to a software release.

3

Repo2Skill-Evo: Maintaining Repository Skills across Releases

Repo2Skill-Evo has two components, a procedure for obtaining a fixed, traceable V1 skill set to maintain, and a release-level task that asks whether agents can keep that set current. We describe the skill set first, then the maintenance task, scaffold, and evaluation. Figure 2 summarizes the full pipeline.

Repo2Skill and Repo2Skill-Evo Taking CLI-Anything v0.2.0 → v0.3.0 as an example. 1) Repo2Skill Admit a skill only if its citations trace back to source

Repo V1 (CLI-Anything v0.2.0)

3) Repo2Skill-Evo Remove obsolete lines the patch proves invalid; keep the rest

2) Release Update Official release: CLI-Anything v0.2.0 to v0.3.0

Changelog (README.md news

Static analysis → symbol/path index

Docs skills/cli-anything-inkscape/ SKILL.md: "The CLI now seeds a new in-memory document instead of failing immediately"

Code

read

rejected

cluster

draft submit

cites

A

automatic candidate discovery

B

SKILL.md step4 (L19) loads project when --project points at an existing file via doc_mod.open_document

manual verification

C

SKILL.md example (L68) action = "open_document" if exists else "requires_existing_file"

D

scripts/project_preload.py (L8) "action = 'open_document(project_path)'" references/api-details.md (L7–L9) "-s / --save — not present" "--dry-run — not present" "Root params: ctx, use_json, project_path"

Test

- proj = doc_mod.open_document(project_path) # v0.2.0: path must exist

Distillation agent

Gold obsolete set shows representative items SKILL.md paragraph (L10) "One-shot mutations do not offer -s/--save or --dry-run"

(inkscape_cli.py) _load_or_seed_project(); @click.option("-s","-save","auto_save")

schematic hunk inkscape/…/inkscape_cli.py

search

index

Unresolved in either index or evidence

A. Build Gold Obsolete Set (from patch evidence)

list, added in v0.3.0) "2026-04-07 — Inkscape autosave fix (#193, fixes #182)"

test_core.py::TestCliBootstrap: expects exit_code==0 AND loaded["name"]=="fresh"

E

+ _load_or_seed_project(project_path) # v0.3.0: seeds a new doc if absent + @click.option("-s", "--save", "auto_save", ...) # new flags: --save / --dry-run

scripts/root_options.py 4 stale root-signature lines (L2, L3, L6, L7) params = [ctx, use_json, project_path] flags = [--json, --project] has_auto_save = False; has_dry_run = False (+ SKILL.md frontmatter L3: "no auto-save flags yet")

Grounding gate ① Found evidence symbols/path ②Scripts and refs in evidence

B. Maintenance Action & Evaluation (illustrative error analysis) This is a simplified CLI hunk.

(output)

Maintenance Agent (read/search/edit) 50-turn budget

Sv1(frozen) skill: inkscape-auto-save (v0.2.0, root group & project loading) SKILL.md: "root callback loads project when -project points at an existing file" references/api-details.md: "-s/--save | not present; --dry-run | not present" scripts/project_preload.py: "action = 'open_document(project_path)'"

eval/01_inkscape-project-seed-and-autosave.patch - SKILL.md: paragraph + step 4 +example (A/B/C) - scripts: project_preload.py line 8 (D) - api-details: -s/--save, --dry-run 2 lines (D) (baseline omits root_options.py 4lines + frontmatter (E); see item E)

agent deletions (removed/modified V1 lines)

Missed (example): agent deleted the project_preload.py line but left SKILL.md step 4. (low Recall)

Sv1 now stale

Correct: agent removed items in (e.g. step 4, example comment, api-details rows, ...) (high recall and precision)

Legend: = manually verified gold obsolete set = maintenance agent's proposed deletions

Exploration is free, admission is deterministic.

Recall = Precision = F1 =

Over-edit (example): agent also removed the still-valid --json / --project flag rows. (low precision)

NL Judge: Complementary

Figure 2 Repo2Skill and Repo2Skill-Evo on a release example. Using a simplified CLI-Anything v0.2.0 → v0.3.0 update, the figure illustrates three stages: (1) Repo2Skill generates candidate V1 skill packages, after which an expert selects, revises, and verifies one to produce the fixed V1 skill set; (2) the release patch changes a documented schema behavior; and (3) Repo2Skill-Evo asks a maintenance agent to remove obsolete lines while preserving valid content. Evaluation uses the patch-grounded gold obsolete set G∆ (Section 3.4). The same construction is applied across all 105 release transitions.

4

3.1

Repo2Skill: Distilling a Traceable Skill Set

Before asking whether agents can keep a skill set current, we first need a fixed V1 skill set whose repositoryspecific claims can be traced to source evidence. Repo2Skill serves this role by distilling repository-specific procedural knowledge into grounded skill packages (Figure 2, left), providing the common starting point for the maintenance experiments that follow. Skill representation. A skill is a self-contained package that documents a reusable procedure in SKILL.md, with optional runnable scripts and supporting references. Repo2Skill adapts this format to repository settings by distilling repository-specific procedures into a skill set S = {s1 , . . . , sm }. Provenance is recorded during distillation. The agent logs the source paths and symbols behind each claim as external metadata, which supports the grounding gate and later tracing of content affected by repository changes. The distillation agent. The agent’s analyze stage performs static analysis to build a symbol index Isym that maps repository symbols and file paths to source locations. This index anchors subsequent logic analysis and material collection. The distillation agent then runs a think–act–observe loop over a workspace state wt = (Ht , At , Lt , Mt , St ), where Ht is the interaction history and At , Lt , Mt , and St are the accumulated analysis report, logic read-map, material documents, and skill package. At each step, the agent chooses from Adistill = {analyze, logic, materials, draft, validate, finish}. Analyze runs first to construct Isym , after which logic derives a routing read-map and materials collects repository-grounded evidence over the same repository partitions. Draft jointly uses these intermediate products to author the skill package, validate applies the grounding gate, and finish terminates the run once validation succeeds. The stages follow this order by default, with earlier stages revisited only for recovery after a failure or validation rejection. Figure 2 illustrates this process through a simplified worked example. Grounding gate. After draft, validate applies structural and grounding checks to the candidate skill set by verifying the source paths and symbols referenced in the generated SKILL.md files and scripts against the V1 repository. For candidates meeting the minimum reference count, validation fails once the fraction of unverifiable references exceeds a preset threshold, and only the candidates that pass proceed to expert selection and refinement. The gate therefore establishes source traceability, and its scope stops at reference resolution, since a referenced path or symbol can resolve even when the surrounding guidance uses it incorrectly. Semantic correctness and procedural coverage accordingly fall to the expert, who reviews both during selection and refinement. Candidate generation and selection. For each transition, Claude-opus-4.6 [3] and GPT-5.4 [25] each produce one candidate V1 skill set under the same staged Repo2Skill procedure, using only the pre-release repository. An expert selects the stronger candidate, then revises and verifies it against the V1 repository. The resulting SV1 is frozen before gold-label construction begins and is shared by all maintenance agents. Appendix B.2 details this process.

3.2

Maintenance Task

Once a repository advances from V1 to V2 , parts of its V1 skill set may no longer match the updated interfaces, defaults, file layouts, or procedures. Maintenance therefore requires mapping patch-level changes to the affected guidance and making targeted updates while preserving content that remains valid. Repo2Skill-Evo operationalizes this problem over official release transitions. The right panel of Figure 2 summarizes the maintenance setting and evaluation.

5

Task definition. Each maintenance instance is a tuple (SV1 , ∆V1 →V2 , Umaint ) formed by the fixed V1 skill set SV1 , the official release patch ∆V1 →V2 , and the maintenance tool interface Umaint available to the agent. The agent produces an updated skill set SV2 and is evaluated on the set-level transformation SV1 → SV2 . Successful maintenance requires the three capabilities below. • Patch-to-skill localization. Identify the skill content affected by patch-level changes to APIs, files, commands, configurations, or usage procedures. • Obsolete-knowledge removal. Remove or revise stale references, signatures, examples, tables, and prose that are no longer valid under V2 . • Information retention. Preserve skill content that remains valid, avoiding unsupported rewrites, dropped context, or wholesale regeneration that discards useful repository-specific knowledge. These requirements motivate our patch-grounded removal metric and complementary final-state NL Judge.

3.3

Agent Scaffold

To attribute performance differences to model behavior, we evaluate every model in the same deliberately minimal single-agent environment. The scaffold provides six generic tools covering shell execution, bounded file discovery and inspection, string-based editing, and termination, without any task-specific localization or repair module. Each agent receives the same fixed V1 skill set, official release patch, and standard operating procedure (SOP) prompt, and edits the workspace to produce SV2 . All runs use a pre-specified turn budget. The budget is stated at initialization, and the remaining budget is reported to the agent throughout execution. If the budget is exhausted before the agent finishes, the resulting partial skill set is still evaluated. This controlled scaffold enables comparison of localization, editing, and convergence behavior under the same external constraints.

3.4

Evaluation Metric

The headline evaluation is a strictly patch-grounded removal metric. It scores whether the agent removes or modifies the V1 knowledge that the release patch directly shows to be obsolete, while penalizing edits to V1 lines outside the patch-supported obsolete set. Since valid V2 content may admit multiple realizations, additions and reformulations in the updated skill set are assessed by the complementary NL Judge described below. Gold obsolete set. For each transition, we construct a gold obsolete set G∆ containing the V1 skill lines that the official V1 → V2 release patch directly shows to be no longer valid under V2 . This includes content invalidated by changes to APIs, signatures, defaults, files, commands, configurations, schemas, behavioral contracts, or other repository behavior exposed by the patch. Obsolete content may appear in prose, examples, reference tables, or scripts. Throughout the metric, a line denotes one physical line of a skill file, and diff matches are made against the original V1 lines without merging, splitting, or normalization. G∆ is defined on the V1 side of the transition: it identifies lines that require deletion or revision while leaving the form of a valid V2 replacement unconstrained. We construct G∆ by automated candidate discovery followed by line-level manual verification against direct patch evidence (Appendix B.3). Removal precision, recall, and F1 . We evaluate the V1 -side removal target defined by G∆ . Let R denote the set of V1 skill lines that appear on the deletion side of the skill-set diff between SV1 and the agent-produced SV2 . Thus, R captures both deletions and revisions to existing guidance, while newly added V2 lines are not included. A gold obsolete line counts as hit when its original form appears on the deletion side of the SV1 → SV2 diff. We deliberately do not match newly generated “+” lines, since a valid post-release replacement may admit multiple semantically equivalent realizations, and final-state quality is instead assessed by the NL Judge. We compute Recall =

|G∆ ∩ R| , |G∆ |

Precision =

6

|G∆ ∩ R| , |R|

F1 =

2|G∆ ∩ R| . |G∆ | + |R|

Recall measures the fraction of patch-verified stale lines that are removed or revised, whereas precision measures the fraction of changed V1 lines that belong to G∆ . Their harmonic mean balances stale-content coverage against edit restraint under the minimal-edit objective. All 105 transitions have |G∆ | > 0, so recall is always defined. When R ̸= ∅ but G∆ ∩ R = ∅, precision is zero, and when R = ∅, recall and F1 are zero while precision is undefined. For each transition and metric, avg@3 is the mean over three runs and best@3 is the maximum over the three runs, with undefined precision values omitted. We then macro-average these transition-level scores across the corpus. Recall and F1 therefore include all 105 transitions, whereas macro precision excludes a transition only when precision is undefined in all three runs. Our headline score is avg@3 macro F1 , with macro recall and precision reported separately. Complementary NL Judge. Because the patch-grounded metric scores the V1 -side removal target, the semantic correctness of replacement and newly added V2 content falls to a complementary NL Judge [12, 20, 40]. For each run, the judge reads the release patch, the V1 skills, the produced SV2 skill set, and limited run metadata. The judge assigns five 0–2 scores. API checks whether the produced skills describe the correct V2 API, Files checks whether the skill prose, scripts, and references are mutually consistent, Info checks whether still-valid V1 guidance was retained, Prompt checks adherence to the patch-grounded maintenance instructions, and NL checks whether the prose describes the post-release state directly. API correctness, file consistency, and retention are aggregated across impacted skills, while prompt adherence and NL correctness are package-level judgments. The Sum score (0–10) is the total over these five dimensions, with structurally invalid skill sets form-gated to zero.

4

Experiments

Our experiments center on whether agents can maintain repository skills across releases. We first conduct a focused utility study to assess whether the fixed V1 skills encode repository-specific knowledge with meaningful downstream utility, thereby motivating their maintenance. We then evaluate maintenance performance and diagnose the sources of failure.

4.1

Setup

Maintenance data. Repo2Skill-Evo contains 105 selected official release transitions from 57 public GitHub

repositories, with one fixed V1 skill set per transition and 1,158 individual skills in total. Each transition pairs official V1 and V2 tags with the corresponding source diff. The corpus focuses on the fast-moving AI/ML and LLM-agent ecosystem, covering agent and RAG frameworks, training and inference stacks, and supporting infrastructure, while retaining several mature general-purpose libraries for breadth. We prioritize transitions with visible source-level changes that may affect repository-specific procedural knowledge (Appendix B.1). Across the corpus, the 105 gold obsolete sets contain 12,217 patch-verified stale V1 skill lines, with individual sets ranging from 5 to 375 lines and a median of 92 (Appendix D.3). Models and protocol. We evaluate Claude-opus-4.6, GLM-5.1 [7], GPT-5.4, Kimi-K2.5 [30], Doubao-Seed2-

pro [29], and MiniMax-M2.5 [24]. Each model runs three times per transition under the same agent scaffold and 50-turn budget. For each transition, all runs receive the same fixed V1 skill set and official V1 → V2 release patch in a standardized tool environment.

Corpus composition. Figure 3 summarizes the corpus along two axes: repository domain and observed

maintenance difficulty. By domain, the corpus includes 33 transitions from agent and RAG frameworks, 32 from training and inference infrastructure, 17 from data and classic ML libraries, 16 from compiler, kernel, and systems software, and 7 from general-purpose libraries and applications. For each transition, we average avg@3 F1 across the six maintenance models and classify the resulting observed difficulty as Easy (≥ 0.65), Medium ([0.40, 0.65)), or Hard (< 0.40). Appendix D.4 provides the classification criteria, per-transition assignments, per-model scores, and domain-level outcomes.

7

Figure 3 Composition of the Repo2Skill-Evo corpus, covering 105 release transitions across 57 repositories. The outer ring shows repository domain, and the inner ring shows observed maintenance difficulty, defined by the mean avg@3 F1 across the six maintenance agents. Per-transition assignments and the full score matrix appear in Appendix D.4.

4.2

Motivating Study: Fixed Skills Provide Meaningful Repository Utility

Before evaluating whether repository skills can be maintained across releases, we first ask whether the fixed V1 skills encode repository-specific knowledge worth maintaining. Prior work has shown that skill benefits vary substantially across tasks [8, 16]. We therefore evaluate the downstream utility of the Repo2Skill-Evo skills and examine where their gains are largest. Experimental protocol. We randomly sample ten repositories from the corpus and define one artifact-generation task per repository, together with a repository-specific utility rubric that remains hidden from the model. We evaluate GPT-5.4 in a 2 × 2 ablation over source and skill access, with three runs in each of the four conditions baseline, skill-only, source-only, and source+skill. Appendix A defines these conditions along with the three contrasts and reports the per-repository scores. We use baseline utility as a coarse proxy for prior repository knowledge and split the repositories at the median into low- and high-baseline groups (Table 1). Table 1 Repository-skill utility by baseline-utility group. Values are group means of artifact utility scores (0–10), manually scored under repository-specific rubrics, with three runs per repository and condition. Groups are formed by a median split of baseline utility. Skill and source deltas are relative to baseline; skill ∆src is relative to source-only. Appendix A provides per-repository scores and cost statistics. Group All repositories Low-baseline repositories High-baseline repositories

#repos

baseline

skill-only

source-only

source+skill

skill ∆

source ∆

skill ∆src

10 5 5

5.88 4.48 7.28

8.68 8.46 8.89

8.64 8.46 8.82

9.01 8.86 9.15

+2.80

+2.76 +3.98 +1.54

+0.37 +0.40 +0.33

+3.98 +1.61

Skills provide substantial downstream utility. Across all ten repositories, skill-only raises mean utility from

5.88 to 8.68, a gain of 2.80 points, and nearly matches source-only at 8.64. The fixed skills therefore provide task-relevant repository knowledge that the model does not consistently recover without external context. Gains are largest at low baseline utility. Skill-only raises mean utility in the low-baseline group from 4.48 to

8.46, a gain of 3.98 points, compared with 1.61 points in the high-baseline group. Within this sample, skills provide the largest absolute gains on repositories where the model performs worst without source or skill access. Skills provide comparable utility at substantially lower cost. Skill-only and source-only achieve nearly identical

8

mean utility (8.68 versus 8.64), but skill-only uses 51,821 mean tokens and 3.9 iterations, compared with 272,620 tokens and 10.8 iterations for source-only. When source is already available, adding skills increases mean utility from 8.64 to 9.01 while modestly reducing mean token usage (Appendix A). Together, these results show that the fixed V1 skills encode useful, compact repository knowledge, motivating their maintenance across releases.

4.3

Agents Do Not Reliably Maintain Repository Skills across Releases

We now evaluate the maintenance task defined in Section 3.2. Table 2 reports the headline patch-grounded removal scores. The metric rewards the removal or revision of obsolete V1 lines while penalizing edits to V1 lines outside G∆ . Maintenance remains far from solved. Claude-opus-4.6 achieves the highest avg@3 macro F1 at 69.7%,

followed by GLM-5.1 at 64.3%, while the remaining four models fall below 60%. GPT-5.4 obtains the highest avg@3 macro recall at 74.6%, but its substantially lower macro precision of 55.7% limits its avg@3 macro F1 to 58.8%. Kimi-K2.5, Doubao-Seed2-pro, and MiniMax-M2.5 achieve avg@3 macro F1 scores of only 47.0%, 39.8%, and 29.9%, respectively. These score profiles have direct maintenance consequences. Low recall leaves patch-verified stale guidance in the produced skill set, whereas low precision reflects broader editing outside the patch-grounded obsolete set and therefore weaker adherence to the task’s minimal-edit objective. Repository-level macro averaging preserves the model ordering, and 95% paired repository-clustered bootstrap intervals for the three adjacent avg@3 F1 differences among the top four models all exclude zero (Appendix D.2). Table 2 Patch-grounded maintenance scores (%) over 105 release transitions and three runs per model. R, P, and F1 denote transition-macro recall, precision, and F1 . avg@3 averages the three runs for each transition, whereas best@3 takes the per-transition maximum over the three runs before macro-averaging. Macro F1 is averaged directly over transitions and is therefore not the harmonic mean of the displayed macro R and P. Models are sorted by avg@3 macro F1 ; the best result in each column is shown in bold. avg@3 macro Model Claude-opus-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 Doubao-Seed2-pro MiniMax-M2.5

best@3 macro

F1

R

P

F1

70.4 75.7 69.7 64.1 72.4 64.3 74.6 55.7 58.8 40.4 71.6 47.0 31.9 73.3 39.8 22.6 74.9 29.9

79.3 78.1

82.2 80.9 67.9 84.9 85.0

76.1

R

P

87.9 55.0 44.7 36.6

73.9 70.4 60.7 53.4 86.8 46.0

Models occupy different points on the recall--precision tradeoff. GPT-5.4 removes or revises a broader

set of V1 lines and achieves the highest recall, but many of these changes fall outside G∆ . Claude-opus-4.6 and GLM-5.1 maintain more balanced recall–precision profiles. The lower-scoring models tend toward high precision and low recall: MiniMax-M2.5, for example, reaches 74.9% precision but removes or revises only 22.6% of the verified stale content. High precision in this regime does not imply successful maintenance; it coexists with substantial under-editing. Section 4.4 connects these outcome profiles to affected-file coverage and edit-selection diagnostics derived from the execution traces and output diffs. Repeated attempts reveal capability but not reliability. Selecting the highest F1 among three runs for each

transition raises macro F1 by 6.4 points for Claude-opus-4.6 and by 9.6–16.1 points for the remaining models. These gains indicate that models sometimes produce substantially better updates than their mean single-run performance suggests. However, even best@3 reaches only 76.1% for the strongest model, and no other model exceeds 73.9%. Moreover, best@3 selects the strongest of three observed outcomes and therefore reflects attainable performance across repeated attempts rather than single-attempt reliability. Together with the within-transition run-to-run variability reported in Appendix D.2, the generally larger best@3–avg@3 gaps for the lower-scoring models indicate that maintenance quality is unstable across repeated attempts.

9

Complementary evaluation identifies the same upper tier. Table 3 reports five-dimensional NL Judge scores

assigned by GPT-5.4 and Claude-opus-4.6. Both judges rank Claude-opus-4.6 first, GPT-5.4 second, and GLM-5.1 third. They therefore identify the same top-three set as the patch-grounded evaluation. All models produce at least some structurally invalid skill sets, with 7–26 invalid runs out of 315 per model. The NL Judge complements the headline metric by assessing the final skill set across five rubric dimensions, whereas the patch-grounded removal metric provides a direct and auditable measure of stale-content removal and off-target editing. Full dimension-level results appear in Appendix D.1. Table 3 Complementary NL Judge scores over 315 runs per maintenance model. Each judge reports the mean 0–10 Sum score, with structurally invalid outputs form-gated to zero before averaging. Invalid reports the number of such outputs. The best score in each judge column is shown in bold. Model

GPT-5.4 judge

Claude-opus-4.6 judge

Invalid

7.54

7.86

7.09 7.18 6.49 6.54 6.01

6.98 7.31 6.24 6.22 5.27

9 7 26 14 15 8

Claude-opus-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 Doubao-Seed2-pro MiniMax-M2.5

The maintenance shortfall extends across the corpus. Transition difficulty varies substantially, but poor

performance is not confined to a few isolated cases. Under the observed-difficulty stratification defined in Section 4.1, 25 of the 105 transitions fall below 0.40, 60 fall in [0.40, 0.65), and only 20 reach at least 0.65 when averaging the six models’ avg@3 F1 scores. Thus, 85 transitions fall below the Easy threshold. Appendix D.4 provides the full stratification and representative high- and low-scoring transitions.

4.4

Failure Diagnosis: Coverage and Editing Bottlenecks

Maintenance can fail because agents miss affected skill files, edit too little or too broadly, or run out of turns before finishing. We analyze all 1,890 runs using execution traces and maintained-output diffs. Table 4 summarizes the resulting model-level diagnostics. Affected-file coverage measures the fraction of gold-affected skill files localized during a run. Removal load, |R|/|G∆ |, measures the volume of changed V1 lines relative to the gold removal target, while off-target editing measures the fraction of changed V1 lines outside G∆ . Together, these diagnostics distinguish missed localization and under-editing from broad, poorly targeted editing. They are used only for post hoc analysis. Appendix C.1 provides the full definitions. Table 4 Post hoc diagnostics over 315 runs per model. Coverage and removal load are mean ratios; full coverage, off-target editing, no-removal runs, and budget exhaustion are percentages; turns is the mean number of turns per run. Model Claude-opus-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 Doubao-Seed2-pro MiniMax-M2.5

mean file cov.

full file cov.

removal load

off-target

no-removal

budget exh.

turns

0.95 0.95 0.93 0.74 0.53 0.51

75.9 78.1 69.2 25.7 10.5 9.5

1.01 1.01 2.84 0.60 0.42 0.30

24.0 27.6 44.3 28.6 25.9 22.7

0.6 0.6 0.0 2.9 4.1 20.0

1.9 41.3 0.0 47.3 14.6 68.6

25.3 42.4 17.8 43.5 33.1 47.0

Coverage is strongly associated with maintenance quality. Across all 1,890 runs, affected-file coverage

correlates with both F1 (r = 0.650) and recall (r = 0.701), and these associations persist after removing additive model and transition effects (Appendix C.1). Runs with full affected-file coverage average 67.3% F1 and 72.1% recall, compared with 41.0% F1 for partial-coverage runs. These results consistently associate incomplete localization with missed stale content. Oracle localization improves performance but leaves substantial residual errors. We test localization more

directly through an oracle ablation on the hardest-20 subset, restricted to transitions with at least 20 gold

10

obsolete lines. We append only the gold-affected skill roots and file paths to the original prompt, without providing patch hunks, line numbers, obsolete-line labels, or edit instructions. Table 5 Oracle skill-file localization on the hardest-20 subset. Values are avg@3 macro scores (%). Gain columns report mean per-transition oracle-minus-baseline differences. Baseline Model

R

P

Oracle F1

R

P

Gain F1

∆R

∆P

∆F1

Claude-opus-4.6 55.4 64.4 53.6 63.5 68.2 62.3 +8.1 +3.8 +8.7 GLM-5.1 46.2 65.2 48.7 54.8 66.0 55.9 +8.6 +0.7 +7.2 GPT-5.4 65.6 47.9 48.2 73.3 50.5 54.6 +7.7 +2.6 +6.3 Kimi-K2.5 14.2 50.4 19.0 31.8 61.9 36.6 +17.6 +11.5 +17.6 Doubao-Seed2-pro 11.5 58.6 16.9 30.2 68.9 37.6 +18.7 +10.3 +20.8 MiniMax-M2.5 6.7 50.0 10.1 18.0 69.4 23.0 +11.2 +19.1 +12.9

Mean

33.3 56.1 32.8 45.3 64.1 45.0 +12.0

+8.0 +12.2

Averaged across the six models, avg@3 macro F1 increases from 32.8% to 45.0%, a mean transition-level gain of 12.2 points (95% paired bootstrap CI: [+6.8, +17.8]), while recall increases by 12.0 points. Recall improves for every model. Yet even with oracle paths, the best model reaches only 62.3% F1 on this subset. File-level localization is therefore important but insufficient: agents must still identify the affected claims within each file and select the appropriate edits. Appendix C.2 provides the full experimental details. Models exhibit distinct failure profiles. Figure 4 visualizes the transition-level recall–precision tradeoff behind

the model averages.

1.0

Removal precision

0.8 F1=0.8

0.6

F1=0.6

0.4 Claude-opus-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 Doubao-Seed2-pro MiniMax-M2.5

0.2

F1=0.4

F1=0.2

0.0 0.0

0.2

0.4

0.6

0.8

1.0

Removal recall

Figure 4 Removal recall versus precision for each model–transition pair after avg@3 aggregation. Pairs with undefined precision are omitted. Diamonds mark per-model means, and light curves show iso-F1 contours.

The diagnostics reveal different routes to failure. GPT-5.4 achieves high affected-file coverage but has a removal load of 2.84 and the highest off-target editing rate, consistent with its high-recall, low-precision profile. In contrast, Kimi-K2.5, Doubao-Seed2-pro, and MiniMax-M2.5 show substantially lower coverage and removal loads, indicating that many failures arise before enough affected content is reached and edited. Kimi-K2.5 and MiniMax-M2.5 also frequently exhaust the turn budget. GLM-5.1 achieves high coverage and a near-unit removal load but exhausts the budget in 41.3% of runs, showing that finding the affected files does not by itself ensure successful completion. Claude-opus-4.6 exhibits the most balanced profile, combining high coverage, a near-unit removal load, and infrequent budget exhaustion. Overall, the results point to two 11

complementary maintenance bottlenecks: localizing the affected skill content and selecting edits that are sufficiently complete without becoming overly broad.

5

Limitations

Our dataset is a curated staleness-positive challenge set, so our results characterize maintenance difficulty on releases known to affect skills, with the prevalence of staleness across releases forming a separate question (Appendix B.1). Our evaluation measures patch-grounded maintenance fidelity, leaving the downstream utility of maintained skills to future work. Although the NL Judge provides a complementary assessment of final-state quality, we do not rerun the maintained skill sets on downstream repository tasks.

6

Conclusion

Repository-grounded skills turn repository-specific procedures into reusable external knowledge, but their validity does not persist automatically. As repositories evolve, previously correct guidance can become misleading while remaining loadable and retrievable. We call this release-driven failure mode silent staleness. Repo2Skill-Evo operationalizes this lifecycle problem as a measurable maintenance task by pairing fixed V1 skill sets with official release patches and evaluating the resulting updates against patch-grounded targets. Across 57 repositories and 105 selected release transitions, even the strongest of the six evaluated agents reaches only 69.7% avg@3 macro F1 . Trace and output analyses characterize two complementary maintenance bottlenecks: incomplete localization of affected skill files and imperfect edit selection. Oracle skill-file localization improves performance but leaves substantial residual errors, indicating that localization alone is insufficient. Together, these findings suggest that repository skills should be treated as versioned knowledge assets that carry the release they were distilled from and require maintenance as the repository evolves.

12

7

Contributions

Project Lead(α − β order)

Chenyuan Duan (duanchenyuan26@stu.pku.edu.cn) Ge Shi(23121732@bjtu.edu.cn) Core Contributors(α − β order)

Zineng Mao, Ge Zhang Contributors(α − β order)

Hao Liang, Yinzhu Piao, Yuchen Wu, Zhixin Yao Corresponding(α − β order)

Kaiyu Huang, Wenhao Huang, Linzhuang Sun Shen Yan(sheny@bytedance.com) Wentao Zhang(wentao.zhang@pku.edu.cn)

13

References [1] Agent Skills. Agent Skills overview, 2026. URL https://agentskills.io/home. [Accessed 04-07-2026]. [2] Hussein Alrubaye, Mohamed Wiem Mkaouer, and Ali Ouni. Migrationminer: An automated detection tool of third-party java library migration at the method level. In 2019 IEEE International Conference on Software Maintenance and Evolution, ICSME 2019, Cleveland, OH, USA, September 29 - October 4, 2019, pages 414–417. IEEE, 2019. doi: 10.1109/ICSME.2019.00072. URL https://doi.org/10.1109/ICSME.2019.00072. System card: Claude Opus 4.6, [3] Anthropic. claude-opus-4-6-system-card. [Accessed 04-07-2026].

2026.

URL

https://www.anthropic.com/

[4] Jialong Chen, Xander Xu, Hu Wei, Chuan Chen, and Bing Zhao. SWE-CI: evaluating agent capabilities in maintaining codebases via continuous integration. CoRR, abs/2603.03823, 2026. doi: 10.48550/ARXIV.2603.03823. URL https://doi.org/10.48550/arXiv.2603.03823. [5] Linfeng Fan, Yuan Tian, Ziwei Li, and Zhiwu Lu. Skill drift is contract violation: Proactive maintenance for LLM agent skill libraries. CoRR, abs/2605.10990, 2026. doi: 10.48550/ARXIV.2605.10990. URL https: //doi.org/10.48550/arXiv.2605.10990. [6] Haoyu Gao, Jai Lal Lulla, Hong Yi Lin, Sebastian Baltes, Christoph Treude, and Mansooreh Zahedi. From registry to repository: How ai agent skills are written, adapted, and maintained, 2026. URL https://arxiv.org/ abs/2607.00911. [7] GLM-Team. GLM-5: from vibe coding to agentic engineering. CoRR, abs/2602.15763, 2026. doi: 10.48550/ ARXIV.2602.15763. URL https://doi.org/10.48550/arXiv.2602.15763. [8] Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, and Lijie Hu. Swe-skillsbench: Do agent skills actually help in real-world software engineering? CoRR, abs/2603.15401, 2026. doi: 10.48550/ARXIV.2603.15401. URL https://doi.org/10.48550/arXiv.2603.15401. [9] Mohayeminul Islam, Ajay Kumar Jha, Ildar Akhmetov, and Sarah Nadi. Characterizing python library migrations. Proc. ACM Softw. Eng., 1(FSE):92–114, 2024. doi: 10.1145/3643731. URL https://doi.org/10.1145/3643731. [10] Yanna Jiang, Delong Li, Haiyu Deng, Baihe Ma, Xu Wang, Qin Wang, and Guangsheng Yu. Sok: Agentic skills - beyond tool use in LLM agents. CoRR, abs/2602.20867, 2026. doi: 10.48550/ARXIV.2602.20867. URL https://doi.org/10.48550/arXiv.2602.20867. [11] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. [12] Seungone Kim, Jamin Shin, Yejin Choi, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=8euJaTveKw. [13] Raula Gaikovina Kula, Ali Ouni, Daniel M. Germán, and Katsuro Inoue. An empirical study on the impact of refactoring activities on evolving client-used apis. Inf. Softw. Technol., 93:186–199, 2018. doi: 10.1016/J.INFSOF. 2017.09.007. URL https://doi.org/10.1016/j.infsof.2017.09.007. [14] Man Ho Lam, Chaozheng Wang, Hange Liu, Jingyu Xiao, Haau-sing Li, Jen-tse Huang, Terry Yue Zhuo, and Michael R. Lyu. Swe-chain: Benchmarking coding agents on chained release-level package upgrades. CoRR, abs/2605.14415, 2026. doi: 10.48550/ARXIV.2605.14415. URL https://doi.org/10.48550/arXiv.2605.14415. [15] Hanyu Li, Yichi Zhang, Speed Zhu, Hang Su, Jun Zhu, and Yinpeng Dong. Repomirage: Probing repository context reasoning in code agents with perturbations. CoRR, abs/2605.26177, 2026. doi: 10.48550/ARXIV.2605.26177. URL https://doi.org/10.48550/arXiv.2605.26177. [16] Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, Shuyi Wang, Qunhong Zeng, Di Wang, Xuandong Zhao, Yuanli Wang, Roey Ben Chaim, Zonglin Di, Yipeng Gao, Junwei He, Yizhuo He, Liqiang Jing, Luyang Kong, Xin Lan, Jiachen Li, Songlin Li, Yijiang Li, Yueqian Lin, Xinyi Liu, Xuanqing Liu, Haoran Lyu, Ze Ma, Bowei Wang, Runhui Wang, Tianfu Wang, Wengao Ye, Yue Zhang, Hanwen Xing, Yiqi Xue, Steven Dillmann, and Han-Chung Lee. Skillsbench: Benchmarking how

14

well agent skills work across diverse tasks. CoRR, abs/2602.12670, 2026. doi: 10.48550/ARXIV.2602.12670. URL https://doi.org/10.48550/arXiv.2602.12670. [17] Zhiwei Li and Yong Hu. Skillhone: A harness for continual agent skill evolution through persistent decision history. CoRR, abs/2606.08671, 2026. doi: 10.48550/ARXIV.2606.08671. URL https://doi.org/10.48550/arXiv.2606. 08671. [18] Tianyang Liu, Canwen Xu, and Julian J. McAuley. Repobench: Benchmarking repository-level code autocompletion systems. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=pPjZIOuQuF. [19] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=zAdUB0aCTQ. [20] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 2511–2522. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.EMNLP-MAIN.153. URL https://doi.org/10.18653/v1/2023.emnlp-main.153. [21] Yuxuan Liu, Zhaochen Su, Lingyun Xie, Yuhao Zhang, Qing Zong, Jiahe Guo, Zhongwei Xie, Yiyan Ji, Yauwai Yim, Hongyu Luo, Xiyu Ren, Ruan Chenyu, Haoran Li, and Yangqiu Song. Skillrevise: Improving LLM-authored agent skills via trace-conditioned skill revision. CoRR, abs/2606.01139, 2026. doi: 10.48550/ARXIV.2606.01139. URL https://doi.org/10.48550/arXiv.2606.01139. [22] Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. SKILL0: in-context agentic reinforcement learning for skill internalization. CoRR, abs/2604.02268, 2026. doi: 10.48550/ARXIV.2604.02268. URL https://doi.org/10.48550/arXiv.2604.02268. [23] Zijian Lu, Yiping Zuo, Yupeng Nie, Xin He, Weibei Fan, Lianyong Qi, and Shi Jin. Contractskill: Repairable contract-based skills for multimodal web agents. CoRR, abs/2603.20340, 2026. doi: 10.48550/ARXIV.2603.20340. URL https://doi.org/10.48550/arXiv.2603.20340. [24] MiniMax. MiniMax-M2.5 model card, 2026. URL https://huggingface.co/MiniMaxAI/MiniMax-M2.5. [Accessed 04-07-2026]. GPT-5.4 thinking system card, [25] OpenAI. gpt-5-4-thinking-system-card/. [Accessed 04-07-2026].

2026.

URL

https://openai.com/index/

[26] Hongji Pu, Xinyuan Song, and Liang Zhao. Skillops: Managing LLM agent skill libraries as self-maintaining software ecosystems. CoRR, abs/2605.13716, 2026. doi: 10.48550/ARXIV.2605.13716. URL https://doi.org/ 10.48550/arXiv.2605.13716. [27] PyTorch. Repository agent skills in the PyTorch codebase, 2026. URL https://github.com/pytorch/pytorch/ tree/main/.claude/skills. Public skill directory maintained under the repository’s .claude/skills/ path. [Accessed 04-07-2026]. [28] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.cc/paper_ files/paper/2023/hash/d842425e4bf79ba039352da0f658a906-Abstract-Conference.html. [29] Bytedance Seed. Seed2.0 model card: Towards intelligence frontier for real-world complexity. CoRR, abs/2607.00248, 2026. doi: 10.48550/ARXIV.2607.00248. URL https://doi.org/10.48550/arXiv.2607.00248. [30] Kimi Team. Kimi K2.5: visual agentic intelligence. CoRR, abs/2602.02276, 2026. doi: 10.48550/ARXIV.2602.02276. URL https://doi.org/10.48550/arXiv.2602.02276. [31] Cédric Teyton, Jean-Rémy Falleri, Marc Palyart, and Xavier Blanc. A study of library migrations in java. J. Softw. Evol. Process., 26(11):1030–1052, 2014. doi: 10.1002/SMR.1660. URL https://doi.org/10.1002/smr.1660.

15

[32] Minh V. T. Thai, Tue Le, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. SWE-EVO: benchmarking coding agents in long-horizon software evolution scenarios. CoRR, abs/2512.18470, 2025. doi: 10.48550/ARXIV. 2512.18470. URL https://doi.org/10.48550/arXiv.2512.18470. [33] Fengcai Wen, Csaba Nagy, Gabriele Bavota, and Michele Lanza. A large-scale empirical study on code-comment inconsistencies. In Yann-Gaël Guéhéneuc, Foutse Khomh, and Federica Sarro, editors, Proceedings of the 27th International Conference on Program Comprehension, ICPC 2019, Montreal, QC, Canada, May 25-31, 2019, pages 53–64. IEEE / ACM, 2019. doi: 10.1109/ICPC.2019.00019. URL https://doi.org/10.1109/ICPC.2019.00019. [34] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying llm-based software engineering agents. Proc. ACM Softw. Eng., 2(FSE):801–824, 2025. doi: 10.1145/3715754. URL https://doi.org/10.1145/ 3715754. [35] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.cc/ paper_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract-Conference.html. [36] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview. net/forum?id=WE_vluYUL-X. [37] Hanrong Zhang, Shicheng Fan, Henry Peng Zou, Yankai Chen, Zhenting Wang, Jiayu Zhou, Chengze Li, WeiChieh Huang, Yifei Yao, Kening Zheng, Xue Liu, Xiaoxiao Li, and Philip S. Yu. Coevoskills: Self-evolving agent skills via co-evolutionary verification. CoRR, abs/2604.01687, 2026. doi: 10.48550/ARXIV.2604.01687. URL https://doi.org/10.48550/arXiv.2604.01687. [38] Xing Zhang, Yanwei Cui, Guanghui Wang, Ziyuan Li, Wei Qiu, Bing Zhu, and Peiyang He. Library drift: Diagnosing and fixing a silent failure mode in self-evolving LLM skill libraries. CoRR, abs/2605.19576, 2026. doi: 10.48550/ARXIV.2605.19576. URL https://doi.org/10.48550/arXiv.2605.19576. [39] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. In Maria Christakis and Michael Pradel, editors, Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, pages 1592–1604. ACM, 2024. doi: 10.1145/3650212.3680384. URL https://doi.org/10.1145/3650212.3680384. [40] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llmas-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ 91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html. [41] Yifan Zhou, Zhentao Zhang, Ziming Cheng, Shuo Zhang, Qizhen Lan, Zhangquan Chen, Zhi Yang, Qianyu Xu, Ronghao Chen, Huacan Wang, and Sen Hu. Skillgenbench: Benchmarking skill generation pipelines for LLM agents. CoRR, abs/2605.18693, 2026. doi: 10.48550/ARXIV.2605.18693. URL https://doi.org/10.48550/ arXiv.2605.18693. [42] Yingli Zhou, Wang Shu, Yaodong Su, Wenchuan Du, Yixiang Fang, and Xuemin Lin. A comprehensive survey on agent skills: Taxonomy, techniques, and applications. CoRR, abs/2605.07358, 2026. doi: 10.48550/ARXIV.2605. 07358. URL https://doi.org/10.48550/arXiv.2605.07358.

16

Appendix A

Skill-Gain Utility Study

Section 4.2 reports the aggregate findings of the motivating utility study. Here we provide the experimental protocol, per-repository results, baseline-utility grouping procedure, and execution costs. Experimental protocol. Each of the ten sampled repositories is represented once. GPT-5.4 performs three runs in each cell of a 2 × 2 ablation over access to the fixed V1 skill set and the V1 repository checkout: • baseline: neither source nor skill is available; • skill-only: the skill set is available and the source is unavailable; • source-only: the source is available and the skill set is unavailable; • source+skill: both the source and the skill set are available. The four conditions use the same repository-specific task, evaluation rubric, agent scaffold, and execution budget, differing only in source and skill access. Skill ∆ is skill-only minus baseline, source ∆ is source-only minus baseline, and skill ∆src is source+skill minus source-only. These contrasts measure, respectively, the utility of skills without source access, the utility of source access without skills, and the incremental utility of skills when source is already available. Table 1 reports grouped means over the same per-repository scores shown in Table 6. Table 6 Per-repository artifact utility scores for the ten-repository motivating study. Scores are 0–10 means over three runs. Skill ∆ is skill-only minus baseline, source ∆ is source-only minus baseline, and skill ∆src is source+skill minus source-only. Rows are ordered by baseline utility, with the median split separating the low- and high-baseline groups. Repository

baseline skill-only source-only source+skill skill ∆ source ∆ skill ∆src

agentmemory oh-my-pi harbor deer-flow hermes-agent

3.43 4.33 4.70 4.73 5.20

8.57 8.73 8.20 8.47 8.33

8.83 7.90 8.50 9.07 8.00

9.07 9.10 8.63 9.17 8.33

+5.13 +4.40 +3.50 +3.73 +3.13

+5.40 +3.57 +3.80 +4.33 +2.80

+0.23 +1.20 +0.13 +0.10 +0.33

browser-use lightrag mem0 langgraph pydantic

6.37 6.83 7.17 7.83 8.20

8.70 8.67 8.60 9.17 9.33

8.73 9.13 8.67 8.70 8.87

9.00 9.17 9.00 9.20 9.40

+2.33 +1.83 +1.43 +1.33 +1.13

+2.37 +2.30 +1.50 +0.87 +0.67

+0.27 +0.03 +0.33 +0.50 +0.53

Mean

5.88

8.68

8.64

9.01

+2.80

+2.76

+0.37

Baseline-utility grouping. We use baseline utility as a coarse operational proxy for repository-specific prior knowledge and split the repositories at the median. The five low-baseline repositories are agentmemory, oh-my-pi, harbor, deer-flow, and hermes-agent. The remaining five form the high-baseline group. Group means appear in Table 1, and the per-repository values from which they are computed appear in Table 6. Because baseline utility may also reflect task difficulty, and because lower-baseline tasks have more headroom on the bounded 0–10 scale, we interpret this grouping descriptively. Execution cost. Table 7 reports utility and execution cost for all four conditions. Each row aggregates 30 runs: three runs on each of ten repositories. Table 7 quantifies the cost reduction noted in Section 4.2. Adding skills on top of source access reduces mean tokens from 272,620 to 237,020 and mean iterations from 10.8 to 9.3, so the +0.37-point utility gain of source+skill over source-only comes at slightly lower cost.

17

Table 7 Artifact utility and execution cost across the four access conditions. Values are means over ten repositories and three runs per repository and condition.

B

Condition

Utility

Mean tokens

Mean iterations

Baseline Skill-only Source-only Source+skill

5.88 8.68 8.64 9.01

21,057 51,821 272,620 237,020

4.4 3.9 10.8 9.3

Maintenance Dataset Construction

Repo2Skill-Evo is constructed before any maintenance model is evaluated. The construction proceeds in four steps: 1. Select release transitions. We identify official releases with publicly available source diffs (Appendix B.1). 2. Materialize release instances. Each transition records its repository, V1 and V2 version tags, commit hashes, and official V1 → V2 release patch. 3. Construct the fixed V1 skill set. Repo2Skill produces candidate skill sets from the pre-release repository, after which an expert selects, revises, and verifies the final SV1 used by all maintenance agents (Appendix B.2). 4. Construct the gold obsolete set. We identify the V1 skill lines that the release patch directly shows to be invalid under V2 , producing the patch-grounded gold set G∆ (Appendix B.3). Transitions with G∆ = ∅ are excluded, yielding the final staleness-positive corpus. Gold annotations are stored separately from the maintenance instances and are used only for evaluation. Maintenance agents receive neither G∆ nor its annotation-side representation.

B.1

Transition Selection and Construction

Source and scope. Beyond the corpus-level counts given in Section 4.1, each transition records the repository, the V1 and V2 version tags, the base and target commit hashes, and the corresponding public source diff, making every instance traceable to an upstream release. Selection favors actively maintained projects; among their releases we prefer diffs that touch interfaces, defaults, file layouts, commands, configurations, or documented repository behavior, since these are the change classes that can invalidate procedural guidance. The complete repository and release list is included in the released materials. Recency and contamination. The selected transitions span a broad range of version histories, including releases that postdate the knowledge cutoffs of some evaluated models. We do not assume that the repositories themselves were absent from model pretraining. However, the maintenance task turns on aligning a fixed V1 skill set with the changes in one specific V1 → V2 release patch, so performance depends on the evidence that the patch itself supplies. We release version tags and commit hashes to support independent dating and auditing.

B.2

Fixed V1 Skill Set

Section 3 states the candidate-generation and expert-selection procedure. Here we record the two properties that matter for the validity of the benchmark. First, isolation: both candidates are generated from the pre-release repository alone, and the expert revision draws on the same V1 evidence, so the release patch and the gold obsolete labels remain outside the construction of SV1 from end to end. Second, review scope: the expert reviews the package as a whole, including SKILL.md, scripts, and references. Cited paths and symbols are checked against the repository, procedural guidance is reviewed for factual consistency, and the package must satisfy the structural and grounding requirements before it is frozen.

18

B.3

Gold Obsolete Set Construction

Candidate discovery and manual verification. In the automated stage, a script extracts repository-specific interfaces referenced by the V1 skill set, including symbols, paths, commands, and configuration keys, and matches them against changed lines in the release patch. This stage surfaces potentially affected skill content and is used only to assist candidate discovery. For every transition, the V1 skill set is then inspected together with the corresponding release patch. Candidate lines whose documented guidance remains valid under V2 are removed, while stale content missed by the automatic pass is added, including semantically expressed claims and lines whose validity depends on an affected interface or behavior. Every final line in G∆ is manually verified against direct patch evidence before being used as a gold removal target. Representative obsolete-content categories. Table 8 summarizes common sources of skill invalidation. These categories sample the range of changes observed across transitions, and membership in G∆ follows from direct patch evidence for each individual line. Table 8 Representative sources of V1 skill obsolescence captured by G∆ . Repository change

Example effect on V1 guidance

API or interface change Signature or default change File or entry-point change Command or configuration change Behavioral contract change

Referenced functions, classes, modules, or interfaces are removed or renamed Arguments, defaults, return contracts, or invocation patterns become invalid Referenced paths, scripts, modules, or entry points move or disappear Flags, commands, configuration keys, or schemas change Previously documented behavior is contradicted by the updated implementation

The resulting stale knowledge is not limited to the line that explicitly names the changed repository element. When a patch-supported change also invalidates dependent instructions, examples, table entries, or script statements, those lines are included in G∆ as well. Example. Consider the transition flask-2.2.5-2.3.0. A V1 skill, app-env-config, describes environment mode using FLASK_ENV, the ENV configuration key, and the app.env property. The release removes this environment-mode surface, making the corresponding V1 guidance obsolete. G∆ therefore includes the affected reference-table rows, prose instructions that use app.config["ENV"], and script lines that set os.environ["FLASK_ENV"] or access app.env. This example illustrates how a single release change can invalidate repository-specific guidance expressed across multiple files and content forms within a skill package.

C

Evaluation Protocol and Diagnostics

C.1

Behavioral Diagnostics

Table 9 defines the post hoc diagnostics used in Section 4.4. Affected-file coverage is derived from agent traces and maintained outputs, while the remaining diff-based diagnostics are computed from the SV1 → SV2 diff. A gold-affected skill file is a file in SV1 containing at least one line in G∆ , and a file counts as localized when the agent directly reads it, explicitly names or targets it in the interaction trace, or modifies it in the maintained output. Localization therefore requires evidence that the agent attended to the file, which places paths appearing only in tool-returned search results outside the criterion. Affected-file coverage is the fraction of gold-affected skill files meeting this criterion. Budget exhaustion covers runs that reach the 50-turn limit while still acting on the workspace, so a run whose final turn is a standalone finish counts as a completed run even on turn 50. Partial outputs from budget-exhausted runs enter the evaluation on the same terms as complete ones. Coverage-association robustness. The pooled correlations reported in Section 4.4 use all 1,890 runs. As a robustness check, we first average the three runs for each model–transition pair, yielding 630 observations. We then residualize affected-file coverage, F1 , and recall with respect to additive model and transition fixed effects and compute correlations between the resulting residuals. 19

Table 9 Definitions of the post hoc diagnostics reported in Table 4. Metric

Definition

mean file cov. full file cov.

Run-level affected-file coverage, reported as the mean across runs. Run-level indicator that every gold-affected skill file is localized, reported as the percentage of runs satisfying the indicator. Run-level ratio |R|/|G∆ |, reported as the mean across runs. Run-level fraction |R \ G∆ |/|R|, defined when R ̸= ∅ and reported as the mean over defined runs. Run-level indicator that R = ∅, reported as the percentage of runs satisfying the indicator. Run-level indicator that run_outcome=max_iterations, corresponding to 50 agent turns without a standalone finish, reported as the percentage of runs satisfying the indicator. Number of agent turns in a run, reported as the mean across runs.

removal load off-target no-removal budget exh. turns

Coverage remains positively associated with F1 (r = 0.477) and recall (r = 0.455). The associations are attenuated relative to the pooled run-level correlations in Section 4.4, indicating that the relationship holds within models as well as across them. Removal-load stratification. Runs with removal load in [0.75, 1.25) average 77.3% F1 , compared with 25.0% for runs below 0.5. Runs with removal load at or above 1.25 achieve 80.8% recall but 44.7% precision. These patterns further distinguish insufficient editing from broad editing that improves stale-content coverage at the expense of edit selectivity.

C.2

Oracle Skill-File Localization Ablation

The oracle ablation in Section 4.4 tests the contribution of skill-file localization more directly. We first restrict the 105 transitions to those with at least 20 gold obsolete lines. Among the eligible transitions, we rank them by the mean baseline avg@3 F1 across the six maintenance models and select in ascending order, allowing at most one transition per repository, until 20 transitions are retained. This produces the hardest-20 subset without allowing multiple releases from the same repository to dominate it. For each selected transition and model, we keep the original maintenance scaffold, tools, 50-turn budget, release patch, and fixed V1 skill set unchanged, so the oracle hint described in Section 4.4 is the only intervention. Agents must still inspect the identified files, trace the relevant release changes, and determine the required updates. Each of the six models is run three times per transition, yielding 360 oracle runs. We compare the oracle runs with the original baseline runs restricted to the same 20 transitions using the identical patch-grounded removal metric. Per-model confidence intervals are bootstrapped over the 20 transition-level ∆F1 values. For the overall interval, we first average ∆F1 across the six models within each transition and bootstrap the resulting 20 transition-level means. Baseline and oracle precision are each aggregated over transitions for which precision is defined in the corresponding condition. The reported ∆P instead averages per-transition oracle-minus-baseline differences where both values are defined, and therefore need not equal the difference between the two displayed aggregate precision values.

C.3

Maintenance and Judge Instructions

We release the full instructions with the accompanying materials. Baseline maintenance agents receive the fixed V1 skill set and official release patch alone, with gold-affected paths supplied only in the oracle localization ablation of Appendix C.2. Maintenance instruction. The maintenance agent is instructed to read the existing skill set and official release patch, make minimal patch-grounded updates, cover affected prose, scripts, and references, and describe the post-release state directly rather than narrating the migration. It should preserve unaffected package structure unless the release requires a change, and it should finish within the fixed 50-turn budget. The total and remaining turn budget are reported throughout execution.

20

NL Judge instruction. The NL Judge works from the official release patch, the fixed V1 skill set, the produced V2 skill set, and limited run metadata alone, so its assessment stays independent of G∆ and the patch-grounded removal scores. The full rubric is included in the released materials.

D

Additional Results and Corpus Statistics

D.1

Full NL Judge Results

Table 10 reports the five dimension-level scores underlying the NL Judge results in Table 3. Form-gated outputs contribute zero to each dimension and to the Sum score. Values are averaged over all 315 runs for each maintenance model. Table 10 Full NL Judge results. Sum is the 0–10 total over five 0–2 dimensions: API correctness, Files consistency, Info retention, Prompt adherence, and NL current-state prose correctness. Invalid reports the number of form-gated runs. Dimension scores (0–2)

D.2

Total Count

Model

API Files Info Prompt

NL

GPT-5.4 judge Claude-opus-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 Doubao-Seed2-pro MiniMax-M2.5

1.38 1.20 1.25 1.06 1.05 0.93

1.00 1.03 1.04 0.92 0.96 1.02

1.75 1.76 1.52 1.74 1.81 1.86

1.65 1.37 1.66 1.20 1.23 0.92

1.76 1.74 1.70 1.57 1.49 1.28

Claude-opus-4.6 judge Claude-opus-4.6 1.38 GLM-5.1 1.20 GPT-5.4 1.24 Kimi-K2.5 0.99 Doubao-Seed2-pro 0.94 MiniMax-M2.5 0.86

1.24 1.09 1.26 0.89 0.90 0.79

1.60 1.61 1.32 1.53 1.54 1.53

1.84 1.36 1.76 1.24 1.33 0.86

1.81 1.72 1.73 1.59 1.52 1.23

Sum Invalid

7.54 7.09 7.18 6.49 6.54 6.01

7.86 6.98 7.31 6.24 6.22 5.27

9 7 26 14 15 8 9 7 26 14 15 8

Statistical Robustness of the Main Results

The headline metric gives equal weight to the 105 release transitions, but 28 of the 57 repositories contribute more than one transition. We therefore examine robustness to repository-level clustering, alternative repositorylevel aggregation, and run-to-run variability. Repository-clustered confidence intervals. We resample the 57 repositories with replacement, retaining all transitions from each sampled repository, and recompute avg@3 macro F1 over 10,000 bootstrap samples. Table 11 reports the resulting percentile 95% confidence intervals. Repository-level macro average. As an alternative aggregation, we first average transition-level avg@3 F1 within each repository and then average across the 57 repositories. This removes the additional weight given to repositories that contribute multiple transitions. The resulting repository-level macro scores preserve the model ordering and differ from the transition-level scores by only +0.9 to +3.4 points. Paired repository-clustered bootstrap. We further assess the three adjacent differences among the top four models using the same repository-level resampling while pairing scores within each transition. All three 95% confidence intervals exclude zero: Claude-opus-4.6 exceeds GLM-5.1 by 5.4 points ([+2.8, +8.0]), GLM-5.1 exceeds GPT-5.4 by 5.4 points ([+2.0, +8.7]), and GPT-5.4 exceeds Kimi-K2.5 by 11.9 points ([+8.1, +15.8]). Run-to-run variance. For each model and transition, we compute the standard deviation of F1 across the three runs and then average over transitions. The resulting values are 6.1 for Claude-opus-4.6, 9.0 for GLM-5.1, 11.6 for GPT-5.4, 12.0 for Kimi-K2.5, 11.9 for Doubao-Seed2-pro, and 14.7 for MiniMax-M2.5. Run-to-run variability is generally larger for the lower-scoring models, providing a complementary view of maintenance reliability alongside avg@3 and best@3. 21

Table 11 Robustness of the headline avg@3 macro F1 (%). “Transition-macro” is the main-text aggregation with equal weight per transition; the 95% CI is obtained by repository-clustered bootstrap. “Repository-macro” averages transition-level scores within each repository before averaging across repositories. Model

Transition-macro

95% CI (cluster)

Repository-macro

69.7 64.3 58.8 47.0 39.8 29.9

[65.7, 73.8] [60.5, 68.2] [55.1, 62.8] [42.8, 51.3] [35.3, 44.8] [26.0, 34.0]

72.7 67.2 62.3 49.0 40.7 31.8

Claude-opus-4.6 GLM-5.1 GPT-5.4 Kimi-K2.5 Doubao-Seed2-pro MiniMax-M2.5

D.3

Obsolete-Set Size Distribution

Table 12 summarizes the size of the patch-verified obsolete set G∆ across the 105 release transitions. The corpus contains 12,217 obsolete V1 skill lines in total, ranging from 5 to 375 per transition, with a mean of 116.4 and quartiles of 46, 92, and 156. Only 7 transitions contain fewer than 20 obsolete lines, whereas 48 contain at least 100. Thus, most selected transitions require updating a nontrivial amount of skill content. Table 12 Distribution of the patch-verified obsolete-set size |G∆ | over the 105 release transitions. |G∆ | (obsolete V1 skill lines)

D.4

Transitions

Share

5–19 20–49 50–99 100–199 ≥ 200

7 22 28 29 19

6.7% 21.0% 26.7% 27.6% 18.1%

All

105

100%

Corpus Composition and Difficulty

Section 4.1 summarizes the corpus along repository-domain and observed difficulty axes. Here we describe the corresponding grouping criteria and provide representative transitions. The released materials include the full per-transition assignments and underlying model scores. Difficulty stratification. For each transition, we compute avg@3 F1 for each of the six maintenance models and average the six values to obtain an observed tractability score. We classify transitions as Hard (< 0.40), Medium ([0.40, 0.65)), and Easy (≥ 0.65), yielding 25, 60, and 20 transitions, respectively. The thresholds fall in relatively sparse regions of the observed score distribution and are used only for descriptive analysis. Mean tractability scores are 29.9% for Hard, 53.0% for Medium, and 74.5% for Easy. Neither release-patch size nor |G∆ | enters the definition. Table 13 lists representative transitions from the two ends of this observed difficulty spectrum.

22

Table 13 Representative hardest and easiest transitions ranked by mean avg@3 F1 across the six maintenance models. Gold lines reports |G∆ |. Group

Transition

Gold lines

Mean F1

Hard Hard Hard Hard Hard

DeepSpeed-0.17.6-0.18.0 peft-v0.15.0-v0.16.0 agentmemory-0.9.13-0.9.16 pydantic-v2.7.1-v2.9.1 datasets-3.0.0-4.0.0

9 29 5 29 144

5.9 11.5 12.1 18.7 21.6

Easy Easy Easy Easy Easy

openhuman-v0.53.35-v0.53.40 langchain-langchain-core==1.0.7-langchain-core==1.1.0 milvus-v2.5.10-v2.5.15 oh-my-pi-15.1.3-15.2.4 warp-v1.8.0-v1.8.1

35 97 72 60 199

88.9 85.3 83.1 79.9 76.8

Domain composition. Each of the 57 repositories is assigned to one of five domains, and every transition inherits the domain of its repository. The corpus contains 19 agent and RAG framework repositories with 33 transitions, 15 training and inference infrastructure repositories with 32 transitions, 9 data and classic ML repositories with 17 transitions, 9 compiler, kernel, and systems repositories with 16 transitions, and 5 general-purpose library and application repositories with 7 transitions. Domain-level outcomes. Mean F1 varies across these domains: 53.8% for agent and RAG frameworks, 46.4% for training and inference infrastructure, 57.8% for data and classic ML, 51.6% for compiler, kernel, and systems software, and 49.5% for general-purpose libraries and applications. Training and inference infrastructure is disproportionately represented in the Hard stratum, contributing 9 of the 25 Hard transitions but only 2 of the 20 Easy transitions. These domain assignments are used only for descriptive corpus analysis and do not enter the construction of G∆ , the maintenance runs, or any headline metric.

23

Related documents

• TEACH and CARE: How Community-Centered Archival Practice May Transform Education, Archives, and Community History#167469
• Contingent Labour — The National Archives — Hays Specialist Recruitment Limited#608307
• SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution#203136
• Mentorship resources and citation-elite journal publication trajectories after training: Evidence from bioscience mentor-mentee networks#1070055
• 42 U.S.C. § 10135 — Review of repository site selection#590324