SkillMOO: Multi-Objective Optimization of Agent Skills for Software Engineering Jingzhi Gong1 , Ruizhen Gu2 , Zhiwei Fei3 , Yazhuo Cao1 , Lukas Twist1 , Alina Geiger4 , Shuo Han5 , Dominik Sobania6 , Federica Sarro5 , and Jie M. Zhang1
arXiv:2604.09297v1 [cs.SE] 10 Apr 2026
1
King’s College London {jingzhi.gong, yazhuo.cao, lukas.twist, jie.zhang}@kcl.ac.uk 2 Queen’s University Belfast [email protected] 3 Nanjing University [email protected] 4 Johannes Gutenberg University Mainz [email protected] 5 University College London {shuo.han.25, f.sarro}@ucl.ac.uk 6 University of Duisburg-Essen [email protected]
Abstract. Agent skills provide modular, task-specific guidance for LLMbased coding agents, but manually tuning skill bundles to balance success rate, cost, and runtime is expensive and fragile. We present SkillMOO, a multi-objective optimization framework that automatically evolves skill bundles using LLM-proposed edits and NSGA-II survivor selection: a solver agent evaluates candidate skill bundles on coding tasks and an optimizer agent proposes bundle edits based on failure analysis. On three SkillsBench software engineering tasks, SkillMOO improves pass rate by up to 131% while reducing cost up to 32% relative to the best baseline per task at low optimization overhead. Pattern analysis reveals pruning and substitution as primary drivers of improvement, suggesting effective bundles favor minimal, focused content over accumulated instructions. Keywords: SBSE · GenAI · AI Agents · Agent Skills · SE4AI · AI4SE
1
Introduction
Large language model (LLM)-based coding agents have emerged as powerful tools for software engineering (SE) tasks, from bug fixing to code migration [5]. As introduced by Anthropic, agent skills are portable folders that package instructions, scripts, and other resources that an agent can load on demand to improve performance on specialized tasks [2]. To systematically evaluate their utility, SkillsBench [4] benchmarks agent skills across diverse domains and shows that skills can improve task performance overall. However, SkillsBench finds that skill gains on SE tasks are modest (+4.5%) [4], and practitioners similarly observe that skill flexibility creates uncertainty about what works best, poor skill bundles may waste agent overhead, and skills need continuous refinement around failure cases [7]. Manual tuning of skill bundles to address these challenges is expensive, fragile, and often requires domain expertise that may not transfer across tasks.
2
Jingzhi Gong et al. NSGA-II survivor selection
Pass rate
SE tasks
Cost SkillsBench
Task Solver
Skill bundles LLM-based skill edits
Optimizer skill
Skill Optimizer
Error traces
Yes
Budget left? No
Final candidate for reporting
Fig. 1. SkillMOO workflow: solver-optimizer loop with evolving skill bundles.
This motivates search-based, data-driven skill optimization, and thereby we propose SkillMOO, a multi-objective optimization (MOO) framework that automatically evolves task-specific skill bundles for SE tasks using LLM-proposed edits and NSGA-II survivor selection on pass rate and cost: a task solver agent evaluates candidate skill bundles on coding tasks and a skill optimizer agent proposes bundle edits based on failure analysis. We evaluate SkillMOO on three SkillsBench SE tasks against original skill bundles and no skill baselines, and in summary, our main contributions are: – We propose SkillMOO, a MOO framework that combines LLM-proposed edits and NSGA-II survivor selection in a solver-optimizer loop. – We provide empirical evidence on three SkillsBench tasks demonstrating that skill optimization can simultaneously improve pass rate and reduce cost. – We analyze recurring skill-edit patterns and identify pruning and substitution as the primary drivers of observed improvements, offering practical insights for skill bundle design. – We release a replication package with data, scripts, and skill artifacts to support further research. Related Work. SkillsBench [4] introduced the first systematic paired evaluation of skills across diverse tasks, and SWE-Skills-Bench [3] found that many public SWE skills provide little benefit in practical SE tasks. SkillRouter [11] frames skill usage as a retrieval problem and shows that scalable deployment requires accurate routing over large skill pools. Meanwhile, recent skill-evolution approaches like EvoSkill [1], Meta Context Engineering [8], and EvoSkills [10] demonstrate that skills can themselves be optimized through iterative refinement. Our work builds on these directions but targets a different gap: (1) we target SE tasks in SkillsBench, and (2) we cast improvement as multi-objective search using LLM-proposed bundle edits and NSGA-II survivor selection on pass rate and cost, plus overhead and edit-pattern analysis.
2
SkillMOO
SkillMOO uses a two-agent iterative workflow with an evolving optimizer skill, as illustrated in Figure 1. Generation 0 starts from the original skill bundle, and the
SkillMOO: Multi-Objective Optimization of Agent Skills
3
Table 1. Tasks, Skill Bundles, and Benchmark Tests for the Evaluation of SkillMOO ID
Task Name (Description)
Task 1
fix-build-agentops (repair Python build failures and produce a valid patch + run note) python-scala-translation (translate Python pipeline logic to Scala with equivalent behavior) spring-boot-jakartamigration (migrate Spring Boot app to Jakarta APIs without regressions)
Task 2
Task 3
# Skill Skills (Count) Bundle 8
6
5
build/dependency triage (3); pytest/testing diagnosis (3); patch+diff workflow (1); CI/replay/report hygiene (1) collections/libraries mapping (2); functional/OOP structure guidance (2); syntax mapping (1); language idioms (1) framework/dependency modernization (2); namespace migration (1); security migration (1); REST client migration (1)
Benchmark Tests (Count) env execution coverage (18); patch safety (7); runtime instrumentation (7); regression tests (5); artifact/replay sanity (3) source/API contract tests (20); behavioral tests (17); compile/unit-test gates (3) RestClient migration tests (10); namespace migration tests (9); security migration tests (8); feature preservation tests (6); dependency update tests (5); build/compile tests (2)
task solver agent executes the task to obtain pass rate, cost, and error traces. Guided by the current optimizer-skill prompt and the failure evidence from the current parent candidate, the skill optimizer agent proposes an edited child skill bundle through operations such as pruning, substitution, reordering, and rewriting, and updates the optimizer skill itself for the next generation. The edited child bundle is then re-evaluated by the task solver, added to the search archive, and the loop repeats until the generation budget is reached. During search, SkillMOO optimizes candidate skill bundles(B) with a biobjective formulation: min f (b) = [−pass(b), cost(b)] b∈B
where b is the skill, pass(b) is test pass rate and cost(b) is LLM inference cost. After each child evaluation, all archived candidates are ranked with NSGAII survivor selection using non-dominated sorting and crowding-distance tiebreaking on these two objectives; the non-dominated candidates form the current Pareto front. We also enforce a pass-preservation guard that rejects a proposed child when its pass rate drops by more than 0.05 from its parent. Runtime is tracked as a secondary metric and used when choosing the final reporting seed via lexicographic order.
3
Experimental Setup
We selected three tasks with the largest original skill bundles from SkillsBench [4] to focus on settings where manual skill tuning is hardest and quality-cost-runtime tradeoffs are most pronounced. We limited the study to three tasks due to time constraints. Because the default verifier1 coverage in this subset is minimal (roughly 3-10 tests), we enhanced each task to 40 tests via GPT-5.4 to provide denser behavior and safety assertions while preserving compile/build gates, as summarized in Table 1. We use GLM-5 [9] for both task solver and skill optimizer agents. Due to time constraints, the search uses population size 1 and runs for 5 generations 1
The set of automatic tests used to judge each solution.
4
Jingzhi Gong et al.
Table 2. Effectiveness against baselines, reported with mean ± standard deviation (SD) and Scott-Knott ESD rank for pass rate (rp ). Task
Method
#Skills Used
Pass rate
Cost (USD)
Runtime (s)
rp
Task 1
SkillMOO ori_skill no_skill
4 8 0
0.37±0.16 0.16±0.08 0.10±0.00
1.10±0.33 1.61±0.41 1.06±0.71
1060.9±280.0 1388.8±97.8 989.2±265.5
1 2 3
Task 2
SkillMOO ori_skill no_skill
2 6 0
0.51±0.12 0.39±0.06 0.39±0.16
1.06±0.25 1.12±0.36 0.85±0.31
612.4±214.2 665.3±189.2 455.2±232.7
1 2 2
Task 3
SkillMOO ori_skill no_skill
5 5 0
0.99±0.01 0.97±0.00 0.91±0.01
1.25±0.49 1.55±0.27 1.14±0.23
317.4±36.7 452.0±216.0 685.5±533.1
1 2 3
(generation 0 seed evaluation plus generations 1–4 optimization steps). Timeout settings are 900 seconds for both agents. Final selection from the Pareto front follows preference order: maximize pass rate, then minimize cost, then runtime. Each method/task is executed 10 times, and Scott-Knott Effect Size Difference (ESD) pass-rate ranks [6] are computed from these 10-run vectors. We evaluate three methods, SkillMOO, ori_skill (original skill bundle), and no_skill (no guidance), with three research questions (RQs): ➤ RQ1. How do final SkillMOO candidates compare with baselines on pass rate, cost, and runtime? ➤ RQ2. Is optimization economically worthwhile accounting for overhead? ➤ RQ3. Which skill-edit patterns recur in improved candidates?
4
Results and Analysis
RQ1: Effectiveness of SkillMOO. Table 2 reports ten-run means with GLM-5. SkillMOO achieves higher pass rate than ori_skill on all three tasks, with improvements ranging from 2.1% (Task 3) to 131.2% (Task 1). On Task 1, SkillMOO raises pass rate from 0.16 to 0.37 (+131.2%) while reducing cost by 31.7% (1.10 vs 1.61 USD) and runtime by 23.6% (1060.9s vs 1388.8s). On Task 2, pass rate improves from 0.39 to 0.51 (+30.8%), with cost reduced by 5.4% (1.06 vs 1.12 USD) and runtime by 8.0% (612.4s vs 665.3s). On Task 3, pass rate rises from 0.97 to 0.99 (+2.1%), while cost drops by 19.4% (1.25 vs 1.55 USD) and runtime drops by 29.8% (317.4s vs 452.0s). Scott-Knott ESD assigns SkillMOO the top pass-rate rank for each task, as in table 2. Against no_skill, SkillMOO retains higher pass rate on every task, but cost and runtime are often higher. This pattern is expected because SkillMOO runs with substantive skill guidance rather than an empty prompt. RQ1 answer: Against the best static baseline per task, SkillMOO improves pass rate on all tasks (+2.1% to +131.2%) while lowering cost (-5.4% to 31.7%), with statistical significance.
SkillMOO: Multi-Objective Optimization of Agent Skills
5
Table 3. SkillMOO optimization overhead (generation 1–4), two-objective HV improvement (∆HV (%)), and optimization efficiency (Cost/∆HV%). Task Task 1 Task 2 Task 3
Opt. Cost (USD)
Opt. Runtime (s)
HV (ori_skill)
HV (SkillMOO)
∆HV (%)
Cost/∆HV%
2.2676 1.8611 1.7641
2305.00 1545.08 867.40
0.0081 0.0056 0.0078
0.1783 0.0296 0.0311
2110 430 301
0.0011 0.0043 0.0059
Table 4. Pattern analysis of skill-bundle operations over all available candidates. Operation Description Bundle pruning (remove skill blocks) Bundle substitution (swap skill blocks) Bundle expansion (add skill blocks) Remove RestTemplate mentions (incl. comments/Javadocs) Remove unused guidance sections (never-invoked) Add OOP translation guidance (class/trait/component) Dead-weight pruning (keep effective, drop unused) Migration sequence reorder (bootstrap/build first) Functional-style translation guidance Explicit remove_skill (telemetry only) Explicit reorder_bundle (telemetry only)
#Edits Pass↑ Cost↓ Time↓ 7 7 5 2 2 3 1 1 1 1 1
5/7 5/7 0/5 2/2 0/2 0/3 0/1 1/1 0/1 1/1 0/1
7/7 7/7 5/5 2/2 2/2 3/3 1/1 1/1 1/1 1/1 1/1
4/7 4/7 5/5 1/2 2/2 3/3 1/1 0/1 1/1 1/1 1/1
RQ2: Optimization Efficiency. Table 3 reports the full optimization cost/runtime and resulting hypervolume (HV) uplift over ori_skill, where HV is computed in a two-objective space: passed tests (maximize) and cost (minimize). The last column shows Cost/∆HV% of $0.0011 for Task 1, $0.0043 for Task 2, and $0.0059 for Task 3, so each percentage point of relative HV gain costs well under one U.S. cent in search spend. Notably, Task 2 has the lowest baseline HV (0.0056) yet remains modest at $0.0043 per HV% improvement because relative uplift in pass–cost space stays large. RQ2 answer: SkillMOO improves multi-objective quality on all tasks (HV gains of 301%–2110%), with optimization efficiency strongest on Task 1 ($0.0011 per %HV gain). RQ3: Skill-Edit Pattern Evidence. For pattern analysis, we enumerate all logged skill-bundle edits and group them by operation description text produced by the optimizer agent during search; the RQ3 summarization step then groups them deterministically with GPT-5.4. Table 4 reports operation description, edit frequency, and improvement counts measured against ori_skill baselines. Table 4 reveals that bundle pruning and bundle substitution are the most frequent operations with 7 edits each, and both achieve cost reduction in 7/7 cases. Bundle expansion shows 0/5 pass improvements, indicating that adding new guidance rarely helps pass rate. These observational patterns suggest that optimization often benefits from removing irrelevant guidance and selectively replacing misaligned content, rather than adding new instructions. Task-specific patterns also emerge2 . For Task 3, removing RestTemplate mentions (including comments/Javadocs) directly addresses strict verifier assertions 2
Detailed task-specific results are provided in the replication package.
6
Jingzhi Gong et al.
(2/2 pass improvements). For Task 2, adding OOP translation guidance targets missing structural elements but does not improve pass rate (0/3). RQ3 answer: Pruning and substitution operations dominate successful edits; cost reductions are consistent across operations, while pass improvements depend on task-specific misalignment in the original bundle. Threats to Validity. First, our study uses three selected tasks from a fixed 16-task SE pool, limiting external validity beyond this setting. Second, RQ3 provides observational pattern evidence, not causal attribution of individual edits. Third, verifier expansion to 40 tests via GPT-5.4 may introduce model-linked bias even though compile/build gates are preserved. Finally, we report results for one model (GLM-5) and the three tasks with the largest original bundles; generalization to other LLMs and smaller tasks requires further study.
5
Conclusion
We presented SkillMOO, a framework that evolves agent skill bundles with LLMproposed edits and NSGA-II survivor selection. The study demonstrates the effectiveness and efficiency of using multi-objective search over skills to improve software engineering agent outcomes compared with fixed bundles, and suggests pruning and substitution matter more than simply adding more guidance. Availability. Replication package at: https://doi.org/10.5281/zenodo.19488943.
References 1. Alzubi, S., et al.: Evoskill: Automated skill discovery for multi-agent systems (2026), https://arxiv.org/abs/2603.02766 2. Anthropic: Introducing agent skills. https://claude.com/blog/skills, published October 16, 2025 3. Han, T., Zhang, Y., Song, W., Fang, C., Chen, Z., Sun, Y., Hu, L.: Swe-skillsbench: Do agent skills actually help in real-world software engineering? (2026), https://arxiv.org/abs/2603.15401 4. Li, X., Chen, W., Liu, Y., et al.: Skillsbench: Benchmarking how well agent skills work across diverse tasks (2026), https://arxiv.org/abs/2602.12670 5. Liu, J., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., Lou, Y.: Large language model-based agents for software engineering: A survey. ACM Transactions on Software Engineering and Methodology (2024) 6. Scott, A.J., Knott, M.: A cluster analysis method for grouping means in the analysis of variance. Biometrics pp. 507–512 (1974) 7. Shihipar, T.: Lessons from building claude code: How we use skills. https://x. com/trq212/status/2033949937936085378, published March 17, 2025 8. Ye, H., He, X., Arak, V., Dong, H., Song, G.: Meta context engineering via agentic skill evolution (2026), https://arxiv.org/abs/2601.21557 9. Zeng, A., Lv, X., Hou, Z., Du, Z., et al.: Glm-5: from vibe coding to agentic engineering (2026), https://arxiv.org/abs/2602.15763
SkillMOO: Multi-Objective Optimization of Agent Skills
7
10. Zhang, H., Fan, S., Zou, H.P., Chen, Y., Wang, Z., Zhou, J., Li, C., Huang, W.C., Yao, Y., Zheng, K., Liu, X., Li, X., Yu, P.S.: Evoskills: Self-evolving agent skills via co-evolutionary verification (2026), https://arxiv.org/abs/2604.01687 11. Zheng, Y., Zhang, Z., Ma, C., Yu, Y., Zhu, J., Wu, Y., Xu, T., Dong, B., Zhu, H., Huang, R., Yu, G.: Skillrouter: Skill routing for llm agents at scale (2026), https://arxiv.org/abs/2603.22455