SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

2026-8-4

PREPRINT

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Zelin Tan† , Yiqun Zhang, Hao Li, Zhiyao Cui, Hejia Geng, Shao Zhang, Hangfan Zhang, Yang Chen, Xiaosong Wang, Lilong Wang, Zhenfei Yin, Shuyue Hu‡ , Chen Zhang∗ , Lei Bai‡∗ Shanghai Artificial Intelligence Laboratory ∗ Corresponding authors.

‡ Project Lead.

arXiv:2608.02287v1 [cs.AI] 3 Aug 2026

Abstract Agent Skills have become an important mechanism for equipping language-model agents with reusable procedural knowledge. However, providing skills alone does not guarantee that current models can identify, apply, and coordinate them effectively. To improve models’ skill-use capabilities, we introduce SKT, a verified data synthesis pipeline that constructs skill-grounded tasks and executable trajectories from large collections of Agent Skills. SKT selects suitable single-skill and multi-skill configurations, synthesizes tasks through rule-based and agent-based verification with feedback-guided repair, and retains only successful trajectories that substantively use every required skill. Using 2,000 public skills, SKT produces 4,000 task packages and 27,164 verified trajectories. Based on the same pipeline and a disjoint task pool, we further construct SkillEval, a held-out executable benchmark for evaluating skill use. Experiments across diverse models, benchmarks, and agent harnesses show that supervised fine-tuning on SKT-generated trajectories consistently improves skill-use performance. Verification ablations, cross-harness evaluation, and scaling experiments further show that these gains depend on high-quality supervision, extend beyond a single agent interface, and increase with broader skill coverage. Together, these results establish verified data synthesis as an effective and scalable approach to skill-use training. Hugging Face: https://huggingface.co/datasets/Artemis0430/skilleval-v1 # Contact: {zhangchen1,bailei}@pjlab.org.cn

1. Introduction An Agent Skill is an organized package of instructions, metadata, and optional resources (e.g., scripts and templates) that can be discovered, loaded, and utilized by a large language model (LLM) (Zhang et al., 2025). By encapsulating domain knowledge, procedural workflows, and auxiliary tools, Agent Skills extend model capabilities beyond their intrinsic knowledge, adapt them to domain-specific tasks, facilitate the sharing and reuse of expert knowledge, and support the composition of multiple skills for solving complex tasks. According to public records, since October 2025, more than 600,000 skills have been made publicly available, and this number continues to grow (Qu, 2026; Zhang et al., 2025). However, the availability of a skill does not necessarily imply that a model can effectively utilize it. Although recent studies have demonstrated that proper skill utilization can substantially improve task success rates, a great number of LLMs do not inherently possess this ability (Han et al., 2026; Li et al., 2026b). Effective skill utilization remains a non-trivial challenge: a model must be able to identify a skill when it is relevant and applicable, understand and follow its operational constraints, coordinate it with other complementary skills, and correctly execute the prescribed procedures following skill instructions while effectively utilizing the associated tools (Li et al., 2026a).

† Work done during an internship at Shanghai Artificial Intelligence Laboratory.

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

In this paper, we investigate how to enable LLMs to effectively utilize skills. Despite the growing interest in Agent Skills, this fundamental question remains largely underexplored. Existing studies have primarily focused on efficiently retrieving skills from skill ecosystems (Li et al., 2026a), internalizing skills into models to reduce reliance on external skill documents (He et al., 2026; Lin et al., 2026; Lu et al., 2026; Zhu et al., 2026a), and leveraging skills to support the self-improvement of LLM-based agents (Shi et al., 2026; Vishe et al., 2026; Wang et al., 2025; Xia et al., 2026). However, these approaches mainly address how skills are retrieved, represented, or incorporated into models, rather than how models can effectively understand, coordinate, and leverage skills to solve diverse tasks. To this end, we introduce Skill-use Training (SKT), a framework for synthesizing high-quality training data to equip LLMs with effective skill-use capabilities. The key idea of SKT is to construct skill-grounded tasks that are solvable, verifiable, and complexity-controllable, enabling models to learn when and how to apply skills across diverse scenarios. SKT first curates skills from public skill repositories (Qu, 2026) and constructs a candidate skill pool based on quality, diversity, and composability. Based on this pool, SKT synthesizes skill-grounded tasks with controlled complexity, ranging from single-skill scenarios to multi-skill compositions. Each task includes not only structured specifications but also reference solutions and verification mechanisms, ensuring task solvability and verifiability. These tasks later undergo automatic assessment to ensure appropriate difficulty and genuine dependence on the designated skills. Finally, SKT collects and filters execution trajectories from a strong LLM through an interaction harness, retaining only successful trajectories with faithful skill utilization as high-quality supervision for skill-use training. Using SKT, we construct a skill-use training corpus from 2,000 public skills, producing 4,000 synthetic tasks and 27,164 verified execution trajectories. We use these trajectories as supervision to train Qwen3.5-9B (Qwen Team, 2026) and Gemma 4 E4B-IT (Gemma Team, 2026), and evaluate the resulting models on four benchmarks: SkillsBench (Li et al., 2026b), MolBench-Bind (Zhang et al., 2026), AgentSkillOS-bench (Li et al., 2026a) , and SkillEval. The latter is constructed using SKT

Before SKT

After SKT

LLM Agent Cannot Effectively Use Skills

LLM Agent Can Effectively Use Skills

to-prd.md

markdown.md

gif-creator.md

frontend.md

Task: Invoice Fraud Detection

1.pdf-scan-extract.md

Detect fraudulent invoices by checking vendor, amount, IBAN, and PO information

2.xlsx-load-lookup.md

Invoices.pdf

vendors.xlsx porders.csv

xlsx.md

3.Fuzzy-math-vendor.md 4.Validat Fields 5.Generate Reports

canvas.md

LLM

pdf.md

… pptx.md

LLM docx.md

Which skill? When to use? How to use? …

Figure 1 | SKT is a multi-agent-based data synthesis pipeline for skill-use tasks and trajectories. Training on data synthesized by SKT can effectively improve agents’ ability to use Agent Skills.

2

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

from a held-out skill pool with no overlap with the training corpus, demonstrating that SKT can also support scalable benchmark construction. As shown in Figure 1, models trained with SKT consistently achieve improvements in skill-use performance across different harnesses and benchmarks. Further analyses, including ablations of verification components, cross-harness evaluations, and scaling studies, demonstrate that these gains arise from high-quality supervision, generalize across agent interfaces, and scale with broader skill coverage. Our key contributions are threefold: • We introduce SKT, a framework that empowers LLMs with the ability to effectively utilize Agent Skills. • We develop a scalable data synthesis pipeline that generates skill-grounded training tasks with solvability, verifiability, and complexity control, and supports the construction of held-out benchmarks. • We present SkillEval, a new benchmark for skill-use evaluation, and conduct extensive experiments across models, harnesses, and benchmarks, demonstrating the effectiveness, scalability, and generalizability of SKT.

2. Related Work Learning and evolving external skills. A line of work treats skills as persistent, editable memories that co-evolve with an agent. SAGE accumulates executable skills across sequential rollouts and rewards both skill creation and reuse (Wang et al., 2025). SkillRL distills successful demonstrations and failure lessons into a hierarchical SkillBank, then updates the bank from validation failures as the policy learns (Xia et al., 2026). Skill1 more tightly couples the lifecycle by training one policy to select, use, and distill skills from a shared outcome signal (Shi et al., 2026). Skill-R1 instead freezes the task model and trains a lightweight editor to revise textual skills over multiple verifier-scored generations (Vishe et al., 2026). Skill internalization. Another line uses external skills as temporary training scaffolds. SKILL0 progressively withdraws skills according to their measured on-policy helpfulness (Lu et al., 2026), whereas SkillC turns paired rollouts with and without skills into a direct contrastive credit-assignment signal (Lin et al., 2026). SIRI mines skills from the policy’s own successful rollouts, validates their utility, and distills only beneficial skill-guided actions into a skill-free policy (He et al., 2026). Skill0.5 adopts a hybrid strategy, internalizing general skills while enforcing the use of task-specific skills to improve out-of-distribution transfer (Zhu et al., 2026a). Parameterized skill representations. Weight-space approaches replace repeated skill-text injection with modular parameters. Skill-to-LoRA synthesizes skill-guided demonstrations and distills each SKILL.md into a separately loadable adapter (Zhang and Qi, 2026). LatentSkill and ParametricSkills instead train hypernetworks that map textual skills to LoRA weights without per-skill backbone tuning; both investigate parameter-space composition, while ParametricSkills also trains on singleand multi-turn skill-exploitation trajectories and supports test-time skill evolution (Yu et al., 2026; Zhao et al., 2026). Evaluating Agent Skills. SkillsBench measures the marginal value of curated skills through paired skill/no-skill execution across diverse domains and shows that focused skill sets can outperform

3

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

exhaustive bundles (Li et al., 2026b). SWE-Skills-Bench applies controlled, execution-based evaluation to real software repositories and finds that gains are highly dependent on skill specificity and contextual compatibility (Han et al., 2026). AgentSkillOS studies skill discovery and DAG-based orchestration at ecosystem scale, together with an artifact-production benchmark (Li et al., 2026a); MolClaw organizes drug-discovery tools into a three-level skill hierarchy and introduces MolBench to evaluate the resulting agent on long-horizon workflows (Zhang et al., 2026). Agentic data synthesis. STEPS organizes capabilities into a hierarchical skill taxonomy and samples coherent skill combinations for compositional instruction synthesis (Wei et al., 2026). AgentSynth composes independently generated subtasks into long-horizon computer-use tasks, with complexity controlled by the number of subtasks (Xie et al., 2026). For terminal agents, TermiGen jointly synthesizes executable environments and trajectories containing error-recovery behavior (Zhu et al., 2026b), while CLI-Universe constructs and verifies containerized tasks sampled from a multidimensional capability taxonomy (Hua et al., 2026). SkillSynth samples workflows from a scenario-mediated skill graph to diversify required execution paths (Fan et al., 2026). Terminal-World directly uses Agent Skills to co-generate task instructions, environments, and teacher trajectories, and composes skills into teams and graphs for broader task coverage (Cheng et al., 2026).

3. Method This section presents our three-stage pipeline SKT. It first selects suitable single- and multi-skill configurations, then synthesizes and verifies executable task packages, and finally collects and validates skill-use trajectories. We also describe how the verified trajectories are used for supervised fine-tuning. Figure 2 summarizes the overall process.

Skills Curation

Tasks Synthesis

Template-driven Synthesis

Suitability

Selected

Skill

Library

or

Skill

Open

Re

Skills

Multi-skill

-

Template

Task

G

en

TaskRepair

instruction.md

Task

I

nstruction

synthesis

Composability

environment/

Rule-based Verifier

Trajectory Synthesis

Feedback

Rubric-based LLM Filter

Structure

Reference

Reproduce

Skill

N

Runtime Conte

x

t

data+SetupFile/ o-Leakage

skills/ task.toml

Agent-based Verifier

Configuration

Trajectory

Model

Re-synthesis

Task

Leakage

Skill

Multi-skill

Description

Check

Dependence

necessity

✖️

Difficulty Controller

LLM as Validator

Accepted

Harness

Valid Trajectory Set

Evaluator

test.sh

Rule-based

Validator

test/

Task Set

Too easy

Trajectory

Analysis

Pass

Agent

solution/ solve.sh

Reference Solution

Pass Rate Estimation

Figure 2 | Overview of SKT. A rubric-based filter selects suitable single- and multi-skill configurations. Template-driven synthesis produces executable task packages, which pass through rule-based, agentbased, and difficulty checks; failed tasks return to the author through a feedback loop. Accepted tasks are then solved by a model–harness pair, and a second verification stage retains only correct and faithful skill-use trajectories for SFT; rejected rollouts are freshly resampled without verifier feedback. The right panel expands the task-package template.

4

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

3.1. Stage I: Skill Curation From a candidate collection C, a rubric-based LLM judge retains skills that can ground executable, objectively checkable tasks. We then sample a skill set S ⊆ C with configurable cardinality 𝑘 = |S|. For 𝑘 > 1, a composition judge retains S only if its skills form a coherent workflow with distinct roles; rejected sets are resampled. For 𝑘 = 1, skills are sampled directly. Our experiments use 𝑘 ∈ {1, 2, 3}. The selected skills are then passed to the task synthesis stage. 3.2. Stage II: Task Synthesis Template-driven authoring. Given S, TaskGen reads every skill and fills the fixed template in Figure 2. The task package T contains the instruction, isolated runtime, execution settings, and executable evaluator, and reference solution. It must be objectively gradable, and assign an intended role to every selected skill. Rule-based task verifier. We first apply deterministic checks to failures that can be established from the package itself. The verifier confirms that all required components are present, file paths are valid, evaluator definitions are supported, and the authoring trace records access to every skill in S. It then executes the reference solution in a clean workspace initialized only from the task inputs; the produced artifacts must receive full evaluator credit. Finally, it compares solver-visible task materials against the references and recorded skill requirements to detect reference values or skill rules copied verbatim outside the supplied skill files. Agent-based task verifier. Deterministic checks cannot establish whether a task is semantically well posed. An LLM verifier therefore reviews the task package and checks that the instruction clearly specifies the required artifact and output location, all information needed for completion is available in the local environment, and solver-visible materials outside the supplied skill files do not semantically reveal the hidden answer, evaluation logic, or decisive skill-derived requirements. Skill dependence. The verifier uses paired rollouts to check the task’s dependence on skill: S is available in one rollout and withheld in the other, while the task and all remaining settings are fixed. The evaluator scores both outputs, after which the verifier compares the score difference to determine whether skill access benefits task completion at the bundle level. Difficulty control. To filter tasks that are valid but too easy, a fixed solver and harness attempt T with the selected skills available for 𝑁 independent rollouts. If 𝑟 𝑗 is the evaluator score of attempt 𝑗, we estimate 𝑁 1 ∑︁ 𝑝pass (T , S) = 1[ 𝑟 𝑗 = 1] . 𝑁

𝑗=1

A task with 𝑝pass ≥ 𝜃easy is sent back for repair. The controller analyzes its rollouts to identify shortcuts and suggest how to increase reasoning or execution demands. Feedback-guided task repair. The first failing gate sends TaskRepair concrete rule errors, semantic agent feedback, or trajectory-grounded difficulty suggestions. After the package is revised, validation restarts at the rule-based gate so that new defects are detected. Only task–configuration pairs that pass all three gates enter Dtask ; candidates that exhaust the repair budget are discarded.

5

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

3.3. Stage III: Trajectory Synthesis For each (T , S) ∈ Dtask and each configured teacher–harness pair ( 𝑚, ℎ), teacher 𝑚 solves the task through harness ℎ with its files and selected skills available. We retain the complete trajectory 𝜏, including model messages, tool calls and results, and output artifacts. The rule-based validator requires full evaluator credit, normal and complete termination, a wellformed tool trace, and explicit skill access. The LLM validator then checks that each skill was consulted before the action it should guide, affected concrete decisions or operations, and was applied correctly. A failed rollout is resampled from the same task without exposing the failure or verifier feedback to the next attempt. The first rollout that passes both validators for a task–teacher–harness pairing enters Dtraj ; exhausting its rollout budget yields no training example for that pairing. 3.4. Training on Verified Trajectories The valid trajectory set at the output of Figure 2 is converted into harness-native training examples. We apply masked autoregressive supervised fine-tuning; for a tokenized trajectory 𝜏 = ( 𝑥1 , . . . , 𝑥𝑇 ), the objective is ∑︁ LSFT = − log 𝑝𝜃 ( 𝑥𝑡 | 𝑥 <𝑡 ) , 𝑡 ∈ A (𝜏)

where A ( 𝜏) contains all assistant-generated reasoning, tool-call, and response tokens. System and task messages and tool observations remain in the conditioning context but are masked from the loss. Thus the model learns the complete skill-use process—from locating relevant instructions to executing and checking skill-guided actions—rather than only the final answer.

4. Experiments 4.1. Experimental Setup Models and Training. We study Qwen3.5-9B (Qwen Team, 2026) and Gemma 4 E4B-IT (Gemma Team, 2026), using full-parameter supervised fine-tuning on the end-to-end trajectory objective from Section 3. All runs use LLaMA-Factory (Zheng et al., 2024) for one epoch, with a learning rate of 5 × 10−6 , a cosine schedule, a 3% warmup ratio, bfloat16 precision, and an effective batch size of eight. We retain complete trajectories whose tokenized input targets contain at most 64k tokens. Data Synthesis. From the public skills.sh library (Qu, 2026), we select 2,000 distinct skills and synthesize 4,000 accepted task packages: 1,520 single-skill, 1,295 two-skill, and 1,185 three-skill tasks. Skill selection, task generation and repair, skill-dependence probing, and task and trajectory verification use DeepSeek V4 Pro (DeepSeek-AI, 2026) with a Claude Agent harness (Anthropic, 2025); difficulty control uses Qwen3.5-35B-A3B with OpenCode (Anomaly, 2026). We run five difficulty rollouts and repair a task if at least three receive full credit ( 𝑁 = 5, 𝜃easy = 0.6). Trajectory Collection. We pair four teachers—MiniMax-M2.5 (MiniMax, 2026), GLM-5 (GLM-5 Team, 2026), Qwen3.5-397B-A17B, and DeepSeek V4 Pro—with both DeepAgents (LangChain, 2025) and OpenCode. From the 32,000 candidate task–teacher–harness combinations, verification retains 14,277 DeepAgents and 12,887 OpenCode trajectories. Each is one complete, uninterrupted rollout.

https://openreward.ai/benchflow/skillsbench

6

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Public Benchmarks. We evaluate on three public benchmarks. SkillsBench (Li et al., 2026b) uses the 77-task test split of the OpenReward implementation. It evaluates multi-step, skill-based problem solving across engineering, science, finance, software development, and data processing. Each task runs in an isolated container and receives a normalized reward from task-specific tests, which can assign partial credit; no LLM grader is used. MolBench-Bind is the 37-task MS2 binding-affinity subset of MolBench (Zhang et al., 2026). Each task presents a protein target and two candidate molecules as SMILES strings and asks the agent to select the molecule with higher or lower affinity under the specified 𝐾𝑖 direction. AgentSkillOS-Bench (Li et al., 2026a) contains 30 multi-format artifactproduction tasks, with six tasks each in data computation, document creation, motion/video, visual creation, and web interaction. SkillEval. We construct SkillEval using the same task-synthesis pipeline that produces the tasks underlying our training trajectories, but from a separately generated pool. Neither these tasks nor trajectories collected on them are included in the SFT data. SkillEval is a cross-domain executable benchmark of skill-grounded agent capabilities. It covers software development and debugging, data analysis and machine learning, security, and finance and business analysis. The benchmark contains 100 tasks over 100 distinct skill groups: 30 single-skill, 46 two-skill, and 24 three-skill tasks. Metrics and Evaluation Protocol. For SkillsBench and SkillEval, we report the mean normalized reward over all tasks. MolBench-Bind reports exact-match accuracy, while AgentSkillOS-Bench reports the mean normalized artifact-based score. We express every metric on a 0–100 scale, with higher values indicating better performance. Unless a no-skill control is explicitly stated, each task is evaluated with its designated external skills available. Each task is executed in a fresh, isolated session and workspace, with no model context or task-generated state carried over between tasks. We run every complete benchmark four times with temperature=0.7 for each model–harness condition and report the arithmetic mean and sample standard deviation (with 𝑛 − 1 in the denominator) of the four run-level scores. 4.2. Overall Effectiveness Across Models and Harnesses We first examine whether training on verified trajectories consistently improves skill-augmented agents when the training and evaluation harnesses are matched. For each backbone–harness combination, Original denotes the off-the-shelf checkpoint, whereas SKT denotes the same backbone fine-tuned on verified trajectories collected with the corresponding evaluation harness. Both checkpoints receive exactly the same task-designated skills at inference time. This paired comparison keeps the backbone, agent harness, benchmark, and inference-time skill access fixed, thereby directly measuring the effect of verified trajectory training in the aligned setting. Table 1 reports results for two backbones, two agent harnesses, and four benchmarks, yielding 2 × 2 × 4 = 16 comparisons. SKT improves the mean score in every comparison, with absolute gains ranging from 3.20 to 18.91 points. The smallest gain occurs for Gemma 4 E4B-IT with OpenCode on SkillsBench, where performance increases from 7.08 to 10.28. The largest occurs for Qwen3.5-9B with DeepAgents on SkillEval, where the score rises from 51.62 to 70.53. Thus, the overall result is not driven by a particular backbone or runtime: all eight Qwen3.5-9B comparisons, all eight Gemma 4 E4B-IT comparisons, and every comparison under both OpenCode and DeepAgents improve after training.

7

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Model

Harness

OpenCode Qwen3.5-9B DeepAgents OpenCode Gemma 4 E4B-IT DeepAgents

Checkpoint

General-Purpose Skill Use

Molecular Science

Artifact Production

SkillsBench

SkillEval

MolBench-Bind

AgentSkillOS

Original SKT Original SKT

5.80 ± 1.03 15.79 ± 1.69 4.94 ± 0.70 13.62 ± 0.82

55.24 ± 0.72 72.48 ± 1.32 51.62 ± 1.16 70.53 ± 1.24

33.11 ± 2.59 44.59 ± 1.56 31.76 ± 1.35 47.30 ± 1.56

79.61 ± 1.64 84.70 ± 2.17 74.03 ± 2.03 79.84 ± 1.82

Original SKT Original SKT

7.08 ± 1.33 10.28 ± 1.16 5.07 ± 0.95 9.24 ± 1.05

61.01 ± 2.05 72.85 ± 1.30 56.14 ± 1.73 66.39 ± 2.11

34.46 ± 1.35 45.27 ± 1.35 31.76 ± 2.59 41.89 ± 1.56

49.41 ± 1.28 56.74 ± 0.93 42.23 ± 0.85 47.38 ± 1.47

Table 1 | Matched-harness performance with externally supplied skills. Values are mean ± sample standard deviation over four complete benchmark runs on a 0–100 scale. Boldface marks the higher mean within each model–harness pair. The gains also extend across benchmark categories. On SkillsBench, SKT improves performance by 3.20–9.99 points across the four model–harness settings, while the corresponding gains on SkillEval reach 10.25–18.91 points. Improvements on MolBench-Bind are consistently double-digit, ranging from 10.13 to 15.54 points, demonstrating that the benefit extends to specialized molecular-science decisions. AgentSkillOS likewise improves by 5.09–7.33 points across all settings. Notably, Qwen3.59B with OpenCode rises from 79.61 to 84.70 on AgentSkillOS despite the already strong Original checkpoint, showing that SKT remains beneficial even at a comparatively high starting score. 4.3. Understanding the Improvement Dependence on External Skills We next test how much of the SFT advantage remains without inference-time skill access. On SkillsBench and SkillEval, we compare Original and SKT under both harnesses with the designated skill files either withheld or provided; all other settings remain fixed. Figure 3 shows gains of only 0.53–5.69 points when skills are withheld, versus 8.68–18.91 points when they are provided. This gap appears on both benchmarks and under both harnesses. Verified vs. Unverified Data Pipelines We compare the full quality-controlled pipeline against raw synthesis by constructing a comparable set of tasks and trajectories without verification or repair, then fine-tuning Qwen3.5-9B with OpenCode Original

OpenCode

20

SKT

DeepAgents

OpenCode

15.8

SkillEval

SkillsBench

6.4 5.0 5.8

0

Withheld

Provided

1.9 1.4

Withheld

50

42.1

55.2

Provided

51.6 35.0

36.4 4.9

70.5

70

13.6

10

DeepAgents

72.5

30.3

30 Withheld

Provided

Withheld

Provided

Figure 3 | Absolute Original and SKT scores for Qwen3.5-9B with task-designated skills withheld or provided. 8

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Unverified SFT

SKT

Original = 0

-1.9

SkillsBench MolBenchBind

-6.1

SkillEval

-5.6

+10.0

+11.5

+17.2

AgentSkillOS--19.5 Bench -20

+5.1

-10

0

10

20

Score change from Original (points) Figure 4 | Change from Original after Qwen3.5-9B–OpenCode SFT on unverified or SKT trajectories (four-run means). on either dataset. Unverified SFT lowers all four benchmark means, while SKT raises all four (Figure 4); the resulting gaps are 11.91–24.61 points. 4.4. Harness Transfer and Mixed-Harness Training A training trajectory captures not only the task solution and the use of external skills, but also the interaction protocol imposed by the agent harness. We therefore ask whether the learned skill-use behavior transfers across harnesses or remains specific to the environment in which the trajectories were collected. We study this question through cross-harness evaluation and mixed-harness training. All evaluations provide the task-designated skills. Cross-Harness Transfer We evaluate Qwen3.5-9B on SkillsBench and SkillEval under both OpenCode and DeepAgents. For each evaluation harness, Cross-harness SFT denotes a checkpoint trained on trajectories collected with the other harness, whereas Matched-harness SFT uses trajectories collected with the same harness used for evaluation. Original, Cross-harness SFT, and Matched-harness SFT are therefore evaluated on the same tasks, with the same designated skills and target-harness configuration; only the source of the SFT trajectories differs. Cross-harness SFT improves over Original in all four comparisons (Figure 5). On SkillsBench, transferring from DeepAgents training to OpenCode evaluation raises the score from 5.80 to 11.60, compared with 15.79 for Matched-harness SFT, retaining 58.1% of the matched gain. In the reverse direction, OpenCode training raises DeepAgents performance from 4.94 to 9.29, compared with 13.62 under matched training, corresponding to 50.1% retention. The same pattern holds on SkillEval: the transferred checkpoints retain 49.1% of the OpenCode matched gain and 52.1% of the DeepAgents matched gain. Across the four settings, Cross-harness SFT yields absolute improvements of 4.35–9.86 points

9

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Original

Cross-harness

SkillsBench OpenCode

Matched-harness

11.6 (58%)

5.8

15.8 9.3 (50%)

DeepAgents

4.9

13.6

0

5

10

15

SkillEval

20

63.7 (49%)

OpenCode

55.2

72.5 61.5 (52%)

DeepAgents

51.6

0

20

40

70.5

60

80

Score (0–100) Figure 5 | Qwen3.5-9B cross-harness transfer. Cross uses the other harness for SFT; Matched uses the evaluation harness. Labels report retained matched gain from four-run means. and retains 49.1%–58.1% of the corresponding Matched-harness SFT improvement. Its consistent advantage over Original shows that a substantial component of the learned skill-use behavior transfers across agent interfaces. At the same time, Matched-harness SFT remains stronger in every setting, indicating an additional benefit from aligning the training trajectories with the target execution environment. This combination of transferable and harness-aligned gains motivates training a single checkpoint on trajectories from both harnesses. Mixed-Harness Training The partial transfer observed above motivates a practical question: can a single checkpoint support both harnesses without maintaining a separate specialist for each one? We train Qwen3.5-9B on a mixture of verified OpenCode and DeepAgents trajectories, and evaluate the resulting Mixed checkpoint under both harnesses. Table 2 compares it with Original and the corresponding Specialist, which is trained only on trajectories from the evaluation harness. OpenCode

Benchmark SkillsBench MolBench-Bind SkillEval AgentSkillOS

DeepAgents

Original

Specialist

Mixed

Original

Specialist

Mixed

5.80 ± 1.03 33.11 ± 2.59 55.24 ± 0.72 79.61 ± 1.64

15.79 ± 1.69 44.59 ± 1.56 72.48 ± 1.32 84.70 ± 2.17

14.24 ± 0.64 45.27 ± 2.59 74.05 ± 1.38 82.25 ± 1.49

4.94 ± 0.70 31.76 ± 1.35 51.62 ± 1.16 74.03 ± 2.03

13.62 ± 0.82 47.30 ± 1.56 70.53 ± 1.24 79.84 ± 1.82

13.93 ± 1.15 44.59 ± 1.56 69.96 ± 1.62 77.40 ± 2.83

Table 2 | Mixed-harness training for Qwen3.5-9B. Original is the off-the-shelf checkpoint, Specialist uses trajectories from the evaluation harness, and Mixed uses trajectories from both harnesses. Scores report mean ± sample standard deviation over four complete runs; bold marks the highest mean within each evaluation-harness group.

10

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Original

SkillEval score (0–100)

75

SKT 70.8

70

72.5

67.4

65 59.8

60 55.2

55 0

100

500

1k

2k

Training-skill budget Figure 6 | SkillEval by training-skill budget for Qwen3.5-9B–OpenCode (four-run mean ± standard deviation; equal x spacing). The Mixed checkpoint improves over Original in all eight benchmark–harness comparisons, with gains ranging from 2.64 to 18.81 points. The largest improvements appear on SkillEval, where Mixed raises the OpenCode score from 55.24 to 74.05 and the DeepAgents score from 51.62 to 69.96. Substantial gains also hold on SkillsBench and MolBench-Bind under both harnesses. Mixed remains close to the matched Specialist throughout the table 2: their mean scores differ by at most 2.71 points. Mixed exceeds the Specialist in three comparisons—by 0.68 points on MolBench-Bind and 1.57 points on SkillEval under OpenCode, and by 0.31 points on SkillsBench under DeepAgents. In the remaining five comparisons, the gap is limited to 0.57–2.71 points. These results show that joint training largely recovers the benefit of harness-specific alignment while producing one checkpoint that can operate effectively with either harness. Together with the cross-harness results, this suggests that shared skill-use behavior and harness-specific interaction patterns can be learned within a unified model, reducing the need to train and maintain a separate checkpoint for each execution environment. 4.5. Scaling the Synthetic Training Pool We jointly scale the synthetic pool from 100 to 2,000 skills and its corresponding verified OpenCode trajectories, train Qwen3.5-9B under the same recipe, and evaluate on SkillEval with its designated skills. Original is the checkpoint at zero training-skill budget. SkillEval rises monotonically from 55.24 for Original to 72.48 at 2,000 training skills (Figure 6). 4.6. Performance by Skill Cardinality Finally, we examine whether SKT remains effective across different numbers of supplied skills. We partition the 77 SkillsBench tasks into three groups according to the number 𝐾 of task-designated skill artifacts: 𝐾 = 1 (𝑛 = 25), 𝐾 = 2 (𝑛 = 19), and 𝐾 ≥ 3 (𝑛 = 33). Both checkpoints use Qwen3.5-9B with OpenCode and receive exactly the same designated skills for each task.

11

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Original score

+7.4

9.54

K=1 (n=25)

K=2 (n=19)

3.95

K≥3 (n=33)

4.04

0

Gain from SKT

16.94

+16.8

+8.0

5

20.72

12.08

10

15

20

Mean normalized reward (0–100) Figure 7 | SkillsBench reward by skill count for Qwen3.5-9B–OpenCode. Gray is Original; blue is the SKT gain. Labels give task counts and final scores. As shown in Figure 7, Original obtains mean normalized rewards of 9.54, 3.95, and 4.04 for 𝐾 = 1, 𝐾 = 2, and 𝐾 ≥ 3, respectively. SKT increases these scores to 16.94, 20.72, and 12.08, corresponding to absolute gains of 7.40, 16.77, and 8.04 points. The improvement is present across all three skill-cardinality groups and is particularly pronounced for 𝐾 = 2. Taken together, this breakdown shows that the aggregate benefit of SKT is not concentrated in single-skill tasks. Substantial improvements also appear when multiple skills are supplied, including an 8.04-point gain for tasks with three or more skills. These results support the effectiveness of SKT in strengthening the model’s ability to leverage task-provided guidance across different skill-composition settings. 4.7. Summary of Findings Our experiments demonstrate that SFT on SKT trajectories improves model performance across diverse skill-use benchmarks. Instead of enabling the model to internalize skills, SKT enhances its capability to leverage externally supplied skills. Additionally, data ablation studies reveal that these performance gains hinge on training data quality. Directly employing raw synthetic training data without pipeline verification and repair can instead impair skill utilization. While the training data and corresponding performance improvements are harness-specific, the learned benefits generalize across different harnesses. Training on mixed harness data furthermore yields a single checkpoint compatible with multiple harnesses. We additionally observe that skill-use performance continues to improve as the number of training skills increases.

5. Conclusion We presented SKT, a three-stage pipeline for creating reliable skill-use tasks and producing verified execution trajectories for training LLMs to use Agent Skills. By combining skill selection, task synthesis with feedback-guided repair, and trajectory synthesis with skill-use verification, SKT produces executable examples spanning single- and multi-skill workflows. Across diverse models, agent harnesses, and application domains, SKT consistently improves skill-use performance. The learned behavior also 12

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

transfers across harnesses, while mixed-harness training enables a single checkpoint to perform effectively across different harnesses. Performance further improves as synthetic skill coverage expands. The results establish verified, composition-aware data synthesis as a practical route toward more reliable skill-using agents.

13

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

References Anomaly. OpenCode: The open source ai coding agent. Software repository, 2026. URL https: //github.com/anomalyco/opencode. Accessed: 2026-07-28. Anthropic. Claude Agent SDK. Software documentation, 2025. URL https://code.claude.com/docs/ en/agent-sdk/overview. Accessed: 2026-07-28. Zihao Cheng, Hongru Wang, Zeming Liu, Xinyi Wang, Xiangrong Zhu, Yuhang Guo, Wei Lin, Jeff Z. Pan, and Yunhong Wang. Terminal-World: Scaling terminal-agent environments via agent skills. arXiv preprint arXiv:2605.20876, 2026. DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. URL https://arxiv.org/abs/2606.19348. Zhiyuan Fan, Tinghao Yu, Yuanjun Cai, Jiangtao Guan, Yun Yang, Dingxin Hu, Jiang Zhou, Xing Wu, Zhuo Han, Feng Zhang, and Lilin Wang. Toward scalable terminal task synthesis via skill graphs. arXiv preprint arXiv:2604.25727, 2026. URL https://arxiv.org/abs/2604.25727. Gemma Team. Gemma 4 Technical Report. arXiv preprint arXiv:2607.02770, 2026. URL https: //arxiv.org/abs/2607.02770. GLM-5 Team. GLM-5: From vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. URL https://arxiv.org/abs/2602.15763. Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, and Lijie Hu. SWESkills-Bench: Do agent skills actually help in real-world software engineering? arXiv preprint arXiv:2603.15401, 2026. URL https://arxiv.org/abs/2603.15401. Zhongyu He, Yuanfan Li, Fei Huang, Tianyu Chen, Siyuan Chen, Xingyang Li, Meng Hsuan Yu, Xiangrong Liu, Leyi Wei, Lu Pan, Ke Zeng, and Xunliang Cai. SIRI: Self-internalizing reinforcement learning with intrinsic skills for LLM agent training. arXiv preprint arXiv:2606.02355, 2026. Zhanbo Hua, Yifan Yao, Weihao Xie, Yongchi Zhao, Minghao Liu, Ruizhi Qiu, Zhewei Huang, Zun Wang, Yiyan Ji, Yunhai Ye, Letian Zhu, Xinping Lei, Han Li, Zhiyuan Ma, Zili Wang, Zhaoxiang Zhang, and Jiaheng Liu. CLI-Universe: Towards verifiable task synthesis engine for terminal agents. arXiv preprint arXiv:2606.22883, 2026. URL https://arxiv.org/abs/2606.22883. LangChain. Deep Agents: The batteries-included agent harness. Software repository, 2025. URL https://github.com/langchain-ai/deepagents. Accessed: 2026-07-28. Hao Li, Chunjiang Mu, Jianhao Chen, Siyue Ren, Zhiyao Cui, Yiqun Zhang, Lei Bai, and Shuyue Hu. Organizing, orchestrating, and benchmarking agent skills at ecosystem scale. arXiv preprint arXiv:2603.02176, 2026a. URL https://arxiv.org/abs/2603.02176. Xiangyi Li, Yimin Liu, Wenbo Chen, et al. SkillsBench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026b. URL https://arxiv.org/abs/2602.12670. Hongxiang Lin, Zhirui Kuai, Erpeng Xue, and Lei Wang. SKILLC: Learning autonomous skill internalization in LLM agents via contrastive credit assignment. arXiv preprint arXiv:2605.27899, 2026. Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. SKILL0: In-context agentic reinforcement learning for skill internalization. arXiv preprint arXiv:2604.02268, 2026. 14

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

MiniMax. The MiniMax-M2 series: Mini activations unleashing max real-world intelligence. arXiv preprint arXiv:2605.26494, 2026. URL https://arxiv.org/abs/2605.26494. Andrew Qu. Introducing Skills, the open agent skills ecosystem. Vercel Changelog, January 2026. URL https://vercel.com/changelog/introducing-skills-the-open-agent-skills-ecosystem. Accessed: 2026-07-28. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/ blog?id=qwen3.5. Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao, Shugui Liu, Qi Gu, Xunliang Cai, Xiang Wang, and An Zhang. Skill1: Unified evolution of skill-augmented agents via reinforcement learning. arXiv preprint arXiv:2605.06130, 2026. Yash Vishe, Rohan Surana, Xunyi Jiang, Zihan Huang, Xintong Li, Nikki Lijing Kuang, Tong Yu, Ryan A. Rossi, Jingbo Shang, Julian McAuley, and Junda Wu. Skill-R1: Agent skill evolution via reinforcement learning. arXiv preprint arXiv:2605.09359, 2026. URL https://arxiv.org/abs/2605.09359. Jiongxiao Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, and Lin Lee Cheong. Reinforcement learning for self-improving agent with skill library. arXiv preprint arXiv:2512.17102, 2025. URL https://arxiv.org/abs/2512.17102. Yifan Wei, Li Du, Xiaoyan Yu, Yang Feng, and Angsheng Li. Towards compositional generalization of LLMs via skill taxonomy guided data synthesis. arXiv preprint arXiv:2601.03676, 2026. Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. SkillRL: Evolving agents via recursive skill-augmented reinforcement learning. arXiv preprint arXiv:2602.08234, 2026. Jingxu Xie, Dylan Xu, Xuandong Zhao, and Dawn Song. AgentSynth: Scalable task generation for generalist computer-use agents. In The Fourteenth International Conference on Learning Representations, 2026. URL https://iclr.cc/virtual/2026/poster/10010827. Aofan Yu, Chenyu Zhou, Tianyi Xu, Zihan Guo, Rong Shan, Zhihui Fu, Jun Wang, Weiwen Liu, Yong Yu, Weinan Zhang, and Jianghao Lin. LatentSkill: From in-context textual skills to in-weight latent skills for LLM agents. arXiv preprint arXiv:2606.06087, 2026. Barry Zhang, Keith Lazuka, and Mahesh Murag. Equipping agents for the real world with agent skills. Anthropic Engineering Blog, October 2025. URL https://www.anthropic.com/engineering/ equipping-agents-for-the-real-world-with-agent-skills. Lisheng Zhang, Lilong Wang, Xiangyu Sun, Wei Tang, Haoyang Su, Yuehui Qian, Qikui Yang, Qingsong Li, Zhenyu Tang, Haoran Sun, Yingnan Han, Yankai Jiang, Wenjie Lou, Bowen Zhou, Xiaosong Wang, Lei Bai, and Zhengwei Xie. MolClaw: An autonomous agent with hierarchical skills for drug molecule evaluation, screening, and optimization. arXiv preprint arXiv:2604.21937, 2026. URL https://arxiv.org/abs/2604.21937. Tianyi Zhang and Zhonghao Qi. Skill-to-LoRA: From using skills to learning behaviors for tokenefficient LLM agents. arXiv preprint arXiv:2606.16769, 2026. Xuan Zhao, Haonan He, Qingyu Yang, Minglei Li, Jingqi Ye, Zelin Tan, Bo Wan, and Peng Ye. Parametric skills. arXiv preprint arXiv:2606.30015, 2026. URL https://arxiv.org/abs/2606.30015.

15

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400–410, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. acl-demos.38. URL https://aclanthology.org/2024.acl-demos.38/. Jiapeng Zhu, Jianxiang Yu, Yibo Zhao, Chengcheng Han, Qi Gu, Xunliang Cai, Xiang Li, and Weining Qian. Skill0.5: Joint skill internalization and utilization for out-of-distribution generalization in agentic reinforcement learning. arXiv preprint arXiv:2605.28424, 2026a. URL https://arxiv.org/ abs/2605.28424. Kaijie Zhu, Yuzhou Nie, Yijiang Li, Yiming Huang, Jialian Wu, Jiang Liu, Ximeng Sun, Zhenfei Yin, Lun Wang, Zicheng Liu, Emad Barsoum, William Yang Wang, and Wenbo Guo. TermiGen: High-fidelity environment and robust trajectory synthesis for terminal agents. arXiv preprint arXiv:2602.07274, 2026b. URL https://arxiv.org/abs/2602.07274.

16

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

A. End-to-End SKT Procedure The main paper presents the three stages of SKT separately. Here we formalize their control flow, including candidate resampling, task repair, trajectory retry, and the conditions under which a candidate leaves the pipeline. The task and trajectory verifiers are treated as abstract operations; this section specifies when they are invoked and how their outputs affect the pipeline. A.1. Inputs and Outputs Let C be a collection of externally represented Agent Skills, K the requested skill cardinalities, and P the configured teacher–harness pairs. A skill configuration S ⊆ C contains 𝑘 distinct skills for some 𝑘 ∈ K. The pipeline first constructs a set of accepted task–configuration pairs Dtask , then collects their verified execution trajectories Dtraj . Each retained trajectory preserves its task, skill, teacher, and harness provenance. For each 𝑘, 𝑄 𝑘 is the target number of accepted tasks and 𝐵select,𝑘 bounds the number of candidate configurations examined. 𝐵task is the maximum number of repairs after initial task generation, and 𝐵traj is the maximum number of fresh rollouts for one task–teacher–harness pairing. A.2. Algorithm Algorithm 1 gives the complete control flow. QualifyTask invokes the rule-based, agent-based, and difficulty gates in that order and returns the first blocking diagnosis. The agent-based gate includes both semantic review and the paired skill-dependence test. QualifyTrajectory combines the deterministic outcome-and-trace check with the model-based skill-use check. A trajectory rejection never changes the task package, and its replacement is sampled without failure or verifier feedback. Appendix B expands the acceptance criteria used by these two operations. A.3. Feedback and Stopping Policy Algorithm 1 uses feedback only for task repair: the first blocking task diagnosis is returned to TaskRepair, whereas a rejected trajectory triggers a fresh retry without verifier feedback. The finite repair and rollout budgets define when each branch stops. Appendix B specifies the corresponding acceptance evidence and retention rules. A.4. High-Level Experimental Configuration Table 3 records the configuration needed to interpret Algorithm 1. Training, inference, hardware, serialization, and software details are reported separately with the experimental reproducibility settings.

B. Detailed Qualification Criteria Algorithm 1 leaves QualifyTask and QualifyTrajectory abstract. This section specifies their decision boundaries and recorded evidence. We report the verifier interfaces and pass conditions rather than reproducing prompt templates or implementation source code. The resulting specification is independent of a particular authoring model or agent harness.

17

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Algorithm 1 End-to-end construction of verified skill-use trajectories. Require: Skills C; cardinalities K; targets { 𝑄 𝑘 }; teacher–harness pairs P; budgets 𝐵select,𝑘 , 𝐵task , 𝐵traj ; difficulty parameters 𝑁, 𝜃easy Ensure: Tasks Dtask and trajectories Dtraj 1: U , Dtask , Dtraj ← ∅ 2: for all 𝑠 ∈ C do 3: if Suitable( 𝑠) then 4: U ← U ∪ {𝑠} 5: end if 6: end for 7: for all 𝑘 ∈ K do 8: 𝑛𝑘 , 𝑐 ← 0 9: while 𝑛𝑘 < 𝑄 𝑘 and 𝑐 < 𝐵select,𝑘 do 10: 𝑐 ← 𝑐 + 1; sample 𝑘 distinct skills S from U 11: if 𝑘 = 1 or Composable(S) then 12: T ← TaskGen(S) 13: for 𝑏 = 0, . . . , 𝐵task do 14: ( 𝑣, 𝑓 ) ← QualifyTask(T , S , 𝑁, 𝜃easy ) 15: if 𝑣 = pass then 16: Add (T , S) to Dtask ; 𝑛𝑘 ← 𝑛𝑘 + 1 17: break 18: else if 𝑏 < 𝐵task then 19: T ← TaskRepair(T , S; 𝑓 ) 20: end if 21: end for 22: end if 23: end while 24: end for 25: for all (T , S) ∈ Dtask do 26: for all ( 𝑚, ℎ) ∈ P do 27: for 𝑎 = 1, . . . , 𝐵traj do 28: 𝜏 ← Rollout(T , S , 𝑚, ℎ) 29: if QualifyTrajectory( 𝜏, T , S) then 30: Add (T , S , 𝑚, ℎ, 𝜏) to Dtraj ; break 31: end if 32: end for 33: end for 34: end for 35: return Dtask , Dtraj

B.1. Verifier Inputs and Hidden Assets For a task package T , let 𝑉 (T ) contain the solver-visible instruction, setup artifacts, execution settings, and output contract. The selected skills S are mounted separately. Hidden assets 𝐻 (T ) contain the evaluator specification, executable reference solution, and task-authoring metadata that records the intended role of each skill. A solver receives 𝑉 (T ) and, in the with-skill condition, S, but never receives 𝐻 (T ). Task verifiers may inspect both visible and hidden surfaces; trajectory verifiers additionally inspect the realized tool trace, output artifacts, and evaluator outcome. In the paired skill dependence test, only access to S changes between the two solver runs.

18

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Component

Experimental setting

Skill cardinalities Task-side model components Difficulty controller Difficulty parameters Teacher models

K = {1, 2, 3} DeepSeek V4 Pro with a Claude Agent harness Qwen3.5-35B-A3B with OpenCode 𝑁 = 5, 𝜃easy = 0.6 MiniMax-M2.5, GLM-5, Qwen3.5-397B-A17B, and DeepSeek V4 Pro DeepAgents and OpenCode

Trajectory harnesses

Table 3 | High-level configuration used for SKT data construction. Check

Pass condition

Rule-based: package and execution

The package is complete and parseable; paths are valid; evaluator definitions are supported; and the authoring trace records access to every skill in S. In a clean workspace initialized only from task inputs, the reference solution executes successfully and its produced artifacts receive full evaluator credit. Visible task materials do not copy reference values or required skill rules verbatim. The instruction clearly identifies the required artifact and output location; the local environment contains all information needed for completion; and solver-visible materials outside the supplied skills do not semantically reveal the hidden answer, evaluation logic, or decisive skill-derived requirements. Paired with-skill and without-skill rollouts hold the task, solver, harness, and all remaining execution settings fixed. Writing their evaluator scores as 𝑟 + and 𝑟 − , respectively, the scores must satisfy 𝑟 + > 𝑟 − , establishing that skill access benefits task completion at the bundle level. This paired test does not require independent leave-one-skill-out necessity for each member. With the selected skills available, a fixed solver and harness attempt the task for 𝑁 independent rollouts. If 𝑟 𝑗 is the evaluator score, the task passes only when Í 𝑝pass (T , S) = 𝑁 −1 𝑁𝑗=1 1[ 𝑟 𝑗 = 1] < 𝜃easy . We use 𝑁 = 5 and 𝜃easy = 0.6; hence three or more full-credit outcomes mark the task as too easy and return it for repair.

Agent-based: semantic validity

Agent-based: skill dependence

Difficulty control

Table 4 | Task-side qualification checks and pass conditions. B.2. Task-Side Qualification For the reported corpus of 4,000 accepted task packages, acceptance requires passing the same three top-level gates in sequence: rule-based verification, agent-based verification, and difficulty control. The agent-based gate comprises both semantic task review and the paired skill-dependence test. Table 4 expands the conditions used by QualifyTask; every listed row is mandatory. B.3. Feedback-Guided Repair Qualification stops at the first failing top-level gate. Its structured diagnosis contains concrete package errors, semantic feedback, or trajectory-grounded difficulty suggestions. TaskRepair revises the complete task package rather than patching a solver trajectory. The revised package then restarts qualification at the rule-based gate, because any change can invalidate earlier evidence. A task enters Dtask only after one package version passes all three gates; a candidate that exhausts its repair budget is discarded.

19

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

B.4. Trajectory-Side Qualification Trajectory qualification consists of a deterministic validator followed by an LLM validator. The deterministic validator uses harness-appropriate terminal signals but applies the same four requirements in both harnesses: full evaluator credit, normal and complete termination, a well-formed tool trace, and explicit access to the required skills. An execution failure, incomplete termination, malformed trace, or non-full evaluator outcome is blocking. The LLM validator examines the complete realized trace and checks every designated skill individually. Each skill must be consulted before the action it should guide, affect a concrete decision or operation, and be applied correctly. Evidence may include a skill-derived convention, formula, API pattern, threshold, ordering constraint, resource, verification step, or repair that changes the produced artifact or workflow. Merely opening or paraphrasing a skill file, citing it after the relevant action, or completing the task entirely through generic reasoning is insufficient. For a multi-skill task, all designated skills must make distinct substantive contributions within one coherent workflow. A trajectory passes only if both validators pass. B.5. Retry, Retention, and Recorded Evidence A rejected rollout is discarded. The next attempt for the same task–teacher–harness pairing begins in a fresh session and workspace and does not receive the previous trace, evaluator outcome, failure reason, or verifier feedback. The first rollout that passes both validators is retained; if the rollout budget is exhausted, that pairing contributes no training example. The retained example contains the uninterrupted solver interaction, while hidden references, evaluator internals, and verifier analyses remain outside the solver trace. For each task attempt, the verification record stores the gate verdict, a concise failure category, and the associated task and skill identifiers. For each trajectory attempt, it stores the teacher and harness identifiers, evaluator outcome, terminal status, validator verdicts, and the skill-use evidence used for the decision. These fields support aggregate auditing of the reported criteria without exposing hidden task assets, verifier prompts, or implementation source code. Token-length filtering, serialization, and loss masking are training-data conversion decisions and are specified separately in the SFT reproducibility section.

C. Dataset Composition and Trajectory Statistics This appendix reports aggregate properties of the dataset used in the main experiments. The counts follow the same accepted task pool as the main experimental setup: 4,000 task packages built from 2,000 skills selected from skills.sh and paired with four teachers and two execution harnesses. C.1. Task and Skill Composition Table 5 breaks down the accepted task pool by skill cardinality. Each accepted task is eligible for eight trajectory-collection pairings, one for each teacher–harness combination, yielding 32,000 candidate pairings before trajectory-side verification.

20

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Functional family

Scope

Representative skill families

Software engineering and quality

Developing, testing, debugging, reviewing, and maintaining software.

Infrastructure, systems, and security

Deploying, operating, monitoring, and securing systems and services.

Data, ML, and scientific computing Documents, content, and media

Transforming structured data, building learned systems, and performing quantitative or scientific computation. Creating, transforming, inspecting, or validating human-facing artifacts.

Business, finance, and operations

Supporting organizational decisions and recurring professional workflows.

Research, communication, and productivity

Gathering, synthesizing, communicating, and organizing information.

Agent tooling and workflow automation

Building tool-using agents and automating interactions with external services.

Code review; test repair; API and framework implementation; Git and repository workflows. CI/CD; containers and Kubernetes; cloud and observability; vulnerability assessment. ETL and SQL; forecasting and statistics; computer vision; scientific analysis. PDFs, spreadsheets, and presentations; technical writing; UI design; image, audio, and video processing. Market and financial analysis; CRM and sales; product analysis and planning; project and people operations. Literature and citation workflows; deep research; email, calendar, and knowledge organization. MCP and tool construction; prompt, context, and memory workflows; multi-agent and browser automation.

Table 6 | An overlapping functional taxonomy of the selected 2,000-skill pool. The examples are illustrative rather than exhaustive labels; skills and tasks are not forced into mutually exclusive categories.

𝑘

Tasks Skill refs. Pairings

𝑘 = 1 1,520 𝑘 = 2 1,295 𝑘 = 3 1,185

1,520 2,590 3,555

12,160 10,360 9,480

Total 4,000

7,665

32,000

Table 5 | Accepted task composition by skill cardinality. The Skill refs. column counts designated task–skill associations. Table 6 gives an illustrative, overlapping organization of functions represented in the selected pool, based on package names and descriptions and, where available, optional metadata. For example, generating a statistical report can combine data analysis with document creation. Because these families overlap, we keep them separate from the compositionality statistics above. The 𝑘 = 1, 2, 3 tiers count designated skill artifacts, not functional domains. A multi-skill task may combine skills within one family or across several families. C.2. SkillEval Benchmark Composition We construct SkillEval with the same task-synthesis pipeline used for the training task pool, but from a separately generated candidate pool. SkillEval is reserved for evaluation: neither its 100 task packages nor trajectories collected by executing them are included in the SFT data. The resulting separation is therefore at the task-package and trajectory levels.

21

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

Task pool Tasks Training SkillEval

𝑘=1

𝑘=2

𝑘=3

4,000 1,520 1,295 1,185 100 30 46 24

Table 7 | Skill-cardinality composition of the training task pool and SkillEval. Harness DeepAgents OpenCode

𝑁

Assistant turns

Tool calls

Skill-load requests

14,277 12,887

12 / 17 9 / 13

18 / 26 11 / 16

5/9 3/7

Table 8 | Descriptive statistics of the collection-level retained trajectories. Interaction-count entries report median / 90th percentile. Table 7 compares the skill-cardinality composition of the two task pools. Each of SkillEval’s 100 tasks is associated with one designated skill group. These groups are distinct from one another within SkillEval and are drawn from a held-out skill pool disjoint from the training skill pool. Of the 100 tasks, 30 are single-skill, 46 are two-skill, and 24 are three-skill. The benchmark spans several functional areas, including software development and debugging, data analysis and machine learning, security, and finance and business analysis. During evaluation, each task is executed in a fresh isolated session and workspace with its designated external skills available, except in the explicit no-skill controls. We report mean normalized reward over all 100 tasks. For each model–harness condition, we run the complete benchmark four times, following the protocol in the main experimental setup. C.3. Trajectory Measurements We summarize the retained verified trajectory pools at the collection level. The 𝑁 values therefore match the trajectory counts in the main experimental setup rather than denoting a per-backbone count after target-specific rendering. Table 8 reports the median and 90th percentile for each interactioncount measure. We count each assistant-role message as one assistant turn and each function invocation as one tool call; parallel calls count separately. Skill-load requests are native skill calls for OpenCode and read_file calls targeting /skills/.../SKILL.md for DeepAgents. Repeated requests and zeros remain in the distributions. These syntactic counts do not themselves establish successful skill loading, substantive skill use, or trajectory acceptance.

D. Training and Evaluation Details We report data-conversion, optimization, and evaluation details for Qwen3.5-9B and Gemma 4 E4B-IT. The tokenizer and chat template are model-specific, while the tool representation remains harness-native. D.1. Trajectory Serialization and Supervision Each retained rollout is serialized end to end while preserving its harness’s native tool schema and action structure. Post-termination acknowledgements and logging events are excluded, and the two 22

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

harnesses are not mapped to a common tool vocabulary. After rendering with the target tokenizer and chat template, we supervise model-authored reasoning, tool calls, and responses, while masking system and task messages and tool observations as in the main SFT objective. We use neither history masking nor packing and discard, rather than truncate, records longer than 64k tokens. D.2. Optimization and Checkpoint Selection We perform full-parameter SFT with LLaMA-Factory and DeepSpeed ZeRO-3, using the settings in Table 9. Records are shuffled with a fixed seed; runs neither resume from optimizer state nor use validation or benchmark scores for selection. Evaluation loads the final checkpoint after one epoch. Setting

Value

Epochs Optimizer Learning rate Global batch

1 Fused AdamW; 𝛽1 = 0.9, 𝛽2 = 0.999, 𝜖 = 10−8 , weight decay = 0 5 × 10−6 ; cosine decay with a 3% warmup ratio 8 (8 workers × 1 example per worker × 1 gradient-accumulation step) bfloat16; gradient checkpointing; ZeRO-3; label-only fused cross-entropy Global norm 1.0 No packing or token-level truncation; a whole-record limit of 64k tokens Final one-epoch checkpoint; no validation-based selection

Precision and memory Gradient clipping Sequence handling Checkpoint rule

Table 9 | Common SFT implementation settings. Figure 8 reports the corresponding one-epoch optimization traces for Qwen3.5-9B. The per-step losses are noisy, while the EMA-smoothed curves decrease across the DeepAgents, OpenCode, and balanced mixed-harness runs. The environment uses Transformers 5.6.0, PyTorch 2.10.0 with CUDA 12.8, Datasets 4.0.0, and Tokenizers 0.22.2; eight GPU workers implement the reported effective batch size. Per-step loss

EMA-smoothed loss

DeepAgents

Training loss

OpenCode

14,277 trajectories 1,785 steps

0.35

Mixed

12,887 trajectories 1,611 steps

24,000 trajectories 3,000 steps

0.30 0.25 0.20 0.15 0.10 0

500

1000

Training step

1500

0

500

1000

Training step

1500

0

1000

2000

3000

Training step

Figure 8 | Qwen3.5-9B training-loss curves for the DeepAgents, OpenCode, and mixed-harness SFT runs. Thin lines show per-step loss and thick lines show the exponential moving average. Labels give the number of training trajectories and optimizer steps in each run.

23

SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation

D.3. Inference and Evaluation Protocol Models are served through OpenAI-compatible endpoints and invoked through harness-native loops. DeepAgents reads materialized /skills/.../SKILL.md files, whereas OpenCode uses its native skill tool. Formal evaluation uses temperature 0.7; other decoding settings are fixed, while timeout watchdogs and interaction limits remain benchmark- and harness-specific. Original and SFT receive identical task assets and designated skills except in explicit no-skill controls. Cross-harness evaluation changes the agent loop and skill interface, not the checkpoint. Every task starts in a fresh isolated session and workspace with no carried context or state. We use each benchmark’s native evaluator and the 0–100 metrics defined in the main setup. For each model–harness condition, four complete runs yield four aggregate scores; we report their arithmetic mean and sample standard deviation with 𝑛 − 1, rather than computing dispersion over task-level scores. D.4. Mixed-Harness and Scaling Variants For mixed-harness training, we seed-sample 12,000 trajectories from each harness’s final pool, yielding a balanced set of 24,000 verified records. We preserve each harness’s native message and tool-call representation and inject no harness identifier into the training messages. The checkpoint otherwise uses the common SFT settings in Table 9. For scaling, we train Qwen3.5-9B with the common recipe on verified OpenCode trajectories from 100, 500, 1,000, or 2,000 skills, then evaluate on SkillEval with designated skills. Zero budget denotes the off-the-shelf Original, not a separate SFT run.

24

Related documents

• Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining#635875
• Controlling memorability of face images with generative models.#220900
• Scaling Synthetic-Image Pre-Training for Federated Fine-Tuning of Large Vision Models#366225
• SkillOS: Learning Skill Curation for Self-Evolving Agents#168351
• Self-Trained Verification for Training- and Test-Time Self-Improvement#238603