SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

softwarearchitecture softwareengineering testing

software engineering, software architecture, testing

SkillJuror

SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior Zhiyu Chen1,2,† , Zihan Guo2,3,† , Bo Huang2,4 , Bingwei Lu4 , Jianghao Lin4∗ , Yuanjian Zhou2∗ , Weinan Zhang2,4∗ Tongji University 2 Shanghai Innovation Institute Sun Yat-sen University 4 Shanghai Jiao Tong University † Equal contribution. ∗ Corresponding author. 1

arXiv:2606.11543v1 [cs.AI] 10 Jun 2026

[email protected], [email protected], [email protected]

Abstract Agent Skills augment large language model (LLM) agents with procedural knowledge at inference time, but current benchmarks rarely distinguish what a Skill says from how it is organized. We study this distinction through Progressive Disclosure, where a concise root file points agents to supporting resources on demand, and compare it with a normalized flat baseline. We present SkillJuror, a framework for evaluating Skill writing paradigms through semantically controlled variants, matched multi-trial evaluations, and trajectory evidence while holding task knowledge fixed. In an 82-task SkillsBench study, Progressive Disclosure changes runtime behavior before aggregate outcomes: distinct Skill resources touched per trajectory rise from 1.18 to 3.85, and effective uptake events rise from 1.33 to 3.92. It also yields 17 additional verifier-passing trials out of 410 matched trials (+4.1%) over the normalized flat baseline. The benefit is task-dependent. Progressive Disclosure helps when supporting resources guide implementation, checking, or repair, but is weaker when success hinges on exact output conventions, numerical thresholds, or long artifact-generation pipelines. These results show that Skill organization is not mere presentation: it can change how agents search and apply procedural knowledge, while outcome gains depend on whether the exposed resources are actionable for the task. Code is available at https://github.com/zhiyuchen-ai/skill-juror. Keywords: Agent Skills, Skill Evaluation, Runtime Analysis, Harness Engineering

Introduction

Agent Skills are becoming a practical mechanism for giving LLM agents task-specific procedural knowledge at inference time. A Skill is a self-contained navigable runtime artifact rather than a static prompt. An agent reads a root SKILL.md, opens supporting references, inspects templates, and may invoke helper scripts while working on a task, as described by the specification1 . Since agents autonomously choose which resources to open and which helpers to invoke, Skill organization governs runtime trajectories instead of simply formatting documentation. Once Agent Skills are adopted, an immediate practical question arises: how should they be written? Published style guides advocate paradigms like progressive disclosure, modular organization, and scriptization (Anthropic, 2025). While these recommendations are intuitive, rigorous empirical validation remains elusive. Existing evidence mostly focuses on adoption, comparing whether Skills are present, absent, or self-generated, rather than how the same knowledge is organized within a Skill (Li et al., 2026). Furthermore, natural Skill collections inevitably confound core knowledge 1 https://agentskills.io/specification.

SkillJuror

Figure 1: From entangled Skill comparisons to controlled runtime evidence. Natural Skill comparisons can vary task knowledge, artifact organization, and runtime behavior simultaneously, making outcome differences hard to attribute. SkillJuror instead constructs knowledge-matched variants that vary organization under matched runtime conditions, then pairs verifier outcomes with process evidence for cost, trajectory shape, and resource uptake.

coverage, individual author styles, and structural organization. Consequently, a better-performing Skill may owe its advantage to superior organization, to richer task knowledge, or to both. This attribution gap makes authoring recommendations difficult to evaluate without controlled, task-knowledge-matched variants. Addressing this challenge is critical because a knowledge-agnostic organization paradigm, if effective across diverse domains, would suggest that Skill layout can systematically reshape agent runtime behavior independently of task-specific content coverage. This would make organization itself a controllable mechanism for guiding how agents retrieve, apply, and reuse procedural knowledge during execution. Isolating organization from knowledge allows us to treat structural layout as an experimentally isolated runtime intervention, establishing a scalable foundation for controlled Skill evaluation. This paper isolates Skill organization as a controlled intervention at runtime. Our focus is Progressive Disclosure, the practice of keeping the root file concise and pointing agents to supporting resources on demand. Holding procedural knowledge constant, this contrast asks whether reorganizing a flat, self-contained instruction into an on-demand resource bundle changes how the agent behaves. Progressive Disclosure is therefore a behavioral intervention whose outcome effects may vary by task. A shorter root may reduce cognitive load at entry and encourage targeted reference access, while its benefits depend on whether agents access and apply the supporting resources that contain task-critical knowledge. We introduce SkillJuror, an evaluation framework that isolates the behavioral and outcome effects of Skill organization by factoring out interfering variables like knowledge coverage and author style. To disentangle organization from task knowledge, SkillJuror constructs controlled, semantically matched Skill variants from a shared source while preserving identical knowledge such as operational obligations, constraints, and helpers. By executing these knowledge-matched pairs under identical, reproducible runtime environments, the framework correlates macro-level pass/fail outcomes with granular trajectory evidence, such as resource access breadth and Effective Resource Uptake (ERU). This dual-layered measurement allows us to systematically audit whether and how a structural layout alteration modifies an agent’s runtime reasoning and execution path. This work makes three contributions:

SkillJuror

1. The first controlled evaluation of Agent Skill organization. We are, to our knowledge, the first to treat Skill organization as an experimentally isolated runtime variable, comparing task-knowledge-matched variants rather than naturally different Skills to decouple content from structure. 2. SkillJuror: a framework for controlled Skill-variant evaluation. We introduce an audited construction and matched-runtime evaluation pipeline that preserves task knowledge while varying organization, links outcome differences to trajectory-level evidence, and enables controlled comparisons of Skill-writing paradigms. 3. Empirical evidence that organization changes behavior before outcomes. We conduct an 82-task Progressive Disclosure study and find large shifts in resource access and effective uptake (from 1.33 to 3.92 events per trajectory). While these behavioral changes lead to a modest aggregate pass-rate gain (+4.1%), the resulting outcome effects remain highly taskdependent.

Related Work

2.1

Agent Skills as Runtime Procedural Artifacts

Agent Skills encapsulate reusable procedural knowledge as agent-facing artifacts rather than as oneoff prompts. The Agent Skills specification defines a Skill as a directory centered on a SKILL.md file, with optional scripts, references, assets, and routing metadata. Anthropic’s guidance further frames Skills around progressive disclosure: metadata supports selection, the root file provides the entry point, and supporting resources are loaded on demand (Anthropic, 2025). Recent proceduralmemory work similarly treats skills as runtime units with activation and execution conditions, as opposed to passively retrieved text (Mi et al., 2026). This view makes Skill authoring an organization problem. The same task knowledge can be exposed as a flat instruction bundle or as a concise root with supporting files. Existing specifications describe such choices as best practices, but they do not empirically test whether organization itself changes downstream behavior under controlled task semantics. 2.2

From Skill Adoption Benchmarks to Controlled Skill-Variant Comparison

The most closely related work evaluates whether providing Skills helps agents solve tasks. SkillsBench (Li et al., 2026) compares no-Skill, curated-Skill, and self-generated-Skill conditions over verifiable tasks, showing that curated Skills can improve pass rate but that effects are heterogeneous and self-generated Skills can be ineffective or even harmful. SWE-Skills-Bench extends this adoption question to repository-based software engineering tasks (Han et al., 2026), while work on larger Skill libraries shows that failures may arise from trajectory-level Skill-selection errors, not just context overhead (Song et al., 2026). Other benchmarks study how Skills are learned, generated, or verified. SkillLearnBench evaluates continual skill-learning methods (Zhong et al., 2026); SkillGenBench evaluates pipelines that distill executable Skills from repositories or documents (Zhou et al., 2026b); and SkillGen synthesizes auditable Skills from successful and failed trajectories (Ma et al., 2026). These efforts broaden the source of Skills, but their unit of analysis remains adoption, acquisition, or generation quality. SkillJuror instead compares variants of the same task Skill while holding task scope, helper availability, workflow obligations, and output contracts fixed, shifting the evaluation grain from Skill availability to controlled Skill-variant comparison. 2.3

Trajectory- and Resource-Aware Agent Evaluation

Agent evaluation has increasingly moved beyond final task success. Outcome-centric benchmarks and surveys provide scalable pass-rate or reward signals, but they often hide why an agent succeeded 3

SkillJuror

or failed (Luo et al., 2025). Trajectory-aware benchmarks address this gap by evaluating intermediate behavior, including tool-use diagnostics, reasoning trajectories, and step-level process quality (He et al., 2025; Kim et al., 2025; Fan et al., 2026). Cost-aware evaluation further argues that success should be interpreted together with resource consumption (Erol et al., 2025). To properly isolate these behavioral factors, contemporary harness engineering serves as a unifying runtime layer to eliminate uncontrolled environment variation (Zhou et al., 2026a), allowing SkillJuror to hold the execution substrate strictly invariant for controlled attribution. Within this controlled framework, SkillJuror uses trajectory evidence to complement verifier outcomes by showing whether changing Skill layout changes runtime resource access and uptake. To evaluate nuanced resource-use behaviors within these trajectories, SkillJuror adopts an LLM-as-ajudge approach (Gu et al., 2024; Shi et al., 2025). This choice is motivated by the need for semantic judgment when capturing effective resource uptake, allowing us to leverage LLM-assisted labels as valuable process evidence that programmatic metrics alone would miss. 2.4

Constrained Optimization of Agent-Facing Artifacts

A related line of work automatically improves agent-facing artifacts. Prompt-sensitivity work shows that even meaning-preserving formatting choices can change model behavior (Sclar et al., 2024), while prompt optimization searches over instructions, examples, or LM-program components to maximize validation performance (Zhou et al., 2022; Yang et al., 2024). Skill-level optimization similarly edits Skill documents using rollout feedback and held-out validation (Yang et al., 2026). Tool-interface and procedural-memory systems improve how agents call APIs or accumulate reusable behavior (Schick et al., 2023; Song et al., 2023; Shinn et al., 2023; Zhao et al., 2024; Wang et al., 2023). These methods search for better artifacts or accumulate new behavior from experience. SkillJuror uses transformation differently by constructing controlled counterfactuals for evaluation. This transformation is designed to isolate whether the same task knowledge behaves differently when organized under a specified writing paradigm. In this paradigm-constrained setting, runtime differences can be interpreted as effects of organization rather than as unconstrained artifact optimization.

Method

3.1

Evaluation Problem

SkillJuror evaluates whether changing a Skill’s organization changes agent behavior while holding task knowledge fixed. Let T denote the task set, where each task T consists of an instruction, an initial environment, and a verifier. For each task T and runtime condition c, BT,c denotes the Skill bundle exposed to the agent. A bundle may contain metadata, a root SKILL.md, executable helpers, and non-executable support material. The controlled comparison uses only bundles that pass the same construction protocol and are accepted as preserving task semantics. In the main run, the primary writing-paradigm contrast is the normalized flat baseline (Baseline) versus the Progressive Disclosure (PD) variant. Each trial rT,c,j is an independent execution of task T under condition c, and repeated trials form RT,c . For each evaluation dimension d, such as outcome, efficiency, paradigm realization, or routing quality, SkillJuror computes a task-level summary Ad (T, c) instead of collapsing all evidence into one score. The main within-task Progressive Disclosure contrast is ∆d (T ) = Ad (T, PD) − Ad (T, Base), with benchmark- and group-level results aggregating these task-level comparisons. 3.1.1

Overall Pipeline

SkillJuror follows the three-stage pipeline in Figure 2. Taking a source Skill bundle and a target writing paradigm as input, it rewrites the Skill under a knowledge-preservation constraint so that

SkillJuror

accepted variants differ primarily in organization rather than in knowledge content. It then executes2 each task–condition pair in repeated, matched trials and maps the resulting artifacts into dimensionspecific summaries for outcome, efficiency, paradigm realization, and routing quality. Controlled Variant Construction

Task

Skill Bundle

Runtime Evidence Collection

Multi-Dimensional Evaluation Analysis Dimensions

Inputs

Evaluation Configuration Writing Paradigm Outcome

Paradigm-Constrained Rewrite

Paradigm

Routing

Harbor Framework Task-level Summaries

revise

Validation Gates

Validated Skill Bundles

Efficiency

Within-Task Contrasts

Trial Set and Artifacts Paradigm Effect

Runtime Mechanism

Translation Patterns

Figure 2: Overview of the SkillJuror method pipeline. Tasks, source Skill bundles, and target writing paradigms are transformed into accepted controlled Skill variants under knowledge-preservation checks. Accepted variants are executed under fixed runtime settings and mapped into four analysis dimensions: outcome, efficiency, paradigm realization, and routing quality. Failed construction candidates loop back for revision before entering runtime evaluation.

3.2

Controlled Skill Variant Comparison

Directly evaluating raw Skill collections inevitably confounds their structural organization with varying author styles, content coverage, and implicit helper implementations. To cleanly isolate organization as an independent runtime intervention, SkillJuror transforms source artifacts into a semantically matched pair: a flat Baseline and a structurally reorganized Progressive Disclosure (PD) variant, as exemplified in Figure 3 to offer an intuitive understanding of their concrete archetypes. Following established industrial Agent Skill guidance from Anthropic (Anthropic, 2025), the PD paradigm minimizes the agent’s immediate cognitive load by serving a lightweight root document (SKILL.md) that routes to partitioned supporting resources only on demand. Conversely, the Baseline flattens this identical knowledge base into a single, self-contained file. To strengthen semantic control, all generated candidate pairs undergo a rigorous two-layer verification pipeline before entering the evaluation engine: 1. Structural Adherence: The PD variant must feature a concise root SKILL.md, maintain physically decoupled supporting directories, and provide explicit navigational hyperlinks within SKILL.md. 2. Knowledge Preservation: Both variants must hold the underlying task scope, workflow obligations, hardware/software dependencies, exact numeric thresholds, and verifier contracts invariant relative to the source. This pipeline ensures that any downstream deviations in agent trajectories or pass rates are strictly attributable to structural layout rather than content drift. 3.2.1

Skill-for-Skill Transformation Pipeline

We implement construction through Skill-for-Skill transformations. Specifically, we employ an LLM agent equipped with a dedicated transformation Skill to ingest the source artifact and produce a new 2 Harbor provides the containerized sandbox for both Skill artifact generation and task execution.

SkillJuror

Original source bundle

Baseline bundle

Progressive Disclosure bundle

Source bundle

Controlled variant

Skill root S

Single skill root

threat-detection/ MD

SKILL.md root 1.3k

normalize structure

dapt-intrusion-detection/

SKILL.md root 7.5k reorganize structure

SKILL.md

support content

pcap-analysis/ MD

SKILL.md root 3.9k

support files PY

SKILL.md root 0.9k support files 7.0k

references/ 2 md files

Skill root S

Single skill root, modular resources

dapt-intrusion-detection/ S

pcap_utils.py support 2.3k

Inline code

Detailed instrucrtions

network-analysis.md 3.5k threat-detection.md 1.3k

pcap-analysis/ PY pcap_utils.py 2.3k

source guidance folded into single SKILL.md

Retrieving resources is guided by SKILL.md.

2 skill roots, 3 files, 7.5k tokens

1 skill root, 1 file, 7.5k tokens

1 skill root, 4 files, 7.9k tokens

Figure 3: Example controlled Skill-bundle transformation. For illustration, we select a SkillsBench source bundle with multiple Skill roots and show how it is converted into the Baseline artifact, then reorganized into a Progressive Disclosure bundle with a concise entry point and on-demand support files. The example is drawn from dapt-intrusion-detection; token counts are approximate tokenizer counts over file contents and are shown only to indicate relative size.

Skill artifact under the target organization profile. These transformations are executed as Harborbacked construction runs, so the Baseline and PD artifacts are produced inside the same sandboxed task materialization and verifier boundary used by the later runtime study. This keeps construction scalable while avoiding manual rewriting and host-specific file-system assumptions. For the Baseline artifact, the transform flattens the source Skill bundle into a controlled comparison anchor while preserving commands, helper contracts, script contents, schemas, warnings, thresholds, output formats, and workflow obligations. For the Progressive Disclosure variant, the transform starts from the Baseline artifact and reorganizes the same content into a short root SKILL.md and supporting files. In the notation above, construction is a two-step task-local transformation: fpd

fbase

BTsrc −−−→ BT,Base −−→ BT,PD . The second step is interpreted as a constrained reorganization of the Baseline artifact, preserving the same task scope, helper contracts, constraints, and output requirements. Constructing PD from the Baseline artifact is deliberate. If Baseline and PD were generated independently from the original source, differences in wording, coverage, or helper recovery could make it difficult to attribute observed differences specifically to organization. By treating PD as a constrained reorganization of the Baseline artifact, we reduce the opportunity for uncontrolled content changes. The PD transform may move detailed material into references or support files while preserving functional domains, concrete commands, helper implementations, task scope, and task-critical constraints. 3.2.2

Construction Validation Mechanism

To increase confidence that the constructed variants provide rigorous semantic control, SkillJuror implements a multi-layered verification framework rather than relying on a single, unconstrained judge model. By coupling rigid programmatic execution with flexible semantic auditing, the pipeline offsets the blind spots of code-based rules with the contextual nuance of model evaluation. This integrated pipeline sequentially enforces structural, semantic, and programmatic invariance through three progressive validation tiers: 6

SkillJuror

• Deterministic Gating: Fast, programmatic filters validate low-level artifact integrity, checking mechanical properties such as directory layout, file-path hygiene, resource reachability, and behavior-unit diffs. • Rubric-Based Semantic Auditing: An automated semantic assessment scans the remaining candidates against localized rubrics to ensure the invariant preservation of task scopes, execution constraints, and input/output contracts. • Human-in-the-Loop Override Policy: Initial validation anomalies or flags are routed to an evidence-backed review protocol. These are systematically classified and resolved as either source-inherited idiosyncrasies, packaging artifacts, or handled via controlled source-to-target repairs. A final pre-runtime eligibility filter drops semantic or scale outliers to insulate the downstream benchmarking from execution confounding. The resulting verified artifact pairs establish a clean, standardized foundation where downstream performance variances can be confidently attributed to structural layout rather than construction drift. 3.3

Runtime Evidence Collection

The runtime layer converts accepted Skill variants into matched execution traces. For each task, SkillJuror rebuilds the same non-Skill execution substrate – instruction, workspace, environment, and verifier – in a sandboxed execution task, then swaps only the exposed Skill artifact across conditions. This standardizes the non-Skill environment and keeps the comparison focused on whether the agent receives no task Skill, Baseline, or the Progressive Disclosure variant. Each task–condition pair is executed with the same agent harness, model family, reasoning configuration, verifier, and timeout budget. We use multiple independent trials because agent execution is stochastic even when the task and Skill are fixed, and each trial starts from the same materialized task state and becomes one observation in RT,c = {rT,c,j }nj=1 . Main-run task counts, condition counts, and trial counts are specified in the experimental setup. Every trial preserves the trajectory and verifier-side execution record needed for replay, audit, and aggregation. Given these comparable artifacts, the evaluation layer defines outcome, efficiency, paradigm-realization, and resource-routing measurements. 3.4

Multi-Dimensional Skill Evaluation

The evaluation layer maps each repeated trial set RT,c into task-level summaries Ad (T, c) along four dimensions: outcome utility, efficiency trade-off, paradigm realization, and resource-routing quality. The aggregation rule is tied to the evidence type: verifier outcomes summarize task completion, resource measures can be viewed per attempt or per strict pass3 , and trajectory evidence summarizes process behavior. Outcome Utility. Outcome utility measures whether the agent completes the task. We keep runtime completion separate from verifier success so that execution failures are not conflated with incorrect solutions. Efficiency Trade-off. Efficiency measures the resource burden of attempting or completing a task. The per-trial view captures deployment cost across all attempts, while the per-pass view captures the yield-normalized cost of obtaining strict passes. 3 Throughout the main text, “pass” denotes a strict verifier pass, counted only when the verifier reward equals 1. Partial rewards are retained only as diagnostics in Appendix C; broad-acceptance diagnostics are reported in Appendix E, Table 16.

SkillJuror

Paradigm Realization. Paradigm realization asks whether the intended Skill organization is visible in runtime behavior. It uses trajectory attribution to test how much, when, and in what pattern the agent engages with Skill material. Resource Routing Quality. Resource Routing Quality asks whether Skill-resource access becomes useful local work. The main metric is Effective Resource Uptake (ERU), which counts a resource or helper signal only when it is consumed into observable implementation, validation, correction, or credible blocker diagnosis. ERU labels are assigned by an LLM-as-judge audit over extracted trajectory events and interpreted as bounded process evidence. Appendix D reports the event categories, boundary rules, and aggregate count summaries used to interpret this metric. 3.5

Task Attribute Schema

Task-level contrasts are expected to be heterogeneous: the same resource exposure can help one task, remain locally useful but outcome-irrelevant in another, or add work in a long artifact pipeline. To support task-dependent analysis, SkillJuror attaches a compact attribute schema to each task. The schema combines source-provided contextual labels, when available, with mechanism-facing labels introduced by the analysis. Contextual labels preserve broad task provenance, while workflow type and validation type describe what the agent must operationally do and how success is accepted. These labels give the later analyses a common vocabulary for grouping task-level contrasts. They are used to examine when additional resource uptake is paired with outcome or efficiency gains, and when it remains local process behavior that does not close the verifier contract.

Experiments

We organize the experiments around four questions that move from input control to outcomes, execution processes, and task-dependent heterogeneity: RQ1(Construction Reliability): Can we reliably construct and audit semantically controlled Skill variants that instantiate different writing paradigms? RQ2(Outcome Effects): How do different Skill paradigms affect task success and efficiency? RQ3(Process Effects): How do different Skill paradigms affect agents’ execution processes and resource-use behavior? RQ4(Task-Dependent Effects): How do task properties influence the outcome and process effects of different Skill paradigms? Together, these questions test whether multi-dimensional evaluation reveals Skill-paradigm effects that pass/fail evaluation alone would miss. 4.1

Experimental Setup

The primary run reported in the paper instantiates the notation above with a main task set Tmain of 82 SkillsBench tasks after applying pre-runtime eligibility filters for external dependencies, size-tail artifacts, and environment comparability. The main condition set is {∅, Base, PD}, corresponding to no Skill, Baseline, and Progressive Disclosure. We run n = 5 trials for every task–condition pair, yielding |Tmain |n = 410 trials per condition for the main RQ2/RQ3 estimates. Full task accounting, exclusion rationale, and the layout-sensitivity supplement are reported in Appendices A and E. Tokens/pass reports non-cached input, cache-creation input, and output tokens, excluding cachedread tokens from the display total. Cost/pass uses the same yield-normalized denominator as minutes/pass, after separating non-cached input, cache-creation input, cached-read input, and output tokens under the GPT-5.4 standard-rate assumption. Appendix C gives the pass criterion and computation protocol. 8

SkillJuror

Table 1: Runtime setup for the main comparison. Setting

Main-text value

Harness / backend Runtime model Primary trial grid Primary conditions Secondary layout checks

Codex runtime with Harbor-backed sandbox materialization. GPT-5.4, high reasoning, fixed across conditions. 82 tasks × 3 primary conditions × 5 trials = 1,230 trials. No Skill, Baseline, Progressive Disclosure. Origin on the 82-task set; Origin-flat on the 56-task multi-Skill subset.

4.2

Construction Reliability (RQ1)

RQ1 asks whether SkillJuror can build two Skill versions that keep the same task meaning while differing in organization. This construction-reliability step establishes that RQ2–RQ4 compare organizational differences rather than drift in task scope, helper availability, command semantics or output contracts. The construction pipeline completed both Baseline and Progressive Disclosure artifacts for all 88 construction-eligible tasks. Table 2 reports the three validation tiers introduced in Section 3. Deterministic gating verified that the materialized artifacts satisfied style, path, and behavior-unit checks before runtime evaluation. Rubric-based semantic auditing then checked the accepted variants against preservation and Progressive Disclosure criteria across all 968 rubric items under a policy-aware Harbor GPT-5.4 audit, yielding 3/968 negative rubric checks. Finally, the human-inthe-loop override policy adjudicated those three cases, leaving no unresolved construction-validity issue. Appendix B reports the rubric criteria and gate outcomes. Table 2: Main construction-reliability summary for the controlled Skill artifacts. The table follows the three validation tiers defined in Section 3; detailed gate outcomes are reported in Appendix B.

Gate

Result

Deterministic Gating Rubric-Based Semantic Auditing

88 tasks validated; semantic content preserved 176 variants evaluated; 965/968 items passed automated review 3 items required manual review; all accepted under documented policy

Human-in-the-Loop Adjudication

The artifacts can therefore be used as controlled inputs for runtime evaluation: the accepted variants preserve task requirements while differing in organization. RQ2–RQ4 then test whether that organizational difference changes outcomes, execution behavior, and task-dependent effects. RQ answer. The controlled artifacts have passed the construction-reliability protocol. Both variants were generated for all 88 eligible tasks, all deterministic gates were passed, and semanticaudit findings were resolved through evidence-based review.

4.3

Outcome Effects (RQ2)

RQ2 asks whether controlled Skill paradigms change task success and efficiency. We use verifier pass rate as the primary outcome measure. Aggregate pass rate is the fraction of trials in a condition that receive a strict verifier pass, and the headline controlled effect is the Progressive Disclosure pass rate minus the Baseline pass rate. Efficiency has several useful denominators. The headline columns in Table 3 use a yield-normalized denominator: total time, display tokens, or estimated cost across all attempts divided by the number of strict passes. This metric estimates the resource burden of obtaining one strict pass under repeated attempts, so it includes failed trials in the numerator. It is thus distinct from both the mean resource 9

SkillJuror

use per attempted trial and the mean resource use among successful trials only; Table 4 reports all three views for time and estimated dollar cost. Table 3: Pass outcomes and yield-normalized per-pass efficiency for the main RQ2 comparison. Bold condition names report the primary 82-task conditions; bold numeric entries mark the best primarycondition value in the column, with higher better for pass/rate/∆ and lower better for resource columns. Italic rows show secondary Origin and Origin-flat layout checks under the same verifier-pass criterion. ∆ is relative to the comparable Baseline where applicable. Min/pass, Tokens/pass, and Cost/pass divide total resource use across all attempts by strict passes. Token/cost details and layout accounting appear in Appendices C and E.

Condition

Estimate set

Pass

Rate

∆ Min/pass Tokens/pass Cost/pass

No Skill 82 tasks 119/410 29.0% -53 Baseline 82 tasks 172/410 42.0% – Progressive Disclosure 82 tasks 189/410 46.1% +17 Secondary Origin and Origin-flat layout checks

29.8 20.1 17.8

0.34M 0.22M 0.21M

$2.05 $1.28 $1.31

82 tasks 56 tasks

17.8 20.1

0.31M 0.37M

$1.68 $1.92

Origin Origin-flat

172/410 42.0% 112/280 40.0%

0 –

Table 4: Three efficiency views for the primary 82-task RQ2 conditions. “Attempt” averages over all trials; “success path” averages only strict-passing trials; “yield” divides total resource use across all trials by strict passes, matching the Min/pass and Cost/pass columns in Table 3. Bold numeric entries mark the lowest value in each column. Condition No Skill Baseline Progressive Disclosure

Attempt Success Yield Attempt Success Yield min/trial path min min/pass cost/trial path cost cost/pass 8.64 8.42 8.19

6.80 7.40 6.30

29.76 20.07 17.76

$0.60 $0.54 $0.61

$0.53 $0.50 $0.50

$2.05 $1.28 $1.31

Table 3 supports three outcome-efficiency patterns. First, both Skill conditions improve strict pass rate relative to No Skill: Baseline reaches 42.0%, and Progressive Disclosure reaches 46.1%, compared with 29.0% for No Skill. The controlled PD–Baseline gap is 17 additional strict passes out of 410 matched trials (+4.1%), with a paired task-level 95% CI half-width of ±6.0%, so the aggregate gain is positive but modest. Second, the efficiency profile shows a trade-off, not uniform dominance. Progressive Disclosure lowers yield-normalized wall-clock burden from 20.1 to 17.8 minutes per strict pass, but cost/pass remains nearly tied with Baseline ($1.31 vs. $1.28) partially because higher pass yield offsets higher per-attempt cost. Third, the secondary Origin and Origin-flat rows contextualize the source-layout explanation: they match the comparable Baseline pass count while requiring more displayed tokens and estimated cost per pass. Full layout accounting appears in Appendix E. The aggregate effect is positive but heterogeneous. As Figure 4 shows, Progressive Disclosure improves over Baseline on 23 tasks, ties on 44 tasks, and underperforms on 15 tasks under the pass metric. Most nonzero differences are one or two trials, but both positive and negative tails are present. This pronounced heterogeneity underscores a key insight: looking only at final pass rates gives an incomplete picture. Because the aggregate outcome gain is modest and highly task-dependent, binary success metrics fail to capture how organization actually shapes agent reasoning across different tasks. This layout change operates primarily as a behavioral intervention, altering how an agent searches for and applies knowledge throughout its execution trace, before it ever hits a verifier. This disconnect motivates the process analysis in RQ3 and the task-type analysis in RQ4, where we look 10

SkillJuror

baseline higher

PD higher

Tasks

40 30 20 11

10 1

-4

-3

-2

-1

PD - baseline strict passes (of 5 trials)

Figure 4: Per-task pass-count deltas for Progressive Disclosure relative to Baseline. Counts cover 82 tasks with five trials per condition. Positive and negative bars show task-level heterogeneity around the aggregate gain.

beyond final endpoints to audit how structural layout reshapes runtime trajectories and resource uptake. RQ answer. Progressive Disclosure yields a modest aggregate pass-rate gain and faster successful completions with only a marginal cost increase, though the task-level effect remains heterogeneous.

4.4

Process Effects (RQ3)

RQ3 examines whether altering a Skill’s structure modifies an agent’s navigation and consumption of procedural knowledge. While pass rates capture final outcomes, the behavioral impact of the layout intervention is most transparent here, demonstrating that Progressive Disclosure fundamentally reshapes agent trajectories regardless of final task success. We operationalize this behavioral audit through two lenses: macro trajectory properties (Paradigm Realization) and granular resource routing (Resource Routing Quality). 4.4.1

Paradigm Realization

Table 5 shows that the layout intervention is visible in runtime behavior. Progressive Disclosure roughly doubles Skill-step share (5.4% to 10.8%) and increases mean resource fanout from 1.18 to 3.85 resources per trial, while support-file and helper evidence rise sharply. These measures capture routing and access behavior. The ERU analysis below examines how often the additional resource interactions are taken up in concrete local work. Table 5: RQ3 Paradigm Realization signals. Signals come from deterministic trajectory attribution for the main Baseline–Progressive Disclosure comparison. “Fanout” is the mean number of distinct Skill resources touched per trial. These signals describe runtime behavior changes; ERU below examines how often accessed resources are taken up in concrete work. Condition Baseline Progressive Disclosure

Skill step Skill token Support read share share 5.4% 10.8%

9.0% 11.5%

Script

Helper Fanout

66/410 31/410 3/410 364/410 101/410 118/410

1.18 3.85

The aggregate table hides an important temporal difference. Figure 5 plots where Skill-related steps occur along normalized trajectory position. Baseline is concentrated near the beginning of the trajectory, consistent with early Skill intake followed by general work. Progressive Disclosure 11

SkillJuror

remains early-heavy, but it spreads Skill use farther into the middle and late phases. The split across trajectory thirds makes this shift explicit: Condition

Early

Middle

Late

Baseline Progressive Disclosure

69.5% 59.4%

21.9% 25.1%

8.5% 15.5%

Baseline

0.50 0.25

Progressive Disclosure 0

100

0.00

Normalized trajectory position (%)

Skill-related steps per trial

The distributed-skill-trial rate also rises from 20.7% to 48.4%. Progressive Disclosure changes the work loop. Agents do not only read a larger instruction once, but return to support resources while implementing, checking, and repairing.

Figure 5: Skill-step timing in the Baseline–Progressive Disclosure comparison. Columns are 5% bins of normalized trajectory position. Color encodes absolute Skill-related steps per trial. The main visible shift is that PD carries more Skill use into middle and late trajectory phases while also increasing the absolute amount of Skill use within each phase.

4.4.2

Resource Routing Quality

The Paradigm Realization layer shows that Progressive Disclosure routes agents into more Skill resources, while ERU asks whether those additional accesses become usable work. An agent can scan directories, probe wrong paths, read generic background material, or invoke a helper without using the resulting signal. We therefore use ERU as a stricter Resource Routing Quality measure over extracted resource events from each trial. A resource event contributes to ERU only when the agent consumes the resource content, helper output, or diagnostic signal and turns it into observable local progress, validation, correction, or credible blocker diagnosis. Table 6 reports the ERU summary over 410 trajectories per condition. The headline pattern is exposure plus uptake intensity: Progressive Disclosure increases extracted resource events from 717 to 1,902, and mean ERU-positive uptake events per trajectory from 1.33 to 3.92. The main RQ3 signal is that agents open more files and that more of those resource signals become observable implementation, validation, correction, or diagnosis steps. Table 6: RQ3 Resource Routing Quality evidence. Counts cover the full Baseline–Progressive Disclosure comparison. “Events” counts extracted Skill-resource events; “Mean yes” counts only ERU-positive events consumed into local implementation, validation, correction, or credible diagnosis. Condition Baseline Progressive Disclosure

ERU rate Events Mean events Mean yes /traj. /traj. 76.0% 84.6%

717 1902

1.75 4.64

≥2 yes traj.

≥5 yes traj.

1.33 95/410 11/410 3.92 335/410 118/410

Task-level ERU rates remain heterogeneous, but the aggregate pattern shows that Progressive Disclosure increases resource interaction, later revisitation, and observable local uptake. The RQ3 finding reveals that the PD layout drives agents to open more support resources, revisit them later, and turn more of those signals into concrete local work. Crucially, this is where the layout intervention is most visible in the entire framework: even when final outcome translation is incomplete or blocked by task-specific constraints, the PD paradigm is behaviorally realized, thereby changing the 12

SkillJuror

agent’s runtime reasoning and execution path. This metric is therefore best read as process evidence for how the layout changes resource use. RQ answer. Progressive Disclosure is reflected in runtime behavior, driving agents to access more distinct files, execute downstream revisitations, and more frequently translate those interactions into ERU-positive local uptake.

4.5

Task-Dependent Effects (RQ4)

RQ4 asks when the resource-use changes observed in RQ3 translate into the outcome and efficiency changes observed in RQ2. This task-level heterogeneity underscores that pass-rate metrics alone do not capture the subtle dynamics of the layout intervention, necessitating a more granular analytical lens. To isolate when and how the expanded resource routing effectively translates into macro outcome gains, we employ task attributes as analytical moderators through a dual-coordinate schema, as illustrated in Figure 6. While retaining source-provided domain labels for broad context, we introduce two mechanismfacing analysis labels to systematically audit the alignment between process indicators and verifier contracts: workflow type, which captures the operational nature of the agent’s work, and validation type, which defines the strictness of the evaluation boundary. This integrated stratification framework permits a granular look at the experimental data based on how success is operationally achieved and accepted, thereby revealing distinct empirical translation archetypes. Mechanism-facing refinement Domain 11 SkillsBench-inherited labels

SkillsBench Tasks

Software Eng.

Office & White Collar

Natural Science

Media & Content

Workflow Type

Validation Type

what the agent must do

how success is accepted

code repair / software

exact / structural

data / doc / office Cybersecurity

Finance

scientific computation

Robotics

Mathematics

media processing

Energy

Manufacturing

optimization / control other engineering

Healthcare

security

exact artifact / file numeric tolerance code / security tests performance threshold

Figure 6: Task-type labeling schema for the SkillsBench instantiation. Domain labels are inherited from the source benchmark, while workflow type and validation type are added as mechanism-facing labels for interpreting task-dependent process–outcome patterns.

4.5.1

Type-Level Translation Patterns

Figure 7 visualizes the localized interaction between process intensity (the change in ERU-positive events) and final outcome variance across our stratified coordinates, with the highly interpretable regions clustered into four empirical translation archetypes in Table 7. We treat these recurring profiles as conditional diagnostics reflecting the common structural pathways through which enhanced resource routing is either processed into verifier-visible action, absorbed as redundant overhead, or decoupled from final validation requirements. Appendix E reports the selective support strata, weak-signal rationale, and compact bridge traces behind these descriptive cases. The resulting distribution demonstrates that aggregate pass-rate variance is merely a surface-level reflection of deeper task-type constraints, which Table 7 groups into distinct empirical profiles based on how local behavior aligns with the verifier contract. As illustrated, code and security frequently exhibit a favorable alignment where elevated process uptake directly supports executable, verifier13

SkillJuror

(a) By validation type

(b) By workflow type

Code repair

Code/security

Strict-pass Δ (pp)

More uptake higher pass

Security

Performance threshold

Optimization/control Exact/structured Data/document

Other engineering

Artifact/file Numeric tolerance More uptake lower pass

Media

Science

−15 0

Δ ERU-positive uptake / trajectory Δpass > 0

Δpass = 0

Δpass < 0

n=8

n=15

n=30

Figure 7: Process–outcome alignment across selected task strata. Points are validation-type and workflow-type strata from the RQ4 support analysis. The x-axis shows the PD-minus-Baseline change in ERU-positive events per trajectory, and the y-axis shows the strict-pass-rate delta. Color marks the passdelta sign, and marker area marks the number of tasks. Table 7: Selected RQ4 task-type empirical translation patterns. Deltas are Progressive Disclosure minus Baseline. The table foregrounds interpretable, high-signal diagnostic trends rather than invariant causal groupings; detailed deltas and support strata are reported in Appendix E. Translation pattern

Where observed

Main signal

Diagnostic mechanism reading

Code/security tests; code repair/software; security Uptake without success Numeric tolerance; scientific computation; Natural Science Fanout tax Media processing; exact artifact/file validation

Pass/runtime ↑; min/token per pass ↓; ERU ↑

Routed resources become executable, verifier-visible code and test actions.

ERU ↑; pass/runtime ↓

Completion with risk

Runtime completion ↑; pass given runtime success ↓

Resources support local work, but numeric and output contracts remain binding. Resources expand brittle exact-artifact pipelines instead of stabilizing the final output. Resources help reach executable completion while threshold, schema, or oracle alignment remains separate.

Targeted efficiency

Performance threshold; optimization/control

Fanout/helper use ↑; artifact success ↔ or ↓

visible actions. Conversely, media-processing and exact-artifact domains tend to manifest a behavioral decoupling; the layout successfully prompts local resource interaction, but these expanded pathways often introduce a fanout tax without stabilizing brittle serialization requirements. Similarly, in numeric, scientific, and optimization tasks, an organizational paradigm can be locally actionable and assist the agent along a completion path, yet final success remains independently governed by precise numerical tolerances or strict oracle alignment. These stratified trends highlight that enhanced resource routing alters runtime execution pathways across the board, but its final outcome utility remains conditional on the nature of the task’s evaluation boundary. 4.5.2

Case Studies

We next examine representative trajectories to interpret the operational mechanics behind these empirical profiles. Each case traces a specific resource signal from local behavior to intermediate action, illustrating how that execution path interacts with the verifier contract. These qualitative traces offer a descriptive diagnostic for why certain structural layouts alter outcomes, complementing the macro tendencies observed in the aggregate statistics. Targeted efficiency gain. In simpo-code-reproduction, the PD layout routes the agent from the concise root skill into the task-specific references/nlp-reproduction.md file. The trajectory 14

SkillJuror

translates this reference into an executable feedback loop: the agent aligns the environment, inspects the SimPO trainer, and executes unit_test/unit_test_1.py with PYTHONPATH=/root/SimPO while monitoring /root/loss.npz. Because the test suite provides immediate feedback, the agent efficiently operationalizes the localized instructions, converting a 0/5 Baseline pass rate into a 5/5 strict pass under PD while reducing average latency. Uptake without success. In exoplanet-detection-period, the PD variant exhibits high process engagement with supporting material, but this intense local uptake fails to translate into the verifier’s numeric contract. Trajectories show effective consumption of references/preprocessing.md, references/box-least-squares.md, and references/lomb-scargle.md to construct a multistage period search. However, strict passes drop from 5/5 to 3/5. Logs reveal a process-outcome decoupling: the final output file contains a logical period (2.79991), but it is rejected because the verifier strictly demands an oracle value of 5.35699 within a 0.01 tolerance. This diagnoses a persistent vulnerability where layouts facilitate local procedural reasoning yet remain decoupled from unexposed validation boundaries. Fanout tax. In video-tutorial-indexer, modular transcription helpers appear locally useful, but the decentralized layout inadvertently expands the agent’s interaction surface within a fragile media pipeline. The agent reads references/transcription.md and speech-to-text/scripts/transcribe.py, but becomes entangled in repairing helper paths and debugging slow automated speech recognition (ASR) runs. While runtime completion holds at 5/5, strict passes fall from 5/5 to 2/5 as execution steps and tokens rise substantially. This illustrates a compounding fanout overhead. When the final contract requires exact structural serialization, dispersing instructions into on-demand files can introduce path-brittleness rather than stabilizing the generated output. Completion-path gain with alignment risk. In manufacturing-equipment-maintenance, the PD layout lowers the entry-level cognitive load to establish a runnable compliance workflow where Baseline uniformly fails. The concise root directs the agent to references/compliance-computation.md, which successfully guides the implementation of ramp computations, TAL interpolations, and thermocouple tie-breaking. Although runtime completion rises from 0/5 to 5/5 and strict passes recover to 3/5, failure logs expose the remaining boundary clearly: one run outputs a rounded value of 1.91 where the oracle mandates 1.9, and another violates the specific ordering for runner_up_run_ids. This profile demonstrates that a paradigm can optimize workflow execution paths, yet final success remains independently gated by rigid output schemas. These case studies provide descriptive reference for interpreting how task attributes shape the translation of local behavior into verifier-visible outcomes. The granular trajectory evidence suggests that the final efficacy of a Skill writing paradigm remains substantially shaped by the structural properties and feedback density of the task’s evaluation boundary, highlighting the qualitative variance that macro pass rates hide. RQ answer. Task properties govern the alignment between process and outcome patterns. In contexts with actionable, well-defined targets, increased local uptake frequently manifests as passrate gains, whereas tasks bound by rigid numeric conventions or exact output schemas are closely correlated with a process–outcome disconnect.

Discussion

The evaluation of SkillJuror demonstrates that Skill organization operates as an active runtime intervention rather than passive documentation. By holding task knowledge constant, the shift to Progressive Disclosure (PD) may alter the agent’s trajectory from a single-shot “read-and-execute” workflow to an iterative “implement-verify-repair” loop. The concentrated late-stage resource inter15

SkillJuror

actions indicate that agents treat modular, offloaded references as on-demand assets for debugging and error recovery rather than upfront context. This behavioral shift provides direct engineering implications for both Skill authors and harness designers. First, modularity must prioritize actionable routing over aesthetic brevity, treating Skill loading and layout as dynamic runtime control surfaces rather than static formatting. The root SKILL.md should decouple high-level strategy from low-level details, functioning strictly as a lean entry point while offloading granular constraints, templates, and specifications into supporting files. Specifying exactly when and why to access a file anchors the agent’s tool-use loop, prompting resource invocation precisely when execution friction occurs. Practically, the observed heterogeneity suggests that PD is not a universal solution, requiring deployment systems to match layout paradigms to the task’s validation strictness. Designers should consider PD in exploration-heavy domains (e.g., code repair, system diagnostics) where tasks involve fluid environments and success depends on surviving runtime errors through late-stage troubleshooting. Conversely, flat layouts may be preferable for constraint-bound domains (e.g., scientific computation, strict schema serialization) where success hinges on rigid numeric thresholds or exact formatting. In these fixed-boundary tasks, distributing constraints across files adds a fanout tax and risks omission as keeping information centralized may reduce the risk that critical limits are missed.

Limitations

This study isolates one writing-paradigm contrast: Progressive Disclosure versus Baseline. SkillJuror is designed to evaluate broader Skill-writing interventions. The empirical scope of this paper is the controlled comparison reported here, rather than scriptization, metadata design, multi-Skill modularization, or other authoring paradigms. The controlled variants also provide auditable semantic control as opposed to a formal proof of equivalence. Deterministic gates, behavior-unit diffs, rubric review, and an independent GPT-5.4 construction audit reduce construction drift, while subtle differences in wording, emphasis, or recoverability of task constraints may remain. The process analyses have their own boundary. ERU and bridge traces rely on LLM-assisted semantic judgment and are best interpreted as bounded mechanism evidence rather than human-groundtruth annotation or causal mediation estimates. Generalization is also limited by the evaluation substrate: the main run uses SkillsBench tasks, Harbor-backed sandbox execution, one primary agent/model configuration, and five trials per task–condition pair. Marketplace Skills, organizationinternal workflows, live APIs, human-evaluated tasks, and different agent platforms may exhibit different reuse patterns, failure modes, and sensitivity to Skill organization.

Conclusion

SkillJuror reframes Agent Skill evaluation by shifting the focus from skill availability alone to controlled structural comparison. Our study shows that how procedural knowledge is organized substantially reshapes an agent’s runtime trajectory, even when task semantics are held fixed under the construction audit. Effective skill writing is therefore not merely about brevity, but about actionable routing—transforming passive instructions into conditional guidance that actively supports execution, validation, and error repair. Ultimately, evaluating agents primarily through final pass/fail outcomes can obscure the mechanics of their success. Future benchmarks should pair verifier outcomes with granular trajectory evidence, because structural layout reshapes how an agent searches and applies knowledge before those effects appear in aggregate outcome metrics.

References Anthropic. Equipping agents for the real world with agent skills, October 2025. Anthropic Engineering Blog.

SkillJuror

Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, and James Zou. Cost-of-pass: An economic framework for evaluating language models. arXiv preprint arXiv:2504.13359, 2025. Shengda Fan, Xuyan Ye, Yupeng Huo, Zhi-Yuan Chen, Yiju Guo, Shenzhi Yang, Wenkai Yang, Shuqi Ye, Jingwen Chen, Haotian Chen, et al. Agentprocessbench: Diagnosing step-level process quality in tool-using agents. arXiv preprint arXiv:2603.14465, 2026. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. The Innovation, 2024. Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, and Lijie Hu. Swe-skills-bench: Do agent skills actually help in real-world software engineering? arXiv preprint arXiv:2603.15401, 2026. Pengfei He, Zhenwei Dai, Bing He, Hui Liu, Xianfeng Tang, Hanqing Lu, Juanhui Li, Jiayuan Ding, Subhabrata Mukherjee, Suhang Wang, et al. Traject-bench: A trajectory-aware benchmark for evaluating agentic tool use. arXiv preprint arXiv:2510.04550, 2025. Wonjoong Kim, Sangwu Park, Yeonjun In, Sein Kim, Dongha Lee, and Chanyoung Park. Beyond the final answer: Evaluating the reasoning trajectories of tool-augmented agents. arXiv preprint arXiv:2510.02837, 2025. Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026. Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460, 2025. Yuchen Ma, Yue Huang, Han Bao, Haomin Zhuang, Swadheen Shukla, Michel Galley, Xiangliang Zhang, and Stefan Feuerriegel. Skillgen: Verified inference-time agent skill synthesis. arXiv preprint arXiv:2605.10999, 2026. Qirui Mi, Zhijian Ma, Mengyue Yang, Haoxuan Li, Yisen Wang, Haifeng Zhang, and Jun Wang. Skill-pro: Learning reusable skills from experience via non-parametric ppo for llm agents. arXiv preprint arXiv:2602.01869, 2026. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551, 2023. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In International Conference on Learning Representations, volume 2024, pp. 25055–25083, 2024. Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic study of position bias in llm-as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pp. 292–314, 2025. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023. Hongwen Song et al. More skills, worse agents? skill shadowing degrades performance when expanding skill libraries. arXiv preprint arXiv:2605.24050, 2026.

SkillJuror

Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, et al. Restgpt: Connecting large language models with real-world restful apis. arXiv preprint arXiv:2306.06624, 2023. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, volume 2024, pp. 12028–12068, 2024. Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, et al. Skillopt: Executive strategy for self-evolving agent skills. arXiv preprint arXiv:2605.23904, 2026. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19632–19642, 2024. Shanshan Zhong, Yi Lu, Jingjie Ning, Yibing Wan, Lihan Feng, Yuyi Ao, Leonardo FR Ribeiro, Markus Dreyer, Sean Ammirati, and Chenyan Xiong. Skilllearnbench: Benchmarking continual learning methods for agent skill generation on real-world tasks. arXiv preprint arXiv:2604.20087, 2026. Chenyu Zhou, Huacan Chai, Wenteng Chen, Zihan Guo, Rong Shan, Yuanyi Song, Tianyi Xu, Yingxuan Yang, Aofan Yu, Weiming Zhang, et al. Externalization in llm agents: A unified review of memory, skills, protocols and harness engineering. arXiv preprint arXiv:2604.08224, 2026a. Yifan Zhou, Zhentao Zhang, Ziming Cheng, Shuo Zhang, Qizhen Lan, Zhangquan Chen, Zhi Yang, Ronghao Chen, Huacan Wang, Sen Hu, et al. Skillgenbench: Benchmarking skill generation pipelines for llm agents. arXiv preprint arXiv:2605.18693, 2026b. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The eleventh international conference on learning representations, 2022.

SkillJuror

Appendix A

Task Set and Exclusions

Table 8 records the task-count relationships used in the experiments. The construction protocol succeeded on 88 construction-eligible tasks. The main runtime comparison uses 82 of these tasks and excludes six tasks before runtime comparison because their artifacts or environments would make the controlled estimate unreliable. Table 8: Task-set accounting for the controlled runtime study. Set

Count Role

Construction-eligible tasks Main runtime tasks Main trials Layout-sensitivity tasks

88 Source tasks for Baseline and Progressive Disclosure construction. 82 Main RQ2/RQ3 task set after pre-run exclusions. 1,230 82 tasks × 3 conditions × 5 trials. 56 Aligned multi-skill supplement for Origin/Origin-flat; not merged into the 82-task main estimate.

Table 9: Main-run exclusions from the 88 construction-eligible tasks. Excluded task

SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior

Related documents