ConceptioArchivearXiv CS
arXiv CSopen access

Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows

Unknown · 2026 · arxiv_cs
arXiv CS · Papers · License: Open Access · 2026
Open Source ↗Direct PDF ↓
artificialintelligenceknowledgerepresentationreasoning
artificial intelligence, reasoning, knowledge representation

Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows Wei-Jung Huang Independent Researcher United States

arXiv:2607.07504v1 [cs.AI] 8 Jul 2026

Abstract Product data scientists often ask LLM-based agents to help with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results. Reusable skill files are meant to avoid prompting from scratch by packaging guidance for a task family. Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck. We ask whether LLMgenerated skills offer a useful low-curation alternative: do they improve performance over the task prompt alone? We test this question across four lifecycle stages: data preparation, data extraction, statistical analysis, and reporting, using one generated skill per stage. We find no reliable improvement from full generated skills over No-Skill prompting. We then ask whether any part of the skill is useful by ablating different skill components. The main ablation covers 56 tasks, nine model configurations, and three providers, yielding 7,560 runs. Compared with prompting using the task alone, neither the full generated skill nor any ablated skill variant significantly improves performance; all 𝑝-values are at least 0.396, and the total spread across variants is only 1.2 pp. A supplemental token-matched control adds 1,512 runs and finds that Full skills perform similarly to task-irrelevant skill-formatted content. The results caution against using one LLM-generated skill per data-science workflow as a default single-shot prompting strategy.

CCS Concepts • Computing methodologies → Artificial intelligence.

Keywords large language models, data-science agents, agent skills, prompt engineering, ablation study, data-science automation

1

Introduction

Product data scientists often ask LLM-based agents to help with recurring execution tasks: cleaning data, writing SQL, choosing statistical tests, computing effect sizes, and formatting reports. We refer to these systems as data-science agents, following recent benchmarks for data-science automation [5, 15]. For these systems, model choice is only part of the design problem. The other part is how to supply domain knowledge. Recent agent platforms and benchmarks use reusable skill files, such as SKILL.md, to package task instructions, examples, and reference notes [1, 8]. For data-science agents, this makes a tempting workflow: write one skill for recurring task families such as data preparation, SQL generation, statistical analysis, and reporting, then prepend it to future tasks.

Expert-written skills can encode high-quality guidance, but they require practitioners to decide what to include, write examples and reference notes, and keep the content current as tools and task conventions change. This manual process scales poorly when teams need guidance for many data-science task families. LLMgenerated skills offer a lower-curation alternative: generate taskfamily guidance once and reuse it across related tasks. The question is whether this low-curation version works in practice. Existing evidence suggests that this is not guaranteed. SkillsBench [8] (86 tasks, 7,308 trajectories) found that human-curated skills improve performance substantially (+16.2 pp), while LLMgenerated skills provide no aggregate benefit. However, SkillsBench does not specifically evaluate reusable skills for data-science workflows. That null result also raises a component-level question: when an LLM-generated skill fails to improve performance, is every component unhelpful, or are useful sections canceled out by harmful ones? For builders of data-science agents, it is useful to ask both whether LLM-generated skills help on data-science workflows and which failure patterns appear when they do not. We study these questions across four execution-facing datascience lifecycle stages. Each stage is represented by a task family covering a distinct workflow: data preparation (null handling, deduplication, type coercion), data extraction (SQL query formulation), statistical analysis (hypothesis testing, effect sizes), and reporting (structured JSON reporting). These workflows represent common execution tasks for data-science agents, and their outputs can be checked by deterministic verifiers. Our evaluation has two parts. First, we compare task-only prompting with one full LLM-generated skill per stage. Second, we ablate the same skills to test whether any section helps on its own. We use a simplified single-shot setup so that the comparison focuses on the skill content itself. Section 3 describes this design choice. We make three contributions: • We evaluate low-curation LLM-generated skills for datascience agents across four lifecycle stages, 56 tasks, nine model configurations, and 7,560 main-ablation runs. • We ablate procedures, examples, and reference notes, then add a 1,512-run token-matched control to separate datascience content from prompt-length overhead. • We observe no reliable improvement from either full skills or ablated variants, and analyze failure patterns that differ across task regimes.

2

Related Work

Our study sits at the intersection of prompt-level knowledge injection, reusable skills, and LLM benchmarks for data-science tasks.

AIDataSci ’26, 2026, Jeju, Korea

The SKILL.md specification is one recent form of reusable domain instruction for LLM prompts [1]. Related ways of supplying task context include retrieval-augmented generation [6] and fewshot prompting [2]. Prior work also shows that prompt components can affect model behavior in non-obvious ways: Min et al. [10] found that label correctness in demonstrations matters less than format and input distribution, and Lu et al. [9] showed that example ordering can significantly affect performance. We move from individual prompt elements to reusable skill files and ask whether their content helps when injected into data-science tasks. Reusable skills have also been evaluated directly. SkillsBench [8] found that focused human-curated skills improve performance, while LLM-generated skills provide no aggregate benefit. SkillLearnBench [16] studies continual skill-learning methods and finds gains over no-skill baselines, but no method dominates across tasks and models. SkVM [3] treats skills as compiled capabilities across heterogeneous LLM backends. These studies motivate testing reusable skills, but they do not ask which parts of generated skills help or hurt in data-science workflows. On the benchmark side, DS-1000 [5], DataSciBench [15], and LLM4DS [11] evaluate LLM code generation for data-analysis tasks and show that strong models can solve many data-science tasks from the task prompt alone. That creates a hard test for reusable skills: if task-only prompting already covers many common datascience procedures, skills have less room to help and may still add cost or conflict with task-specific instructions. These benchmarks evaluate data-science capability, but they do not test whether reusable skills improve performance in this domain. Recent data-preparation and table-reasoning systems also point to a different form of support than static front-loaded instructions. PrepBench evaluates natural-language-driven data preparation and emphasizes ambiguous intents, imperfect real-world data, interactive disambiguation, code generation, and workflow translation [13]. AutoDCWorkflow generates data-cleaning workflows from a raw table and analysis purpose, then evaluates answer, data, and workflow quality [7]. Chain-of-Query uses multi-agent collaboration and clause-by-clause SQL generation for table understanding [12]. These systems do not evaluate reusable skill files directly, but they suggest one limitation of flat prompt injection: data-science agents may need task-specific orchestration, validation, and feedback.

3 Experimental Design 3.1 Skill Construction We generated one skill per lifecycle stage (four skills total) using a Gemini 2.5 Pro coding agent with extended thinking enabled. The agent received the target workflow, required section structure, and SKILL.md schema, and it could research the target domains and related benchmarks before writing each Full skill. Each skill was produced in one autonomous session with no human editing, candidate selection, or iterative refinement. This is the low-curation workflow we study: generate one skill for a task family, then reuse it as written. Each skill contains four sections: • Routing (∼50 tokens): activation triggers specifying when to apply the skill

Wei-Jung Huang

• Core Procedure (∼190 tokens): step-by-step workflow instructions • Worked Examples (∼225 tokens): concrete input→output demonstrations • Reference Notes (∼225 tokens): supplementary heuristics and conventions These sections correspond to the content types in Anthropic’s Agent Skills documentation [1]. Unlike Anthropic’s recommended architecture, which stores reference materials in separate files loaded on demand, we inject all sections as a single flat file. We make this choice to isolate content: the experiment tests whether generated skill content helps when the model sees it, not whether a separate loader can retrieve the right file at the right time. For this reason, the study should not be read as an evaluation of selective loading or progressive disclosure. The format also matches a common low-curation implementation: a single markdown document produced without explicit tooling for multi-file packages. The ablation variants are produced by mechanically deleting sections from the Full skill, not by separate generation runs. This matters for causal interpretation: the skill variants differ only in which sections are present. Sections are written to be self-contained, so deletion does not leave dangling cross-references. This design lets us compare the full skill against variants that remove supporting content, while keeping the basic task procedure and generation source fixed.

3.2

Ablation Conditions

Our ablation treats Worked Examples and Reference Notes as two independent binary factors, with Routing and Core Procedure held constant. We keep Routing and Core Procedure together because, without activation triggers and procedural steps, the remaining sections lack the context needed to function as a coherent skill. Routing is also redundant in our setup because each skill is only paired with tasks from its matching lifecycle stage. This yields four skill conditions plus a No-Skill baseline: (1) No-Skill: task prompt only (2) Core-Only: Routing + Core Procedure (3) Core+Examples: Routing + Core Procedure + Worked Examples (4) Core+Refs: Routing + Core Procedure + Reference Notes (5) Full: all four sections Skill content is injected unconditionally as a user-message prefix using a fixed template ([SKILL START] ... [SKILL END], followed by the task prompt) across all conditions and providers. The No-Skill baseline contains only the task prompt, with no injected skill wrapper or control text. Thus, the comparison is between the same task prompt alone and the same task prompt with generated skill content prepended. Length-Control tests a simpler alternative explanation: the model may react to a long, skill-formatted prefix rather than to useful data-science guidance. To test this, we add a supplemental LengthControl condition. For each lifecycle stage, we create a task-irrelevant office-supply setup skill with the same injection wrapper, markdownstyle organization, and tokenizer length within 1% of the corresponding Full skill.

Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows

We use coherent office-supply guidance rather than random strings, which could create an unrealistic distraction. The control gives the model plausible procedural instructions, examples, and notes, but none are intended to help with data cleaning, SQL, statistical analysis, or reporting. We treat it as a control, not a harmless placebo, since unrelated instructions can still change model behavior. Comparing Full against Length-Control estimates whether the generated data-science content helps beyond adding similarly long, skill-formatted context. Generic skill guidance can conflict with the task prompt. If the main problem is priority ambiguity, a short rule telling the model to follow the task over the skill should recover some failures. To test this, we define a supplemental Full+Priority condition: it uses the same full.md skill files as Full, but inserts a user-message directive after [SKILL END] and before the task prompt stating that task instructions override conflicting skill guidance. This is a prompt-level task-over-skill directive, not a system-level instruction hierarchy or retrieval-gated priority mechanism. These supplemental conditions are narrow checks around the main flat-injection ablation. They do not test selective loading, system-level instruction priority, or schema/tool constraints. The design also omits an expert-written or task-specific positivecontrol skill. This keeps the focus on low-curation LLM-generated skills, but it means the study does not test whether this benchmark would detect gains from carefully authored skill content.

3.3

Tasks and Lifecycle Mapping

We evaluate 56 tasks (14 per lifecycle stage), mapped to four datascience workflows: • Data Preparation (14 tasks): CSV cleaning, including null imputation, deduplication, type coercion, and date standardization • Data Extraction (14 tasks): SQL query formulation against relational databases, using schemas from Spider [14] and standard benchmark databases • Statistical Analysis (14 tasks): hypothesis testing, effectsize computation, and statistical inference using standard datasets (Fisher Iris [4], mtcars, PlantGrowth, ToothGrowth) • Reporting (14 tasks): generating structured JSON reports from API responses (GitHub Events, OpenWeatherMap, REST Countries) These tasks cover the verifiable execution layer of product datascience work rather than the full product decision-making lifecycle. We intentionally exclude upstream product framing and experiment design because those tasks often depend on business context, stakeholder constraints, engineering feasibility, and judgment calls that are not well captured by the deterministic verifiers used here. Tasks are split evenly between self-authored and externally grounded examples to reduce bias from task authorship. The selfauthored subset contains 28 tasks (7 per stage). A Gemini 2.5 Pro coding agent with extended thinking enabled drafted the prompts and initial gold outputs for these tasks.

AIDataSci ’26, 2026, Jeju, Korea

The externally grounded subset also contains 28 tasks (7 per stage) and uses public datasets, benchmark schemas, or public APIs rather than purely synthetic inputs.1 Gold outputs were then validated by scripts that recompute expected results from source data where applicable. Scoring uses fixed gold files and deterministic verifiers, so Gemini does not act as the evaluator at test time. All task prompts and gold outputs were finalized before skill generation, preventing post-hoc alignment between task requirements and skill content.

3.4

Verification

All outcomes are binary pass/fail, evaluated by fully automated, deterministic verifiers with no human judgment involved. Each lifecycle stage uses a verifier matched to its output type: • Data Preparation: the model’s output CSV is parsed into a DataFrame and compared against the gold CSV. Exactmode tasks check row-level values, ignoring column order but preserving case sensitivity. Structural-mode tasks check expected columns, row counts, absence of nulls, and dateformat compliance. • Data Extraction: the model’s SQL is executed against an in-memory SQLite database seeded with task-specific test data. The result set is compared to the gold query output with order-insensitive row matching. • Statistical Analysis: numeric outputs (test statistics, pvalues, effect sizes) are extracted and compared against gold values with tolerance 𝜖 = 0.01. Both absolute and relative tolerance are applied to accommodate rounding differences. • Reporting: JSON output is validated against a task-specific schema, including required keys, value types, nested structure, and selected field values where specified. Exact-match verifiers can conflate reasoning failures with format errors. To check this risk, we programmatically identified all condition-flip cases, where a task passes under one condition but fails under another, and reviewed the associated verifier messages. In these cases, failures reflected substantive errors, such as wrong imputation method or wrong case convention due to competing instructions, rather than trivial formatting mismatches.

3.5

Models

We select nine model configurations spanning three providers (Table 1), varying along three dimensions. First, model capability: we include compact models (GPT-4o-mini, Gemini Flash, Claude Haiku) and frontier default models (GPT-4o, Gemini Pro, Claude Sonnet) to test whether stronger models use skills differently. Second, provider diversity: sampling from OpenAI, Google, and Anthropic guards against provider-specific artifacts. Third, reasoning mode: Gemini 2.5 Flash and Claude Sonnet 4 are each tested with extended thinking disabled and enabled, and o3-mini provides an always-on reasoning baseline. For stratified analysis, we use three pre-specified groups: compact default models, 1 SQL tasks use schemas from Spider [14] and Chinook/Northwind-style databases;

Data Preparation tasks use the Melbourne Housing, Titanic, UCI Wine Quality, Auto MPG, and student performance datasets; Statistical Analysis tasks use Fisher Iris [4], mtcars, PlantGrowth, ToothGrowth, and the sleep dataset; Reporting tasks use the GitHub Events, OpenWeatherMap, and REST Countries APIs.

AIDataSci ’26, 2026, Jeju, Korea

Wei-Jung Huang

frontier default models, and explicit-reasoning configurations. Gemini 2.5 Pro remains in the frontier default group because it was run only in its default configuration, with no paired disabled-thinking condition. Moving it into the explicit-reasoning group would mix model scale with reasoning-mode treatment and break the balanced design. Model

Provider

Reasoning Mode

Group

GPT-4o-mini GPT-4o o3-mini Gemini 2.5 Flash Gemini 2.5 Flash+ Gemini 2.5 Pro Claude Haiku 4.5 Claude Sonnet 4 Claude Sonnet 4+

OpenAI OpenAI OpenAI Google Google Google Anthropic Anthropic Anthropic

No No Native Disabled Enabled Default No Disabled Enabled

Compact Frontier Explicit-reasoning Compact Explicit-reasoning Frontier Compact Frontier Explicit-reasoning

Table 1: Model configurations. Note. Groups are balanced by design. Explicit-reasoning denotes configurations run through explicit reasoning-mode settings or a reasoning-only API; it does not imply that other models lack internal reasoning. Gemini 2.5 Pro is treated as frontier default because there is no paired disabled-thinking Gemini Pro condition. Temperature is set to 0 where supported; o3-mini and Claude Sonnet 4+ use provider-required reasoning settings. Temperature 0 reduces sampling variation but does not make calls deterministic.

Exact API model identifiers are recorded in the evaluation harness. The + variants use the same base identifiers as their non-+ counterparts, with extended thinking enabled. Each of the 56 × 5 × 9 = 2,520 main-ablation cells is run three times, yielding 7,560 runs. The supplemental Length-Control and Full+Priority conditions each add 56 × 1 × 9 × 3 = 1,512 runs, for 10,584 total runs. We use three repetitions because API providers can exhibit minor non-determinism even under low-variation decoding settings, due to factors such as batching and quantization. Of the 2,520 main-ablation cells, 93.4% are unanimous (all three repetitions produce the same pass/fail outcome) and only 6.6% show a split verdict. We use majority vote (2-of-3 pass) as the cell-level outcome to avoid treating repetitions as independent observations.

3.6

Analysis

We analyze results with four complementary methods: • Mixed-effects model. We fit a linear mixed-effects model with condition as a fixed effect, task as a random-intercept grouping factor, and model as a random-effect variance component. This estimates condition effects after accounting for task difficulty and model strength. We report the linear model because its coefficients directly express percentage-point differences; a logistic GEE and a non-parametric permutation test give substantively identical conclusions. • Group-stratified bootstrap analysis. We estimate skill effects separately for compact, frontier, and explicit-reasoning groups to test whether effects differ across model configurations. • Bootstrap confidence intervals. We compute percentile bootstrap intervals with 10,000 iterations over the 56 tasklevel paired differences.

• McNemar tests. For each skill condition, we count taskmodel pairs where skill content flips the outcome from fail to pass versus pass to fail, then test whether these counts differ. We analyze Length-Control and Full+Priority as supplemental paired controls against No-Skill and Full, rather than as skill components in the main ablation.

4 Results 4.1 LLM-Generated Skills Do Not Improve Performance Table 2 presents pass rates by model and condition. The aggregate spread across all five conditions is just 1.2 pp (67.3% to 68.5%). The mixed-effects model (Table 3) gives the same conclusion: no condition reaches significance at 𝛼=0.05, and all four skill conditions fall well short (all 𝑝≥0.396). Paired McNemar tests tell a similar story: for each skill condition, skills help and hurt on roughly equal numbers of task×model pairs (e.g., Core+Examples: 20 helped vs. 19 hurt, 𝑝=1.000). Full skills consume ∼4.5× the input tokens of No-Skill (mean 1,293 vs. 287 input tokens), so the tested knowledge-injection mechanism adds cost without measurable accuracy gain.

4.2

Task Source Does Not Explain the Null

Because half of the tasks are self-authored and half are externally grounded, we check whether the null result is an artifact of task construction. Table 4 shows that the source split does not change the conclusion. Full skills are slightly below No-Skill for both self-authored tasks (−0.8 pp, 95% CI [−5.2, +3.2]) and externally grounded tasks (−0.8 pp, 95% CI [−5.2, +2.8]). Averaging across all four skill conditions yields the same pattern: −0.7 pp on selfauthored tasks and −0.5 pp on externally grounded tasks. Thus, the aggregate null is not driven by one source group.

4.3

Token-Matched Control

We additionally compare Full skills against a token-matched LengthControl condition containing task-irrelevant office-supply guidance. This analysis helps separate the effect of generated data-science content from the effect of adding similarly long, skill-formatted context. Length-Control reaches 66.9% pass rate, compared with 67.5% for Full and 68.3% for No-Skill. The paired comparisons in Table 5 provide no evidence that the data-science skill content improves over the matched control: Full exceeds Length-Control by only +0.6 pp (95% CI [−2.2, +3.4], 𝑝=0.775). Length-Control also does not show strong evidence of harm relative to No-Skill (−1.4 pp, 95% CI [−3.4, +0.6], 𝑝=0.311), although the point estimate is negative. Thus, the data do not support a simple prompt-length explanation for the null, nor is there measurable evidence that the generated data-science content adds useful information beyond a similarly sized, skill-formatted control. We treat this as a control analysis, not as proof that irrelevant content has no effect.

Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows

AIDataSci ’26, 2026, Jeju, Korea

No-Skill

Full

Core-Only

Core+Ex

Core+Refs

GPT-4o-mini GPT-4o o3-mini Gemini 2.5 Flash Gemini 2.5 Flash+ Gemini 2.5 Pro Claude Haiku 4.5 Claude Sonnet 4 Claude Sonnet 4+

57.1% 64.3% 80.4% 55.4% 58.9% 69.6% 67.9% 83.9% 76.8%

57.1% 64.3% 80.4% 58.9% 57.1% 67.9% 66.1% 80.4% 75.0%

57.1% 58.9% 78.6% 58.9% 60.7% 71.4% 66.1% 82.1% 73.2%

51.8% 60.7% 82.1% 62.5% 58.9% 76.8% 66.1% 82.1% 75.0%

58.9% 60.7% 82.1% 57.1% 57.1% 69.6% 66.1% 82.1% 71.4%

Average

68.3%

67.5%

67.5%

68.5%

67.3%

Table 2: Pass rates by model and condition. Note. Each cell aggregates 56 tasks with three repetitions by majority vote. Bold indicates the best condition in each row; underline indicates the worst. Across model-condition cells, task-level standard deviations range from 0.37 to 0.50.

Condition

𝛽ˆ

𝑝

Full Core-Only Core+Ex Core+Refs

−0.008 −0.008 +0.002 −0.010

0.497 0.497 0.865 0.396

Table 3: Mixed-model condition effects. Note. 𝑛 =2,520 majority-vote outcomes; No-Skill is the reference. 𝛽ˆ is the effect in pass-rate points. The model includes task random intercepts and a model random-effect variance component. Logistic GEE and permutation tests yield substantively identical conclusions (all 𝑝>0.45). Task Source Self-authored Externally grounded

Tasks

No-Skill

Full

Avg Skill

Skill Diff.

28 28

70.2 66.3

69.4 65.5

69.5 65.8

− 0.7 [ − 4.3, +2.6] − 0.5 [ − 4.3, +2.7]

Table 4: Task-source split. Note. Entries are percentages based on majority-vote outcomes. Avg Skill averages Full, Core-Only, Core+Examples, and Core+Refs. Skill Diff. reports Avg Skill minus No-Skill, with 95% bootstrap CIs over task-level paired differences.

Comparison Length-Control vs. No-Skill Full vs. Length-Control Full vs. No-Skill

Diff.

95% CI

Help/Hurt

− 1.4 pp +0.6 pp − 0.8 pp

[ − 3.4, +0.6] [ − 2.2, +3.4] [ − 3.8, +2.0]

14/21 26/23 19/23

4.5

No Component Shows a Reliable Benefit

𝑝 0.311 0.775 0.644

Table 5: Paired length-control comparisons. Note. Each condition has 504 majority-vote outcomes. Differences are percentage-point changes for the left condition relative to the right. Help/Hurt counts task × model pairs where the left condition changes a failure to a pass versus a pass to a failure. Confidence intervals bootstrap over task-level paired differences; 𝑝 values use two-sided McNemar tests.

4.4

No-Skill by +0.4 pp (95% CI [−2.4, +3.0], 𝑝=0.875). In the mixedeffects model, the Full+Priority coefficient relative to No-Skill is +0.004 (𝑝=0.744). The cell transitions suggest a tradeoff rather than a clean repair. Full+Priority recovers 12 of the 23 task×model cells where No-Skill passes but Full fails, consistent with the directive resolving some skill-task conflicts. But it also introduces 8 new failures among cells where both No-Skill and Full pass, and loses 4 cells where Full alone helped. The added failures are mostly format or output-discipline errors, suggesting that the priority rule can pull the model back toward the task while weakening useful discipline supplied by the skill. The directive also does not materially repair the main Data Preparation harm. Data Preparation rises from 50.8% under Full to 51.6% under Full+Priority, still below the 57.1% No-Skill baseline. A simple user-message priority sentence is therefore not enough to remove the conflict-heavy failure mode. Stronger mechanisms, such as system-message placement, retrieval-gated skill snippets, or schema/tool-level constraints, remain open.

Prompt-Level Priority Control

We next test whether a prompt-level task-over-skill directive mitigates the suspected skill-task conflicts. Full+Priority reaches 68.7% pass rate, compared with 67.5% for Full and 68.3% for No-Skill. The paired difference is small and not statistically reliable: Full+Priority exceeds Full by +1.2 pp (95% CI [−1.0, +3.6], 𝑝=0.362) and exceeds

The aggregate null could still hide component-level cancellation: one section might help while another hurts. The component-level comparisons are small and non-significant, but they help characterize why the aggregate effect is flat: • Reference Notes. Core+Refs (67.3%) falls below No-Skill (68.3%) by −1.0 pp, and below Core-Only (67.5%) by −0.2 pp. • Worked Examples. Core+Examples (68.5%) is closest to No-Skill (68.3%), differing by just +0.2 pp. • Full Skill. Full (67.5%) matches Core-Only, suggesting that adding both Examples and Reference Notes to the base procedure produces no net change. Because the design crosses Worked Examples and Reference Notes, we can also estimate their interaction. Adding Reference Notes on top of Core+Examples changes performance by −1.0 pp, close to the −0.2 pp change from adding Reference Notes on top of Core-Only, yielding an interaction of −0.8 pp. This small value gives little evidence that a large positive effect from examples is being canceled by a large negative effect from reference notes. It

AIDataSci ’26, 2026, Jeju, Korea

Wei-Jung Huang

Record · ID 349669 · SHA-256 5eb10a09ab1c4742
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.