Personalized Worked Example Generation from Student Code Submissions using Pattern-based Knowledge Components Griffin Pitts
Muntasir Hoq
Peter Brusilovsky
North Carolina State University Raleigh, NC, USA
North Carolina State University Raleigh, NC, USA
University of Pittsburgh Pittsburgh, PA, USA
Narges Norouzi
Arto Hellas Juho Leinonen
arXiv:2604.24758v1 [cs.HC] 27 Apr 2026
University of California, Berkeley Berkeley, CA, USA
Aalto University Aalto, Finland
Bita Akram North Carolina State University Raleigh, NC, USA
Abstract
1
Adaptive programming practice often relies on fixed libraries of worked examples and practice problems, which require substantial authoring effort and may not correspond well to the logical errors and partial solutions students produce while writing code. As a result, students may receive learning content that does not directly address the concepts they are working to understand, while instructors must either invest additional effort in expanding content libraries or accept a coarse level of personalization. We present an approach for knowledge-component (KC) guided educational content generation using pattern-based KCs extracted from student code. Given a problem statement and student submissions, our pipeline extracts recurring structural KC patterns from students’ code through AST-based analysis and uses them to condition a generative model. In this study, we apply this approach to worked example generation, and compare baseline and KC-conditioned outputs through expert evaluation. Results suggest that KC-conditioned generation improves topical focus and relevance to learners’ underlying logical errors, providing evidence that KC-based steering of generative models can support personalized learning at scale.
Introductory programming courses often use short learning activities that support students’ conceptual understanding through practice and explanation. One common activity type is the worked example, which typically presents students with a related problem statement, a code solution similar to what they are working on, and step-by-step instructional explanations that walk through the solution process [13, 15]. Access to a well-matched worked example at points of struggle can help students interpret errors, connect ideas to code, and keep moving forward [4]. Learning to program can be cognitively demanding, and grounded in cognitive load theory, worked examples can reduce unnecessary load by making intermediate steps explicit and limiting the amount of inference required to follow a solution [1, 13, 17]. In practice, producing high-quality worked examples at the level of detail needed for instruction is costly and time-intensive. Instructors must craft a correct solution, decide how to segment it into meaningful steps, and write explanations that are accurate, readable, and appropriate for novice programmers [8, 16]. Large repositories can reduce the need to author every example from scratch, however identifying examples that align with the specific task and difficulty a student is experiencing remains challenging [4]. In many cases, a course repository may not contain a worked example with the needed fine-grained relevance, and selecting one that does fit requires targeted retrieval or matching methods [4]. Learners’ prior knowledge further complicates authoring and selection because explanatory detail that helps novices can become redundant or obstructive for more advanced learners, consistent with the expertise reversal effect [9]. Large language models (LLMs) offer a way to reduce authoring burden by generating worked examples on demand [8, 14]. Prior evaluations in introductory programming report that expert reviewers judged LLM-generated worked examples to have clear, suitable explanations and coherent step structure, and that students found them helpful for learning and making progress on practice tasks [8, 16]. However, existing approaches often provide limited adaptation to learner differences such as skill level, which can lead to examples that introduce unfamiliar concepts for novices [16]. Moreover, they may not align the example with the specific logical error reflected in a student’s partial solution [6], so the example may be less direct in addressing the student’s current misconception.
CCS Concepts • Social and professional topics → Computing education; • Computing methodologies → Natural language processing; Knowledge representation and reasoning; Information extraction.
Keywords Worked example generation, programming education, knowledge components, AST-based code analysis, LLMs, AI in education ACM Reference Format: Griffin Pitts, Muntasir Hoq, Peter Brusilovsky, Narges Norouzi, Arto Hellas, Juho Leinonen, and Bita Akram. 2026. Personalized Worked Example Generation from Student Code Submissions using Pattern-based Knowledge Components. In Proceedings of the Thirteenth ACM Conference on Learning @ Scale (L@S ’26), June 29–July 3, 2026, Seoul, Republic of Korea. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3774398.3811577
This work is licensed under a Creative Commons Attribution-NonCommercialNoDerivatives 4.0 International License. L@S ’26, Seoul, Republic of Korea. © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2293-6/2026/06 https://doi.org/10.1145/3774398.3811577
Introduction
L@S ’26, June 29–July 3, 2026, Seoul, Republic of Korea.
Knowledge components (KCs) represent discrete units of knowledge or skill that support successful task performance [10]. In KCbased learner models, student performance is interpreted in terms of which KCs appear mastered versus still learning, and systems can use that information to select what to practice or explain next. In this work, we operationalize KCs as the concepts that a worked example should conceptually target. Specifically, following prior work [5], we extract pattern-based KCs from a student’s submission via identifying common programming patterns in abstract syntax trees (AST) found in a cohort of students’ submissions. As these KCs are derived from recurring AST substructures in the learner’s own code, they ground generation in constructs students are actively using and may be applying incorrectly or incompletely [5]. Using these pattern-based KCs, we then condition an LLM to generate a worked example whose code and explanations explicitly address the intended KC targets found in the students’ code. We compare the KC-conditioned approach to an otherwise identical generator that omits KC information, our baseline. Both approaches are evaluated through expert review using a rubric adapted from prior work [8] that assesses worked-example quality and relevance to the learner’s likely knowledge gap. For outputs guided by the pattern-based KCs, we additionally assess whether the intended KCs appear in the example code and are explicitly discussed in the step explanations. Altogether, these evaluations address the following research questions: RQ1: Does KC-conditioned generation improve worked-example quality and relevance compared to our baseline? RQ2: How reliably does our KC-conditioned pipeline produce worked examples that include and explicitly explain the intended KC patterns?
2
Related Work
Recent work has used LLMs to generate programming learning activities. Focusing on scaffolded Parsons problems, CodeTailor [7] uses a learner’s incorrect code to prompt an LLM for a corrected reference solution, then constructs a personalized Parsons problem by aligning the learner’s original code with the generated correction. Similarly, PuzzleMakerPy [2] uses an LLM to generate Parsons problems that learners can customize by thematic context and targeted concepts. For LLM-generated worked examples, prior work has examined how prompting strategies affect content quality. Jury et al. [8] studied worked example generation for CS1 students with both expert and student evaluation, reporting that LLMs can produce generally coherent worked examples, while also noting LLM outputs often lacked novice-friendly detail in step structure and commenting. Related work has evaluated similar generation in a CS2 context [12] and explored controllable rewriting that produces novice, intermediate, and advanced variants from a base worked example by adjusting step granularity and explanation detail [16]. In this work, we condition LLM-based worked-example generation on pattern-based KCs extracted from learner’s code submissions, ensuring these KCs appear in the generated example’s code and step explanations. This connects the generation of a worked example to an explicit representation of a learner’s likely logical error, supporting more personalized, instructionally-relevant generation of worked examples at scale.
G. Pitts et al.
3
Methodology
Our methodology has three stages. First, we extract pattern-based KCs from student submissions and enrich them with short, humanreadable descriptions. Second, for each submission we generate two worked examples using a baseline prompting pipeline and a KCconditioned prompting pipeline. Third, we evaluate the generated worked examples through expert review.
3.1
Dataset and Sampling
We use a publicly available CodeWorkout dataset containing 57,670 anonymized Java submissions from 368 CS1 students collected in Spring 2019, including 18,787 correct and 38,883 incorrect submissions [3]. For this study, we randomly selected two out of fifty problems from the dataset, repeatEnd and fix45 1 . repeatEnd is a string task involving substring extraction and repetition (repeat the last 𝑛 characters 𝑛 times), while fix45 is an array task involving constrained rearrangement (reorder elements so each 4 is immediately followed by a 5 without moving the 4s). For each problem, we identify a student’s last incorrect attempt by timestamp, since incorrect submissions provide the clearest evidence of students’ code patterns involving logical errors our worked-example generation pipeline was designed to address. From these candidate submissions, we randomly sample 50 submissions per problem, yielding 100 unique submissions overall.
3.2
Pattern-based KC Discovery and Labeling
We adopt the pattern-based KC discovery pipeline of Hoq et al. [5] to derive submission-level KC targets from student code. The pipeline operationalizes a KC as a recurring AST-subtree pattern that corresponds to a programming construct or a common combination of constructs that appears across many student solutions. We run the pattern-based KC discovery pipeline on the full Codeworkout dataset following the procedure in [5], and use the resulting trained components to infer KC targets for the sampled submissions. Step 1: Identify important subtrees with attention. For each submission, we parse the program into an AST and extract candidate subtrees ranging from small local constructs to larger statementlevel structures. These subtrees are fed to a Subtree-based Attention Neural Network (SANN) trained to predict whether a submission is correct or incorrect. SANN encodes each subtree into a fixed-length vector and assigns an attention weight indicating its importance for the prediction [5]. Following Hoq et al. [5], attention weights are computed with a sigmoid activation so multiple subtrees can be highlighted within a program, identifying influential patterns in the code responsible for correctness. Step 2: Normalize subtree tokens for abstraction. We normalize each retained subtree by replacing identifiers and literals (e.g., variable names, constants, method names) with placeholders [5]. This reduces sensitivity to surface-level variation that does not change program logic and supports grouping similar patterns across different student implementations. Step 3: Learn representations for clustering. We train a variational autoencoder (VAE) over sequences of high-attention subtrees to 1 https://codeworkout.cs.vt.edu/gym/exercises
Personalized Worked Example Generation using Pattern-based Knowledge Components
L@S ’26, June 29–July 3, 2026, Seoul, Republic of Korea.
learn representations that reflect how each subtree appears in program context [5]. The VAE is trained on high-attention subtrees from correct submissions so that structurally similar correct patterns embed nearby in the latent space [5]. After training, we use the encoder’s latent vectors as inputs to clustering.
Item
Description
Formatting
Step 4: Cluster latent vectors into a KC inventory, then map submissions to KCs. We cluster the VAE latent vectors from correct submissions using K-means (k = 50 as in prior work) [5]. At inference time, each high-attention subtree from either a correct or incorrect submission is assigned to its nearest cluster centroid to obtain a KC ID. In our study, these submission-level KC sets are the targets used to condition worked-example generation.
Clear explanations
Does the output follow the required worked-example structure (3–10 steps), where each step pairs code with a matching written explanation and includes helpful inline // comments? Is the worked example easy to follow, with a clear, linear flow and little assumed prior knowledge? Is the code correct by inspection, and would it produce the intended behavior for the problem? Are the steps broken into manageable parts, with each step covering a specific part of the solution and avoiding large jumps? Does the worked example address a misconception or logical error suggested by the student submission? For KC-conditioned outputs, do the provided KC targets appear in the code and are they addressed in the step text in a way that clearly connects each target to the specific code where it appears?
Step 5: Enrich KC targets for prompting. Because cluster IDs, generated following [5], are not human-readable, we add an enrichment layer that attaches short labels and one-sentence descriptions to each submission’s KC targets for use in prompts. For each KC assigned to a submission, we provide GPT-5.2 (gpt-5.2-chat-latest) with (i) the problem statement, (ii) the student submission, and (iii) a code snippet aligned to the high-attention subtree that triggered that KC assignment, then prompt it to produce a concise KC label (2–6 words) and a one-sentence description of the pattern.
3.3
Worked Example Generation
With each student submission, we derive a submission-level KC target set from the code using the Hoq et al. [5] pipeline described in Section 3.2, and generate human-readable labels and a singlesentence description for every pattern-based KC. Using these KCs, we generate two worked examples per student submission with GPT-5.2 (gpt-5.2-chat-latest), one generated with the KC targets included in the prompt and one generated that omits KC information. Prompting and problem context. The two generation pipelines (baseline and KC-conditioned) use a shared prompt template that specifies a persona as an introductory programming tutor and lays out the task, response structure, and formatting guardrails. In both variants, the prompt provides the problem statement and the student’s code, and instructs the model to infer what the student is struggling with and generate a worked example for a related but different problem that practices similar skills. The KC-conditioned variant additionally provides the student submission’s extracted pattern-based KCs, as labels with single-sentence descriptions, and instructs the model to use these KCs as constraints when inferring the student’s logical error and selecting what the worked example should practice and explain. Both pipelines enforce the same output format, based on prior work [8], where the LLM is instructed to produce a worked example with 3–10 steps, and each step pairs a brief explanation with a corresponding code fragment. Parameters for the GPT 5.2 models are left at default values, and full prompt templates are provided in Appendix A.
3.4
Expert Evaluation
Experts evaluated baseline and KC-conditioned outputs using the rubric outlined in Table 1, adapted from prior work [8]. Each rubric item is scored on a 0–2 scale, where 0 indicates the item criteria is not met, 1 indicates partial fulfillment (meets the item criteria in some aspects but with omissions), and 2 indicates the item criteria
Correctness Step structure
Relevance to the student KC coverage
Table 1: Expert-evaluation rubric for worked examples.
is fully met. For KC-conditioned outputs, experts additionally score KC coverage, which reflects whether the provided KC targets are instantiated in the worked example code and step explanations. Further, for each paired comparison, experts recorded their preference for the baseline or KC-conditioned worked example, or indicated no preference. Per student submission sampled, experts are shown the problem statement, the student’s code, and the paired worked examples. For worked examples generated with pattern-based KCs, the KC labels and descriptions are also shown. The evaluation followed a procedure recommended by [11]. Two experts first jointly coded an initial subset of generated worked examples (10%), then computed Cohen’s Kappa (𝜅) on that subset to quantify agreement. Inter-rater agreement exceeding 𝜅 = 0.80 was achieved on the first round (𝜅 = 0.90), after which the full set of generated worked examples was coded and analysis conducted. Paired Wilcoxon signed-rank tests and Holm corrected p-values were used when comparing baseline and KC-conditioned ratings.
4
Results
RQ1: Worked example quality and relevance. Across 200 generated worked examples (100 baseline, 100 KC-conditioned), experts scored outputs on five rubric items, scaled 0–2 (Table 1). For KCconditioned outputs, experts also rated a sixth item, KC coverage. The largest differences appeared on two items: novice-friendly explanations and student relevance (Table 2). KC-conditioned worked examples scored higher on student relevance (22% higher; 𝑝 = 0.001), indicating that they more often addressed the logical error suggested by the student submission rather than offering a generic solution. KC-conditioned outputs also scored higher on novicefriendly explanations (13% higher; 𝑝 = 0.027), which aligns with our observations that their step text more consistently explains the KC-targeted parts of the solution at a level of detail appropriate for novices. The remaining rubric items showed no statistically significant differences. Both pipelines consistently followed the intended worked example format and produced correct code at
L@S ’26, June 29–July 3, 2026, Seoul, Republic of Korea.
Rubric item Formatting Clear explanations Correctness Step Structure Relevance to the student
G. Pitts et al.
Baseline 𝑀
KC-conditioned 𝑀
𝑝 -value
2.00 1.81 1.97 1.99 1.67
2.00 1.94 1.99 1.94 1.89
1.00 0.027 0.635 0.076 0.001
Table 2: Baseline vs. KC-conditioned ratings (0–2 scale). 𝑀 denotes the mean expert rating across 200 generated worked examples.
Student code excerpt (problem fix45, code_id=43552) 1 2 3 4 5 6 7 8 9
if (i == 0 && nums[i] == 5 || nums[i] == 5 && nums[i-1] != 4) { int fiveSpot = i; for (int m = i; m < nums.length; m++) { if (nums[m] == 4 && nums[m+1] != 5) { int otherNum = nums[m+1]; nums[m+1] = 5; nums[fiveSpot] = otherNum; break; }}}
1 2 3
Pattern-based KC from the student submission: Boolean operator precedence
1
Question: Write a method that counts how many numbers in an array are either negative OR are zeros that do not come immediately after another zero. Be careful to correctly combine && and || in your condition.
Extracted pattern-based KC (label + description) near perfect levels. However, one tradeoff we observed was that KC-conditioned outputs received slightly lower ratings in regard to their step-structure (5% lower; 𝑝 = 0.076), which may reflect cases where incorporating additional KC targets leads the model to pack more content into each step, making step boundaries less distinct and step labels less informative. The preferences of the expert evaluators were considered, and were observed to be consistent with the rubric item ratings. Across 100 comparisons, experts had no preference for 50, preferred the KC-conditioned worked examples for 41, and preferred the baseline for 9. Based on our observations, preferences for the KC-conditioned output were typically tied to the outputs being more specific toward what the student likely misunderstood and more direct in addressing that gap, instead of presenting a broadly correct but generic solution. RQ2: KC Coverage. For KC-conditioned outputs, we asked whether the provided KC targets appear in the code and are explained in the step text in a way that link each target to the code where it appears. Alignment was high (mean = 1.92 on a 0–2 scale), and of 100 examples, 92 fully met this criterion, with 8 partially aligned. Partial alignment occurred primarily when multiple KC targets were provided, where not all targets were addressed clearly in both the code and the step explanations. In these cases, the worked example often focused on one KC and mentioned others only briefly, and step boundaries were sometimes less distinct when the model attempted to address several targets at once. This suggests the prompt may need additional guidance for prioritizing multiple KC targets, including which targets to emphasize and how to allocate them across steps.
5
Discussion and Future Work
This work proposes a KC-guided worked example generation pipeline that uses pattern-based KCs extracted from student code (via AST subtree discovery) as constraints in the generation process. This is done so that the resulting worked examples generated practice and conceptually focus on the targeted pattern-based KCs from the students code. Across expert ratings, the KC-conditioned pipeline significantly improved evaluations of explanation clarity and relevance to the student’s likely logical error. This result suggests that the use of KCs as a structured representation of what a learner needs to practice can support more targeted personalization of generated learning content. However, the partial KC-coverage cases also suggest that KC-guided generation needs mechanisms for deciding which targets to emphasize when a submission maps to several plausible KCs. In future work, these KC targets could be integrated into a broader learner modeling framework that tracks students’ mastery over time and uses that knowledge state to select and sequence KC-guided examples across practice opportunities.
Description: The student combines && and || in a single condition without parentheses, so the intended grouping may differ from Java's evaluation order.
KC-conditioned worked example excerpt
2 3
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Overview: This program will loop through an integer array and count values that meet a combined condition using both && and ||. We will use parentheses to make the logic clear and avoid mistakes with operator precedence. Step 1: Create the method and a counter variable. public int countSpecial(int[] nums) { int count = 0; // Keeps track of how many values meet the condition Step 2: Loop through the array so we can check each element. for (int i = 0; i < nums.length; i++) { // Go through every index Step 3: Use parentheses to clearly group && and || conditions. // We want to count: 1) Any negative number OR // 2) A zero that is NOT immediately after another zero if (nums[i] < 0 || (nums[i] == 0 && (i == 0 || nums[i - 1] != 0))) { count++; // Increase count if condition is true } Step 4: Finish the loop and return the result. } return count; // Return the final total }
Figure 1: Example of a high-level pattern-based KC extracted from student code and used to guide worked-example generation, targeting explicit grouping of operations with parentheses.
Future work will prioritize establishing potential learning impacts of our KC-conditioned worked examples. While expert evaluations indicate that KC inclusion can steer LLM-generation toward clearer and more instructionally-relevant instruction, an important question remains whether these examples improve learning and transfer. We plan future classroom studies where an incorrect submission triggers either a baseline or a KC-guided worked example, and we will measure improvement on a second attempt, performance on a related follow-up problem targeting the same KCs, and brief concept checks aligned with those targets. A limitation of our approach is that KC-guided generation quality depends on the quality and granularity of the discovered KC inventory and its labels, and on how accurately a new submission is mapped to the most instructionally-relevant targets. This can be challenging when a submission yields many plausible KC targets. We plan to address this by incorporating model attention weights in the generation process to prioritize KC targets, validating the human-readable KC descriptions with instructors, and testing robustness across additional problems and programming topics.
Personalized Worked Example Generation using Pattern-based Knowledge Components
Acknowledgments This research was supported by the U.S. National Science Foundation (NSF) under Grant #2426837. Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect views of the NSF.
References [1] Paul Chandler and John Sweller. 1991. Cognitive load theory and the format of instruction. Cognition and instruction 8, 4 (1991), 293–332. [2] Andre del Carpio Gutierrez, Paul Denny, and Andrew Luxton-Reilly. 2024. Automating personalized parsons problems with customized contexts and concepts. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1. 688–694. [3] Stephen H Edwards and Krishnan Panamalai Murali. 2017. CodeWorkout: short programming exercises with built-in data collection. In Proceedings of the 2017 ACM conference on innovation and technology in computer science education. 188– 193. [4] Muntasir Hoq, Atharva Patil, Kamil Akhuseyinoglu, Peter Brusilovsky, and Bita Akram. 2025. An automated approach to recommending relevant worked examples for programming problems. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1. 527–533. [5] Muntasir Hoq, Griffin Pitts, Tirth Bhatt, Aum Pandya, Andrew Lan, Peter Brusilovsky, and Bita Akram. 2025. Pattern-based Knowledge Component Extraction from Student Code Using Representation Learning. arXiv preprint arXiv:2508.09281 (2025). [6] Muntasir Hoq, Ananya Rao, Reisha Jaishankar, Krish Piryani, Nithya Janapati, Jessica Vandenberg, et al. 2025. Automated Identification of Logical Errors in Programs: Advancing Scalable Analysis of Student Misconceptions. In Proceedings of the 18th International Conference on Educational Data Mining. 90–103. [7] Xinying Hou, Zihan Wu, Xu Wang, and Barbara J Ericson. 2024. Codetailor: Llm-powered personalized parsons puzzles for engaging support while learning programming. In Proceedings of the Eleventh ACM Conference on Learning@ Scale. 51–62.
L@S ’26, June 29–July 3, 2026, Seoul, Republic of Korea.
[8] Breanna Jury, Angela Lorusso, Juho Leinonen, Paul Denny, and Andrew LuxtonReilly. 2024. Evaluating llm-generated worked examples in an introductory programming course. In Proceedings of the 26th Australasian computing education conference. 77–86. [9] Slava Kalyuga, Paul Ayres, Paul Chandler, and John Sweller. 2003. The Expertise Reversal Effect. Educational Psychologist 38, 1 (2003), 23–31. arXiv:https://doi.org/10.1207/S15326985EP3801_4 doi:10.1207/S15326985EP3 801_4 [10] Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science 36, 5 (2012), 757–798. [11] J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174. [12] Vincent Lindvall and Sofia Marcus. 2024. Using Large Language Models to Generate Worked Examples of CS2-Level Programming Questions. [13] Kasia Muldner, Jay Jennings, and Veronica Chiarelli. 2022. A review of worked examples in programming activities. ACM Transactions on Computing Education 23, 1 (2022), 1–35. [14] Griffin Pitts, Anurata Prabha Hridi, and Arun Balajiee Lekshmi Narayanan. 2025. A Survey of LLM-Based Applications in Programming Education: Balancing Automation and Human Oversight. In Proceedings of the Fourth Workshop on Bridging Human-Computer Interaction and Natural Language Processing (HCI+ NLP). 255–262. [15] Ben Skudder and Andrew Luxton-Reilly. 2014. Worked examples in computer science. In Proceedings of the Sixteenth Australasian Computing Education ConferenceVolume 148. 59–64. [16] Yfke Smit. 2025. Personalising LLM-Generated Worked Examples based on Skill Level in Introductory Programming. Master’s thesis. [17] John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science 12, 2 (1988), 257–285.
A
Prompt Templates
https://osf.io/4h9dn/overview?view_only=97189a73e56b4254bd22 98669b6eabc4