Practical Implementation Report on Introducing Spec-Driven Development Using AI Agents in Software Development PBL

arXiv:2608.30572v1 [cs.SE] 31 Aug 2026

Practical Implementation Report on Introducing Spec-Driven Development Using AI Agents in Software Development PBL Hidetake Tanaka1[0009−0003−7766−7639] , Hiroshi Igaki2[0009−0009−0810−983X] , Kazumasa Shimari3[0000−0001−8837−5090] , Kiyoshi Honda2[0000−0002−7725−5031] , and Naoki Fukuyasu2[0009−0006−8102−6692] 1

Nara Institute of Science and Technology, 8916-5 Takayama-cho, Ikoma, Nara 630-0192, Japan tanaka.hidetake.te0@naist.ac.jp 2 Osaka Institute of Technology, 1-79-1 Kitayama, Hirakata, Osaka 573-0196, Japan {hiroshi.igaki,kiyoshi.honda,naoki.fukuyasu}@oit.ac.jp 3 Wakayama University, Sakaedani 930, Wakayama, Wakayama 640-8510, Japan shimari@wakayama-u.ac.jp

Abstract. In recent years, autonomous AI agents such as GitHub Copilot and Claude Code have been rapidly gaining popularity. This study reports on the practical implementation of Spec-Driven Development, a software development methodology premised on AI agents, within a Software Development Project-Based Learning (SDPBL) course for thirdyear undergraduate students. We defined a workflow consisting of four phases, namely investigation, planning, implementation, and review. We also established an environment tailored for the SDPBL course where AI agents generate documentation and code during each phase. We analyzed the results from three perspectives, namely students’ subjective AI usage, implementation throughput, and code comprehension. The analysis reveals that AI usage patterns varied across development phases and teams. Moreover, while AI agent utilization increased implementation throughput, it also tended to encourage students to proceed with development without fully understanding the code. This study demonstrates that regular verification of code comprehension by instructors and appropriate feedback are essential for maintaining educational effectiveness when introducing SDD into SDPBL. Keywords: Software Engineering Education · Project-based Learning · AI Agents · Large Language Models · LLMs

1

Introduction

In recent years, software development utilizing Large Language Models (LLMs) and AI agents—which possess the capability to autonomously execute various tasks—has become increasingly widespread. In the field of software engineering education, the integration of LLMs into programming exercises [4,10,12] and

2

H. Tanaka et al.

Software Development Project-Based Learning (SDPBL) is also actively progressing [9,13]. In particular, the introduction of LLMs into SDPBL is expected to mitigate long-standing challenges, such as the gap in implementation skills among students and the disproportionate contribution of specific members to the development process [8]. On the other hand, there are still few reports on how AI agents, which provide advanced support across various stages of software development, actually impact SDPBL in practice. The year 2025 has been called the inaugural year of AI agents, with autonomous AI tools such as Devin4 , GitHub Copilot’s agent mode5 , and Claude Code6 being released between late 2024 and early 2025. These agentic coding assistants can autonomously perform complex tasks, including code editing across multiple files, compilation, execution, and bug fixing, toward given goals. An analysis by Li et al. [11] revealed that as of June 22, 2025, AI agents were active in 61,453 repositories and had submitted 456,535 pull requests on GitHub. As AI agent adoption continues to grow, determining how to effectively and educationally incorporate them into SDPBL is expected to become a critical challenge in software engineering education. Therefore, in this paper, we introduce SpecDriven Development (SDD)—a development methodology designed for the use of AI agents—into an actual SDPBL course to clarify how students utilize AI agents within the development process. Recently, SDD has been proposed as a development methodology based on AI agents7 . SDD involves defining detailed specifications clearly at the early stages of development and having AI agents generate code based on those specifications. This methodology requires AI agents to proceed with development based on structured specification documents. While AI agents create various specifications and documents, developers are required to understand and verify them, thereby maintaining code quality and consistency throughout the development process. We aim to identify the realities of AI agent utilization in the context of team development education, including both benefits (such as implementation throughput improvements and development support) and potential drawbacks (such as development proceeding without adequate understanding and the burden associated with AI agent usage). To achieve this goal, we address the following research questions: RQ1. To what extent do students utilize AI in each phase of the development process (investigation, planning, implementation, review, etc.) in SDPBL with SDD introduced? RQ2. Does the widespread use of LLMs/AI agents affect students’ implementation throughput (in terms of added lines of code) in SDPBL? RQ3. How does the level of AI usage relate to students’ code comprehension? In SDPBL where SDD has been introduced, the extent to which students tend to utilize AI tools, including AI agents, in different development phases 4

https://devin.ai/ https://github.blog/news-insights/product-news/github-copilot-the-agent-awakens/ 6 https://www.claude.com/product/claude-code 7 https://kiro.dev/blog/from-chat-to-specs-deep-dive/ 5

Spec-Driven Development Using AI Agents in SDPBL

3

(investigation, planning, implementation, review, etc.) remains unclear. We investigate the activity status of each student and team through weekly surveys and analyze how LLMs are utilized across different phases. In these surveys, we define the level of AI usage on a five-point scale, ranging from level 0 (no use) to level 4 (very helpful), based on students’ perceived effectiveness in each phase. Next, we analyze the impact of LLM usage on implementation throughput. While it has been reported that utilizing LLMs in software development improves code contributions [15], other studies suggest that LLM contributions are limited to experienced developers and have limitations for inexperienced developers [16]. In this study, we aim to quantitatively analyze how students, who are unfamiliar with team development in SDPBL contexts, are affected in their implementation throughput (specifically, added lines of code) by using LLMs. Regarding the impact of AI usage, it has also been suggested that students may pay less attention to code and have limited understanding when developing with LLMs [15]. In the SDPBL we conducted, we paid particular attention to this point and conducted one-on-one interviews with each student at each lecture session to assess their code comprehension on a four-level scale. By comparing these interview results with the previously defined levels of AI usage, we analyze how the level of AI usage relates to students’ ability to understand and explain their own code. The remainder of this paper is organized as follows. Section 2 summarizes related work on LLM-based software development, AI agent-based software development, SDD, and generative AI in software engineering education. Section 3 details the application of SDD to our SDPBL course and its implementation status. Section 4 presents our findings. Section 5 discusses implications for introducing SDD to SDPBL and threats to validity. Section 6 concludes the paper.

2

Related Work

2.1

LLM-Based Software Development

Software development utilizing Large Language Models (LLMs) has become widespread in recent years, with a development methodology known as “Vibe Coding” gaining traction. This approach was popularized by AI researcher Andrej Karpathy through a post on X8 , representing a concept where developers proceed with development through dialogue with generative AI without closely examining code details. This methodology is particularly suited for rapid Proof of Concept (POC) and prototype development. Developers can quickly transform ideas into tangible forms without spending extensive time on traditional detailed implementations. Conversely, ensuring the quality and consistency of generated code is challenging in Vibe Coding because developers do not directly verify implementation details [17,19]. A study by Hou et al. [7] highlights that code generated 8

https://x.com/karpathy/status/1886192184808149383

4

H. Tanaka et al.

by LLMs often selects generic designs without considering project-specific coding conventions or design patterns. Furthermore, it identifies issues such as the accumulation of technical debt due to the disregard for architectural patterns. 2.2

AI Agent-Based Software Development

AI agents have been increasingly applied to various software engineering phases. Bouzenia et al. [2] proposed RepairAgent, an agent that autonomously selects and executes appropriate tools for bug fixing. It iteratively performs tasks ranging from bug information collection to fixing and verification based on gathered data and feedback from previous attempts. Evaluations using the Defects4J dataset demonstrate its capability to autonomously repair 164 bugs. Additionally, Batole et al. [1] introduced LocalizeAgent to maintain design quality. This multi-agent framework summarizes program analysis results into LLMunderstandable formats and identifies design issues through prompt generation tailored to specific refactoring types. It achieved up to a 206% improvement over the baseline in localization accuracy for issues such as modularity. According to a survey by He et al. [5], AI agents are evolving beyond singletask assistance toward automating entire development processes. Their importance is expected to grow further across various software development tasks, including requirements definition [14], design [21], implementation [18,20], testing [3], and code review [6]. 2.3

Spec-Driven Development

To address the challenges of Vibe Coding, Spec-Driven Development (SDD) has been proposed 7 . This approach involves defining detailed specifications clearly at the early stages of development and having AI agents generate code based on those specifications. The distinguishing feature of SDD is that AI agents proceed with development based on structured specification documents rather than ambiguous natural language instructions. These specifications explicitly describe architectural patterns, coding conventions, security requirements, and testing strategies. AI agents recognize these specifications as constraints and generate code compliant with them. The AWS Kiro documentation9 , which introduced SDD to the world in July 2025, defines a workflow where AI agents sequentially execute phases such as requirements analysis, design, and implementation planning. During each phase, documents such as requirements.md, design.md, and tasks.md are generated by AI agents. Developers continuously verify these documents at each step to ensure alignment with their intent. By continuously verifying and refining these specifications, developers can ensure quality consistency and maintainability throughout the subsequent implementation phase. Many AI agents, including GitHub Copilot, can interpret these workflows and project-specific coding conventions from Markdown documents called custom instructions. For instance, by describing workflows and project-specific rules 9

https://kiro.dev/docs/specs/

Spec-Driven Development Using AI Agents in SDPBL

5

(e.g., generating outputs in Japanese or adhering to specific coding conventions) in a file named copilot-instructions.md, developers can enable AI agents to perform SDD according to the specified instructions. In SDD, although AI agents create various specifications and documents, developers are required to understand and verify them. Consequently, SDD demands greater expertise compared to Vibe Coding. While receiving support from AI agents, developers must understand the generated documents and source code and take responsibility for the final artifacts. Consequently, for SDPBL aimed at cultivating application development skills, SDD is more suitable than Vibe Coding. This is because students must understand the specifications and assume responsibility for the AI agent’s outputs, including both documentation and source code. 2.4

Generative AI and Software Development Education

Surveys of university programming education instructors regarding generative AI highlight conflicting views, ranging from prohibiting tools like ChatGPT to actively integrating them with educational considerations for students’ future careers [10]. Furthermore, Guner et al. [4] analyzed ChatGPT usage patterns in programming and identified five distinct student profiles (e.g., AI-dependent code generators), reporting that appropriate instructional intervention significantly improves both exam performance and usage patterns. The integration of LLMs into team-based development, such as SDPBL, is also advancing [9,13]. However, few studies have comprehensively investigated the effectiveness of AI agents when integrated into team development across various development phases. Determining how to effectively incorporate AI agents into team-based projects like SDPBL is expected to be a significant challenge in software engineering education. Therefore, this paper describes the application methodology of Spec-Driven Development (SDD) in our SDPBL course for third-year undergraduate students at Osaka Institute of Technology and presents an analysis of the students’ implementation status.

3

Application of Spec-Driven Development to SDPBL

This section describes the application of SDD to our SDPBL course. First, we provide an overview of the target course, followed by a detailed explanation of the materials, focusing on a specific case of introducing an AI agent into SDPBL within the course. Then, we describe the evaluation methodology used throughout the course and present the approach for addressing the RQs. 3.1

Overview of the SDPBL Course

This paper focuses on our SDPBL course, conducted as part of the “Information System Development Exercise” course for the Department of Information Systems at Osaka Institute of Technology. This elective course is offered in the

6

H. Tanaka et al.

second semester of the third year and consists of 14 weekly lectures, each with two consecutive 100-minute periods (2 credits). The objective of this PBL is for teams to develop a web application with a database, applying the knowledge of information system implementation acquired in previous coursework. Through hands-on experiences with web frameworks, server deployment, and agile processes, students aim to acquire the essential skills and knowledge required for system engineers. While participating students possess a foundational understanding of Java syntax and have received prior instruction on Git and GitHub, they typically have minimal experience in team-based development for Spring Boot or web applications. The first five of the 14 lectures consist of tutorials on the Spring Boot framework. Starting from the 6th lecture, students are divided into teams of three to four for the SDPBL, which culminates in a final presentation in the 14th lecture. The development environment consists of Visual Studio Code, Git, and GitHub to support collaborative work. Teams follow the GitHub Flow10 and submit weekly progress reports. These reports document each team member’s development plan and implementation details for the preceding week, including branch names, corresponding pull requests (PRs), and review results. Each lecture begins with a 30-minute retrospective using the KPT (Keep, Problem, Try) method, the results of which are included in the reports. Following the retrospectives, instructors review the reports and interview students to provide feedback on their progress and KPT results. Since teams are required to demonstrate a working version of their application during each weekly progress report, design and implementation activities are carried out in almost every lecture. 3.2

Preparation and Lecture Content for Application of Spec-Driven Development

To apply SDD to the SDPBL course, we implemented the following preparations in the Information System Development Exercise course in the 2025 academic year. Lecture on Spec-Driven Development Instructors explain the concepts of GitHub Copilot (agent mode), Vibe Coding, and SDD (as discussed in Sections 2.1 and 2.3) to the students. Additionally, they present a workflow for applying SDD to the SDPBL course (Figure 1). As shown in Figure 1, the development process is divided into four phases: investigation, planning, implementation, and review. In each phase, AI agents generate artifacts, including investigation reports and implementation plans. Specifically, the requirements analysis and design phases of the existing SDD workflow are redefined as the investigation phase in our workflow, and a review phase is added after implementation. This modification aims to generalize and simplify the workflow by focusing on investigating current specifications for various purposes, such as feature additions and debugging, rather than just analyzing new requirements. The review phase is explicitly 10

https://docs.github.com/en/get-started/using-github/github-flow

Spec-Driven Development Using AI Agents in SDPBL

1$#$,*.-$"& +2$.)2)&'*"

/$#'$0

!"#$%&'()&'*"

!"#$%&'()&'*"+ ,$-*,&

!"#$"%#&"'( )*#+,)(-.

+,)""'"(

<$#'$=+,$-*,&

94*"$:+,$-*,&;+ %-$8%234

7

./)" 0&)%1%2345

!-.,$-$"&)&'*" 6*7,8$+8*4$

Fig. 1. Workflow of Spec-Driven Development in the SDPBL course

included to have both AI agents and students confirm that the implementation matches the specifications and follows coding conventions. Instructors emphasize the importance of carefully reviewing and understanding each AI-generated artifact to ensure consistent development. Distribution of the Template Project Following the lecture, a template project directory for Visual Studio Code is distributed to each team. Figure 2 illustrates the directory structure; AI-generated artifacts are stored in the docs/ directory. The done/, investigate/, and review/ subdirectories contain the corresponding reports. Each team renames the RenameMe directory to their project name, creates a Spring Boot project in the NewSpringBootProject directory, and registers it as a GitHub repository. copilot-instructions.md contains the workflow and project-specific rules as custom instructions. Demo of Spec-Driven Development Following the explanation of the AI agents and the workflow, a demonstration was conducted where two instructors performed one full cycle of the workflow on the template project in front of the students. The demonstration used the prompts shown in Figure 3 to guide the AI agent through the investigation, planning, and implementation phases. The demonstration began with an investigation prompt and a review of the generated report, followed by a planning prompt to generate tasks.md. After verifying the plan for functional requirements and testing procedures, the instructors formulated an implementation prompt. Upon receiving the implementation instruction, the AI agent edits the source files and updates specs.md. In the demonstration, the instructors verified that the login functionality was correctly implemented according to tasks.md and explained the verification points

8

H. Tanaka et al.

RenameMe/ .github/ copilot-instructions.md docs/ reports/ done/ investigate/ review/ specs.md tasks.md NewSpringBootProject/

Fig. 2. Directory structure of the template project

to the students. The instructors emphasized that students must be able to explain any AI-generated code. Following implementation, the AI agent reviews the code to ensure that requirements are met, coding conventions are followed, and specs.md is updated. The resulting review report evaluates functionality, security, code quality, and documentation consistency, and provides prioritized improvement suggestions. If issues are identified, the process returns to the investigation phase. The project developed by the instructors with the AI agent based on this workflow is published on GitHub11 (written in Japanese). 3.3

Evaluation Methodology

As described in Section 3.1, the SDPBL started from the 6th lecture, and we analyze team activities from November 11, 2025 to January 13, 2026. To address RQ1–RQ3, we collect and analyze data on (1) student perception of AI (including AI agents or other LLM services) utilization (RQ1), (2) the number of lines added (LOC) by each student per lecture (RQ2), and (3) students’ comprehension of the implemented content for each lecture (RQ3). To monitor AI utilization, we conduct weekly student surveys. While our primary research focus is on AI agents, we recognize that students may employ a broad range of AI-related services beyond these specific tools. Therefore, we intentionally designed the surveys to capture the overall status of AI utilization, ensuring that any AI-assisted activities are documented regardless of the platform used. First, we assess students’ AI use up to the 5th lecture—before the introduction of SDD by the instructors. From the 6th lecture onwards, the same questions are administered every lecture. Since the deadline for these surveys is set for the day before the next lecture, the responses for a given lecture (e.g., lec06) reflect activity from the day of that lecture until the day before the subsequent one. The survey asked students to describe how they utilized Generative 11

https://github.com/kiyoshi-honda/InuzukiProject

Spec-Driven Development Using AI Agents in SDPBL Prompt for the investigation phase When accessing http://localhost:8080/, the login page should be displayed, and after logging in with ID yamada / PW taro, the page should be displayed. Investigate the implementation method.

Prompt for the planning phase

Prompt for the implementation phase

Based on the investigation results, create a plan for implementing login with in-memory user definitions (ID: yamada, PW: taro).

Implement the login functionality. Encode the password using BCrypt. Use the default login page for the custom login page.

9

Fig. 3. Examples of prompts for the investigation, planning, and implementation phases

AI in their team development activities during the week, without restricting the scope to specific AI agents. Specifically, students rated the extent to which they used AI tools for tasks such as investigation, planning, implementation, review, debugging, and explanation (of code, etc.). Responses were provided on a 0–4 scale, where 0 indicated “No use,” and 1–4 ranged from “Used but not helpful at all” to “Used and very helpful.” This allowed us to analyze not only whether the tools were used but also the students’ perceived effectiveness. Regarding the number of lines added, we retrieve the Git repositories and aggregate the LOC added by each student for all commits on the main branch during each lecture’s period. A lecture’s period is defined from the day of the lecture until the day before the next lecture. Files that were clearly not new implementation, such as external library installations or accidentally included log files, were manually excluded by the authors. To assess students’ understanding of the code and architecture, we provided opportunities for each student to explain the code they committed to the repository during progress reports. Instructors directly assessed understanding based on a four-level scale, where 0 denotes no understanding, 1 denotes partial understanding, 2 denotes substantial understanding, and 3 denotes full understanding.

4

Results

4.1

RQ1. To what extent do students use AI in each phase of the development process in SDPBL with SDD?

Each graph in Figure 4 shows how much students used AI in each phase: Investigation, Planning, Implementation, Review, and Debugging. Students reported their AI use on a 5-point scale from 0 to 4, and these reports were aggregated by team. lec01–lec05 show the results from the individual exercise period before

H. Tanaka et al.

2 1

3 2 1

3 2 1

le

c0 le

c0 le

Debug Phase Mean Usage Level

Review Phase 4 3 2 1

3 2 1

le

c0 le

105 le c0 6 le c0 7 le c0 8 le c0 9 le c1 0 le c1 1 le c1 2 le c1 3

0

105 le c0 6 le c0 7 le c0 8 le c0 9 le c1 0 le c1 1 le c1 2 le c1 3

0

4

c0

Mean Usage Level

4

0

105 le c0 6 le c0 7 le c0 8 le c0 9 le c1 0 le c1 1 le c1 2 le c1 3

0

105 le c0 6 le c0 7 le c0 8 le c0 9 le c1 0 le c1 1 le c1 2 le c1 3

0

4

105 le c0 6 le c0 7 le c0 8 le c0 9 le c1 0 le c1 1 le c1 2 le c1 3

3

Implementation Phase Mean Usage Level

Planning Phase Mean Usage Level

Mean Usage Level

Investigation Phase 4

c0

10

Fig. 4. Trend of AI Utilization

team development started. These results show which phases students used AI in and how much they used it during this period. lec06 and later show the results at the end of each week. Students self-reported how much they used AI in each phase during that lecture week. These results show, in the Implementation and Debugging phases, which relate to source code generation and understanding, all teams used AI to some extent. However, in other phases, there are large differences in AI use between teams. In the Investigation and Planning phase, Team 25A and Team 25B used AI effectively. In the Review phase, all teams did not use AI much. In this SDPBL, Team 25A and 25B used AI effectively throughout all phases. Team 25C used AI less compared to other teams. Team 25D showed large differences in AI use depending on the phase. Team 25D hardly used AI in Investigation and Planning. However, Team 25D used AI effectively in Implementation and Debugging. These results suggest that when SDD is introduced into SDPBL, how much students use AI in each phase is likely to differ by team. Especially in phases other than source code generation and comprehension, such as Investigation and Planning, AI use tends to differ greatly by team and by student. 4.2

RQ2. Does the widespread use of LLMs/AI agents affect student’s implementation throughput (added LOC) in SDPBL?

ChatGPT was released in November 2022. Since that time, generative AI has been able to generate source code. However, until OpenAI announced the "Code Interpreter" function in July 2023, advanced programming support with code execution and debugging was difficult. Therefore, we estimate that generative AI was rarely used in 2022 for Spring Boot framework development, which is the target of our SDPBL. After 2023, students who used generative AI for coding in SDPBL gradually increased. In this course, until 2024, active use of generative AI was not recommended. Also, no instruction was given to prohibit the

Spec-Driven Development Using AI Agents in SDPBL

11

use of generative AI. In 2025, it became possible to use AI agent functions in GitHub Copilot. As described in Section 3, we explained SDD in the course. We recommended that students actively use AI agents for development. Based on the above background, we conducted an analysis to evaluate the impact of LLM and AI agent use on our SDPBL. We analyzed the number of added LOC per student per lecture from 2022 to 2025. The number of students (and teams) who took our SDPBL course in each academic year was 38 (10 teams) in 2022, 28 (7 teams) in 2023, 25 (7 teams) in 2024, and 14 (4 teams) in 2025. The data collection method is shown below: 1. We collected the GitHub repository URLs of all teams that participated in SDPBL in each year. We also collected account information for each student. 2. We obtained the Git repositories of all teams. We analyzed all commits included in the period for each lecture targeting the main branch. We obtained a list of added LOC for each file included in the commits. Here, the period for each lecture is from the day of the lecture to the day before the next lecture. 3. We manually removed unnecessary files from the list of added LOC for each file. The files we removed are as follows: external library files, files mistakenly included in commits (such as log files), manuals such as README, Markdown files generated by AI agents, other files that do not affect program execution. 4. We aggregated the remaining list of added LOC for each file by student and by lecture. Figure 5 shows the number of added LOC for all students per lecture from 2022 to 2025 as box plots. This figure shows a comparison of added LOC between years. The horizontal axis of each graph shows the year. The vertical axis of each graph shows the number of added LOC. We examined year-to-year differences in the number of added LOC during eight lectures spanning the 2022–2025 academic years. Table 1 summarizes the Steel-Dwass test results; the reported p-value is the minimum pairwise p-value for each lecture, and the corresponding pair is highlighted in bold. The SteelDwass tests revealed statistically significant differences among academic years in seven of the eight lectures: lec06, lec07, lec08, lec09, lec10, lec12, and lec13. Only lec11 showed no significant differences. The Steel-Dwass pairwise comparisons identified specific year pairs exhibiting significant differences (Table 1). A consistent pattern is that earlier years (2022, 2023) have fewer added LOC than later years (2024, 2025). For example, Year 2022 is significantly lower than Year 2024 in lec06, lec07, lec08, lec10, and lec12, and Year 2023 is significantly lower than Year 2024 in lec08 and lec13. No instances were observed where Years 2024 or 2025 had significantly fewer added LOC than Years 2022 or 2023, suggesting an increase in students’ implementation throughput (added LOC per lecture) that corresponds temporally with the proliferation of LLM technologies.

12

H. Tanaka et al.

Fig. 5. Year-to-year Comparison of added LOC for Each Lecture with Steel-Dwass test Results Note: We treated values exceeding 1000 lines as extremely large values even among outliers. We did not draw these values in the graph. The p-values shown in each graph correspond to the minimum pairwise p-value identified by the Steel-Dwass test.

4.3

RQ3. How does the level of AI usage relate to students’ code comprehension?

In Figure 6a, average code comprehension levels are plotted for four AI usage levels throughout the Implementation phase. Level 1 is omitted because no students reported AI usage level 1 during this phase. The analysis distinguishes three periods: Early (lec07–08), Mid (lec09–10), and Late (lec11–12). The average was calculated from each student’s code comprehension level. In this figure, we can see that students at AI level 0, that is, students who implemented code without using AI, maintained consistently high code comprehension levels throughout all periods. On the other hand, students at AI level 3 implemented code with very little comprehension, especially during the Mid period (lec09–10). Figure 6b shows a graph that plots the average code comprehension level for each team across lectures. In Figure 6b, we can see that Team 25C, which used AI the least during the Implementation phase, maintained consistently high code comprehension scores throughout all periods. Furthermore, we compared the transitions of AI usage levels during the Implementation phase in Figure 4 with the code comprehension trends in Figure 6b. By comparing these two figures, we obtained the following observations: 1. For all teams except Team 25C (Team 25A, Team 25B, and Team 25D), code comprehension levels fluctuated significantly across lectures. Specifically, these team’s code comprehension levels dropped in lec09. At the same

Spec-Driven Development Using AI Agents in SDPBL

13

Table 1. Summary of Significant Year Differences by Lectures lecture p-value Significant Pairs lec06 lec07 lec08 lec09 lec10 lec11 lec12 lec13

0.0077 0.0052 0.0292 0.0237 0.0021 0.1608 0.0260 0.0063

22 < 24 22 < 23, 22 < 24, 22 < 25 22 < 24, 23 < 24, 23 < 25 22 < 23 22 < 23, 22 < 24, 22 < 25 Not significant 22 < 24 23 < 24

Note: p-values are the minimum pairwise p-values obtained from the Steel-Dwass test for each lecture. The pair shown in bold corresponds to this minimum p-value. YearA < YearB indicates that YearA has significantly fewer added LOC than YearB (p < 0.05, Steel-Dwass test).

time, these teams’ AI usage levels during the Implementation phase (lec09) became very high. 2. We observed that most teams’ code comprehension levels improved in lec08 and lec10. At this time, two patterns existed. The first pattern was teams that improved their code comprehension levels while maintaining high AI usage levels (ex. Team 25B). The second pattern was teams that improved their code comprehension levels while reducing their AI usage levels (ex. Team 25A). 3. We observed that in lec11 and lec12, most teams showed high code comprehension levels. In this SDPBL, instructors check each student’s code comprehension level one-on-one every week from lec07 onward. Therefore, it is highly likely that students and teams pointed out by instructors for having low code comprehension tried to understand the code as much as possible in the following week. Furthermore, we observed that code comprehension levels in the Late period (lec11–12) showed improvement trends regardless of AI usage. We believe that when introducing LLMs into SDPBL, it is important to have some form of mechanism to assess whether students understand the content of AI-generated code.

5

Discussion

Based on our observations from the SDPBL course, this section presents practical considerations for instructors planning to introduce AI agents into SDPBL. We then discuss the threats to the validity of our findings. 5.1

Considerations for Introducing AI Agents in SDPBL

Based on the findings from RQ1–RQ3, we discuss three key considerations for instructors planning to adopt AI agents in SDPBL.

14

H. Tanaka et al.

Mean Code Comprehension LV

(a) Code Comprehension Trends by AI Usage Level Over Lectures

3.0 2.5 2.0 1.5 1.0 0.5 0.0

lec07

lec08

lec09

lec10

lec11

lec12

(b) Code Comprehension Trends by Team Over Lectures Fig. 6. Analysis of Code Comprehension and AI Usage (Implementation Phase)

Balancing Code Comprehension and AI Usage RQ1 and RQ3 explored the relationship between code comprehension and the utilization level of AI. Our observations indicate that students who did not use AI at all tended to exhibit high code comprehension scores. However, across the student body as a whole, no clear statistical trend was confirmed between AI usage levels and code comprehension. Several factors likely contributed to this result. First, the limited sample size may have reduced the statistical power to detect a definitive trend. Second, the Standard Error of the Mean (SEM) was relatively large across all levels, reflecting high individual variability in how students integrated AI into their learning. Furthermore, the results may have been influenced by timely instructor interventions, such as one-on-one interviews intended to provide targeted guidance to students with lower comprehension levels. A potential example of this impact was observed in lec09, where AI usage peaked and comprehension scores dropped; however, following instructor support, comprehension showed a measurable recovery in lec10. These interventions may have encouraged students to prioritize logic over simple code generation, potentially mitigating the risk of over-reliance on AI and narrowing the usage gap across proficiency levels.

Spec-Driven Development Using AI Agents in SDPBL

15

Meanwhile, the RQ2 results indicate an increase in implementation throughput (added LOC per student) from 2022 to 2024–2025. This trend suggests that AI may accelerate the pace of development. However, this increased output does not necessarily imply deeper learning, as students might prioritize task completion over a thorough understanding of the produced code. Balancing this increased implementation speed with the necessity of ensuring fundamental code comprehension remains a key pedagogical consideration. We recommend three approaches to address these challenges: – Foundation-first approach: In our SDPBL, the first five of the 14 sessions were designated as a tutorial period, covering the fundamentals of Web application development and team collaboration. By providing reference sample applications, we aimed to support students in understanding and implementing code independently. This foundation appears to be essential for building basic skills before students begin to utilize AI tools more extensively. – Continuous monitoring with timely intervention: Regular assessments are necessary to detect comprehension drops early. As demonstrated in lec09, proactive intervention is effective when applied promptly after identifying that AI usage is replacing rather than supporting comprehension. – Accountability for AI-generated code: Although a significant negative correlation between AI usage and code comprehension was not observed, the instructor’s one-on-one interviews may have played a role in maintaining student engagement. These interviews can serve as an opportunity for students to demonstrate accountability for their work. Rather than using AI as a shortcut, students are encouraged to explain the logic behind any code they did not manually write, which may help ensure that implementation speed does not compromise their underlying understanding.

Technical Infrastructure To support the integration of AI agents into the SDPBL, we distributed standardized project templates followed by a 30-minute lecture and a one-hour demonstration. Based on our observations, these project templates appeared to help students smoothly initiate their development tasks using AI agents. These standardized templates should include pre-configured AI instructions (copilot-instructions.md), specification templates, and development rules, such as coding conventions. Such tools may reduce cognitive load, potentially allowing students to focus on high-level design and learning rather than environment setup. Moreover, the provided templates appear to serve dual purposes: for students with less experience, they offer concrete examples of how to structure instructions for AI agents; for more motivated students familiar with AI tools, they may provide a foundation for experimenting with customization and refinement. In this way, these materials can serve as valuable learning resources adapted to different skill levels.

16

5.2

H. Tanaka et al.

Threats to Validity

Internal Validity: In this study, we did not conduct a controlled experiment with a concurrent control group that abstained from using AI agents. Although we compared the results with historical data from 2022 to 2024, we cannot rule out the possibility that year-to-year variations in student skill sets, differences in cohort size, or subtle changes in the educational environment served as confounding factors. Therefore, care must be taken in concluding that the observed changes are solely attributable to the introduction of AI agents. Construct Validity: We used “added LOC” as the primary metric for development efficiency. However, efficiency in software engineering is a multifaceted concept that encompasses class design quality, refactoring, and the removal of redundant code. Since added LOC cannot fully capture these qualitative activities, the results may not entirely reflect the comprehensive nature of software development efficiency. Additionally, the code comprehension scores rely on instructors’ subjective evaluations. While this approach lacks the rigor of objective tests, it enables the assessment of the depth of students’ understanding through dialogue, which is difficult to capture with purely quantitative metrics. External Validity: This study focuses on a specific course using Java and Spring Boot and is limited to an experiment involving only four teams. Consequently, the results cannot be fully generalized to all contexts. However, they provide relevant and informative guidance for similar PBL courses and team developments that employ widely adopted technologies.

6

Conclusion

This study examined the integration of AI agents into our SDPBL course, analyzing student activities through self-reported AI usage, added lines of code, and code comprehension levels. Our findings provide several key insights. Students actively utilized AI agents across all development phases, with usage patterns varying between teams. We observed a notable increase in implementation throughput, with students in 2024–2025 adding more lines of code compared to those in 2022–2023. Regarding the relationship between AI usage and code comprehension, although mean values suggest that students with lower comprehension might rely more on AI, this trend was not statistically significant. The lack of a significant correlation may be attributed to our pedagogical approach, where one-on-one instructor interviews and timely interventions potentially encouraged students to maintain their comprehension of AI-generated code, thereby mitigating the risk of over-reliance. We conclude that while AI agents can enhance implementation speed in SDPBL, their introduction should be accompanied by careful educational management. Regular monitoring of code comprehension and proactive feedback appear to be important for maintaining educational effectiveness. Providing proper guidance may help ensure that students use AI as a supportive tool rather than a substitute for deep understanding.

Spec-Driven Development Using AI Agents in SDPBL

17

Future work should focus on three areas. First, developing effective methods to assess and improve students’ understanding of AI-generated code. Second, investigating optimal workflows that balance AI assistance with learning outcomes. Third, conducting longitudinal studies to examine how early exposure to AI agents affects students’ long-term software engineering skills. Acknowledgments. This work has been supported by JST BOOST, Japan Grant Number JPMJBS2423, JSPS KAKENHI Grant Number 26K15097 and 26K21197, and Support Center for Advanced Telecommunications Technology Research. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.

References 1. Batole, F., OBrien, D., Nguyen, T.N., Dyer, R., Rajan, H.: An llm-based agentoriented approach for automated code design issue localization. In: Proceedings of the IEEE/ACM 47th International Conference on Software Engineering. p. 1320–1332 (2025). https://doi.org/10.1109/ICSE55347.2025.00100 2. Bouzenia, I., Devanbu, P., Pradel, M.: Repairagent: An autonomous, llmbased agent for program repair. In: Proceedings of the IEEE/ACM 47th International Conference on Software Engineering. p. 2188–2200 (2025). https://doi.org/10.1109/ICSE55347.2025.00157 3. Feldt, R., Kang, S., Yoon, J., Yoo, S.: Towards autonomous testing agents via conversational large language models. In: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering. p. 1688–1693 (2024). https://doi.org/10.1109/ASE56229.2023.00148 4. Güner, H., Er, E.: Ai in the classroom: Exploring students’ interaction with chatgpt in programming learning. Education and Information Technologies 30(9), 12681–12707 (Jan 2025). https://doi.org/10.1007/s10639-025-13337-7 5. He, J., Treude, C., Lo, D.: Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Trans. Softw. Eng. Methodol. 34(5) (May 2025). https://doi.org/10.1145/3712003 6. Heander, L., Söderberg, E., Rydenfält, C.: Support, not automation: Towards ai-supported code review for code quality and beyond. In: Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. p. 591–595. Association for Computing Machinery (2025). https://doi.org/10.1145/3696630.3728505 7. Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., et al.: Large language models for software engineering: A systematic literature review. ACM Trans. Softw. Eng. Methodol. 33(8) (Dec 2024). https://doi.org/10.1145/3695988 8. Igaki, H., Fukuyasu, N., Saiki, S., Matsumoto, S., Kusumoto, S.: Quantitative assessment with using ticket driven development for teaching scrum framework. In: Companion Proceedings of the 36th International Conference on Software Engineering. p. 372–381 (2014). https://doi.org/10.1145/2591062.2591162 9. Kharrufa, A., Alghamdi, S., Aziz, A., Bull, C.: Llms integration in software engineering team projects: Roles, impact, and a pedagogical design space for ai tools in computing education. ACM Trans. Comput. Educ. 26(2) (Jan 2026). https://doi.org/10.1145/3779296

18

H. Tanaka et al.

10. Lau, S., Guo, P.: From "ban it till we understand it" to "resistance is futile": How university programming instructors plan to adapt as more students use ai code generation and explanation tools such as chatgpt and github copilot. In: Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1. p. 106–121 (2023). https://doi.org/10.1145/3568813.3600138 11. Li, H., Zhang, H., Hassan, A.E.: The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering(arxiv) (2025). https://doi.org/https://doi.org/10.48550/arXiv.2507.15003 12. Liu, R., Zenke, C., Liu, C., Holmes, A., Thornton, P., Malan, D.J.: Teaching cs50 with ai: Leveraging generative artificial intelligence in computer science education. In: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1. p. 750–756 (2024). https://doi.org/10.1145/3626252.3630938 13. Rasnayaka, S., Wang, G., Shariffdeen, R., Iyer, G.N.: An empirical study on usage and perceptions of llms in a software engineering project. In: Proceedings of the 1st International Workshop on Large Language Models for Code. p. 111–118 (2024). https://doi.org/10.1145/3643795.3648379 14. Sami, M.A., Waseem, M., Zhang, Z., Rasheed, Z., Systä, K., Abrahamsson, P.: Early results of an ai multiagent system for requirements elicitation and analysis. In: Product-Focused Software Process Improvement: 25th International Conference, PROFES 2024, Tartu, Estonia, December 2–4, 2024, Proceedings. p. 307–316 (2024). https://doi.org/10.1007/978-3-031-78386-9_20 15. Shihab, M.I.H., Hundhausen, C., Tariq, A., Haque, S., Qiao, Y., Mulanda, B.W.: The effects of github copilot on computing students’ programming effectiveness, efficiency, and processes in brownfield coding tasks. In: Proceedings of the 2025 ACM Conference on International Computing Education Research V.1. p. 407–420. Association for Computing Machinery (2025). https://doi.org/10.1145/3702652.3744219 16. Stray, V., Moe, N.B., Ganeshan, N., Kobbenes, S.: Generative ai and developer workflows: How github copilot and chatgpt influence solo and pair programming. In: Proceedings of the 58th Hawaii International Conference on System Sciences. pp. 7381–7390 (2025). https://doi.org/10.24251/HICSS.2025.883 17. Tambon, F., Moradi-Dakhel, A., Nikanjam, A., Khomh, F., Desmarais, M.C., Antoniol, G.: Bugs in large language models generated code: an empirical study. Empirical Softw. Engg. 30(3) (Feb 2025). https://doi.org/10.1007/s10664-025-10614-4 18. Wang, H., Liu, Z., Wang, S., Cui, G., Ding, N., Liu, Z., et al.: INTERVENOR: Prompting the coding ability of large language models with the interactive chain of repair. In: Ku, L.W., Martins, A., Srikumar, V. (eds.) Findings of the Association for Computational Linguistics: ACL 2024. pp. 2081–2107 (Aug 2024). https://doi.org/10.18653/v1/2024.findings-acl.124 19. Wang, Y., Jiang, T., Liu, M., Chen, J., Mao, M., Liu, X., et al.: Beyond functional correctness: Investigating coding style inconsistencies in large language models. Proc. ACM Softw. Eng. 2(FSE) (Jun 2025). https://doi.org/10.1145/3715749 20. Zhang, H., Cheng, W., Wu, Y., Hu, W.: A pair programming framework for code generation via multi-plan exploration and feedback-driven refinement. In: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. p. 1319–1331 (2024). https://doi.org/10.1145/3691620.3695506 21. Zhou, X., Li, R., Liang, P., Zhang, B., Shahin, M., Li, Z., et al.: Using llms in generating design rationale for software architecture decisions. ACM Trans. Softw. Eng. Methodol. (Dec 2025). https://doi.org/10.1145/3785010

Related documents

• A Lean and Spec-Driven AI-Assisted Software Development Lifecycle for Applied AI Education: The AI-SDLC Approach#1050527
• A Lean and Spec-Driven AI-Assisted Software Development Lifecycle for Applied AI Education: The AI-SDLC Approach#1028765
• Practical Implementation Report on Introducing Spec-Driven Development Using AI Agents in Software Development PBL#799762
• AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applications#130756
• Remember Your Trace: Memory-Guided Long-Horizon Agentic Framework for Consistent and Hierarchical Repository-Level Code Documentation#1141539