arXiv:2605.06464v1 [cs.SE] 7 May 2026
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study Shota Sawada
Tatsuya Shirai
Yutaro Kashiwa
National Institute of Technology, Nara College Yamatokoriyama, Japan [email protected]
Nara Institute of Science and Technology Ikoma, Japan [email protected]
Nara Institute of Science and Technology Ikoma, Japan [email protected]
Ken’ichi Yamaguchi
Hiroshi Iwata
Hajimu Iida
National Institute of Technology, Nara College Yamatokoriyama, Japan [email protected]
National Institute of Technology, Nara College Yamatokoriyama, Japan [email protected]
Nara Institute of Science and Technology Ikoma, Japan [email protected]
Abstract
1
LLM-based autonomous coding agents have reshaped software development. While these agents excel at code generation, open questions persist about the long-term maintainability of AI-generated code. This study empirically investigates the maintenance extent, human involvement, and modification types of AI-generated files versus human-authored code. Using the AIDev dataset of AI-generated pull requests and GitHub, we analyzed over 1,000 files and approximately 3,200 changes from 100 popular repositories. Our findings show that: (i) AI-generated files receive less frequent maintenance than human-authored code, with updates affecting only a small fraction of file size; (ii) the most frequent modifications to AI code are feature extensions, whereas human updates focus on bug fixes, and (iii) human developers perform the large majority of this maintenance.
Software maintenance is critical in the software development lifecycle to keep systems reliable, stable, and compatible with evolving technology. Developers spend considerable time on maintenance activities, including bug fixing, refactoring, and feature extensions. Previous studies [1, 2] have investigated software maintenance to understand and reduce these costs. Dehaghani et al. [3] report that approximately 90% of the software development life cycle is related to maintenance activities, and that these costs have increased by 50% over the past two decades. Most of these studies, however, were conducted before the emergence of large language models (LLMs). In particular, Agentic Coding has transformed software development. Agentic Coding is an approach in which AI agents autonomously decompose high-level instructions into subtasks and perform coding activities such as writing, debugging, and refactoring with minimal human intervention. While Agentic Coding accelerates development, prior studies indicate it introduces potential risks to code quality [4–6]. He et al. [7] revealed that although agentic coding produces a sharp immediate increase in development velocity, it results in a 30% rise in static analysis warnings and a 41% rise in code complexity over the long term. Sankhe et al. [8] confirmed that while AI assistance improved productivity by 31.4%, it also led to a 23.7% increase in security vulnerabilities and a notable rise in code duplication. These agent-related studies have primarily focused on shortterm effects, such as before and after the change. The maintenance activities required after AI-generated files are introduced have received little attention, and the extent to which human intervention is required is still unclear. This study empirically investigates the maintainability of AI-generated files and the participation of AI in maintenance activities within projects that adopt autonomous coding agents. Specifically, we identify files and commits added by four major AI agents (Copilot, Claude, Devin, and Cursor) from the repositories and PR history recorded in the AIDev dataset [9], and compare them with human-generated files and commits to clarify the maintenance activities required. Our empirical analysis of 3,238 commits shows that AI-generated files receive significantly less maintenance than human files, yet
CCS Concepts • Software and its engineering → Automatic programming; Software evolution; Maintaining software.
Keywords Agentic Coding, Software Maintenance, AI-generated code ACM Reference Format: Shota Sawada, Tatsuya Shirai, Yutaro Kashiwa, Ken’ichi Yamaguchi, Hiroshi Iwata, and Hajimu Iida. 2025. To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study. In Proceedings of The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). ACM, New York, NY, USA, 6 pages. https://doi.org/XXXXXXX. XXXXXXX
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. EASE 2026, Glasgow, United Kingdom © 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/XXXXXXX.XXXXXXX
Introduction
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
human developers perform about 83% of it. Modifications to AIgenerated files are mostly feature extensions, while human-generated files focus on bug fixes. Replication Packages: To facilitate replication and further studies, we provide the scripts and data used in our replication package.1
2
Related Work
While Agentic Coding accelerates code implementation [10, 11], a growing body of work [12, 13] has begun to examine its broader effects on development tasks. Cihan et al. [14] reported that introducing automated code review tools increased the average time required to close pull requests. Becker et al. [15] conducted a controlled experiment and found that AI tool usage led to a 19% increase in task completion time. The quality of generated code has drawn considerable attention [16]. Paul et al. [4] compared code smells in human-written and AI-generated code, finding that AI-generated code contains 63% more code smells on average. He et al. [7] reported that although development speed improves temporarily, technical debt accumulates through static analysis warnings and code complexity, leading to a decline in future velocity. From a security perspective, several studies indicate that AI-generated code is more prone to vulnerabilities than human-written code [6, 17]. Pearce et al. [18] evaluated code generated by GitHub Copilot against security scenarios from MITRE’s “CWE Top 25”2 and found that 40% of the generated programs contained vulnerabilities. Maintainability issues, such as modularity and readability, have also been identified [19]. Liu et al. [5] investigated repetition in AI-generated code and revealed prevalent redundant repetitions at both the character and block levels, which degrade readability and efficiency. Kravchuk-Kirilyuk et al. [20] reported that while AI-generated code can mimic superficial modular structures, it tends to violate principles that sustain maintainability, such as encapsulation, and introduces hidden dependencies. Watanabe et al. [21] analyzed agent-generated pull requests and found that 9.9% of the generated methods are eventually deleted during review, placing an unnecessary cognitive burden on human reviewers. Beyond generation-time quality, the maintenance activities surrounding AI-generated code present additional challenges. Haque et al. [22] reported that test code included in initial PRs authored by AI agents is often insufficient and frequently requires additional updates after the initial PR. Ottenhof et al. [23] conducted an empirical study on agentic refactoring and found that while human developers perform diverse structural improvements, refactorings by AI agents are dominated by superficial changes such as adding or modifying annotations. AI agents also struggle with continuous revisions during review. Minh et al. [24] analyzed agent-authored PRs and found that although agents excel at narrow automation, they frequently fail at iterative refinement, leading to “ghosting” (abandonment of PRs) when faced with subjective human feedback. These prior studies primarily evaluate code quality at the point of generation or examine short-term effects immediately after AI tool adoption. However, once AI-generated files are merged into a codebase, human developers must continue to maintain them
Sawada, et al.
over extended periods. This long-term maintenance burden has received little empirical investigation. Our study addresses this gap by tracking the maintenance activities that follow the introduction of AI-generated files and quantifying the extent of human intervention required to sustain them.
3
Data Collection
To analyze the maintenance of AI-generated code, we need to identify files created by AI agents and track their subsequent commit histories. We used the AIDev dataset [9], which contains more than 456,000 pull requests made by five autonomous coding agents (OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code) from approximately 61,000 repositories. All PRs in this dataset were created between December 2024 and July 2025, following the release of agentic coding tools. To ensure the quality of the analyzed projects, we utilized the repository list provided alongside the AIDev dataset, which consists of 2,807 repositories that have already been filtered to include only those with more than 100 stars. From this repository dataset, we identified AI-generated files through the following steps. First, we extracted files created in these PRs by identifying files marked as “added” in the Git file status of each PR. Second, we verified that each file was created by an AI agent by examining the committer name. Although the dataset guarantees that PRs are created by agents, individual commits may be made by human developers. We matched committer names against AI-agent account identifiers (i.e., bots): “claude[bot]”3 for Claude Code, “Cursor Agent”4 for Cursor, “Copilot”5 for GitHub Copilot, and “devin-ai-integration”6 for Devin. We excluded Codex PRs because it does not appear as a commit owner, making it difficult to determine whether a commit was created by an agent or a developer [25]. Using this approach, we collected files generated by agents from 100 repositories. We restricted the sample to 100 repositories to reduce the high computational cost and address API limits. Due to a large number of files from specific projects, we randomly sampled up to ten AI-generated files from each repository. For comparison, we also sampled up to ten human-generated files from the same repository created during the same period. When fewer than ten files were available in either category, we extracted all available files and balanced the dataset by selecting an equal number from the other category. For each selected file, we collected all commits recorded through January 31, 2026. This ensures a maintenance observation period of at least six months for all files because we obtained files created before July 31. In total, we collected 508 AI-generated files and 508 humangenerated files from 100 repositories, along with 1,543 commits modifying AI-generated files and 1,695 commits modifying humangenerated files. We excluded the initial file-creation commits from this analysis, as we focus on subsequent maintenance activities.
3 https://github.com/claude 4 https://github.com/apps/cursor 5 https://docs.github.com/en/enterprise-cloud@latest/copilot/concepts/agents/codin
1 https://anonymous.4open.science/r/AI-Code-Maintainability-83CD/README.md 2 https://cwe.mitre.org/top25/
g-agent/about-coding-agent 6 https://github.com/apps/devin-ai-integration
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study
Filter repos w/ AI-generated files
100 repositories
Random sampling
Extract top 100 repos w/ AI-generated files
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
1,016 files 508 AI-generated files 508 Human-generated files
2,807 repositories in repository Dataset
3,238 commits Extract all commits
1,543 Commits on AI-generated files 1,695 Commits on Human-generated files
RQ1: Maintainance
---------------
+
RQ2: Commit types +
?
?
---------------
feature? or fix?
RQ3: Commit count +
---------------
or Human?
AI?
Figure 1: Overview of our data collection process and research questions
4 Research Questions 𝑅𝑄 1 : To what extent is AI-generated files maintained? Motivation. Prior studies have indicated that AI-generated code tends to be lower quality compared to human-written code [4, 6, 7]. If AI-generated code demands significantly more maintenance, the productivity benefits of using AI agents may be offset by longterm maintenance costs. However, little is known about how AIgenerated code is actually maintained after its initial creation. Approach. In this study, we analyzed the extent of maintenance of AI-generated files from two perspectives: the maintenance frequency and the magnitude of the maintenance. For the maintenance frequency, we compared the number of commits applied to AI-generated files with those applied to humangenerated files to evaluate the maintenance activity associated with AI-generated code. Specifically, we aggregated commits for each file and grouped them into one-month intervals relative to the creation date. We then focused on the first six months, a period common to all files, and visualized the distributions using violin plots. We use only six months because the observation periods after file creation vary across files, making direct comparison beyond this window unreliable. Note that as we described in the Data Collection section, we ensured that at least six months have passed since the creation of all the studied files. Additionally, we did not include the first commit that created the file as maintenance. For the magnitude, we examined whether there were differences in the number of lines of code changed per commit. However, the raw number of changed lines depends on file size, and larger files tend to have more lines modified. We normalize the changes by the size of the files to measure the percentage of changes. Results. Figure 2a shows the number of commits for each file. The red plots represent AI-generated files, while the blue plots represent human-generated files. The horizontal axis indicates the time elapsed since file creation. To mitigate the influence of extreme values, outliers are excluded from the visualizations. AI-generated files exhibited approximately half the commit count of human-generated files during the first month, with maintenance activity gradually declining over the subsequent three months. The distribution shows that maintenance continues at a reduced frequency rather than stopping entirely. These findings imply that
AI-generated code does not impose an immediate, severe burden on developers, and that more maintenance effort is currently devoted to human-generated code. From the fourth month onward, the gap with human-generated files narrows. The commit count stays small but non-zero, indicating that maintenance continues over the long term to a similar extent as for human-generated files. In summary, although the need for maintenance immediately after generation is lower for AI-generated files, ongoing maintenance is still required. Next, Figure 2b shows the distribution of the magnitude of maintenance for each month after file creation. The results reveal that human-generated files exhibit a larger magnitude compared to AIgenerated files. The smaller magnitude of changes in AI-generated files implies that maintenance is often limited to minor changes or slight modifications. Furthermore, a decreasing trend was observed in the modification ratio of AI-generated files. This indicates that the volume of maintenance decreases over time. In contrast, the higher ratio in human-generated files likely reflects more fundamental structural changes and active refactoring. Ultimately, AI-generated code does not impose an immediate, severe maintenance burden on developers. Moreover, although maintenance is required in the long term, its magnitude is smaller and less burdensome compared to human-generated files. Answer to RQ1. AI-generated files require less maintenance than humangenerated files in both frequency and magnitude. This suggests that AI-generated code does not impose a significant maintenance burden on developers.
𝑅𝑄 2 : What types of maintenance activities are applied to AI-generated files? Motivation. Watanabe et al. [26] show that agents perform bug fixing more frequently than feature addition. However, their study focuses only on commits within PRs generated by AI agents. It remains unclear what types of changes are made in subsequent maintenance commits after file creation. In this RQ, we examine whether modifications to AI-generated files are primarily driven by constructive activities (e.g., feature additions) or by corrective actions (e.g., bug fixes or refactoring).
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom