Context Rot in AI-Assisted Software Development: Repurposing Documentation Consistency for AI Configuration Artifacts Christoph Treude
Sebastian Baltes
Singapore Management University Singapore, Singapore [email protected]
Heidelberg University Heidelberg, Germany [email protected]
arXiv:2606.09090v1 [cs.SE] 8 Jun 2026
Abstract Developers increasingly provide AI coding assistants with persistent context through configuration files such as CLAUDE.md, AGENTS.md, and .cursorrules. These files describe code elements, architecture, and development conventions, forming the context that guides AI tool behavior across sessions. As software evolves, this context can become stale, a phenomenon we call context rot. While AI configuration artifacts are new, the underlying consistency problem connects to decades of software documentation research. Researchers have built tools to check consistency between documentation and code, spanning README files, code comments, API documentation, architecture descriptions, and installation instructions. We argue that this existing toolbox is an immediate starting point for detecting context rot, and we present a research roadmap mapping documentation consistency approaches to corresponding problems in this new setting. As preliminary evidence, applying an existing README/wiki consistency checker to a statistically representative sample of 356 repositories identifies stale code element references in 23.0% of repositories, showing that traditional documentation consistency tools can already surface context rot.
Keywords AI coding assistants, documentation consistency, configuration artifacts, context rot
1
Introduction
AI coding assistants are now a routine part of software development workflows. Tools such as Claude Code, GitHub Copilot, Cursor, and Gemini CLI generate code, answer architectural questions, and work through unfamiliar codebases on behalf of developers. Central to their effectiveness is persistent, project-specific context, supplied by configuration files that describe the codebase to the model before any interaction begins. These configuration files take many forms, and most tools support multiple formats [5]. Anthropic’s Claude Code reads CLAUDE. md files placed at the repository root and in subdirectories. OpenAI’s Codex reads AGENTS.md and AGENTS.override.md. GitHub Copilot reads .github/copilot-instructions.md and instructions/*.md files, and additionally recognizes CLAUDE.md and AGENTS.md. Cursor reads AGENTS.md and formerly .cursorrules, which has since been deprecated in favor of AGENTS.md. Google’s Gemini CLI reads GEMINI.md. Despite their different naming conventions, the function is the same. Each gives the model project-specific knowledge that persists across sessions. These files describe many aspects of a project: conventions, contribution guidelines, architecture, build commands, and testing practices [10]. A CLAUDE.md might state that authentication logic
lives in src/auth/middleware.py, that the primary API client is HttpClient, or that the project uses MAX_RETRIES as a global constant. Beyond specific code references, these files also describe architectural patterns, required tools, dependency versions, and workflow expectations. Lulla et al. show that the presence of AGENTS.md files is associated with lower agent runtime and reduced token consumption [9], suggesting that up-to-date configuration contributes to more efficient AI assistance. When stale, such descriptions can degrade AI assistance without any visible error. The model references a function that no longer exists, generates code that imports a deleted module, or enforces conventions the team abandoned months ago. We call this context rot 1 : the gradual divergence between what a configuration file says about a codebase, its tools, its architecture, or its conventions, and what actually holds. While AI configuration artifacts are new, the underlying problem is not. The software engineering community has studied documentation consistency for decades, building tools to detect precisely this kind of mismatch: between code and README files [15], between method implementations and their comments [11–13], between APIs and their documentation [17, 18], between architectural descriptions and deployed systems [1, 8], and between installation instructions and build configurations [3, 6]. These tools predate AI coding assistants, but the consistency problems are structurally similar: take a claim made in a text artifact about the state of software, check it against the actual software, and flag divergences. This paper makes three contributions. First, we introduce context rot as the divergence between AI configuration files and the codebases, tools, architectures, and workflows they describe. Second, we provide preliminary empirical evidence for the feasibility of detecting context rot by applying an existing README/wiki consistency checker to AI configuration files without modification, finding stale code element references in 23.0% of 356 repositories, sampled to be representative of the eligible dataset at a 95% confidence level with a 5% margin of error. Third, we present a research roadmap mapping five classes of documentation consistency technique onto open problems in AI configuration artifacts. Specifically, we instantiate one part of a broader research roadmap: referential rot, where a configuration file refers to code elements that no longer exist. We apply DOCER [15], an existing consistency checker for README files and wikis, to AI configuration files without adapting its extraction rules or staleness definition.
1 The term context rot is currently used mainly for degradation within a model’s input
context window [7]. We extend it to AI-specific, repository-versioned artifacts that are sent to models as project context or configure how AI coding tools operate. Their usefulness can degrade as they become stale relative to the evolving codebase.
Christoph Treude and Sebastian Baltes
Our aim is not to advance DOCER itself, but to test whether a traditional documentation consistency tool can identify stale code references in AI configuration artifacts. The remainder of this paper is organized as follows. Section 2 provides background on AI configuration artifacts and the documentation consistency literature. Section 3 describes the preliminary study and results. Section 4 presents the research roadmap. Section 5 concludes.
2
Background
We situate our work in two bodies of prior art: the emerging practice of AI configuration artifacts, and decades of research on software documentation consistency.
2.1
AI Configuration Artifacts
AI coding tools offer multiple configuration mechanisms, including context files, skills, subagents, and hooks [5]. Context rot can arise in any of these artifact types. Our preliminary study focuses on context files — versioned Markdown files that developers maintain to provide persistent, project-specific context to an AI coding assistant — as the most widely studied artifact type. Unlike interactive prompts, these files live in the repository alongside source code and are read automatically at the start of each session. Their content ranges from high-level architecture descriptions to precise references to specific functions, files, and constants. Galster et al. collected a dataset of 9,470 context files from 4,463 GitHub repositories, spanning all major AI coding tool formats [4]. The dataset reveals that these files are substantive: they are not mere pointer files but contain detailed descriptions that AI tools consume directly. The most common types are AGENTS.md (42.7% of files), CLAUDE.md (30.3%), and copilot-instructions.md (13.7%). The content and purpose of such files have been examined from several angles: Mohsenimofidi et al. [10] study context engineering practices for AI agents in open-source software, Galster et al. [5] conduct an exploratory study of configuration practices across tools, and Lulla et al. [9] show empirically that AGENTS.md files improve AI coding agent efficiency, making the accuracy of these files relevant.
2.2
Documentation Consistency Research
Software documentation consistency has been an active research area since at least the early 2000s. The central challenge is that documentation and code evolve at different rates and through different processes. Code changes are continuously exercised by compilers, automated tests, and continuous integration, so drift is caught quickly. Documentation changes are not, so documentation is updated manually, opportunistically, and often incompletely. Researchers have addressed consistency across multiple documentation artifact types, each targeting a distinct class of divergence. DOCER [15] targets README files and GitHub wikis, addressing stale code element references by extracting identifiers using regular expressions and checking their presence in the source code at successive repository snapshots. A reference is stale if the element was present when the documentation was first committed but has since been removed. Code comment consistency checkers address inconsistencies between code and comments, detecting
when a method’s implementation diverges from its Javadoc or inline comment [11–13]. API documentation checkers address API contract violations, comparing parameter names, types, and descriptions in published documentation against actual method signatures [17, 18]. Architecture consistency tools address architectural drift, checking whether implementation, deployment, or recovered architecture matches stated architectural descriptions [1, 8]. Installation and dependency consistency checkers address stale environment assumptions, verifying that setup instructions match actual build configurations and dependency manifests [3, 6]. AI configuration artifacts can exhibit all of these classes of divergence: they describe not only specific code elements but also architectural patterns, API tool parameters, dependency versions, and development conventions. The common thread across these consistency problems is that a claim made in a text artifact about the state of software can be operationalized as a query against the actual software, then flagged if it diverges.
3
Preliminary Study
In this preliminary study, we focus on one specific form of context rot: referential rot, where a configuration file refers to functions, classes, constants, scripts, or file paths that no longer exist in the repository.
3.1
Dataset
We use the configuration files from the dataset of Galster et al. [4]. After excluding empty files and pointer files (files that merely reference another file), and retaining only files with a recorded first commit SHA (required for the two-snapshot comparison), 8,213 files from 4,420 repositories remain as candidates. We randomly sample 356 repositories (random seed 42), retaining all configuration files present in each repository. This means that a repository containing both a CLAUDE.md and an AGENTS.md contributes both files to the analysis. The sample size was chosen to be statistically representative of the 4,420 eligible repositories at a 95% confidence level and a 5% margin of error; we therefore interpret repository level estimates as estimates for the eligible dataset rather than only for the sampled repositories. The sample includes all major configuration file types present in the dataset.
3.2
Applying DOCER to AI Configuration Files
We apply DOCER’s detection logic as follows for each configuration file: (1) Clone. We clone the repository locally. (2) Extract. We run DOCER’s regular expressions against the current (HEAD) version of the configuration file using git grep -howIP -f regex_list.txt, extracting candidate code elements. (3) Verify at first commit. We search for each candidate in the source code at first_commit_sha (when the configuration file was first committed) using git grep -howFI, excluding README files and all AI configuration file types from the search scope. (4) Verify at HEAD. We repeat the search at the current HEAD.
Context Rot in AI-Assisted Software Development
Table 1: Stale code element references detected across 356 randomly sampled repositories.
Repositories analyzed with ≥1 stale reference Configuration files analyzed Candidate elements extracted Verified references (found at first commit) still valid at HEAD stale (present then, absent now)
Count
%
356 82 612 29,454 18,048 17,818 230
N/A 23.0 N/A N/A 100 98.73 1.27
Table 2: Verified and stale references by configuration file type. Copilot instructions includes copilot-instructions.md and other *.instructions.md files. Config file type
Files
Verified
Stale
Stale %
CLAUDE.md AGENTS.md Copilot instructions GEMINI.md .cursorrules Other
147 234 211 6 9 5
5,423 6,762 5,436 133 127 167
77 70 77 1 0 5
1.42 1.04 1.42 0.75 0.00 2.99
Total
612
18,048
230
1.27
(5) Classify. Elements present at first commit but absent at HEAD are stale. Elements present at HEAD are valid. Elements absent from both snapshots are noise and discarded. For this short paper, we deliberately do not tune DOCER for AI configuration files. Its regular expressions, verification strategy, and staleness definition are left unchanged. This allows us to ask what an existing documentation consistency checker can already reveal about AI configuration artifacts. Because we preserve DOCER’s traditional two-snapshot design, our measurement captures only references that were valid when the configuration file was first committed and later disappeared from the source. References introduced in subsequent edits to the configuration file fall outside this scope; we return to this and the opposing false-positive bias when interpreting the results.
3.3
Results
Table 1 summarizes findings across 356 repositories and 612 configuration files. DOCER’s regular expressions extracted 29,454 candidate code elements in total. DOCER detects 230 stale code element references across 82 repositories, or 23.0% of those analyzed (95% CI 18.8–27.2%). Among the 82 affected repositories, the median number of stale references was 1 and the maximum was 20. Table 2 breaks them down by configuration file type. Stale references appear across all major types, with per-reference stale rates between 1.0% and 1.4%; these differences are small, and we do not test them for significance. To assess how well DOCER detects genuine context rot in this new setting, one author manually inspected a random sample of 50
Table 3: Manual inspection of 50 randomly sampled elements classified as stale. Classification Genuine referential rot False positive Ambiguous
Count
%
32 12 6
64 24 12
elements classified as stale. As Table 3 shows, 32 (64%) were genuine referential rot and the remainder false positives or ambiguous; five representative genuine cases appear in Table 4. The false positives arise from DOCER’s broad regular expressions matching common English words or generic tokens that coincidentally appeared in the source at some point. They are informative for the transfer argument: they mark which parts of traditional documentation consistency tooling work directly here and which would benefit from artifact-specific tuning, motivating RQ2 (Section 4). As a threat to validity, the inspection was performed by a single annotator; inter-rater agreement was not assessed. More broadly, the 23.0% figure carries two opposing biases. DOCER’s two-snapshot criterion captures only references that were valid at the configuration file’s first commit and misses any introduced by later edits, which deflates the estimate. Its broad regular expressions also match non-code tokens, so 36% of elements classified as stale were false positives or ambiguous on inspection, which inflates it. We therefore read 23.0% as a feasibility signal rather than a precise prevalence; our sample also covers only public GitHub repositories that contain AI configuration files. For practitioners, this check is available today [14]. The lightweight two-snapshot git grep procedure used here needs no new tooling and runs in continuous integration, flagging configuration file references that have vanished from the source. The drift it surfaces is concrete, such as a renamed function, a deleted script, or a dropped dependency (Table 4). Treating configuration files as code, and reviewing them alongside refactors that rename or delete elements, prevents much of this rot at the source.
4
Research Roadmap
Our results show that referential rot appears in the sampled repositories and can be detected with existing tooling. The preliminary study addresses only one part of a larger problem. We identify four further transfer opportunities where prior work in documentation consistency maps onto open problems in AI configuration artifacts, summarized in Table 5. Each opportunity pairs an established technique with an open problem (Table 5): code comment consistency checkers [11–13], API documentation checkers [17, 18], architecture consistency tools [1, 8], and installation and dependency checkers [3, 6]. This list is not exhaustive; AI configuration files also contain natural language descriptions of team conventions that are hard to operationalize as code queries. But it marks concrete starting points where prior work already exists. We now state the research questions this agenda raises.
Christoph Treude and Sebastian Baltes
Table 4: Five representative stale code element references detected in AI configuration files. Each element was present in the repository source at the configuration file’s first commit but absent at HEAD. Repository
Config file
Stale element
Type
Observed drift
microsoft/pr-metrics
copilot-instructions.md
tsyringe
library
Absent from source and build config at HEAD
dagu-org/dagu
ui/CLAUDE.md
runDAGs
function
Function absent at HEAD (apparently renamed during UI rewrite)
smartystreets/ smartystreetspython-sdk
CLAUDE.md
send_risk_lookup
function
Method absent from SDK at HEAD
rolldown/rolldown
AGENTS.md
packages/rolldown/ src/binding.d.ts
file path
File path absent at HEAD
EricLBuehler/ mistral.rs
AGENTS.md
scripts/ convert_awq_ marlin.py
script
Script absent from repository at HEAD
Table 5: Documentation consistency techniques and their analogues in AI configuration artifacts. The first row is the present study; the remaining rows are open transfer opportunities (RQ2). “Existing tools” names representative prior work; full citations appear in Section 2 and below. Consistency technique
Existing tools
AI configuration analogue
Transfer status
Referential (code/README)
DOCER
Code elements (functions, files, constants) that no longer exist
Direct; shown here
Code comment
iComment, @tComment, deep JIT detection
Behavioral instructions that reference renamed or deleted elements
Open (RQ2)
API documentation
Directive defect and API documentation checkers
MCP tool descriptions that diverge from their implementation
Open (RQ2)
Architecture
Traceability link recovery
Architectural claims that drift as the system evolves
Open (RQ2)
Installation / dependency
FindICI; installation instruction adaptation
Runtime and dependency versions that no longer hold
Open (RQ2)
RQ1: What kinds of context rot occur in AI configuration artifacts? Context rot extends well beyond stale code element references. AI configuration files also contain architectural claims, tool use guidance, dependency and runtime assumptions, and behavioral conventions. Establishing the full taxonomy in practice requires large-scale, longitudinal analysis of the Galster et al. corpus [4], coding how configuration files diverge from their repositories over time. Our preliminary study addresses one subcategory, referential rot; the full taxonomy motivates the transfer opportunities in Table 5. RQ2: Which traditional documentation consistency techniques transfer to AI configuration artifacts without modification? DOCER transfers directly for referential rot; each remaining opportunity in Table 5 requires evaluation. Do code comment checkers flag stale behavioral instructions? Do API documentation checkers detect drift between MCP tool descriptions and their implementations? Each transfer attempt can follow the recipe used here: apply the
unmodified tool to a corpus of the relevant configuration files, measure how often it fires, and manually validate precision. The result is an empirical estimate of transferability and evidence about which techniques work directly, which need tuning, and which require new approaches. RQ3: Which forms of context rot affect AI assistant behavior? A configuration file can be stale without affecting the AI tool’s output. Conversely, if the AI acts on a stale reference and generates code that imports a deleted module, the cost is real. Measuring this calls for controlled experiments comparing AI output quality across repositories with and without injected or naturally occurring context rot, extending the design of Lulla et al. [9], who compare agents run with and without an AGENTS.md, to injected stale references. Such experiments may reveal that some categories (file paths, function names, library names) matter more than others. RQ4: How can detected context rot be repaired or mitigated? Deterministic detection and language model repair are complementary.
Context Rot in AI-Assisted Software Development
A tool like DOCER identifies the specific stale element; a language model can then inspect the git history or the current codebase to suggest the updated reference. This hybrid sidesteps the precision concerns of LLM-only detection while keeping the flexibility needed for natural language repair, and it can be evaluated on whether the suggested replacement matches the element’s actual rename in the history. Mitigation can also be preventive, alerting developers when a configuration file is not updated alongside related code changes.
5
Conclusion
AI configuration files guide coding assistants through the codebases, tools, architectures, and conventions of the projects they assist with. As software evolves, these descriptions go stale, a phenomenon we call context rot. Rather than building entirely new detection techniques, we argue that existing documentation consistency tools are a ready starting point: checkers for README files, code comments, API documentation, architecture descriptions, and installation instructions all address forms of divergence with direct analogues in AI configuration artifacts. As preliminary evidence, we applied DOCER, a consistency checker built for README files and wikis, to AI configuration files without modification, finding stale code element references in 23.0% of 356 repositories. We identify four further transfer opportunities (Table 5): behavioral instructions, MCP tool descriptions, architectural claims, and dependency references. As AI coding assistants become standard development infrastructure, keeping their configuration consistent with the evolving codebase is a software quality problem the community already has the tools to address.
Data Availability The online appendix [16] provides the analysis script, full perelement results, the repository versions analyzed, DOCER’s regular expressions, the list of 230 stale references, and the 50 manual annotations. Galster et al. describe the input dataset of AI configuration files [4]; the dataset is archived on Zenodo [2].
References [1] Nour Ali, Sean Baker, Ross O’Crowley, Sebastian Herold, and Jim Buckley. 2018. Architecture Consistency: State of the Practice, Challenges and Requirements. Empirical Software Engineering 23, 1 (2018), 224–258. [2] Sebastian Baltes, Seyedmoein Mohsenimofidi, Levi Böhme, Jai Lulla, Muhammad Auwal Abubakar, Christoph Treude, and Matthias Galster. 2026. A Dataset of Agentic AI Coding Tool Configurations. doi:10.5281/zenodo.19375880 [3] Nemania Borovits, Indika Kumara, Dario Di Nucci, Parvathy Krishnan, Stefano Dalla Palma, Fabio Palomba, Damian A Tamburri, and Willem-Jan van den Heuvel. 2022. FindICI: Using Machine Learning to Detect Linguistic Inconsistencies Between Code and Natural Language Descriptions in Infrastructure-as-Code.
Empirical Software Engineering 27, 7 (2022), 178. [4] Matthias Galster, Seyedmoein Mohsenimofidi, Levi Böhme, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, and Sebastian Baltes. 2026. A Dataset of Agentic AI Coding Tool Configurations. In Proceedings of the 3rd ACM International Conference on AI-Powered Software (Montreal, Canada) (AIware ’26). ACM, New York, NY, USA. To appear. [5] Matthias Galster, Seyedmoein Mohsenimofidi, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, and Sebastian Baltes. 2026. Configuring Agentic AI Coding Tools: An Exploratory Study. In Proceedings of the 3rd ACM International Conference on AI-Powered Software (Montreal, Canada) (AIware ’26). ACM, New York, NY, USA. To appear. [6] Haoyu Gao, Christoph Treude, and Mansooreh Zahedi. 2025. Adapting Installation Instructions in Rapidly Evolving Software Ecosystems. IEEE Transactions on Software Engineering 51, 4 (2025), 1334–1357. doi:10.1109/TSE.2025.3552614 [7] Kelly Hong, Anton Troynikov, and Jeff Huber. 2025. Context Rot: How Increasing Input Tokens Impacts LLM Performance. Technical Report. Chroma. Technical report. https://research.trychroma.com/context-rot. [8] Jan Keim, Sophie Corallo, Dominik Fuchß, and Anne Koziolek. 2023. Detecting Inconsistencies in Software Architecture Documentation Using Traceability Link Recovery. In 2023 IEEE 20th International Conference on Software Architecture (ICSA) (L’Aquila, Italy). IEEE, Piscataway, NJ, USA, 141–152. doi:10.1109/ICSA56044.2023.00021 [9] Jai Lal Lulla, Seyedmoein Mohsenimofidi, Matthias Galster, Jie M. Zhang, Sebastian Baltes, and Christoph Treude. 2026. On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents. In Proceedings of the Journal Ahead Workshop (Rio de Janeiro, Brazil) (JAWs ’26). ACM, New York, NY, USA. To appear. [10] Seyedmoein Mohsenimofidi, Matthias Galster, Christoph Treude, and Sebastian Baltes. 2026. Context Engineering for AI Agents in Open-Source Software. In Proceedings of the 23rd International Conference on Mining Software Repositories (Rio de Janeiro, Brazil) (MSR ’26). ACM, New York, NY, USA. To appear. [11] Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J Mooney. 2021. Deep Just-In-Time Inconsistency Detection Between Comments and Source Code. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. AAAI Press, Palo Alto, California, USA, 427–435. Issue 1. doi:10.1609/aaai.v35i1. 16119 [12] Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. iComment: Bugs or Bad Comments?. In Proceedings of the Twenty-First ACM SIGOPS Symposium on Operating Systems Principles. ACM, New York, NY, USA, 145–158. [13] Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T Leavens. 2012. @tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation (Montreal, QC, Canada). IEEE, Piscataway, NJ, 260–269. doi:10.1109/ICST.2012. 106 [14] Wen Siang Tan, Markus Wagner, and Christoph Treude. 2023. Wait, wasn’t that code here before? Detecting Outdated Software Documentation. In Proceedings of the International Conference on Software Maintenance and Evolution (Bogotá, Colombia). IEEE, Los Alamitos, CA, 553–557. doi:10.1109/ICSME58846.2023. 00071 [15] Wen Siang Tan, Markus Wagner, and Christoph Treude. 2024. Detecting Outdated Code Element References in Software Repository Documentation. Empirical Software Engineering 29, 1 (2024), 5. [16] Christoph Treude and Sebastian Baltes. 2026. Online Appendix: Context Rot in AI-Assisted Software Development. doi:10.5281/zenodo.20588740 [17] Gias Uddin and Martin P Robillard. 2015. How API Documentation Fails. IEEE Software 32, 4 (2015), 68–75. [18] Yu Zhou, Ruihang Gu, Taolue Chen, Zhiqiu Huang, Sebastiano Panichella, and Harald Gall. 2017. Analyzing APIs Documentation and Code to Detect Directive Defects. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) (Buenos Aires, Argentina). IEEE, Piscataway, NJ, USA, 27–37. doi:10.1109/ ICSE.2017.11