Do They Do as They Are Told? Investigating Coding Agents' Adherence to Explicit Instructions in Context Files

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

arXiv:2611.13709v1 [cs.SE] 10 Nov 2026

GIANMARIO VORIA, University of Salerno, Italy YUTARO KASHIWA, Nara Institute of Science and Technology, Japan GEMMA CATOLINO, University of Salerno, Italy FABIO PALOMBA, University of Salerno, Italy Developers increasingly rely on agent context files (ACFs), persistent, project-scoped artifacts such as AGENTS.md, to steer the behavior of autonomous coding agents without renegotiating expectations at every interaction. Yet this governance mechanism rests on an untested assumption: that instructions written into an ACF are actually carried out, rather than ignored or loosely followed. Existing studies describe what ACFs contain, but not whether individual instructions are followed, leaving open which guidance can be trusted. We address this gap through an empirical study of real-world ACF snapshots and their linked agent-generated commits: characterizing the explicit, actionable instructions ACFs encode, operationalizing them into executable measurement scripts, and using these scripts to assess instruction-level compliance. We find that most observable instructions can be operationalized and that agents follow ACF guidance more often than not, but adherence splits sharply between instructions followed consistently and those never respected, with differences across instruction categories and coding agents. This split shows where ACF guidance works and where it does not, pointing to instruction types that may need rewriting and to agents that may need enforcement beyond the ACF. These findings lay the groundwork for systematic approaches to specifying, verifying, and enforcing project-specific instructions in autonomous software development. Additional Key Words and Phrases: Autonomous Coding Agents; Empirical Software Engineering; Agent README; Automated Software Engineering. ACM Reference Format: Gianmario Voria, Yutaro Kashiwa, Gemma Catolino, and Fabio Palomba. 2026. Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files. ACM Trans. Softw. Eng. Methodol. X, X, Article X (August 2026), 32 pages. https://doi.org/10.1145/XXXXXXX.XXXXXXX

1

Introduction

The emergence of autonomous coding agents is progressively transforming how software engineering (SE) activities are organized and carried out [13]. When provided with appropriate instructions and contextual guidance, these agents can complete sophisticated development tasks with relatively limited human oversight [20]. Consequently, contemporary software development is increasingly moving toward a collaborative human–agent setting, where developers primarily specify goals, requirements, and constraints, while autonomous agents are responsible for translating them into executable development actions [16]. Authors’ Contact Information: Gianmario Voria, gvoria@unisa.it, University of Salerno, Fisciano, Italy; Yutaro Kashiwa, yutaro.kashiwa@naist.jp, Nara Institute of Science and Technology, Nara, Japan; Gemma Catolino, gcatolino@unisa.it, University of Salerno, Fisciano, Italy; Fabio Palomba, fpalomba@unisa.it, University of Salerno, Fisciano, Italy. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1557-7392/2026/8-ARTX https://doi.org/10.1145/XXXXXXX.XXXXXXX ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:2

Voria et al.

In this context, a new class of software artifact has emerged: agent context files (ACFs). These artifacts (e.g., AGENTS.md, CLAUDE.md, or copilot-instructions.md) act as agent-oriented counterparts of traditional README files, explicitly defining how an AI agent should behave within a given software project [4, 5]. ACFs encode project-specific knowledge, including architectural overviews, build and test procedures, coding conventions, and operational rules governing tool usage and external service interaction [5]. Modern AI-assisted development tools, including Claude Code, OpenAI Codex, and GitHub Copilot, are designed to automatically load these files at the beginning of a session and use them as a stable reference when planning and executing code changes [4]. Consequently, ACFs play a central role in shaping how agents interpret a codebase, adhere to development practices, and perform non-trivial software engineering tasks. Despite the growing adoption of ACFs, little empirical evidence exists on whether the instructions they encode are actually reflected in practice. The presence of guidance in a context file does not guarantee that an agent will follow it, and recent studies of agent-generated tests and of testing guidance in ACFs both name this question as future work [2, 14]. The question matters because ACFs are expected to act as persistent, project-specific sources of guidance [23] that let developers steer agent behavior in a controlled and predictable way. This expectation breaks down if instructions are systematically ignored, partially applied, or inconsistently interpreted, and recent evidence on agents’ compliance with documented repository rules suggests that such failures do occur [37]. Answering it also matters beyond confirming that ACFs work, because an instruction-level account of compliance has implications for two related problems in agent governance. Knowing which instructions are consistently followed and which are essentially never respected shows how effective specific forms of guidance are, and points to instruction types that may need reformulation or reinforcement beyond the context file. Differences in compliance across coding agents show how agents differ in interpreting project-level constraints, and expose instruction classes for which particular agents are less reliable. This gives a basis for agent-specific guidance and for verification mechanisms that complement natural-language instructions. An empirical account of instructionlevel compliance thus yields design and engineering implications, not only a check on whether ACFs work. A fundamental challenge in addressing this problem lies in the nature of the instructions themselves. Existing studies describe ACF contents through high-level categories (e.g., testing, implementation, or quality) that aggregate heterogeneous forms of guidance with different meanings and implications [4]: a testing-related instruction, for instance, may mean running an existing test suite, adopting a specific framework, or generating new test cases. Such variation makes it difficult to reason about what it means for an agent to “follow” a given instruction type, or to identify observable evidence of such behavior in software artifacts. Hence, advancing our understanding of ACFs requires moving beyond high-level categorizations toward a more precise characterization of the instructions they encode—focusing on guidance that is explicit and actionable, directly stated in the ACF and prescribing concrete development behavior, and establishing whether each such instruction can be operationalized through measurable software artifacts. We address this gap by examining the relationship between explicit, actionable instructions in ACFs and the observable behavior of autonomous coding agents in real-world development. Rather than assuming the presence of guidance implies compliance, or treating compliance as a single aggregate figure, we structure the investigation as a three-phase empirical pipeline: characterizing the instructions ACFs encode, establishing whether they are concretely measurable through software artifacts, and assessing how closely agents follow them, at a granularity fine enough to attribute compliance variation to specific instruction types and agents rather than to ACFs overall.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:3

◎ Main Objective. This study aims to empirically investigate whether instructions documented in agent context files are reflected in agent-generated software artifacts. Specifically, we aim to (1) characterize the explicit and actionable instructions encoded in real-world ACFs, (2) establish whether such instructions can be operationalized through executable measurement scripts derived from their observable behavioral descriptors, and (3) assess the extent to which these instructions are manifested in downstream changes produced by autonomous agents in real-world development settings. We conduct a three-phase empirical study grounded in a dataset of 621 unique ACF snapshots linked to 6,541 agent-generated commits across 268 repositories. First, we combine manual calibration with LLM-assisted classification to extract and characterize explicit, actionable instructions from real-world ACFs, validating the results on a statistically representative sample. In the second phase, we generate executable measurement scripts via test-driven code generation: a code LLM independently samples candidate scripts and test fixtures from the instruction’s behavioral descriptor, and a script is accepted only if it passes a minimum number of fixtures and is corroborated by at least one independently generated script. In the third phase, we apply the validated scripts to the reconstructed commit dataset and analyze the resulting compliance observations through descriptive statistics and comparative tests across instruction categories and agents. Throughout, compliance denotes artifact-level evidence, produced by a script that operationalizes the instruction’s own specified behavioral descriptor, that an applicable commit satisfies what the instruction prescribes. Applying this pipeline to the corpus, we find that most explicit, actionable, and observable instructions can be operationalized into executable measurement scripts and that, where compliance can be measured, agents follow ACF guidance more often than not—though adherence is far from uniform, splitting sharply between instructions followed consistently and those essentially never respected, with the gap shaped by both instruction category and coding agent. We use this split throughout the paper as evidence in its own right, indicating where current ACF content already functions reliably, where it does not, and which instruction classes and agents most warrant reinforcement beyond natural-language guidance alone. To sum up, this paper offers the following main contributions: (1) A three-criteria characterization of the explicit, actionable, and observable instructions encoded in real-world ACFs, distinguishing genuinely verifiable guidance from high-level content categories used in prior work. (2) An empirical evaluation of whether such instructions can be operationalized into executable measurement scripts and whether they are actually reflected in agent-generated commits, showing that compliance, while common, is neither guaranteed nor uniform across instruction types and agents. (3) A discussion of the implications of these instruction-level compliance patterns for ACF authoring practices and for the engineering of agentic systems, identifying instruction types that may warrant reformulation and the classes of guidance for which verification beyond natural language appears most warranted. (4) A publicly available replication package[33], including the extraction and classification pipeline, the generated measurement scripts, and the compliance dataset, to support future work on verifying and enforcing agent adherence to project-specific guidance.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:4

2

Voria et al.

Background and Related Work

Software engineering practice is shifting from single-turn code completion toward autonomous coding agents that decompose high-level goals into multi-step plans and execute them with limited human intervention [13]. Wang et al. [35] characterize such systems through a perception–memory– action framework, in which agent behavior is governed in part by an explicit form of procedural memory encoded in the agent’s configuration. This shift has broadened the range of activities agents are expected to perform, including automated program repair [3], autonomous testing [10], and agentic refactoring [15], moving agents progressively closer to end-to-end ownership of development tasks. Greater autonomy, however, introduces governance challenges. Empirical studies report integration defects and security risks specific to agentic settings [29], and consistently identify the absence of authoritative, project-specific knowledge as a primary failure mode [35]. This failure mode is not merely theoretical: agents contributing to real-world pull requests still require substantial human revision because they fail to follow project-specific conventions [36], indicating that capability alone does not guarantee conformance to project-level expectations. Maintaining human governance over autonomous agents has consequently emerged as a practical concern for professional developers. Huang et al. [17] find that experienced developers deliberately resist delegating full control to agents, relying instead on explicit strategies—detailed prompts, staged plans, and continuous supervision—to retain authority over design and implementation decisions. Their findings show that governance is not a theoretical abstraction: practitioners actively constrain agent autonomy because unconstrained agents produce outputs that violate quality, safety, and design requirements. This motivates the need for mechanisms through which developers can encode such constraints persistently and systematically, rather than re-asserting them at every interaction. Agent Context Files as Governance Artifacts. In response to this need, developers have adopted persistent, project-scoped artifacts to communicate intent to agents [19, 30]. Agent context files (ACFs)—named CLAUDE.md, AGENTS.md, .cursor/rules, or copilot-instructions.md depending on the tool—capture guidelines, constraints, and conventions that remain active across sessions [4, 5]. Modern agentic tools load these files automatically at the start of a session and treat them as a stable reference throughout planning and execution [4]. Framed in terms of the perception– memory–action view introduced above, ACFs represent an explicit, developer-authored form of procedural memory: a mechanism through which project-specific knowledge is injected into the agent’s context rather than inferred from the codebase alone. Chatlatanagulchai et al. [4, 5] conducted the first large-scale empirical study of ACFs, analyzing 2,300 files across 1,925 repositories. They find that ACF content skews heavily toward functional concerns—build and run commands (62.3%), implementation details (69.9%), and architecture (67.7%)—while non-functional requirements such as security (14.5%) and performance (14.5%) are rarely specified. Mohsenimofidi et al. [23] report substantial variation in how developers express instructions across projects, finding no established structural standard across the 466 projects examined. Galster et al. [11] identify eight distinct configuration mechanisms ranging from static context files to executable subagents. Ardic et al. [2] study testing guidance in the ACFs of popular repositories with agent-authored pull requests. In a sample of 300 repositories, 249 include test commands, while guidance on test strategy, mocking, coverage, or naming is much less common. Beyond characterizing content, recent work has begun to quantify the operational impact of ACFs: Lulla et al. [21] show that the presence of an AGENTS.md file is associated with a 28.6% reduction in agent runtime and a 16.6% reduction in token consumption. Together, these studies establish ACFs as a widespread and influential class of software artifacts and confirm that their content measurably shapes agent behavior. Existing characterizations, ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:5

however, remain descriptive: they document what ACFs contain, typically through high-level content categories, rather than the extent to which the specific instructions they encode are actually followed. Instruction Following and the Compliance Gap. Whether the presence of an instruction in an ACF translates into compliant agent behavior remains largely unexamined. Adjacent evidence suggests it should not be assumed. Studies of AI-generated code quality show that functional correctness does not predict adherence to convention: code that passes all tests can still carry latent code smells and violate project-specific expectations [27], and even where agentic refactoring yields measurable structural improvements [15], these gains are evaluated against generic quality metrics rather than against the developer-authored guidance the agent was given. Instruction following has also been studied as a general LLM capability, including for code. CodeIF-Bench [34] measures whether generated code satisfies constraints stated in a single prompt, and AgentIF [26] extends this to long, multi-constraint system prompts in agentic settings. This work shows that instruction adherence is measurable and imperfect even for state-of-the-art LLMs. It evaluates instructions given within a single interaction, however, not a persistent, project-scoped artifact, and it does not look at the artifacts produced in real development. Studies of agents in real repositories come closer. Watanabe et al. [36] find that agent pull requests need substantial revision to meet project conventions, and Ehsani et al. [8] characterize at scale why agentic pull requests fail, without tying failures to the instructions an ACF documents. Hora and Robbes [14] show that agents add mocks to tests more often than non-agents (36% versus 26% of test commits) and call for mocking guidance in agent configuration files, but they only check a few repositories anecdotally for whether such guidance is followed. Gloaguen et al. [12] evaluate whether ACFs help agents resolve issues, and find slightly lower task success for LLM-generated context files, slightly higher for developer-written ones, and about 20% higher cost. Their trace analysis also shows that tools named in a context file are used far more often than tools that are not named. This measures tool usage in controlled issue-resolution tasks. We instead check, for individual instructions, whether their content shows up in the artifacts of real agent-generated commits. Closest to our setting, Yang et al. [37] directly measure coding agents’ compliance with documented rules. Their RepoComplianceBench covers repository-level AI-contribution policies (bans, disclosure requirements, verification gates) and uses a controlled set of 106 issues across four agents. It does not cover the broader development guidance that ACFs encode, nor compliance observed in real, mined commits. Existing work therefore establishes two main aspects. First, ACFs are widely adopted as behavioral governance artifacts. Second, agents do not reliably adhere to project conventions even without explicit, structured guidance. In this context, what remains missing is a systematic, artifact-grounded assessment of whether the specific instructions documented in an ACF are reflected in the commits an agent subsequently produces—and, more fundamentally, which of these instructions can be verified from produced artifacts at all. Novelty and Contribution. Prior work on ACFs is limited in two ways that this paper directly addresses. First, existing characterizations describe ACF content through high-level categories [4] that aggregate heterogeneous forms of guidance, leaving it unclear which instructions are explicit and actionable enough to support downstream verification. Second, no study measures whether individual ACF instructions are reflected in the commits agents produce in real projects. Our

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:6

Voria et al.

work addresses both gaps through a three-phase empirical pipeline: characterizing explicit and actionable ACF instructions, operationalizing them as executable measurement proxies, and assessing their manifestation in real agent-generated commits. 3

Empirical Study

The goal of this empirical study is to investigate the relationship between instructions encoded in agent context files and the observable behavior of autonomous coding agents in real-world software development, and to understand whether explicit guidance documented in these artifacts is reflected in the software modifications agents generate. The purpose is to characterize the types of instructions that can be empirically operationalized and to assess how far such instructions show up in downstream agent-generated commits. The study is conducted from the perspective of software engineering researchers and practitioners who want to understand how effective agent-oriented guidance is in practice. Rather than focusing on agent intent or internal reasoning, we take an artifact-centric view: we look at whether instructions are reflected in the observable software artifacts agents produce in real-world development. 3.1

Research Questions

Despite the growing use of agent context files to guide autonomous coding agents, it remains unclear what types of instructions they actually encode in a way that can be measured. Existing categorizations are typically too high-level and group together very different forms of guidance that cannot be directly measured in development artifacts [4]. To support a rigorous empirical investigation, we first need to identify which instructions are sufficiently explicit and actionable, meaning they are directly stated in the ACF and describe a concrete development behavior an agent can reasonably perform. Our first RQ builds a fine-grained dataset of explicit and actionable instructions from real-world ACFs, moving beyond high-level categories toward concrete guidance that supports downstream analysis. RQ1 . What types of explicit and actionable instructions are documented in agent context files? Characterizing explicit and actionable instructions is necessary, but not enough for empirical measurement. We also need to establish whether each instruction can be verified through observable development artifacts, meaning whether compliance can be checked by inspecting produced or modified files, without inferring agent intent or internal reasoning. Treating observability as a purely classificatory label is risky: an instruction judged observable in the abstract may still turn out impossible to measure in practice. Our second RQ investigates whether observable measurement proxies can actually be built for the instructions identified in RQ1 . For each instruction, we generate a measurable indicator that can determine compliance from commit-level artifacts. Instructions for which no such indicator can be built are considered non-operationalizable and excluded from downstream analysis. This process produces a refined dataset of instructions paired with validated measurement proxies, which is the empirical foundation for RQ3 . RQ2 . To what extent can observable measurement proxies be operationalized for explicit and actionable instructions documented in agent context files? Given a validated dataset of instructions with matching measurement proxies, the central question is whether autonomous coding agents actually follow the guidance in their ACFs. It remains unclear whether these instructions show up in the software artifacts agents produce during real development, or whether they are systematically ignored, partially applied, or inconsistently interpreted. Our third RQ investigates how far observable instructions are reflected in downstream agent-generated ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

AIDev Dataset

{ } ACF Snapshots (unique ACF versions)

Refine Strict AgentAuthored Commits Mine Commit-Level Changes

Agent Context Files Dataset (ACFs)

Filter Repositories with ACFs

Extracted per commit: • Diff patches • Metadata • Files before/after commit • ACF snapshot at commit time

{ } Explicit & Actionable Instructions

{ } Instructions & Observable Measures

Test-Driven Script & Fixture Generation

Instruction–Commit Applicability Filtering

Script Validation & Filtering

Compute Compliance per Instruction–Commit Pair

Manual Validation (Stratified Sample)

Manual Spot-Check of Accepted Scripts

Aggregate & Compare by Category / Agent

RQ1

RQ2

RQ3

Manual Calibration & Coding Guide LLM-Assisted Classification

N scripts × M fixtures (CodeT)

correctness + corroboration

X:7

file patterns from fixtures

Kruskal-Wallis, Cliff's δ

Fig. 1. Overview of the research process.

commits. For each instruction-commit pair, we apply the matching measurement proxy to the code changes and check whether observable evidence of compliance is present. RQ3 . To what extent are observable instructions documented in agent context files reflected in downstream agent-generated commits? Figure 1 summarizes the research method we followed. We first mined and combined data on agent context files and agent-generated code from two complementary datasets [4, 20]. We then built context file snapshots, unique versions of an agent context file defined by content and linked to the commits generated under that version, and classified their instructions using a mix of manual calibration and LLM-assisted qualitative analysis (RQ1 ). Next, we built a measurable proxy for each identified instruction, keeping only those for which a working indicator could be generated (RQ2 ). Finally, we applied the resulting measurement scripts to the reconstructed commit dataset and analyzed how far agent-generated code reflects the instructions in the associated ACF (RQ3 ). 3.2

Data Collection

To investigate the relationship between agent context files and agent-generated development artifacts, we built a dataset by combining two complementary sources: (i) an existing dataset of ACFs mined from GitHub repositories [4], and (ii) a large-scale dataset of agent-authored commits [20]. Combining them required extra preprocessing and reconstruction steps to support fine-grained instruction-artifact analysis. 3.2.1 Agent Context Files and Context Snapshots. We started from the ACF dataset introduced in prior work [4], which collects files from GitHub repositories based on the official naming conventions of widely used coding agents (e.g., CLAUDE.md, AGENTS.md, and copilot-instructions.md). We kept only repositories associated with three agents, Copilot, Claude Code, and OpenAI Codex, using the classification procedures from the original dataset for consistency. Agent context files change over time as developers update agent guidance. To capture the instructions available at the time of each commit, we built context snapshots: unique versions of an ACF, deduplicated by content, so identical file contents map to the same snapshot and any change in content produces a new one. Each snapshot was then linked to every commit generated while that version was active. This let us associate each commit with the exact instruction set available to the agent at generation time, giving 676 unique context snapshots reused across 10,763 commits and spanning 307 repositories. Context files varied a lot in size, with a median of 638 words (110 lines) and a maximum exceeding 16K words. 3.2.2 Agent-Authored Commits. To capture agent behavior in real-world development, we relied on the AIDev dataset [20], which attributes pull requests and commits to autonomous coding agents using signals such as bot accounts, branch naming conventions, and co-authorship markers. These ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:8

Voria et al.

Table 1. Characteristics of the analyzed commit dataset. Metric Repositories Commits Changed files Median changed files per commit Mean changed files per commit Maximum changed files in a commit

Value 268 6,541 25,822 1 3.95 1,056

Commits touching source files Commits touching test files Commits touching documentation files Commits touching configuration files Commits touching CI files Distinct programming languages

4,067 1,381 1,072 329 245 26

signals differ across agents, so we distinguished two confidence levels. Strict agent commits are those for which authorship can be inferred with high confidence from strong indicators, bot identities or unambiguous authorship metadata, following the same rules as AIDev. Non-human-suspected commits are those with no clear sign of human authorship but with less certain attribution. This distinction was needed because agents expose different authorship patterns: some appear as direct commit authors via dedicated bot accounts, others (e.g., Claude Code) appear as co-authors in commit messages, and tools such as GitHub Copilot may rely on indirect signals such as branch naming. All analyses reported in this paper use only the strict agent subset, favoring attribution precision over recall; non-human-suspected commits are not analyzed further. 3.2.3 Dataset Integration and Reconstruction. The two datasets were combined at the repository level, keeping only commits from repositories with at least one available ACF. Starting from 21,513 commits, 8,600 had context information, of which 6,580 also had file-level change data. After excluding 39 planning and merge commits (planning commits, 3,149 overall, mostly carry no file changes and had already been removed by this filter), the analysis-ready dataset is 6,541 commits across 268 repositories. For each selected commit, we rebuilt the full development context by cloning the corresponding Git repository and extracting commit metadata, diff patches, file-level change information, and the content of all modified files before and after the commit. The before version came from the parent commit; the after version is the current commit. All extracted artifacts were stored in a structured local format with automated completeness checks for missing commits, incomplete snapshots, and failed retrievals. All data and scripts are available in our replication package [33]. 3.2.4 Final Dataset. Table 1 summarizes the final dataset. The corpus spans a wide range of development activity: 62.2% of commits modify source code, 21.1% involve test files, and 16.4% affect documentation, with smaller shares covering configuration and CI artifacts. It covers 26 programming languages, including C#, TypeScript, Python, and Rust. With a median of 1 modified file per commit and a mean of 3.95, most commits are small, focused changes, which makes instruction-level compliance assessment tractable at the commit level. 3.3

RQ1 — Data Analysis

To address RQ1 , we used a three-phase approach combining manual qualitative analysis with LLM-assisted classification. The goal was to identify explicit and actionable instructions in agent context files, grounded in the high-level ACF categorization from prior work [4]. We retain that work’s 19 categories as defined, including partially overlapping ones (e.g., Build and Build and Run, ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:9

Management and Project Management), so that our results remain directly comparable to it rather than introducing a new taxonomy. The unit of analysis was the context snapshot: a distinct version of an agent context file, deduplicated by content and linked to the commits generated while that version was active. After removing empty and malformed snapshots, the final population was 621 unique context snapshots. 3.3.1 Phase 1 — Manual Calibration. We started with an iterative manual analysis of randomly selected context snapshots to set stable extraction procedures. Rather than fixing a sample size up front, we proceeded incrementally following qualitative saturation criteria [32]: we continued until the extraction procedure, coding guidelines, and decision rules were stable enough to support prompt construction for automated classification. This phase covered 15 ACF snapshots, after which no substantially new decision patterns appeared, so we considered the procedure stable. During this phase, we extracted and labeled candidate instructions, grounding them in the existing high-level ACF categories from prior work rather than building a new taxonomy. Each candidate instruction was judged against three criteria. • Explicitness: the instruction must be directly and clearly stated in the ACF, not inferred from examples or context. • Actionability: the instruction must describe a concrete development behavior an agent can reasonably perform during software modification. • Observability: compliance must be checkable through produced or modified development artifacts, such as source code, documentation, configuration files, or repository structure, without inferring agent intent or internal reasoning. The analysis focused only on behavioral guidance directed at agents; repository descriptions, architectural overviews, code examples, and other informational content were excluded. Disagreements and ambiguous cases were discussed among all authors and used to refine the coding guidelines. 3.3.2 Phase 2 — LLM-Assisted Classification. Once the coding procedure was stable, we used an LLM annotator to classify instructions across the full dataset [1, 31]. Manually coding all 621 snapshots would have taken substantial effort and introduced coder fatigue and inconsistency, so the LLM acted as a structured coding assistant rather than a source of ground truth. Annotation Pipeline. Each context snapshot was processed independently using a fixed fewshot prompt with three parts: (i) a system prompt defining the annotation task, the three coding criteria, and the output schema; (ii) a curated set of few-shot examples from the calibration phase, covering clear inclusion cases, clear exclusion cases, and borderline cases; and (iii) the target ACF content. Snapshots that exceeded the model’s context budget were split into paragraph-level chunks, annotated independently, and merged back at the snapshot level. For each extracted instruction, the model produced a JSON object recording the raw instruction text, high-level category, normalized label, coding decisions for each criterion, a candidate observable behavior field describing how compliance could be verified in produced artifacts, an inclusion decision, and a mandatory justification note. Prompt Engineering. The prompting strategy was developed iteratively through pilot runs on the calibration examples. Early versions systematically over-included vague or process-oriented statements as observable. The final version strengthened the observability criterion: an instruction now counts as observable only if a reasonably simple verification function could inspect the generated or modified files and determine compliance, with no access to agent reasoning, command execution, or internal workflow state. All prompt versions, model configurations, execution scripts, and few-shot examples are available in the replication package [33]. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:10

Voria et al.

Model Selection. We evaluated two candidate annotators, Ministral 3 14B Reasoning and Qwen 3.5 9B, both run locally via LMStudio with fixed inference parameters (32k-token context, temperature 0.1, top-p 0.85, GGUF Q4_K_M quantization). A cross-model comparison on a shared set of annotations gave an overall agreement rate of 74.8%, with 79.5% include-agreement and 76.1% observable-label agreement. Manual inspection of the disagreements showed that Ministral systematically over-extracted candidate instructions, accepting vague or process-oriented statements that Qwen correctly excluded. We therefore chose Qwen 3.5 9B as the primary annotator for the final extraction. 3.3.3 Phase 3 — Manual Validation. We did not assume LLM-generated annotations were equivalent to human judgment. Following established practice for LLM-based annotation [1, 31], we validated the automated classifications against the predefined coding criteria using a human annotator on a statistically representative sample. Sample Selection. The validation population was the 621 processed snapshots. Standard finitepopulation sampling at a 95% confidence level and 10% margin of error gave a validation sample of 84 context snapshots (13.5% of the population). We set the sampling unit at the snapshot level rather than the instruction level on purpose: validating a whole snapshot, rather than isolated instructions sampled across snapshots, lets the annotator check whether the extraction procedure correctly identifies all relevant instructions in a given ACF, including omissions that an instructionlevel sample would miss. To ensure representativeness, we used proportional stratified random sampling based on (i) the agent associated with the snapshot (Copilot, Claude Code, OpenAI Codex) and (ii) instruction density (low: 1–10 instructions; medium: 11–30; high: >30). The 84 sampled snapshots together contain 2,202 individual instructions for validation, both included and excluded by the LLM, corresponding to 14.8% of all 14,902 extracted instructions. This is the raw count of candidates the LLM extracted from these snapshots, before human review. The results (Section 4.1.2) also report a narrower count of 2,196 candidates that received a recorded validation decision, and a still-narrower count of 2,114 instructions confirmed as genuine. Each of these three counts excludes a different subset (undecided candidates, and false positives, respectively) and each is used where it fits that specific analysis. This instruction-level coverage is well above the minimum sample size needed for 95% confidence and 10% margin of error at the instruction level for a population this size (96 instructions), giving strong grounding for the instruction-level metrics we report. Validation Protocol. The validation was done by the first author, who had developed and applied the coding guide during calibration. Following Ortloff et al. [24], we used a single annotator for this well-bounded classification task: their study shows that for tasks with a narrow interpretive range, one experienced coder can be enough, and may even give higher internal validity than multi-coder setups with reconciliation noise. This fits our setting because each of the three coding criteria is a binary decision grounded in explicit textual evidence, and because the coding guide had already been refined with all coauthors before validation began. Low-confidence decisions during validation were still discussed with all coauthors until we reached a shared interpretation. The resulting data were used to compute extraction-level precision, recall, and F1-score, and label-level accuracy for explicitness, actionability, observability, inclusion decisions, and category assignment. We also validated the candidate observable behavior field: high accuracy here shows that the LLM-generated behavioral descriptors are reliable enough to use as input for the measurement proxy generation step in RQ2 . 3.3.4 RQ1 Output. The output of RQ1 is a validated dataset of explicit and actionable instructions extracted from agent context files, grounded in the existing high-level ACF categorization. For each ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:11

instruction, the dataset records its textual formulation, category, per-criterion coding decisions, and candidate observable behavior. This dataset gives a fine-grained picture of the concrete guidance documented in ACFs and how it is spread across repositories, agents, and instruction categories. Instructions validated as observable, together with their candidate behavioral descriptors, are the foundation for RQ2 . 3.4

RQ2 — Data Analysis

To address RQ2 , we investigated whether the explicit and actionable instructions from RQ1 can be turned into executable measurement scripts. Each instruction in our dataset is, by construction, explicit and describes a concrete development behavior verifiable through software artifacts. This makes each instruction suited to a functional specification: a well-defined procedure that takes commit-level artifacts as input and returns a binary compliance signal as output. Turning a natural language specification into such a procedure is a code generation task, and LLMs have shown strong, well-documented capability at this [18]. We treat an instruction as operationalizable only if a correct measurement script can be generated and validated for it; instructions for which this fails are excluded from the final RQ3 analysis. 3.4.1 Measurement Script Specification. Each measurement script is generated independently for a single instruction. We first considered grouping instructions that share a topic (e.g., naming conventions, indentation style) under a shared canonical representative before generation, on the idea that behaviorally equivalent instructions should map to the same measurement procedure. A closer look at the RQ1 dataset showed that this equivalence rarely holds: instructions on the same high-level topic often prescribe different, mutually exclusive requirements. For example, one instruction may require camelCase for method identifiers, another PascalCase for type names, and a third snake_case for file names, all under “naming conventions.” Collapsing these into one canonical representative would either drop the specific requirement each instruction mandates, or require per-instance parametrization of a shared script, which carries the same overhead as generating a script per instruction directly while adding a new point of failure (an instruction wrongly assigned to a template family, or a parameter wrongly extracted). We therefore generate one measurement script per instruction. For each instruction retained by RQ1 (explicit, actionable, and observable), we use its candidate observable behavior field as the functional specification for script generation. This field, produced during RQ1 extraction and confirmed on the manually validated sample, describes in concrete terms how compliance can be verified by inspecting files modified or added in a commit. For an instruction prescribing camelCase naming for method identifiers, for example, the candidate observable behavior states that all method names added or modified in the commit must match a camelCase pattern. 3.4.2 Test-Driven Measurement Script Generation. Script generation follows a test-driven approach, building on evidence that pairing a formal specification with independently generated test cases substantially improves the correctness of LLM-generated code [9, 22]. Rather than generating one script and repairing it against failures, we adopt the sampling-and-selection strategy from Chen et al.’s CodeT [6]: for each instruction, the same code LLM independently generates 𝑁 candidate measurement scripts and 𝑀 candidate test fixtures, where each fixture is a pair of synthetic commits, one compliant and one non-compliant, synthesized from the candidate observable behavior specification. Each synthetic commit uses the same four-field commit object that scripts take as input, files_before, files_after, message, and branch, so a script is validated on exactly the input format it later receives for real commits. Fixtures carry an intended label since they are generated from an explicit compliant/non-compliant prompt, but we do not treat this label as ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:12

Voria et al.

ground truth without checking it, since the LLM can misread an ambiguous instruction when building a fixture, mirroring the test-case correctness issues reported for LLM-generated tests in general [6]. Every candidate script is run against every candidate fixture. In CodeT’s original dual execution agreement procedure [6], scripts that give exactly the same verdicts across all fixtures form a consensus group, and the group scoring highest on the product of its size and the number of fixtures it passes is selected. We retain this principle, that a script is trusted only when independently generated scripts agree with it on independently generated tests, but replace exact-verdict grouping with the overlap-based acceptance criterion described in Section 3.4.3, because exact agreement proved too sparse at our sampling budget. 𝑁 and 𝑀 are small compared to the sampling budgets used in general-purpose code generation benchmarks, since the measurement scripts here target a narrower and more uniform class of checks than the algorithmic problems CodeT was originally evaluated on. We therefore use the exhaustive variant of the agreement procedure, comparing all script-fixture pairs directly rather than sampling from them. We use Qwen2.5-Coder-14B-Instruct as the generation model, matching the local execution setup used for instruction extraction in RQ1 (LMStudio, 32k context window, GGUF Q4_K_M quantization) to keep the infrastructure consistent and fully reproducible on the same hardware. We confirmed this configuration ran successfully on the available hardware before the full RQ2 run. Unlike the extraction stage, which favors deterministic, high-precision output (temperature 0.1), script and fixture generation needs diversity across independently sampled candidates for the dual execution agreement mechanism to work, so we set temperature to 0.8, the value Chen et al. [6] validated for this purpose, and kept top-p at 0.85. We fixed the sampling budget to 𝑁 = 𝑀 = 10: ten candidate scripts and ten candidate fixture pairs per instruction, compared exhaustively (100 script-fixture pairs). This is deliberately smaller than the budgets in CodeT’s original evaluation (50–1,000, depending on benchmark difficulty), which targeted open-ended competitive programming problems with many structurally different correct solutions [6]. In our setting, each instruction comes with an explicit candidate observable behavior specification that narrows the space of correct implementations considerably, so a smaller candidate pool is enough for the corroboration mechanism to reach a stable consensus. We did not tune 𝑁 and 𝑀 separately (the pilot described in Section 3.4.3 concerned the acceptance criterion, not the budget); the operationalization rate reported in RQ2 (the share of instructions for which a script is accepted) works as the direct empirical check on whether the fixed budget is adequate. 3.4.3 Script Validation and Filtering. A script is accepted only if it clears two bars: a minimumcorrectness bar (it must pass a minimum number of fixtures outright) and a corroboration bar (at least one other independently generated script must agree with it on a large share of the fixtures it passes). Neither bar alone is enough. The first alone would accept a script that happens to satisfy one fixture by chance, and the second alone, agreement with no floor on correctness, would accept two scripts that are consistently wrong the same way. Requiring both keeps CodeT’s core principle, that an accepted solution must be corroborated and not just self-consistent, while staying feasible at the sampling budget used here (𝑁 = 𝑀 = 10). In a pilot run of the pipeline on a subset of instructions before the full RQ2 run, exact-match convergence across independently generated scripts and fixtures essentially never happened, leaving almost every instruction without an accepted script. We set the minimum-correctness bar to two fixtures passed and the corroboration bar to 60% overlap with the best-performing script’s passed-fixture set. Both values were fixed on that same pilot rather than through a separate calibration study: a lower correctness floor risks accepting a script that only happens to satisfy a single fixture, while a much higher overlap threshold brings back the sparsity problem that led us to abandon exact-match agreement in the first place. Instructions ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:13

for which no script clears both bars are excluded from the RQ3 analysis. Rather than falling back to manually authored scripts, we treat exclusion as final, keeping one fully automated generation pipeline. Each exclusion is logged with the instruction’s high-level category, originating ACF, and coding agent, so we can check exclusions for patterns rather than treat them as missing at random. This pass/fail rule is our operational definition of operationalizability: an instruction resists operationalization if no script clears both bars within the sampling budget, regardless of whether it was classified as observable in RQ1 . Relaxing exact-match agreement to overlap-based corroboration lowers the bar for what counts as agreement, which could in principle raise the risk of accepting a script that is subtly wrong. To check this, we manually inspect a sample of accepted scripts and verify that each one correctly implements its instruction’s candidate observable behavior, using the same single-annotator validation protocol as RQ1 . We report the agreement rate between this manual check and the automated acceptance decision as a direct, empirical measure of how reliable the accepted scripts are, alongside the operationalization rate itself. We size the spot-check sample using the same finite-population-corrected Cochran formula, 95% confidence level, and 10% margin of error used for RQ1 ’s 84-snapshot validation sample (Section 3.3), applied here to the population of 4,123 accepted scripts instead of context snapshots. This gives a target sample size of 94. We stratify by category and allocate the 94-script target proportionally to each category’s share of accepted scripts, using the largest-remainder method to hit the target exactly and a floor of 1 script per category so none is left entirely unchecked. Within each category, scripts are drawn uniformly at random without replacement, using a fixed random seed for reproducibility. Each sampled script is reviewed by one author alongside its originating instruction’s raw text and candidate observable behavior specification, and judged agree (the script correctly implements the specified behavior) or disagree (it does not), with free-text notes for any judgment that is not immediately obvious from the two-way label alone. We refer to an accepted script together with the scripts that corroborate it as its consensus group. 3.4.4 RQ2 Output. The output of RQ2 is a validated set of instructions paired with executable measurement scripts. For each instruction, the dataset records the candidate observable behavior descriptor, the generated script, the fixture set used for validation, and the operationalizability verdict. This dataset is the measurement infrastructure used in RQ3 . Beyond the script artifacts themselves, we report the operationalization rate, the share of RQ1 ’s explicit, actionable, and observable instructions for which a validated script was obtained, both overall and broken down by high_level_category and by coding agent. This breakdown shows which classes of instructions are measurable at all, which directly bounds what RQ3 can say about adherence. If the operationalization rate were markedly lower for substantive categories (e.g., testing, security, architecture) than for surface-level categories (e.g., formatting, naming), that would mean RQ3 ’s adherence results, however precise, describe compliance for a subset of instructions skewed toward superficial conventions rather than the full space of explicit and actionable directives in ACFs. We report this breakdown as a required result of RQ2 , independent of RQ3 ’s findings, and discuss what it means when interpreting the RQ3 results. 3.5

RQ3 — Data Analysis

To address RQ3 , we applied the measurement scripts from RQ2 to the mined commit dataset and checked whether the instructions in each ACF are reflected in the matching agent-generated code changes. The analysis works at the level of individual (instruction, commit) pairs: for every commit generated while a given context snapshot was active, we consider each instruction from ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:14

Voria et al.

that snapshot for which RQ2 produced an accepted measurement script. Instructions without an accepted script are excluded from this analysis, by RQ2 ’s own definition of operationalizability. This is the coverage boundary discussed in Section 4.2.1 and revisited in Section 4.3. 3.5.1 Instruction–Commit Applicability. Design. Not every operationalized instruction is worth checking against every commit. A naming-convention instruction for Python method identifiers has nothing to say about a commit that only touches a YAML configuration file. Running its script against such a commit is not a neutral no-op: LLM-generated scripts tend to be permissive by default. They are written to look for evidence of a violation and report compliance when none is found, rather than requiring positive evidence that the check actually applied. So running a script against a commit it has nothing to check skews toward compliance with no real evidence behind it, rather than a neutral null result (Section 4.2.2 reports how common this default is among accepted scripts). Evaluating every instruction against every commit without filtering would inflate every adherence figure in this study with this kind of empty compliance, not genuine adherence. An applicability filter, run before script execution, is a methodological necessity, not an optional extra. We first considered setting applicability from hand-authored, category-level rules (e.g., “namingconvention instructions apply to commits touching source files,” “testing instructions apply to commits touching source or test files”). We reject this approach: it needs inventing and validating a new mapping from category to file type at a level of detail too high-level to be reliable. A category such as Testing covers instructions targeting Python test files, Go test files, JavaScript configuration files, and CI pipeline definitions alike, so no single category-level rule captures this. Implementation. We instead derive applicability directly from each instruction’s own accepted evidence, rather than from a newly written rule. Every accepted instruction in RQ2 ’s output carries the fixture set its script was validated against (fixtures_used), each fixture a realistic before/after commit example built specifically for that instruction. For each accepted instruction, we take the union of file paths in the files_before and files_after of all fixtures in its winning consensus group, and pull out from each path either its extension (e.g., .py, .json) or, for extensionless conventional filenames (e.g., Dockerfile, Makefile), its exact basename. This gives a small set of file patterns per instruction, grounded in the same artifacts RQ2 already validated the script against, rather than a newly introduced heuristic. A commit is applicable to an instruction if it touches at least one file matching one of these patterns. Instructions whose fixtures have empty files_before and files_after throughout, meaning their compliance criterion concerns only the commit message or branch name, are treated as universally applicable, since every commit has a message and a branch regardless of which files it touches. Instruction-commit pairs that fail the applicability check are excluded from evaluation entirely; they contribute no observation, compliant or otherwise. We report, per instruction, the number of applicable commits found, and separately flag any accepted instruction with zero applicable commits in the mined dataset. These cannot contribute an adherence observation despite being operationalizable, and are excluded from the statistical analysis in Section 3.5.3 on that basis. This is a further, separate loss beyond RQ2 ’s operationalization funnel, and we report it with the same transparency. 3.5.2 Instruction–Commit Compliance Evaluation. We use compliance to mean the presence, in an applicable commit’s artifacts, of the behavioral evidence a script derives directly from the instruction’s own candidate observable behavior (Section 3.4). That field is written from the instruction’s own text during RQ1 extraction, and the script is generated and validated against fixtures grounded in it before ever being applied to a real commit, so a compliance verdict reflects what the instruction itself specifies as satisfying behavior. For every instruction-commit pair that passes ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:15

the applicability filter, we run the instruction’s accepted script against that commit’s artifacts and record a binary compliance signal. The script’s input is exactly the four-field commit object defined in Section 3.4, files_before, files_after, message, branch, populated from the mined commit dataset rather than a synthetic fixture. Scripts do not additionally receive a line-level diff or patch; they were generated and validated only against full before/after file content, and extending the contract after the fact to include diff-level information would invalidate the already-generated and already-validated script population. This means instructions whose true compliance criterion depends on what was specifically added or removed, rather than the file’s final state, are evaluated on a weaker signal than they ideally would be. We treat this as a limitation of the current design, discussed further under construct validity in Section 6, rather than fix it by regenerating scripts. 3.5.3 Aggregation and Statistical Analysis. We do not compare instruction categories or coding agents directly on the raw (instruction, commit) observations, because these observations are not independent: commits from the same context snapshot share the same instruction set, and instructions evaluated within the same commit share the same agent and repository. Treating each pair as an independent sample would understate variance and risk finding significance that is not really there. We instead aggregate to one compliance rate per instruction, the share of its applicable commits in which it was followed, before making any cross-group comparison, so each instruction contributes exactly one observation regardless of how many commits it was evaluated against. We report descriptive statistics on this instruction-level compliance-rate dataset: the overall mean compliance rate, the rate by high_level_category, and the rate by coding agent (Copilot, Claude Code, OpenAI Codex), matching the reporting structure used for operationalization rate in RQ2 . To test whether compliance rate differs across categories, we compare the distributions of per-instruction compliance rates using the Kruskal-Wallis test, a good fit given the expected non-normality of rate data bounded in [0, 1] and the unequal group sizes across categories. Where this test is significant, we run post-hoc pairwise Mann-Whitney U tests between categories with Bonferroni correction for the number of comparisons, and report Cliff’s delta as the effect size for each significant pair, so the size of a difference is reported independently of the sample size behind it. We repeat the same procedure with coding agent as the grouping variable, to test whether adherence patterns differ by agent and not only by instruction category. Many instructions are evaluated against very few applicable commits: the median evaluated instruction has only 3, and 540 (26.3%) have exactly 1, which forces their individual compliance rate to 0% or 100% regardless of true adherence. To check whether this thin-evidence tail is driving the bimodal split and the two cross-group comparisons, we run a sensitivity analysis repeating the full statistical procedure above restricted to instructions with at least 3, and separately at least 5, applicable commits, and report whether the comparisons found significant in the full sample stay significant as this low-evidence tail is excluded. 4 4.1

Analysis of the Results RQ1 — What types of explicit and actionable instructions are documented in agent context files?

We answer RQ1 by applying the classification pipeline from Section 3.3 to all 621 context snapshots. Section 4.1.1 reports how the extracted instructions split across the three coding criteria: explicitness, actionability, and observability. Section 4.1.2 then reports the manual validation we use to check how reliable this automated classification is. 4.1.1 Instruction Extraction Across the Three Coding Criteria. Table 2 shows how the 14,902 candidate instructions extracted from the 621 context snapshots (24.0 per snapshot on average) are ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:16

Voria et al.

Table 2. Instruction funnel across the three coding criteria, for the full corpus and for the validated sample. The corpus column counts raw candidates extracted by the LLM. The sample column counts only the 2,114 instructions confirmed as genuine during manual validation (2,113 true positives plus 1 instruction the LLM missed); this excludes 82 false positives that the LLM extracted but the human annotator rejected. Stage Total Explicit + Actionable + Observable

Sample, confirmed (𝑁 = 2,114) 2,114 (100.0%) 1,403 (66.4%) 1,145 (54.2%) 585 (27.7%)

Corpus, extracted (𝑁 = 14,902) 14,902 (100.0%) 10,324 (69.3%) 8,726 (58.6%) 6,552 (44.0%)

classified by the three coding criteria. It also reports the same breakdown for the 84-snapshot validated sample, restricted to the instructions confirmed as genuine during manual validation (Section 4.1.2). Applied in sequence to the full corpus, the criteria narrow the candidates to 10,324 (69.3%) that are explicit, 8,726 (58.6% of all candidates) that are also actionable, and 6,552 (44.0% of all candidates) that are also observable. This final set, which satisfies all three criteria, forms the RQ1 dataset. It covers 592 of the 621 snapshots (95.3%), and 615 snapshots (99.0%) contain at least one explicit and actionable instruction. Explicitness is the first and, in absolute terms, the largest filter, discarding 30.7% of all candidates. Among the criteria applied to already explicit instructions, observability is the more selective: it removes about a quarter of the explicit-and-actionable instructions at the corpus level (24.9%) and close to half at the sample level (48.9%), whereas actionability removes 15.5% and 18.4% of the explicit ones, respectively. The much sharper observability cut on the validated sample than on the corpus is consistent with the per-criterion accuracies reported in Section 4.1.2, where observability is the hardest criterion for the LLM to judge correctly and its errors lean toward over-inclusion. Table 3 reports the corpus-wide category distribution at two funnel stages: explicit-and-actionable (𝑁 = 8,726) and the final, fully-qualified dataset (𝑁 = 6,552). It also reports a commit-weighted view of the final dataset, which scales each category’s count by the number of commits that reuse the snapshot it appears in (𝑁 = 106,923 weighted occurrences across the 10,763 commits reusing the corpus’s snapshots; this weighting uses all commits linked to the snapshots, before the file-level filtering of Section 3.2 that yields the 6,541 analysis commits). Implementation Details is the largest category at every stage and grows further once observability is applied, from 42.8% of explicitand-actionable instructions to 50.0% of the final dataset. Together with Testing and Documentation, these three categories make up 76.6% of the final dataset, and this concentration is even sharper (83.2%) in the commit-weighted view. Other categories shrink more once observability is applied: Build and Run drops from 8.8% of explicit-and-actionable instructions to 2.9% of the final dataset, and Build from 3.8% to 2.5%. This shows that process and execution guidance in these categories is often explicit and actionable, but rarely verifiable from the artifacts it leaves behind. For example, “run the linter” is a clear, concrete instruction, but nothing in the produced files shows whether the agent actually ran it. The agent distribution stays fairly stable across funnel stages: Copilot accounts for 55.4% of explicit-and-actionable instructions and 56.9% of the final dataset, Claude Code for 38.7% and 37.4%, and OpenAI Codex for 5.9% and 5.7%, roughly matching each agent’s share of ACF snapshots in the corpus. 4.1.2 Reliability of the Automated Classification. To check how reliable this classification is, we manually validated a stratified sample of 84 context snapshots (13.5% of the population), covering 2,196 individually scored instruction-level decisions. This is the 2,202 raw candidates the LLM extracted from these snapshots (Section 3.3), plus the 1 instruction it missed, minus 7 candidates for which no decision was recorded. Table 4 reports extraction precision, recall, and F1-score, overall and by agent. The LLM annotator correctly identified 2,113 genuine instructions (TP), incorrectly ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:17

Table 3. Corpus-wide category distribution at the explicit-and-actionable and final (explicit, actionable, and observable) funnel stages, with a commit-weighted view of the final stage. All 19 categories are listed, matching the level of detail used for the category-level statistical tests in RQ3 (Section 4.3.2). Category Implementation Details Testing Documentation Build and Run Config. and Environment Development Process Build Quality Architecture Security Management Maintenance Debugging Performance Project Management UI/UX DevOps General AI Integration Total

Expl.+Act. 3,738 (42.8%) 1,231 (14.1%) 952 (10.9%) 767 (8.8%) 484 (5.5%) 373 (4.3%) 328 (3.8%) 222 (2.5%) 152 (1.7%) 118 (1.4%) 104 (1.2%) 89 (1.0%) 46 (0.5%) 37 (0.4%) 28 (0.3%) 21 (0.2%) 14 (0.2%) 13 (0.1%) 9 (0.1%) 8,726

Final 3,279 (50.0%) 927 (14.1%) 810 (12.4%) 189 (2.9%) 356 (5.4%) 221 (3.4%) 164 (2.5%) 133 (2.0%) 131 (2.0%) 99 (1.5%) 65 (1.0%) 66 (1.0%) 36 (0.5%) 25 (0.4%) 21 (0.3%) 14 (0.2%) 4 (0.1%) 11 (0.2%) 1 (0.0%) 6,552

Weighted 58,944 (55.1%) 14,943 (14.0%) 15,086 (14.1%) 1,772 (1.7%) 5,698 (5.3%) 2,352 (2.2%) 1,975 (1.8%) 1,395 (1.3%) 1,148 (1.1%) 610 (0.6%) 979 (0.9%) 996 (0.9%) 333 (0.3%) 155 (0.1%) 162 (0.2%) 248 (0.2%) 18 (0.0%) 88 (0.1%) 21 (0.0%) 106,923

Table 4. Extraction-level performance of the LLM-assisted instruction extraction, computed on the validated sample, overall and by agent. Group Overall Copilot Claude Code OpenAI Codex

Snap. 84 50 29 5

TP 2,113 1,210 784 119

FP 82 23 49 10

FN 1 1 0 0

Prec. 0.963 0.981 0.941 0.922

Rec. 0.999 0.999 1.000 1.000

F1 0.981 0.990 0.970 0.960

extracted 82 non-instructions (FP), and missed only 1 instruction that the human annotator later found during full-snapshot review (FN). This gives an overall precision of 96.3%, recall of 99.9%1 , and F1-score of 98.1%. Performance is consistent across agents: Copilot has the highest F1-score (99.0%) and OpenAI Codex the lowest (96.0%), the latter computed over a smaller group of 5 snapshots. Beyond the extraction itself, Table 5 reports label-level accuracy for each of the six coding decisions, computed over the 2,113 confirmed genuine instructions. This covers explicitness, actionability, and observability, alongside inclusion, category, and the candidate observable behavior descriptor. Accuracy is high and consistent for explicitness (95.5%), actionability (97.3%), category assignment (96.6%), and the candidate observable behavior descriptor (97.5%). The last of these is the field reused as the specification input for measurement script generation in RQ2 , so its high accuracy supports using it as a reliable input there. Observability (85.0%) and the resulting inclusion decision (85.2%) are noticeably less accurate. This confirms that telling artifact-verifiable guidance apart from process-oriented or vague statements is the hardest of the three coding decisions for the LLM annotator, matching the large share lost by process and execution categories at the observability stage of the funnel (Table 3). Over 90% of the disagreements on these two fields corrected an LLM yes/include decision to no/exclude, so the annotator’s errors lean toward over-inclusion rather 1 Recall is computed from a single recorded omission across the entire sample, so while the point estimate is high, it rests on

very few observed misses. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:18

Voria et al.

Table 5. Label-level accuracy of the LLM’s per-criterion coding decisions on confirmed genuine instructions (𝑁 = 2,113), overall and by agent. Criterion Explicitness Actionability Observability Inclusion Category Cand. behavior

Overall 0.955 0.973 0.850 0.852 0.966 0.975

Copilot 0.952 0.967 0.859 0.855 0.967 0.988

Claude Code 0.962 0.983 0.841 0.839 0.980 0.954

Codex 0.941 0.975 0.824 0.908 0.874 0.983

than going in a random direction. This means the corpus-wide counts at the observable/final funnel stage (Table 2) should be read as upper bounds. OpenAI Codex shows lower category accuracy (87.4% versus 96.7–98.0% for the other two agents), again over a small group of snapshots. Overall, the validation shows that the automated classification closely reproduces the explicitness and actionability criteria set during manual calibration (Section 3.3), and reproduces observability less accurately and with a systematic lean toward over-inclusion. This gives us enough confidence to treat the LLM-classified corpus as the basis for the RQ1 dataset, keeping in mind the upper-bound caveat on the observable/final stage noted above. Answer to RQ1 . Applied to the full corpus of 621 context snapshots, the classification pipeline identifies 10,324 explicit instructions (69.3% of all 14,902 candidates), of which 8,726 (58.6% of candidates) are additionally actionable, and of which 6,552 (44.0% of candidates, an upper-bound estimate per the validation above) are additionally observable. This fully-qualified dataset spans 95.3% of all snapshots. Among explicit instructions, observability is the most selective criterion, and it cuts process- and execution-oriented categories hardest (Build and Run falls from 8.8% of explicit-and-actionable instructions to 2.9% of the final dataset), so that Implementation Details, Testing, and Documentation together make up over three-quarters of the final dataset. Manual validation on a representative sample shows the extraction is reliable (96.3% precision, 99.9% recall) and the explicitness, actionability, category, and candidate-behavior labels are accurate (95.5–97.5%), while observability and the resulting inclusion decision are the least reliable (85.0–85.2% accuracy, biased toward over-inclusion). 4.2

RQ2 — To What Extent Can Observable Measurement Proxies Be Operationalized for Explicit and Actionable Instructions Documented in Agent Context Files?

4.2.1 Operationalization Outcomes. Table 6 reports the outcome of the script generation pipeline from Section 3.4, applied to the 6,552 instructions RQ1 classified as explicit, actionable, and observable at the corpus level. This is the same final-stage population reported in Table 3. Of these, 4,123 (62.9%) reach an accepted measurement script under the sampling budget and corroboration criteria fixed in Section 3.4. The remaining 2,429 (37.1%) resist operationalization within that budget and are excluded from RQ3 . Category-Level Variation. Operationalization rate is not the same across instruction categories, and the pattern only partly matches what a simple superficial-versus-substantive split would predict. Among the twelve categories with at least 50 instructions, Configuration and Environment operationalizes at the highest rate (269 out of 356, 75.6%), followed by Maintenance (48 out of 66, 72.7%), Architecture (95 out of 131, 72.5%), Build and Run (130 out of 189, 68.8%), and Testing (626 out of 927, 67.5%), all above the corpus-wide average. This runs counter to a concern raised ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:19

Table 6. Operationalization outcome by instruction category. Total reproduces the final-stage, explicit+actionable+observable counts from Table 3, listing all 19 categories.

Category Implementation Details Testing Documentation Configuration and Environment Development Process Build and Run Build Quality Architecture Security Maintenance Management Debugging Performance Project Management UI/UX General DevOps AI Integration Total

Total Accepted Rate 3,279 2,008 61.2% 927 626 67.5% 810 505 62.3% 356 269 75.6% 221 130 58.8% 189 130 68.8% 164 95 57.9% 133 76 57.1% 131 95 72.5% 99 48 48.5% 66 48 72.7% 65 31 47.7% 36 19 52.8% 25 15 60.0% 21 8 38.1% 14 7 50.0% 11 10 90.9% 4 3 75.0% 1 0 0.0% 6,552 4,123 62.9%

when this pipeline was designed, that automated operationalization would systematically favor superficial formatting conventions over substantive engineering categories: Architecture and Testing instructions are, in practice, no harder to operationalize than average, and in this sample somewhat easier. At the opposite end, Management (31 out of 65, 47.7%) and Security (48 out of 99, 48.5%) operationalize at the lowest rates among these categories, more than 14 percentage points below the average. Implementation Details, the largest single category by volume (3,279 instructions, 50.0% of the final-stage corpus), operationalizes close to the overall average at 61.2% (2,008 out of 3,279). Because it is so large, it dominates the aggregate rate regardless of how other categories behave. Agent-Level Consistency. Operationalization rate varies little by coding agent: 2,367 out of 3,729 (63.5%) for Copilot, 1,520 out of 2,449 (62.1%) for Claude Code, and 236 out of 374 (63.1%) for OpenAI Codex, a spread of only 1.4 percentage points. Which agent’s context file an instruction came from has essentially no bearing on whether it can be operationalized. The category-level variation above is the main source of difference in coverage, not the originating agent. 4.2.2 Exclusion Patterns and Generation Reliability. Table 7 breaks down the 2,429 excluded instructions by the bar they failed. Most of them (1,392, 57.3%) fail the minimum-correctness bar: for 1,006 (41.4%) no generated script passed even one fixture, and for 386 (15.9%) the best script passed only one. A further 997 (41.1%) clear the correctness bar, but no independently generated script agrees with the best one. Only 40 (1.6%) fail at candidate generation, with fewer than two syntactically valid scripts. This shows that generation-stage reliability is high, and exclusion is mostly a correctness or agreement problem, not a formatting or parsing one. Across all 6,552 instructions, the pipeline obtained the full requested budget of ten candidate scripts for 6,223 (95.0%) and the full ten ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:20

Voria et al.

Table 7. Exclusion reason for the 2,429 instructions that did not reach an accepted script. Percentages may not sum to 100% due to rounding.

Exclusion Reason No script passed any fixture No corroborating script Top script below min. passed Insufficient valid candidates Total excluded

Count 1,006 997 386 40 2,429

% of Exclusions 41.4% 41.0% 15.9% 1.6% 100.0%

candidate fixtures for 5,786 (88.3%). Truncated or malformed candidates are the exception, not the rule. Among the 4,123 accepted instructions, the winning consensus group has a mean size of 6.3 out of 10, and this splits into two clusters: 825 instructions (20.0% of accepted) reach full unanimous agreement across all ten sampled scripts, while 589 (14.3%) are accepted on the minimum admissible corroboration of exactly two agreeing scripts. The remaining 65.7% are spread fairly evenly in between. Operationalizability is therefore not a clean two-way split: many instructions have a candidate observable behavior specific and mechanical enough that independently sampled scripts converge on it readily, but the corroboration threshold is decisive for a substantial share of cases, both among accepted instructions (the 14.3% accepted on exactly two agreeing scripts) and among excluded ones (the 997 instructions whose correct-looking script found no corroborating peer). Consistent with the permissive-default tendency noted in Section 3.5 as the reason for the applicability filter, 2,569 of the 4,123 accepted scripts (62.3%) return True with no supporting evidence when checked against a commit that touches no relevant files. The applicability filter is built to exclude exactly these uninformative pairings from the RQ3 analysis, so this default does not by itself bias the compliance rates reported there. It does mean, though, that any instruction-commit pair that slips past the filter despite being irrelevant will count as compliant rather than as a neutral non-observation. The RQ3 compliance figures should therefore be read as an optimistic estimate: the reported non-compliance rate is a floor on true non-adherence, not a precise ceiling, since any leakage through the applicability filter can only push apparent compliance up, never down. The manual spot-check described in Section 3.4 finds that 67 of the 94 sampled accepted scripts (71.3%) correctly implement their instruction’s candidate observable behavior. Conversely, roughly three in ten accepted scripts do not faithfully implement their specification, so passing both acceptance bars is evidence of, not a guarantee of, a correct operationalization. The disagreements cluster around a small number of recurring patterns rather than spreading evenly. The most common is a script that hardcodes an example from the ACF itself, a specific filename, type name, or argument value used to illustrate a convention, as a literal requirement instead of generalizing the underlying rule. Genuine implementation bugs, such as a condition that can never be true, a missing import, or a regex typo that silently disables half a check, are comparatively rare in the sample. Because these errors are systematic rather than random, they cannot be assumed to cancel out: a hardcoded example typically makes a script stricter than the instruction it implements, which would bias the affected instructions’ RQ3 compliance downward. The RQ3 rates should therefore be read with this measurement error in mind, alongside the optimistic bias from permissive defaults discussed above. Answer to RQ2 .

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:21

A majority, 4,123 out of 6,552 (62.9%), of the explicit, actionable, and observable instructions identified in RQ1 can be operationalized as executable measurement scripts through testdriven, consensus-based generation, although a manual spot-check confirms that only 71.3% of accepted scripts faithfully implement their instruction’s specification. Operationalization rate is essentially unaffected by which coding agent’s context file an instruction came from, but varies meaningfully by category: Configuration and Environment, Architecture, and Testing instructions operationalize above the corpus average, while Management (47.7%) and Security (48.5%) instructions operationalize at markedly lower rates. Most exclusions (57.3%) happen because no generated script is correct enough, and 41.1% because no second script agrees with the best one. Almost none (1.6%) come from missing or malformed candidates. Because the operationalized subset is not a uniform sample of RQ1 ’s instruction population, particularly for Security and Management, the adherence results reported for RQ3 describe compliance within this operationalizable subset, not the full space of explicit and actionable directives documented in ACFs. 4.3

RQ3 — To What Extent Are Observable Instructions Documented in Agent Context Files Reflected in Downstream Agent-Generated Commits?

4.3.1 Evaluation Coverage. Table 8 reports how many of the 4,123 instructions operationalized in RQ2 could actually be checked against a real commit. An instruction contributes an adherence observation only if at least one commit generated under its originating context snapshot touches a file matching that instruction’s own applicability pattern, derived in Section 3.5 from the instruction’s accepted fixtures. Of the 4,123 operationalized instructions, 2,053 (49.8%) have at least one applicable commit. The remaining 2,070 (50.2%) are operationalizable in principle but never meet a commit that touches the relevant file type in the mined dataset, and are excluded from the compliance analysis on that basis. This is a further loss beyond RQ2 ’s own operationalization funnel: across the full pipeline, only 2,053 of the original 6,552 explicit, actionable, and observable instructions from RQ1 (31.3%) end up producing an adherence observation. The 2,053 evaluated instructions are checked against 18,594 instruction-commit pairs in total. Category-Level Coverage. Coverage is uneven across categories and does not track the operationalization rate from RQ2 . Among the nine categories with at least 50 operationalized instructions, Documentation reaches the highest coverage (340 out of 505, 67.3%), while Configuration and Environment (99 out of 269, 36.8%) and Development Process (50 out of 130, 38.5%) fall well below the corpus-wide average of 49.8%; among smaller categories, Maintenance (15 out of 48, 31.2%) and Security (18 out of 48, 37.5%) are similarly low. This happens even though Configuration and Environment had the highest operationalization rate of any category in RQ2 (Table 6). This gap shows that being operationalizable and being evaluable are governed by different things: an instruction can be easy to write a correct measurement script for, and still rarely meet a matching commit in the mined dataset, because the file types it concerns are touched infrequently relative to how often the convention could in principle be checked. Agent-Level Coverage. Coverage is close to uniform by originating agent: 799 out of 1,520 (52.6%) for Claude Code, 1,137 out of 2,367 (48.0%) for Copilot, and 117 out of 236 (49.6%) for OpenAI Codex, a spread of 4.6 percentage points. As with RQ2 ’s operationalization rate, which agent’s context file an instruction came from has little bearing on whether it is ever evaluable. Category composition, not agent, is again the main source of variation. 4.3.2 Compliance Outcomes. Across the 2,053 evaluated instructions, the mean per-instruction compliance rate is 59.8%, but the median is 100.0%. This gap between mean and median points ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:22

Voria et al.

Table 8. Evaluation coverage and compliance outcome by instruction category. Operationalized reproduces the RQ2 accepted counts from Table 6. Evaluated is the number of operationalized instructions with at least one applicable commit; Mean/Median are computed over the per-instruction compliance rate among evaluated instructions only. AI Integration is omitted from the row listing (0 operationalized instructions in Table 6, hence 0 evaluated) but is included in the Total row for completeness; it also does not contribute a group to the category-level significance test in Section 4.3.2, which is accordingly computed over 18 categories.

Category Implementation Details Testing Documentation Configuration and Environment Build and Run Development Process Architecture Build Quality Maintenance Security Management Debugging Performance General Project Management UI/UX DevOps Total

Operat. Eval. Cov. Comp. Mean Comp. Median 2,008 1,022 50.9% 63.6% 100.0% 626 270 43.1% 49.2% 37.5% 505 340 67.3% 60.2% 100.0% 269 99 36.8% 56.3% 100.0% 130 62 47.7% 69.9% 100.0% 130 50 38.5% 34.0% 0.0% 95 40 42.1% 38.2% 0.0% 95 48 50.5% 67.5% 100.0% 76 37 48.7% 63.5% 100.0% 48 15 31.2% 85.3% 100.0% 48 18 37.5% 58.6% 98.7% 31 24 77.4% 44.6% 0.0% 19 4 21.1% 21.9% 0.0% 15 9 60.0% 65.1% 100.0% 10 7 70.0% 71.4% 100.0% 8 3 37.5% 93.3% 100.0% 7 3 42.9% 91.7% 100.0% 3 2 66.7% 50.0% 50.0% 4,123 2,053 49.8% 59.8% 100.0%

to a strongly bimodal distribution rather than a spread of partial compliance: 1,116 instructions (54.4%) are followed in every applicable commit checked, 766 (37.3%) are followed in none, and only 171 (8.3%) fall strictly in between. Part of this pattern comes from how thin the evidence often is: the median instruction has only 3 applicable commits, and 540 instructions (26.3%) have exactly one, so many of the 0% or 100% rates rest on a single observation. This does not bias the group comparisons that follow, since those operate on per-instruction rates precisely to avoid counting the same evidence twice (Section 3.5.3). But it does mean individual instruction-level rates should be read as rough signals, not precise estimates. Category-Level Variation. A Kruskal-Wallis test across the 18 instruction categories with at least one evaluated instruction finds a significant difference between them (𝐻 = 59.13, 𝑝 < .001); AI Integration contributes no group to this test, since it has zero operationalized instructions in RQ2 (Table 6) and zero evaluated instructions here (Table 8). Post-hoc pairwise Mann-Whitney U tests with Bonferroni correction across the 153 category pairs identify four pairs that remain significant. Three involve Development Process as the lower-compliance side; the fourth compares the two largest categories of the final dataset, Implementation Details and Testing. Effect sizes are reported as Cliff’s 𝛿 of the first-named group against the second. Development Process instructions (mean 34.0%, median 0.0%) comply significantly less often than Build and Run (𝑝 Bonf = .013, 𝛿 = −0.39), Documentation (𝑝 Bonf = .037, 𝛿 = −0.29), and Implementation Details (𝑝 Bonf = .002, 𝛿 = −0.32), each a small-to-medium effect. Implementation Details also complies significantly more often than Testing (𝑝 Bonf = .002, 𝛿 = 0.15), a small effect. Architecture (mean 38.2%, median 0.0%) and ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:23

Debugging (mean 21.9%, median 0.0%) show similarly low compliance to Development Process on paper, but neither survives Bonferroni correction against any other category: Architecture because so many comparisons were tested, Debugging because its support (4 evaluated instructions) is too small to carry statistical power. We report both as descriptive only. Taken together, the surviving comparisons single out Development Process as the one category with a significant compliance gap relative to several others in the full sample, rather than pointing to broad, category-driven variation across the whole corpus; Section 4.3.3 shows that this gap, unlike the Implementation Details-versus-Testing difference, does not survive once thin-evidence instructions are excluded. Agent-Level Variation. Table 9 reports compliance by originating agent. A Kruskal-Wallis test across the three agents finds a significant difference between them (𝐻 = 41.58, 𝑝 < .001). Copilot instructions comply at a mean rate of 66.0% (median 100.0%), significantly higher than both Claude Code (52.2% mean, 𝑝 Bonf < .001, 𝛿 = 0.15, a small effect) and OpenAI Codex (51.2% mean, 𝑝 Bonf = .018, 𝛿 = 0.13, a negligible-to-small effect). Claude Code and OpenAI Codex are statistically indistinguishable from each other (𝑝 Bonf = 1.0, 𝛿 = −0.0005). So the finding is not that three agents diverge from one another in three directions, but that Copilot-authored commits follow their own context file’s instructions somewhat more often than Claude Code’s or Codex’s do, while the latter two are, on this measure, identical. OpenAI Codex’s estimate rests on the smallest sample of the three (117 evaluated instructions against 799 and 1,137 for the other two agents), so while the Copilot-versus-Codex difference reaches significance, it should be read with that imbalance in mind. 4.3.3 Sensitivity to Applicable-Commit Thresholds. Because 540 of the 2,053 evaluated instructions (26.3%) have exactly one applicable commit, which forces their individual compliance rate to 0% or 100% by construction, we check whether the results above hold up once these thin-evidence instructions are excluded. We repeat the bimodal-split computation and both Kruskal-Wallis analyses restricted to instructions with at least 3 (n = 1,144) and at least 5 (n = 819) applicable commits. The bimodal shape itself holds up well. The share of instructions at a clean extreme, 0% or 100%, drops only a little as the thin-evidence tail is excluded: from 91.7% in the full sample to 87.3% at the ≥ 3 threshold and 86.2% at ≥ 5. The share falling strictly in between rises to match, from 8.3% to 12.7% to 13.8%. Even at the strictest threshold, most instructions are still either always or never followed, not spread across a range. The category-level difference across groups also stays significant at both thresholds (𝐻 = 40.79, 𝑝 < .001 at ≥ 3; 𝐻 = 43.97, 𝑝 < .001 at ≥ 5), but the specific pairs behind it do not all hold up. None of Development Process’s three significant comparisons in the full sample (versus Build and Run, Documentation, Implementation Details) survive Bonferroni correction at either threshold, even though its mean compliance stays the lowest among categories with enough support at each threshold (34.0% at 𝑛 = 50, 25.0% at 𝑛 = 14, 21.4% at 𝑛 = 7). Only the Implementation Details-versusTesting comparison stays significant at every threshold; this is the one category-level finding that fully holds up under this check. Since Development Process’s own evaluated sample shrinks from 50 to 14 to 7 instructions as we filter, this looks like a loss of statistical power on an already small group, not a reversal of the underlying gap. The agent-level result splits the same way. The Copilot-versus-Claude Code gap stays significant at every threshold (𝑝 Bonf < .001, .002, .004 at the full, ≥ 3, and ≥ 5 samples, 𝛿 consistently between 0.12 and 0.15), so this is the one agent-level comparison that fully holds up. The Copilot-versusOpenAI Codex gap, by contrast, is significant only in the full sample (𝑝 Bonf = .018) and not at ≥ 3 (𝑝 Bonf = .165) or ≥ 5 (𝑝 Bonf = .102), even though the direction of the effect is unchanged and its magnitude grows (𝛿 = 0.13, 0.14, 0.23) as Codex’s already small evaluated sample shrinks ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:24

Voria et al.

Table 9. Compliance outcome by coding agent. Operationalized and Evaluated counts as in Table 8.

Agent Operat. Copilot 2,367 Claude Code 1,520 OpenAI Codex 236 Total 4,123

Eval. Cov. 1,137 48.0% 799 52.6% 117 49.6% 2,053 49.8%

Mean Median 66.0% 100.0% 52.2% 66.7% 51.2% 100.0% 59.8% 100.0%

further, from 117 to 56 to 22 instructions, under filtering. Claude Code and OpenAI Codex remain statistically indistinguishable at every threshold, matching the full-sample result. Taken together, this check supports the paper’s core compliance finding, the bimodal split and the Copilot-versus-Claude Code gap, as holding up once thin-evidence instructions are excluded. It also shows that Development Process’s specific compliance gap and Copilot’s specific edge over OpenAI Codex point in the same direction but are not statistically solid at stricter evidence thresholds, and we read them accordingly in what follows. Answer to RQ3 . Of the 4,123 instructions operationalized in RQ2 , 2,053 (49.8%) are ever checked against a real commit, yielding 18,594 instruction–commit observations; the remainder are operationalizable but never meet an applicable commit in the mined dataset. Among evaluated instructions, mean compliance is 59.8%, but the distribution is strongly bimodal: 54.4% of instructions are followed in every applicable commit and 37.3% in none, rather than spread across a range of partial adherence. Compliance varies significantly by category (𝑝 < .001), but few pairwise differences survive correction for multiple comparisons. Implementation Details instructions are followed more often than Testing ones, the only category-level difference that holds at every evidence threshold. Development Process instructions are the least compliant among categories with enough support and differ significantly from three others in the full sample, though this gap is not solid under a sensitivity check that restricts the comparison to instructions with more applicable-commit evidence (Section 4.3.3). Compliance also varies significantly by agent (𝑝 < .001): Copilot-authored commits comply significantly more often than Claude Code’s, a gap that holds up under the same sensitivity check, while Copilot’s edge over OpenAI Codex specifically does not; Claude Code and OpenAI Codex are statistically indistinguishable from one another at every threshold tested. Overall, explicit and actionable ACF instructions are reflected in agent behavior more often than not, but adherence is neither uniform across instruction categories nor across coding agents. 5

Discussion and Implications

In this section, we discuss our findings further and draw practical implications for practitioners using autonomous coding agents in their workflow. To the best of our knowledge, this is among the first articles to evaluate ACF governance at the level of the single instruction, rather than the whole file. Prior studies describe ACF content through high-level categories [4, 5], describe large differences in structure across projects [23], or treat the ACF as an all-or-nothing treatment, measuring the effect of its presence rather than of what it says [21]. Our breakdown into explicit, actionable, and observable instructions shows that this broader view hides substantial differences: indeed, within the same category, individual instructions differ sharply in whether they can be verified at all, let alone whether they are followed.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:25

Researchers studying agent governance should treat the single instruction, not the file or the category, as the unit of analysis. Aggregate compliance claims summarize a strongly bimodal population (Section 4.3.2), not a stable property of the artifact itself. 5.1

On Agent Instructions in the Wild

Of the 14,902 candidate instructions extracted from the corpus, 69.3% are explicit and only 58.6% are also actionable (Table 2). These figures describe how ACF authors write guidance in practice, independent of whether an agent later complies with it. Prompt engineering work argues that instruction clarity and specificity are key to reliable LLM behavior [7, 28]: vague or non-actionable guidance puts the burden of interpretation on the model. Read against this, our extraction funnel suggests that a large share of ACF content in the wild is written more like documentation for human readers than instruction for an agent: nearly a third of statements are not explicit, and one in six of the explicit ones does not prescribe a concrete, performable action. This holds regardless of the compliance question we answer in RQ3 . An instruction can still be followed well by an agent that is capable of resolving its ambiguity, but low explicitness or actionability is still an authoring weakness. It limits how reliably an instruction steers behavior across different agents and models. Observability is a separate concern from explicitness and actionability. An instruction can be perfectly explicit and actionable to carry out, like “run the linter before committing,” while leaving no trace in the artifacts an outside observer can check. This matters for building automated compliance mechanisms, since only instructions that leave such a trace can ever be audited without looking at an agent’s internal reasoning. Among explicit instructions, observability is the most selective of our criteria, and it hits process-oriented guidance hardest: Build and Run instructions drop from 8.8% of explicit-and-actionable instructions to 2.9% of the final dataset. The losses continue further down the pipeline: only 62.9% of observable instructions turn into a working, corroborated script (Section 4.2.1), and only 49.8% of those ever meet an applicable commit (Section 4.3.1). In the end, just 2,053 of the original 14,902 candidate instructions (13.8%) become the basis for an actual, artifact-grounded compliance observation. 5.2

On Instruction Compliance

Compliance is strongly bimodal: 54.4% of the 2,053 evaluated instructions are followed in every applicable commit checked, 37.3% are followed in none, and only 8.3% fall strictly in between (Section 4.3.2). Based on our artifact-level proxies, ACF-based governance behaves less like a dial agents turn to a partial setting, and more like a switch. Part of this shape is mechanical: each commit yields a binary verdict and the median instruction is checked against only 3 commits, so intermediate rates are hard to obtain. But the split survives when instructions with fewer than 3 or 5 applicable commits are excluded (Section 4.3.3), so it is not only an artifact of thin evidence. This has a practical meaning: non-compliance is concentrated in instructions that are essentially never followed, rather than spread thinly across all of them. Checking a handful of rules therefore gives little assurance about the rest: a review that happens to sample consistently followed instructions will miss the 37.3% that are never respected. The picture is also likely optimistic rather than pessimistic: most compliance-checking scripts are permissive, so any leftover leakage through the applicability filter can only push compliance figures up, never down. Reported non-compliance is a floor, not a ceiling. Non-compliance is also not spread evenly across the corpus. Development Process instructions comply significantly less often than Build and Run, Documentation, and Implementation Details (mean 34.0%, median 0.0%), though this specific gap does not hold up under a sensitivity check ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:26

Voria et al.

that restricts the comparison to instructions with more applicable-commit evidence (Section 4.3.3), so we treat the direction as suggestive rather than firmly established. This is plausibly because workflow and collaboration conventions, such as commit formatting or branching practices, sit at the edge of what any artifact-based proxy, or the agent itself, reliably treats as a binding rule rather than as background context. At the other end, Implementation Details and Documentation, two of the three categories that together make up 76.6% of the final RQ1 dataset (Table 3), are followed in a majority of cases (means of 63.6% and 60.2%, medians of 100%). The third, Testing, complies at a noticeably lower rate (mean 49.2%, median 37.5%) and significantly less often than Implementation Details, the one category-level difference that holds at every evidence threshold (Section 4.3.3). ACF-based governance works best for code-level conventions, and less well for testing and process-oriented guidance. Agent identity matters too, as a separate axis from category. Copilot-authored commits comply with their own context file at a significantly higher mean rate (66.0%) than either Claude Code (52.2%) or OpenAI Codex (51.2%), which are statistically indistinguishable from each other (Section 4.3.2). Operationalization and coverage rates are nearly uniform across agents (Sections 4.2.1, 4.3.1), so the gap does not stem from which instructions could be measured for each agent. It is consistent with real differences in how each agent’s tooling loads, prioritizes, or re-asserts ACF content during a session, but our agent-level comparison does not control for the different category mix of each agent’s ACFs, so part of the gap may reflect what those ACFs ask for rather than how faithfully each agent follows them. This is consistent with Huang et al. [17], who find that developers already compensate for this uncertainty through active, ongoing supervision rather than trusting agents by default. We treat this as an association, not an established cause: category mix, differences in repositories and languages across agents, and the uneven attribution precision across agents noted under external validity, remain plausible contributing factors we cannot fully rule out. Of the two pairwise gaps behind this result, Copilot’s edge over Claude Code holds up under a sensitivity check that restricts the comparison to instructions with more applicable-commit evidence; its edge over OpenAI Codex specifically does not (Section 4.3.3), and should be weighted accordingly. 5.3

Implications for ACF Design

Together, the extraction-funnel and compliance results speak directly to how ACFs should be written. The gap between explicit and actionable instructions (Section 4.1) shows that a meaningful share of ACF content is written closer to documentation than to agent-directed instruction. This is an authoring problem with a concrete fix: ACF authors should write for the agent, not the reader. State one concrete behavior per rule, use imperative phrasing, and avoid vague qualifiers such as “when appropriate” or “if needed,” which put the burden of interpretation on the model instead of resolving it in the file itself. Observability adds a second, mostly separate authoring concern. An instruction can be explicit and actionable and still leave no trace an outside observer, human or automated, can check. Since only observable instructions can be audited without looking at an agent’s internal reasoning, developers should prefer instructions that are checkable from the artifacts an agent modifies, such as file properties, naming conventions, or structural rules, over process-only directives whose execution leaves nothing behind. Tool vendors could support this directly at authoring time, flagging non-observable instructions before the gap is discovered later, as our own extraction pipeline does after the fact. The category-level compliance results (Section 4.3.2) point to a further, more tentative design implication, offered with the caveat that the pairwise comparison behind it is not solid under a sensitivity check (Section 4.3.3). Development Process instructions are descriptively both harder to ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:27

observe and less complied with than Implementation Details, Testing, or Documentation. Workflowlevel conventions, such as commit message formatting or branching discipline, are plausibly the kind of guidance that natural-language phrasing in an ACF is least suited to carry reliably. If this pattern holds up under further evidence, authors may be better served by encoding such conventions directly into project tooling, such as commit hooks, branch-protection rules, or CI configuration, than by relying on prose an agent may or may not follow consistently. We flag this as a direction worth checking further, not a settled recommendation.  ACF authors should write one concrete, imperative, artifact-checkable behavior per rule, avoiding vague qualifiers and process-only directives whose execution leaves no trace. Tool vendors should flag non-observable instructions at authoring time. And workflow-level conventions, Development Process guidance in particular, though this specific category-level result is descriptive rather than statistically solid (Section 4.3.3), may be better encoded in project tooling than left as ACF prose. 5.4

Implications for Engineering Agentic Systems

Beyond how ACFs are written, our results speak to how agentic systems should be built to govern the instructions ACFs contain. The bimodal shape of compliance (Section 4.3.2) means spot-checking is a poor substitute for systematic verification. Because non-compliance concentrates in instructions that are essentially never followed, checking a few rules says little about the others: sampling rules that happen to be respected will miss those that are consistently ignored. This points toward automated, continuously-run compliance checks over occasional review, aimed specifically at the instruction categories and agents our results flag as least reliable, rather than applied evenly across an ACF whose instructions are, as shown above, far from equally reliable. The agent-identity effect strengthens this point beyond day-to-day monitoring: it bears on how verification mechanisms should be designed in the first place. Compliance differences across Copilot, Claude Code, and OpenAI Codex do not stem from differences in what could be operationalized or evaluated for each agent, so reliability of ACF adherence may be better understood as a property of the ACF and agent together than of the ACF alone, though, as noted above, we cannot fully rule out category-mix, repository-, or language-level confounds. This reliability should not be assumed to carry over if a team switches to a different agent or adopts a new one. Verification mechanisms built on top of an ACF therefore cannot be agent-agnostic by default. They need to be calibrated, or re-checked, per agent. Our own measurement pipeline (Section 3.4) is a concrete example of the kind of mechanism this argues for. Sampling scripts and fixtures independently from each instruction’s behavioral descriptor, and accepting only scripts that are both minimally correct and independently corroborated, is itself a verification layer grounded in artifacts, sitting alongside the ACF’s natural-language content rather than inside it. That 62.9% of observable instructions could be operationalized this way suggests that automated, per-instruction compliance checking is practical at scale for a large share of ACF content today, not only a theoretical possibility. It is not yet hands-off: our spot-check found roughly three in ten accepted scripts implementing their instruction incorrectly (Section 4.2.2), so generated checks still need human review before they are trusted as enforcement. Taken together, these results support the broader point that ACFs are one layer of governance, not sufficient on their own. Natural-language instructions are still needed to communicate intent, but for the categories and agents our findings show to be unreliable, they need to be paired with independent, artifact-grounded enforcement, such as automated compliance checks, CI-integrated verification, or agent-specific safeguards, rather than trusted on their own.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:28

Voria et al.

Practitioners should pair ACFs with automated, continuously-run compliance checks aimed at the least reliable categories and agents rather than spot-checking. Verification mechanisms should be calibrated per ACF-and-agent pair rather than assumed to carry over across agents. And our own test-driven script-generation pipeline shows that artifactgrounded, per-instruction verification is practical today for a majority of observable ACF content, provided generated checks are reviewed before being used as enforcement. 6

Threats to Validity

This section outlines potential threats to the validity of our study and the mitigation strategies implemented. Internal validity concerns whether observed relationships can be attributed to the studied factors. A first threat is the absence of a counterfactual: we never observe what an agent would produce in the absence of a given instruction, so compliance may partly reflect ecosystem or model defaults rather than the ACF itself. This is tempered by our explicitness/actionability criteria, which favor project-specific directives over generic best practices, for which alternative explanations are weaker; a full ablation, however, is out of scope. A second threat concerns snapshot–commit linking: if a reconstructed snapshot does not match what the agent actually used at generation time, compliance may be misattributed. We mitigate this through content-based deduplication of snapshots and by linking each commit to the snapshot version active when it was generated (Section 3.2), backed by automated completeness checks. A third threat concerns the agent-level comparison in RQ3 : per-instruction compliance rates are compared across agents without controlling for instruction category, and the agents’ ACFs differ in category mix. Because compliance also varies by category, part of the observed agent gap may reflect what each agent’s ACFs ask for rather than how faithfully each agent follows them; together with repository- and language-level differences across agents, this is why we interpret the agent-level result as an association rather than a causal effect of agent identity. Another threat is residual noise in agent-authorship attribution, which relies on AIDev’s heuristic signals [20]; we mitigate this by restricting analysis to the strict agent subset, AIDev’s higherconfidence tier, without claiming a validated precision figure. Finally, the LLM components of our pipeline are inherently stochastic, and prior work shows LLMs can produce different code even under low-temperature or greedy decoding [25]; this is especially relevant for RQ2 , where script and fixture generation deliberately uses a higher temperature (0.8) for candidate diversity. We bound this threat with fixed inference parameters throughout, and, for RQ2 specifically, by construction: accepted scripts must clear the dual-bar criterion described below, so validity rests on empirical pass/fail evidence rather than on generation determinism. Construct validity concerns how well measures capture the intended concepts. A primary threat lies in the acceptance criterion used to select measurement scripts in RQ2 . We adapt CodeT’s dual execution agreement from exact-set consensus to a dual-bar criterion, consisting of a minimumcorrectness bar and a 60% overlap-based corroboration bar, because a pilot showed that exact agreement essentially never occurs at our sampling scale. Both thresholds were fixed from that same pilot rather than calibrated independently. In addition, an accepted script reflects only one valid operationalization of an instruction’s behavioral descriptor, so instructions that can be satisfied through several legitimate realizations may be undercounted, whereas the vacuous-compliance risk discussed below overcounts them. We partially mitigate this by manually spot-checking a statistically sized, category-stratified sample of accepted scripts (Section 3.4) and reporting the agreement rate between this check and the automated decision as a direct reliability measure. That check, however, also quantifies the residual threat: 67 of 94 sampled scripts (71.3%) correctly ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:29

implement their specification, so roughly three in ten accepted scripts measure something other than what the instruction prescribes. The most frequent error, hardcoding an example from the ACF as a literal requirement, makes a script stricter than its instruction and would bias the affected compliance rates downward, partly offsetting the optimistic bias of permissive defaults discussed below; we cannot quantify the net effect on RQ3 , and the spot-check itself relies on a single annotator. A related threat concerns the input given to scripts. Scripts receive the full before/after content of changed files, but not a line-level diff (Section 3.5). Instructions whose compliance criterion concerns only what a commit added or modified are therefore evaluated on the final state of each touched file, so pre-existing violations in a file the agent touched may be counted against the agent, and pre-existing compliant content may mask a violation the agent introduced elsewhere. Another threat is that compliance is recorded as a binary signal, which cannot capture partial or graded adherence; we treat this as a deliberate scope decision tied to our artifact-based observability criterion rather than an oversight. One more threat concerns reliance on the LLM-generated candidate observable behavior field as the functional specification for script generation; this is partially mitigated by its validation during RQ1 (Section 3.3), though that validation covers only the sampled subset. Finally, LLM-generated scripts tend to be permissive by default, reporting compliance absent contrary evidence; we mitigate the resulting risk of vacuous compliance by deriving instruction– commit applicability directly from each instruction’s own validated fixture file patterns (Section 3.5) rather than from coarser, hand-authored category-level rules, though this filter is only as complete as the fixture set it is derived from. It also matches files by extension or conventional basename, so a commit touching any file of the relevant type counts as applicable even when the touched file is unrelated to the instruction’s target; a more precise, content-aware applicability check would reduce the remaining risk of vacuous compliance. External validity concerns generalizability. Our findings are scoped to three agents—Copilot, Claude Code, and OpenAI Codex—and may not generalize to other coding agents. The strict agent subset used for our analyses, while improving attribution precision, is likely imbalanced across agents, since some expose clean bot identities while others rely on indirect signals such as branch naming; this could skew per-agent comparisons toward agents with more reliably attributable commits. Our corpus is further limited to public GitHub repositories containing an ACF, excluding enterprise or private agentic workflows that may use these artifacts differently. The pace of change in agent tooling and ACF conventions also means our results reflect a snapshot in time and may not hold for future agent versions. Finally, we rely on open-weight, locally executed models (Qwen 3.5 9B for annotation, Qwen2.5Coder-14B-Instruct for script generation) rather than frontier proprietary models; extraction quality and operationalization outcomes could differ with a different underlying model. Conclusion validity concerns the reliability of statistical inferences. Applicability filtering (Section 3.5) reduces the effective sample available per instruction, and instructions with zero applicable commits are excluded from the statistical analysis entirely, which may underpower comparisons involving rarer instruction types. Aggregating to one compliance rate per instruction before cross-group comparison guards against pseudo-replication, since commits sharing a snapshot or an agent are not independent observations; however, this aggregation still yields coarse, highvariance rates for instructions evaluated against very few applicable commits. We assess this directly: restricting both cross-group comparisons to instructions with at least 3 and at least 5 applicable commits (Section 4.3.3) leaves the bimodal compliance split and the Copilot-versus-Claude Code gap intact, but the Development Process category deficit and the Copilot-versus-OpenAI Codex gap ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:30

Voria et al.

do not survive Bonferroni correction at either threshold, indicating these two specific comparisons are underpowered rather than firmly established. Group sizes across categories and agents are also likely imbalanced, which we account for by using Kruskal-Wallis and post-hoc Mann-Whitney U tests with Bonferroni correction and Cliff’s delta effect sizes (Section 3.5), rather than assuming normality or equal-sized groups. Finally, the RQ1 dataset underlying all downstream statistics carries residual labeling noise: validation was performed by a single annotator, following Ortloff et al. [24]’s rationale for wellbounded coding tasks, and cross-model agreement between the two candidate LLM annotators reached only 76.1% on the observable label; this measurement error propagates into the RQ2 and RQ3 results built on top of it. 7

Conclusion

This paper investigated whether instructions documented in agent context files (ACFs) are reflected in the behavior of autonomous coding agents, structured around three research questions covering instruction characterization, operationalization, and compliance. Analyzing 621 ACF snapshots and 6,541 agent-generated commits across 268 repositories, we found that 6,552 instructions are explicit, actionable, and observable; that 62.9% of these can be operationalized into executable measurement scripts; and that, among the 2,053 instructions ultimately evaluable against real commits, mean compliance is 59.8%, with a strongly bimodal split between instructions followed universally and instructions never followed. Compliance varies significantly by instruction category and by originating agent, indicating that adherence to ACF guidance is neither uniform nor automatic. This study is scoped to publicly available repositories using three widely adopted agents and relies on LLM-assisted extraction and generation validated through manual sampling; results should be interpreted within this context rather than as universal claims about agent behavior. Future work will extend this pipeline to additional agents and a broader set of ACF conventions, investigate the underlying causes of category- and agent-level compliance gaps, and explore mechanisms for enforcing ACF adherence at generation time rather than only measuring it after the fact. Acknowledgment We gratefully acknowledge the financial support of JSPS KAKENHI (JP24K02921, JP25K03100), JST ASPIRE (JPMJAP2415), and JST CREST (JPMJCR23M1, JPMJCR26X7). We also acknowledge the use of GPT and Claude to ensure the manuscript’s linguistic accuracy and readability and to partially generate and correct the data collection and analysis scripts. References [1] Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs replace manual annotation of software engineering artifacts?. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 526–538. [2] Baris Ardic, Mitchell Olsthoorn, and Andy Zaidman. 2026. Uncovering AGENTS. md: What Testing Guidance Open Source Projects Provide to Coding Agents. In International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, Benevento, Italy. [3] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188–2200. [4] Worawalan Chatlatanagulchai, Hao Li, Yutaro Kashiwa, Brittany Reid, Kundjanasith Thonglek, Pattara Leelaprute, Arnon Rungsawang, Bundit Manaskasemsak, Bram Adams, Ahmed E Hassan, et al. 2025. Agent READMEs: An empirical study of context files for agentic coding. ACM Transactions on Software Engineering and Methodology (2025). [5] Worawalan Chatlatanagulchai, Kundjanasith Thonglek, Brittany Reid, Yutaro Kashiwa, Pattara Leelaprute, Arnon Rungsawang, Bundit Manaskasemsak, and Hajimu Iida. 2025. On the use of agentic coding manifests: An empirical ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Do They Do as They Are Told? Investigating Coding Agents’ Adherence to Explicit Instructions in Context Files

X:31

study of claude code. In Proceedings of the 27th International Conference on Product-Focused Software Process Improvement (PROFES). Springer, 543–551. [6] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. CodeT: Code Generation with Generated Tests. arXiv:2207.10397 [cs.CL] https://arxiv.org/abs/2207.10397 [7] Antonio Della Porta, Gianmario Voria, Andrea Abbate, Raffaele Sulipano, Stefano Lambiase, Gemma Catolino, and Fabio Palomba. 2026. Toward measuring prompt quality: A preliminary investigation on prompt smells. In 2026 IEEE International Conference on Software Analysis, Evolution and Reengineering-Companion (SANER-C). IEEE, 293–300. [8] Ramtin Ehsani, Sakshi Pathak, Shriya Rawal, Abdullah Al Mujahid, Mia Mohammad Imran, and Preetha Chatterjee. 2026. Where do ai coding agents fail? an empirical study of failed agentic pull requests in github. 807–811 pages. [9] Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K Lahiri. 2024. Llm-based testdriven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering 50, 9 (2024), 2254–2268. [10] Robert Feldt, Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Towards autonomous testing agents via conversational large language models. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1688–1693. [11] Matthias Galster, Seyedmoein Mohsenimofidi, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, and Sebastian Baltes. 2026. Configuring Agentic AI Coding Tools: An Exploratory Study. arXiv:2602.14690 [cs.SE] https://arxiv.org/abs/2602.14690 [12] Thibaud Gloaguen, Niels Mündler-Sasahara, Mark Niklas Müller, Veselin Raychev, and Martin Vechev. 2026. Evaluating AGENTS. md: Are repository-level context files helpful for coding agents? arXiv preprint arXiv:2602.11988 (2026). [13] Ahmed E Hassan, Hao Li, Dayi Lin, Bram Adams, Tse-Hsun Chen, Yutaro Kashiwa, and Dong Qiu. 2025. Agentic software engineering: Foundational pillars and a research roadmap. arXiv preprint arXiv:2509.06216 (2025). [14] Andre Hora and Romain Robbes. 2026. Are coding agents generating over-mocked tests? an empirical study. In Proceedings of the 23rd International Conference on Mining Software Repositories. 335–346. [15] Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, and Ahmed E Hassan. 2025. Agentic Refactoring: An Empirical Study of AI Coding Agents. arXiv preprint arXiv:2511.04824 (2025). [16] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79. [17] Ruanqianqian Huang, Avery Reyna, Sorin Lerner, Haijun Xia, and Brian Hempel. 2025. Professional Software Developers Don’t Vibe, They Control: AI Agent Use for Coding in 2025. arXiv preprint arXiv:2512.14012 (2025). [18] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2026. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology 35, 2 (2026), 1–72. [19] Nam Le Hai, Dung Manh Nguyen, and Nghi DQ Bui. 2025. On the impacts of contexts on repository-level code generation. In Findings of the Association for Computational Linguistics: NAACL 2025. 1496–1524. [20] Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. 2025. The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering. arXiv:2507.15003 [cs.SE] https://arxiv.org/ abs/2507.15003 [21] Jai Lal Lulla, Seyedmoein Mohsenimofidi, Matthias Galster, Jie M. Zhang, Sebastian Baltes, and Christoph Treude. 2026. On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents. arXiv:2601.20404 [cs.SE] https: //arxiv.org/abs/2601.20404 [22] Noble Saji Mathews and Meiyappan Nagappan. 2024. Test-driven development and llm-based code generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1583–1594. [23] Seyedmoein Mohsenimofidi, Matthias Galster, Christoph Treude, and Sebastian Baltes. 2025. Context engineering for AI agents in open-source software. arXiv preprint arXiv:2510.21413 (2025). [24] Anna-Marie Ortloff, Matthias Fassl, Alexander Ponticello, Florin Martius, Anne Mertens, Katharina Krombholz, and Matthew Smith. 2023. Different researchers, different results? analyzing the influence of researcher experience and data type during qualitative analysis of an interview and survey study on security advice. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21. [25] Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2025. An empirical study of the non-determinism of chatgpt in code generation. ACM Transactions on Software Engineering and Methodology 34, 2 (2025), 1–28. [26] Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li. 2025. Agentif: Benchmarking instruction following of large language models in agentic scenarios. [27] Abbas Sabra, Olivier Schmitt, and Joseph Tyler. 2025. Assessing the Quality and Security of AI-Generated Code: A Quantitative Analysis. arXiv preprint arXiv:2508.14727 (2025). [28] Yuya Sasaki, Hironori Washizaki, Jialong Li, Dominik Sander, Nobukazu Yoshioka, and Yoshiaki Fukazawa. 2024. Systematic literature review of prompt engineering patterns in software engineering. In 2024 IEEE 48th Annual

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:32

Voria et al.

Computers, Software, and Applications Conference (COMPSAC). IEEE, 670–675. [29] Yuchen Shao, Yuheng Huang, Jiawei Shen, Lei Ma, Ting Su, and Chengcheng Wan. 2024. Are llms correctly integrated into software systems? arXiv preprint arXiv:2407.05138 (2024). [30] Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. 2023. Repository-level prompt generation for large language models of code. In International Conference on Machine Learning. PMLR, 31693–31715. [31] Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 conference on empirical methods in natural language processing. 930–957. [32] Christoph Treude. 2024. Qualitative data analysis in software engineering: Techniques and teaching insights. In Handbook on Teaching Empirical Software Engineering. Springer, 155–176. [33] Gianmario Voria, Yutaro Kashiwa, Gemma Catolino, and Fabio Palomba. [n. d.]. Replication Package. https://github. com/gianwario/Agents-Context-Files-Analysis [34] Peiding Wang, Li Zhang, Fang Liu, Lin Shi, Minxiao Li, Bo Shen, and An Fu. 2025. CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation. arXiv:2503.22688 [cs.SE] https://arxiv.org/abs/2503.22688 [35] Yanlin Wang, Wanjun Zhong, Yanxian Huang, Ensheng Shi, Min Yang, Jiachi Chen, Hui Li, Yuchi Ma, Qianxiang Wang, and Zibin Zheng. 2025. Agents in software engineering: Survey, landscape, and vision. Automated Software Engineering 32, 2 (2025), 70. [36] Miku Watanabe, Hao Li, Yutaro Kashiwa, Brittany Reid, Hajimu Iida, and Ahmed E Hassan. [n. d.]. On the use of agentic coding: An empirical study of pull requests on github. ACM Transactions on Software Engineering and Methodology ([n. d.]). [37] Wenhao Yang, Runzhi He, and Minghui Zhou. 2026. A First Look at Coding Agents’ Compliance with AI Contribution Rules in Open-Source Communities.

Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Related documents

• What May an Agent Change About Itself? A Containment Floor for Self-Configuring Agent Runtimes#1202693
• Agent trajectories as programs: fingerprinting and programming coding-agent behavior#282865
• Towards a Middle-Ground Theory of Agency for Artificial Intelligence#749198
• When Agent Context Goes Stale: Incoherence in Volatile Agent Context#1210389
• When Agent Context Goes Stale: Incoherence in Volatile Agent Context#1201294