Do AI Coding Agents Log Like Humans? An Empirical Study
arXiv:2604.09409v1 [cs.SE] 10 Apr 2026
YOUSSEF ESSEDDIQ OUATITI, Queen’s University, Canada MOHAMMED SAYAGH, ETS - Québec University, Canada HAO LI, Queen’s University, Canada AHMED E. HASSAN, Queen’s University, Canada Software logging is essential for maintaining and debugging complex systems, yet it remains unclear how AI coding agents handle this non-functional requirement. While prior work characterizes human logging practices, the behaviors of AI coding agents and the efficacy of natural language instructions in governing them are unexplored. To address this gap, we conduct an empirical study of 4,550 agentic pull requests across 81 open-source repositories. We compare agent logging patterns against human baselines and analyze the impact of explicit logging instructions. We find that agents change logging less often than humans in 58.4% of repositories, though they exhibit higher log density when they do. Furthermore, explicit logging instructions are rare (4.7%) and ineffective, as agents fail to comply with constructive requests 67% of the time. Finally, we observe that humans perform 72.5% of post-generation log repairs, acting as “silent janitors” who fix logging and observability issues without explicit review feedback. These findings indicate a dual failure in natural language instruction (i.e., scarcity of logging instructions and low agent compliance), suggesting that deterministic guardrails might be necessary to ensure consistent logging practices. CCS Concepts: • Software and its engineering → Software development techniques. Additional Key Words and Phrases: Software logging, coding agents, agentic coding, large language models ACM Reference Format: Youssef Esseddiq Ouatiti, Mohammed Sayagh, Hao Li, and Ahmed E. Hassan. 2026. Do AI Coding Agents Log Like Humans? An Empirical Study. 1, 1 (April 2026), 24 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
Large Language Models (LLMs) are transforming software engineering by enabling AI coding agents to generate and submit code changes. Unlike simple code completion tools, these agents interpret high-level goals, plan tasks, and execute pull requests (PRs) with minimal human intervention [33, 35, 39]. However, as agents take on more responsibility, they must adhere not only to functional requirements (i.e., passing tests) but also to non-functional requirements (NFRs) such as observability. Observability is a critical NFR for diagnosing failures and monitoring system health [13, 28, 37], and it is primarily realized through logging. Yet, in traditional development, logging practices are often informal and learned through experience or tribal knowledge [26]. Additionally, developers must balance the trade-off between providing sufficient context and avoiding excessive noise [13, 37]. For instance, a lack of logging leads to limited runtime information and a reduced ability to diagnose Authors’ Contact Information: Youssef Esseddiq Ouatiti, [email protected], Queen’s University, Kingston, Canada; Mohammed Sayagh, ETS - Québec University, Montreal, Canada, [email protected]; Hao Li, Queen’s University, Kingston, Canada, [email protected]; Ahmed E. Hassan, Queen’s University, Kingston, Canada, ahmed@cs. queensu.ca. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM XXXX-XXXX/2026/4-ART https://doi.org/10.1145/nnnnnnn.nnnnnnn , Vol. 1, No. 1, Article . Publication date: April 2026.
2
Ouatiti et al.
issues [38]. Logging too much, however, can cause system overhead and make logs noisy and difficult to analyze [36]. It remains unknown whether AI agents can navigate these trade-offs or whether they simply replicate insecure or overly verbose patterns present in their training data and the repository environments in which they operate. This uncertainty presents a significant gap in our understanding of agentic logging. While recent studies have examined the functional correctness and acceptance rates of agentic PRs [7, 32], the observability gap remains unaddressed. For instance, it is unknown whether agents mimic human logging habits or whether developers effectively instruct agents to maintain logging and observability standards in the first place. Without this knowledge, practitioners risk integrating agents that produce opaque and unmaintainable code. To address this gap, we conduct an empirical study of logging practices in agent-generated code. We leverage the AIDev dataset [15] to analyze 4,550 agentic PRs and 3,276 human PRs across 81 well-maintained, mature, and popular repositories. We combine quantitative metrics with qualitative analysis of instructions and review comments to characterize the entire lifecycle of agentic logging. Our study addresses the following research questions (RQs): RQ1. How do logging practices in agentic pull requests differ from those in human pull requests? We observe that agents change logging less often than humans in 58.4% of the studied repositories. However, in repositories where both agents and humans add logs, agents introduce 30% more logs per 1,000 lines of code. While these agents successfully mimic human error-logging patterns, they are less consistent in matching human use of informational context (e.g., INFO level statements). RQ2. How prevalent are explicit logging instructions in issue descriptions and repository agent-instruction files? We find that logging instructions are rare (4.7%) and largely ineffective. For instance, agents fail to comply with logging requests 67% of the time, regardless of how specific those logging instructions are. RQ3. Is agentic logging regulated post generation, and by whom? We observe a hidden maintenance burden, as humans perform 72.5% of post-generation logging repairs. This regulation is mostly implicit, with humans fixing logging and observability issues in subsequent commits rather than requesting changes during code review. This paper contributes the first comparative analysis of logging practices between human and agentic contributors across mature and popular software repositories. Furthermore, it highlights the logging instruction gap and evaluates the compliance gap between human instructions and agent actions regarding logging. Finally, the paper offers a lifecycle analysis of post-generation logging regulation, quantifying the hidden maintenance burden placed on human reviewers. We share a replication package [20] which includes our code for conducting the studied experiments, so that others in the research community can replicate or extend our work. 2 Background & Related Work This paper targets the empirical characterization of logging practices within agentic workflows in Open Source Software (OSS). We analyze how the introduction of AI coding agents impacts the implementation, instruction, and governance of software logging. We discuss the following research directions as they are the closest to our work. 2.1 Software Logging Practices Several studies have characterized how developers implement and maintain logging in real-world systems. Fu et al. [4] analyzed logging in large Microsoft systems and found that logging is highly contextual, typically appearing in specific scenarios such as exception handling, return value verification, and critical logic branches. Pecchia et al. [24] examined industrial safety-critical , Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
3
systems, observing that logging practices are largely informal and driven by individual developer expertise rather than standardized guidelines. Yuan et al. [37] analyzed failure data from distributed systems (e.g., Hadoop) and found critical gaps in logging coverage, noting that many software failures occurred without generating any log entries. Regarding the maintenance of logging code, Kabinna et al. [8] investigated the stability of logging statements in open-source projects, reporting that 20% to 45% of logging statements are modified over their lifetime, with many changes occurring shortly after introduction. Li et al. [16] highlighted the prevalence of duplication, finding widespread identical static messages that complicate automated analysis. Finally, qualitative studies by Li et al. [13] and Rong et al. [26] confirmed that while developers view logging as indispensable for debugging, they struggle with the trade-offs regarding code complexity and performance overhead. Our research extends this direction by investigating whether coding AI agents adhere to these established human patterns. While prior work characterizes human logging as an informal and unstable activity, it is unknown if AI agents replicate these behaviors (e.g., similar churn rates or coverage gaps) or exhibit distinct “machine-native” logging practices. We address this by investigating agentic logging practices against human baselines. 2.2
LLMs for Software Engineering
The application of Large Language Models (LLMs) has expanded to cover tasks ranging from code completion to the automation of more complex activities, including aspects of non-functional requirements (NFRs) such as observability. Mastropaolo et al. [19] introduced LANCE, a T5-based model that treats logging as a translation task. While it achieved 65.9% accuracy in placement, it struggled with semantic content, achieving only a 15.2% exact match for log messages. Xu et al. [34] advanced this work with UniLog, demonstrating that in-context learning and few-shot prompting can significantly improve message quality (BLEU-4 score of 27.1) without the cost of fine-tuning. However, recent empirical evaluations by Rodriguez et al. [25], utilizing GPT-4o, revealed a persistent bias toward “over-logging.” They found that while modern LLMs match human placement accuracy in approximately 64% of cases, they exhibit an over-logging rate of nearly 83%, often placing redundant instrumentation at the start or end of functions. Beyond logging, Licorish et al. [17] observed that while LLMs produce functionally correct code, they frequently introduce verbose structures with higher cyclomatic complexity. Additionally, Sandoval et al. [27] identified that LLMs are prone to reproducing insecure patterns present in their training data, such as hard-coded credentials. Our research complements these benchmark-driven studies with an in-situ analysis of how logging is actually produced by AI agents in real pull requests. While prior evaluations rely on isolated datasets (e.g., LANCE, UniLog), it remains unexamined how these “over-logging” and verbose tendencies manifest in active agentic workflows where humans must review and merge the code. We address this by characterizing agent-generated logging in real-world software projects. 2.3
AI-Assisted Development and Agentic Contributions in OSS
Recent empirical studies have begun to characterize the growing footprint of AI-generated contributions in open-source ecosystems. Watanabe et al. [32] analyzed 567 pull requests generated by the Claude Code agent, reporting an acceptance rate of 83.8% (comparable to human contributors) while noting that agents primarily focused on maintenance tasks such as refactoring and documentation, with 54.9% of PRs merged without human modification. He et al. [6] conducted a study of the Cursor assistant and found that although adoption yielded a transient 3 to 5 times increase in development activity, it coincided with a persistent 30% rise in static-analysis warnings and a 41% increase in code complexity, highlighting a trade-off between speed and quality. Wang et al. [31] identified a “programmatic bias” in agentic workflows, observing that agents resort to code-based solutions for , Vol. 1, No. 1, Article . Publication date: April 2026.
4
Ouatiti et al.
93.8% of tasks, often diverging from the GUI-driven workflows preferred by human developers. Finally, Tufano et al. [29] examined developer interactions with LLM-based bots in review processes, finding that while bots are frequently delegated review responsibilities, developers remain skeptical of their suggestions for non-trivial logic changes. Our work differs from this line of research in that, rather than focusing on functional correctness, code structure, or acceptance rates, we study logging as a mechanism for achieving the nonfunctional requirement of observability. Specifically, we analyze how logging is produced, explicitly instructed, and regulated within AI-authored pull requests in open-source projects. 3
Data Collection and Processing
As the goal of our paper is to understand whether AI agents introduce logging instructions in the same way as human developers, we leverage the AIDev dataset [15] that contains human and agentic Pull Requests (PRs). From the dataset, we select a set of repositories along with their agentic and human PRs (Section 3.1). From these PRs, we study the logging statements identified using a keyword-based approach (Section 3.2). To better understand the agentic behavior in PRs, we analyze how developers instruct agents in terms of the creation and maintenance of logging statements (Section 3.3). 3.1
Repository and PR Selection
Our data collection pipeline, shown in Figure 1, results in a set of 4,550 agentic and 3,276 human PRs across 81 repositories. This dataset is obtained from the AIDev dataset between December 2024 and July 2026. We leverage AIDev-pop, a subset of the AIDev dataset that includes repositories with at least 100 stars, comprising 33,596 agentic PRs and 6,618 sampled human PRs. We further apply a filter restricting to repositories with at least 500 stars to ensure that repositories contain both human and agentic PRs for project-level comparison. The repositories that are in the intersection of the two datasets account for 810 repositories, which together consists of 9,750 agentic and 6,569 human PRs. To enable a sound comparison between human and agentic PRs at the project level, we focus on repositories with at least 10 agentic PRs and 10 human PRs. This filtering step yields 130 repositories, with 6,843 agentic and 4,784 human PRs. We further restrict the dataset to repositories whose primary programming language is Python, Java, or JavaScript/TypeScript, similar to prior work on software logging [2, 14, 16, 21, 22]. We focus on these programming languages since they have well-defined strategies for identifying the logging statements (as further discussed in Section 3.2 below). For each PR, we collect its patch to determine whether it includes changes to logging statements. For each agentic PR, we also collect the associated instructions. To identify PRs with logging changes, we use the GitHub API to retrieve patches for human PRs, while patches for agentic PRs are already available in the AIDev dataset. Further details on logging statement identification are provided in the next subsection. As our study covers how agents are instructed, we further collect the instructions given to the agents. These instructions can be in the form of issues linked to PRs, repository-level instructions files (e.g., ./github/copilot-instructions.md) at the time of the PR creation, or comments provided during the review for agents to adjust their generated code. Note that instructions can also be provided through other channels that are not publicly available for collection and analysis. 3.2
Logging Detection Strategy
We identify logging statement changes within code diffs using a regex-based strategy adapted from prior logging studies [14, 16, 21, 22]. As detailed in Tables 1 and 2, we use regex expressions tailored , Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
5
Agentic PRs 33,596 (2807 repos)
Agentic PRs 9,750 (810 repos)
Agentic PRs 6,843 (130 repos)
Human PRs 6,618 (810 repos)
Human PRs 6,569 (810 repos)
Human PRs 4,784 (130 repos)
(1) Initial AIDev-pop (>100 Stars)
Filter shared repositories
PR Analysis Logging Presence (Human & Agent PRs)
Shared repos only
At least 10 agentic and human PRs each
Repos with significant amount of agentic & human PRs
Agentic PRs 4,550 (81 repos) Human PRs 3,276 (81 repos)
Agent Instructions Linked issues
Language filter (Py,Java,JS/TS)
Repo instructions Reviews
Final PR dataset
Fig. 1. Overview of our data collection pipeline. Table 1. Regex expressions used to identify logging statements (case-insensitive; re.I). Language Regex Pattern Python
\b(?:logging|logger|_logger)\. (?P<level>debug|info|warning|warn|error| critical|exception)\s*\(
Java
\b(?:logger|log|LOG|LOGGER)\. (?P<level>trace|debug|info|warn|warning| error|fatal)\s*\(
JS/TS
\bconsole\. (?P<level>log|info|warn|error|debug)\s*\(
to each programming language. These expressions are executed on source files based on their extensions, such as .py for Python. We explicitly exclude build artifacts (e.g., dist/, node_modules/), binary assets, and minified code to reduce noise from auto-generated files. The regex patterns capture logging framework invocations in Python (e.g., logging.info), object-oriented styles in Java and JavaScript (e.g., LOGGER.warn), and console logging in JavaScript/TypeScript (e.g., console.log). Generic print statements (e.g., System.out.println) are excluded, as they do not represent typical production-level logging. To ensure the robustness of our regex expressions, we perform a manual analysis of a representative sample of 380 diffs (95% confidence interval and 5% margin of error) and find that they achieve a precision of 96% and a recall of 94%, demonstrating the reliability of our regex-based approach. Note that we exclude 4 repositories in which neither agentic nor human PRs contain logging changes, resulting in a final dataset of 77 repositories. , Vol. 1, No. 1, Article . Publication date: April 2026.
6
Ouatiti et al.
Table 2. File extensions scanned and path/suffix exclusions used to reduce noise during logging statement identification. Language Extensions
Exclusions (Paths & Suffixes)
Python
.py
Dirs: build/, dist/, site-packages/, vendor/ Files: _test.py, test_*.py
Java
.java
Dirs: target/, bin/, build/ Files: Test.java, *Test.java
JS/TS
.js, .jsx, .ts, .tsx
Dirs: node_modules/, dist/, public/, vendor/ Suffixes: .min.js, .map, .gz, .bundle.js, .worker.js
Table 3. Agent instruction files and their corresponding regex patterns used for identification. Agent
Regex pattern
Devin
**/PULL_REQUEST_TEMPLATE/DEVIN_PR_TEMPLATE.md; **/PULL_REQUEST_TEMPLATE/devin_pr_template.md .cursor/*; .cursorrules; **/*.mdc .github/copilot-instructions.md; .github/instructions/* .claude/*; CLAUDE.md; .github/workflows/claude*.yml **/AGENTS.override.md; **/TEAM_GUIDE.md; **/.agents.md **/AGENTS.md
Cursor Copilot Claude Codex Common
3.3
Studying Agent Instructions
We collect the available instructions for the studied repositories at the creation time of each individual agentic PR, as discussed in Section 3.3.1. We then leverage an LLM-as-judge multi-agent approach to identify the logging intent of developers (e.g., creation of a new logging statement), as discussed in Section 3.3.2. 3.3.1 Collection of Agent Instructions. In this paper, we study how developers instruct agents to generate and maintain logs. The instruction dataset consists of three sources, as described below. • Linked issues: Developers can create an issue describing a task and assign it to an agent, which then addresses the task through a PR. Issues associated with agentic PRs are available in the AIDev dataset and are used to analyze how developers provide logging-related instructions. Not all agentic PRs have associated issues, as agents may co-author PRs offline with human developers who subsequently submit them. • Repository-level agents’ instruction files: Developers can guide agents through repositorylevel instruction files (e.g., CLAUDE.md and .github/copilot-instructions.md). For each agentic PR, we retrieve the instruction files present in the repository at the time the PR was created. These files are identified using the regex patterns shown in Table 3. • Review comments on agentic PRs: Developers can leave review comments under a PR to guide agents. We collect review comments available in the AIDev dataset as a means of capturing instructions provided to the AI agent to adjust its generated code. 3.3.2 Identifying Developers’ Logging-Related Intents. To identify the intents of developers behind logging instructions (i.e., Add, Remove, or Modify), if any, we use a multi-agent LLM-as-judge protocol [11, 12, 30] on review comments, instruction files, and associated issues. Each of these , Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
7
Table 4. Log levels categorized by verbosity across analyzed languages. Verbosity
Python
Java
JS/TS
Highest ↓ ↓ Lowest
debug info warning error, critical
trace, debug info warn error, fatal
debug info, log warn error
Table 5. Mapping of language-specific keywords to unified syntactic contexts. Unified Context
Python Keywords
Java & JS/TS Keywords
Conditionals Loops Try/Catch Unnested
if, elif, else if, else, switch, case for, while for, while, do try, except, finally try, catch, finally Logging statement is not enclosed in any of the above blocks
data points is studied separately to identify whether it has a logging instruction. If so, which of the three possible typical intents (Add, Remove, or Modify) is provided. To do so, we prompt three frontier models (GPT-4o, GLM-4.7, and DeepSeek-V3.2) to independently classify a text input (e.g., a code review comment) using the prompt shown in Figure 2, whose construction is discussed below. From the three votes, we assign the final label for each instruction source (e.g., a review comment) by majority voting. To construct our prompt, we follow a similar approach to previous work [11, 33]. We first establish ground truth labels for 100 samples through manual annotation. Using this sample, we iteratively refine the jury prompt, measuring the agreement between the jury’s majority-vote prediction and our manually curated ground truth. We finalize the prompt once this agreement reaches Cohen’s 𝜅 = 0.83, ensuring that the automated classification reliably mirrors human intent. 4
Results
RQ1. How do logging practices in agentic pull requests differ from those in human pull requests? Motivation: The goal of this research question is to determine whether AI agents mimic human developers in the creation and maintenance of logs within a given project. In other words, if humans frequently insert logs in a particular way, do agents follow the same logging practices? Understanding this behavior helps identify whether agents are capable of automatically recognizing and following developers’ practices in creating logging statements, or whether they neglect such practices, potentially diminishing the observability of a software system. The results of this research question motivate the need to equip AI agents with tools that analyze and respect project-specific observability practices, if agents do not follow human logging conventions, and to encourage developers to be more explicit about their logging expectations. Approach: To identify whether AI agents mimic developers in the creation and maintenance of logging statements, we compare agentic PRs with human PRs within the same project. For each of the 77 repositories in our dataset, we calculate the following metrics to characterize logging practices for both agentic and human PRs: , Vol. 1, No. 1, Article . Publication date: April 2026.
8
Ouatiti et al.
LLM Jury Prompt for Instructions (Issues, Reviews & Repo Files) System Message: You are an expert reviewer analyzing text files for logging-related directives. The text file could either be an issue body, a repository-level agent instruction file, or a PR review comment. User Message Template: Task: Analyze the provided text. Does it contain an explicit instruction regarding logging, tracing, observability? Output Labels (Choose one): • ADD: Requests for new logs (quantity control). Examples: “Add debug logs”, “Log all errors”, “Ensure observability”, “Include logging statements”, “Always log exceptions”. • REMOVE: Requests to suppress logs (noise control). Examples: “No console.log in production”, “Reduce verbosity”, “Don’t add debug logs”, “Avoid excessive logging”. • MODIFY: Requests to modify existing logs (quality control). Examples: “Use slf4j instead of System.out”, “Change info to debug level”, “Use structured logging format”, “Follow our logging conventions”. • none: No specific logging instructions found. Important: • Only classify as ADD, REMOVE, or MODIFY if there is an EXPLICIT instruction about logging behavior. • Vague mentions of “good practices” or “code quality” without specific logging guidance should be NONE. • If the instruction says to USE logging (e.g., “Use console.log for debugging”), that is ADD. • If the instruction specifies HOW to log but not whether to add/remove, that is MODIFY. Return a compact JSON object: {“label”: “...”, “rationale”: “...”}. Type of text: {metadata} Text: {text}
Fig. 2. The prompt used to identify logging instructions. The metadata injection provides the model with the source type (i.e., issue, repo_instruction, or review_comment)
• Logging Prevalence: We measure the percentage of PRs that explicitly introduce, modify, or remove at least one logging statement. This metric is computed separately for human and agentic PRs. • Log Density: The number of modified logging statements per 1,000 modified lines of code (LOC). • Message Characteristics: We study message verbosity and log levels. Verbosity is measured as the number of characters in extracted literal log message text. Log level distributions are computed using language-specific logging patterns (Table 4). At the repository level, log message length comparisons are performed only when both agentic and human PRs contain , Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
9 100 Humans more (score < 0.5)
Repos: 77 Median Human: 23.5% Median Agent: 18.5%
Agents more (score > 0.5) Equal (score = 0.5) Equality baseline (score = 0.5)
80
80
Agent Logging Prevalence (%)
Logging Prevalence Across Repositories (%)
100
60
40
40
20
20
0
60
0
Human
Agent
(a) Distribution of repository-level logging prevalence for human and agentic PRs.
Mean score: 0.46 Median score: 0.45 Agents more: 29 repos Humans more: 45 repos Equal: 3 repos Total: 77 repos
0
20
40 60 Human Logging Prevalence (%)
80
100
(b) Paired repository-level comparison of logging prevalence (Human on x-axis, Agent on y-axis).
Fig. 3. Repository-level comparison of logging prevalence in human and agentic PRs. Panel (a) shows the distribution across repositories; panel (b) shows the paired per-repository comparison.
at least one extractable log message. Consequently, repositories without extractable messages on at least one side (e.g., those with only variable-based or dynamically constructed messages) are excluded from the verbosity analysis. For project-level comparisons, we use the median log message length and the median log level percentages computed across PRs. • Syntactic Context: The distribution of log placement within control-flow constructs (e.g., if, try/catch, and Unnested). To account for language differences in our diff-based analysis, we map language-specific keywords found in the diff context to unified categories, as shown in Table 5. For example, both Python’s except and Java’s catch are mapped to the unified try/catch category. For project-level metrics, we calculate the median across all PRs of that project. For each control-flow category (e.g., try/catch), we calculate the median percentage of logs placed in that construct across all PRs in the project. We focus on these metrics to characterize logging practices, as they capture the main aspects of logging statements and align with prior literature on the development and maintenance of software logging [1]. Maintenance effort, log density, verbosity, log level distribution, and syntactic context have all been studied in prior work [1, 3, 14, 16]. For each project and PR type (i.e., human or agentic), we calculate one project-level metric value as the median across all PRs of that type (e.g., median logging prevalence across all human PRs, and separately across all agentic PRs). We compare agentic to human PRs using a normalized score Agentic computed as Agentic+Human . This score maps each repository to a common 0–1 scale and remains defined as long as at least one side is non-zero. A score of 0.5 indicates parity, values above 0.5 indicate higher agentic values, and values below 0.5 indicate higher human values. Results: In 58.4% of the studied repositories, human pull requests change logging (i.e., add, modify, or remove) more often than agent pull requests, as shown in Figure 3. This means that, within the same repository, humans are more likely to add, remove, or modify logging statements when they change code. Specifically, in 45 out of 77 repositories (58.4%), the agentic-to-human logging prevalence score is below 0.5, indicating that agents touch logging in a smaller share of their PRs than humans do. In contrast, 29 repositories (37.7%) show the opposite trend, where , Vol. 1, No. 1, Article . Publication date: April 2026.
10
Ouatiti et al.
agents touch logging more often than humans. This difference between agent and human logging prevalence across the same repositories is statistically significant (𝑝 = 0.019). Moreover, the median score is 0.45, suggesting that for a typical project in our dataset, agents change logging about 16% less often than humans. Finally, we observe that among the 22 repositories (28.6%) with similar logging prevalence scores (from 0.44 to 0.55), logging prevalence varies substantially, ranging from 5.6% to 66.7%, with medians of 26.5% for agents and 25.3% for humans. In 50.6% of the studied repositories, agentic pull requests have higher log density than human pull requests, as shown in Figure 4. However, the paired repository-level difference in log density is not statistically significant (𝑝 = 0.274). The median agentic-to-human density score is 0.51. Consequently, in a typical project, agent and human log density are nearly balanced, with a slight tilt toward agents. For example, in microsoft/ApplicationInsights-JS the mean log density is 12.90 for agents versus 1.03 for humans (score 0.93). Restricting the comparison to the 67 repositories (87.0%) where both agents and humans have logging-changing PRs reveals a stronger pattern as the median score rises to 0.56, which means that, in repositories where both sides actively add logs, agents produce about 30% more logging changes per 1,000 changed LOC than humans. However, as illustrated in Figure 5, this density gap is largely a composition effect across PR sizes rather than a fundamental difference in logging behavior. Because log density naturally decreases as PR size increases for both groups, the overall density metrics are skewed by the fact that agents typically make much smaller modifications (median 1,279 LOC versus 2,770.5 LOC for humans). Agent log-adding PRs are heavily concentrated in smaller ranges (46.8% for agents vs. 33.0% for humans for LOC ≤1,000), which naturally yield denser patches. Indeed, in the 48 repositories where agents make smaller changes than humans, they add 65% more logs per 1,000 LOC. Conversely, human log-adding PRs are concentrated in massive changes (52.7% for humans vs. 40.7% for agents for LOC >2,500). When controlling for this size disparity, the logging behaviors converge. In the 19 repositories where agents make larger changes than humans, agents become more conservative, adding 21% fewer logs. Similarly, when examining only large PRs (>2,500 LOC), median densities between the two groups become nearly identical (1.64 vs. 1.59 logs per 1,000 LOC). This pattern strongly supports a dynamic of selective delegation [33] as developers predominantly trust agents with smaller, tightly bounded tasks, while humans handle larger architectural integrations. Agents and humans write log messages of similar length at the repository level, as shown in Figure 6. Across the studied 77 repositories, log message length (measured in number of characters) is centered at parity, with a median score of 0.50. Furthermore, 63.6% (49 out of 77) of the repositories show similar log message lengths between agents and humans. In contrast, 22.1% (17 out of 77) of the repositories show agents writing substantially longer messages, while 14.3% (11 repositories) show humans writing substantially longer messages. For example, wix/react-nativeui-lib has a median agentic message length of 37 compared to 14 for humans (score 0.73), while jina-ai/node-DeepResearch shows the opposite pattern (17 vs. 52, score 0.25). The median score of 0.50 suggests that agents write messages nearly identical in length to humans, indicating they largely follow existing human practices. Agents largely mirror human conventions for most log levels, with notable exceptions for INFO and WARN messages, as shown in Figure 7. Specifically, agents show high adherence to project norms for general-purpose and error logging. In fact, the usage rates of the standard JS/TS logging method console.log1 and the ERROR log level are similar in 71.4% and 53.2% of our studied repositories respectively. The DEBUG log level also shows high alignment (similar in 64.9% of the repositories), with agents using it more in 22.1% of repositories and humans in only 13.0%. However, 1 Serves as the default, severity-neutral logging method in JS/TS
, Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
102
11
Repos: 77 Median Human: 1.29 Median Agent: 2.58
Humans denser (score < 0.5) Agents denser (score > 0.5) Equal (score = 0.5)
10
Agent Log Density (logs per 1K LOC)
Log Density (logs per 1K LOC)
Equality baseline (score = 0.5)
101
1
100
10−1
10−2
100
10−1
Mean score: 0.54 Median score: 0.51 Agents more: 39 repos Humans more: 38 repos Equal: 0 repos Total: 77 repos
10−2
Human
10−2
Agent
(a) Distribution of repository-level log density for human and agentic PRs.
10−1 100 Human Log Density (logs per 1K LOC)
101
(b) Paired repository-level comparison of log density (Human on x-axis, Agent on y-axis).
Fig. 4. Repository-level comparison of log density in human and agentic PRs. Panel (a) shows the distribution across repositories; panel (b) shows the paired per-repository comparison. Agent binned median Human binned median
Log Density (log changes per 1K LOC)
103
102
101
100
10−1
101
102
104
103
105
PR Size (LOC Changed)
Fig. 5. PR-level relationship between change size and log density. Points represent individual PRs (transparent). Lines represent binned medians: PR size is partitioned into 12 log-spaced bins over the combined agent and human LOC range, and for each group we plot the median LOC (x-axis) and median log density (y-axis).
divergence appears with INFO and WARN, as INFO is the level where humans most often exceed agents (24.7% of repositories), while WARN shows the lowest overall similarity (48.1%), with agents overusing it in 29.9% of repositories and humans in 22.1%. A manual inspection suggests that part of this gap may come from program-state confirmation messages (e.g., “operation completed”), which are more common in human-authored logs. Agents largely mirror human log placement conventions, with divergence in conditional and iterative contexts, as shown in Figure 8. Specifically, agents show high adherence to project norms for error-handling and top-level logging. In fact, the placement of logs in TRY_CATCH blocks and UNNESTED (top-level function body) contexts are nearly identical in 58.4% and 59.7% of our studied repositories respectively. However, divergence appears in control-flow contexts. , Vol. 1, No. 1, Article . Publication date: April 2026.
Ouatiti et al.
Repos: 77 Median Human: 30.0 Median Agent: 33.0 Median Score: 0.50
80
Humans longer (score < 0.5) Agents longer (score > 0.5) Equal (score = 0.5)
80
Agent Median Message Length
Median Log Message Length per Repository (characters)
12
60
40
20
Equality baseline (score = 0.5)
60
40
20
0
0
Human
Agent
(a) Distribution of repository-level log-message length for human and agentic PRs.
Mean score: 0.50 Median score: 0.50 Agents more: 37 repos Humans more: 31 repos Equal: 9 repos Total: 77 repos
0
20
40 60 Human Median Message Length
80
(b) Paired repository-level comparison of logmessage length (Human on x-axis, Agent on y-axis).
Fig. 6. Repository-level comparison of log-message length in human and agentic PRs. Panel (a) shows the distribution across repositories; panel (b) shows paired per-repository medians. Results are shown for the same 57 repositories with extractable message text on both sides.
console.log
71.4%
ERROR
15.6%
53.2%
DEBUG
27.3%
64.9%
INFO
15.6%
48.1%
0
20
19.5%
22.1%
59.7%
WARN
13.0%
24.7%
29.9%
40
60
13.0%
22.1%
80
100
Projects (%)
Fig. 7. Percentage of repositories in three categories for each log level: gray = similar usage, blue = agents use more, and tan = humans use more.
For instance, we observe for the CONDITIONAL blocks (if/else/switch) that only 46.7% of repositories show similar usage, with humans placing more logs in conditionals in 28.6% of repositories. The gap widens for LOOP contexts, where humans log significantly more in 32.5% of repositories. These log placement patterns resemble the log level findings, as agents match human practices for error-related contexts but are more conservative in locations where informational logging typically occurs (e.g., Loops). , Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
13
similar 46.7%
Conditional
28.6%
24.7%
similar 58.4%
Try/Catch
18.2%
23.4%
similar 41.5%
Loop
32.5%
26.0%
similar 59.7%
Unnested
16.9%
40
30
20
10
23.4%
0
10
Humans more Agents more 20
30
40
Projects (%)
Fig. 8. Aggregated syntactic-context comparison across repositories. Left bars show repositories where humans use the context more; right bars show repositories where agents use it more; center labels show the share of repositories with similar usage.
Summary of RQ1 In 58.4% of repositories, agents change logging in fewer PRs than humans (median prevalence score 0.45). Conversely, when both agents and humans do add logs, 58.2% of repositories show higher agent log density, often associated with smaller change sizes. Moreover, message length, log levels, and syntactic placement are broadly similar between agentic and human PRs. We recommend developers to review logging explicitly in agentic PRs, checking in particular for missing instrumentation and overly dense logging in small PRs. RQ2. How prevalent are explicit logging instructions in issue descriptions and repository agent-instruction files? Motivation: The goal of this research question is to determine how often human developers explicitly request logging when instructing an AI agent to perform a task. In other words, we quantify the prevalence of explicit logging requirements in the two primary instruction channels that typically guide an agent: the task specification (i.e., the linked issue description) and repository instruction files (e.g., AGENTS.md or CLAUDE.md). This distinction matters because, in traditional development, logging is often governed by implicit norms and learned practices. For example, a junior engineer may naturally add error logs in a catch block without being explicitly instructed to do so. Agents, however, largely rely on their training and what is stated in instructions (prompts). This difference creates an important ambiguity when interpreting the logging characteristics observed in RQ1. Specifically, it remains unclear whether the observed agentic logging practices are an intrinsic behavior built into the models themselves, or if agents require explicit, lateral logging instructions from developers to implement proper observability. Understanding how frequently humans specify logging, and how agents respond to these instructions, helps disentangle whether logging is a built-in property of the agent or a prompted action. The results of this research question inform whether improving observability in agentic contributions should primarily focus on better , Vol. 1, No. 1, Article . Publication date: April 2026.
14
Ouatiti et al. Repository-level Agent instructions Taxonomy Classification of logging instructions
xN
YES related to logging? NO
LLM Jury Our AIDev subset (popular projects with human and agentic PRs )
Extract instruction sources
Logging instructions: Add, Change, Remove
Specification/instructions not related to logging
Linked issues
Fig. 9. Overview of our approach to characterize agentic logging characteristics.
agent support for repository-specific logging conventions, on encouraging developers to be more explicit about their logging expectations, or on both. Approach: To characterize how humans instruct agents on logging, we analyze two instruction channels: (1) Task Specifications from linked issue descriptions (Copilot PRs), and (2) Repository Instructions from global instruction files (e.g., AGENTS.md, CLAUDE.md), as illustrated in Figure 9. We compute metrics at two units of analysis: instruction-level and PR-level. • Instruction-level metrics. For each detected logging instruction, we measure: (i) Intent (Add, Modify, Remove) using the LLM Jury protocol (Section 3.3), and (ii) Strength (Strong, Weak) via manual labeling by the first two authors (Cohen’s 𝜅 = 0.96). • PR-level metrics. For each agentic PR, we measure: (i) whether it is log-instructed (at least one explicit logging instruction from either Task Specifications or Repository Instructions) or log-uninstructed, and (ii) whether the PR changes logging. For log-instructed PRs, we additionally measure compliance, i.e., whether the final diff matches the instruction intent (Add, Modify, Remove). These measures capture whether humans provide logging instructions, what behavior they request, and how strong those instructions are. They also allow us to test whether explicit logging instructions are associated with different logging behavior in agentic PRs. Results: Explicit logging instructions are rare in the two instruction channels we analyze, as shown in Table 6. Among the 1,308 agentic PRs where at least one instruction channel is observable (a linked issue or a repository instruction file), only 4.7% (61 out of 1,308) are associated with any logging instruction. Furthermore, we observe zero overlap between instruction sources, as 15 PRs receive instructions solely from linked issues, and 46 receive them solely from repository instruction files. Moreover, we observe that repository instructions often act as cleanup rules, telling agents to delete logs or debug output before finishing. All 10 Remove instructions from repository files originate from a single project (dropseed/plain) and provide the same directive: use statements for debugging, but remove them before committing. This instruction does not ban logging. Instead, it guides the agent to use logs temporarily (for its internal debugging) and then clean them up. Indeed, we observe zero debug statements in the final code. However, we note that this 100% compliance with removal might be inflated by vacuous compliance, where agents may have simply never added debug statements in the first place, rather than actively removing them. Agents show a compliance gap regardless of instruction strength. In our analysis of task specifications (n=15) and repository instructions (n=46), we find that concrete wording alone does not ensure compliance. As shown in Table 7, at the issue level, 73.3% (11 out of 15) of logging instructions are strong, yet compliance among these strong instructions is only 27.3% (3 out of 11). At the repository-file level, all 46 labeled instructions are strong, but overall compliance remains , Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
15
Table 6. PR-level breakdown of logging instructions and agent compliance by instruction channel (Task Specifications vs. Repository Instructions). Count denotes the number of agentic PRs with at least one explicit logging requirement in that channel.
Instruction channel
Intent
Count
Compliant PRs
Compliance (%)
Task Specifications
Add Modify Remove
5 8 2
2 3 0
40.0% 37.5% 0.0%
Repository Instructions
Add Modify Remove*
36 0 10
3 0 10
8.3% 0.0% 100.0%
*“Remove debug instrumentation before commit.” Zero debug instrumentation added = 100% compliance.
Table 7. Impact of instruction strength on agent compliance (n=15 issue instructions).
Instruction Strength
Definition
Strong Weak
Specifics (files, levels, frameworks) Generic (“add logs”, “ensure observability”)
Count
Compliance
11 4
27.3% 50.0%
Table 8. PR-level logging prevalence by log-instruction status.
Group
Logging-change PRs
Total PRs
Prevalence (%)
Log-instructed Log-uninstructed
9 932
61 4489
14.8% 20.8%
All PRs
941
4550
20.7%
low (6.5%, 3 out of 46). Note that while task specifications are Copilot-linked and therefore visible to the model before generation, the visibility of repository instructions may differ across agent workflows. So non-compliance with repository instructions specifically may come from two causes: the file is not surfaced to the agent, or it is surfaced but ignored. Consequently, having a logging instruction in one of the two channels (i.e., task specifications or repository instructions) is not associated with a higher rate of logging changes. In fact, we find no statistical difference in logging behavior between log-instructed and log-uninstructed PRs. Agents receiving instructions changed logging in 14.8% of cases, while uninstructed agents changed logging in 20.8% of cases, as shown in Table 8. A Pearson’s 𝜒 2 test confirms this null result (𝜒 2 =1.32, p=0.25). Thus, simply adding a logging instruction to an issue or repository agent instruction file does not reliably alter the agent’s logging behavior. , Vol. 1, No. 1, Article . Publication date: April 2026.
16
Ouatiti et al. Review comments Automated/bot logging instructions
Extract Extract logging instructions (same as RQ2)
Our AIDev subset (popular projects with human and agentic PRs )
Human logging instructions
Filter logging instructions
Extract Agent logging survival analysis Extract the history of within-PR logging changes
git blame
Changed & Unchanged agentic logging
Code Diffs
Logging change attribution
(a) Overview of our approach to tracking post-generation log regulation. Agentic PR creation
t=0
Review comment
t=1
...
Commit to the PR
Merged PR
t=k
t=N
(b) Overview of the PR lifecycle considered for post-generation analysis. Fig. 10. RQ3 methodology and lifecycle setup.
Summary of RQ2 Our analysis identifies both a specification gap and a compliance gap in agentic logging. Specifically, 98.7% of agentic PRs lack logging instructions, and even when explicitly instructed, agents do not comply with requests 67% of the time. Crucially, this low compliance persists despite 73% of logging instructions being detailed (specifying log levels, files, or frameworks), these strong instructions yield only 27% compliance. Practically, this suggests that relying on instruction specificity alone might not ensure observability. Instead, we recommend enforcing logging standards through deterministic CI/CD checks or linters that block uninstrumented code. RQ3. Is agentic logging regulated post generation, and by whom? Motivation: The goal of this research question is to understand the lifecycle of agentic PRs postgeneration. We specifically investigate how logging is regulated during code review and subsequent commits. Unlike functional correctness, which automated tests and CI pipelines can verify, logging quality (e.g., missing context, noise) rarely breaks the build. Consequently, logging issues can easily evade automated scrutiny. Therefore, it remains unclear whether agent-authored logging enters the codebase unexamined, or whether human reviewers and automated bots intervene to request changes and refine logging and observability. Approach: To quantify the post-generation regulation of agentic logging, we analyze the version history and review comments of our dataset, as shown in Figure 10a. Specifically, we perform three primary analyses: • Lifecycle Tracking: We reconstruct the history of every agent-introduced logging statement from its initial commit (𝑡 = 0) to the final merged state, as shown in Figure 10b. We attribute , Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
17
All Agentic PRs with Logging 941 PRs with logging changes
Post-Generation Revision Status Unchanged: 215 (22.8%)
Revised: 726 (77.2%)
Revision Actor (among 726 revised PRs) Human only 396 (54.5%)
Bot only 255 (35.1%)
Both 75 (10.3%)
(a) Agentic PRs with logging changes. All Human PRs with Logging 766 Human PRs with logging changes
Post-Generation Revision Status Unchanged: 141 (18.4%)
Revised: 625 (81.6%)
Revision Actor (among 625 revised PRs) Bot only: 10 (1.6%)
Human only 611 (97.8%)
Both: 4 (0.6%)
(b) Human PRs with logging changes. Fig. 11. Post-generation logging revision flow for agentic and human PRs.
every modification or deletion to either a human or a bot using commit authorship metadata. This allows us to quantify how much logging churn is driven by automated iteration versus human intervention. • Logging Instructions Analysis: We analyze review comments across all 4,550 agentic PRs to identify explicit logging feedback and who provides it. A PR is counted as having explicit logging feedback if at least one review comment is labeled as logging-related by our LLM Jury protocol (Section 3.3). We identify reviewer type from GitHub author metadata (bot account vs. human account). We then classify logging-related comments into Add (coverage-seeking), Modify (quality-refining), or Remove (noise-control). Finally, we report the prevalence of these logging-related comments over both (i) all agentic PRs and (ii) the subset of PRs with logging changes. • Survival Analysis: We use Kaplan-Meier survival analysis [9] to study how long agentgenerated logs remain unchanged after their introduction. We include only PRs in which the first commit already contains logging changes. The time axis is defined as the number of , Vol. 1, No. 1, Article . Publication date: April 2026.
Ouatiti et al.
Estimated probability of no later logging change
18 1.0
Agent Human
0.8
0.6
0.4
0.2
0.0
0
5
10
15
20
25
30
Commits after first commit
Fig. 12. Kaplan–Meier survival curves for post-generation logging stability in shared repositories (Agent vs Human). The data includes PRs whose first commit already contains logging changes and an event is the first later commit that modifies logging.
subsequent commits within the same PR. We mark an event at the first subsequent commit that edits logging (Add, Modify, or Remove). If logging is never edited again, the PR is treated as unchanged up to its last commit. We estimate the survival curve and 95% confidence intervals using scikit-survival.2 To contextualize our findings, we apply the exact same three-step approach to our dataset of human-authored PRs. Similar to RQ1, the human PRs considered are from the same repositories and the same time frames as the agentic PRs. This allows us to isolate whether the post-generation regulation and review patterns observed in agentic PRs are unique to AI-generated code, or if they simply reflect the standard review lifecycle of modern software engineering. Results: Post-generation logging revisions are common in both agentic and human PRs, but the revision actors differ, as shown in Figure 11. Among PRs that introduce logging changes, 77.2% (agentic) and 81.6% (human-authored) of the PRs are revised in later commits. For the revised PRs, we observe that the actor mix differs sharply. While agentic PRs are mainly split across humanonly (54.5%), bot-only (35.1%) revisions, human PRs are almost entirely revised by humans only (97.8%). At the logging-statement level, humans contribute with 72.5% post-generation modifications to agentic PRs, versus 99.5% for human PRs. Furthermore, our survival analysis suggests that post-generation logging revisions occur primarily early in the PR lifecycle for both human and agentic PRs, as shown in Figure 12. This means that when logging is revised, it is usually adjusted in the first few follow-up commits, while later logging revisions become less common. Notably, we observe a distinct gap in iteration frequency between the two groups (i.e., humans vs. agents). Specifically, the human survival curve drops faster and lower than the agent curve, indicating that human-authored logging undergoes more frequent and rapid post-generation revision. Although agents are still subject to post-generation regulation, their initial logging implementations are more “sticky” and less likely to be churned across subsequent commits than those introduced by human developers. Explicit logging feedback in review text is rare in both human and agentic PRs, as shown in Table 9. We observe no meaningful difference in the prevalence of explicit logging feedback 2 https://scikit-survival.readthedocs.io/
, Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
19
Table 9. Explicit logging feedback rate in review comments (PR-level).
PR Group
With Feedback
Total
Rate
All agentic PRs All human PRs
99 71
4,550 3,276
2.18% 2.17%
Agentic PRs with logging changes Human PRs with logging changes
55 46
941 766
5.80% 6.00%
PR Size (LOC changed, log scale)
104 4390 2702
103 250
231
102
101
Agent Unchanged
Agent Revised
Human Unchanged
Human Revised
Fig. 13. PR size versus post-generation logging revisions in agentic and human PRs.
between agentic (2.18%) and human-authored PRs (2.17%). Even when restricting to PRs that already contain logging changes in their initial commit, the rates remain low at 5.80% for agentic PRs and 6.00% for human PRs. This suggests that logging corrections are usually applied directly in later commits rather than explicitly requested in review text. Furthermore, when logging feedback appears, it is mostly automated and primarily requests modifications. Specifically, bots account for 75.6% of logging feedback in agentic PRs and 81.1% in the human cohort. In the agentic subset, human and bot reviewers show similar intent distributions, with Modify as the largest category (47.4% for humans, 44.9% for bots), followed by Remove (28.9% vs. 24.6%) and Add (23.7% vs. 30.5%). In the human subset, human reviewers also focus on Modify (55.0%), while bot feedback is more skewed toward Add (44.2%). Logging regulation is mainly concentrated in large PRs, as shown in Figure 13. Specifically, post-generation logging regulation is not evenly distributed, and is concentrated in larger changes. For agentic PRs, those that undergo post-generation log changes are substantially larger (median 2,702 LOC) than those whose logging remains unchanged (median 231 LOC). This difference is statistically significant (𝑝 < 0.001) with a large effect size (Cliff’s 𝛿 = 0.688). We observe the same pattern for human PRs, as the ones with revised logging have a large effect size of 4,390 LOC, versus 250 LOC for unchanged PRs (𝑝 < 0.001, Cliff’s 𝛿 = 0.726). We also find a strong positive association between PR size and the amount of post-generation logging modification in both types of PRs (agentic: Spearman 𝜌 = 0.648, 𝑝 < 0.001, human: Spearman 𝜌 = 0.667, 𝑝 < 0.001). Finally, explicit logging feedback is more likely in larger PRs for both groups, and this size effect is stronger for human PRs. In the agentic subset, PRs with feedback have a median size of 320 LOC versus , Vol. 1, No. 1, Article . Publication date: April 2026.
20
Ouatiti et al.
130 LOC without feedback (𝑝 < 0.001), while the gap is 890.5 LOC versus 114 LOC in the human subset (𝑝 < 0.001). Summary of RQ3 Our analysis reveals that agentic PRs do not significantly reduce the post-generation effort required to regulate logging. Logs introduced by both agents and humans are frequently revised before merging (77.2% and 81.6%, respectively). Crucially, the regulation of agentic code remains heavily human-driven as 54.5% of revised agentic PRs are changed exclusively by humans, who perform 72.5% of all post-generation log modifications. This persistent reliance on manual intervention demonstrates that current agentic workflows fail to alleviate the human maintenance burden for logging. 5
Implications
Our empirical findings reveal a disconnect between how AI coding agents generate code and how software logging observability is maintained. In this section, we discuss the practical implications of these findings for tool builders, software practitioners, and project maintainers. 5.1
For Tool Builders: Transitioning to Deterministic Guardrails
Our RQ2 results demonstrate that natural language instruction might be an unreliable mechanism for guiding agentic logging. Developers rarely provide explicit logging instructions (4.7% of the time). Even when they do, agents fail to comply with constructive requests 67% of the time. Consequently, tool builders cannot rely solely on prompt engineering or context files (e.g., AGENTS.md) alone to enforce non-functional requirements such as observability. This aligns with the broader findings in the literature demonstrating that the underlying Large Language Models (LLMs) powering these code agents frequently struggle to adhere to strict constraints and complex instructions [18, 40]. To address this, the design of agentic tools should shift from natural language guidance to guardrail-driven development. Tool builders should integrate deterministic enforcement mechanisms directly into the agent’s workflow. For example, agents should be required to pass observabilityfocused static analysis (e.g., linters) or CI/CD checks before submitting a pull request. By treating logging as a hard, verifiable constraint rather than an optional prompt piece, tool builders can ensure that agents produce code that is maintainable by default. 5.2
For Researchers: Training Agents for Proactive Observability
Our RQ1 findings show that agents can mimic human error-logging patterns, particularly in exception-handling blocks. However, they significantly underutilize INFO-level logging compared to human developers. This indicates that current models view logging primarily as a reactive mechanism for capturing failures, rather than a proactive tool for tracking normal system states. This behavioral skew highlights a critical gap in how underlying LLMs are trained or fine-tuned for software engineering tasks. Future research should focus on developing specialized training datasets or reward models that emphasize the semantic value of state-transition logging. For instance, models could be aligned using Reinforcement Learning from Human Feedback (RLHF) [23] to capture qualitative developer preferences regarding log clarity, placement, and verbosity. Concurrently, Reinforcement Learning with Verifiable Rewards (RLVR) [10] could leverage static analyzers or CI/CD pipelines as objective reward signals to automatically penalize uninstrumented controlflow paths. Ultimately, researchers should train agents not only to write code that passes unit , Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
21
tests, but also to generate the necessary footprints that allow human operators to understand the system’s runtime narrative. 5.3
For Practitioners: Mitigating the Hidden Maintenance Tax
While agentic coding tools promise increased development velocity, our RQ3 results reveal a hidden maintenance tax. We find that humans perform 72.5% of post-generation log repairs. Crucially, these interventions are largely implicit, and humans act as “silent janitors” who fix logging issues in subsequent commits rather than requesting corrections during code review. This dynamic is unsustainable for long-term project health. Practitioners and engineering managers should update their code review protocols to explicitly account for agentic contributions. Non-functional requirements, particularly observability, should become first-class items on PR review checklists. Reviewers should be encouraged to reject uninstrumented agentic PRs and prompt the agent to fix the missing logs, rather than silently absorbing the technical debt. Shifting this maintenance burden back to the agent is essential to realizing the full productivity benefits of AI-assisted development. 6
Threats to Validity
6.1
Internal Validity
Potential subjectivity in classifying the intent of logging instructions poses a threat. To mitigate this risk, we did not rely on a single classifier. Instead, we implemented a rigorous LLM Jury protocol (comprising GPT-4o, GLM-4.7, and DeepSeek) to triangulate the final labels. We iteratively refined our prompting strategy until we achieved substantial agreement (Cohen’s 𝜅 = 0.83) against a manually annotated ground truth, ensuring that our classification is robust and reproducible. An alternative explanation for non-compliance with repository-level instructions (e.g., AGENTS.md) is that the agent may not have accessed the file due to context window limitations. Our dataset is derived from frontier models (e.g., Claude 3.5 Sonnet, GPT-4o), which feature massive context windows (128k+ tokens) that can easily accommodate typical instruction files. Furthermore, modern agentic scaffolding employs context compaction strategies (e.g., Claude compact). Therefore, the non-compliance we observed is more likely a behavioral alignment issue than a resource limitation. Ephemeral instructions provided by users before PR generation introduce another threat. We acknowledge a blind spot regarding such instructions delivered via IDE chat interfaces, which are not captured in our dataset. However, we argue that this does not undermine our core finding of a logging gap. If invisible chat instructions were both prevalent and effective, we would expect higher logging prevalence in the final agent-generated code. The fact that agentic PRs still exhibit reduced logging activity suggests that chat-based instructions, if present, fail to drive observability as much as persistent instructions. 6.2
Construct Validity
One threat is our reliance on regex-based static analysis to identify logging statements. While pragmatic and well established in logging research, this approach carries the risk of missing dynamic logging patterns or custom wrappers (e.g., a project-specific MyLogger.track() or non-standard libraries). To minimize this threat, we leveraged specialized patterns tailored to the idioms of each target language (Python, Java, and JS/TS) and explicitly excluded build artifacts. We empirically validated the reliability of this approach on a statistically significant sample of 380 code diffs, achieving 96% precision and 94% recall. , Vol. 1, No. 1, Article . Publication date: April 2026.
22
6.3
Ouatiti et al.
External Validity
Our study focuses on repositories using Python, Java, and JavaScript/TypeScript. We selected these languages because they have mature tools for studying logging and represent dominant ecosystems in modern software development and AI training datasets [5]. Additionally, we restricted our dataset to repositories with at least 100 stars. While our findings may not generalize to smallscale repositories, our conclusions are derived from mature, active projects where logging and observability are typically genuine concerns. Finally, while specific compliance rates may shift as LLM capabilities evolve, the fundamental specification gap we observed, where humans fail to request logging, is a behavioral pattern likely to persist across model generations. 7
Conclusion
This paper presents an empirical study of logging practices in agent-generated code, analyzing 4,550 PRs from 81 open-source repositories. We investigate how agents implement logging compared to humans, how they respond to instructions, and how their work is regulated post generation. We find that agents generally mimic human logging mechanics but exhibit a significant prevalence gap, modifying logs less often than humans in 58.4% of the studied repositories. Furthermore, natural language instruction is largely ineffective. Explicit logging instructions are rare, appearing in only 4.7% of PRs, and agents ignore them 67% of the time. Finally, we identify a hidden maintenance cost, as humans perform 72.5% of post-generation log repairs, acting as “silent janitors” to ensure observability. These findings suggest that natural language instruction faces a double hurdle: humans rarely provide logging prompts (specification gap), and agents frequently ignore them (compliance gap). Consequently, relying on optional prompts is insufficient to ensure observability. Future work should explore deterministic enforcement mechanisms, such as CI/CD linters, to guarantee that agent-generated code meets production logging standards. References [1] Mohamed Batoun, Mohammed Sayagh, Roozbeh Aghili, Ali Ouni, and Heng Li. 2024. A literature review and existing challenges on software logging practices. Empirical Software Engineering 29 (06 2024). doi:10.1007/s10664-024-10452-w [2] Yi-Hung Chou, Yiyang Min, April Yi Wang, and James A. Jones. 2025. Learning from Mistakes: Understanding Ad-hoc Logs through Analyzing Accidental Commits . In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE Computer Society, Los Alamitos, CA, USA, 1–13. doi:10.1109/MSR66628.2025.00017 [3] Patrick Loic Foalem, Foutse Khomh, and Heng Li. 2024. Studying logging practice in machine learning-based applications. Information and Software Technology 170, C (2024), 17 pages. [4] Qiang Fu, Jieming Zhu, Wenlu Hu, Jian-Guang Lou, Rui Ding, Qingwei Lin, Dongmei Zhang, and Tao Xie. 2014. Where do developers log? an empirical study on logging practices in industry. In Companion Proceedings of the 36th International Conference on Software Engineering. 24–33. [5] GitHub. 2024. The State of Open Source and AI: The 2024 Octoverse Report. https://github.blog/news-insights/ octoverse/octoverse-2024/. Accessed: 2025-02-09. [6] Hao He, Courtney Miller, Shyam Agarwal, Christian Kästner, and Bogdan Vasilescu. 2025. Does AI-Assisted Coding Deliver? A Difference-in-Differences Study of Cursor’s Impact on Software Projects. arXiv:2511.04427 [cs.SE] https: //arxiv.org/abs/2511.04427 [7] Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, and Ahmed E. Hassan. 2025. Agentic Refactoring: An Empirical Study of AI Coding Agents. arXiv:2511.04824 [cs.SE] https://arxiv.org/abs/2511.04824 [8] S. Kabinna, W. Shang, C. Bezemer, and A. E. Hassan. 2016. Examining the Stabity of Logging Statements. In Proceedings of the 23rd International Conference on Software Analysis, Evolution, and Reengineering, Vol. 1. 326–337. [9] E. L. Kaplan and Paul Meier. 1958. Nonparametric Estimation from Incomplete Observations. J. Amer. Statist. Assoc. 53, 282 (1958), 457–481. arXiv:https://www.tandfonline.com/doi/pdf/10.1080/01621459.1958.10501452 doi:10.1080/ 01621459.1958.10501452 [10] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. In Advances in Neural Information , Vol. 1, No. 1, Article . Publication date: April 2026.
Do AI Coding Agents Log Like Humans? An Empirical Study
23
Processing Systems (NeurIPS), Vol. 35. 21314–21328. [11] Hao Li, Cor-Paul Bezemer, and Ahmed E. Hassan. 2025. Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models . In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE Computer Society, Los Alamitos, CA, USA, 307–318. doi:10.1109/ICSE-SEIP66354.2025.00033 [12] Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. arXiv:2412.05579 [cs.CL] https://arxiv.org/abs/2412.05579 [13] Heng Li, Weiyi Shang, Bram Adams, Mohammed Sayagh, and Ahmed E. Hassan. 2021. A Qualitative Study of the Benefits and Costs of Logging From Developers’ Perspectives . IEEE Transactions on Software Engineering 47, 12 (2021), 2858–2873. [14] Heng Li, Weiyi Shang, and Ahmed E. Hassan. 2017. Which log level should developers choose for a new logging statement? Empirical Software Engineering 22, 4 (2017), 1684–1716. doi:10.1007/s10664-016-9456-2 [15] Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. 2025. The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering. arXiv:2507.15003 [cs.SE] https://arxiv.org/ abs/2507.15003 [16] Zhenhao Li, Heng Li, Tse-Hsun Chen, and Weiyi Shang. 2021. DeepLV: Suggesting Log Levels Using Ordinal Based Neural Networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering. 1461–1472. [17] Sherlock A. Licorish, Ansh Bajpai, Chetan Arora, Fanyu Wang, and Kla Tantithamthavorn. 2025. Comparing Human and LLM Generated Code: The Jury is Still Out! arXiv:2501.16857 [cs.SE] https://arxiv.org/abs/2501.16857 [18] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. [19] Antonio Mastropaolo, Luca Pascarella, and Gabriele Bavota. 2022. Using deep learning to generate complete log statements. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 2279–2290. doi:10.1145/3510003.3511561 [20] Youssef Esseddiq Ouatiti. 2026. agentic_logging_RP. https://github.com/YoussefEssDS/agentic_logging_RP/tree/main. Replication package, accessed April 1, 2026. [21] Youssef Esseddiq Ouatiti, Mohammed Sayagh, Noureddine Kerzazi, Bram Adams, and Ahmed E. Hassan. 2024. The impact of concept drift and data leakage on log level prediction models. Empirical Software Engineering 29, 5 (July 2024), 37 pages. doi:10.1007/s10664-024-10518-9 [22] Youssef Esseddiq Ouatiti, Mohammed Sayagh, Noureddine Kerzazi, and Ahmed E. Hassan. 2023. An Empirical Study on Log Level Prediction for Multi-Component Systems. IEEE Transactions on Software Engineering 49, 02 (2023), 473–484. [23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. 27730–27744. [24] Antonio Pecchia, Marcello Cinque, Gabriella Carrozza, and Domenico Cotroneo. 2015. Industry Practices and Event Logging: Assessment of a Critical Software Development Process. In Proceedings of the 37th International Conference on Software Engineering. 169–178. [25] Mayra Sofia Ruiz Rodriguez, SayedHassan Khatoonabadi, and Emad Shihab. 2025. Automated File-Level Logging Generation for Machine Learning Applications using LLMs: A Case Study using GPT-4o Mini. arXiv:2508.04820 [cs.SE] https://arxiv.org/abs/2508.04820 [26] Guoping Rong, Shenghui Gu, Haifeng Shen, He Zhang, and Hongyu Kuang. 2023. How Do Developers’ Profiles and Experiences Influence their Logging Practices? An Empirical Study of Industrial Practitioners. In 2023 IEEE/ACM 45th International Conference on Software Engineering. 855–867. [27] Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, and Brendan Dolan-Gavitt. 2023. Lost at C: a user study on the security implications of large language model code assistants. In Proceedings of the 32nd USENIX Conference on Security Symposium (Anaheim, CA, USA) (SEC ’23). USENIX Association, USA, Article 124, 18 pages. [28] Weiyi Shang, Meiyappan Nagappan, and Ahmed E. Hassan. 2015. Studying the Relationship between Logging Characteristics and the Code Quality of Platform Software. (2015), 1–27. [29] Rosalia Tufano, Antonio Mastropaolo, Federica Pepe, Ozren Dabic, Massimiliano Di Penta, and Gabriele Bavota. 2024. Unveiling ChatGPT’s Usage in Open Source Projects: A Mining-based Study. In Proceedings of the 21st International Conference on Mining Software Repositories (Lisbon, Portugal) (MSR ’24). Association for Computing Machinery, New York, NY, USA, 571–583. doi:10.1145/3643991.3644918 [30] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv:2404.18796 [cs.CL] https://arxiv.org/abs/2404.18796
, Vol. 1, No. 1, Article . Publication date: April 2026.
24
Ouatiti et al.
[31] Zora Zhiruo Wang, Yijia Shao, Omar Shaikh, Daniel Fried, Graham Neubig, and Diyi Yang. 2025. How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations. arXiv:2510.22780 [cs.AI] https://arxiv.org/abs/2510.22780 [32] Miku Watanabe, Yutaro Kashiwa, Bin Lin, Toshiki Hirao, Ken’Ichi Yamaguchi, and Hajimu Iida. 2024. On the Use of ChatGPT for Code Review: Do Developers Like Reviews By ChatGPT?. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (Salerno, Italy) (EASE ’24). Association for Computing Machinery, New York, NY, USA, 375–380. doi:10.1145/3661167.3661183 [33] Miku Watanabe, Hao Li, Yutaro Kashiwa, Brittany Reid, Hajimu Iida, and Ahmed E. Hassan. 2025. On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub. arXiv:2509.14745 [cs.SE] https://arxiv.org/abs/2509.14745 [34] Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. 2024. UniLog: Automatic Logging via LLM and In-Context Learning. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal). Article 14, 12 pages. [35] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: agent-computer interfaces enable automated software engineering. In Proceedings of the 38th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 1601, 125 pages. [36] Ding Yuan, Yu Luo, Xin Zhuang, Guilherme Renna Rodrigues, Xu Zhao, Yongle Zhang, Pranay U. Jain, and Michael Stumm. 2014. Simple testing can prevent most critical failures: an analysis of production failures in distributed data-intensive systems. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation (Broomfield, CO) (OSDI’14). 249–265. [37] Ding Yuan, Soyeon Park, Peng Huang, Yang Liu, Michael M. Lee, Xiaoming Tang, Yuanyuan Zhou, and Stefan Savage. 2012. Be conservative: enhancing failure diagnosis with proactive logging. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation (Hollywood, CA, USA) (OSDI’12). 293–306. [38] D. Yuan, S. Park, and Y. Zhou. 2012. Characterizing logging practices in open-source software. In Proc. of the 34th Int. Conf. on Software Engineering. 102–112. [39] Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Yun Yang, Weisong Sun, Shengcheng Yu, and Zhenyu Chen. 2024. A Survey on Large Language Models for Software Engineering. arXiv:2312.15223 [cs.SE] https://arxiv.org/abs/ 2312.15223 [40] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-Following Evaluation for Large Language Models. arXiv preprint arXiv:2311.07911 (2023).
, Vol. 1, No. 1, Article . Publication date: April 2026.