arXiv:2605.02273v1 [cs.SE] 4 May 2026
These Aren’t the Reviews You’re Looking For How Humans Review AI-Generated Pull Requests Kacper Duma
Patryk Wróblewski
Jagoda Bobińska
Nicolaus Copernicus University Toruń, Poland [email protected]
Nicolaus Copernicus University Toruń, Poland [email protected]
Nicolaus Copernicus University Toruń, Poland [email protected]
Julia Winiarska
Piotr Przymus
Nicolaus Copernicus University Toruń, Poland [email protected]
Nicolaus Copernicus University Toruń, Poland [email protected]
Abstract We analyze code review interactions for AI-generated pull requests (PRs) on GitHub using the AIDev dataset and compare them to human-authored PRs within the same repositories. We find that most AI-generated PRs receive no review and, when reviewed, are largely dominated by AI agents rather than humans. Humanauthored PRs are more likely to receive human-only review and to attract direct human feedback. In contrast, reviews of AI-generated PRs more often take the form of automation-mediated interaction, with human involvement frequently expressed through agent steering rather than standalone evaluation. These results indicate systematic differences in how review activity is structured in agentic workflows and raise challenges for interpreting review metrics as indicators of human oversight in large-scale mining studies. ACM Reference Format: Kacper Duma, Patryk Wróblewski, Jagoda Bobińska, Julia Winiarska, and Piotr Przymus. 2026. These Aren’t the Reviews You’re Looking For How Humans Review AI-Generated Pull Requests. In Proceedings of The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/ nnnnnnn.nnnnnnn
1
Introduction
Code review is a central practice in software engineering, traditionally described as an evaluator-driven process in which humans inspect code, discuss alternatives, and iteratively converge toward improved solutions. Empirical work, however, shows that real-world reviews often diverge from this idealized model. Contemporary reviews tend to be lightweight and brief discussions, few reviewers, and rapid approvals rather than in-depth technical critique [10]. A recent survey further highlights that modern practice prioritizes speed, pragmatism, and asynchronous, tool-supported workflows Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. EASE 2026, Glasgow, United Kingdom © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
over exhaustive evaluation [4]. As a result, even human-authored pull requests (PRs) frequently receive limited substantive feedback. The emergence of AI-generated pull requests adds another dimension to this streamlined landscape. Coding agents now autonomously propose changes across open-source repositories, but little is known about how humans respond to these contributions. Prior work provides early evidence on the variability of automated review tools [14], yet almost nothing is known about human review behaviour when the author is an AI agent. Do developers offer meaningful technical feedback on these changes, or do interactions primarily consist of acknowledgments, workflow comments, or procedural steering? And how does this compare to reviews of human-authored PRs within the same repositories? This problem is important because AI-generated pull requests are becoming increasingly common, yet we don’t have a clear understanding of how developers engage with and evaluate contributions authored by AI agents. Without this knowledge, it is difficult to assess the quality of collaboration and broader impact of AI on software development practices. To investigate these questions, we conduct a large-scale empirical study of human review activity on AI-generated (“agentic”) PRs in GitHub repositories with at least 100 stars avalible in AIDev dataset [9]. For analysis, we distinguish review interactions that clearly fall into non-evaluative categories—such as CI-related messages, or steering commands for AI agents—from all remaining comments, which we treat as evaluative for the purposes of quantification (this category varies in depth but is adequate for capturing broad review differences). Research Question: RQ1: What characterizes review activity for AI-generated PRs? RQ2: How does this activity differ from review of human-authored pull requests in the same repositories? Contributions: • Large-scale analysis of human review activity on AI-generated and human-written PRs fromSS AIDev repositories (≥ 100★). • Classification of review interactions into non-evaluative vs. other comments to quantify human involvement. • Replication package: https://github.com/ncusi/reviewing-ai-generatedprs-ease2026-short
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Kacper Duma, Patryk Wróblewski, Jagoda Bobińska, Julia Winiarska, and Piotr Przymus
Dataset
Repositories
2 Related Work 2.1 AI-Assisted Code Review Prior research explores automated support for code review, including defect detection, style enforcement, and maintainability improvement [3, 4, 7, 12]. Recent studies focus on LLM-based reviewers integrated into pull request (PR) workflows. Sun et al. [14] analyze over 22,000 AI-generated review comments on GitHub and show that actionable feedback containing concrete code suggestions is significantly more likely to trigger code changes. Other studies report mixed productivity effects: while AI reviewers can reduce manual effort, they may also increase comment volume without shortening review latency [1, 6, 18]. Tufano et al. [16] conduct a qualitative analysis of automated code review techniques and characterize the types of changes for which current approaches succeed or fail. They report that existing models handle simple modifications more reliably than changes requiring broader contextual understanding of the code base. Turzo et al. [17] propose a deep learning-based classifier for review comments and show that fine-grained categorization provides better informative code review analytics than coarse-grained metrics based on counts.
2.2
Human–AI Interaction in Code Review
Several works investigate how developers perceive and use AIassisted review. Qualitative studies indicate that AI feedback can reduce social friction but introduce additional cognitive load related to validating AI suggestions [2]. Developers therefore tend to treat AI comments as advisory rather than authoritative, selectively integrating them based on context and trust.
2.3
Agentic Software Engineering
The concept of agentic software engineering extends AI assistance toward autonomous agents acting as participants in development workflows. Tang et al. [15] propose a multi-agent system capable of iterative and communicative code review. Conceptual work emphasizes the need for new evaluation criteria for such systems, including transparency and alignment with human team norms [11]. Gong et al. [8] analyze the inconsistency in agent-authored PR and report that PRs with misaligned descriptions are less likely to be accepted and take longer to merge.
2.4
Review Dynamics in Pull Requests
Empirical studies of human code review analyze review latency, reviewer participation, and comment characteristics, providing important baselines for understanding collaborative review processes [5, 10]. However, direct comparisons between traditional human review workflows and AI-driven agent review processes remain limited. Pirouzkhah et al. [13] examine PR descriptions at scale and report that certain description elements, especially explicit statements of desired feedback, are associated with greater reviewer engagement and higher likelihood of merge.
2.5
Review Comment Classification
Prior work has proposed taxonomies to describe the content of code review comments. Those classifications typically distinguish
Pull Requests Code Reviews
AIDev
(Not in AIDev, fetched from Github API)
Figure 1: Overview of the AIDev-based data collection pipeline. Pull requests and code reviews are mined from GitHub repositories, combining AIDev annotations with additional review data retrieved via the GitHub API. between functional issues, maintainability concerns, documentation related remarks and discussion oriented feedback. More recent studies explore automated classification of review comments to support large scale analytics [17]. In contrast, our study does not aim to introduce a fine-grained semantic taxonomy of comment types. Rather than categorizing review content by technical topic, we focus on distinguishing interaction modes—agent steering, automationrelated interaction, and other human-driven review. This coarser scheme aligns with our objective of analyzing how human oversight is expressed in agentic workflows, rather than evaluating the substantive technical quality of review feedback.
3 Dataset and Methodology 3.1 Dataset We use the AIDev dataset [9], which contains over 932k pull requests together with metadata identifying agentic and human authorship. From AIDev, we derive the sets of AI-generated pull requests 𝑃𝐴𝐼 and human-authored pull requests 𝑃ℎ𝑢𝑚𝑎𝑛 , as well as the repository sets 𝑅𝑝𝑜𝑝 (repositories containing at least one agent-authored PR) and 𝑅ℎ𝑢𝑚𝑎𝑛 (repositories containing at least one human-authored PR). We further define their intersection as 𝑅∩ = 𝑅𝑝𝑜𝑝 ∩ 𝑅ℎ𝑢𝑚𝑎𝑛 . AIDev includes full review activity for repositories in 𝑅𝑝𝑜𝑝 . For repositories in 𝑅ℎ𝑢𝑚𝑎𝑛 whose reviews were not covered by AIDev, we retrieve the corresponding review and discussion records via the GitHub REST API. Let 𝐶 denote the union of all collected review comments. We partition 𝐶 into 𝐶𝐴𝐼 (review comments authored by agents) and 𝐶ℎ𝑢𝑚𝑎𝑛 (review comments authored by humans). For any PR set 𝑃 we write 𝐶 (𝑃) for all reviews targeting PRs in 𝑃, with 𝐶 (𝑃) = 𝐶𝐴𝐼 (𝑃) ∪ 𝐶ℎ𝑢𝑚𝑎𝑛 (𝑃). These sets define the slices used in our research questions: RQ1 𝐶 (𝑃𝐴𝐼 | 𝑅𝑝𝑜𝑝 ) RQ2 𝐶 (𝑃𝐴𝐼 | 𝑅∩ ) vs. 𝐶 (𝑃ℎ𝑢𝑚𝑎𝑛 | 𝑅∩ ), 𝑅∩ = 𝑅𝑝𝑜𝑝 ∩ 𝑅ℎ𝑢𝑚𝑎𝑛 . Table 1 summarizes the analyzed pull requests after filtering. Filtering details follow.
3.2
Filtering
3.2.1 Filtering Human PRs. The AIDev dataset provides a stratified sample of human-authored pull requests designed to match
These Aren’t the Reviews You’re Looking For How Humans Review AI-Generated Pull Requests
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Table 1: PR-level overview of code review presence and reviewer composition. Review presence indicates whether a PR received any review. For reviewed PRs, reviewer composition distinguishes between agentic-only, human-only, and mixed (human + agentic) participation. R - indicates repositories subset. Percentages are relative to row totals. Review Presence
Reviewer Composition
PR author
R
# PRs
# no review
# with review
AI
Human
Human + AI
AI
Human
AI agent AI agent Human
𝑅𝑝𝑜𝑝 𝑅∩ 𝑅∩
33596 9616 5574
20621 (61.38%) 2781 (28.92%) 1924 (34.52%)
12975 (38.62%) 6835 (71.08%) 3650 (65.48%)
7625 (58.77%) 3939 (57.63%) 1932 (52.93%)
1316 (10.14%) 552 (8.08%) 920 (25.21%)
4034 (31.09%) 2344 (34.29%) 798 (21.86%)
2.35 ± 1.95 2.66 ± 2.16 2.05 ± 1.66
2.19 ± 2.37 2.19 ± 2.46 2.57 ± 3.32
the characteristics of AI-generated pull requests. During manual inspection, however, we observed that some pull requests labeled as human-authored were in fact created by automated agents. To address this issue, we applied a two-step filtering procedure to clean the set 𝑃 human . First, we identified accounts that were mislabeled as human but in fact corresponded to bots. These accounts were initially detected based on the presence of the “[bot]” pattern in the username. Importantly, every account matching this pattern was manually inspected and verified to ensure correct classification. Second, we filtered out pull requests whose bodies contained AIrelated keywords (e.g., PR-Codex, Copilot, Claude, Devin, Cursor). All pull requests matching these patterns were subsequently manually inspected to confirm that they were in fact created by AI agents. Overall, we removed 1,044 pull requests from 𝑃human . 3.2.2 Filtering Human Code Reviews. Similarly, we cleaned the set of human-authored review comments to obtain a reliable 𝐶 human . Although AIDev explicitly labels many bot accounts, some automated review mechanisms appear under the account type “User” and are therefore are not identified as “Bot”. To detect such cases, we applied a heuristic filtering procedure. We first restricted attention to accounts with user type = “User”. Among these, we flagged usernames exhibiting bot-like patterns (e.g., “[bot]”, “ci”, “automation”). To reduce false positives, we additionally required that such accounts have at least 𝑘 interactions (e.g., 𝑘 = 10). All flagged accounts were then manually inspected to confirm their automated nature. Repositories involving such accounts were excluded, yielding a human-only set 𝑅human used for comparisons with 𝑅pop and 𝑅∩ . All identified accounts were labeled as “Bot Other”. In total, we removed 588 misclassified human-authored comments from 𝐶 (𝑃𝐴𝐼 ) and 539 from 𝐶 (𝑃human ).
4
Reviews (mean ± std)
Classification of Comments
To help us perform our review analysis, we designed a rule-based classifier to assign each human-authored comment to one of three categories. In this paper, the classifier is used only as a helper tool. It is based on regular expressions derived from a data-driven analysis of the corpus. To construct the rules, we grouped comments by normalized text, inspected the most frequent forms, and iteratively expanded the patterns to cover common variants. Human-authored review classifier assignees each comment exactly one category: 𝜅 : 𝐶ℎ𝑢𝑚𝑎𝑛 → {agent, automation, human_review}.
Agentic. Agent-related comments correspond to human steering of LLM-based coding assistants. These are detected by pairing agent names (e.g., copilot, coderabbit, devin, claude, gemini, sourcery-ai) with imperative verbs such as “run”, “fix”, “rebase”, “test”, or “update”. We include both inline commands (e.g., “@coderabbit fix lint failure”) and comments that start with an agent address. Automation. Automation/CI comments are matched using a separate set of patterns targeting chatops commands, CI-service prefixes, and known bot output formats. These include commands such as “/azp run”, “/test”, “/rebase”, auto-formatting bots, CLA reminders, coverage annotations, and other system-generated diagnostic notices. The defining characteristic is interaction with infrastructure rather than with LLM agents. Human Review. All remaining human-authored comments that do not match the agent or automation patterns are labeled as human review. In the initial design, workflow-related remarks and substantive evaluative feedback were treated as separate categories. However, empirical analysis of the corpus revealed substantial lexical and functional overlap between these two types of comments, and attempts to separate them introduced unstable boundaries without analytical benefit. We therefore merged them into a single category. This merged class includes both detailed code-review comments about functionality, design, correctness, performance, style, and maintainability, as well as short process-oriented statements that facilitate the pull-request workflow (e.g., “lgtm”, “done”, “thanks”, pings such as “@user”, merge/close notices, reminders to rerun tests or formatting, and instructions to clean or revert specific files). This category is treated as the default class for human-driven interaction within pull requests.
4.1
Classification rules
Throughout development, we repeatedly sampled false positives and false negatives to refine category boundaries and ensure consistent separation between agent, human review and automation comments. Particular attention was paid to ambiguous cases where infrastructure and agent - triggering commands, and human-authored feedback might overlap. The complete implementation, all regularexpression patterns, and representative examples for each category are included in our replication package. The classifier is implemented using structured regular expressions that encode lexical and structural signals observed in the corpus. Agent-related comments are detected through the co-occurrence of a closed lexicon of agent identifiers (e.g., Copilot, Claude, Gemini, Devin, CodeRabbit) and a predefined set of imperative action verbs (e.g., run, fix, rebase, test, update, review). A comment is classified
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Kacper Duma, Patryk Wróblewski, Jagoda Bobińska, Julia Winiarska, and Piotr Przymus
Table 2: Distribution of comment-level review comments per PR (with ≥1 review), distinguishing agentic vs. human authorship. Human comments are categorized (𝐶 human ). 𝑅 denotes the repository subset. Percentages are coresponding to row totals. Reviews PR author
R
All
AI agent AI agent Human
𝑅𝑝𝑜𝑝 𝑅 𝑅
39122 23081 10001
𝐶𝐴𝐼 28004 (71.58%) 16727 (72.47%) 5589 (55.88%)
Human review categories (𝜅 on 𝐶 human ) 𝐶ℎ𝑢𝑚𝑎𝑛
Agentic
Automation
Human review
11118 (28.42%) 6354 (27.53%) 4412 (44.12%)
3154 (28.37%) 1647 (25.92%) 72 (1.63%)
789 (7.1%) 543 (8.55%) 212 (4.81%)
7175 (64.53%) 4164 (65.53%) 4128 (93.56%)
as agent-directed if it matches either an address–command structure (e.g., “@agent fix lint”) or a command–entity co-occurrence pattern within the same textual span. Automation and CI-related comments are identified using prefixbased and format-based expressions capturing chatops commands (e.g., “/azp run”, “/rebase”), infrastructure triggers, and standardized bot output markers such as coverage annotations, conflict warnings, or CLA reminders. These patterns rely on initial markers and service specific tokens specific for infrastructure interaction. All regular expressions are evaluated deterministically and in fixed order. Agent patterns are applied first, followed by automation patterns. Any comment that does not match these patterns is assigned to the default human review category. This default class therefore encompasses both substantive evaluative feedback (e.g., comments on functionality, design, correctness, or style) and procedural workflow-related remarks (e.g., “lgtm”, merge notices).
4.2
Validation of the Classifier
5
Table 3: Regex classifier confusion matrix. Agentic
Automation
Human
189 0 11
0 200 0
10 7 383
Overall accuracy equals 96.5% (772/800). Given the large sample size, residual classification errors are unlikely to affect the observed aggregate proportions. Per-class performance is consistently high: Agentic (precision 94.5%, recall 95.0%), Automation (precision 100%, recall 96.6%), and Human Review (precision 95.8%, recall 97.2%). Misclassifications occur mainly between agentic and human review, reflecting lexical overlap between imperative forms and informal discourse. Confusion with bot is negligible.
Results
In this study, we first analyze AI-generated pull requests from repositories in 𝑅𝑝𝑜𝑝 as from AIDev dataset [9]. We then compare code review practices between agent-authored and human-authored pull requests originating from the same set of repositories (𝑅∩ ; see Section 3 for details). Table 1 summarizes pull-request–level review presence and reviewer composition, while Table 2 reports comment-level review activity and the distribution of human review categories. Across all settings, review activity is generally lightweight, with reviewed pull requests receiving on average 2–3 comments with standard deviations below 3.5.
5.1
To ensure reliability, we conducted a manual validation of the regular-expression rules on the datasets used for both RQs. For each dataset, we drew stratified random samples with equal numbers of instances from each predicted category of human comments. In total, 800 comments were manually inspected and re-labeled to assess classification correctness. After merging workflow and review into a single human review category, the aggregated three-class confusion matrix (agentic, bot, human review) is as follows:
Agentic Automation Human
These indicate that the regex rules are consistent and that the three-class scheme preserves strong discriminative performance with a simpler structure.
Agentic Pull Requests in 𝑅𝑝𝑜𝑝
We first examine review activity for agent-authored pull requests in popular repositories (𝑅𝑝𝑜𝑝 ). In total, 33,596 agent-authored PRs are included in this subset. At the pull-request level (see Table 1), 61.38% (20,621) receive no recorded review activity, while 38.62% (12,975) receive at least one review. Among those agent-authored PRs that are reviewed, 58.77% (7,625) are reviewed exclusively by agents, 10.14% (1,316) receive human-only review, and 40.34% (4,034) involve mixed human–agent participation. Viewed from the perspective of observable human involvement, 84.0% (28246/33596) of agent-authored PRs either receive no recorded review or are reviewed exclusively by agents, whereas 15.9% (5350/33596) exhibit some form of human participation. We note that the absence of recorded review activity does not imply the absence of human oversight; maintainers may inspect pull requests without leaving traceable comments. Our results capture observable review interaction as reflected in the PR history. At the comment level (see Table 2), review discussions are dominated by agent-authored comments. Of 39,122 total review comments, 71.58% (28,004) are authored by agents and 28.42% (11,118) by humans. Within human comments, 64.53% (7,175) are categorized as direct human review (evaluative and workflow-related feedback), 28.37% (3,154) correspond to agent-steering interactions, and 7.10% (789) are automation-related messages.
5.2
Agentic and Human Pull Requests in 𝑅∩
We compare agent-authored and human-authored pull requests within the same repositories (𝑅∩ ) using Pearson’s chi-square tests
These Aren’t the Reviews You’re Looking For How Humans Review AI-Generated Pull Requests
on categorical count data. To account for multiple comparisons, we apply the Benjamini–Hochberg false discovery rate correction and adopt a conservative significance threshold of 𝑝 < 0.001. We report the chi-square statistic together with its degrees of freedom (number of independent categories in the contingency table) and effect sizes using Cramér’s 𝑉 . At the pull-request level (see Table 1), agent-authored pull requests are reviewed slightly more often than human-authored pull requests (71.08% vs. 65.48%). This difference is statistically significant but small in magnitude (𝜒 2 (1) = 51.44, 𝑝 < 10−3 , 𝑉 = 0.06). Viewed from the perspective of observable human involvement, 69.9% (6720/9616) of agent-authored PRs in 𝑅∩ either receive no recorded review or are reviewed exclusively by agents, whereas 30.1% (2896/9616) exhibit some form of human participation. For human-authored PRs in the same repositories, 69.2% (3856/5574) either receive no recorded review or are reviewed exclusively by agents, while 30.8% (1718/5574) involve observable human participation. As before, the absence of recorded review activity does not imply the absence of human oversight; maintainers may inspect pull requests without leaving traceable comments. Reviewer composition differs much more strongly between author types. For agent-authored pull requests, most reviewed cases are agent-only (57.63%), with mixed Human+AI reviews also common (34.29%), and human-only reviews relatively rare (8.08%). For human-authored pull requests, agent-only reviews remain frequent (52.93%), but mixed reviews are less common (21.86%), and humanonly reviews are substantially more frequent (25.21%). This difference is statistically significant with a moderate effect size (𝜒 2 (2) = 629.4, 𝑝 < 10−3 , 𝑉 = 0.25). At the comment level (see Table 2), reviews of agent-authored pull requests are dominated by agentic comments (72.47%), while human-authored pull requests attract a substantially larger share of human comments (44.12%). This difference is statistically significant with a moderate effect size (𝜒 2 (1) = 873.7, 𝑝 < 10−3 , 𝑉 = 0.16). The strongest divergence concerns the distribution of human review comment categories. For agent-authored pull requests, comments unrelated to steering neither agents nor bots account for 65.53% of human review comments, compared to 93.56% for humanauthored pull requests. Conversely, agent-steering commands are far more common when reviewing agent-authored pull requests (25.92%) than human-authored ones (1.63%). This difference is statistically significant with a large effect size (𝜒 2 (3) = 1280, 𝑝 < 10−3 , 𝑉 = 0.34). Results remain significant under the adopted threshold.
6 Discussion 6.1 RQ1: What characterizes review activity for AI-generated PRs? AI-generated pull requests exhibit limited and automation-heavy review activity. In 𝑅𝑝𝑜𝑝 , 61.38% of agent-authored PRs receive no recorded review activity. Overall, 22.6% (7625/33596) are reviewed exclusively by agents and 15.9% ( 1316+4034/33596) involve observable human participation. Thus, most agent-authored PRs either receive no recorded review or are evaluated exclusively through automated participation. At the comment level, review discussions are similarly dominated by agents: 71.58% of review comments are authored by agents.
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Human comments account for 28.42% and consist primarily of direct human review (64.53%), alongside agent-steering (28.37%) and automation-related interaction (7.10%). We initially distinguished workflow comments (e.g., “LGTM”) from other types of reviews, but due to frequent annotator disagreement, we merged this into direct human review. Answer to RQ1 AI-generated pull requests are frequently unreviewed and, when reviewed, are predominantly handled by agents or through mixed human–agent participation. Observable human involvement is limited and includes both direct review feedback and agent-steering interaction.
6.2
RQ2: How does this activity differ from review of human-authored pull requests in the same repositories?
Within the same repositories (𝑅∩ ), agent-authored PRs are reviewed slightly more often than human-authored PRs, although this difference is small (𝑉 = 0.06). Importantly, the overall rate of observable human participation is nearly identical: 30.12% for agent-authored PRs and 30.83% for human-authored PRs. The primary differences concern reviewer composition and interaction structure rather than the mere presence of human involvement. Human-authored PRs are substantially more likely to receive human-only review (25.21%) than agent-authored PRs (8.08%), whereas mixed human–agent participation is more common for agent-authored PRs (34.29% vs. 21.86%) (𝑉 = 0.25). At the comment level (ie. 𝐶𝐴𝐼 vs 𝐶𝐻𝑢𝑚𝑎𝑛 ), human-authored PRs attract a larger share of human comments (44.12% vs. 27.53%) (𝑉 = 0.16). The strongest divergence concerns human comment categories (𝑉 = 0.34): for human-authored PRs, 93.56% of human comments constitute direct human review and agent-steering is nearly absent (1.63%). In contrast, reviews of agent-authored PRs include a substantially larger share of agent-steering (25.92%) and a lower proportion of direct human review (65.53%). Answer to RQ2 Compared to human-authored PRs in the same repositories, AI-generated PRs exhibit similar overall levels of observable human participation but differ markedly in how review effort is structured, with fewer human-only reviews and substantially more agent-steering interaction. Interpretation and Consequences. The strongest differences are structural rather than quantitative. While human participation occurs at comparable overall rates within the same repositories, its form changes when the author is an AI agent. Reviews of agentauthored PRs more frequently involve mixed participation and steering-oriented interaction, whereas human-authored PRs are predominantly evaluated through direct human review. Compared to traditional human-centric workflows, configurations with no recorded human review activity provide limited evidence of explicit human evaluation in the PR history. Although maintainers may inspect pull requests without leaving comments, the absence of documented interaction reduces the traceability of
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
review decisions and complicates the interpretation of review metrics as indicators of oversight. Observable review activity reflects interaction patterns rather than documented reasoning. Taken together, these findings suggest that AI authorship is associated with changes in the observable structure of review workflows, even when overall levels of human participation remain similar. Our results describe recorded interaction patterns and should not be interpreted as definitive evidence of reduced human oversight.
7
Threats to Validity
Construct Validity. Agent-authored pull requests are identified using the AIDev dataset, with additional filtering and manual verification to remove automation accounts from the human-authored set. Mislabeling may still occur. Review comments are classified using deterministic rules; short or ambiguous comments may be misclassified. Such errors may affect exact proportions but are unlikely to change overall trends. The absence of review comments does not imply that the code was not reviewed (e.g., it may have received a silent approval). Nevertheless, we classify all PRs without comments as not reviewed, as there is no empirical basis to distinguish between these cases. Internal Validity. Differences in review activity may be influenced by factors other than authorship, such as pull request size, complexity, or repository-specific workflows. These factors are not controlled for in this study. External Validity. Our analysis is limited to GitHub repositories in the AIDev dataset and, for RQ2, to repositories containing both agent-authored and human-authored pull requests. Results may not generalize to other settings. Conclusion Validity. We report descriptive results and do not assess code quality or review effectiveness. Findings should be interpreted as differences in review activity.
8
Conclusion
This study investigates how AI-generated pull requests are reviewed in practice using large-scale AIDev dataset [9], comparing review activity for agent-authored and human-authored PRs within the same repositories. AI-generated pull requests are frequently unreviewed and, when reviewed, are predominantly evaluated by agents or through mixed human–agent configurations. Note, that within the same repositories, overall levels of observable human participation are similar for agent-authored and human-authored PRs. At the comment level, reviews of agent-authored PRs are dominated by agent-generated comments, while human participation includes both direct review feedback and a substantial share of steering-oriented interaction. In contrast, human-authored PRs are far more likely to receive human-only review and to attract predominantly direct human feedback, whereas AI-generated PRs more often involve mixed participation and significantly higher levels of agent-steering interaction. These findings indicate that AI-generated contributions are associated not with the disappearance of human involvement, but with measurable changes in how review effort is expressed and documented. In agentic workflows, observable review activity does not necessarily correspond to direct human evaluation, and silent
Kacper Duma, Patryk Wróblewski, Jagoda Bobińska, Julia Winiarska, and Piotr Przymus
or undocumented reviews limit the traceability of human oversight. While using PR comments for agent steering can reuse existing infrastructure and shorten feedback loops, it also blurs the boundary between evaluation and interaction. As a result, conventional review metrics should be interpreted with care when used as indicators of human oversight and, where possible, should distinguish between human review, automation, and agent-steering activity. Future work should examine the implications of the distinct review configurations identified here. Fully automated review loops, delegated steering-based interaction, and augmentative AI participation in human-authored PRs represent different modes of review that may vary in effectiveness and oversight characteristics. Integrating interaction-pattern analysis with measures of review quality and post-merge outcomes could help clarify these effects and inform the development of appropriate metrics for assessing human oversight in increasingly automated development environments.
References [1] Fannar Steinn Aðalsteinsson, Björn Borgar Magnússon, Mislav Milicevic, Adam Nirving Davidsson, and Chih-Hong Cheng. Rethinking Code Review Workflows with LLM Assistance: An Empirical Study, May 2025. [2] Adam Alami and Neil A. Ernst. Human and machine: How software engineers perceive and engage with ai-assisted code reviews compared to their peers, 2025. [3] Alberto Bacchelli and Christian Bird. Expectations, outcomes, and challenges of modern code review. In 2013 35th International Conference on Software Engineering (ICSE), pages 712–721, San Francisco, CA, USA, May 2013. IEEE. [4] Deepika Badampudi, Michael Unterkalmsteiner, and Ricardo Britto. Modern Code Reviews—Survey of Literature and Practice. ACM Trans. Softw. Eng. Methodol., 32(4):107:1–107:61, May 2023. [5] H. Alperen Çetin, Emre Doğan, and Eray Tüzün. A review of code reviewer recommendation studies: Challenges and future directions. Science of Computer Programming, 208:102652, August 2021. [6] Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, and Eray Tüzün. Automated code review in practice, 2024. [7] Nicole Davila and Ingrid Nunes. A systematic literature review and taxonomy of modern code review. Journal of Systems and Software, 177:110951, July 2021. [8] Jingzhi Gong, Giovanni Pinna, Yixin Bian, and Jie M. Zhang. Analyzing messagecode inconsistency in ai coding agent-authored pull requests, 2026. [9] Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering. arXiv preprint arXiv:2507.15003, 2025. [10] Peter C. Rigby and Christian Bird. Convergent contemporary software peer review practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages 202–212, New York, NY, USA, August 2013. Association for Computing Machinery. [11] Ariel Rokem. Ten simple rules for scientific code review. PLOS Computational Biology, 20(9):e1012375, September 2024. [12] Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. Modern code review: a case study at google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP ’18, page 181–190, New York, NY, USA, 2018. Association for Computing Machinery. [13] Alberto Bacchelli Shirin Pirouzkhah, Pavlína Wurzel Gonçalves. The value of effective pull request description, 2026. [14] Kexin Sun, Hongyu Kuang, Sebastian Baltes, Xin Zhou, He Zhang, Xiaoxing Ma, Guoping Rong, Dong Shao, and Christoph Treude. Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions, August 2025. [15] Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawende F. Bissyande. CodeAgent: Autonomous Communicative Agents for Code Review, September 2024. [16] Rosalia Tufano, Ozren Dabić, Antonio Mastropaolo, Matteo Ciniselli, and Gabriele Bavota. Code review automation: Strengths and weaknesses of the state of the art, 2024. [17] Asif Kamal Turzo, Fahim Faysal, Ovi Poddar, Jaydeb Sarker, Anindya Iqbal, and Amiangshu Bosu. Towards automated classification of code review feedback to support analytics, 2023. [18] Miku Watanabe, Yutaro Kashiwa, Bin Lin, Toshiki Hirao, Ken’Ichi Yamaguchi, and Hajimu Iida. On the Use of ChatGPT for Code Review: Do Developers Like Reviews By ChatGPT? June 2024.