arXiv:2606.22721v1 [cs.SE] 21 Jun 2026
Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code Haoran Yu
Lifei Liu
Xiaochong Jiang
Independent Researcher Seattle, WA, USA [email protected]
Independent Researcher Seattle, WA, USA [email protected]
Independent Researcher Seattle, WA, USA [email protected]
Yuwen Jia
Su Wang
Pin Qian
Independent Researcher Santa Clara, CA, USA [email protected]
Carnegie Mellon University Pittsburgh, PA, USA [email protected]
Carnegie Mellon University Pittsburgh, PA, USA [email protected]
Yihang Chen Georgia Institute of Technology Atlanta, GA, USA [email protected]
Abstract As AI coding agents (e.g. GitHub Copilot, Devin, OpenAI Codex, Cursor) submit pull requests to open-source repositories at scale, a key question arises: do human reviewers gradually lower their scrutiny for AI-generated code over time? We conduct a longitudinal within-reviewer analysis using the AIDev dataset, studying 400 repeat reviewers who collectively submitted 11,429 reviews over a seven-month observation period. Comparing each reviewer’s early and late review episodes, we observe a population-level shift in approval rate from 30.1% to 36.8% (Wilcoxon signed-rank 𝑝 < 10−6 on paired shifts). Pooled by within-reviewer experience decile, the cumulative gap reaches +14.5 pp from first to tenth decile. This shift is experience-driven (persists after controlling for calendar time), agent-specific (human PR approval rates decline over the same period), and not explained by PR difficulty (median PR size is flat). However, review latency increases rather than decreases (+3.5×), while inline comment volume decreases (−22%, 𝑝 = 0.0014), suggesting reviewers spend more time in queue but less time actively inspecting code. The combination of rising approval, declining comment effort, and increasing queue time is most consistent with reflexive habituation under growing workload rather than rational trust calibration alone.
CCS Concepts • Software and its engineering → Collaboration in software development; Software evolution.
Keywords code review; AI agents; reviewer behavior; longitudinal study; agentic SE
1
Introduction
Code review is the primary human gate that separates an AI agent’s output from deployed software [1, 9]. This gate is only meaningful if reviewers maintain consistent scrutiny. Prior work examines whether AI-generated PRs pass review [8] and has explored the
inverse direction of using AI as a reviewer of human code [6, 10], but almost no work asks whether human reviewers change their behavior as they accumulate experience with AI agents. The rubber-stamping hypothesis posits that reviewers, after repeatedly approving agent-generated code with few defects, gradually reduce their inspection effort and begin approving PRs reflexively. This mirrors habituation effects documented in security auditing, aviation checklists, and automated-testing interactions [2, 11]. If it occurs for AI agent PRs, it would undermine the human oversight layer that AI-safety arguments often rely upon. Our central question is: Do individual reviewers become systematically more approving of AI agent pull requests as they accumulate reviewing experience? Concretely, we contribute: • A within-reviewer longitudinal analysis of 11,429 reviews from 400 repeat reviewers in the AIDev dataset [5]. • Evidence of a statistically significant population-level increase in approval rates (+14.5 pp across experience deciles), driven by reviewer experience rather than calendar time, accompanied by a 22% decline in inline review comments (𝜌 = −0.556 with approval shift). • A calendar-based human PR control showing that the trend is agent-specific: human PR approval rates decline over the same period. We cannot establish causality: agent code may genuinely improve over the observation window (all four agents received updates during 2025), confounding reviewer behavior change with rational re-calibration. However, the joint pattern of rising approval and declining comment effort provides suggestive evidence toward habituation rather than pure rational updating.
2 Method 2.1 Dataset We use AIDev [5], a dataset of pull requests submitted by AI coding agents to GitHub repositories with at least 100 stars. The dataset
Haoran Yu, Lifei Liu, Xiaochong Jiang, Yuwen Jia, Su Wang, Pin Qian, and Yihang Chen
contains 16,895 human reviews across 2,494 unique reviewers, covering five agent systems: GitHub Copilot Autofix, Devin (Cognition AI), OpenAI Codex CLI, Cursor, and Claude Code (Anthropic). We focus our per-agent analysis on the four agents with sufficient repeat-reviewer coverage; Claude Code contributes <2% of repeatreviewer reviews. For this study we focus on the 400 repeat reviewers who each reviewed 10 or more agent PRs, yielding 11,429 reviews. Within this cohort, 52 heavy reviewers contributed 50 or more reviews each, with individual observation spans up to 178 days (the full dataset covers 207 days). Each review record contains: reviewer identifier, agent identity, review outcome (approved, changes_reqested, commented), review timestamp, and associated PR metadata.
2.2
Longitudinal Design
For each repeat reviewer we sort their reviews chronologically and split them at the temporal midpoint into an early and a late episode. We compute per-reviewer approval rates (ARearly , ARlate ) and the individual shift ΔAR = ARlate − ARearly . For the decile analysis, we pool all reviews from repeat reviewers, order them by the reviewer’s within-reviewer review index (1 = first review of this reviewer, 𝑛 = latest), map indices to deciles, and compute the mean approval rate per decile.
2.3
Table 1: Approval rate by experience decile (pooled repeat reviewers). Decile 1 = reviewer’s first reviews; Decile 10 = latest reviews. AR (%)
CRR (%)
1 (earliest) 2 3 4 5 6 7 8 9 10 (latest)
27.9 27.1 30.0 28.1 33.3 35.6 32.6 34.2 38.6 42.4
11.2 11.3 8.0 7.9 6.2 7.0 7.3 6.6 6.9 5.6
Total shift +14.5 pp AR (27.9%→42.4%); −5.6 pp CRR
Approval rate CR rate (rescaled)
40%
Proxy Cross-Agent Control
To probe whether any shift is reviewer-general or agent-specific, we identify 108 reviewers who reviewed PRs from at least two distinct agents. For each such reviewer we compute their approval rate shift separately for each agent they reviewed. If the shift were agent-specific (e.g. because one agent’s code genuinely improved), we would expect the shift magnitude to differ markedly across agents.
35%
30%
28% decile 1
2.4
Decile
Outcome Measures
• Approval rate (AR): fraction of reviews with outcome approved. • Change-request rate (CRR): fraction with outcome “changes requested.” • Review latency: hours between PR opening and review submission (median per episode). We use the Wilcoxon signed-rank test on paired (ARearly , ARlate ) values across all 400 reviewers.
3 Results 3.1 Population-Level Longitudinal Trend Table 1 shows an overall increase in approval rate and decrease in change-request rate across experience deciles, with local nonmonotonicity due to binning noise. The early-vs-late split analysis (400 reviewers, median 17 reviews each) shows aggregate AR rising from 30.1% to 36.8% (+6.7 pp), with the change-request rate declining from 11.2% to 5.6%.1 The Wilcoxon signed-rank test on paired shifts yields 𝑝 < 10−6 , confirming the trend is not due to a small number of influential reviewers and that the decile 1 Aggregate rates are review-weighted (pooling all reviews). Per-reviewer means give
similar results: 30.5% → 36.6% (+6.1 pp).
2
3
4
5
6
7
8
9
10
Figure 1: Approval rate (solid circles, left axis) and changerequest rate (dashed, rescaled 2× for visualization) across 10 experience deciles. Approval rises overall from 27.9% to 42.4% (+14.5 pp) with local fluctuations; CR rate declines from 11.2% to 5.6%.
trend reflects genuine within-reviewer shifts rather than reviewercomposition artifacts. With 𝑁 = 400 pairs, the test has 80% power to detect Cohen’s 𝑑 ≥ 0.14 (approximately 2.1 pp given observed SD(ΔAR) = 0.25); our observed 𝑑 = 0.25 is well above this threshold. At the individual level, 52% of reviewers showed a positive ΔAR (i.e. became more approving) while only 28% became less approving and 20% showed no change (ΔAR = 0). Review latency also changes markedly: reviewers take a median of 3.9 hours in their early episode but 13.5 hours in their late episode. This apparently contradictory finding—approving more yet taking longer—may reflect reviewers accumulating a larger review backlog over time, deprioritizing routine agent PRs, or changes in PR submission timing rather than slower active inspection.
Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code
Monthly Approval Rate: Agent PRs vs Human PRs (Same reviewer pool, same repositories)
Table 2: Early-to-late approval rate shift per agent (repeat reviewers only). 𝑛 = number of repeat reviewers with ≥5 reviews for that agent.
50
Agent PRs Human PRs
𝑛
ARearly (%)
ΔAR (pp)
95% CI
Copilot Devin Codex Cursor
241 95 51 18
28.1 33.0 35.5 39.8
+9.1 +3.8 +1.8 −1.4
[+5.8, +12.3] [−0.8, +8.3] [−6.4, +9.6] [−14.8, +10.3]
Approval Rate (%)
45 Agent
40 35 30 25 20 15
Review Effort: Inline Comment Analysis
To probe whether rising approval reflects declining inspection effort, we analyze inline review comments from the same reviewers. Pooled by experience decile, the mean number of inline comments per review drops from 1.01 (first decile) to 0.79 (tenth decile), a 22% decline. Total comment word count per review decreases from 18.6 to 13.5 words (−28%). Both declines are statistically significant in paired early-vs-late tests (Wilcoxon 𝑝 = 0.0014 for comment count, 𝑝 = 0.0029 for word count). The within-reviewer correlation between approval-rate shift and comment-effort shift is strongly negative: Spearman 𝜌 = −0.556 (𝑝 < 10−4 ) for ΔAR vs. Δ(comments/review). Among reviewers who drifted upward (ΔAR > +0.10), inline comments decreased by 0.48 per review on average; among stable reviewers (|ΔAR| < 0.05), comments were unchanged (+0.10). This pattern—more approvals accompanied by less commenting—is consistent with declining inspection effort rather than unchanged effort with a rationally raised threshold.
3.3
Per-Agent Breakdown
Table 2 reports the approval-rate shift for each agent. Copilot reviewers exhibit the strongest positive shift (+9.1 pp), consistent with Copilot being the most mature and widely deployed agent in our observation window. Devin shows a moderate shift (+3.8 pp). Codex and Cursor are near-stable (≤2 pp change); however, both have small repeat-reviewer populations (𝑛 ≤ 51), limiting statistical power. We caution against reading these between-agent differences causally: Copilot’s larger shift may reflect the larger and more diverse pool of Copilot reviewers rather than a property of the agent itself.
3.4
Proxy Cross-Agent Control
To assess whether the trend is reviewer-specific or agent-specific, we examine the 108 reviewers who reviewed PRs from two or more agents. Among the 75 (reviewer, agent) pairs that each had at least five reviews, the mean approval-rate shifts are: Codex +9.4 pp, Copilot +9.8 pp, Devin +8.7 pp. Cursor (𝑛 = 10 pairs) shows −10.3 pp, but this estimate is extremely noisy given the small sample. The similarity of shifts across agents for the same set of reviewers is consistent with a reviewer-general phenomenon rather than one driven by a single agent’s code quality improving. That is, a reviewer who becomes more approving of Copilot PRs also tends to become more approving of Devin and Codex PRs.
Jan 6
Reviews (K)
3.2
4
Feb
Mar
Apr
May
Apr
May
Jun
Jul
Agent Human
2 0
Jan
Feb
Mar
2025
Jun
Jul
Figure 2: Monthly approval rate for agent PRs (solid) vs. human PRs (dashed) in overlapping repositories with 95% binomial CI bands (shaded). Same reviewer pool. Bottom panel shows review volume. Human PR volume drops sharply in July (𝑛 = 207); the late-period divergence has wide uncertainty.
We stress, however, that this cross-agent control is underpowered: only 1–2 reviewer–agent pairs overlap for direct withinreviewer, within-agent comparison, making a fully controlled test infeasible with the current dataset.
3.5
Calendar-Based Human PR Control
To distinguish reviewer-general leniency from agent-specific adaptation, we crawled review records for 6,618 human-authored PRs from the same repositories via the GitHub API, obtaining 11,415 reviews from 1,851 reviewers. Of these, 728 reviewers appear in both the agent and human PR review pools. Figure 2 compares monthly approval rates for agent and human PRs across the same reviewer population and repository set. In January 2025, agent PRs were approved less often than human PRs (30.7% vs. 37.8%, Δ = −7.1 pp). Over the following months, the two trends diverge: agent AR rises to 41.7% by July while human AR declines to 29.1% by June (22.2% in July, though with only 𝑛 = 207 reviews). By June 2025, reviewers approve agent PRs 10 pp more readily than human PRs from the same repositories. This calendar-based comparison strengthens the agent-specific interpretation: if reviewers were simply becoming more lenient overall, human PR approval rates should rise in tandem. Instead, human AR is flat or declining while agent AR rises, consistent with reviewers developing differential trust in agent-generated code. We note two caveats: (1) human PR review volume drops sharply after June 2025 (𝑛 = 207 in July), making the late-period human AR estimate noisy; and (2) this is a population-level comparison, not a
Haoran Yu, Lifei Liu, Xiaochong Jiang, Yuwen Jia, Su Wang, Pin Qian, and Yihang Chen
within-reviewer paired test (the per-reviewer calendar split yields only 26 agent-side reviewers with sufficient data in both periods).
4 Discussion 4.1 What the Data Show We observe a consistent pattern: reviewers who accumulate experience reviewing agent PRs become more approving over time (+14.5 pp from first to tenth decile). Three sensitivity checks help locate the source of this shift: Experience vs. calendar time. We fit a logistic regression separately for each of the 343 reviewers with ≥10 reviews for whom the model converges (57 reviewers with no outcome variation—52 never approved, 5 always approved—are excluded), predicting approval from both within-reviewer experience index and calendar date (days since January 1). Pooling coefficients: experience carries the signal (mean coefficient +0.11, 65% of reviewers positive) while calendar time contributes little (mean −0.007, 46% positive). The trend is driven by individual reviewer accumulation, not by a global temporal shift. PR size is flat. Monthly median PR size (lines changed) does not decline over the observation window (Spearman 𝜌 = +0.02, 𝑝 = 0.009, negligible magnitude). Later agent PRs are not systematically simpler; the approval increase cannot be explained by declining PR difficulty. Agent-specific, not global leniency. The calendar-based human PR control (Figure 2) shows agent AR rising while human AR declines over the same period. If reviewers were simply becoming more lenient across all PRs, human PR AR should rise in parallel. It does not.
4.2
Three Interpretations
These results are consistent with three accounts: (1) Progressive trust calibration—reviewers rationally raise their approval threshold based on accumulated positive evidence; (2) Deliberation under workload—growing backlog pressure produces longer latency but ultimately resolves toward approval (𝜌¯ = +0.08 between latency and approval, 𝑝 ≈ 0.21); (3) Reflexive habituation—reviewers reduce inspection effort after repeated positive experiences. The inline comment analysis provides partial disambiguation. The strong negative correlation between approval shift and comment-effort decline (𝜌 = −0.556, 𝑝 < 10−4 ) is more consistent with (3) than (1): under pure trust calibration, reviewers would approve more but maintain inspection depth. Instead, rising approval co-occurs with 28% less commenting. The latency increase is best explained by longer queue time but shorter active review—the nuanced variant of (3). We cannot fully exclude (1), as some effort reduction may be rational, but the magnitude suggests effort reduction beyond rational updating.
4.3
Per-Agent Asymmetry
Among agents with 𝑛 > 90 reviewers, both show clear positive shift: Copilot (+9.1 pp, 𝑛 = 241) and Devin (+3.8 pp, 𝑛 = 95). Codex (𝑛 = 51) and Cursor (𝑛 = 18) are underpowered for reliable agentspecific conclusions.
4.4
Implications for Practice
While the comment-effort analysis favors habituation over pure trust calibration, the practical outcome is the same regardless of mechanism: over time, a growing majority of reviewers approve agent PRs at higher rates with less scrutiny. Among those with meaningful change (|ΔAR| ≥ 0.05), 65% moved toward more approval and 35% toward less. Teams relying on human review as a quality gate should consider: • Rotation policies that prevent any single reviewer from accumulating an excessive share of agent PR reviews. • Streak audits that flag long consecutive approval runs from a single reviewer for secondary inspection. • Trend dashboards that show reviewers their personal approval-rate trajectory alongside downstream defect data.
4.5
Limitations
The AIDev dataset covers only repositories with >100 stars; enterprise repositories may differ. Our observation window spans at most 207 days. Cursor (𝑛 = 18) and Codex (𝑛 = 51) have small reviewer populations. The human PR control (Figure 2) loses volume after June 2025 (𝑛 = 207 in July), adding noise to the late-period comparison. We cannot rule out that the latency increase reflects systematic changes in PR submission timing rather than reviewer behavior. Crucially, we lack a direct measure of agent code quality over time; if agents genuinely improved, rising approval would be rational rather than habituative.
5
Related Work
Code review quality and efficiency have been studied extensively [1, 7, 9]. Kalliamvakou et al. [3] warned of biases in mining GitHub data, a caveat we inherit. Wessel et al. [11] showed that code review bots affect contributor behavior in OSS projects; our work extends this to AI coding agents and reviewer-side behavior. Kononenko et al. [4] linked review quality and participation to defect outcomes; tracking review quality longitudinally would complement our outcomebased analysis. Cassee et al. [2] found that CI adoption changes reviewer behavior; AI agents represent a larger-scale intervention in the same vein.
6
Conclusion
We report a longitudinal within-reviewer analysis of human code review behavior for AI agent pull requests. Across 400 repeat reviewers and 11,429 reviews, we observe a statistically significant increase in approval rates (+14.5 pp across experience deciles) and a corresponding decline in change-request rates. This shift is experiencedriven (not calendar-driven), agent-specific (human PR approval rates decline over the same period in the same repositories), and not explained by declining PR difficulty (median PR size is flat). The concurrent increase in review latency (+3.5×) coupled with a decline in inline comment effort (−22%, 𝜌 = −0.556 with approval shift) suggests that reviewers spend more time in queue but less time actively inspecting, consistent with reflexive habituation under growing workload. This pattern raises actionable concerns for
Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code
teams relying on human oversight of agentic software development. We release our analysis scripts to support replication and extension.2
References [1] Alberto Bacchelli and Christian Bird. 2013. Expectations, Outcomes, and Challenges of Modern Code Review. In Proceedings of the 35th International Conference on Software Engineering (ICSE ’13). IEEE Press, 712–721. doi:10.1109/ICSE.2013. 6606617 [2] Nathan Cassee, Bogdan Vasilescu, and Alexander Serebrenik. 2020. The Silent Helper: The Impact of Continuous Integration on Code Reviews. In Proceedings of the 27th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’20). IEEE, 423–434. doi:10.1109/SANER48275.2020.9054818 [3] Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. German, and Daniela Damian. 2014. The Promises and Perils of Mining GitHub. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR ’14). ACM, 92–101. doi:10.1145/2597073.2597074 [4] Oleksii Kononenko, Olga Baysal, Latifa Guerrouj, Yaxin Cao, and Michael W. Godfrey. 2015. Investigating Code Review Quality: Do People and Participation Matter?. In Proceedings of the 31st IEEE International Conference on Software Maintenance and Evolution (ICSME ’15). IEEE, 111–120. doi:10.1109/ICSM.2015. 7332457 [5] Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. 2025. The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering. arXiv:2507.15003 [cs.SE] https://arxiv.org/ abs/2507.15003
[6] Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. 2022. Automating Code Review Activities by Large-Scale Pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’22). ACM, 1035–1047. doi:10.1145/3540250.3549081 [7] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. 2016. An Empirical Study of the Impact of Modern Code Review Practices on Software Quality. Empirical Software Engineering 21, 5 (2016), 2146–2189. doi:10.1007/ s10664-015-9381-9 [8] Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv preprint arXiv:2302.06590 (2023). [9] Peter C. Rigby and Christian Bird. 2013. Convergent Contemporary Software Peer Review Practices. In Proceedings of the 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE ’13). ACM, 202–212. doi:10.1145/2491411.2491444 [10] Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawendé F. Bissyandé. 2024. CodeAgent: Autonomous Communicative Agents for Code Review. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP ’24). 11279–11313. doi:10.18653/v1/2024.emnlp-main.632 [11] Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, and Marco Aurélio Gerosa. 2020. Effects of Adopting Code Review Bots on Pull Requests to OSS Projects. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME ’20). IEEE, 1–11. doi:10.1109/ ICSME46990.2020.00011 2 https://anonymous.4open.science/r/reviewer-behavioral-shift-FE86