(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

arXiv:2610.13668v1 [cs.SE] 9 Oct 2026

CHENYANG YANG∗ , Carnegie Mellon University, USA HAO HE and TI-CHUNG CHENG, Microsoft, USA JENNA BUTLER and EMERSON MURPHY-HILL, Microsoft, USA CARMEN BADEA and CHRISTIAN BIRD, Microsoft Research, USA With the advances of coding agents, software developers are spending more time reviewing code than writing code. As human review velocity falls behind agentic code generation, it is tempting to build automated review agents towards a future of “lights-off” software factory with little to no human supervision. To assess the practicality of such a fully autonomous software factory, we construct CRBugBench, a new benchmark that evaluates whether review agents detect human-verified bugs. We build a data pipeline that mines bugs from repository history, and pair each bug with its introducing PR to form a review task. CRBugBench consists of 1,000 review tasks across 98 open-source GitHub repositories, and 800 additional tasks from proprietary industry repositories at one large company. We then use the benchmark to study agent review quality: We found that SOTA review agents do well: the best (claude-opus-5 with copilot-cli) catches 71.9% of bugs within its top-5 comments, up from 47.7% (opus-4.6) in five months of model evolution. Yet agents remain less effective on complex changes and bugs requiring non-local reasoning, with sharp performance drops on proprietary industry repositories where project context is more often needed. To improve bug detection, we found scaling compute helps most through prompt changes: exploration-nudging prompts gain up to 5.8% performance at 10% higher cost. From these findings, we derive concrete guidance for developers to adopt review agents – scaling review effort to PR complexity, steering agents by project priorities, supplying repository knowledge, dispatching models by PR characteristics, and monitoring quality as models evolve.

1

Introduction

Software engineering is increasingly automated. From code completion [11], chat assistants [23], to coding agents [3], the task of writing code has been shifted from human developers to AI. Developers are now increasingly doing more verification work, commonly through code review [1]. Yet the velocity of human reviews cannot catch up with agent-generated code. It is tempting to further automate humans away, by using AI for code reviews [4]. The extreme version of this would be a so-called “lights-off” software factory [36], where AI both writes and reviews code, with little to no human supervision in the loop. Are current repositories ready to become lights-off repositories? To answer that question, we first need to understand how well AI can review code. Without good review capability, the engineering loop cannot be fully closed. While many code review benchmarks exist (e.g., CodeReviewer [22], SWR-Bench [49], c-CRAB [51]), they typically treat human comments as ground truth and expect AI-generated reviews to match them, lexically or semantically. Human comments, without filtering, are a poor proxy of review ground truth. They can be arbitrary, containing nitpicking or less useful comments that are accepted due to social dynamics [13, 35]. They can also be too narrow, missing valid points that AI reviews can bring up. Therefore, assessing review quality based on the ground truth of human reviews is misleading: AI may provide high-quality review, just differently than human reviewers [41]. It would be hard to cleanly interpret the outcomes from these benchmarks. ∗ Work done at Microsoft Research.

Authors’ Contact Information: Chenyang Yang, Carnegie Mellon University, USA; Hao He; Ti-Chung Cheng, Microsoft, USA; Jenna Butler; Emerson Murphy-Hill, Microsoft, USA; Carmen Badea; Christian Bird, Microsoft Research, USA. , Vol. 1, No. 1, Article . Publication date: October 2026.

2

Yang et al.

Pull request to review

Sandboxed evaluation Review agent Sandboxed repo: pandas @pk4v6XY + diff Future commits removed; restricted Internet access Write: review.md ## Issue 2: `astype(squeeze=True)` raises `TypeError` whenever the target dtype is a NumPy dtype **File:** `/app/pandas/pandas/core/ internals/blocks.py:724-728` **Severity:** Critical **Problem:** `squeeze` is passed unconditionally (`squeeze=b.ndim != 1`) for every per-column astype, but squeezing is only correct when the *target* dtype is a 1D-only EA. […] This turns very ordinary `convert_dtypes()` calls into hard errors. **Evidence**: […]

LLM judges Match each identified issue with the ground-truth bug

Matched: Yes Justification: […]

Fig. 1. CRBugBench identifies historical human-verified bugs, and traces them back to the bug-inducing PR commit. Review agents are provided with a sandboxed repository at the commit to explore and experiment, producing a final review.md file, with findings self-ranked by severity. A panel of LLM judges then compares each finding in the review to the ground-truth bug, and labels whether there are matches.

To provide a more objective assessment of review quality, we curated a code review benchmark that evaluates how well a PR review detects human-verified bugs. The intuition is that, a good PR review, should at least catch the bugs that a human has confirmed and fixed. We design, iterate, and validate a scalable data pipeline to collect diverse bugs from repository history. We applied this data pipeline to both open-source repositories on GitHub and industry proprietary code repositories at one large company and collected verified bugs from two sources: First, from addressed human comments in PR discussions – these are bugs that are identified and fixed before the PR is merged. We identified them by finding all human review threads and filtering them down to the ones that leave bug-related comments and are addressed later. Second, from human-submitted bug-fix PRs – these are bugs that are identified after a PR is merged. We identified them by finding all bug-fixing issues and PRs, and traced them back to the PRs that introduced the bugs. These bugs then serve as the ground truth that a review agent should identify from the introducing PRs. In the end, we curated CRBugBench, a bug-grounded code review benchmark with 1,800 code review tasks, across 98 GitHub open-source repositories, 2 large industry repositories, and 5 programming languages. These tasks cover diverse PR sizes and types, as well as diverse bugs with different categories, sources, and complexity (Figure 4). We then use CRBugBench to systematically study agent review quality. First, we evaluate standalone review agents with different models, and ask: How well do review agents detect PR bugs (RQ1)? We found that review agents have a good success rate on detecting bugs in GitHub PRs: The best review agent (with claude-opus-5-xhigh) catches 79.6% of bugs that humans found before merge, and 64.2% of bugs that humans found after merge, within top-5 review comments. This bug-catching review capability has steadily increased over model evolution, going from 47.7% , Vol. 1, No. 1, Article . Publication date: October 2026.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

opus-5

80% Opus 5

70

opus-5 (Claude Code)

gpt-5.5 + compr. opus-5 + compr.

70%

65 GPT-5.4

gpt-5.6-sol + compr.

Opus 4.6

Average recall@5

opus-4.8 + compr.

Opus 4.8 GPT-5.5

GPT-5.6

50%

40%

30%

20%

Average recall@5 (%)

60%

3

60

55

Mar

May

Jul

Model release date GitHub OSS Anthropic

Industry proprietary OpenAI

gpt-5.6-sol + self-refine

opus-4.8 (Claude Code) sonnet-5

gpt-5.4

gpt-5.5

opus-4.6 max

45

gpt-5.6-sol ensemble@3

gpt-5.6-sol

50

35

0%

gpt-5.6-sol ensemble@5

gpt-5.6-luna

ensemble@3: mixed models

gpt-5.6-sol + sub-agents

gpt-5-mini high

40

10%

opus-4.8

gpt-5.6-luna max

gpt-5.6-luna medium

gpt-5.6-luna low

haiku-4.5

30 $0.05

$0.1

$0.2

$0.5

$1

$2

$5

$10

$20

Average cost per task (log scale) Single agent OpenAI

Comprehensive prompt Anthropic

Multi-agent

Alternative harness

Mixed

(a) (b) Fig. 2. Frontier models’ review capabilities have steadily increased over six months in 2026, with claude-opus-5 as the best-performing model (Left). The performance-cost Pareto front is dominated by small OpenAI models (gpt-5.6-luna) for cheaper choices, and large Anthropic models (opus-4.8, opus-5) for most-performant options. Multi-agent configurations provide improvements at a large cost (Right).

(opus-4.6) to 71.9% (opus-5) from Feb to July 2026 (Figure 2a). However, despite the promising trend, agents still miss 20%+ bugs human found, indicating a need for human-in-the-loop review. We corroborate our results with CRBugBench’s proprietary portion built from industry repositories that are (1) industry-scaled and (2) certainly not part of models’ training data. We found that review agents struggle much more with the proprietary portion, dropping up to 25.5% performance, but with a similar promising trend of increased performance over model evolution (Figure 2a). The sharp drop is partially explained by different task distribution: tasks sourced from industry repositories contain far more bugs that are not directly inferrable from the diffs – they require more project contexts to be confirmed. Next, we study the conditions for successful agent reviews. We collected features on repositories, PRs, and bugs, and ask: How do different factors impact review quality (RQ2)? We modeled the probability that a review detected a bug using multivariable logistic regression, with our collected features. We found that review agents are less effective for catching bugs in larger and more heterogeneous PRs, or bug-fixing PRs. Non-functional or less severe bugs are harder to detect, as well as bugs that require non-local reasoning. This shows that review quality is heterogeneous, and developers need to form calibrated trust on review agents based on the PR context. We then examine agent trajectories, and compare the errors different agents have made, and ask Where do agents spend time and why do they fail (RQ3)? We found that agents spend most of their turns reading and searching code but rarely execute it, with the best model claude-opus-5 executing code in 9.7% of turns (vs. gpt-5.6-sol in 6.9%). Agents also differ substantially in their persistence: half of claude-opus-5 runs end by turn 35, compared with turn 17 for gpt-5.6-sol. Our qualitative analysis also reveals cases where agents trust misleading PR descriptions or code , Vol. 1, No. 1, Article . Publication date: October 2026.

4

Yang et al.

comments, or fail to account for repository-wide behavior and external library semantics, reflecting limitations of current frontier models. Finally, we experiment with various agent design dimensions, and ask: What agent design choices improve bug-detection performance (RQ4)? We found that scaling compute is effective, and most effective through prompt changes. Exploration-nudging prompts improve 5.8% performance (1.10x cost), while multi-agents improve 4.7% performance (5.55x cost). Increased reasoning efforts can improve model performance from 36% to 57% at 23.22x cost. There is a clear trade-off to make in the cost-performance frontier (Figure 2b). Based on empirical findings from our benchmark, we make recommendations for developers to adopt code review agents (Section 5). We recommend that developers scale their (automated and human) review efforts based on PR complexity, and steer agent review directions based on project or PR priorities. They should also incorporate repository knowledge for better review quality, especially when they work on large proprietary repositories. If cost is a concern, they should consider dispatching different models and agent configurations based on PR characteristics. With evolving model behaviors, it would be a good idea to monitor review quality and adapt the harness periodically. We also recommend developers to apply the methodology of CRBugBench and build their own customized benchmark of their software repositories. To summarize, this paper contributes: • A scalable data construction and evaluation pipeline that transforms a repository into a suite of PR review tasks with human-verified bugs (Section 3). • CRBugBench, a constructed benchmark from open-source repositories (Section 3). • A comprehensive evaluation of review agents, with an analysis of when review agents work and when they do not (Section 4). • Actionable guidance for developers to adopt review agents (Section 5). 2 2.1

Background and Related Work Modern code review

Code review originated in formal, synchronous inspection processes with prescribed roles and meetings [17]. Modern code review is a more lightweight, asynchronous process, usually involving rapid feedback from a small number of reviewers [5, 33, 34]. There is empirical evidence that code review contributes to software quality, but its outcomes are heterogeneous. Greater review participation is associated with fewer post-release defects [28], while the usefulness of a particular comment depends on both its technical content and its context. Large-scale studies show that useful feedback includes defects, design alternatives, and maintainability guidance, and that larger changes tend to receive a lower proportion of useful comments [13, 46]. We intend CRBugBench to focus on the more objective aspects (bug-finding) of code review, but leave the more subjective or domain-specific aspects to human reviewers. 2.2

Benchmarking code review

The most popular approach to evaluate code review is to collect human review comments and evaluate generated comments against them, with text-overlap metrics such as BLEU or CodeBLEU or LLM-based evaluators: Early benchmarks [22, 44, 45] construct method-level code paired with human review comments, while more recent benchmarks like SWR-Bench [49], ContextCRBench [19], and c-CRAB [51] have moved to full-project, PR-level evaluation. In particular, c-CRAB converts human comments into executable tests and measures whether a separate coding agent can turn a review into a test-passing repair. These benchmarks are grounded primarily in reviewer comments before PR merge, and their reported scores can artificially penalize valid review findings that the , Vol. 1, No. 1, Article . Publication date: October 2026.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

5

original reviewers missed. An alternative approach is to use LLMs to help augment and evaluate code review. AACR-Bench [50] augments reviewer comments with human-validated LLM-generated review comments, but this approach cannot assess performance on bugs that LLMs fail to identify in the first place. DeepCRCEval [26] and CRScore [30] use LLM-based evaluators to evaluate various aspects of code review (e.g., conciseness, comprehensiveness), but their evaluations are limited by the reliability of LLM evaluators and do not measure historical confirmed-bug recovery. Our benchmark takes a complementary approach: we identify bugs verified by human-authored change, including defects that were caught or escaped the original review, and then evaluate directly whether a review agent can identify that defect in the introducing PR. This provides a recall-oriented test of whether review agents recover known, consequential issues across realistic repository contexts, instead of a test of textual similarity or a general measure of comment style. Our approach connects to the established literature on post-release defects and defect prediction [12, 15], and similar to recent work that evaluates code review using discovered defects: Lu et al. [25] reconstruct introducing merge requests from industrial fault reports, while CR-Bench traces SWE-bench fixes to introducing PRs and evaluates reviews against the known bugs [31]. Compared to these works, we contribute a scalable recipe to curate diverse bugs and task environments from software repositories, and apply the recipe to create a benchmark grounded both in public and proprietary repositories. 2.3

Automating code review

Automation has long supported individual parts of code review, including static analysis, reviewer recommendation, change prioritization, and suggested revisions. For example, Review Bot combined static analysis with reviewer recommendation to surface findings inside the review workflow [6]. Neural approaches later learned edits or comments from historical review data [22, 44, 45]. LLMbased systems extend this line of work to open-ended agentic review that can inspect repository context, use development tools, and produce several candidate findings [40, 41]. Recent evidence nevertheless cautions against over-trust of LLM-generated reviews. A controlled study with expert developers found that LLM-generated reviews redirected reviewers’ attention and increased the number of low-severity issues found, but did not increase detection of high-severity issues, save review time, or improve confidence [43]. An industrial study of 4,335 pull requests found that developers acted on many automated comments, while also reporting faulty, unnecessary, and irrelevant feedback and observing longer average PR closure times [14]. A recent roadmap consequently frames the field around context-aware, value-driven, human-AI review rather than replacement of human reviewers [48]. Our work advances this agenda by evaluating end-to-end review agents on bug detection, characterizing the change and repository conditions under which they fail, to inform when to allocate human oversight. 3

Data Collection

In this section, we describe our data collection methodology, from how we select repositories, how we identify human-verified bugs, to how we create task environment for review agent evaluation (an overview is provided in Figure 3). 3.1

Repository selection

We selected five popular programming languages: Python, Rust, C#, JavaScript/TypeScript, and Java, based on StackOverflow’s survey [39]. For each language, we identified the most popular, well-maintained engineering repositories with substantial activities. We first identified top-1,000 starred GitHub repositories, and then filtered down to ones with more than 2,000 pull requests (PRs) and more than 200 contributors. We further filtered out repositories that are not engineeringfocused (e.g., tutorials), that are created after 2025, or have no activities within a year, to focus on , Vol. 1, No. 1, Article . Publication date: October 2026.

6

Yang et al.

1

Identify repository • Python, Rust, C#, JS/TS, Java • Top 1k stars • >2k PRs, >200 contributors • Created before 2025 • Activity within a year • Engineering-focused

Domains: Web, AI, dev-tools, … One per app. domain x language

106 repos

2

Identify bugs

3

Bugs identified before merge

Bugs identified after merge

Identify merged PRs with comments

Identify bug-fixing PRs with closed issues

Identify bugmentioning comments that are resolved

Trace bugs back to the inducing PR; filter out PRs with low SZZ confidence

Filter out non-bug comments with LLMs

Filter out issues that are not bugs or poorly described; Remove SZZ noise

2 private industry mono-repos

Create tasks Create Docker containers from PR base commit Create diff-to-review from base to buginducing commit Create ground-truth from comments or issues for LLM judges 1000 tasks, 500 per track 800 tasks, 400 per track

Fig. 3. CRBugBench tasks are created from human-verified bugs in repository history. We identify bugs from human comments and bug-fixing PRs, trace them back to the inducing PR, and create containerized tasks.

established, actively maintained software projects. From the remaining repositories, we annotated their application domains (e.g., AI/ML, databases, security, DevOps, web frameworks, gaming, embedded, etc.) with an LLM, manually verified that the annotations are reasonable, and then sampled one repository per domain for each language. This gives us a final of 106 repositories1 spread across 31 application domains (statistics in Figure 4). 3.2

Bug identification

Bugs identified before merge. First, we identify bugs that developers have caught during PR review – i.e., they are mentioned in PR discussion and fixed before merge. We collect these tasks in 2 steps: (1) Mining. We first collect all merged PRs and their discussion threads from GitHub API. We filter out PRs without any review threads. This yields 304,601 PRs with 1,436,120 review threads. We then apply deterministic quality filters to remove low-quality comments, filtering out comments that are bot-authored, self-authored, too short, on non-source-code files, or are rejected. After filtering, 472,039 candidate threads remain across 117,522 PRs. (2) Sampling and LLM filtering. Because the surviving population is far larger than what we can evaluate, we draw a stratified sample of 90 threads per repository. Within a repository we order PRs by merge time, split them into 10 equal-count temporal strata, and make sample spans equally across the repository’s history. This yields 8,639 threads. We then use an LLM to further label and filter comments that are not bugs (e.g., style, naming, tests, nits, refactors, and concerns the replies show were clarified or declined). This results in 865 vetted bugs across 99 repositories. Bugs identified after merge. Second, we identify bugs that developers caught after a PR is merged – i.e., they are mentioned in a later issue and fixed in a later PR. We collect these tasks in 4 steps: (1) Mining. We first collect all merged PRs with fix-related keywords in their titles or labels. This yields 195,075 candidate PRs. We then identify closing issues for each PR, and remove the ones without substantive descriptions (less than 80 characters), yielding 57,725 fix PRs. (2) Sampling and LLM filtering. Because the surviving population is far larger than what we can evaluate, we again draw a stratified sample of 200 PRs per repository. We then have an LLM label and filter issues that are not bugs and hence unpreventable (new features, refactors, documentation, performance work). This yields 10,784 fix PRs. 1We collected the repositories in June 26th, 2026.

, Vol. 1, No. 1, Article . Publication date: October 2026.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

7

(3) Linking to bug-inducing PR. We then trace each defect back to its inducing PR with SZZ [37]: We identify the deleted/modified lines and use git blame to obtain the commits that last touched them; and map those commits to PR numbers using the repository’s referencing convention. The primary inducing PR is the one accounting for the largest share of blamed lines. This yields 8,846 fix–inducing pairs. We keep the top 12 pairs per repository ranked by SZZ blame share, resulting in 1,168 pairs. Since SZZ is noisy (a refactor that last-touched a line can be attributed as if it had introduced the defect), we further use an LLM (claude-opus-4.8) for quality control. The LLM is asked to classify the type of the inducing change and whether the blamed lines introduce the bugs, yielding 736 pairs. (4) Ground truth screening. Finally, we filter out poorly-described issues with an LLM. We drop issues with no reported faulty behaviors (screenshot-only reports, unfilled templates, bare one-liners), non-bug issues that are vague or off-target, and issue/fix mismatches. This yields a final set of 689 fix–inducing PR pairs. Data quality control. We iterated our bug identification pipeline over multiple weeks, through validating the quality of curated bugs and adding additional deterministic or LLM filters. We randomly sampled 40 identified bugs from the final pipeline and assessed whether each reported bug (i) represented a concrete defect and (ii) could be detected by reviewers using the available context. A task was labeled valid only if both criteria were satisfied. Overall, 37 tasks (92.5%) were valid, with an approximate 95% Wilson confidence interval of 80.1-97.4%. 3.3

Task creation

From the identified and verified bugs and bug-inducing PRs, we construct our review tasks. For each PR, we identify the base commit and the reviewed commit. For bugs identified before merge, the reviewed commit is the commit that carries the ground-truth bug comments. For bugs identified after merge, the reviewed commit is the final commit before merge. We then construct the baseto-reviewed-commit diff for review. We create a DockerFile per task and remove ones with build failures. We then sample the tasks down to 500 per track. Task format. Each task instance contains a Docker snapshot of the repository at the inducing PR’s base commit, with history through that commit intact but all later references deleted to prevent leakage. The PR title and description are provided as context, and the change is provided both as a file and uncommitted working-tree modifications. The agent reviews the change without being told that a bug exists and writes its review to review.md. The review is structured as a list of review items, self-ranked by the agent based on severity, following popular review agent design [4]. We restrict Internet access and explicitly prompt the agent to avoid cheating behaviors. We later conducted an AI-assisted auditing of agent trajectories and did not find any cheating behaviors. Task scoring. Each review finding is scored by an LLM-as-judge panel against the ground truth (bug-mentioning comments or issues) and annotated if there is a match. To mitigate judge biases towards same-family models and single-judge instability [52], we use three judge models: gpt-5.5, claude-opus-4.8, and gemini-3.1-flash, and take the majority vote as the final result. We independently annotated 40 samples of agent reviews and found that we achieved almost perfect human-LLM agreement (𝜅 = 0.89). We also observed that judges are fairly consistent with each other, achieving almost perfect across-LLM agreement (Fleiss’ 𝜅 = 0.95). For evaluation, we measure recall@k as the proportion of tasks for which the ground-truth issue is identified among the agent’s top-k findings. Varying k captures different limits on developer attention and team review budgets. , Vol. 1, No. 1, Article . Publication date: October 2026.

8

Yang et al.

Repositories

a Repository corpus 20 20 10

0

0 1K

10K

100K

1M

1K

Repositories

Stars

10K

100K

1M

Pull requests 40

10 20

5 0

0 12 3 4 5 6 7 8 91011121314151617 18

Age (years)

10 100

1K

Data science 4 Developer tools 4 Gaming 4 LLM apps 4 Networking 4 Utilities 4 Web app 4 Web framework 4 GUI desktop 3 Mobile 3 Prog. language 3 AI agents 2 Blockchain 2 Computer vision 2 Embedded 2

Data engineering 4 CLI tools 4 Automation 4 Security 5 Finance trading 5 DevOps 5 Database 5 API tools 5 AI / ML 5 Self-hosted 1 Operating system 1 Middleware 1 Game engine 1 Frontend UI 1 Scientific 2 NLP 2

10

10K100K

Source files in snapshot

b Task profile Comment

Fix

Tasks

60

Feature Fix Refactor Performance Build CI

40 50 20 0

0 1

10

100

1K

2015

Changed lines

Functional Reliability Usability Maintain. Efficiency Compat.

2020

2025

In diff

100

High Medium Low 0

Quality impact

200

Critical

Cross-file

200

100

Change type

Domain

Runtime 0

0

Year of reviewed commit

100

200

300

0

Review cue

100

200

300 400

Severity

Fig. 4. Statistics of collected repositories.

3.4

Benchmark characteristics

Applying our data pipeline to GitHub repositories, we collect CRBugBench, with 1,000 tasks across 98 popular and established open-sourced repositories (Figure 4). The tasks are spread out from 2015 to 2026, with most reviewed changes single-language (82%) and a median churn of 189 lines. Task annotations. We in addition annotate various task attributes for downstream analysis. (1) Code metrics. We collect diff size, file kinds, languages, and repository size. We also capture filelevel dependency graph with CodeQL and compute the number of dependent / dependency files. (2) LLM annotations. We further create rubrics and use an LLM to annotate various semantic attributes of code change and bugs: We annotate change type (following taxonomy in Li et al. [21]), bug impact based on ISO/IEC 25010 quality impact [20], plus bug severity and bug visibility. We validated a sample of 40 annotations and found that we have almost perfect human-LLM agreement for change type and bug impact (𝜅 = 0.88, 0.82 respectively), and substantial agreement for bug severity and visibility (𝜅 = 0.64, 0.68). For severity, we group critical / high and medium / low in our analysis, which yields a higher 𝜅 = 0.90. , Vol. 1, No. 1, Article . Publication date: October 2026.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

3.5

9

Proprietary benchmark construction

Besides the public benchmark, we applied the same data pipeline on two industry mono repositories on enterprise platforms for online collaboration and productivity, collecting 800 additional review tasks. These tasks are sourced from repositories with an average of >100k source files, which is a much larger scale compared to open-source repositories (an average of 3k source files). 4

Experiments

Next, we use our benchmark to evaluate review agents’ bug-detection performance. We first evaluate across different model configurations to understand how well review agents detect PR bugs (RQ1). We then use the collected data to do a large-scale empirical analysis, to understand what factors impact review agents’ bug detection capabilities (RQ2). We then analyze the agent trajectories to understand where do agents spend time and why do they fail (RQ3). We finally evaluate with different harness variations, to understand how agent design choices impact bug detection (RQ4). 4.1

Research Design

Evaluating and comparing agentic models (RQ1). We first evaluate across a wide range of large and small models with strong coding capabilities: • Large frontier models: claude-opus-5, claude-opus-4.8, claude-opus-4.6, gpt-5.6sol, gpt-5.5, and gpt-5.4. These represent large, strong coding models available at the time of our experiments. • Smaller and older models: claude-sonnet-5, gpt-5.6-luna, claude-haiku-4.5, and gpt-5mini. These represent decent coding models that are cheaper and smaller. To control for variations in agent harnesses, we use copilot-cli, pinned to 1.0.74, as the agent harness to evaluate and compare different models.2 For the review prompt, we use the official prompt provided by copilot-cli in the /review command. For all models, we use a reasoning effort of xhigh when available. Otherwise, we use the max available reasoning effort for the model. In addition to the main performance metric recall@k, we also report (1) cost metrics, including token usage, estimated cost (in USD), and latency, and (2) review length metrics, including number of findings and words. For the proprietary benchmark portion, we evaluate four top-performing models: claude-opus-5, claudeopus-4.8, gpt-5.6-sol, and gpt-5.5. We compare their results to the open-source portion of our benchmark. Analyzing variation in agent performance (RQ2). To identify the conditions under which review agents are more or less successful, we model the probability that a review detects a bug using multivariable logistic regression. Because the literature on LLM-based code review is still nascent, we draw on the well-established literature on human code review to motivate our hypotheses, under the assumption that factors known to make changes harder for human reviewers are likely to affect review agents in similar ways. Specifically, we draw on findings that larger changes are reviewed less effectively and attract fewer useful comments [8, 13, 32], and composite changes that mix multiple concerns are harder to review [7, 42]. Review quality also varies with the kind of artifact under review: test code receives less review discussion than production code [38], and build and configuration files attract fewer useful comments [13]. Finally, studies show that delocalized defects that require reasoning about code outside the inspected artifact, are substantially harder to 2We ablate this choice with claude-code in RQ4 and observed better performance from copilot-cli.

, Vol. 1, No. 1, Article . Publication date: October 2026.

10

Yang et al.

find [8, 16], and the majority of issues raised in code review concern maintainability rather than functional defects [9, 27]. Motivated by these findings, we test the following hypotheses: H1: Detection probability decreases with higher PR change complexity. We measure change complexity with number of change lines, the number of dependent files, and the number of dependency files. H2: Detection probability is associated with PR change type. H3: Detection probability is associated with PR change language. H4: Detection probability decreases with higher PR change heterogeneity. We measure change heterogeneity with the number of change types within one change (e.g., on source code, tests, build files, etc.), and with the file extension entropy. H5: Detection probability decreases with higher repository complexity. We measure repository complexity with the number of lines in the repository snapshot. H6: Detection probability increases for diff-visible bugs. H7: Detection probability is associated with bug impact. H8: Detection probability is associated with bug severity. All predictors are entered simultaneously, with continuous measures standardized where applicable. We also include the model used for review, task track, and calendar time as adjustment covariates. review_caught ∼ log_num_changed_lines_z_score + log_dependent_files_z_score + log_dependency_files_z_score

+ log_repository_nloc_z_score + num_subchange_types + extension_entropy_z_score + bug_is_local_visible + high_severity_bug + 𝐶 ( change_type, Treatment( ref = feature ) ) + 𝐶 ( language, Treatment( ref = Python ) ) + 𝐶 ( bug_iso_impact, Treatment( ref = Functional Suitability ) ) + 𝐶 ( model_key ) + 𝐶 ( track ) + 𝐶 ( calendar_time )

Because a task could have multiple review executions, observations from the same task may be correlated. We therefore cluster standard errors by task and applied finite-sample and degrees-ofˆ and its freedom corrections. For each scalar predictor, we report the adjusted odds ratio, (exp{ 𝛽}), 95% cluster-robust confidence interval. The adjusted odds ratio describes the change in the odds of the outcome associated with a one-unit increase in the predictor, holding the other predictors constant. We test each scalar predictor using a one-degree-of-freedom cluster-robust Wald (F)-test. For each categorical predictor, we use an omnibus cluster-robust Wald (F)-test to assess whether its levels are jointly associated with the outcome. We report adjusted odds ratios and 95% confidence intervals for individual levels relative to the reference category. We hypothesize that differences among models vary across data subsets. To test this hypothesis, we fit an additional regression model with interactions between model and the other factors. We further examine whether the associations between these factors and bug detection differ between human reviewers and LLMs. For this analysis, we fit a model incorporating human-detection observations identified from bug sources and test interactions between is_human_review and the other factors. To account for multiple testing, we apply the Benjamini-Hochberg procedure [10] to control the false discovery rate at 5%. We also confirmed that the fitted model converged, its design matrix was full rank, and that multicollinearity was within our specified thresholds (condition number < 30, the maximum variance inflation factor < 10). Analyzing agent trajectories (RQ3). To characterize how agents spend their budget, we instrumented the trajectories of all executions. Each tool call is classified into one of six actions by tool name, and, for shell calls, by parsing the command into segments and mapping the head verb: Read (open a file), Probe (search or list), Write (create or edit), Execute (run tests, builds, or ad-hoc scripts), Git (inspect history or diffs), and Other. , Vol. 1, No. 1, Article . Publication date: October 2026.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

11

To characterize why agents fail, we analyze the six frontier models and decompose failures into three groups, by matching file paths explored in the trajectory and the ground truth: whether the agent inspected the file holding the ground-truth defect, whether the review discussed that file, and whether the discussion was judged a correct diagnosis. We then complement this with an inductive open coding of 20 sampled tasks with diverging review agent outcomes. Each case was inspected and annotated on how the agents succeed or make mistakes. Evaluating and comparing agent harnesses (RQ4). We finally evaluate across various agent configurations, testing how the choice of harness, prompt, and orchestration impacts agent performance. • Reasoning efforts. We ablate reasoning efforts for gpt-5.6-luna, with reasoning efforts set to low, medium, and max. • Harness. We ablate the harness with claude-code for claude-opus-4.8 and claude-opus-5 using its /review command. • Prompt. We ablate the default copilot-cli review prompt with a prompt that encourages comprehensiveness. The default prompt optimizes for signal-to-noise ratios (i.e., models are encouraged to only output important findings). • Orchestration. We ablate the default single-agent loop with three ways to scale up compute efforts. The first is through self-refine, where we have a second agent to improve the reviews from the first pass. The second is through candidate-ensemble, where we have an orchestrator agent that collects several independent candidate reviews and consolidates them into one set of findings. We vary both how many candidates are pooled (three or five) and whether those candidates come from the same model or from different models. In the case of different models, we sample candidates from claude-opus-4.8, gpt-5.5 and gpt-5.6-sol. The last is through sub-agents, where we have an orchestrator agent that creates sub-directions to explore, and invoke sub-agents on demand, and create consolidated findings. 4.2

Threats to Validity

Construct validity. Our benchmark only measures review agents’ bug detection capabilities. Our results should therefore be interpreted as recovery of known bugs rather than as overall review usefulness. Our evaluation uses LLM judges, which may be unreliable [52]. We mitigate this by using a three-model panel and calibrate human-LLM agreement. Internal validity. Despite multi-stage filtering, our benchmark construction can still yield incorrect ground truths or unsolvable tasks. We mitigate this with extensive data quality control and manual verification of samples. Agent behavior is also stochastic and sensitive to changes, which we mitigate with pinning the primary harness and holding the prompt and task environment fixed. Our observational regressions estimate adjusted associations, not causal effects. Unmeasured factors and LLM annotation errors may confound estimates or attenuate effects. External validity. We cover popular open-source repositories across domains and two large industry monorepos. These choices improve diversity but do not represent all software development. Our selection favors mature, active, popular projects and source-code changes with detailed textual evidence and reproducible environments. Results may differ for smaller or younger projects, uncommon languages, generated or configuration-heavy code, and other review practices. 4.3

How well do review agents detect PR bugs (RQ1)?

First, we found that review agents are decently strong at detecting bugs (Table 1): The top-5 findings from claude-opus-5 cover 79.6% bugs that are identified by humans before merge, and 64.2% bugs that are identified after merge. This only costs $2.97 per task, with an average latency of 9.8 minutes. We also observe a steady increase of review capability from frontier models: From February to July, , Vol. 1, No. 1, Article . Publication date: October 2026.

12

Yang et al.

Table 1. Bug-detection performance and cost across model configurations on GitHub repositories. Run

Track Recall

Review

Tokens

Cost Latency

@1 @3 @5 @10 Count Words In Cache Out (K) (M) (M) (K)

($)

(min)

Cmt. Fix Cmt. Fix Cmt. Fix Cmt. Fix Cmt. Fix Cmt. Fix

48.3 36.2 45.6 32.0 45.4 27.0 43.5 29.6 43.7 29.0 42.6 28.4

71.2 55.4 65.9 44.6 58.7 35.0 61.9 45.6 61.6 45.4 59.5 42.6

79.6 64.2 70.0 46.6 59.1 36.4 65.6 49.0 63.3 47.8 61.8 45.4

86.0 68.6 71.5 47.8 59.3 37.0 66.6 51.8 63.5 48.6 62.0 45.6

5.76 6.78 2.71 2.84 2.08 2.46 2.72 3.57 2.18 2.66 2.28 2.72

1.08 1.28 0.61 0.67 0.36 0.45 0.34 0.43 0.28 0.33 0.30 0.35

2.43 3.37 1.23 1.66 0.90 1.40 1.01 1.64 1.65 2.17 1.91 2.49

2.34 3.27 1.17 1.59 0.86 1.35 0.86 1.48 1.49 2.00 1.70 2.23

34.3 43.9 24.0 30.4 12.5 17.5 10.4 16.3 11.2 15.3 21.5 26.6

2.55 3.38 1.54 1.98 0.99 1.44 1.50 2.03 1.86 2.33 1.27 1.59

8.6 11.1 6.5 7.9 4.1 6.0 3.7 5.2 4.0 5.3 4.3 5.3

Cmt. Fix gpt-5.6-luna Cmt. Fix claude-haiku-4.5 (none) Cmt. Fix gpt-5-mini (high) Cmt. Fix

47.8 30.6 43.5 30.0 33.0 17.6 38.4 21.4

62.5 43.2 60.4 44.6 41.2 21.8 52.7 31.2

64.7 45.2 63.9 47.8 43.0 23.0 54.7 33.4

65.7 45.6 65.4 50.2 43.8 23.2 55.3 33.8

2.65 3.04 2.76 3.52 2.58 3.16 2.72 3.48

0.69 0.80 0.32 0.40 0.44 0.52 0.54 0.71

3.26 4.56 2.24 3.33 0.91 1.20 1.40 1.57

3.18 4.46 2.04 3.07 0.87 1.15 1.14 1.30

35.1 45.9 17.5 24.9 13.4 16.0 13.5 19.4

1.18 1.60 0.51 0.72 0.21 0.26 0.12 0.14

9.1 11.7 3.1 4.3 2.2 2.7 5.3 5.4

claude-opus-5 claude-opus-4.8 claude-opus-4.6 (max) gpt-5.6-sol gpt-5.5 gpt-5.4

claude-sonnet-5

45 100

40

Share of cases (%)

30 25 20 15 10 5

82.1 71.9

Opus 4.6

60

50.0 40

20

0 1

2

3

4

5

Position of first hit in the review list

6+

GPT-5.4 GPT-5.5 GPT-5.6 Opus 4.8 Opus 5

80

migrate from

Issues found (%), agent at recall@5

migrate to

35

GPT-5.4 GPT-5.5 GPT-5.6 Opus 4.8

0

Human

Agent

5.5

5.7

9.3

11.5

30.4

14.2/8.6

13.6/7.8

16.7/7.3

15.1/3.6

30.8/0.4

0.2

3.8

5.9

24.8

6.9/6.7

9.9/6.1

12.8/6.8

26.0/1.2

3.6

5.7

24.6

8.2/4.6

12.3/6.5

25.7/1.1

2.1

21.0

11.1/8.9

22.2/1.2

18.9 19.7/0.8

Human + agent

Fig. 5. Bug detection performance and model migration gains. (a) Distribution of the first correct finding’s position in the review list, with correct findings concentrated near the top. (b) Detection rates for human review, agent review, and their combination, highlighting their complementary coverage. (c) Pairwise gains from migrating between models, with darker shading indicating larger gains.

the review agents have gone from catching 47.7% to 71.9% (with opus-4.6 vs. opus-5, ***) of bugs, averaged across both pre-merge and post-merge bugs. Noticeably, the models are also fairly strong at ranking their findings: the findings that are a hit are heavily concentrated at the left end (89% of findings ever found is found in top-3 items, Figure 5a). This motivates us to design an improvement strategy, where we explicitly ask models to provide more comprehensive findings and then rank them by severity. This reduces the risk of discarding valid bugs early, and leverages models’ strong ranking capabilities for prioritizing the most consequential findings, which proves to be the strongest improvement (cf. Section 4.6). If we look across smaller and bigger models, we can also see a clear trade-off between performance and cost (Figure 2b). Some small models are very effective at finding bugs at their cost tier – for , Vol. 1, No. 1, Article . Publication date: October 2026.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

Reviewed diff Diff size (+1 SD)

Repository context

Defect nature

***

Dependent files (+1 SD)

Visible in the diff

Dependency files (+1 SD)

High or critical severity

*

Repository NLOC (+1 SD)

File-type entropy (+1 SD) Subchange types (+1)

13

*** **

ISO impact (vs Functional Suitability) PR type (vs Feature)

Language (vs Python)

***

Fix

Reliability

C#

Usability

Refactor

Java

Maintainability

Performance

TypeScript

Performance efficiency

Build

JavaScript

Compatibility

Rust

Portability

*

Other 0.05 0.25

1

4 10

Odds ratio for detection (log scale)

Other / config / rare 0.05 0.25

***

Security 1

4 10

Odds ratio for detection (log scale)

0.05 0.25

1

4 10

Odds ratio for detection (log scale)

Fig. 6. Odds ratios (log scale) with 95% confidence intervals from the multivariable logistic regression of bug detection. Blue marks a significant effect: *** 𝑝 < 0.001, ** 𝑝 < 0.01, * 𝑝 < 0.05; grey is not significant.

example, gpt-5.6-luna matched 78% claude-opus-5 performance at 21% cost. This suggests room to make deliberate model choices based on cost constraints. Opportunities for human-agent collaboration. Our results indicate a clear opportunity for humanagent collaborative code review. By the nature of our benchmark construction, humans alone only catch 50% of the bugs, while best agent alone catches 71.9%.3 If they were to collaborate together and combine their findings, this can catch 82.1% of the bugs. Humans and agents are also observed to complement each other on different PR subgroups, which we will discuss in next subsection. Agents fall short at large industry repositories. Overall, we have seen a significant (up to 25.5%) performance decrease when we apply the same agent on our internal repositories (Figure 2a). As we will show later, this can be partially attributed to their different bug distribution: Tasks in our internal repositories contain fewer diff-visible bugs, but require deeper domain knowledge, inspecting larger contexts, or gathering runtime evidence for successful diagnosis. ○ Key Findings (RQ1): Latest agentic models like claude-opus-5 are strong code reviewers, catching 71.9% bugs within top-5 findings at $2.97. Models’ review capabilities have steadily increased over time, but gaps remain that call for human engagement. Agents struggle more with large industry repositories. 4.4

How are task conditions correlated with review agent performance (RQ2)?

Agents struggle with PRs with larger, more complex diffs (H1, H4). First, we observe that larger diffs significantly decrease detection probability (OR=0.591, ***). This is expected, as larger diffs contain more code to review and hence dilute review attention, making bugs harder to detect. Besides the volume, we also observe that more heterogeneous diffs are harder to review (OR=0.795, *). That is, when the diffs contain multiple types of changes (e.g., on source code, tests, configs), they are harder to review. We hypothesize that this may come from that more complex changes require reviews to reason across modules and interfaces, hence harder to review. Agents struggle with bug-fixing PRs (H2). We found that bug-fixing PRs (OR=0.521, ***) are harder to review than other types of PR, when we adjust for other attributes (e.g., diff size) in regression. This may be a result of when agents review a bug-fixing PR, their attention is diverted to verify the fix is plausible, with less attention to other regressions introduced. 3 From our benchmark construction telemetry, we observe similar amount of pre-merge and post-merge bugs (49.7% vs.

50.3%). This distribution can change for repositories with more (or less) thorough code review policies. , Vol. 1, No. 1, Article . Publication date: October 2026.

14

Yang et al.

Changed lines in the diff 202

≤ 32

198

32–104 198

104–270

Fix

303

Reliability

322

Feature

494

60

Performance

95

27

Performance efficiency

391

1

22

60

Low or medium

5

9

High or critical

20

40

60

459

Visible in diff

Bug severity

4

0

Defect visibility

17

Security

172

3

52

Other

21

Portability

363

2

33 20

Build

Compatibility

Subchange types

93

Refactor

Maintainability

199

> 708

PR type 431

Usability

198

270–708

ISO impact Functional suitability

80 100

891 104

0

Tasks detected (%)

20

40

60

536

Needs extra context 0

20

40

60

80 100

Tasks detected (%)

80 100

Tasks detected (%)

Fig. 7. Bug detection rates by diff size, number of subchange types, ISO impact, bug severity, PR type, and defect visibility. Numbers beside bars indicate task counts.

Meanwhile, we do not observe any significant effects of PR language (H3). This may be because all the languages we study are popular enough that frontier models excel at them at a similar level. Repository size (H5) also has no observed significant effect, when adjusted for diff size. We hypothesize this is because, reviewing a given diff usually does not require extensive inspection and understanding of the whole repository contexts, but only the local contexts – and the size of local contexts are correlated with diff size. Agents struggle with less severe, non-functional bugs (H7, H8). Next, we look into how detection performance diffs across bug attributes. We found that agents have harder time detecting nonfunctional bugs like usability (OR=0.454, ***) bugs, with the same direction but weaker evidence for portability (OR=0.523, 𝑝=0.07) bugs, but easier time with more severe bugs (OR=1.906, **). This likely comes from how the agents are both trained and instructed to focus on high-impact issues, while less-critical issues can get neglected. Agents struggle with bugs that are not directly visible (H6). We also found that agents succeed more with bugs that are directly visible in the diff (OR=2.621, ***), compared to bugs that need additional project knowledge, cross-files check beyond the diff, or run-time checks. Comparing GitHub tasks and industry tasks, we found that PRs in our internal repositories contain far fewer locally visible bugs (22.8% vs. 46.1% in GitHub tasks), but contain many more bugs that require project context understanding. This is a significant factor in explaining why models fall short more in industry tasks: adjusting for bug visibility alone explains 20% of the performance gap. This may also reflect that industry repositories often require deeper domain knowledge that is not part of models’ training data, and the models have to compensate through additional test-time search. Model improvements are not uniform across PR types. We observe that model improvements are generally mixed across model pairs – some net gains come at the cost of 8.9% of regressions (Figure 5c). One noticeable exception comes from claude-opus-5, which provides consistent improvements with few regressions (up to 1.2%). We also observe a significant interaction between models and PR types: opus-5’s edge over the other models is concentrated on feature and bug-fixing PRs, and narrows significantly on refactors (RoR=0.568, *) and other PR types (RoR=0.476, ***).4 Human reviews catch more maintainability issues. Comparing human reviews and agent reviews, we observe a significant interaction between bug impact and reviewer identity: humans are much better (OR=2.702, *) at catching maintainability issues than agent reviewers. Yet overall, humans 4 The ratio of odds ratios (RoR) measures how models’ relative detection odds vary across conditions.

, Vol. 1, No. 1, Article . Publication date: October 2026.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

100

15

claude-opus-5

claude-opus-4.8

claude-opus-4.6

gpt-5.6-sol

gpt-5.5

gpt-5.4

claude-sonnet-5

gpt-5.6-luna

claude-haiku-4.5

gpt-5-mini

80 60 40

Runs still active (%)

20 0

100 80 60 40 20 0

50

100

150

50

100

150

50

100

150

50

100

150

50

100

150

Turn (tool-call round) Read

Write

Probe

Execute

Git

Other

Fig. 8. Tool-use composition and run duration across ten models.

are also worse at identifying more severe bugs (OR=0.450, *) and reliability issues (OR=0.591, *). This suggests that human-agent collaboration is also non-uniform: humans are better at keeping repositories more well-maintained, but agents can more tirelessly scrutinize for reliability issues and subtle yet critical bugs. ○ Key Findings (RQ2): Agents struggle with larger, more heterogeneous, and bug-fixing PRs, and with non-functional, less severe bugs or those requiring non-local reasoning. Differences in bug visibility explain 20% of the GitHub-industry performance gap. We also observe complementary strengths for humans and agents: maintainability for humans, and severity and reliability for agents. 4.5

Where do agents spend time and how do they fail (RQ3)?

We found that different agentic models execute similar loops (Figure 8): within the loop, agents usually start with git operations (64%, mostly git diff and git log), followed by a long inspection phase in which reading and searching account for 75% of all turn mass (39% Read, 36% Probe), while running code accounts only for 8%. Agents tend to establish what the change does by looking at source and history far more than by executing it. Comparing model families, Claude models run code more often than GPT models (9.7% of all turns for claude-opus-5 vs. 6.9% for gpt-5.6-sol). Qualitatively, we observe this helps claude-opus-5 discover a few bugs that are missed by others: on a yt-dlp task opus-5 retrieved roughly 25 archived pages through the Wayback CDX API, ran the extractor, and reported that livestream pages yield IDs for only two of nine tested pages; on a GitExtensions task opus-5 installed the .NET SDK and built an MSBuild probe to prove a DLL was absent from the publish output. More noticeably, different models stay in the loop for different length, resulting in different “effective” budgets. Half of the gpt-5.6-sol runs are over by turn 17 and only 3.5% survive to turn 50, whereas claude-opus-5 reaches its half-life at turn 35. This results in different levels of comprehensiveness of produced review and may explain the result discrepancy partly. Failures come from diagnostics instead of retrieval. We found 95.6% of agent executions inspect the file with defects, but only 80.2% produce a finding for that file, and 56.2% actually hit the bugs. , Vol. 1, No. 1, Article . Publication date: October 2026.

16

Yang et al.

Table 2. Comparisons of different prompting strategies for review agent performance. Model

Track Prompt

claude-opus-5

Cmt. Fix

claude-opus-4.8 Cmt. Fix gpt-5.6-sol

Cmt. Fix

gpt-5.5

Cmt. Fix

Default Comprehensive Default Comprehensive Default Comprehensive Default Comprehensive Default Comprehensive Default Comprehensive Default Comprehensive Default Comprehensive

Recall

Review

Tokens

Cost

Latency

@1

@3

@5

@10

Count Words In Cache Out ($) (K) (M) (M) (K)

(min)

48.3 48.3+0.0 36.2 37.8+1.6 45.6 47.4+1.8 32.0 32.6+0.6 43.5 45.3+1.8 29.6 30.2+0.6 43.7 42.5-1.2 29.0 30.4+1.4

71.2 71.4+0.2 55.4 53.2-2.2 65.9 69.5+3.5 44.6 47.8+3.2 61.9 65.9+4.0 45.6 48.8+3.2 61.6 62.7+1.2 45.4 49.0+3.6

79.6 79.9+0.3 64.2 62.4-1.8 70.0 75.5+5.5 46.6 52.6+6.0 65.6 71.3+5.6 49.0 54.0+5.0 63.3 67.4+4.2 47.8 53.0+5.2

86.0 87.2+1.2 68.6 70.2+1.6 71.5 78.4+6.9 47.8 55.2+7.4 66.6 74.2+7.6 51.8 57.8+6.0 63.5 68.5+5.0 48.6 55.6+7.0

5.76 9.26 6.78 10.83 2.71 4.40 2.84 5.00 2.72 3.90 3.57 5.39 2.18 3.12 2.66 3.95

8.6 10.0+1.4 11.1 13.0+1.9 6.5 7.3+0.8 7.9 9.2+1.3 3.7 5.6+1.9 5.2 8.1+2.8 4.0 4.0+0.1 5.3 5.5+0.2

1.08 1.66 1.28 1.92 0.61 0.81 0.67 0.91 0.34 0.53 0.43 0.71 0.28 0.44 0.33 0.54

2.43 2.41 3.37 3.29 1.23 1.16 1.66 1.72 1.01 0.85 1.64 1.16 1.65 1.30 2.17 1.75

2.34 2.31 3.27 3.17 1.17 1.09 1.59 1.64 0.86 0.74 1.48 1.03 1.49 1.13 2.00 1.63

34.3 39.6 43.9 50.1 24.0 27.1 30.4 36.0 10.4 15.7 16.3 22.1 11.2 14.1 15.3 18.4

2.55 2.76+0.2 3.38 3.60+0.2 1.54 1.67+0.1 1.98 2.21+0.2 1.50 1.36-0.1 2.03 1.81-0.2 1.86 1.82+0.0 2.33 1.97-0.4

The best-performing model, claude-opus-5, improves much more on producing a finding for the defect file (91.9%), but still has gaps in producing a correct diagnosis (75%). Qualitatively, we found that many of the failing tasks require knowledge absent from the diff: some require repository-wide reasoning, and some require external context such as library or protocol semantics. More interestingly, in a few cases, agents are misled by the diff: In a Selenium task, all models endorsed the buggy condition because both the in-code comment and the PR description asserted the intended behavior that the code failed to implement. In one Elasticsearch task opus-5 cited the ground-truth-defective line as its gold standard of correctness and focused on auditing other parts of the same file. These cases suggest an overtrust failure: agents accept author claims or existing code as evidence of correctness without independently verifying behavior, even when those descriptions are wrong. This reflects a larger problem of agents being gullible to its received contexts [18], and that security risks can emerge from automated review processes. ○ Key Findings (RQ3): Review agents run the same read-and-search loop and rarely execute the code they review (8%). They differ more in how long they stay in the loop (from a median of 17-turn to 35-turn). Agents struggle with diagnosis and can be gullible to misleading contexts. 4.6

How do agent design choices affect bug-detection performance (RQ4)?

First, we found that adjusting effort levels is the simplest yet effective strategy to scale up test-time compute for better performance. gpt-5.6-luna can jump from around 36% to 57% detection with efforts set from low to max, at 23.22x the cost (Figure 2b). Second, we found that prompting agents to output more comprehensive findings is fairly effective, especially for less-verbose models like gpt-5.6-sol (+5.3%), with similar cost (Table 2). Models do output more findings, but they are also strong in putting important findings towards the top. In contrast, multi-agent setups provide smaller improvements (+2.8% on average, Table 3), with a much higher cost (3.68x). This shows up in the pareto front (Figure 2b) that comprehensive prompt configurations dominate multi-agent configurations for test-time compute scaling. We, however, do notice that improvements from prompt changes vary across models. Noticeably, for claude-opus-5, the improvements are much smaller or even non-existent compared to earlier , Vol. 1, No. 1, Article . Publication date: October 2026.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

17

Table 3. Comparisons of different multi-agent configurations for gpt-5.6-sol xhigh. Track Configuration

Cmt.

Fix

Base Self-refine Ensemble@3 Ensemble@5 Ensemble@3 (diff. models) Sub-agents Base Self-refine Ensemble@3 Ensemble@5 Ensemble@3 (diff. models) Sub-agents

Recall

Review

Tokens

Cost

Latency

@1

@3

@5

@10

Count Words In Cache Out ($) (K) (M) (M) (K)

(min)

43.5 45.3+1.8 42.7-0.8 44.1+0.6 45.7+2.2 42.8-0.7 29.6 30.8+1.2 31.4+1.8 30.2+0.6 32.4+2.8 33.4+3.8

61.9 63.7+1.8 64.1+2.2 65.1+3.2 62.8+1.0 58.3-3.6 45.6 46.8+1.2 50.0+4.4 49.4+3.8 49.6+4.0 50.2+4.6

65.6 67.1+1.4 68.0+2.4 70.4+4.7 66.8+1.2 61.9-3.7 49.0 52.0+3.0 53.6+4.6 53.6+4.6 52.8+3.8 55.0+6.0

66.6 68.2+1.6 70.4+3.8 72.4+5.8 68.2+1.6 62.8-3.8 51.8 54.8+3.0 58.0+6.2 57.6+5.8 56.0+4.2 56.8+5.0

2.72 3.02 3.27 3.41 2.88 2.88 3.57 4.20 4.43 4.81 3.84 4.23

3.7 9.1+5.4 14.9+11.2 21.8+18.1 15.9+12.1 9.7+6.0 5.2 12.5+7.2 18.6+13.3 28.5+23.3 22.2+17.0 9.7+4.4

0.34 0.36 0.36 0.38 0.33 0.27 0.43 0.49 0.48 0.52 0.44 0.41

1.01 2.30 3.91 5.93 4.09 3.59 1.64 3.45 6.05 9.37 5.85 5.53

0.86 2.02 3.49 5.22 3.63 3.13 1.48 3.13 5.48 8.50 5.41 4.93

10.4 22.9 42.3 62.9 50.4 46.7 16.3 32.7 63.2 95.8 69.5 71.0

1.50 3.10+1.6 5.14+3.6 8.07+6.6 5.57+4.1 5.24+3.7 2.03 4.14+2.1 7.48+5.5 11.47+9.4 6.91+4.9 7.63+5.6

models. This reflects on a constant theme of agent harness design and model capability: many of the encouraged behaviors are gradually incorporated into model training, and hence earlier useful harness designs may become outdated. It is important to periodically revisit harness designs throughout model evolutions. For harnesses, we observe copilot-cli to outperform claude-code, justifying our harness choices (Figure 2b). Noticeably, we observed claude-opus-5 with claude-code comes with a sharp increase in cost. Inspecting the trajectories, we found this configuration much more frequently uses sub-agents (9.0 per run), compared to copilot-cli (0 per run) or claude-code with opus-4.8 (0.1 per run). This matches Anthropic’s official documentation that “Claude Opus 5 delegates to sub-agents more readily than prior models” [2]. Developers need to watch out for such unexpected cost surge with model changes. ○ Key Findings (RQ4): Agents benefit from additional test-time compute through effort-level, prompting or multi-agents. Comprehensive prompting is more cost-effective and performant compared to multi-agents for code review. 5 5.1

Discussion When can a software factory be lights-off?

While coding agents have gained substantial review capabilities, our results suggest that agents are not yet ready for a fully autonomous software factory, especially for large-scale proprietary industry repositories. Even the strongest agent misses known bugs, with heavy performance drops on proprietary repositories, and failures concentrate in diffs that require non-local or domainspecific reasoning. The present evidence therefore supports “lights-off” operation only for narrowly scoped, low-consequence PRs. For consequential changes, review agents are better treated as scalable peers that broaden coverage while an accountable human retains the final decision. While the boundary can move as models improve, developers should calibrate based on repository-specific, continuously monitored evidence rather than aggregate benchmark performance alone. 5.2

Recommendations for best practices when adopting review agents

While review agents are not yet capable of fully replacing human reviewers, they are still quite valuable to be incorporated into the review pipeline to help developers catch many bugs they would otherwise miss (Figure 5b). The human-in-the-loop adoption, however, should be done with care, , Vol. 1, No. 1, Article . Publication date: October 2026.

18

Yang et al.

based on review contexts. Based on our empirical observations, we derive the following actionable recommendations for review agent deployment: Scale review efforts based on PR complexity. An agent’s bug-detection capability decreases as a PR becomes larger or contains more kinds of changes (cf. Section 4.4). Teams should use these properties to allocate both automated and human review effort. Small, homogeneous changes may be reviewed with a cheaper configuration, whereas large, heterogeneous PRs warrant stronger models, higher reasoning effort, comprehensive prompts, and careful human review. Splitting unrelated changes into smaller PRs can also reduce the amount of code competing for reviewer attention. Human reviewers should retain responsibility for project values and long-term maintainability, where our results suggest that their attention is particularly complementary to that of agents. Steer review directions based on priorities. By default, agent reviews focus on more severe, functional bugs. Their review efforts can also be (mis-)directed by surface cues like the types of PR, statements in code comments, or specified behaviors in test cases. Developers should provide custom instructions that reflect their priorities – for example, explicitly asking an agent to examine usability, portability, backward compatibility, or maintainability when those properties matter. Incorporate repository knowledge for better review quality. Agents excel at finding bugs that can be clearly inferred based on local diff contexts. They struggle more when a correct judgment depends on project-specific knowledge, surrounding source files, or runtime evidence, which is often the case for large, proprietary repositories. To mitigate these deficiencies, developers can incorporate repository-specific instructions and knowledge bases to guide and improve review quality. Monitor review quality and adapt harness over model evolution. Review capability changes with both the underlying model and the surrounding environment (e.g., as the codebase evolves). Harnesses that help one model may provide little benefit to a later model. Teams should therefore evaluate the deployed agent configuration periodically. They can apply our data-construction method to construct custom benchmarks for their own repositories, and adapt the harness as needed when model behaviors change. 5.3

Research opportunities

Self-improving review agents. One emerging theme from our analysis is that current review agents are still weak at detecting bugs that require project-level knowledge. While developers can provide explicit scaffold and contextual information, future review agents should also be able to maintain memory that enables continual adaptation to (evolving) software repositories. Such memory can be built from artifacts produced in engineering processes, e.g., design documents, agentic sessions, past review discussions, prior incidents and commits [29, 47]. To build effective review agents, researchers need to create efficient and effective memory representations, such that new information is incorporated and stale information is removed [24]. They will also need to create effective retrieval mechanisms, and train agents to use the retrieval tools to seek relevant information. To enable self-improvement, researchers can build on our work, and create reliable data pipelines to mine task instances to guide memory evolution. Improve human-in-the-loop review efficiency. Our results suggest that human-in-the-loop reviews are still important for bug detection. However, human reviewers struggle to keep up with agentic code generation [1]. To keep humans in the review loop, we need to design effective human-agent collaboration. Instead of having humans manually review every single line of code, we should build systems to assist and speed up human inspection, and maximize the value they provide at minimal cognitive cost. For example, researchers can build code understanding agents, that provide PR walkthrough, potentially tailored to reviewers’ expertise and past interactions. The agents should be able to highlight the parts that are high-risk, high-impact, or they are least certain about. , Vol. 1, No. 1, Article . Publication date: October 2026.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

19

Training review agents that are able to calibrate their uncertainty well and communicate it to humans would be a promising direction. 6

Conclusion

In this work, we introduce CRBugBench and empirically study when and how code review agents detect human-verified bugs. We show that agents are improving rapidly, but still struggle with complex changes and bugs requiring project context, particularly in proprietary repositories. Comprehensive prompting improves detection more cost-effectively than multi-agent setups, though its benefits vary across models. These findings support retaining human oversight and provide guidance for adapting review effort, context, and agent configurations to project needs. Data Availability We are unable to make the replication package publicly available. We provide an open-source version of the benchmark at ‡ https://github.com/malusamayo/CRBugBench. Acknowledgments We want to thank Kaia Newman, Jina Chun, Jaeyeon Kim, Mike Swafford, Vivien Hung, Selma Ikiz, Karl Piteira, Ben Zorn, Chris Trevino, Nathan Evans, Jonathan Larson, Peter Choi, Rudrajit Choudhuri, and Dax Pryce for their feedback on this work. We would also like to thank Jonathan Larson and the Agentic Data Memory team for generously providing the computational resources and infrastructure that made this work possible. References [1] Shyam Agarwal, Courtney Miller, Christian Kästner, and Bogdan Vasilescu. 2026. 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse. arXiv preprint arXiv:2607.07980 (2026). [2] Anthropic. [n. d.]. Prompting Claude Opus 5. Claude Platform Docs. https://platform.claude.com/docs/en/build-withclaude/prompt-engineering/prompting-claude-opus-5 [3] Anthropic. 2025. Claude Code. https://claude.com/product/claude-code [4] Anthropic. 2026. Code Review. https://code.claude.com/docs/en/code-review [5] Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and challenges of modern code review. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 712–721. doi:10.1109/ICSE.2013.6606617 [6] Vipin Balachandran. 2013. Reducing Human Effort and Improving Quality in Peer Code Reviews Using Automatic Static Analysis and Reviewer Recommendation. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 931–940. doi:10.1109/ICSE.2013.6606642 [7] Mike Barnett, Christian Bird, João Brunet, and Shuvendu K. Lahiri. 2015. Helping Developers Help Themselves: Automatic Decomposition of Code Review Changesets. In Proc. 37th International Conference on Software Engineering (ICSE). 134–144. doi:10.1109/ICSE.2015.35 [8] Tobias Baum, Kurt Schneider, and Alberto Bacchelli. 2019. Associating Working Memory Capacity and Code Change Ordering with Code Review Performance. Empirical Software Engineering 24, 4 (2019), 1762–1798. doi:10.1007/s10664018-9676-8 [9] Moritz Beller, Alberto Bacchelli, Andy Zaidman, and Elmar Juergens. 2014. Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?. In Proceedings of the 11th Working Conference on Mining Software Repositories. Association for Computing Machinery, 202–211. doi:10.1145/2597073.2597082 [10] Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Methodological) 57, 1 (1995), 289–300. doi:10.1111/j.25176161.1995.tb02031.x [11] Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. 2022. Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools. Queue 20, 6 (2022), 35–57. [12] Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu. 2011. Don’t Touch My Code! Examining the Effects of Ownership on Software Quality. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. Association for Computing Machinery, 4–14. doi:10.1145/2025113.2025119 , Vol. 1, No. 1, Article . Publication date: October 2026.

20

Yang et al.

[13] Amiangshu Bosu, Michaela Greiler, and Christian Bird. 2015. Characteristics of Useful Code Reviews: An Empirical Study at Microsoft. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 146–156. doi:10.1109/MSR.2015.21 [14] Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, and Eray Tüzün. 2025. Automated Code Review in Practice. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 425–436. doi:10.1109/ICSE-SEIP66354.2025. 00043 [15] Marco D’Ambros, Michele Lanza, and Romain Robbes. 2010. An Extensive Comparison of Bug Prediction Approaches. In 2010 7th IEEE Working Conference on Mining Software Repositories. IEEE, 31–41. doi:10.1109/MSR.2010.5463279 [16] Alastair Dunsmore, Marc Roper, and Murray Wood. 2000. Object-Oriented Inspection in the Face of Delocalisation. In Proc. 22nd International Conference on Software Engineering (ICSE). 467–476. doi:10.1145/337180.337343 [17] Michael E. Fagan. 1976. Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal 15, 3 (1976), 182–211. doi:10.1147/sj.153.0182 [18] Neil Fendley, Zhengyu Liu, Aonan Guan, Jiacheng Zhong, and Yinzhi Cao. 2026. Comment and Control: Hijacking Agentic Workflows via Context-Grounded Evolution. arXiv preprint arXiv:2605.11229 (2026). [19] Ruida Hu, Xinchen Wang, Xin-Cheng Wen, Zhao Zhang, Bo Jiang, Pengfei Gao, Chao Peng, and Cuiyun Gao. 2026. Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice. In Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering, FSE Companion 2026. Association for Computing Machinery, 620–631. doi:10.1145/3803437.3805235 [20] ISO/IEC. 2011. Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models. International Standard ISO/IEC 25010:2011. International Organization for Standardization and International Electrotechnical Commission. https://www.iso.org/standard/35733.html [21] Hao Li, Haoxiang Zhang, and Ahmed E Hassan. 2025. The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering. arXiv preprint arXiv:2507.15003 (2025). [22] Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. 2022. Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for Computing Machinery, 1035–1047. doi:10.1145/3540250.3549081 [23] Jenny T Liang, Chenyang Yang, and Brad A Myers. 2024. A large-scale survey on the usability of ai programming assistants: Successes and challenges. In Proceedings of the 46th IEEE/ACM international conference on software engineering. 1–13. [24] Tobias Lindenbauer, Georg Groh, and Hinrich Schütze. 2025. From knowledge to noise: CTIM-rover and the pitfalls of episodic memory in software engineering agents. In Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025). 411–427. [25] Junyi Lu, Lili Jiang, Xiaojia Li, Jianbing Fang, Fengjun Zhang, Li Yang, and Chun Zuo. 2025. Towards Practical Defect-Focused Automated Code Review. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, 40505–40536. https://proceedings.mlr.press/v267/lu25f.html [26] Junyi Lu, Xiaojia Li, Zihan Hua, Lei Yu, Shiqi Cheng, Li Yang, Fengjun Zhang, and Chun Zuo. 2025. DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation. In Fundamental Approaches to Software Engineering (Lecture Notes in Computer Science, Vol. 15693). Springer, 43–64. doi:10.1007/978-3-031-90900-9_3 [27] Mika V. Mäntylä and Casper Lassenius. 2009. What Types of Defects Are Really Discovered in Code Reviews? IEEE Transactions on Software Engineering 35, 3 (2009), 430–448. doi:10.1109/TSE.2008.71 [28] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. 2014. The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects. In Proceedings of the 11th Working Conference on Mining Software Repositories. Association for Computing Machinery, 192–201. doi:10.1145/2597073.2597076 [29] Fangwen Mu, Junjie Wang, Lin Shi, Song Wang, Shoubin Li, and Qing Wang. 2026. Experepair: Dual-memory enhanced llm-based repository-level program repair. Proceedings of the ACM on Software Engineering 3, FSE (2026), 3954–3976. [30] Atharva Naik, Marcus Alenius, Daniel Fried, and Carolyn Rose. 2025. CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Association for Computational Linguistics, 9049–9076. doi:10.18653/v1/2025.naacl-long.457 [31] Kristen Pereira, Neelabh Sinha, Rajat Ghosh, and Debojyoti Dutta. 2026. CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents. arXiv preprint arXiv:2603.11078 (2026). doi:10.48550/arXiv.2603.11078 [32] Achyudh Ram, Anand Ashok Sawant, Marco Castelluccio, and Alberto Bacchelli. 2018. What Makes a Code Change Easier to Review: An Empirical Investigation on Code Change Reviewability. In Proc. 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 201–212.

, Vol. 1, No. 1, Article . Publication date: October 2026.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection

21

[33] Peter C. Rigby and Christian Bird. 2013. Convergent Contemporary Software Peer Review Practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. Association for Computing Machinery, 202–212. doi:10.1145/2491411.2491444 [34] Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern Code Review: A Case Study at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. Association for Computing Machinery, 181–190. doi:10.1145/3183519.3183525 [35] Anita Sarma and Nina Chen. 2024. Effective teaching through code reviews: Patterns and anti-patterns. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1262–1283. [36] Dan Shapiro. 2026. The Five Levels: From Spicy Autocomplete to the Dark Factory. https://www.danshapiro.com/blog/ 2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/ [37] Jacek Śliwerski, Thomas Zimmermann, and Andreas Zeller. 2005. When do changes induce fixes? ACM sigsoft software engineering notes 30, 4 (2005), 1–5. [38] Davide Spadini, Maurício Aniche, Margaret-Anne Storey, Magiel Bruntink, and Alberto Bacchelli. 2018. When Testing Meets Code Review: Why and How Developers Review Tests. In Proc. 40th International Conference on Software Engineering (ICSE). 677–687. doi:10.1145/3180155.3180192 [39] Stack Overflow. 2025. Technology | 2025 Stack Overflow Developer Survey. https://survey.stackoverflow.co/2025/ technology/. [40] Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawendé F. Bissyandé. 2024. CodeAgent: Autonomous Communicative Agents for Code Review. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11279– 11313. doi:10.18653/v1/2024.emnlp-main.632 [41] Chakkrit Kla Tantithamthavorn, Yaotian Zou, Andy Wong, Michael Gupta, Zhe Wang, Mike Buller, Ryan Jiang, Matthew Watson, Minwoo Jeong, Kun Chen, and Ming Wu. 2026. RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-Based Code Review Automation at Atlassian. In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice. Association for Computing Machinery, 34–45. doi:10.1145/3786583.3786851 [42] Yida Tao, Yingnong Dang, Tao Xie, Dongmei Zhang, and Sunghun Kim. 2012. How Do Software Engineers Understand Code Changes? An Exploratory Study in Industry. In Proc. 20th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE). 51:1–51:11. doi:10.1145/2393596.2393656 [43] Rosalia Tufano, Alberto Martin-Lopez, Ahmad Tayeb, Ozren Dabić, Sonia Haiduc, and Gabriele Bavota. 2025. Deep Learning-Based Code Reviews: A Paradigm Shift or a Double-Edged Sword?. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1640–1652. doi:10.1109/ICSE55347.2025.00060 [44] Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk, and Gabriele Bavota. 2022. Using Pre-Trained Models to Boost Code Review Automation. In Proceedings of the 44th International Conference on Software Engineering. Association for Computing Machinery, 2291–2302. doi:10.1145/3510003.3510621 [45] Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. 2021. Towards Automating Code Review Activities. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 163–174. doi:10.1109/ICSE43902.2021.00027 [46] Asif Kamal Turzo and Amiangshu Bosu. 2024. What Makes a Code Review Useful to OpenDev Developers? An Empirical Investigation. Empirical Software Engineering 29, 1, Article 6 (2024). doi:10.1007/s10664-023-10411-x [47] Boshi Wang, Weijian Xu, Yunsheng Li, Xuemei Gao, Yujia Xie, Huan Sun, and Dongdong Chen. 2026. Improving code localization with repository memory. In International Conference on Learning Representations, Vol. 2026. 111266–111285. [48] Zezhou Yang, Cuiyun Gao, Zhaoqiang Guo, Zhenhao Li, Kui Liu, Xin Xia, and Yuming Zhou. 2026. A Roadmap for Modern Code Review: Challenges and Opportunities. ACM Transactions on Software Engineering and Methodology (2026). Just Accepted. doi:10.1145/3800963 [49] Zhengran Zeng, Ruikai Shi, Keke Han, Yixin Li, Kaicheng Sun, Yidong Wang, Zhuohao Yu, Rui Xie, Wei Ye, and Shikun Zhang. 2026. SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation. Proceedings of the ACM on Software Engineering 3, FSE, Article FSE137 (2026), 23 pages. doi:10.1145/3808144 [50] Lei Zhang, Yongda Yu, Minghui Yu, Xinxin Guo, Zhengqi Zhuang, Guoping Rong, Dong Shao, Haifeng Shen, Hongyu Kuang, Zhengfeng Li, et al. 2026. AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context. arXiv preprint arXiv:2601.19494 (2026). [51] Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen, and Abhik Roychoudhury. 2026. Code review agent benchmark. Preprint. arXiv:2603.23448 [cs.SE] doi:10.48550/arXiv.2603.23448 [52] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36 (2023), 46595–46623.

, Vol. 1, No. 1, Article . Publication date: October 2026.

22

Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

, Vol. 1, No. 1, Article . Publication date: October 2026.

Yang et al.

Related documents

• (When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection#1298164
• Code Broker: A Multi-Agent System for Automated Code Quality Assessment#139027
• OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review#441868
• Scanning the Harness: Configuration Exposures in AI Coding-Agent Supply Chains#1242536
• Is Agent Code Less Maintainable Than Human Code?#1131585