arXiv:2606.18168v1 [cs.SE] 16 Jun 2026
All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code Dipayan Banik
Kowshik Chowdhury
Shazibul Islam Shamim
Danovo Energy Solutions Raleigh, North Carolina, USA [email protected]
Kennesaw State University Kennesaw, Georgia, USA [email protected]
Kennesaw State University Kennesaw, Georgia, USA [email protected]
Abstract—Software practitioners increasingly use AI coding agents that generate test code alongside production code in open-source pull requests (PRs). Recent studies report more than 932,000 agent-authored PRs across more than 116,000 repositories, yet whether their test files contain meaningful verification logic remains underexplored. Test files lacking explicit assertions execute code without verifying behavior, so quality gates based on test-file presence overestimate verification strength. The goal of this paper is to help practitioners assess the verification strength of agent-authored patches by characterizing oracle signals and their link to merge outcomes and review effort. We conduct an empirical study of 86,156 test-file patches from 33,596 agentauthored PRs across 2,807 GitHub repositories produced by five coding agents: OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code. A qualitative analysis of 384 stratified patches informs a syntactic taxonomy of eight oracle signal categories. Applied at scale, 80.2% of test patches contain weak or no explicit oracle signals. While raw merge rates are lower for strong-oracle PRs, a regression analysis adjusting for agent, PR size, repository popularity, task type, and language shows strong oracles significantly improve merge likelihood (OR = 1.28, p < 0.001). Our findings suggest that test file counts substantially overestimate verification strength and that practitioners can adopt oracle-aware quality checks to more accurately evaluate agent-authored contributions. Index Terms—test oracle, test assertions, AI coding agents, empirical software engineering, mining software repositories
I. I NTRODUCTION The landscape of software development is shifting from AIassisted code completion to autonomous code authorship. Interactive assistants such as early GitHub Copilot offered line-level suggestions that developers could accept or ignore, preserving human control over commits [1]. A new generation of autonomous agents, including OpenAI Codex, Devin, Cursor, and Claude Code operates at the repository level: given an issue or prompt, they plan implementations, write production code, generate tests, open pull requests, and in some cases iterate on reviewer feedback without human intervention [2], [3]. Industry projections estimate that 75% of enterprise software engineers will use AI code assistants by 2028 [4]. This transition represents not merely a productivity gain but a fundamental change in who authors the verification logic that guards against regression. As agents take on this authorship, the quality of the test code they produce becomes a critical concern. In a typical pull request, a developer writes a test that calls a function and
then asserts that the returned value matches an expected result. This assertion is the test oracle, which determines whether the test passes or fails [5]. Without it, a test executes code but never checks whether the output is correct. A pull request that includes test files appears well-verified to a reviewer scanning a file list: CI pipelines report coverage, dashboards turn green, and the contribution looks complete. However, practitioners are increasingly reporting a gap between this appearance and the underlying verification strength. Developers describe AIgenerated tests that “confirm your code does what it already does”— a phenomenon called “test theater” [6]. Common patterns include assertion-free functions that execute code paths but never verify the output [7], and test suites that mock the module under test itself, passing by construction rather than by correctness [8]. Research on LLM-generated test oracles reinforces this concern. Controlled experiments show that LLM-generated assertions frequently encode actual program behavior rather than expected behavior, turning bugs into passing tests [9]. Fine-tuned assertion generators detect only a fraction of known faults even at 59.5% exact-match precision [10], and mutationguided feedback improves oracle quality but requires curated prompts unavailable in autonomous agent workflows [11]. These findings expose a gap unaddressed by existing metrics: code coverage correlates only weakly with fault-detection effectiveness [12], with mature test suites showing gaps of up to 51% between executed and checked code [13]. A systematic categorization of what agents actually produce in test patches ranging from no oracle signal to strong behavioral assertions would give both CI tooling and human reviewers a vocabulary to distinguish verified contributions from structural scaffolding. Understanding how these categories relate to merge outcomes and review effort would further clarify whether oracle strength carries practical consequences in realworld development workflows, or whether reviewers treat all test-bearing PRs alike regardless of assertion quality. Existing research has examined oracle quality in controlled settings and test file presence in real-world repositories, but not the oracle content of agent-authored test patches at scale. We address this gap through a large-scale empirical study of 86,156 cumulative test-file patches from 33,596 agentauthored pull requests across 2,807 GitHub repositories from the AIDev dataset [14]. We investigate two research questions:
RQ1: What oracle signal patterns characterize agentauthored test-file patches? • RQ2: How does oracle signal strength relate to PR merge outcomes and human review effort? This paper makes three primary contributions: (1) We propose a syntactic taxonomy of eight oracle signal categories for classifying test-file patches. (2) We characterize oracle signal distributions across five AI coding agents. (3) We analyze the relationship between oracle signal strength and PR merge outcomes through a multivariate logistic regression.
TABLE I O RACLE SIGNAL TAXONOMY CATEGORIES .
•
II. M ETHODOLOGY We use the AIDev-pop subset of the AIDev dataset [14], which contains 33,596 agent-authored pull requests across 2,807 repositories with at least 100 GitHub stars. We extract filelevel patches from the pr_commit_details table, which stores the diff for each file modified in each commit within a pull request. Our analysis scripts, classified dataset, and figures are available at [15]. A. RQ1: Oracle Signal Classification Test File Identification: We identify test files using two criteria applied jointly. First, the file path must match test-related directory or filename patterns, including test/, tests/, __tests__/, testing/, or filenames containing _test., .test., or .spec. Second, the file extension must belong to a set of 30 source code extensions spanning Python, TypeScript, JavaScript, Go, Java, C#, Rust, Ruby, and others. We exclude non-code artifacts (fixtures, snapshots, documentation) that appear in test directories but contain no executable test logic. This filtering yields 103,976 test code patches from the total of 711,923 file patches. Cumulative Patch Construction: A single test file may appear in multiple commits within the same pull request. To avoid double-counting and capture the full oracle signal introduced by the PR, we concatenate all patches for each unique (PR, filename) pair in commit order. This produces 86,156 cumulative patches for our taxonomy analysis. Oracle Signal Taxonomy: For each cumulative patch, we extract added lines and search for assertion patterns from widely used testing frameworks (e.g., JUnit, pytest, Jest, Go testing, RSpec). Through iterative open coding, we find that oracle signals fall along a spectrum – from no assertion to checks that code merely executes, to checks of specific output values or behavior – which we group into eight categories in Table I. Weak categories (W1–W5) confirm execution without verifying output correctness, mirroring the documented gap between executed and output checked code [12], [13]; strong categories (S1–S3) compare output to expected values, verify error handling or types, or combine multiple assertion types. To assess reliability, two authors independently labeled a stratified random sample of 384 patches (95% confidence, 5% margin); we resolved disagreements by discussion and reported agreement with Cohen’s Kappa [16]. B. RQ2: Oracle Signals and PR Outcomes PR-Level Quality Groups: We aggregate patch-level classifications to the PR level by assigning each PR the highest
Code W1 W2 W3 W4 W5 S1 S2 S3
Tier Weak Weak Weak Weak Weak Strong Strong Strong
Definition No assertion pattern present Existence/non-null checks only Boolean asserts only (no value compared) Mock/call-verification only Snapshot match only Value equality or comparison Error, containment, or type checks Two or more distinct strong types
oracle-signal category among its test-file patches. We define three quality groups: all weak (best patch is W1–W5), has strong (best patch is S1 or S2), and has multi strong (best patch is S3). Outcome Variables: We collect merge outcome, review effort (formal reviews + inline + discussion comments), PR size (files changed, lines added), repository popularity (star count), and time to close for each PR. Controlled Comparisons: We compare the three quality groups across all outcome variables. To control for confounding factors, we repeat the comparison within each agent, within each task type, and within PR size buckets (small 1–5 files, medium 6– 20, large >20). We apply size-bucket controls to both merge rate and review effort to distinguish whether oracle signal strength has an independent association with PR outcomes. We additionally fit a multivariate logistic regression including agent, log-transformed PR size, log-transformed repository stars, task type, and primary language as covariates. III. A NALYSIS AND F INDINGS A. RQ1: Oracle Signals in Test-File Patches Figure 1 presents the oracle signal distribution across all 86,156 cumulative test-file patches and for newly created files separately. As shown in the summary header, 80.2% of patches contain weak or no explicit oracle signals (W1–W5), with value assertions (S1) and multi-signal strong oracles (S3) accounting for 11.3% and 5.7%, respectively. Two authors independently labeled 384 stratified patches (95% confidence, 5% margin), reaching Cohen’s κ = 0.77. On the same sample, the classifier matches the human labels for the oracle-signal category in 86.7% of patches. Two patterns emerge from the per-agent breakdown. First, agents differ in how often they introduce oracle signals (χ2 = 2497.3, p < 0.001), with Claude Code and Devin producing stronger oracle profiles than Copilot, Cursor, and OpenAI Codex. Second, newly created files show higher strong-oracle rates than modified files (χ2 = 810.2, p < 0.001), ranging from 18% for OpenAI Codex to 67% for Claude Code. The newly created-file comparison carries the strongest claim because the full file content is the patch and no prior oracle content exists. Key Finding. 80.2% of agent-authored test-file patches contain weak or no explicit oracle signals. On newly added test files, strong-oracle rates range from 18% to 67% across agents.
Overall: 80.2% weak (W1 W5) | S1: 11.3% | S3: 5.7% | N = 86,156 | Newly added: 18% 67% strong across agents training
(a) All Test-File Patches
Claude Code
55
0.1
0.3
3
-
18
4
19
41% strong (n=1,923)
Copilot
73
0.3
0.2
0.5
0.2
17
2
6
26% strong (n=18,209)
Cursor
59
0.2
1
2
2
17
3
16
36% strong (n=2,072)
Devin
71
0.2
0.3
2
0.4
12
3
10
26% strong (n=13,147)
OpenAI Codex
85
0.0
0.3
0.3
0.0
8
3
4
15% strong (n=50,805)
Overall
79
0.1
0.3
0.7
0.2
11
3
6
20% strong (n=86,156)
W1
W2
W3
W4
W5
S1
S2
S3
(b) Newly Added Files Claude Code
24
-
0.2
9
-
30
6
31
67% strong (n=461)
Copilot
55
0.4
0.4
0.3
0.0
29
3
11
44% strong (n=3,475)
Cursor
53
-
1
1
4
17
2
21
40% strong (n=603)
Devin
41
0.2
0.7
4
1
22
6
25
52% strong (n=2,108)
OpenAI Codex
81
0.0
0.5
0.3
-
11
2
5
18% strong (n=15,648)
Overall
71
0.1
0.5
0.8
0.3
16
2
9
27% strong (n=22,295)
W1
W2
W3
W4
W5
S1
S2
S3
Fig. 1. Oracle signal distributions and breakdowns for RQ1 analyses.
B. RQ2: Oracle Strength, Merge Outcomes, and Review Effort S3 PRs merge at 59.7% compared to 72.6% for weak-oracle PRs, but S3 PRs also contain 4.2× more code additions (Mann-Whitney test, p < 0.001, r = 0.38), attract 2.4× more review effort (Mann-Whitney test, p < 0.001, r = 0.36), and reside in repositories with 3.8× more stars. This review effort gap persists within PR size buckets, with ratios ranging from 1.7× for small PRs to 2.5× for large PRs. Figure 2 shows that the aggregate pattern weakens within individual agents. Within Claude Code, S3 PRs merge at 62% compared to 52% for weak-oracle PRs; Copilot shows the same direction at 49% versus 44%. For test-type PRs, S3 PRs merge at 81% compared to 77%. A multivariate logistic regression adjusting for agent, PR size, repository popularity, task type, and primary language confirms that S3 oracles associate with significantly higher merge rates after adjustment (OR = 1.28, p < 0.001). Key Finding. Stronger oracle signals are associated with more complex, heavily reviewed pull requests. After adjusting for agent, PR size, repository popularity, task type, and language, S3 oracles significantly increase the likelihood of merge. IV. D ISCUSSION Our findings highlight a meaningful gap between test-file presence and oracle quality in agent-authored pull requests. This pattern may reflect an optimization mismatch: coding agents are trained on code completion objectives that reward structural plausibility, that is, creating test files, importing frameworks, and defining function signatures rather than behavioral reasoning about expected program output. Generating a correct oracle requires an understanding of intended semantics, a capability that current agent architectures do not explicitly optimize for. The variation across agents (18% to 67% strong-oracle rate on new test files) further suggests that
data, prompt design, and tool integration influence testing discipline. While our data cannot isolate the prompts behind each PR, oracle strength also varies with task type – a proxy for what the agent was asked to do: feature work yields strong oracles in only 18.2% of patches, versus 25.6% for bug fixes and 24.9% for test-focused tasks. This suggests practitioners obtain more meaningful oracles when verification is scoped as an explicit objective rather than bundled into feature PRs, and that adopting oracle-aware CI checks that flag newly added test files lacking assertion patterns can surface this gap early and separate verified contributions from structural scaffolding. V. T HREATS TO VALIDITY Our regex-based taxonomy classifies syntactic oracle signals in patch diffs, not complete source files; a modified-file patch classified as W1 may belong to a file with existing assertions, which we address by reporting newly created files separately. A patch classified as S1 confirms that an equality check exists, not that it checks the right property. The association between oracle strength and merge outcomes may be confounded by unobserved factors such as developer experience, merge-bot interactions, or CI requirements; we mitigate this through multivariate regression adjusting for agent, PR size, stars, task type, and language. Our dataset covers five agents active between December 2024 and July 2025, with OpenAI Codex contributing 65% of PRs, and filters for repositories with 100+ stars; results may not generalize to smaller repositories or ecosystems with different testing conventions. Additionally, our taxonomy does not capture implicit oracles such as crash-on-failure or timeout-based detection, which some testing frameworks use as default verification mechanisms. Our sensitivity analysis shows that reclassifying W4 (mock/interaction) and W5 (snapshot) as strong shifts the overall weak rate by less than one percentage point, indicating that boundary decisions between categories do not materially affect the findings. VI. R ELATED W ORK
Our work intersects three areas: the test oracle problem, LLM-based oracle generation, and empirical studies of agentauthored contributions. The oracle problem has a long research history [5]. Coverage correlates only weakly with faultdetection effectiveness [12], and mature test suites exhibit gaps of up to 51 percentage points between executed and checked code [13]. Jain et al. [17] formalized this as the “oracle gap”, showing that high coverage alone does not capture test adequacy. Our study examines whether this distinction between executing code and verifying its behavior holds for agentauthored test code at scale. In the LLM setting, generated oracles frequently encode actual rather than expected behavior [9], fine-tuned generators detect only a fraction of known faults [10], and mutation-guided feedback improves quality but requires curated prompts [11]. These studies evaluate oracle quality in controlled settings; our work shifts the analysis to real-world agent-authored patches. For real-world agent
(a) Merge Rate by Agent 82 66
62 52 53 44 42
40
79 81
53 45
41
0
80
600
200 Claude Code
Copilot
Cursor
Devin
OpenAI Codex
0 All weak (W1-W5)
77
76
512 306 307 123
184 3
Median Additions
4
7
Mean Review Effort
Has strong (S1-S2)
Median Stars
60
80 81 71
68
400
20
(c) Merge Rate by Task Type
100
1177
800
60 62
49
(b) PR Complexity by Quality Group
1000
Value
Merge Rate (%)
80 60
1200
Merge Rate (%)
100
58
64
63 53
61
50
40 20 0
feat
fix
test
refactor
Has multi-strong (S3)
Fig. 2. Merge outcomes and confounding-factor analyses for RQ2.
behavior, Haque et al. measured test file presence on the AIDev dataset [14], Milanese et al. compared test quality between agent and human PRs using test-smell detection, and Chowdhury et al. analyzed how code-review agents and feedback signal quality relate to PR outcomes [18], [19], [20]. VII. C ONCLUSION This study provides large-scale empirical evidence that testfile presence is an insufficient proxy for verification strength in agent-authored pull requests. By classifying 86,156 cumulative test-file patches across 2,807 repositories, we show that 80.2% contain weak or no explicit oracle signals, with the strong-oracle rate varying from 18% to 67% across agents on newly created files. These findings suggest that coding agents generate test structure far more reliably than they generate oracle logic. Strong oracle signals coincide with larger, more heavily reviewed PRs; once this complexity is controlled, they predict higher merge likelihood, making oracle strength a positive signal masked by confounding rather than a merge disadvantage. R EFERENCES [1] H. Li, H. Zhang, and A. E. Hassan, “The rise of AI teammates in software engineering (SE) 3.0: How autonomous coding agents are reshaping software engineering,” arXiv preprint arXiv:2507.15003, 2025. [2] C. Reddington, “From idea to PR: A guide to GitHub Copilot’s agentic workflows,” https://github.blog/ai-and-ml/github-copilot/from-idea-t o-pr-a-guide-to-github-copilots-agentic-workflows/, 2025, accessed: 2026-03-30. [3] M. Watanabe, H. Li, Y. Kashiwa, B. Reid, H. Iida, and A. E. Hassan, “On the use of agentic coding: An empirical study of pull requests on GitHub,” arXiv preprint arXiv:2509.14745, 2025. [4] Gartner, “Gartner says 75 percent of enterprise software engineers will use AI code assistants by 2028,” https://www.gartner.com/en/newsroom/ press-releases/2024-04-11-gartner-says-75-percent-of-enterprise-softw are-engineers-will-use-ai-code-assistants-by-2028, 2024, press release, April 11, 2024. [5] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015. [6] B. Houston, “The rise of test theater,” https://ben3d.ca/blog/the-rise-o f-test-theater, 2025, accessed: 2026-03-29.
[7] H. Flores, “I let an AI agent write 275 tests. here’s what it was actually optimizing for,” https://dev.to/htekdev/i-let-an-ai-agent-write-275-tes ts-heres-what-it-was-actually-optimizing-for-32n7, 2026, accessed: 2026-03-29. [8] M. Bar-Zeev, “Why testing after with AI is even worse,” https://dev.to/m barzeev/why-testing-after-with-ai-is-even-worse-4jc1, 2026, accessed: 2026-03-29. [9] M. Konstantinou, R. Degiovanni, and M. Papadakis, “Do LLMs generate test oracles that capture the actual or the expected program behaviour?” arXiv preprint arXiv:2410.21136, 2024. [10] S. Primbs, B. Fein, and G. Fraser, “AsserT5: Test assertion generation using a fine-tuned code language model,” in Proceedings of the IEEE/ACM International Conference on Automation of Software Test (AST 2025), 2025, arXiv:2502.02708. [11] A. Moradi Dakhel, A. Nikanjam, V. Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing,” Information and Software Technology, vol. 171, 2024. [12] L. Inozemtseva and R. Holmes, “Coverage is not strongly correlated with test suite effectiveness,” in Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), 2014, pp. 435–445. [13] S. B. Hossain, M. B. Dwyer, S. Elbaum, and A. Nguyen-Tuong, “Measuring and mitigating gaps in structural testing,” in Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE 2023), 2023. [14] H. Li, H. Zhang, and A. E. Hassan, “AIDev: Studying AI coding agents on GitHub,” in Proceedings of the 23rd International Conference on Mining Software Repositories (MSR 2026), 2026, arXiv:2602.09185. [15] D. Banik, K. Chowdhury, and S. I. Shamim, “Replication package for “all smoke, no alarm: Oracle signals in agent-authored test code”,” https: //doi.org/10.6084/m9.figshare.32032107, 2026. [16] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, pp. 159–174, 1977. [17] K. Jain, G. T. Kalburgi, C. Le Goues, and A. Groce, “Mind the gap: The difference between coverage and mutation score can guide testing efforts,” in Proceedings of the 34th IEEE International Symposium on Software Reliability Engineering (ISSRE 2023), 2023, pp. 102–113. [18] S. Haque, S. Ingale, and C. Csallner, “An empirical study of tests in agentic pull requests,” in Proceedings of the 23rd International Conference on Mining Software Repositories (MSR 2026), 2026, arXiv:2601.03556. [19] R. Milanese, F. Salzano, A. Spina, A. Vitale, R. Pareschi, F. Fasano, and M. Fazzini, “Human-agent versus human pull requests: A testingfocused characterization and comparison,” in Proceedings of the 23rd International Conference on Mining Software Repositories (MSR 2026), 2026, arXiv:2601.21194. [20] K. Chowdhury, D. Banik, K. M. Ferdous, and S. I. Shamim, “From industry claims to empirical reality: An empirical study of code review agents in pull requests,” in Proceedings of the 23rd International Conference on Mining Software Repositories (MSR 2026), 2026, arXiv:2604.03196.