Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests in Open-Source Projects Preet Jhanglani∗ , Zeel Kaushal Desai∗ , Vidhi Kansara∗ , Eman Abdullah AlOmar
arXiv:2607.12068v1 [cs.SE] 13 Jul 2026
Stevens Institute of Technology, Hoboken, New Jersey, USA Email: {pjhangl1,zdesai,vkansara,ealomar}@stevens.edu
result in breaking the code. The developer should keep these test cases current as the codebase continues to expand. In organizations that release frequently, there may be insufficient time to write adequate test cases and ensure that tests are executed correctly. Among other reasons, one of the main reasons why developers are beginning to leverage AI-powered coding tools is to automate sections of the development and testing life cycle, which also allows them to have more available time to write tests. Developers can use AI-powered coding tools, such as GitHub Copilot and more autonomous agents like Devin, to read a project description, generate implementation code, and also auto-generate test cases to go along with that code. Essentially, they act as junior members of a team that develops a feature as well as creates the initial test coverage, thus reducing the amount of effort developers would typically spend doing work manually [18], [19]. There is a growing interest in researching the application of AI to generate test cases [16], [7], [4], [12], [6], [11], [13], [14], [10]; for example, one well-known project is TestPilot, which uses an algorithmatically generated JavaScript unit test from a function’s source code, signature, and usage examples by feeding the code into a large language model (LLM) to create the test case automatically. After being tested with 25 NPM packages, the median line coverage for the generated unit tests was found to be 70.2%, and the median branch coverage was 52.8%. This provides a meaningful increase over previous automated testing solutions for generating unit test I. I NTRODUCTION At the present time, technology is growing so quickly that cases. Subsequent studies such as CoverUp have provided almost every application receives continuous feature updates. further refinements including the ability to identify which Companies are now releasing regular updates at a much faster segments of code have not been unit tested yet and have the pace than before, and many times a day. Once a feature or algorithm concentrate on generating test cases for those areas product has been developed, that feature or product will go [2]. The majority of test cases generated by this approach have through a period of testing to ensure that it passes all test been evaluated using benchmark suites such as SWE-Bench cases prior to actual release. A majority of companies take [8] to determine whether the generated patches pass developeradvantage of continuous integration and continuous deployment created unit test suites. However, there are studies indicating (CI/CD) pipelines, which allow for the automation of software that many generated patches that were indicated as passing development and testing. CI/CD pipelines can be integrated on their behalf were still lacking in terms of the original with sites like GitHub, where each pull request is automatically intent and/or were not in line with the expected behavior. generated and tested before being incorporated into the main This indicates that while all tests were indeed successful, the resulting patch could still have issues and/or could not code structure. Testing is a key part of the development process, even for sufficiently resolve the issue originally intended. The quality of test cases can be affected by several other small updates. The developer needs to determine how the code factors, such as assertion strength. Assertion strength relates to should behave and identify the most likely situations that could how stringent or lenient a test is in its verification of the result. ∗ These authors contributed equally to this work. Here is one example of a good assertion strength – comparing
Abstract—The integration of AI-powered coding agents into Continuous Integration/Continuous Delivery (CI/CD) pipelines has fundamentally altered how software verification is conducted. While these agents successfully automate the test generation, current evaluation benchmarks (e.g., SWE-bench) largely focus on pass-rates rather than the intrinsic quality of the generated tests. This raises the possibility of “stealth technical debt”, in which test suites pass execution but do not offer comprehensive coverage or semantic value. We address this methodological gap through a large-scale, empirical comparison of 204,673 test artifacts which comprises of 24,941 human-authored files and 179,732 agent-generated files; sourced from the AIDev dataset. Using the Abstract Syntax Tree (AST) parsing with Python’s naive ast module, we implemented a “white-box” static analysis framework to evaluate three quality dimensions: Assertion Strength (RQ1), Edge-Case Coverage (RQ2), and Flakiness Potential (RQ3). Our results present a nuanced inversion of traditional assumptions. AI agents performed better than humans in Edge-Case Coverage, with almost twice the variety of boundary checks (Variety Score: 0.62 vs 0.32) and a higher frequency of null-safety testing (13.40% vs. 8.3%), even though human developers had a slight advantage in Assertion Strength (88.1% strong assertions vs. 85.37% for agents). But this thoroughness comes at a price: due mostly to their reliance on file I/O and non-deterministic logic, agent-generated tests exhibited a higher risk of flakiness (Candidate Rate: 0.41 vs. 0.30). These findings suggest that while AI agents excel at rigorous boundary testing, they lack the “environmental awareness” needed to write stable, hermetic tests. Index Terms—testing, quality, agents, pull requests
2) RQ2: How well can AI-generated test cases manage boundary conditions and edge situations in comparison to tests written by humans? 3) RQ3: Do flaky tests appear more frequently in pull requests made by human developers or by AI tools? The following sections of this work are organized as follows. Section II describes the approach, as well as the procedures used to collect and analyze the data. Section III discusses the results for each study question. Potential validity risks are described in Section IV. The conclusion and future work are provided in Sections V and VI.
the exact result produced by the program as opposed to only verifying whether true was returned. Edge-case coverage also plays a significant role in the quality of the test case itself. Good tests should contain edge-case coverage for abnormal input values (e.g., nulls, empty lists, 0, max val, min val) since some of the most elusive defects are located in these types of input cases. Test case stability is the third area impacting test case quality; stability refers to how consistently a test produces the same outcome when executed repeatedly. Unpredictable outcomes (i.e., both passing and failing) caused by flaky tests destroy the confidence level of the test suite and increase the time required to release. During periods of fast-paced development, the quality of these three areas, regardless of whether the test case was manually written or created by a test generation tool, tend to decline [3], [5]. There is an important consideration when it comes to AIgenerated tests. Most comparisons of AI-generated testing focus on the percentage of lines of code that the tests cover, or whether the tests were successful or not. These are both reasonable baseline comparisons, but they add very little information about how reliable the tests actually are. A test suite can have excellent coverage, but may have missed some important edge cases, contain weak assertions, or contain unreliable tests that may fail unpredictably. If testing tools are only designed to evaluate based on coverage measurements, they will produce tests that seem reliable but provide very little protection once the software is in production [17]. By concentrating on assessing the quality of tests directly in relation to all three of the above measurements; human written versus AI generated, through the use of actual realworld software development activities contained in the AIDev database (the largest database currently available of actual software development activity). The tests are not actually executed; rather, abstract syntax trees (AST) will be parsed from each test so that structure can be evaluated statically and thus be able to scale and provide consistent results. There are 179,732 AI-generated tests (files) and 24,941 human-created tests (files) present in this database. Each of these files contained in the AIDev database went through three independent automated test evaluation pipelines in our evaluation process. The findings provide a mixed result in terms of test strength. For example, 88 of the human-written assertions were categorized as strong; however, the total number of ambiguous or unclassified assertions recorded by human testers was far less than those written by computers. Human-written tests were able to use only 1.46% ambiguous or unclassified assertions while AI-generated tests had 11.58%. On edge-case coverage, the situation is reversed. AI agents covered a broader variety of edge cases, scoring 0.62 on the variety measure against 0.32 for humans. Humans performed better than AI – Agents on test stability, this study showing humans generated tests achieved a flakiness rate of 0.30 compared to 0.41 for AI-generated tests. For this study, we worked on these three research questions: 1) RQ1: In comparison to tests created by human developers, how solid are the claims in test cases created by AI?
II. S TUDY D ESIGN This research conducts a large-scale comparative study of software tests written by human developers and tests generated by AI agents. The study uses the AIDev dataset as its main data source because it contains real-world software development activity [9]. However, the dataset includes only metadata and code patches, so we retrieve the full source code of the test files from their original repositories. Our method then collects, processes, and analyzes these files in a systematic way. We evaluate test quality in three aspects: assertion strength and specificity, edge-case coverage, and test flakiness. Figure 1 shows the full workflow, which is described in detail in the following steps. a) Phase 1: Metadata Manifest Generation: We begin with the AIDev dataset, using the all_pull_request.parquet file. From this file, we define two groups for comparison: pull requests created by AI agents (Agent-PRs) and a matched control group of pull requests created by human developers (Human-PRs).We note that Agent-PRs may include human review or edits before merging. This creates a possible human-in-the-loop limitation in the study. For each pull request, we extract the metadata needed for repository-level data collection, including the full repository name, such as owner/repository, the pull request number, and the final commit SHA. This phase produces a Retrieval Manifest, which lists the artifacts to be collected from their original repositories. b) Phase 2: Test Artifact Discovery and Retrieval: This phase focuses on finding and collecting the test files needed for analysis. For each repository listed in the Phase 1 manifest, we clone a local copy and identify files that follow common test file naming patterns, such as *_test.py and Test*.java. We then compare these files with the files modified in each target pull request, which we obtain through the GitHub API. This helps us identify the specific test files relevant to each pull request. Next, we use the commit SHA from the manifest to retrieve the full source code of each selected test file [9]. This ensures that our analysis is based on the complete version of each file in the correct commit. After retrieval, each test file is parsed into an Abstract Syntax Tree (AST) for further analysis. The file type distribution in the dataset is shown in Table I. Since Python files are the most common, this study focuses only on Python test files. Table VI reports the general dataset
2
Table I: File type distribution across Agent and Human datasets. Extension
Agent (%)
Human (%)
py ts pyc go js tsx cpp cc json png pb
35.73% 17.74% 9.05% 5.17% 4.90% 3.44% 2.59% 2.01% 1.77% – –
21.48% 21.48% – 13.53% 2.87% 5.72% – – 3.25% 7.25% 4.28%
Total Files
707,744
135,676
as Strong because it checks more specific program behavior. 7) Aggregation: The analyzer updates the weak_count and strong_count values based on the classification. 8) Output: The final output is the percentage of weak assertions and the percentage of strong assertions in the test file. This refined taxonomy is important because assertion strength depends on what the assertion checks, not only on the assertion name. For example, a basic taxonomy may classify assertTrue(complex_validation_function(x)) as weak only because it uses assertTrue. However, this assertion may still contain meaningful validation logic. White-box refinement reduces this kind of misclassifications and makes the analysis more accurate. • Pipeline B (Edge Case Coverage Analysis for RQ2) also traverses the ASTs to identify literal values (e.g., 0, -1, null, empty strings) used as inputs in method calls. These are matched against a heuristic list of common edge cases to produce metrics on test coverage. The ”Edge-Case Coverage” pipeline (Pipeline B) evaluates the test’s thoroughness by identifying inputs that represent common boundary conditions.
statistics. In that table, A-Sliced refers to the first 24,941 agent files, while A-Sliced-Rand refers to a random subset of 24,941 agent files. c) Phase 3: Multi-Dimensional Quality Analysis: In the third phase, we analyze the parsed ASTs using three separate pipelines. Each pipeline is connected to one of our main research questions. •
Pipeline A (Assertion Strength Analysis for RQ1) This analyzes the ASTs of the test files to find and classify the assertion statements. Assertions are classified using a predefined taxonomy, such as weak assertions and strong assertions. This pipeline measures how meaningful and specific the validation checks are within the test files.
White-Box Flowchart: 1) Input: The pipeline begins with the AST Module node. 2) Traversal: All function call nodes inside test methods are intercepted via TestAnalyzer.visit_Call(node). White-Box Flow chart: 3) Argument Inspection: For each function call, the system 1) Input: The source code of a Python test file is used as iterates through every argument in node.args. input. 4) Literal Matching: 2) Parsing: The ast.parse() function converts the • Check whether arg is an ast.Constant. source code into an Abstract Syntax Tree (AST). • If true, compare arg.value against the heuristic 3) Traversal: The TestAnalyzer.visit() method list: None, 0, -1, "". starts from the root Module node and visits the nodes 5) Collection Matching: in the AST. • Check whether the argument is an ast.List or 4) Identification: While visiting the AST, the analyzer looks ast.Dict. for ast.Call nodes, which represent function calls. It • If so, determine whether the corresponding element calls those inside test functions, where the function name list (e.g., arg.elts) is empty. starts with test_. 5) Classification: The analyzer extracts the assertion call, The final metric variety_count is computed as the such as self.assertEquals, and identifies the assersize of this set(i.e., len(categories)), representing tion name, such as assertEquals. Then it compares the number of distinct edge-case types covered in the this name with the predefined WEAK_ASSERTIONS and file. STRONG_ASSERTIONS sets. 6) Aggregation: A set is used to accumulate unique 6) Advanced Predicate Analysis: In the refined taxonedge-case categories identified, such as: NULL_INPUT, omy, calls to self.assertTrue are checked more ZERO_INPUT, EMPTY_COLLECTION. carefully using _is_strong_assertTrue(node). 7) Output: The final metric variety_count is computed as the size of this set (i.e., len(categories)), • If the argument is a simple variable, such as representing the number of distinct edge-case types assertTrue(variable), the assertion is clascovered in the file. sified as Weak. • If the argument contains a comparison, binary This static, literal-based approach is a heuristic. Its primary operation, or another function call, such as limitation, which must be This static, literal-based approach is assertTrue(x > 5), the assertion is classified a heuristic. Its primary limitation,is that it only detects edge
3
Table II: Taxonomy of assertion strength. Assertion Type assertNotNull(x)
Simple Taxonomy Weak
Justification (Simple) Checks for existence, not value.
Advanced Taxonomy Weak
assertTrue(x) assertTrue(variable)
Weak Weak
Checks for truthiness, not value. (See above)
Context-Dependent Weak
assertTrue(x > 5)
Weak
(See above)
Strong
assertTrue(is_valid(x)) Weak
(See above)
Strong
Checks for a specific, non-trivial value. Verifies a specific error-handling path.
Strong
assertEquals(5, x)
Strong
assertRaises(MyError)
Strong
Strong
Justification (Advanced) Confirmed; minimal semantic validation. Predicate must be inspected. Argument is ast.Name; lowvalue check. Argument is ast.Compare; encodes specific domain logic. Argument is ast.Call; encodes specific semantic logic. Confirmed; provides precise state validation. Confirmed; tests non-happy paths.
cases involving literals (e.g., 0, None, ”), and cannot detect This approach follows prior research on test flakiness and variable-driven edge cases (e.g., empty_list = [...]; test smells. Such work often uses AST-based static analysis to myfunc(empty_list)), as this would require complex detect code patterns, such as async waits or resource-related data-flow analysis, which is beyond the scope of this static issues, that are common causes of flaky tests. AST analysis. d) Phase 4: Synthesis and Comparative Evaluation: The final phase involves the synthesis of our findings. We • Pipeline C (Flakiness Analysis for RQ3) uses a hybrid approach to identify possible flaky tests. First, it uses static aggregate metrics from all three analysis pipelines and perform analysis to detect common patterns that may lead to flaky rigorous statistical comparisons between the AgentTest-PRs behavior, such as time.sleep, network calls, file I/O, and HumanTest-PRs cohorts for each quality dimension. These or non-deterministic functions such as random(). Then, results are analyzed to directly answer our research questions, for a sampled subset of tests that contain these patterns, and the findings are synthesized into a final discussion, we use dynamic analysis by running each test multiple conclusions, and a set of actionable recommendations for both times in an isolated environment. For example, each test developers and AI tool builders. may be executed 100 times to check whether the result changes across runs. The final output is a comparative A. “White-Boxing” the Analysis Pipelines flakiness rate for each group. This section moves beyond the high-level overview to detail White-Box Flowchart: exactly how inputs are processed and features are calculated for 1) Input: The input is the AST Module node of a Python each analysis pipeline. The implementation is a Python script test file. utilizing the native AST (Abstract Syntax Trees) module.The 2) Traversal: The TestAnalyzer.visit_Call(node) script defines a single TestAnalyzer class that inherits from method visits function calls inside test methods. ast.NodeVisitor which traverse a Python test file’s AST and Extraction: The helper function collect metrics for all three research questions simultaneously. 3) Path _get_call_path(node) examines node.func and extracts the full call path, such as (’time’, B. Manual Analysis and Exploratory Investigation ’sleep’) or (’open’,). 4) Pattern Matching: The extracted call path is compared Prior to embarking on a full-scale, automated analysis, we with the FLAKINESS_INDICATORS dictionary. For undertake a foundational phase of manual, qualitative analysis. example: This will involve the detailed inspection of a small, randomly • (’time’, ’sleep’) maps to ASYNC_WAIT. selected sample of 50 pull requests from each of our two cohorts • (’random’, ’random’) maps to (Agent-authored and Human-authored). This exploratory stage NON_DETERMINISM. is deliberately structured to follow the established “Explore, • (’open’,) maps to FILE_IO. Analyze, Synthesize” framework for qualitative data analysis, 5) Aggregation: Each matched indicator is added to allowing us to develop a nuanced, human-centric understanding the rq3 _flakiness _indicators list. The of the data before applying automated techniques. stored information also includes the line number, node.lineno, so that the matched code can be C. Automated Analysis Pipelines reviewed later. 6) Output: The candidate rate is calculated as The analytical core of our research is made up of three len(rq3_flakiness_indicators). This different, automated analysis pipelines, which integrate both value represents the number of flakiness indicators found static and dynamic techniques to systematically quantify the in the test file. multifaceted dimensions of test quality.
4
Table III: Heuristic Rules for Identifying Edge-Case Literals (RQ2). Edge-Case Category Null Inputs Empty Collections
Python Literal Value None [] or {}
Empty Strings Numeric Boundaries (Zero) Numeric Boundaries (Negative)
"" or ’’ 0 -1
Target AST Node ast.Constant(value=None) ast.List(elts=[]) ast.Dict(keys=[]) ast.Constant(value="") ast.Constant(value=0) ast.Constant(value=-1)
or
Example Code Fragment my_func(None) my_func([]) my_func("") my_func(0) my_func(-1)
Table IV: Rules for Identifying Flakiness Root Causes (RQ3). Flakiness Root Cause Async Wait
Code Pattern / Anti-Pattern Fixed time delays
Non-Determinism
Use of random functions
Environment / Resource Leak Environment / Resource Leak
Hard-coded File I/O Network Calls
Non-Determinism
System Time Dependency
Python AST Signature (Simplified) ast.Call(func=ast.Attribute(value=ast.Name(id=’time’), attr=’sleep’)) ast.Call(func=ast.Attribute(value=ast.Name(id=’random’), attr=’random’)) ast.Call(func=ast.Name(id=’open’)) ast.Call(func=ast.Attribute(value=ast.Name(id=’socket’), attr=’socket’)) ast.Call(func=ast.Attribute(value=ast.Name(id=’datetime’), attr=’now’))
a) Assertion Strength Analysis: To measure the semantic depth of the tests, we use tree-sitter to generate an Abstract Syntax Tree (AST) for each collected test file. We then traverse each AST to identify method calls that correspond to assertion functions in common testing frameworks, such as assertEquals, assertTrue, and assertIn.
c) Flakiness Analysis: This analysis uses a two-stage process. In the first stage, we perform static analysis on the test source code to identify patterns that are commonly linked to flaky tests. These patterns include the use of non-deterministic functions, such as random(), hard-coded time delays, such as Thread.sleep, interactions with networks or file systems, and dependencies on system time. Any test that contains one or more of these patterns is marked as a flakiness candidate. In the second stage, we perform dynamic analysis on a representative sample of these candidates. For each selected test, we create an isolated execution environment, check the specific software commit linked to the test, and run the test repeatedly, such as N = 100 times. A test is confirmed to be flaky if it produces both passing and failing results across these repeated runs without any changes to the source code.
Each assertion is classified using a predefined strength taxonomy. Assertions that check only basic conditions, such as non-null values using assertNotNull or simple truth values using assertTrue(variable), are classified as Weak. These assertions provide limited information about whether the program behavior is correct. In contrast, assertions that check specific expected values, exact exception types, or deep equality between complex data structures are classified as Strong. Examples include assertEquals(expected, actual) and assertions that compare nested objects or collections. These assertions provide a stronger form of validation because they check more precise program behavior.
III. E XPERIMENTAL R ESULTS
To rigorously evaluate the qualitative and quantitative characteristics of agent-generated software tests, our experimental b) Edge-Case Coverage Analysis: Using the same set of framework is structured around three central research questions ASTs, this pipeline evaluates how well the tests cover edge (RQs). Each of these questions is designed to probe one of the cases and boundary inputs. The pipeline identifies method calls critical dimensions of test quality that were identified in our inside test functions and extracts the literal values passed as study design. arguments. These values may include numbers, strings, and The following section derived from executing the Testnamed constants. Analyzer script on the two AgentTest-PRs cohort against the The extracted values are then compared with a predeestablished baseline provided by the cohorts of 20 test files fined list of common edge cases and boundary values. This each. The raw, aggregated data was sourced from the script’s list includes values such as 0, -1, None, and "", which JSON outputs. HumanTest-PRs cohort. represents an empty string. It may also include languagespecific boundary values such as Integer.MAX_VALUE and A. RQ1: How does the assertion strength of agent-generated Integer.MIN_VALUE. tests compare to human-written tests? a) Methodology: For each test file in both cohorts, we classified every assertion as either “Weak” or “Strong” using the taxonomy in Table V. We then computed the weak-to-total ratio per file. Files where this ratio is high are considered to
We use the frequency and variety of these edge-case values as a quantitative measure of test robustness. A test file with more diverse edge-case inputs is considered more likely to check behavior near the boundaries of the input domain.
5
Figure 1: Overview of our study design.
Percentage of Weak Assertions: (Count of Weak Assertions / Total Count of Assertions) × 100 • Percentage of Strong Assertions: (Count of Strong Assertions / Total Count of Assertions) × 100 e) Results: The numbers in Table VI turned out closer than we expected. Going in, we assumed there would be a clear quality gap between human and agent assertions, but that is not quite what happened. Agents did lean more toward weak assertions (14.63% vs. 11.92% for humans), and humans held a corresponding edge in strong ones (88.08% vs. 85.37%). But this is not the large divide that earlier work on semantic comprehension issues in LLM-generated code might lead you to expect [15]. The more telling result is in the “Unknown” column. Agents produced unrecognized assertion patterns at 11.58%, roughly eight times the 1.46% we saw in human-written tests. What this means in day-to-day terms is that agents frequently reach for
have shallow assertion depth. Aggregating these per-file ratios gives us a distribution for each cohort, which we compare directly. b) Weak Assertions: We treat an assertion as weak if it only checks whether something exists, whether a boolean condition holds, or whether an object is of a certain type, without pinning down the actual value or internal state. Typical examples are assertNotNull, assertTrue, and assertIsInstance [15]. c) Strong Assertions: A strong assertion nails down a specific expected output, confirms a particular state change, or checks a non-trivial behavioral property. Think assertEquals(5, result), assertThrows(SpecificException.class), or assertDeepEquals(expectedObject, resultObject) [15]. d) Metrics:
•
6
Table V: Assertion categories used in our analysis. Category
Assertion Methods
Weak
assertTrue, assertFalse, assertIs, assertIsNot, assertIsNone, assertIsNotNone, assertIn, assertNotIn, assertIsInstance, assertNotIsInstance
Strong
assertEquals, assertNotEquals, assertEqual, assertNotEqual, assertDictEqual, assertListEqual, assertSetEqual, assertTupleEqual, assertRaises, assertRaisesRegex, assertLogs, assertGreater, assertGreaterEqual, assertLess, assertLessEqual, assertAlmostEqual, assertNotAlmostEqual
Reclassified (Advanced)
assertTrue(expr) with logical/comparison or function call expressions (e.g., is_valid(x), x > 0).
Figure 2: Assertion strength comparison across cohorts. Agents exhibit a higher rate of ‘Unknown’ assertion types than humans.
Unknown
Project-specific or misspelled assertions, e.g., assertSomething, assertIsLess, or bare assert.
Table VII: Prevalence of edge cases in assertions.
Table VI: General statistics for human vs. agent cohorts. Metric
Human
Agent
A-Sliced
A-Sliced-Rand
Total Files Files w/Asserts Weak (%) Strong (%) Unknown (%) Cand. Rate Variety
24,941 1,730 11.92 88.08 1.46 0.30 0.32
179,732 29,353 14.30 85.70 10.93 0.44 0.61
24,941 3,412 13.90 86.10 9.22 0.46 0.58
24,941 4,046 14.63 85.37 11.58 0.41 0.62
•
assertion methods that either do not exist in standard libraries or are misspelled. A reviewer looking at one of these test files has to pause and work out whether the assertion is doing something clever with a project-specific helper, or whether the agent just made it up. Across a large codebase, that kind of friction adds up and can quietly erode maintainability. The takeaway here is that agents are not dramatically worse than humans in raw assertion strength. But the gap in discipline, writing assertions that a reader can immediately trust, matters more than the percentages alone would suggest.
Edge Case
Human
Agent
A-Sliced
A-Sliced-Rand
Null Input Zero Input Negative Input Empty String Empty Collection
8.3% 11.2% 0.0% 4.1% 8.1%
13.0% 27.7% 0.0% 5.7% 15.0%
13.5% 23.7% 0.0% 5.4% 15.3%
13.4% 27.7% 0.0% 5.8% 14.9%
Edge Case Variety. The percentage of tests in each cohort that include at least one test targeting our predefined categories: null inputs, empty collections, numeric boundaries (zero, negative values, maximum integers), and strings of zero or very long length.
c) Results: We expected humans to do better here, but the data in Table VII says otherwise. Agents beat humans on nearly every edge case category. For “Null Input,” agents hit 13.4% compared to 8.3% for humans. “Empty Collection” showed a similar gap: 14.9% vs. 8.1%. The biggest difference was in “Zero Input,” where agents reached 27.7%, more than double the human rate of 11.2%. The variety scores tell a similar story. Agents averaged 0.614 on our variety metric, almost twice the human score of 0.318 B. RQ2: To what extent do agent-generated tests cover edge (Table VI). So when agents write tests, they tend to throw a and boundary cases compared to human-written tests? wider net of input types into a single suite. One thing both a) Methodology: We scanned the input values used groups got wrong: neither humans nor agents tested negative in method calls across all collected test files. Our pipeline numeric boundaries at all (0.0% across the board), which is a flags literal values and constants that match known boundary blind spot worth noting. conditions and special cases: zero, negative numbers, null These results push back against the idea that LLMs are bad values, empty collections, and numeric or string length limits. at reasoning about boundary conditions. What seems to be For each test file, we recorded how many of these categories happening instead is that agents default to listing out standard appeared. boundary values (null, zero, empty) almost mechanically, while b) Metrics: human developers are more likely to skip cases they consider • Edge Case Frequencies. The average number of distinct unlikely based on their understanding of the code. Whether edge case patterns found per 1,000 lines of test code. This the agent’s extra coverage is genuinely useful or just noise is gives us a normalized density that is comparable across a separate question, but the raw numbers clearly favor agents files of different sizes. on this metric.
7
Table VIII: Prevalence of specific code patterns. Category Async Wait Non-Determinism File I/O Network I/O
Human
Agent
A-Sliced
A-Sliced-Rand
1.9% 3.1% 3.5% 0.1%
1.3% 5.2% 4.6% 0.2%
1.4% 5.8% 3.9% 0.2%
1.4% 5.2% 4.4% 0.1%
without mocking or cleanup, and that is where most of the extra flakiness risk comes from. Humans write flaky code too, of course, but agents do it at a noticeably higher rate in these two specific areas. Figure 3: Edge-Case coverage comparison. Agents significantly outperform humans in covering Zero Inputs and Null Inputs.
C. RQ3: What is the prevalence of flaky tests in pull requests submitted by AI agents compared to those submitted by humans? a) Methodology: Our two-stage flakiness analysis will first use static analysis to find a complete list of “flakiness candidates” from both the agent- and human-authored groups. This identification relies on the existence of recognized antipatterns and code constructs that exhibit a strong correlation with non-deterministic test behavior, including the utilization of Thread.sleep, interactions with network endpoints, file I/O operations, and invocations of non-deterministic APIs. After this static screening, we move on to a dynamic analysis phase. In this phase, we will run a random sample of 1,000 tests from each group of candidates. We will run each of these tests 100 times in a controlled, separate space. If a test shows both a passing and a failing result at least once during these repeated runs, it will be clear that it is flaky. b) Metrics: • Candidate Rate: The percentage of tests in a cohort that our static analysis flags as potential flakiness candidates. • Confirmed Flakiness Rate: Of the sampled candidates, the percentage that actually turned out to be flaky when run through our dynamic analysis. c) Results: This is the one research question where our initial guess held up. Agents do appear more prone to introducing instability. Their average candidate rate came in at 0.435, well above the human rate of 0.301 (Table VI). Looking at what drives that gap in Table VIII, agents leaned harder on “Non-Determinism” (5.2% vs. 3.1%) and “File I/O” (4.4% vs. 3.5%). Both of these point to agents not being careful about side effects and resource isolation when generating test code. The concurrency picture is less clear-cut. Humans actually used “Async Wait” patterns slightly more (1.9%) than agents did (1.4%), and network I/O was negligible for both groups. So it is not that agents are worse across every flakiness dimension. The problem is narrower than that: agents are specifically bad at handling random values and file system access. They write code that touches the disk or calls non-deterministic APIs
Figure 4: Flakiness indicator rates. Agents exhibit higher rates of Non-Determinism and File I/O usage. IV. T HREATS T O VALIDITY A. Threats to Conclusion Validity The validity of the conclusion concerns the ability to draw a correct conclusion about the treatment (author type) and the outcome (test quality metrics). • Statistical Power and Sample Size: The study’s comparative analysis depends on statistical significance. The ability to draw a statistically significant conclusion (e.g. “AI test has a higher proportion of weak assertions”) depends on the final sample size (number of test files) and the variance within each cohort. The sample size used in the experimental results (N=29491 per cohort) is not enough, and the variance in some cases is low (e.g., RQ3). A larger sample is required to draw more definitive conclusions. • Reliability of Measures: The analysis relies on the automated AST pipelines. A bug in the TestAnalyzer (e.g., misclassifying an assertion) would systematically skew the results and threaten then conclusion. This threat is mitigated by the “white-box” design, which makes all heuristics explicit and renewable. B. Threats to Construct Validity: Construct validity concerns whether the metrics being measured (RQs 1-3) are valid proxies for the construct they are intended to represent (“test quality”).
8
RQ1 (Assertion Strength): The “Advanced Taxonomy” specific AI models prevalent during the dataset’s collection is a significant mitigation for the weak assertTrue (e.g., older versions of GPT) and on the prompts provided construct. However, even this advanced proxy is limited. by their human collaborators. The findings for “AI” may The true strength of assertTrue(is valid(x)) depends on the not be generalizable to future, more advanced models or semantic complexity of the is valid(x) function, which the to different prompting-strategy environments. AST parser does not analyze. The metric is a proxy for strength, not a direct measure of it. This threat, particularly V. C ONCLUSION for RQ1, was confirmed to be the central finding of the experimental analysis. As discussed in earlier section, the To assess assertion strength, edge-case coverage, and flakiconstruct (the TestAnalyzer) was a poor proxy for “test ness, we analyzed more than 200,000 files in a comprehensive quality” in the Human-PR cohort due to its inability to empirical comparison of human-authored and agent-generated parse pytest-style assertions, thereby invalidating a direct test suites. What we found is less a story of one side comparison. being better, and more a trade-off: agents cover more ground • RQ2 (Edge-Case Coverage): The pipeline used in this mechanically, but they are less stable when it comes to study only finds literal constants (e.g., myfunc(0)). It execution environments. The common assumption that AI cannot detect variables-based or context-dependent edge agents simply write worse code than humans does not hold up cases, since data-flow analysis is not performed. Therefore, cleanly. For assertion quality (RQ1), agents roughly matched results may underestimate edge case coverage, particularly humans in the split between strong and weak assertions. Where in more advanced or human-authored tests. A test file they fell short was in what we call “assertion drift”: a high could have 100% comprehensive edge-case coverage using rate of unfamiliar or non-standard validation patterns that do variables, and our tool would incorrectly score it at 0. not belong to any recognized testing library. This implies that Therefore, this metric is a proxy for “developer awareness agents can lack the semantic accuracy necessary for reliable of literal edge cases,” not “true edge case coverage”. verification, even when they can replicate the structure of • RQ3 (Test Flakiness): The construct validity for RQ3 is rigorous testing. stronger, but only because the methodology is careful to Surprisingly, our examination of edge situations (RQ2) define it. The static analysis (Pipeline C) does not measure showed that AI greatly exceeds humans in terms of coverage “flakiness”; it measures “flakiness candidates”. A call to frequency, especially for common boundary conditions such Thread.sleep, for example, is a known anti-pattern but as zero and null inputs. This suggests that LLMs’ probabilistic can be a legitimate part of a test (e.g., testing a timeout). structure serves as a useful “fuzzing” technique by thoroughly The static metric alone is a weak construct. The construct listing cases that human developers frequently ignore because is only valid because it is the first stage of a two-stage of exhaustion or implicit domain assumptions. (static + dynamic) process defined in the methodology. However, the results of our flaking investigation offset this benefit (RQ3). Non-deterministic behaviors were statistically C. Threats to External Validity: External validity concerns the ability to generalize the study’s more likely to be introduced by agent-generated tests, particularly through incorrect file input/output and uncontrolled findings to other contexts. random seed usage. The greatest obstacle to independent • Language-Specific Threat: As detailed in earlier Sections, implementation of agent-based testing is still this lack of the provided implementation uses Python’s AST module, environmental knowledge. Therefore, the findings will only be valid for PythonIn the end, we determine that AI agents now work best as based projects. These results may not carry over to Java, high-volume “test generators” that need human supervision JavaScript, or C# projects, since those languages come rather than as human testers’ substitutes. They struggle with with their own testing frameworks, coding conventions, the depth of reliability (stability), they excel at broadening and AI generation behaviors. The RQ1 findings reinforced the scope of the test suite (coverage). Future research should this concern. The choice of Python, with its idiomatic focus on hybrid workflows in which human developers—or split between unit test and Pytest, was a decisive factor. A specialized static analysis tools—enforce semantic correctness study in a language with a more unified testing framework and environmental isolation while agents suggest edge cases (e.g., Java/JUnit) might yield different results. and assertion skeletons. • Project-Specific Threat: The AIDev dataset and the •
study’s recovery methodology are based on open-source projects. The developers, motivations, the quality asVI. F UTURE W ORK surance processes in open-source software may not be representative of closed-source, industrial, or enterprise The results of this study highlight a number of crucial environments. The findings may not be generalized beyond avenues for the development of automated tests in the future. the open-source context. Although the coverage breadths of current AI agents are • AI-Specific Threat: The study analyzes “Agent-PRs” from impressive, the following areas require focused research due to the AIDev dataset. The quality of these PRs depends on the their lack of environmental awareness and assertion precision.
9
VIII. ACKNOWLEDGMENTS
A. Environmental Modeling and Sandboxed Execution According to our data, the careless use of File I/O and non-deterministic APIs is a major cause of agent-generated flakiness (RQ3). Developing “environment-aware” agents that are specifically trained or prompted to identify the limitations of a CI/CD pipeline should be the main goal of future research. Integrating Sandboxed Execution Environments, where agents can run their generated tests iteratively, identify side effects (such as leftover files or thread leaks.) and self-correct before submission, is a promising approach. As a result, the paradigm changes from “Generate and Pray” to “Generate, Execute, and Refine.” B. Hybrid Assertion Generation (RAG + Rule-Based) In order to address the high frequency of “Unknown” and weak assertions (RQ1), future research should investigate hybrid architectures that integrate static analysis rules with large language models. The target repository’s specific assertion libraries and coding conventions could be dynamically fed to agents through the use of Retrieval-Augmented Generation (RAG). By grounding the agent’s output in legitimate, projectspecific syntax, this would lessen the “assertion drift” that our study found. Additionally, the semantic depth of generated tests could be greatly enhanced by fine-tuning models specifically on “strong” assertion patterns—those that verify state changes rather than just existence. C. Automated Mocking and Dependency Inference Agents are great at counting inputs, but have trouble with the context needed to test them safely, according to the “Edge Case Paradox” (RQ2). One area that deserves more attention is automated dependency inference. The idea is that agents would scan a method’s call graph, figure out which external dependencies are involved (databases, APIs, time providers), and generate the appropriate mocks on their own. Agents would be able to take advantage of their edge-case creativity without adding the instability that comes with integration testing if they went beyond simple input generation to complete “Test Fixture Synthesis.” D. Longitudinal Maintenance Studies Lastly, even though this study examined test development, it is still unknown how much maintenance of agent-generated suites will cost in the long run. Future long-term research should monitor the “survival rate” of agent-authored tests in industrial code bases, i.e., how frequently noise causes them to be removed, rewritten, or disabled. It is crucial to comprehend the “Maintenance-to-Value” ratio in order to ascertain the actual return on investment for software testing with AI. VII. DATA AVAILABILITY Our replication package is available online [1].
10
During the preparation of this work, the authors used the ChatGPT Web interface to improve the language and readability, and used Gemini to create an approach overview figure . After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication. R EFERENCES [1] Replication package. https://zenodo.org/records/20344789, Accessed: December 2025. [2] J. Altmayer Pizzorno and E. D. Berger. Coverup: Effective high coverage test generation for python. Proceedings of the ACM on Software Engineering, 2(FSE):2897–2919, 2025. [3] S. O. Barraood, H. Mohd, and F. Baharom. Test case quality factors. Turkish Journal of Computer and Mathematics Education, 12(3):1683– 1694, 2021. [4] A. Celik and Q. H. Mahmoud. A review of large language models for automated test case generation. Machine Learning and Knowledge Extraction, 7(3):97, 2025. [5] L. Crispin. Driving software quality: How test-driven development impacts software quality. IEEE software, 23(6):70–71, 2006. [6] A. Deljouyi, R. Koohestani, M. Izadi, and A. Zaidman. Leveraging large language models for enhancing the understandability of generated unit tests. pages 1449–1461, 2025. [7] C. Gao, X. Hu, S. Gao, X. Xia, and Z. Jin. The current challenges of software engineering in the era of large language models. ACM Transactions on Software Engineering and Methodology, 34(5):1–30, 2025. [8] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. [9] H. Li, H. Zhang, and A. E. Hassan. The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering, 2025. [10] R. Milanese, F. Salzano, A. Spina, A. Vitale, R. Pareschi, F. Fasano, and M. Fazzini. Human-agent versus human pull requests: A testing-focused characterization and comparison. arXiv preprint arXiv:2601.21194, 2026. [11] F. Molina, A. Gorla, and M. d’Amorim. Test oracle automation in the era of llms. ACM Transactions on Software Engineering and Methodology, 34(5):1–24, 2025. [12] W. C. Ouédraogo, K. Kaboré, Y. Li, H. Tian, A. Koyuncu, J. Klein, D. Lo, and T. F. Bissyandé. Large-scale, independent and comprehensive study of the power of llms for test case generation. arXiv preprint arXiv:2407.00225, 2024. [13] R. Pan, M. Kim, R. Krishna, R. Pavuluri, and S. Sinha. Multi-language unit test generation using llms. arXiv e-prints, pages arXiv–2409, 2024. [14] K. Qiu, N. Puccinelli, M. Ciniselli, and L. Di Grazia. From today’s code to tomorrow’s symphony: The ai transformation of developer’s routine by 2030. ACM Transactions on Software Engineering and Methodology, 34(5):1–17, 2025. [15] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation, 2023. [16] M. L. Siddiq, J. C. Da Silva Santos, R. H. Tanvir, N. Ulfat, F. Al Rifat, and V. Carvalho Lopes. Using large language models to generate junit tests: An empirical study. In Proceedings of the 28th international conference on evaluation and assessment in software engineering, pages 313–322, 2024. [17] Q. Yang, J. J. Li, and D. Weiss. A survey of coverage based testing tools. In Proceedings of the 2006 international workshop on Automation of software test, pages 99–103, 2006. [18] S. Yoshimoto, S. Fujita, K. Horikawa, D. Feitosa, Y. Kashiwa, and H. Iida. Testing with ai agents: An empirical study of test generation frequency, quality, and coverage. arXiv preprint arXiv:2603.13724, 2026. [19] Z. Yuan, Y. Lou, M. Liu, S. Ding, K. Wang, Y. Chen, and X. Peng. No more manual tests? evaluating and improving chatgpt for unit test generation. arXiv preprint arXiv:2305.04207, 2023.