Improving LLM-Driven Test Generation by Learning from Mocking Information Jamie Lee∗ , Flynn Teh∗ , Hengcheng Zhu † , Mengzhen Li ‡ , Mattia Fazzini ‡ , Valerio Terragni ∗ ∗ University of Auckland, Auckland, New Zealand † The Hong Kong University of Science and Technology, Hong Kong SAR
arXiv:2604.19315v1 [cs.SE] 21 Apr 2026
‡ University of Minnesota, Minneapolis, USA Emails: ∗ {eejl773, fteh492, vter674}@aucklanduni.ac.nz, † [email protected], ‡ {li001618, mfazzini}@umn.edu
Abstract—Large Language Models (LLMs) have recently shown strong potential for automated unit test generation. This has motivated us to investigate whether developer-defined test doubles (commonly referred to as mocks) available in existing test suites can be leveraged to improve LLM-driven test generation. To this end, we propose M OCK M ILL, an LLM-based technique and tool that generates test cases by exploiting mocking information automatically extracted from developer-written tests. M OCK M ILL targets components that are replaced by test doubles in existing tests and uses the encoded stubbings and interaction expectations to guide test generation, combined with an iterative generationand-repair process to ensure executable tests. We evaluated M OCK M ILL on 10 open-source classes from six JAVA projects using four LLMs, and compared the generated tests with existing project tests and tests produced by baseline approaches. The results show that M OCK M ILL’s tests cover lines of code and kill mutants that existing tests and baseline-generated tests miss. Overall, our findings provide preliminary evidence that leveraging mocking information is a complementary and effective way to enhance LLM-based test generation. Index Terms—Test Generation, LLMs, Test Doubles, Mocking.
I. I NTRODUCTION
configured to return predefined responses for specific inputs (stubbings) or to check expected interactions (verify operations). Through them, developers implicitly specify how dependent components should behave. Test doubles are central to modern software testing [10], and studies across programming languages show that stubbing and verify operations are widely used in both industry and open-source software [11], [12], [13], [14]. Tests generated from test double information can help verify dependent components when they are part of the software system. Existing LLM-based approaches typically generate tests using the code under test and its associated documentation or comments [8], [9], making the rich usage patterns and expected behaviors encoded in test doubles an untapped resource for improving automated test generation. To bridge this gap, we introduce M OCK M ILL, an LLM-based approach that leverages the mocking information from an existing test suite to generate new tests. Given the software under test and its test suite as inputs, M OCK M ILL generates tests for those components that are replaced by test doubles in existing tests. The approach starts by automatically extracting mocking information from the test suite via static analysis. M OCK M ILL then employs an LLMguided approach to generate tests for targeted components. The approach integrates a generation-and-repair loop: the extracted mock information is incorporated into the LLM input to generate candidate tests, and any compilation or runtime issues in the tests are iteratively fixed via LLM-based corrections. By guiding an LLM with this existing, developer-defined information, M OCK M ILL produces tests that align with the behavioral expectations already encoded in the test suite.
Software underpins modern life, making rigorous testing essential to ensure correctness [1]. Yet writing high-quality tests is labor-intensive [2], motivating decades of automated test generation research [2], [3]. Long-standing approaches range from random testing to search-based methods [4]. Recently, the emergence of Large Language Models (LLMs) has opened a new frontier for automated test generation [5], [6], [7]. LLMs, trained on vast code corpora, can produce useful tests, often with minimal human guidance [8], [9]. This capability has sparked interest in LLM-based testing [9]. However, while the results of LLM-driven test generation are highly We implemented M OCK M ILL and evaluated it on 10 classes promising, many aspects of LLM-based test generation remain from six open-source JAVA projects using four LLMs (GPT– underexplored, particularly the impact of providing specific 4 O M INI, GPT–5 M INI, GPT–5, and C LAUDE S ONNET 4.5). auxiliary artifacts beyond the class under test [9], [8]. We compare the tests generated by M OCK M ILL with existing In this paper, we explore how to improve LLM-driven test project tests and with two baselines: an LLM-based approach generation by leveraging test doubles [10] (or more informally that does not utilize mocking information and random test also referred to as mocks). Test doubles are test artifacts generation (R ANDOOP) [15]. We measured line and mutation commonly used to manage the complexity of interacting coverage to assess effectiveness. The results suggest that components during testing. They are lightweight stand-in M OCK M ILL covers code and kills mutants missed by existing objects that mimic real components, allowing developers to and baseline tests, providing preliminary evidence that it isolate the software under test from its dependencies [10], complements existing or automatically generated tests. We also referred to as mocked components. Test doubles can be released M OCK M ILL’s code and experimental data [16]. This is the authors’ version of the paper published in IEEE International Conference on Software Testing and Verification Workshops (ICSTW) - International Workshop on Artificial Intelligence in Software Testing (AIST).
II. I NTUITION AND RUNNING E XAMPLE
1 // Developer-written test with mocking info 2 @Test 3 void pageQueryAopLogsTest() { 4 ... aopLogRepository = mock(AopLogRepository.class); 5 PageRequestDto pageDto = PageRequestDto.of(1, 10); 6 AopLogQueryDto queryDto = new AopLogQueryDto(); 7 when(aopLogRepository.pageFetchBy(pageDto, queryDto)). thenReturn(mockResult); 8 // ...} 9 10 // Baseline-generated test without mocking info (line 7) 11 @Test 12 void baseline_test() { 13 AopLogRepository repository = new AopLogRepository(); 14 AopLog aopLog = new AopLog(); 15 aopLog.setId(1L); 16 repository.insert(aopLog); 17 AopLogQueryDto queryDto = mock(AopLogQueryDto.class); 18 when(queryDto.getId()).thenReturn(2L); 19 List<AopLog> result = repository.fetchBy(queryDto); 20 assertTrue(result.isEmpty());} 21 22 // MockMill-generated test giving mocking info (line 7) 23 @Test 24 void mockmill_test() { 25 AopLogRepository aopLogRepository = new AopLogRepository (); 26 PageRequestDto pageRequestDto = PageRequestDto.of(0,10); 27 AopLogQueryDto queryDto = new AopLogQueryDto(); 28 Result<Record> mockResult = mock(Result.class); 29 Result<Record> result = aopLogRepository.pageFetchBy( pageRequestDto, queryDto); 30 assertEquals(mockResult, result);}
Our key intuition is that mocking information encodes meaningful knowledge about how software components should be used and this can be leveraged to create useful tests. This work specifically leverages stubbings and verify operations [17] as mocking information (or mocking data), which often specify: (i) which methods of dependent components are invoked, (ii) the arguments passed to the methods, and (iii) the return values or exceptions resulting from those invocations. Stubbings are the natural way to encode (i), (ii), and (iii). Furthermore, verify operations can also specify (i) and (ii) by checking that certain methods have been called with certain arguments during test execution [17]. Such structured behavioral information serves as implicit usage documentation of dependent components. We use dependent components, replaced components, mocked components interchangeably to refer to those components that are replaced by test doubles and on which stubbings and verify operations are defined. When provided to an LLM during test generation, the information can guide the model toward realistic and meaningful tests that improve the coverage of the test suite. M OCK M ILL exploits this insight by extracting mocking information from existing tests and using the information to generate tests for the mocked components. These components are the target components in the test generation process and Fig. 1. Running example based on the AopLogRepository class. are the dependent components in existing tests. Figure 1 shows a running example, illustrating how M OCK M ILL leverages mocking information and the advantage of test generation, and (4) post-generation repair. Phases (1), (3), incorporating it. The figure includes three tests: the original and (4) follow common patterns in prior LLM-assisted test developer-written test pageQueryAopLogsTest (from which generation approaches [8], [9], while phase (2) is novel and M OCK M ILL extracts the mocking data), the test generated extracts realistic usage information about mocked components by an LLM-driven baseline without providing mocking infor- that prior approaches do not explicitly leverage. mation (baseline_test), and the test generated by M OCK Figure 2 provides a high-level overview of M OCK M ILL. M ILL using the extracted mocking data (mockmill_test). M OCK M ILL takes as input a software project configured with pageQueryAopLogsTest contains mocking data for the a standard build automation tool (e.g., M AVEN or G RADLE AopLogRepository class (i.e., the dependent component). for JAVA). It also accepts a configuration file to filter target This component is the target component (i.e., target class) for components (e.g., classes in JAVA), specify LLM settings, baseline_test and mockmill_test. AopLogRepository define the number of repair attempts, and set token limits. is also a subject of our evaluation (Section IV). M OCK M ILL begins with the Project Analysis phase, which finds The baseline approach generated a test that exercises existing tests using stubbings or verify operations and identifies only the fetchBy method, representing the most com- associated components, which are the target components mon and straightforward usage of the repository ob- for test generation. The Mock Extraction phase then gathers ject. The existing developer test pageQueryAopLogsTest structural data (i.e., the specific content of stubbings and mocks AopLogRepository and stubs its pageFetchBy verify operations) about the dependent component. This data is method, which reflects a more advanced usage scenario passed to the Test Generation phase, where the LLM produces involving paging. Providing the mocking information in candidate test cases based on extracted mock data. Finally, pageQueryAopLogsTest enables M OCK M ILL to guide an the Post-Generation Repair phase compiles, executes, and LLM to generate a test that exercises the paging mechanism iteratively repairs the generated tests until they run successfully. of AopLogRepository, thereby covering more lines of code The output of M OCK M ILL consists of automatically generated and killing mutants injected in that method. tests for components that are substituted by test doubles in other tests. The output also includes logs about compilation and III. M OCK M ILL repair attempts, along with code and mutation coverage reports M OCK M ILL is an LLM-based automated test generation that help assess the quality of the generated tests. The rest approach that leverages mocking information to generate test of this section describes M OCK M ILL’s phases in detail. The cases. Its methodology comprises four phases: (1) project LLM prompts used by the approach are omitted due to space analysis, (2) mock extraction (introduced in this work), (3) limitations but are available in the replication package [16].
Fig. 2. High-level overview of M OCK M ILL’s workflow.
A. Project Analysis The Project Analysis phase identifies the dependent components that can be targets for test generation by analyzing the test code in the project under analysis. Specifically, this phase parses the abstract syntax tree (AST) of the test files in the project to identify the components that have been replaced by test doubles and that use stubbings or verify operations. In our implementation, we identified the test cases written using JU NIT 5 and the test doubles created with M OCKITO, since they are the most popular testing and mocking frameworks in the Java ecosystem [14]. Once M OCK M ILL identifies this information, this phase filters out components that are not part of the project (i.e., components from third-party libraries) by keeping only the classes that are defined in the source code of the project (and not in the libraries used by the project). The approach performs this step as generated tests should focus on code that belongs to the project. M OCK M ILL encodes the relevant targets in a structured representation (i.e., a JSON file) that is then passed to the subsequent phase of the approach. Running Example: For the project in Figure 1 (lines 2–9), this phase identifies that the test pageQueryAopLogsTest mocks AopLogRepository and defines a stubbing on it, selecting the class as a candidate for mock-informed test generation. B. Mock Extraction
part of the structured input provided to the generator, ensuring that the LLM is aware of the paging behavior exercised in the developer-written test. C. Test Generation The Test Generation phase constructs LLM prompts for automated test creation based on the target component and the extracted mock information. In this phase, a structured prompt is assembled that provides the LLM with (i) explicit test generation instructions, (ii) the complete source code of the target component (e.g., the source code of the target class), and (iii) the data associated with the relevant stubbings and verify operations. Including the full class source code enables the LLM to reason about the implementation context, dependencies, and behaviors, thereby reducing the likelihood of incorrect assumptions or hallucinations [5], [6]. Prompt design: The test generation prompt targets mutationbased adequacy [18], as mutation coverage is widely recognized as one of the most comprehensive test adequacy criteria [18]. The prompt instructs the LLM to (i) use exact values from stubbings and verify operations; (ii) create precise assertions that fail under realistic mutations (bugs); (iii) test both true and false execution paths and boundary conditions derived from mocking information; and (iv) generate tests that instantiate and exercise real objects of the CUT rather than substituting them with test doubles.
The Mock Extraction phase focuses on collecting the mocking information only relevant to each target component and structuring it into an intermediate JSON representation (which Baselines. The approach also uses a few-shot prompting will be provided to the LLM for test generation). This phase strategy [19] and provides two examples for the requested identifies calls to mocking APIs of the supported mocking task. The examples are based on simple, self-contained JAVA frameworks (e.g., when(...).then(...) or verify() in classes along with corresponding structured mock and stub M OCKITO), along with argument values associated with the data extracted from representative tests. This data captures calls to the API methods by parsing the AST of the test real interaction patterns that reflect how the CUT is used. The files. The analysis traverses both test and setup methods to examples then include executable unit tests that instantiate the record field assignments, verification calls, and stubbing logic CUT, exercise its behavior, and assert on expected outputs. associated with the selected class. This focused extraction We intentionally designed the two few-shot examples to be ensures that the Test Generation phase receives precise input domain-agnostic (and are not based on any of the projects data without including the entire source files of the test, which considered in the evaluation). The prompt also does not target could otherwise overload the LLM context window and add or identify specific mutants to kill [20]. It encourages the LLM noise, reducing the focus and overall quality of generated tests. to produce tests that satisfy mutation-based adequacy criteria Running Example: In Figure 1, the Mock Extrac- in a general sense. Finally, the prompt specifies the required tion phase analyzes pageQueryAopLogsTest and records output format, directing the LLM to produce a compilable test that the method pageFetchBy(pageDto, queryDto) of file that includes all necessary package declarations and import AopLogRepository is stubbed. This extracted detail becomes statements.
TABLE I E VALUATION DATASET OF OUR EXPERIMENTS . T HE TABLE REPORTS THE DETAILS ON THE TARGET COMPONENTS (CUT S ) AND THEIR PROJECTS .
CUT Name of CUT ID
Project Name
Project CUT Methods Max CC Tests w/ TDs Ss VOs Dev LOC LOC Tests? w/ Ss w/ VOs 209 336 100 271 125 205 150 130 92 97
13 15 8 16 6 8 6 19 7 7
9 13 13 10 8 12 6 7 22 10
5 4 20 8 1 2 5 1 5 3
5 8 0 6 11 23 5 16 15 1 2 4 5 5 0 2 5 5 3 3
9 0 12 6 16 4 5 0 5 3
✗ ✓ ✗ ✗ ✓ ✗ ✓ ✓ ✓ ✗
CAS CustomerApplicationService E-Commerce Platform RRA ResourceRightSizingAnalyzer E-Commerce Platform MAS McpAsyncServer MCP Java SDK MUT McpUriTemplateValidator MCP Java SDK MQS MaestroQueueSystem Maestro MPC ModulePermissionConverter Miaocha QPC QueryPermissionChecker Miaocha SMS SemanticSchema SuperSonic ALR AopLogRepository ZhiLu AI Management URE UserRepository ZhiLu AI Management
45.9K 45.9K 27.4K 27.4K 85.7K 37.0K 37.0K 61.5K 7.5K 7.5K
D. Post-Generation Repair
It supports both M AVEN and G RADLE-based projects for automated compilation and test execution. We implemented the static analysis and mock extraction of M OCK M ILL using the JAVA PARSER (https://javaparser.org/) library, which enables AST-based analyses of source and test files. All model calls and repair iterations are logged to facilitate reproducibility and further analysis.
After generation, M OCK M ILL iteratively compiles and repairs tests, following prior LLM-driven test generation [21], [8]. If compilation fails, the resulting error messages are passed to the LLM with targeted repair instructions to produce a corrected test file. This cycle repeats until the tests compile successfully or the retry limit is reached. The same iterative process applies to runtime failures. Once compilation succeeds, M OCK M ILL executes the tests and, upon failure, provides the A. Dataset error logs to the LLM for correction. This loop continues until We evaluated M OCK M ILL on open-source repositories all tests pass or the maximum retry limit is reached. selected from GitHub based on the following criteria. The This design assumes a regression testing scenario [22], [23], repository uses at least JAVA 8, JU NIT 5, and M OCKITO 4 where failing generated tests are attributed to issues in the to ensure compatibility with modern testing practices. To tests themselves rather than bugs in the target component. mitigate data leakage, we included repositories updated after By default, M OCK M ILL treats failures as test issues (e.g., a the latest knowledge cutoff date of the model considered component not properly set up) and attempts to fix them. Users (January 2025). The repository has at least 20 GitHub stars, can also manually inspect failing tests if they suspect the failure indicating community interest and maintenance. The repository originates from bugs in the target component. The final test uses M AVEN or G RADLE to facilitate automated building, suite is executed for coverage and mutation analysis to evaluate dependency resolution, and test execution. The repository adequacy and fault-detection capability. contains at least one mocked class. Prompt design: The repair prompt includes the error message Mining GitHub using the above criteria yielded 24 candidate along with the source code of the target component and the repositories. We manually examined each repository to identify generated tests. It explicitly forbids explanations or comments target components (which we also refer to as components under and requires the LLM to use a significant change in approach test or CUTs) based on the following criteria. The CUT has on repeated attempts. These constraints reduce verbosity and at least 50 lines of code, ensuring sufficient implementation focus the LLM on producing concise and executable tests. complexity. The CUT implements at least five methods, with at least one having cyclomatic complexity ≥ 5, as done in related IV. E VALUATION work [4]. The CUT is replaced by a test double with stubbings This section presents the evaluation of M OCK M ILL, which or verify operations in other tests, allowing M OCK M ILL to is based on the following research questions (RQs): extract and reuse its defined mocking behavior. RQ1: Effectiveness – To what extent can M OCK M ILL generThis selection methodology yielded ten CUTs from six ate effective tests? unique repositories. Table I provides details on the projects and RQ2: Complementarity – To what extent do tests generated the CUTs. Column “Max CC” denotes the highest cyclomatic by M OCK M ILL complement those generated by baseline complexity of the methods in the CUT. Columns under “Tests approaches and existing tests? w/ TDs” report the number of tests that define a stubbing RQ3: Cost – What is the cost of generating tests using (“w/ S”) or a verify operation (“w/ VO) on the test doubles M OCK M ILL? replacing the CUTs. Column “Dev Tests” indicates whether To evaluate M OCK M ILL, we implemented it in a prototype the repository contains developer-written tests that directly test tool for JAVA projects that use JU NIT [24] and M OCKITO [25]. (i.e., exercise) the CUTs.
TABLE II LLM S USED IN THE EVALUATION AND TOKEN COSTS . Provider
Model
O PENAI O PENAI O PENAI A NTHROPIC
GPT–4 O M INI GPT–5 M INI GPT–5 C LAUDE S ONNET 4.5
Input Cost
Output Cost
$0.15 / 1M tok $0.25 / 1M tok $1.25 / 1M tok $3.00 / 1M tok
$0.60 / 1M tok $2.00 / 1M tok $10.00 / 1M tok $15.00 / 1M tok
B. Methodology Baselines. To evaluate the complementarity of M OCK M ILL, we considered three baselines: an LLM-based test generation approach (which we refer to as LLM), R ANDOOP [15] (i.e., an approach based on random testing), and the developerwritten tests. LLM uses the same prompt and few-shot examples as M OCK M ILL but omits mock and stub information. This configuration allows for a controlled comparison that isolates the relative contributions of mock information in the LLM-driven test generation process. Although E VOSUITE is generally more effective than R ANDOOP [4], we used R ANDOOP because it provided better support for the recent JAVA versions associated with the projects considered. Across projects, developer-written tests were found for only five CUTs. Models. In RQ1, we evaluated M OCK M ILL in combination with four LLMs. Table II lists the models we used and the token costs used at the time we ran the experiments (September 2025). At that time, GPT–4 O M INI was recognized as a lightweight, cost-efficient model, GPT–5 M INI and GPT–5 represented newer reasoning-capable variants, and C LAUDE S ONNET 4.5 provided a cross-provider comparison. All models were used with default parameters. In RQ2, we used GPT–5 M INI, as it was a top-performing model in RQ1 and it was cheaper than the other top-performing model C LAUDE S ONNET 4.5. Selecting this model allowed us to focus the comparison on the relative benefits of M OCK M ILL while controlling experimental costs and maintaining consistency across runs. Metrics. We considered four classes of metrics: generation, compilation, execution, and quality. In terms of generation, we report the average number of TeSt generaTed per CUTs (TST). Compilation metrics assess whether M OCK M ILL produces tests that can be built automatically. These metrics capture feasibility boundaries: percentage of tests that Compiles on the First Try (CFT) demonstrates immediate usability, while percentage of tests that Compiles EVentually (CEV) reflects the effectiveness of M OCK M ILL’s repair loop. Execution metrics measure how many executable tests M OCK M ILL generates. TestS Passed (TSP) reports the percentage of tests that pass. Quality metrics assess how effectively the generated tests detect faults and exercise code, since quantity and executability alone do not ensure usefulness. We report the minimum (MIN), median (MED), maximum (MAX), and standard deviation (STDEV) of mutation score (fault detection) and line coverage (structural coverage) [26]. We use PIT [27] (with all mutations enabled) and JAC O C O [28] to measure mutation score and line coverage, respectively. For fault detection, we also report Unique Mutations Killed, which isolates the exclusive contributions of
a given technique, revealing complementarity that aggregate percentages can hide. For structural coverage, Unique Lines Covered highlights lines reached only by a technique. Together, these metrics provide a nuanced picture, as it is possible to identify when approaches add non-overlapping value relative to others. Setup. We executed M OCK M ILL and LLM ten times to account for randomness in LLM outputs. We repeated the same procedure for R ANDOOP. C. Results 1) RQ1: Effectiveness: – To what extent can M OCK M ILL generate effective tests? Table III reports the results associated with RQ1. The average number of generated tests (TST) varies markedly across models. While GPT–4 O M INI, GPT–5 M INI, and GPT–5 generate between 8.5 and 13.2 tests on average, C LAUDE S ONNET 4.5 produces a much larger number of tests (45.7 on average). However, this larger test volume does not translate into clearly better quality than other models in terms of median mutation score and line coverage. M OCK M ILL is generally successful at producing compilable tests across all considered models. Although first-try compilation (CFT) varies across models, eventual compilation (CEV) is consistently high, ranging from 92% for GPT–4 O M INI to 100% for the remainder. This result indicates that the generation-and-repair loop of M OCK M ILL is effective at overcoming a substantial portion of the issues that arise during initial test generation. The execution results further support the effectiveness of M OCK M ILL. In terms of pass rate, all models except GPT–4 O M INI achieve near-perfect results, with TSP values between 98.6% and 99.7%. Taken together, these results indicate that M OCK M ILL is able to generate not only compilable but also executable tests with high reliability, especially when paired with GPT–5 M INI, GPT–5, or C LAUDE S ONNET 4.5. The mutation score results show that M OCK M ILL can generate tests with strong fault-detection capability. All models reach high maximum mutation scores, ranging from 85% for GPT–4 O M INI to 100% for GPT–5, GPT–5 M INI, and C LAUDE S ONNET 4.5. The median mutation score provides a more robust picture of typical performance across runs. Here, GPT–5 M INI and C LAUDE S ONNET 4.5 obtain the strongest medians, with 84% and 89%, respectively, while GPT–5 reaches 62% and GPT–4 O M INI 43%. At the same time, the standard deviation values are relatively large for all models, indicating variability across runs and CUTs. Still, the high medians and maxima for GPT–5 M INI, GPT–5, and C LAUDE S ONNET 4.5 show that M OCK M ILL can often generate tests that kill a substantial fraction of mutants. Among these models, GPT–5 M INI stands out because it combines a high median mutation score with much stronger compilation and execution behavior than C LAUDE S ONNET 4.5 (and at much lower cost, see RQ3). The line coverage results further support the effectiveness of M OCK M ILL. Median line coverage is high for the three
TABLE III E VALUATION RESULTS OBTAINED BY RUNNING M OCK M ILL ON THE CUT S OF THE DATASET (RQ1).
Model GPT–4 O M INI GPT–5 GPT–5 M INI C LAUDE S ONNET 4.5
Generation
Compilation
Execution
Mutation Score (%)
Line Coverage (%)
TST
CFT (%)
CEV (%)
TSP (%)
MIN
MED
MAX
STDEV
MIN
MED
MAX
STDEV
8.5 13.2 11.4 45.7
55 43 88 44
92 100 100 100
81.6 98.6 99.7 99.7
0 16 3 5
43 62 84 89
85 100 100 100
27.0 26.7 25.6 33.1
0 79 27 39
58 91 93 94
93 100 100 100
33.4 6.4 11.4 12.7
strongest models: 91% for GPT–5, 93% for GPT–5 M INI, R ANDOOP (2.61% for RRA). Mock information yields a small and 94% for C LAUDE S ONNET 4.5. Moreover, all three but measurable advantage, exposing a few additional lines not models reach 100% maximum line coverage, while GPT–4 O reached by other techniques. However, its overall impact on M INI reaches a maximum of 93%. GPT–4 O M INI shows coverage is less pronounced than on mutant detection. substantially weaker typical performance, with a median line Developer tests contribute the fewest unique mutations coverage of 58%, whereas the other three models consistently (2.56% for RRA, none elsewhere), indicating that LLMexercise a large portion of the target code. Although C LAUDE generated tests can reveal faults missed by human developers. S ONNET 4.5 attains the highest median line coverage, its Mock information enhances this advantage by improving fault advantage over GPT–5 M INI is marginal (94% vs. 93%). detection diversity. For line coverage, differences are minor. Thus, considering line coverage together with compilation M OCK M ILL-generated tests again show the highest exclusive and execution reliability, GPT–5 M INI emerges as the most coverage (e.g., 2.61% for RRA, 1.56% for MQS), while nopractical and effective choice. mock and developer tests contribute little additional coverage. Overall, the results provide evidence that M OCK M ILL is Although the differences are minor, there is evidence that mock effective at generating useful tests. Across the stronger models, information can help LLMs reach certain lines missed by LLM, M OCK M ILL almost always produces executable tests after R ANDOOP, and developer tests. repair and often achieves high line coverage and mutation 3) RQ3: Cost – What is the cost of generating tests using scores, indicating that the generated tests are not merely M OCK M ILL?: Table V presents the average test generation runnable but also meaningful from a testing perspective. Among cost (USD) and input/output token usage for M OCK M ILL and the evaluated models, GPT–5 M INI offers the best overall tradeLLM across LLMs. Costs were derived from the input and off. It achieves the highest first-try compilation rate, perfect output tokens consumed in each run and the corresponding eventual compilation, near-perfect pass rate, and strong median model-specific API pricing. mutation score and line coverage. At the same time, it is At the prompting level, providing mock information considerably smaller and cheaper than GPT–5 and C LAUDE (M OCK M ILL vs. LLM) slightly increases input tokens and thus S ONNET 4.5, making it the most effective and practical model cost by increasing the prompt length. These effects typically for M OCK M ILL based on our evaluation. add around 5-15% of cost within the same model, and start to 2) RQ2: Complementarity – To what extent do tests gener- be noticeable with models where token rates are higher. ated by M OCK M ILL complement those generated by baseline At the model level, cost increases predictability with model approaches and existing tests?: Across CUTs, M OCK M ILL size and capabilities. C LAUDE S ONNET 4.5 has the highest achieves higher unique mutation kill rates than both LLM cost, followed by GPT–5 (˜$0.08–$0.11), whereas the mini and R ANDOOP only (shown in Table IV). CAS and RRA variants are quite efficient (typically under $0.02), with GPT–5 are prime examples of this, showing a M OCK M ILL detection M INI being the most cost-effective model in terms of cost and rate of 24.56% and 25.64%, compared to LLM ( 0-12.82%). quality of generated tests. M OCK M ILL achieves a strictly higher unique mutation killed rate than LLM for 40% of CUTs and R ANDOOP for 50% V. D ISCUSSION of CUTs. R ANDOOP rarely kills any unique mutations, with killing 2.56% only on RRA. For MAS, MUT, QPC, and Interpretation of Results. The results indicate that M OCK SMS, the table shows 0% across all approaches, indicating M ILL can guide LLM models toward useful tests. By leveraging that M OCK M ILL performs similarly across all cases. This mock information, M OCK M ILL provides contextual cues that suggests that mock information generally helps identify unique help uncover behaviors and faults missed by a vanilla LLM, mutation cases missed by other tools. The low R ANDOOP R ANDOOP, or developer-written tests. This is supported by the performance indicates that a simple traditional test generation moderate number of unique mutations killed and unique lines approach might struggle to detect nuanced faults without the of code covered by M OCK M ILL. Because each experiment was contextual and semantic reasoning available to LLMs. Unique repeated ten times, we are confident that the observed benefits line coverage is generally low across all CUTs, with only are consistent and not due to random variation in LLM outputs. minor contributions from M OCK M ILL (e.g., 2.61% for RRA, Mock-informed test generation consistently killed mutations 1.56% for MQS, and 1.79% for URE) and one case for that were otherwise undetected. This suggests that stubbings
TABLE IV BASELINE C OMPARISON WITH LLM, R ANDOOP AND DEVELOPER - WRITTEN TESTS BY CUT S (RQ2)
CUT ID M OCK M ILL CAS RRA MAS MUT MQS MPC QPC SMS ALR URE
Unique Mutations Killed (%) LLM R ANDOOP
24.56 25.64 0.00 0.00 10.53 1.23 0.00 0.00 4.26 0.00
1.75 12.82 0.00 0.00 5.26 0.00 0.00 0.00 4.26 0.00
Dev Tests
M OCK M ILL
– 2.56 – – 0.00 – 0.00 0.00 0.00 –
0.00 2.61 0.00 0.00 1.56 0.83 0.00 0.00 0.00 1.79
0.00 2.56 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
TABLE V AVERAGE TEST GENERATION COST AND TOKEN USAGE (RQ3)
Model GPT–4 O M INI GPT–5 M INI GPT–5 C LAUDE S ONNET 4.5
M OCK M ILL
LLM
Cost Input Output USD Tokens Tokens
Cost Input Output USD Tokens Tokens
$0.0076 $0.0176 $0.1028 $0.5132
36,761 16,477 15,228 78,713
3,506 6,723 8,377 18,471
$0.0062 $0.0171 $0.0786 $0.4358
26,805 13,352 9,043 58,918
3,632 6,852 6,733 17,272
and verify operations activate distinct reasoning pathways within models and help them focus on different faults. Complementary. We applied a Kruskal–Wallis test to assess whether mutation scores and line coverage differ significantly across techniques. All H values (0.19–1.94) and p-values (≥ 0.584) indicate no statistically significant differences between M OCK M ILL and the BASELINE. As such, M OCK M ILL complements existing approaches. It can enhance existing test suites by generating additional tests that detect more faults and expand behavioral coverage. Indeed, the contextual cues derived from stubbing and verify operations that guide test generation may reduce test diversity, since the model tends to follow the provided interaction patterns. Future LLM-driven test generation could consider combining prompts both with and without mock information when available. Cost and Practical Implications. The cost analysis shows that M OCK M ILL incurs only a modest cost overhead (about 5–15% higher than LLM generation without mock information) while killing unique bugs and covering additional code. This trade-off is favorable for practical adoption, as M OCK M ILL can deliver measurable testing benefits at an affordable additional computational cost.
Unique Lines Covered (%) LLM R ANDOOP 1.18 0.65 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 2.61 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Dev Tests – 2.61 – – 0.00 – 0.00 0.00 0.00 –
Metric limitations. The metrics we used (generation, compilation, execution, and coverage) capture only part of test quality. Generated tests that compile and pass may still need improvement. Future work should include a manual assessment of the quality of generated tests. Model bias and training data leakage. To mitigate data leakage, we selected projects updated after the training cutoff date of the selected models. Despite this precaution, it remains difficult to guarantee that the results stem purely from reasoning rather than memorization of data (possibly from similar projects) seen during training. VI. R ELATED W ORK
Traditional Test Generation Techniques. Random testing [15] and search-based software testing [29] reduce manual effort by automatically producing tests to maximize structural coverage. Similar to M OCK M ILL, these techniques leverage information in source code to enhance test generation. MS EQ G EN [30] mines method call sequences from code repository to achieve higher coverage. Fraser and Zeller [31] mine object usage patterns from existing code to generate meaningful tests. Our work complements them by utilizing mocking information to guide test generation toward better tests. Automated Test Generation with LLMs. LLM-based approaches generate unit tests via prompt engineering and structured context. Studies on zero-shot, few-shot, and Chainof-Thought prompting show that richer, code-aware inputs (e.g., signatures, existing tests, stack traces) outperform plain natural language prompts [6]. Accordingly, pipelines typically collect project context, query an LLM, and iteratively compile/execute and repair failing outputs [8], [9], [21], [7]. Similar to prior work, M OCK M ILL relies on prompt design and post-processing. However, unlike approaches that build context solely from A. Threats to Validity source code and project metadata, M OCK M ILL also leverages Generalizability. We evaluated our approach on ten classes developer-written mocks and stubs as structured input to guide from six projects. Future work is needed to assess whether the the LLM in generating realistic tests for mocked and stubbed approach generalizes across other projects, languages, and classes. Although prior tools may generate tests with test mocking frameworks. Although the current prototype and doubles, none explicitly exploit existing mocking information experiments focus on M OCKITO and JAVA-based projects, the to guide test generation. To the best of our knowledge, approach can be extended to other languages and frameworks. M OCK M ILL is the first to do so. For example, to P YTHON projects using UNITTEST. MOCK or Test Doubles in Software Testing. Recent work has introJAVA S CRIPT projects using J EST. duced tools that analyze, generate, or refactor mocks and
stubs to improve testing quality and maintainability. M OCK S NIFFER [32] characterizes recommends mocking decisions. Empirical studies on mock assertions [14] suggest that mocks capture developers’ behavioral intent and domain knowledge. M OCK M ILL directly utilize this encoded intent to inform LLM-driven test generation. S TUB C ODER [33] generates and repairs stub code via evolutionary search to keep tests passing as production code evolves. RICK [34] records production executions and generates tests that mimic observed behavior using test doubles. ARUS [35] improves maintainability by detecting and removing unnecessary stubbings. Compared with these approaches, M OCK M ILL does not aim to generate or improve test doubles but instead leverages the behavioral information already encoded in them to guide LLMs in generating new tests. VII. C ONCLUSIONS AND F UTURE W ORK This paper introduced M OCK M ILL, the first LLM-based test generation approach that leverages developer-written mocking information to guide test creation. M OCK M ILL generates tests that uncover lines of code and mutants missed by baseline approaches, showing that mock information provides valuable contextual cues for producing realistic and useful tests. The approach complements developer-written suites, traditional tools such as R ANDOOP, and an vanilla LLM-based approach. The cost overhead of 5–15% with respect to the LLM baseline considered is acceptable given the corresponding improvements in test quality and fault detection capability. Our findings suggest that future LLM-driven test generation should incorporate available mocking information as contextual guidance to produce more comprehensive and realistic tests. Rather than replacing existing methods, mock-informed generation should complement them within a holistic LLMbased testing framework that integrates both mock-informed and non–mock-informed prompts. As the first work of its kind, M OCK M ILL opens several promising research directions. Future work includes ablation studies to assess the relative impact of mocking information, automatic extraction of project-specific few-shot examples to improve performance, dynamic collection of mocking data to capture richer behaviors, and establishing traceability links between test doubles and generated tests to better understand their influence on test creation. R EFERENCES [1] G. J. Myers, C. Sandler, and T. Badgett, The art of software testing. John Wiley & Sons, 2011. [2] A. Bertolino, “Software testing research: Achievements, challenges, dreams,” in FOSE. IEEE, 2007, pp. 85–103. [3] S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, P. McMinn, A. Bertolino et al., “An orchestrated survey of methodologies for automated software test case generation,” JSS, vol. 86, no. 8, pp. 1978–2001, 2013. [4] G. Jahangirova and V. Terragni, “Sbft tool competition 2023 - java test case generation track,” in SBFT, 2023, pp. 61–64. [5] A. Bodicoat, G. Jahangirova, and V. Terragni, “Understanding llm-driven test oracle generation,” in ACM AIWARE, 2025. [6] W. C. Ouédraogo, K. Kaboré, H. Tian, Y. Song, A. Koyuncu, J. Klein, D. Lo, and T. F. Bissyandé, “Large-scale, independent and comprehensive study of the power of llms for test case generation,” arXiv, 2024.
[7] V. Terragni, A. Vella, P. Roop, and K. Blincoe, “The future of ai-driven software engineering,” ACM TOSEM, 2025. [8] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” IEEE TSE, vol. 50, no. 1, pp. 85–105, 2023. [9] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,” IEEE TSE, vol. 50, no. 4, pp. 911–936, 2024. [10] G. Meszaros, xUnit test patterns: Refactoring test code. Pearson Education, 2007. [11] S. Mostafa and X. Wang, “An empirical study on the usage of mocking frameworks in software testing,” in QSIC. IEEE, 2014, pp. 127–132. [12] D. Spadini, M. Aniche, M. Bruntink, and A. Bacchelli, “Mock objects for testing java systems: Why and how developers use them, and how they evolve,” ESEM, vol. 24, pp. 1461–1498, 2019. [13] M. Fazzini, C. Choi, J. M. Copia, G. Lee, Y. Kakehi, A. Gorla, and A. Orso, “Use of test doubles in android testing: An in-depth investigation,” in ICSE, 2022, pp. 2266–2278. [14] H. Zhu, V. Terragni, L. Wei, S.-C. Cheung, J. Wu, and Y. Liu, “Understanding and characterizing mock assertions in unit tests,” ACM PACSE, vol. 2, no. FSE, pp. 554–575, 2025. [15] C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, “Feedback-directed random test generation,” in ICSE. IEEE, 2007, pp. 75–84. [16] Lee, Jamie and Teh, Flynn and Zhu, Hengcheng and Li, Mengzhen and Fazzini, Mattia and Terragni, Valerio, “MockMill Replication Package,” https://doi.org/10.5281/zenodo.19490389, 2026. [17] Mockito Framework, “Mockito javadoc,” https://javadoc.io/doc/org. mockito/mockito-core/latest/org.mockito/org/mockito/Mockito.html, 2025, accessed: March 2026. [18] M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman, “Mutation testing advances: an analysis and survey,” in Advances in computers. Elsevier, 2019, vol. 112, pp. 275–378. [19] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. [20] C. Foster, A. Gulati, M. Harman, I. Harper, K. Mao, J. Ritchey, H. Robert, and S. Sengupta, “Mutation-guided llm-based test generation at meta,” arXiv preprint arXiv:2501.12862, 2025. [21] R. Ravi, D. Bradshaw, S. Ruberto, G. Jahangirova, and V. Terragni, “Llmloop: Improving llm-generated code and tests through automated iterative feedback loops,” ICSME. IEEE, 2025. [22] S. Shamshiri, R. Just, J. M. Rojas, G. Fraser, P. McMinn, and A. Arcuri, “Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges,” in ASE. IEEE, 2015, pp. 201–211. [23] V. Terragni, G. Jahangirova, P. Tonella, and M. Pezzè, “Evolutionary improvement of assertion oracles,” in FSE, 2020, p. 1178–1189. [24] JUnit Team, “Junit 5: The next generation of junit,” https://junit.org/ junit5/, 2025, accessed: March 2026. [25] Mockito Framework, “Mockito: Tasty mocking framework for unit tests in java,” https://site.mockito.org/, 2025, accessed: March 2026. [26] V. Terragni, P. Salza, and M. Pezzè, “Measauring Software Testability Modulo Test Quality,” in ICPC, 2020. [27] PIT Mutation Testing, “Pit: State of the art mutation testing for java,” https://pitest.org/, 2025, accessed: March 2026. [28] JaCoCo Project, “Jacoco: Java code coverage library,” https://www.jacoco. org/jacoco/, 2025, accessed: March 2026. [29] G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in FSE, 2011, pp. 416–419. [30] S. Thummalapenta, T. Xie, N. Tillmann, J. de Halleux, and W. Schulte, “Mseqgen: object-oriented unit-test generation via mining source code,” in FSE 2009, 2009, pp. 193–202. [31] G. Fraser and A. Zeller, “Exploiting common object usage in test case generation,” in ICST 2011, 2011, pp. 80–89. [32] H. Zhu, L. Wei, M. Wen, Y. Liu, S.-C. Cheung, Q. Sheng, and C. Zhou, “Mocksniffer: Characterizing and recommending mocking decisions for unit tests,” in ASE, 2020, pp. 436–447. [33] H. Zhu, L. Wei, V. Terragni, Y. Liu, S.-C. Cheung, J. Wu, Q. Sheng, B. Zhang, and L. Song, “Stubcoder: Automated generation and repair of stub code for mock objects,” ACM TOSEM, vol. 33, no. 1, 2023. [34] D. Tiwari, M. Monperrus, and B. Baudry, “Mimicking production behavior with generated mocks,” IEEE TSE, 2024. [35] M. Li and M. Fazzini, “Automatically removing unnecessary stubbings from test suites,” in ICST. IEEE, 2024, pp. 233–244.