arXiv CSopen access

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

arXiv:2607.19682v1 [cs.SE] 22 Jul 2026

JUNJIE CHEN, Tianjin University, China ZIQI WANG, Tianjin University, China LIN YANG, Tianjin University, China CHEN YANG, Tianjin University, China XIAO CHU, Huawei Cloud, China JIANYI ZHOU, Huawei Cloud, China GUANGTAI LIANG, Huawei Cloud, China QIANXIANG WANG, Huawei Cloud, China DONG WANG∗ , Tianjin University, China Automated unit test generation has recently benefited from advances in large language models (LLMs), yet our industrial deployments reveal a persistent gap between promising research results and practical usability. In real-world projects with complex frameworks and cross-file dependencies, LLM-generated tests frequently fail to compile, require costly manual repair, or provide unstable coverage improvements. This paper reports our experience in designing, deploying, and evaluating CATGen, a context-aware workflow for LLM-based unit test generation, informed by repeated industrial failures and refinements. Rather than relying on LLMs to infer incomplete project context, we found that compilation robustness critically depends on making project-level dependencies explicit, stabilizing test class scaffolding, and replacing iterative LLM-based repair with lightweight static analysis. These experience-driven insights shaped CATGen’s multi-stage design, which combines structured context retrieval, deterministic test skeleton construction, and program analysis–based post-processing. We evaluate CATGen on real-world complex focal methods from proprietary industrial projects and additionally on the Defects4J benchmark to assess generalizability. Across both settings, CATGen substantially improves compilation success and structural coverage while significantly reducing generation time and token consumption compared to existing LLM-based approaches. Our results demonstrate that reliable LLM-based unit test generation in practice depends less on prompt engineering alone and more on systematic engineering support grounded in real-world development constraints. CCS Concepts: • Software and its engineering → Software testing and debugging; Software maintenance tools; • Computing methodologies → Natural language generation. Additional Key Words and Phrases: Unit Test Generation, Large Language Model, Code Context ACM Reference Format: Junjie Chen, Ziqi Wang, Lin Yang, Chen Yang, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, and Dong Wang. 2026. Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper). 1, 1 (July 2026), 23 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn ∗ Corresponding Author.

Authors’ Contact Information: Junjie Chen, Tianjin University, Tianjin, China, [email protected]; Ziqi Wang, Tianjin University, Tianjin, China, [email protected]; Lin Yang, Tianjin University, Tianjin, China, [email protected]; Chen Yang, Tianjin University, Tianjin, China, [email protected]; Xiao Chu, Huawei Cloud, Beijing, China, chuxiao1@ huawei.com; Jianyi Zhou, Huawei Cloud, Beijing, China, [email protected]; Guangtai Liang, Huawei Cloud, Beijing, China, [email protected]; Qianxiang Wang, Huawei Cloud, Beijing, China, [email protected]; Dong Wang, Tianjin University, Tianjin, China, [email protected]. 2026. ACM XXXX-XXXX/2026/7-ART https://doi.org/10.1145/nnnnnnn.nnnnnnn , Vol. 1, No. 1, Article . Publication date: July 2026.

2

1

J. Chen, Z. Wang, L. Yang et al.

Introduction

Unit testing is a cornerstone of software quality assurance, detecting bugs early by validating the functionality of each program unit [9, 37, 51, 57]. Manually writing high-quality unit tests for the focal method (a.k.a. the method under test) can be tedious and time-consuming [26]. To reduce this effort, numerous automated test generation methods have been developed over the past decades. Traditional methods often relied on random-based strategies [32], constraint-driven techniques [15, 46], and search-based approaches [11, 22, 23]. Deep learning-based methods have emerged [20, 29, 31], framing unit test generation as a neural machine translation problem. Despite their respective strengths, these methods still face limitations in fully understanding the code’s intent and generating syntactically correct and effective tests. Recently, Large Language Model (LLM)-based techniques have gained significant popularity and have shown potential in automated test generation. Several LLM-based methods have been proposed, including ChatTester [55], ChatUniTest [12], HITS [43], TELPA [48] and RATester [54]. These tools empirically demonstrate the strong capability of LLMs to generate high-coverage tests over traditional ones. For instance, ChatTester, which incorporates an initial test generator and an iterative test refiner, improved the statement coverage from EvoSuite’s 68.0% to 82.3%, evaluated on their datasets. HITS is specifically devised to decompose complex focal methods into slices for LLMs, proving its superiority in generating high-coverage unit tests. However, both prior empirical studies and our industrial experience reveal that compilation failures remain a major barrier to the practical adoption of LLM-based test generation. Large-scale evaluations [53] have shown that a significant portion of LLM-generated unit tests fail to compile, regardless of the prompting strategy used [38, 43]. In practice, such failures severely limit the usefulness of generated tests, as developers must first diagnose and repair compilation errors before any coverage or defect detection benefits can be realized. Through collaboration with a large global industrial partner and deployment of existing LLM-based test generation on real-world projects (detailed in Section 4), we systematically analyzed recurring failure cases. Our investigation suggests that insufficient and improperly utilized project-level context is the primary underlying challenge, which manifests in several recurring forms: • (I) Context mismatch in industrial project environments. Real-world software projects involve complex project-level dependencies, including testing frameworks, mocking libraries, and cross-file interactions, that are often poorly captured by existing methods. As a result, LLMs have to infer these dependencies without adequate context, frequently leading to incorrect imports, unresolved symbols, or incompatible framework usage. In our deployments, these mismatches frequently caused tests to fail at the first compilation, regardless of test logic quality. • (II) Fragility of test scaffolding. Constructing a correct test class skeleton, including imports, annotations, class declarations, mock definitions, and setup logic, requires strict adherence to framework-specific conventions and precise contextual information [53]. Yet existing approaches typically ask LLMs to generate it from scratch, where even the test logic was plausible, minor skeleton deviations (e.g., missing lifecycle hooks or mismatched mocking annotations) often invalidated the entire test class. • (III) Escalating cost of post-generation repair. To compensate for missing context, existing approaches often rely on iterative LLM-based refinement or post-generation repair [10, 28, 30, 33]. In our industrial deployments, these strategies frequently incurred substantial time and token overhead, while yielding limited improvements in compilation robustness. These limitations are magnified in large-scale industrial scenarios with tight release cycles, where manual correction of compilation errors is costly and undermines the productivity gains promised

, Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

3

by automated test generation. In such settings, effective utilization of project-level context becomes critical for practical adoption at scale. Motivated by these observations, this paper reports our experience in designing and applying a context-aware LLM-based unit test generation workflow, which we refer to as CATGen. Our goal is to distill a set of practice-driven design principles informed by repeated failure patterns observed in industrial usage, to advance the practical reliability and applicability of LLM-based unit test generation. The central insight is that compilation robustness improves when LLMs are systematically supported with explicit, structured context, instead of being asked to infer it implicitly. Specifically, CATGen is guided by three experience-derived principles centered on context acquisition and utilization via deterministic program analysis: (I) Instead of leaving LLMs to infer incomplete or missing project-level dependencies, CATGen explicitly retrieves contextual information from project structures and build configurations, ensuring that framework usage and external dependencies are accurately captured. (II) Rather than forcing LLMs to generate fragile test scaffolding code from scratch, CATGen constructs a valid test class skeleton using context-aware templates and systematic mocking strategies, allowing LLMs to focus exclusively on generating test logic. (III) To avoid costly iterative refinement, CATGen applies lightweight program analysis–based post-processing rules gained from developer feedback to deterministically repair common compilation errors and enhance test adequacy. To demonstrate the practical value and generalizability of the proposed approach, we evaluate CATGen in both industrial and open-source environments. In the industrial setting, we construct a benchmark of eight proprietary projects from our global industrial partner, covering diverse realworld scenarios such as advanced Java features, common design patterns, and framework-intensive systems. Together with partner engineers, we curate 183 focal methods jointly prioritized for maintenance risk (evolving logic) and regression risk (failure-prone interactions); the two notions overlap in practice but are not identical sets. These methods reflect substantial industrial complexity: 57.38% involve complex dependencies, and each interacts with 3.05 external files on average. All methods originate from actively maintained production code, representing realistic testing demands rather than artificially difficult cases. Moreover, because the codebases are proprietary, the benchmark avoids potential LLM data-leakage concerns common in open-source datasets. We compare CATGen against six representative or state-of-the-art baselines, including one searchbased technique and five LLM-based approaches, using compilation success rate, line coverage, branch coverage, and passing rate. The results show that CATGen consistently achieves substantially higher compilation success, with improvements ranging from 24.72%-38.05%, while also significantly improving coverage, yielding 17.27%–22.17% gains in line coverage and 15.31%–18.24% gains in branch coverage, together with reduced time and token consumption by 51.27%–69.00% in time and 66.83%–83.86% in token usage. In the open-source setting, we conduct experiments on Defects4J to examine whether the same design principles generalize beyond proprietary projects. CATGen exhibits a consistent performance trend, delivering 10.42%–14.33% improvements in compilation success over existing LLM-based approaches, while also achieving 6.11%–8.39% higher line coverage and 3.27%–10.56% higher branch coverage. Taken together, these findings highlight how practice-driven system design choices can substantially enhance the practicality of LLM-based unit test generation in industrial settings, while also demonstrating strong potential for adoption in open-source ecosystems. Contributions. This experience paper makes the following contributions: ❶ We identify key bottlenecks that limit automated unit test generation in industrial deployments based on real-world experience. To address them, we curate an industry-grounded benchmark with our partner and design CATGen, a context-aware workflow that integrates LLM generation with deterministic program analysis. ❷ We conduct an extensive evaluation in both industrial and open-source settings , Vol. 1, No. 1, Article . Publication date: July 2026.

4

J. Chen, Z. Wang, L. Yang et al.

to assess the effectiveness and efficiency of CATGen against state-of-the-art LLM-based approaches. The results show that CATGen consistently outperforms the baselines across all the evaluated metrics. ❸ We learned that practical LLM-based test generation depends less on prompt engineering alone and more on systematic engineering support: (i) explicitly retrieving project-level context instead of relying on the model to infer dependencies, (ii) constructing a stable test-class skeleton rather than generating fragile initialization code from scratch, and (iii) applying deterministic, analysis-driven post-processing to avoid costly iterative LLM repair loops. 2

Related Work

Automated unit test generation is the process of automatically creating test cases to validate code functionality [53]. Typically, it involves analyzing the source code to identify the focal method and then generating inputs and expected outputs to verify its behavior. Over the past decade, numerous automated unit test generation approaches have been proposed to reduce the manual effort required from developers. Traditional techniques include random-based strategies, searchbased approaches, and model checking. For example, EvoSuite [22], one of the most influential test generation techniques, employs an evolutionary algorithm to generate initial test cases and then iteratively refines them using mutation and selection strategies. To address the readability and maintainability challenges of traditional approaches, DL-based test generation techniques have been developed by framing test generation as a neural machine translation problem [41, 42]. For instance, Watson et al. [44] introduced ATLAS, which leverages neural machine translation to generate meaningful assert statements for test methods automatically. Similarly, Tufano et al. [40] proposed ATHENATEST, a BART Transformer-based approach that generates unit test cases by learning from real-world focal methods and developer-written test cases. Although DL–based techniques show promise, their effectiveness is constrained by the reliance on smaller, general-purpose pre-trained models, and they often struggle to ensure syntactic correctness and executability in complex project settings. LLMs, trained on extensive corpora, have demonstrated impressive capabilities in generating unit tests. Several ChatGPT-based techniques have been developed to leverage its powerful language understanding and code generation skills for generating effective unit tests. For example, Yuan et al. [55] introduced ChatTester, which iteratively generates unit tests through interactive conversations with ChatGPT. El Haji et al. [21] studied GitHub Copilot for Python test generation in developer workflows, and Lemieux et al. [27] combined search-based testing with pre-trained LLMs to escape coverage plateaus. Wang et al. [43] produced HITS by decomposing the focal methods into slices and asking the LLM to generate test cases slice by slice. TELPA [48], RATester [54], and WiseUT [50] further incorporate cross-file context, aiming to improve coverage for complex behaviors. Beyond coverage-oriented generation, RTED [49] improves type error detection through reflective test generation and type constraints, SEGA [47] targets business logic bugs by extracting business semantics from requirement documents, and CLAST [52] enhances the semantic clarity of generated tests. On the other hand, Yang et al. [53] conducted the first large-scale study to investigate multiple open-source LLMs, highlighting the impact of prompt design and model choice. Despite these advances, LLM-based approaches still face significant challenges in practical settings. In particular, while many methods report improvements in coverage or test quality, they often struggle to generate tests that reliably compile and execute in real-world projects. Compared to benchmark datasets (e.g., Defects4J), industrial projects involve more complex dependencies, framework constraints, and cross-file interactions, which frequently lead to compilation failures and unstable test behavior, limiting practical usability as developers must manually fix errors before tests can be executed. , Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

5

Fig. 1. Overview of the proposed CATGen

Beyond academic benchmarks, recent industrial deployments include Meta’s TestGen-LLM [10], Google’s BRT-Agent [13], and Mozilla’s BLAST [25]. Those efforts emphasize integrating LLMs into large-scale developer workflows and reporting productivity-oriented outcomes. They seldom foreground systematic analyses of why generated tests fail to compile in framework-heavy codebases. Our experience paper complements them by studying recurring compilation failures from proprietary deployments and encoding mitigations in CATGen through structured context retrieval, deterministic scaffolding, and analysis-driven repair aimed at executability as well as coverage. 3

Practice-Driven Workflow for LLM-Based Unit Test Generation

Figure 1 depicts the workflow we converged to after pilots on production repositories, iteratively tracing compilation and runtime failures back to missing context or scaffolding. Once recurring failure classes stabilized under explicit stages, we froze this layout. In deployment the limiting factor was seldom “more coverage on one method” but rather producing tests that compile and execute under frameworks, dependencies, and cross-file interactions—settings where generation without explicit project context frequently fails. We structure generation into four stages. Given a focal method, CATGen (1) retrieves project-level context including build configuration and relevant code elements, so that frameworks, imports, and cross-file calls are not left to model inference (Section 3.1); (2) constructs a test class skeleton tailored to the project’s testing/mocking setup, avoiding fragile scaffolding generation from scratch (Section 3.2); (3) generates test methods via skeleton-conditioned completion, where a fixed skeleton anchors the model output and reduces structural drift (Section 3.3); and (4) applies program analysis–based post-processing to deterministically repair common compilation issues and improve robustness without relying on costly iterative LLM refinement (Section 3.4). 3.1

Contextual Information Retrieval

In our early trials, we observed that prompting an LLM with the focal method alone was often insufficient for real-world projects. The generated tests frequently overlooked framework-specific conventions (e.g., annotations and imports), used incompatible mocking APIs, or invoked external methods without matching signatures. These issues typically surfaced as compilation failures and made downstream repair expensive. We therefore converged on a lightweight retrieval step that makes project-level dependencies and cross-file behaviors explicit, rather than leaving them to the model’s inference. Specifically, CATGen collects five complementary types of context: • Inner Class Context. Structural and semantic elements of the focal class, including imports, constructors, fields, and public methods. This helps the model build valid instances, manage object state, and trigger intended logic paths. , Vol. 1, No. 1, Article . Publication date: July 2026.

6

J. Chen, Z. Wang, L. Yang et al.

• Focal Method Context. Details of the focal method, including its parameter/return types and implementation. This helps reduce type mismatches and semantically invalid invocations, which in turn lowers the likelihood of compilation errors and common runtime issues. • External Method Call Information. For methods invoked by the focal method, CATGen extracts their signatures, types, return values, and method bodies. This provides the behavioral and typing constraints needed for consistent stubbing and for reasoning about reachable paths. • Testing Framework. The project’s testing framework determines the correct annotations and assertion conventions. CATGen parses build configuration files (e.g., pom.xml) to detect framework keywords such as JUnit 5. • Mocking Framework. The mocking library (e.g., Mockito) is essential for dependency isolation and avoiding mix-and-match errors, also evident by prior studies [39, 58]. Similar to the testing framework, CATGen infers mocking frameworks by parsing build configuration files. To materialize the five context types, CATGen first scans project build descriptors for declared testing and mocking artifacts, then parses focal files into ASTs using PSI-compatible structures so signatures, fields, and intra-class members remain faithful to the workspace view. For outgoing calls from the focal method, CATGen traverses call expressions to resolve targets where symbol resolution succeeds; it records visibility and modifiers because they constrain whether a dependency must be mocked, injected, or exercised via reflection later. Unresolved or library-only symbols are retained conservatively so that skeleton imports and mock hooks remain aligned with what the compiler must see. Taken together, this retrieved context grounds skeleton construction and post-processing; extraction uses lightweight static analysis. 3.2

Context-Aware Test Class Skeleton Construction

In industrial projects, we found that the test class skeleton is often a decisive factor for usability, as it establishes the execution environment for all test methods. Even minor omissions, such as missing imports, incorrect framework annotations, or incomplete initialization, can invalidate the entire test class, regardless of whether the generated assertions themselves are reasonable. During our early attempts, when LLMs were asked to generate the skeleton from scratch, they frequently hallucinated framework-specific patterns or dependency configurations. Based on these observations, we treat the skeleton as a construction task rather than a free-form generation problem, and instead build it deterministically using the retrieved project context. CATGen constructs a reliable skeleton via a framework-to-template mapping. Each template encodes the framework-required annotations, lifecycle methods, and configuration patterns, curated from official documentation, so that the skeleton consistently conforms to framework conventions. As illustrated in Figure 2a, we use the focal method RestTemplateService.postForEntry(String, String, String) from our industrial benchmark as a running example. This method interacts with external dependencies and includes error-handling logic, making it representative of cases where scaffolding mistakes are common. Figure 2b shows the corresponding skeleton constructed by CATGen, which consists of three parts: Test Class Configuration. Skeleton templates abstract framework-level structural patterns (imports, lifecycle hooks, and mocking APIs) instantiated per detected stack. The configuration establishes the structural and annotative foundation of the test class and ensures compatibility with the selected testing and mocking frameworks. CATGen derives the test class name by appending “Test” to the focal class name, and sets visibility according to framework conventions (e.g., public under JUnit 5) to support test discovery. It then adds class-level annotations based on detected dependencies: for Mockito-based tests, @ExtendWith(MockitoExtension.class) enables @Mock and @InjectMocks; for PowerMock-based tests, when static mocking is required, declaring classes are added to @PrepareForTest. At the field level, Mockito templates introduce @Mock for dependencies , Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

7

(a) Focal method

(b) Test class skeleton Fig. 2. The illustrative focal method, along with its corresponding test class skeleton generated by CATGen

and @InjectMocks for wiring them into the focal object. Finally, lifecycle hooks (e.g., @BeforeEach, @AfterEach) are added to standardize setup/teardown for isolation and repeatability. We provide detailed construction rules in our replication package. The green region in Figure 2b illustrates the resulting configuration. Test Context Initialization. Beyond configuration, the skeleton must correctly initialize the focal object and its dependencies so that generated test methods can execute reliably. CATGen constructs the test context by mapping focal-class member variables to test-class fields and ensuring constructors are invoked or configured appropriately. In the example (blue region in Figure 2b), using JUnit 5 and Mockito, CATGen declares the focal class instance as a field annotated with @InjectMocks (lines 13–14), and introduces @Mock-annotated fields for dependencies (line 11). A @BeforeEach method (lines 16–19) initializes mocks and sets non-mockable or configurationdependent fields via reflection. An @AfterEach method (line 21) is also generated to keep the skeleton complete and facilitate cleanup. Import Statements. Scaffold imports are assembled deterministically from three sources: (i) framework imports implied by the detected testing/mocking stack; (ii) standard assertion and utility imports for the chosen framework; and (iii) dependency imports resolved from focal-method and focal-class types via static analysis and classpath information, rather than being inferred by the LLM (Figure 2b, yellow). When resolution is ambiguous we conservatively prefer project-local and framework-consistent symbols; imports for types introduced only in LLM-completed bodies are supplemented or corrected in Section 3.4. , Vol. 1, No. 1, Article . Publication date: July 2026.

8

3.3

J. Chen, Z. Wang, L. Yang et al.

Fig. 3. Test Class Generated by CATGen

Skeleton-Conditioned Completion for Test Generation

A common LLM-based test generation workflow follows a dialog-style paradigm: users provide a natural language query, and the model outputs candidate tests [12, 43, 55]. While straightforward, our experience suggests that this interaction pattern is fragile in practice. In a pilot study, we observed that even when a pre-constructed test skeleton is provided, LLMs (including advanced models such as GPT-4) often drift from the given structure, producing incomplete or invalid test cases. This drift frequently manifests as missing required boilerplate, redefining class-level elements inconsistently, or returning partial snippets that are hard to compile and integrate. To mitigate these issues, we reformulate test generation as a skeleton-conditioned completion task rather than generating an entire test class from scratch. Concretely, we decompose the input into two complementary components: Contextual Prompt. The prompt provides structured guidance for generating test methods. It includes class-level information (class name, constructors, member fields) to support instantiation and state management, followed by the focal method implementation and related methods to enable behavior reasoning. It also specifies the detected testing and mocking frameworks and instructs consistent handling of static and non-static dependencies. Finally, the prompt requires the model to analyze the branching structure of the focal method before writing tests to encourage systematic exploration of control flow and improve branch coverage. Prefilled Content. We prefill the response with the test class skeleton constructed in Section 3.2. The model is then asked to complete the remaining parts under this fixed scaffold: it writes test methods, configures mock behaviors, and inserts assertions while preserving the skeleton’s structure. In our experience, anchoring generation in this way reduces syntactic errors and structural drift, and increases the likelihood that the output is directly compilable. The full prompt is provided in our replication package. 3.4

Program Analysis-Based Post-Processing

LLM-generated tests often require post-processing before they become usable, and a common strategy is to iteratively re-prompt the model with compiler feedback [12]. In our early deployments, we adopted the same feedback-driven repair loop and ran multiple LLM repair rounds to fix compilation errors. However, this approach proved brittle and cost-unstable in practice: when hallucinations persisted, or project-level dependencies were only partially captured, similar errors , Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

9

repeatedly resurfaced across iterations, while token usage and latency increased rapidly as the loop continued. To make repair predictable and cost-stable, CATGen instead adopts deterministic, lightweight post-processing grounded in program analysis and the retrieved project context. The process consists of two stages: Static Compilation Error Repair. We observe three dominant categories of compilation failures in the LLM-completed test bodies: (I) unresolved dependencies, including missing imports, incorrect package qualifiers, or ambiguous simple names; (II) references to undefined types, methods, or variables; and (III) incorrect initialization or framework usage (e.g., Mockito setup or constructor misuse). CATGen implements a lightweight repair pipeline on top of the project context from Section 3.1 and compiler-visible facts from AST inspection, so that repairs are grounded in the same dependency and signature information as skeleton construction, rather than in additional LLM rounds. Following empirical observations of industrial developer repair workflows and insights from prior work [55], we implement a suite of targeted strategies: (1) Package Declaration Completion, which aligns the test class package with the focal method’s source location; (2) Import Statement Supplementation, which resolves simple names and static members against the focal compilation unit, neighboring files in the same module, and the classpath slice from context retrieval, and inserts or adjusts import declarations when resolution is unique; (3) Class Annotation Rectification, which corrects JUnit and Mockito annotation configurations; (4) Invalid Reference Resolution, which removes or rewrites references that cannot be bound to symbols visible through static analysis; (5) Private Member Access Adaptation, which introduces reflection-based access when private members must be exercised; (6) Method Signature Alignment, which reconciles method names and parameter or return types with their actual signatures; (7) Exception Specification Enhancement, which adds explicit exception declarations or assertions where required for compilation or consistent exception handling; and (8) Fallback Assertion Mechanism, which introduces default assertions so that test methods retain minimal behavioral checks. All eight strategies share the same backbone: each rule is driven by compiler diagnostics and lightweight AST inspection over the project context from Section 3.1, fires in a fixed precedence order, and never re-invokes the LLM, which keeps repeated runs deterministic and cost-bounded. Building on this backbone, the three failure categories are dispatched to complementary rules. For (I), import supplementation extends the scaffold imports of Section 3.2 to types that surface only in LLM-completed bodies; when a binding is ambiguous it conservatively prefers project-local, framework-consistent symbols and otherwise hands the case off to invalid-reference resolution rather than inventing packages. Building on this, category (II) is resolved by the same invalidreference resolution working together with method signature alignment, which together reconcile unbound calls and mismatched names or parameter/return types against the signatures recovered from the focal class and its dependencies. Finally, category (III) is addressed by class annotation rectification, private member access adaptation, exception specification enhancement, and the fallback assertion mechanism, which repair JUnit/Mockito scaffolding, private-access patterns, and throws/assertion structure so that the merged class compiles and exhibits minimally consistent behavior. Overall, this stage is deliberately limited to restoring compilability and structural consistency; broader test adequacy is evaluated via coverage and mutation analyses later in the paper, and the complete rule ordering with worked examples is provided in the replication package. Program Analysis-Based Coverage Enhancement. Even when tests compile, they may miss corner cases. To complement this gap without additional LLM calls, CATGen uses static analysis to identify critical decision points such as null checks, empty-string validations, and explicit exception throws. For each uncovered condition, CATGen synthesizes an additional test following the Given–When–Then paradigm [43, 55]: in the Given phase, it analyzes exception-handling , Vol. 1, No. 1, Article . Publication date: July 2026.

10

J. Chen, Z. Wang, L. Yang et al.

Table 1. Statistics of the evaluation dataset. Scenario

# Focal Methods % Complex Dep. Avg. Dep. Files

Service Orchestration Rules Data Access Persistence Configuration Dependency Wiring Framework Entry Data Transformation Mapping Collection Stream Processing Shared Utilities Libraries Microservice Integration Infrastructure

43 38 12 42 16 11 12 9

34.88 78.95 58.33 52.38 50.00 54.55 66.67 100.00

1.05 4.13 3.42 3.36 3.81 2.45 3.17 5.44

Total

183

57.38

3.05

structures and external dependencies and simulates them via mocking; in the When phase, it injects boundary/invalid inputs (e.g., null or empty strings) to trigger the targeted path; and in the Then phase, it generates value-based and exception-aware assertions (e.g., assertTrue, assertThrows) to validate the expected behavior. Figure 3 illustrates an example where LLM-generated tests miss a null-input scenario, and the enhancement module automatically adds a targeted test to exercise and validate the corresponding exception handling. Finally, CATGen merges analysis-driven tests into the LLM-completed class. A deterministic consolidation step resolves colliding @Test names (suffixing enhancement-only methods when needed), removes near-duplicate bodies, and orders methods for readability; static repair then reconciles imports and signatures on the merged class. 4 4.1

Industrial Evaluation Experimental Design

Industrial Benchmark. To ensure that our evaluation reflects the challenges we encountered in production settings, we construct the benchmark in close collaboration with our industrial partner, capturing the failure modes observed when deploying LLM-generated tests in real repositories. In these codebases, a focal method is rarely self-contained. Its behavior is often shaped by dependency injection and configuration wiring, by contracts that are distributed across files and modules, and by framework managed execution paths where correctness depends on conventions and implicit resources. We also saw many methods with non-linear control and data flow, such as nested conditionals, early returns, and chained transformations, where meaningful branches are difficult to reach with simple happy path inputs. When this surrounding context is missing or only partially inferred, the generated tests tend to either fail to compile, set up mocks inconsistently, or run but exercise little of the intended behavior. These observations are why we treat compilation success and cost stability as first order concerns in this benchmark, not secondary metrics. To capture these practical realities, we curate the benchmark together with engineers from our industrial partner using actively maintained and production-deployed internal Java projects. The projects span multiple architectural layers, including core service logic, data access code, configuration and wiring components, web controllers, shared utilities, and microservice infrastructure. We use project-level proportional stratified sampling to reflect both scale and diversity, and we intentionally bias selection toward difficult methods with deep call chains, cross-module interactions, and framework-managed resources. The benchmark totals 183 focal methods (Table 1): 57.38% have complex dependencies and 3.05 dependent files on average. We count distinct external classes and methods invoked by the focal method as a coupling-oriented proxy aligned with coupling between objects (CBO) [14], and treat ≥ 3 such dependencies as complex dependencies, reflecting dependency-heavy cases where our deployments saw frequent compilation failures. The Microservice Integration Infrastructure scenario is the hardest subset (100%; 5.44 files on average). , Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

11

Compared Techniques. To demonstrate the effectiveness of the proposed workflow, we comprehensively adopt six representative or state-of-the-art test generation techniques as baselines: • EvoSuite [22]. A traditional search-based tool that generates JUnit tests via evolutionary algorithms; it iteratively executes and mutates candidate tests to maximize structural coverage (e.g., line and branch) without using LLMs. • ChatTester [55]. It includes an initial test generator and an iterative test refiner. The initial test generator first leverages an LLM to understand the focal method and generate a test, and the refiner then iteratively fixes the compilation errors. • ChatUniTest [12]. It generates unit tests using an LLM by leveraging the focal method and context extracted through predefined rules as inputs. For tests that fail execution, it utilizes JVM error reports to guide the LLM in correcting them. • HITS [43]. It first decomposes the focal method into slices and creates unit tests for each slice, designed to cover all lines and branches. These unit tests collectively form the initial test suite. For the non-executable test suite, HITS includes a fixer to repair them. • TELPA [48]. It enhances LLM-based test generation with program analysis to target uncovered branches. TELPA extracts object construction sequences and branch-relevant dependencies, and iteratively guides the LLM with coverage feedback. • RATester [54]. It improves repository awareness by querying a language server for symbol definitions and usages. The retrieved global context is injected into the prompt to reduce hallucinations and signature mismatches. For all LLM-based baselines, we use their publicly available open-source implementations. Because some original studies instantiated their methods with proprietary models, we standardize all LLM calls to the same set of open-source models to ensure a fair comparison. This standardization is applied consistently wherever an LLM is involved, including both unit test generation and subsequent repair steps. RATester was originally implemented for Go. To maintain a consistent Java-based evaluation environment, we follow the authors’ Java adaptation and re-implement RATester accordingly, without modifying its underlying algorithmic design. Evaluation Metrics. We evaluated the performance of CATGen and baselines using four commonly adopted metrics: Compilation Success Rate (CSR), Line Coverage (CovL), Branch Coverage (CovB), and Pass Rate (PR). CSR is determined by the ratio of test methods that compile successfully to the total number of test methods generated for all focal methods. CovL is calculated as the number of covered lines divided by the total number of executable lines in the source code, while CovB is defined as the number of covered branches divided by the total number of branches in the control flow graph. PR adapts this metric by using the count of test methods that pass execution as the numerator. For a fair comparison, CATGen and all baselines are required to generate a single test class, but multiple test methods are allowed within that class. For each test class that compiles and executes successfully, we employ the Jacoco tool1 to measure its line and branch coverage. Implementation and Environment. CATGen is implemented in Java. For program structure analysis, it uses PsiMethod from IntelliJ’s PSI to parse ASTs, enabling precise extraction of method signatures, control-flow dependencies, and inter-file relationships. The workflow depends only on AST-level facts; any conforming extractor supplying the same structural signals can substitute. The context-aware skeleton construction component employs standardized templates for JUnit 4 [1]/5 [2], Spock [5], and Spring Boot Test [6], with Mockito [3] and PowerMock [4] for mocking. From an engineering perspective, our goal is to evaluate CATGen under practically deployable open-source LLMs rather than exhaustively benchmark all available models. We therefore select 1 https://github.com/jacoco/jacoco

, Vol. 1, No. 1, Article . Publication date: July 2026.

12

J. Chen, Z. Wang, L. Yang et al.

model families that (i) are publicly accessible and reproducible, (ii) provide instruction-tuned checkpoints suitable for interactive code generation, and (iii) cover both general-purpose and code-specialized variants across multiple parameter scales. We evaluate models from the Llama, DeepSeek, and Qwen families, spanning both general and code-specialized models across multiple scales: CodeLlama-7B (CL-7B) [7], Llama3.1-8B (Lla-8B) [8], DeepSeekCoder-6.7B-Instruct (DC-7B) [16], DeepSeekCoder-33B-Instruct (DC-33B) [17], DeepSeek-R1-Distill-Llama-8B (DRL-8B) [18], DeepSeek-R1-Distill-Qwen-32B (DRQ-32B) [19], Qwen2.5-Coder-7B-Instruct (QC-7B) [36], Qwen2.5-Coder-32B-Instruct (QC-32B) [35], and Qwen2.5-32B (Q-32B) [34]. CATGen is deployed in a hybrid execution environment consisting of 4 NVIDIA A100-PCIE-40GB GPUs (driver version 535.161.08, CUDA 12.2) and an Intel Xeon Gold 6330 CPU. The software stack includes JDK 11.0.8, Gradle 7.4, and Maven 3.3.9 for managing Java modules, with JaCoCo 0.8.11 integrated for code coverage analysis. Python 3.10.12 is used for LLM-related tasks, and LLM inference is performed using vLLM 0.8.4 [56] for efficient and scalable deployment. All experiments use a zero-temperature setting to ensure deterministic outputs. For the EvoSuite baseline, we allocate a search budget of 300 s per focal method [22]. 4.2

Evaluation Results

I. Effectiveness of CATGen in industrial setting. To mitigate stochasticity in LLM operations and guarantee result reliability, we independently ran all LLM-based test generation processes five times with the same configuration and used the per-method average as the final result for each metric. To assess the robustness of the observed improvements, we use Wilcoxon signed-rank tests [45] on paired per-method outcomes after averaging the five repetitions. Finding I: CATGen substantially improves executability, while also delivering higher structural coverage than existing methods in the industrial scenarios. From our experience, a critical requirement in industrial projects is whether generated tests can reliably compile and execute under project-specific configurations, as this directly determines how much behavior the tests can meaningfully exercise and validate. As shown in Table 2, for compilation success rate, CATGen reaches 91.83%, while EvoSuite reaches 75.80%, and the LLM-based baselines achieve 55.88% for ChatTester, 51.70% for ChatUniTest, 51.02% for HITS, 64.11% for TELPA, and 67.11% for RATester. EvoSuite occasionally exhausts its budget without emitting a test on highly coupled focal methods, which we count as compilation failures and which partly explains its lower compilation success rate [22]. For coverage, CATGen reaches 70.10% on line coverage and 63.92% on branch coverage. RATester achieves 48.43% and 46.33%, TELPA achieves 45.99% and 44.32%, and EvoSuite achieves 39.55% and 34.56%. ChatTester achieves 30.76% and 29.75%, ChatUniTest achieves 29.77% and 26.74%, and HITS achieves 36.10% and 36.33%. For pass rate, CATGen reaches 54.63%, while EvoSuite reaches 44.21%, RATester reaches 41.71%, TELPA reaches 39.91%, ChatTester reaches 35.85%, HITS reaches 32.93%, and ChatUniTest reaches 30.24%. Notably, among the LLM-based baselines, TELPA and RATester tend to achieve stronger overall results, which aligns with their designs that explicitly incorporate broader project context and cross-file information when generating tests. The statistical testing results confirm that CATGen consistently outperforms all baselines, with p-values < 0.005 for CSR, CovL, CovB, and PR in all pairwise comparisons. We report unadjusted 𝑝-values for pairwise comparisons; given the consistently small 𝑝-values observed across all subjects, the statistical significance of our results remains robust. As an illustrative pairwise contrast for Wilcoxon reporting, for CovL under QC-32B versus RATester, we obtain 𝑛=183, 𝑊 =1148, 𝑝=0.0026, and 𝑟 =0.339 (medium effect). Finding II: CATGen ensures robust performance across LLMs of varying scales and architectures, with larger models potentially yielding better results. As shown in Table 2, the underlying LLM choice significantly impacts all test generation methods: larger models , Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

13

Table 2. Effectiveness comparison among different models and different methods. Metric Method

CSR

CovL

N/A 32.62% 33.50%

N/A 66.92% 63.53%

N/A N/A 73.03% 62.37% 71.06% 46.09%

N/A 75.00% 71.26%

N/A 79.23% 70.04%

N/A 80.00% 71.48%

HITS TELPA RATester CATGen

51.02% 12.61% 10.01% 64.11% 45.72% 18.94% 67.11% 50.83% 22.37% 91.83% 90.20% 82.73%

27.31% 41.67% 45.94% 84.26%

58.93% 71.45% 75.36% 92.18%

67.68% 78.32% 80.45% 95.61%

61.83% 74.88% 78.12% 95.33%

67.95% 80.14% 83.27% 96.28%

74.60% 82.96% 81.14% 94.91%

78.22% 82.94% 86.55% 94.97%

EvoSuite 39.55% N/A N/A ChatTester 30.76% 17.19% 2.51% ChatUniTest 29.77% 24.75% 3.82% HITS 36.10% 32.89% 5.72%

N/A 13.92% 12.99% 18.75%

N/A 40.10% 35.51% 42.24%

N/A N/A 48.20% 36.76% 45.92% 39.50% 54.47% 43.89%

N/A 43.27% 46.91% 46.67%

N/A 36.93% 29.19% 36.35%

N/A 37.97% 29.35% 43.95%

45.99% 41.20% 12.35% 27.64% 53.83% 60.48% 57.31% 48.43% 45.73% 15.48% 30.52% 49.36% 62.47% 59.81% 70.10% 69.14% 57.02% 63.60% 71.05% 75.79% 71.68%

58.94% 63.87% 75.49%

49.18% 53.02% 52.96% 55.68% 73.15% 74.01%

N/A 32.00% 31.47% 48.02% 55.81%

N/A 47.88% 49.14% 57.44% 60.92%

N/A 32.18% 27.49% 39.48% 48.36%

46.33% 31.67% 12.74% 24.85% 59.32% 59.54% 58.69% 63.92% 62.13% 47.56% 52.26% 66.54% 73.73% 63.44%

65.12% 70.27%

51.44% 53.61% 69.32% 70.06%

N/A 39.19% 31.01% 48.74% 49.88% 54.32%

N/A 52.93% 52.72% 51.53% 56.32% 54.12%

N/A 40.32% 36.96% 38.12% 47.05% 49.87%

54.63% 47.00% 34.71% 37.62% 50.54% 67.77% 64.53%

69.36%

58.34% 61.77%

EvoSuite 34.56% ChatTester 29.75% ChatUniTest 26.74% HITS 36.33% TELPA 44.32% RATester CATGen

PR

CL-7B Lla-8B DRL-8B QC-7B QC-32B Q-32B DRQ-32B DC-7B DC-33B

EvoSuite 75.80% N/A N/A ChatTester 55.88% 29.17% 4.60% ChatUniTest 51.70% 32.92% 5.41%

TELPA RATester CATGen

CovB

Avg

EvoSuite 44.21% ChatTester 35.85% ChatUniTest 30.24% HITS 32.93% TELPA 39.91% RATester 41.71% CATGen

N/A 10.20% 14.92% 16.61% 28.45%

N/A 29.27% 14.22% 11.67% 25.48% 29.76%

N/A 2.52% 3.25% 5.12% 10.32%

N/A 2.27% 3.41% 3.01% 12.92% 10.21%

N/A 13.16% 10.20% 12.27% 25.78%

N/A 16.88% 11.01% 16.50% 22.87% 25.34%

N/A 45.67% 31.51% 50.51% 56.14%

N/A 46.42% 33.54% 35.17% 41.93% 42.25%

N/A 49.60% 46.78% 56.45% 62.83%

N/A 51.57% 45.96% 46.55% 52.41% 55.93%

N/A 34.58% 25.93% 41.04% 50.27%

N/A 43.78% 43.33% 45.07% 50.36% 53.61%

(e.g., 32B–33B) consistently outperform smaller ones like Lla-8B. Among model families, Qwen variants lead overall: QC-32B achieves the highest coverage (CovL: 75.79%, CovB: 73.73%), while DRQ-32B attains the best compilation success rate (96.29%) and pass rate (62.36%). DeepSeek models follow closely, whereas Llama-family models, especially Lla-8B, perform poorly under baselines. On the other hand, results show that code-specialized variants tend to outperform their general-purpose counterparts at the same scale. For instance, QC-32B outperforms Q-32B, and models enhanced via reinforcement learning-based knowledge distillation (e.g., DRQ-32B) further exceed their general counterparts. These findings underscore the significance of domain-specific pretraining and distillation-based capability transfer for generating high-quality tests. Importantly, CATGen maintains consistent superiority across all LLM configurations. This robustness makes it particularly valuable for cost-sensitive or resource-constrained scenarios. Even with a 7B-scale model like DC-7B, CATGen outperforms all baselines across all model sizes, offering a compelling balance between efficiency and effectiveness. , Vol. 1, No. 1, Article . Publication date: July 2026.

14

J. Chen, Z. Wang, L. Yang et al.

Table 3. Effectiveness comparison between CATGen and its variants. Method CATGen Δw/o skeleton Δw/o repair Δw/o enhancement Δw/o all

CSR

CovL

CovB

PR

95.61% -3.04% -16.79% -1.96% -38.88%

75.79% -16.37% -8.95% -10.51% -30.63%

73.73% -10.28% -6.22% -9.51% -24.00%

57.77% -8.86% -8.76% -0.13% -15.98%

II. Contributions of Components in CATGen. We conducted an ablation study to examine the contribution of each core component (i.e., context-aware test class skeleton construction, static compilation error repair, and program analysis-based test enhancement) to the overall effectiveness of CATGen, with the program analysis-based post-processing module evaluated via its two subcomponents separately. Accordingly, we constructed four variants of CATGen: • “w/o skeleton” (without the context-aware test class skeleton construction): This variant omits the rule-based test class skeleton from the prompt, instructing the LLM to generate test code from scratch without any structural guidance. • “w/o repair” (without the static compilation error repair): This variant disables the static repair stage and applies only the program analysis-based test enhancement to the raw test class generated by the LLM. • “w/o enhancement” (without the program analysis-based test coverage enhancement): This variant disables the test coverage enhancement component, retaining test class skeleton construction and static repair, but excluding any analysis-driven enhancements. • “w/o all”: This variant removes all three core components, relying solely on standard promptbased generation without any additional post-processing or enhancement. Based on the RQ1 results, we select QC-32B as the representative LLM for our ablation study, given its superior line coverage and relatively high compilation success rate, which demonstrate its overall effectiveness. To mitigate random variation, we repeat each experiment five times and report the average results across all evaluation metrics. Finding III: Each core component of CATGen serves a unique and critical function in achieving the framework’s overall performance. Table 3 presents the effectiveness comparison between CATGen and its four ablated variants. First of all, removing the context-aware test class skeleton construction (“w/o skeleton”) led to a substantial performance drop: line coverage (CovL) decreased by 16.37%, branch coverage (CovB) by 10.28%, compilation success rate (CSR) by 3.04%, and passing rate (PR) by 8.86%, compared to the full version. This highlights the critical role of the structured test class skeleton in guiding the LLM. Without it, the model is more prone to hallucination, failing to correctly construct class declarations, mock annotations, or instantiate focal classes, resulting in incomplete or invalid test classes. Interestingly, while coverage and correctness decline, the relatively stable CSR indicates that program analysis-based post-processing remains effective in preserving compilability, even when initialization is missing. Second, removing the static compilation error repair stage (“w/o repair”) caused a moderate drop in coverage (CovL: –8.95%, CovB: –6.22%) but a much steeper decline in CSR (–16.79%) and PR (–8.76%). This discrepancy underscores the importance of static repair in addressing syntactic and structural errors that LLMs frequently produce. As this variant retains the other components, including test enhancement, the observed decrease primarily stems from compilability issues rather than changes in the generation workflow. Third, disabling the program analysis-based enhancement (“w/o enhancement”) resulted in a 10.51% and 9.51% drop in CovL and CovB, respectively—comparable to the loss observed when test class initialization is removed. However, CSR and PR remained largely stable (–1.96% and –0.13%), , Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

15

Table 4. Efficiency comparison between CATGen and baselines. Method EvoSuite ChatTester ChatUniTest HITS TELPA RATester CATGen

Gen.Time Gen.Tokens Post.Time Post.Tokens Total.Time Total.Tokens 10,980s 2,805s 1,601s 2,972s 4,386s 5,217s 1,759s

0k 432k 234k 459k 612k 735k 203k

0s 1,666s 2,166s 2,951s 0s 0s 77s

0k 459k 857k 799k 0k 0k 0k

10,980s 4,471s 3,768s 5,924s 4,386s 5,217s 1,836s

0k 891k 1,091k 1,258k 612k 735k 203k

suggesting that the generated tests, although less diverse, remain syntactically and semantically valid. This validates that the enhancement module is instrumental in enriching test diversity and uncovering edge-case behaviors. Finally, the full ablation scenario (“w/o all”)—which removes all three components—resulted in the most severe degradation. CSR and CovL dropped by 38.88% and 30.06%, respectively, roughly equating to the cumulative impact of removing individual components. These results emphasize the complementary roles of CATGen’s core modules: the test class skeleton ensures syntactic structure, static repair improves correctness and compilability, and program analysis-based enhancements enhance coverage through targeted test generation. Collectively, they form a cohesive, multi-stage framework that substantially improves both the quality and reliability of LLM-generated unit tests. III. Efficiency of CATGen. We evaluate two key metrics to assess computational efficiency and resource utilization: execution time (Time) and token consumption (Tokens). These metrics are measured across the generation phase (Gen), the post-processing phase (Post), and their aggregate total (Total). Following the setup in RQ2, we select QC-32B for validation, as it achieved the best overall performance among the evaluated LLMs. Each experiment is repeated five times to mitigate variability, and the average results are reported. Finding IV: CATGen achieves substantial efficiency gains over all baselines in both execution time and token consumption, across both the generation and post-processing phases. Table 4 summarizes the results for efficiency-related metrics. In terms of time efficiency, CATGen completes the full pipeline in 1,836 seconds (the sum of generation and post-processing time), outperforming ChatUniTest (3,768s, –51.27%), ChatTester (4,471s, –58.93%), and HITS (5,924s, –69.00%). It also substantially outperforms TELPA (4,386s, –58.13%) and RATester (5,217s, –64.80%), and is markedly faster than the traditional baseline EvoSuite (10,980s, –83.28%). These improvements stem from two key design choices: (1) a single-round LLM generation strategy (1,759s), which avoids the costly iterative refinement loops used by baselines; and (2) a program analysis-based post-processing module that requires only 77 seconds, more than one order of magnitude faster than the LLM-driven repair stages in ChatTester (1,666s), ChatUniTest (2,166s), and HITS (2,951s). In contrast, TELPA and RATester rely on multi-round LLM generation without an explicit repair phase, leading to high cumulative latency due to repeated model invocations, while EvoSuite spends most of its time on evolutionary search and repeated test executions rather than model inference. CATGen consumes only 203k tokens in total, with zero tokens used in post-processing. This represents a 77.22% reduction compared to ChatTester (891k) and an 83.86% reduction compared to HITS (1,258k). Even ChatUniTest, despite its relatively fast generation time, incurs 1,091k tokens due to heavy reliance on LLM-based post-processing. Compared with TELPA (612k) and RATester (735k), CATGen also achieves substantial token savings, as their multi-round generation strategy requires significantly more LLM interactions. Collectively, these results demonstrate that CATGen’s decoupled architecture, which separates test generation from LLM-dependent post-processing, , Vol. 1, No. 1, Article . Publication date: July 2026.

16

J. Chen, Z. Wang, L. Yang et al.

Table 5. Effectiveness on Defects4J. Metric

Project

EvoSuite

ChatTester

ChatUniTest

HITS

TELPA

RATester

CATGen

CovL

Chart Math Lang Time

48.31% 60.21% 71.32% 69.91%

54.24% 47.89% 80.47% 71.33%

59.10% 51.81% 79.12% 70.21%

62.14% 54.42% 83.52% 75.31%

64.50% 64.56% 81.21% 76.24%

67.53% 67.13% 80.42% 76.31%

75.71% 73.32% 91.32% 82.21%

CovB

Chart Math Lang Time

43.51% 45.17% 62.89% 60.74%

48.14% 38.54% 74.21% 65.33%

51.28% 40.78% 76.32% 67.32%

60.47% 48.21% 77.21% 69.24%

60.74% 61.54% 80.21% 75.87%

62.43% 54.32% 78.21% 70.32%

72.89% 64.74% 84.25% 80.45%

CSR

Chart Math Lang Time

98.21% 97.24% 100.00% 97.21%

64.32% 59.21% 71.54% 69.54%

71.21% 65.78% 76.20% 70.24%

75.41% 67.89% 82.65% 77.54%

76.54% 70.12% 83.41% 75.69%

80.12% 77.32% 82.12% 80.23%

90.45% 91.57% 94.45% 93.14%

PR

Chart Math Lang Time

89.45% 90.21% 89.12% 90.42%

41.23% 44.89% 58.54% 60.87%

59.10% 51.81% 65.32% 59.45%

62.14% 54.42% 69.54% 70.24%

64.50% 64.56% 74.56% 72.21%

60.34% 65.21% 75.32% 72.56%

78.12% 75.78% 83.24% 81.45%

enables deterministic, scalable, and resource-efficient unit test synthesis. This design is particularly well-suited for industrial deployment, where latency and compute cost are critical constraints. 5

Open-Source Software Evaluation

To assess the generalizability of CATGen beyond industrial codebases, we additionally evaluate it on the widely used open-source benchmark Defects4J [24]. Following prior studies [51], we select four representative projects, Chart, Lang, Time, and Math, which span diverse domains such as chart rendering, language utilities, date-time processing, and numerical computation. We reuse the same focal-method evaluation protocol, comparison techniques, metrics, and experimental configuration as in the industrial evaluation. To ensure a controlled comparison in the OSS setting, we instantiate CATGen and all LLM-based baselines with a single underlying model, Qwen2.5-Coder-32B-Instruct (QC-32B), for all LLM invocations. All other factors, including prompts, inference settings, compilation and execution pipelines, and cost accounting, are kept identical. This setup isolates the effect of the benchmark itself, ensuring that any observed differences arise from dataset characteristics rather than changes in experimental configuration. To further evaluate fault-detection capability, we report Mutation Score (MS)—the ratio of killed mutants to total mutants—which directly evaluates fault-detection effectiveness beyond PR. MS is reported only on the open-source benchmark; it is not reported for the industrial setting due to the proprietary nature of the dataset. Finding V: CATGen remains the most effective on open-source benchmarks compared to LLM-based approaches. As shown in Table 5, on Defects4J, existing approaches already achieve strong results in terms of both compilation success and structural coverage, indicating that current methods can generate usable unit tests for open-source projects with relatively stable environments. Under this strong baseline performance, CATGen still achieves the best overall effectiveness among LLM-based methods. In particular, CATGen attains the highest compilation success rate (92.40%) and pass rate (79.65%) among all LLM-based techniques, and remains only slightly below EvoSuite in compilation success (98.17%). This is consistent with practical experience that Defects4J-style , Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

17

Table 6. Mutation Score (MS) on Defects4J.

Project

EvoSuite

ChatTester

ChatUniTest

HITS

TELPA

RATester

CATGen

Chart Math Lang Time

58.2% 63.5% 65.8% 57.6%

50.1% 54.6% 57.9% 48.8%

48.6% 52.1% 55.4% 47.2%

53.4% 57.2% 60.3% 52.1%

57.8% 61.3% 64.7% 55.9%

60.2% 64.1% 66.5% 58.7%

66.9% 71.2% 73.6% 67.4%

Avg

61.3%

52.8%

50.8%

55.9%

59.7%

62.4%

69.8%

projects, which mainly contain computation-oriented logic with limited framework dependencies, are particularly favorable to search-based tools such as EvoSuite. At the same time, CATGen substantially improves test adequacy: it achieves the highest line coverage (80.64%) and branch coverage (75.58%) across all compared methods, surpassing both traditional search-based generation (EvoSuite: 62.44% / 53.08%) and the strongest LLM-based baselines (e.g., RATester: 72.85% / 66.32%). These results indicate that CATGen improves coverage while preserving a high level of executability. From our experience, the open-source results align with what we observed in the industrial evaluation: stabilizing the test class skeleton reduces early compilation friction, and lightweight post-processing helps recover common issues without relying on iterative LLM repair. In opensource projects where the environment is generally cleaner, these mechanisms appear to function as “reliability amplifiers”—they do not replace strong baseline capabilities, but make the generated tests more consistently executable and more thorough in structural exploration. Finding VI: CATGen improves fault-detection capability as measured by mutation score. To further assess whether the generated tests are effective in detecting faults beyond executability, we evaluate the mutation score (MS) on Defects4J. As shown in Table 6, CATGen consistently achieves the highest mutation score across all projects, outperforming the strongest baseline (RATester) by 7.4 percentage points on average. This indicates that the improvements in compilation success and structural coverage also translate into stronger fault-detection capability. Interestingly, although some baselines achieve relatively high line or branch coverage, their mutation scores remain comparatively lower, suggesting that high structural coverage does not necessarily imply effective fault detection. In contrast, CATGen not only generates executable tests but also produces tests that are more effective in killing injected mutants. These results highlight the importance of complementing traditional metrics such as PR and coverage with mutation-based evaluation, and confirm that CATGen improves both executability and fault-detection effectiveness. 6

Discussion

I. Case Study and Practical Insights. We now use the case in Figure 3 (QC-32B) to qualitatively demonstrate how CATGen operates and outperforms baselines. The corresponding focal method is shown in Figure 2a. Based on the structured test class skeleton, CATGen demonstrates clear advantages in test effectiveness over existing baselines. It generates test methods that accurately cover both nominal and exceptional execution paths. In the successful case, CATGen configures restTemplate.postForEntity() via when/thenReturn to return a properly typed ResponseEntity containing expected data (lines 4), thereby exercising the focal method’s main control flow. In the exceptional case, it simulates a network failure by injecting when().thenThrow(new RestClientException("Error")) (lines 11), ensuring that the focal method properly handles the exception and rethrows it as a CommonException (lines 13-14). Additionally, CATGen’s program analysis-based enhancement component identifies unhandled edge conditions, such as null values, which are often overlooked by LLMs, and generates dedicated tests to cover these cases explicitly. By integrating these three test methods into a cohesive and well-structured test class, CATGen achieves comprehensive , Vol. 1, No. 1, Article . Publication date: July 2026.

18

J. Chen, Z. Wang, L. Yang et al.

Fig. 4. Unit test generated by baselines

functional coverage of the focal method. For the same focal method, EvoSuite generated tests that failed to compile in our setting, which reflects a recurring challenge we encountered in industrial repositories where framework wiring, external types, and required initialization must be correct before any test logic becomes executable. When turning to LLM-based baselines in Figure 4, we observed different practical tradeoffs. ChatTester and ChatUniTest tend to exercise the system under real or close-to-real runtime conditions, which makes it difficult to reproduce exception scenarios such as network failures in a controlled test environment without inducing actual faults. HITS does incorporate mocking, yet it can still produce mocks that do not match the intended service behavior, for example returning a raw Map where a ResponseEntity is required, which weakens behavioral fidelity. TELPA is guided by coverage feedback and often tries to reach uncovered branches quickly, but in our setting it can be sensitive to harness stability, as shown by directly constructing RestTemplateService with a null dependency and invoking postForEntry without consistently configuring postForEntity via when and thenReturn or thenThrow. As a result, the generated tests may fail early due to incomplete initialization. RATester enriches generation with repository-level symbol and usage information and does set up Mockito.mock with a when rule for restTemplate.postForEntity, yet the test still hinges on scaffolding and mock semantics, such as returning null for a call that is expected to produce a ResponseEntity. This kind of mismatch can shift execution into a different path and reduces how faithfully the test represents the target service behavior under industrial project conventions. These observations suggest that, for industrial projects, incorporating broader context is helpful, while stabilizing the test class skeleton and initialization remains a prerequisite for turning that context into compilable and behavior-valid tests. Taken together, the quantitative evaluations and qualitative case analysis strongly reinforce the effectiveness of CATGen’s multi-stage architecture in addressing the challenges of LLM-based unit test generation, particularly in industrial scenarios. II. Lessons from Replacing LLM-Based Repair with Static Analysis. Our results show that deterministic program analysis yields more effective post-processing than probabilistic, LLM-based repair. To validate this claim, we further investigate the extent to which CATGen’s program analysis-based post-processing can enhance existing baselines. Specifically, we collect the baseline , Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

19

Table 7. Effectiveness comparison of different post-processing strategies. Method

CovL

CovB

CSR

PR

ChatTester ChatTesterw ChatTesterc

48.20% 43.93% 49.95%

49.60% 43.57% 50.17%

73.03% 63.36% 83.49%

51.57% 44.32% 52.82%

ChatUniTest ChatUniTestw ChatUniTestc

45.92% 38.53% 50.81%

46.78% 40.05% 56.20%

71.06% 59.05% 80.04%

45.96% 38.33% 48.51%

HITS HITSw HITSc

54.47% 48.18% 59.01%

56.45% 50.19% 63.70%

67.68% 65.30% 85.28%

46.55% 41.75% 53.35%

Note: w = without post-processing; c = with CATGen’s post-processing component.

results before post-processing, then substitute each baseline’s original post-processing module with CATGen’s, and re-evaluate the metrics. We use the subscript w to denote results before repair and c for results after applying CATGen’s post-processing. As in previous research questions, we use QC-32B as the underlying LLM for consistency. Results presented in Table 7 demonstrate that integrating CATGen’s post-processing module into baselines consistently enhances performance across all evaluation metrics. For CovL, ChatTester’s performance increases from 48.20% to 49.95%, ChatUniTest’s from 45.92% to 50.81%, and HITS’s from 54.47% to 59.01%. In CovB, ChatTester improves from 49.60% to 50.17%, ChatUniTest from 46.78% to 56.20%, and HITS from 56.45% to 63.70%. CSR rises notably: ChatTester’s CSR increases from 73.03% to 83.49%, ChatUniTest’s from 71.06% to 80.04%, and HITS’s from 67.68% to 85.28%. Similarly, in PR, ChatTester improves from 51.57% to 52.82%, ChatUniTest from 45.96% to 48.51%, and HITS from 46.55% to 53.35%. Notably, all baselines without post-processing exhibit inferior performance, with CSR dropping below 66% and PR decreasing below 46%. This stark contrast highlights the critical importance of post-processing in unit test generation. These results validate that deterministic program analysis outperforms heuristic strategies in post-processing, particularly for tasks requiring rigorous adherence to code correctness and execution stability. This suggests that making repair deterministic and context-grounded is a stronger leverage point than adding more LLM iterations, because it bounds cost while systematically eliminating recurring compiler-level failure modes. III. Future Work. Currently, CATGen primarily targets Java projects; future work will extend its capabilities to support mainstream programming languages such as Python, C++, and JavaScript, enabling broader applicability across diverse software ecosystems. A key focus will be optimizing model inference efficiency by refining inference workflows and post-processing strategies to enhance scalability in large-scale industrial projects, reducing both generation time and computational overhead. To improve test robustness, we will investigate methods for generating more diverse test cases that comprehensively cover edge cases, boundary conditions, and exception scenarios. This includes developing advanced condition-decomposition algorithms and boundary-value analysis techniques to address under-tested scenarios systematically. 7

Threats to Validity

External Validity. A potential threat to external validity arises from the choice of models and the implementation of baselines. To mitigate this, we evaluate CATGen across multiple state-of-the-art LLMs with diverse architectures, including general-purpose and code-specialized models, to ensure findings are not tied to a single model. For baselines, we adopt publicly available implementations to ensure fair and reproducible comparisons. Our evaluation targets reproducible, pipeline-level test , Vol. 1, No. 1, Article . Publication date: July 2026.

20

J. Chen, Z. Wang, L. Yang et al.

generation with locally hosted open-weight models; we do not report head-to-head experiments against interactive coding assistants (e.g., Claude Code and Codex-based tools), which optimize ad hoc developer interaction rather than batch generation under fixed project context and may raise data-security concerns for proprietary code under NDA settings. Internal Validity. Three aspects affect internal validity. First, our coverage metrics rely on Jacoco for runtime instrumentation, which may miss certain execution paths or misreport coverage in edge cases. While Jacoco is standard in Java testing, this limitation is acknowledged. Second, the non-deterministic nature of LLMs may introduce variability in generated tests. To address this, we conduct multiple independent runs for each configuration and apply statistical significance testing to ensure observed improvements are consistent. Third, when contrasting CATGen with baselines in Section 4, we compare outcomes on identical focal methods using paired Wilcoxon signed-rank tests [45], reporting 𝑝-values together with effect sizes (𝑟 ) and the accompanying changes in compilation success rate, line coverage, and branch coverage. Construct Validity. We treat success as compilable, executable tests under project constraints, not metric inflation alone. The industrial benchmark stresses deployment failure modes and may favor compilability-focused workflows; we therefore complement it with Defects4J and mutation-score analysis. Industrial focal methods were not publicly available during LLM training, and generic repair patterns also transferred to C pilots, but all reported experiments are Java-based. Static context extraction uses IntelliJ PSI in our implementation but depends only on AST-level structural signals; any extractor providing equivalent facts can substitute for PSI. 8

Conclusion

LLMs have recently shown promise for automated unit test generation, yet our industrial deployments reveal a persistent gap between encouraging research results and practical usability. In real-world projects, generated tests must not only achieve coverage but also compile and execute under complex project-level constraints. In our experience, we observed that insufficient code context, fragile scaffolding, and costly repair loops frequently dominate the workflow, limiting the benefits of existing prompt-driven approaches. To tackle this, we introduce a context-aware workflow developed through repeated industrial failures. Rather than relying on LLMs to infer missing dependencies, CATGen explicitly retrieves project context, constructs deterministic testclass skeletons, and applies analysis-driven post-processing to ensure compilation robustness. Across both industrial and open-source settings, CATGen consistently generates more compilable and effective tests while significantly reducing generation cost compared to existing LLM-based approaches. Our insights encourage future work to treat LLMs as components within engineered systems rather than standalone solutions, and to prioritize robustness and deployability when bringing AI-assisted testing techniques into real-world development environments. Data Availability Due to confidentiality agreements with our industrial partner, the full source code and proprietary benchmark datasets used in this work cannot be publicly released. However, to maximize reproducibility and transparency under these constraints, we maintain a public replication repository that documents evaluated LLM checkpoints and inference settings; the scope of static context extraction and its portability across representative IDE parser APIs; deterministic, framework-aware test skeleton construction; the compilation-repair strategies in Section 3.4 with worked examples and ordering rationale; consolidation rules for merging LLM-generated fragments with analysis-driven enhancements into one test class; and supporting Java parsing utilities together with auxiliary preprocessing and results-processing scripts released with the package. We additionally provide , Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

21

anonymized prompts, representative focal-method examples, and intermediate results aligned with these materials.2 References [1] 2026. JUnit 4. https://junit.org/junit4 Accessed: 2026-01. [2] 2026. JUnit 5. https://junit.org Accessed: 2026-01. [3] 2026. Mockito. https://site.mockito.org Accessed: 2026-01. [4] 2026. PowerMock. https://powermock.github.io Accessed: 2026-01. [5] 2026. Spock Framework. https://spockframework.org Accessed: 2026-01. [6] 2026. Spring Boot Testing. https://docs.spring.io/spring-boot/reference/testing Accessed: 2026-01. [7] Meta AI. 2023. CodeLlama-7B-Instruct. https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf. [8] Meta AI. 2025. Meta-Llama-3.1-8B-Instruct. https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct. [9] M Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Janis Benefelds. 2017. An industrial evaluation of unit test generation: Finding real faults in a financial application. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). IEEE, 263–272. [10] Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Automated unit test improvement using large language models at meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 185–196. [11] Arianna Blasi, Alessandra Gorla, Michael D Ernst, and Mauro Pezzè. 2022. Call me maybe: Using nlp to automatically generate unit test cases respecting temporal constraints. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–11. [12] Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 572–576. [13] Runxiang Cheng, Michele Tufano, Jürgen Cito, José Cambronero, Pat Rondon, Renyao Wei, Aaron Sun, and Satish Chandra. 2025. Agentic Bug Reproduction for Effective Automated Program Repair at Google. arXiv preprint arXiv:2502.01821 (2025). [14] Shyam R Chidamber and Chris F Kemerer. 1994. A metrics suite for object oriented design. IEEE Transactions on Software Engineering 20, 6 (1994), 476–493. [15] Christoph Csallner, Nikolai Tillmann, and Yannis Smaragdakis. 2008. DySy: Dynamic symbolic execution for invariant inference. In Proceedings of the 30th international conference on Software engineering. 281–290. [16] deepseek ai. 2023. DeepSeek-Coder-6.7B-Instruct. https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct. [17] deepseek ai. 2024. DeepSeek-Coder-33B-Instruct. https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct. [18] deepseek ai. 2024. DeepSeek-R1-Distill-Llama-8B. https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B. [19] deepseek ai. 2024. DeepSeek-R1-Distill-Qwen-32B. https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen32B. [20] Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K Lahiri. 2022. Toga: A neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering. 2130–2141. [21] Khalid El Haji, Carolin E. Brandt, and Andy Zaidman. 2024. Using GitHub Copilot for Test Generation in Python: An Empirical Study. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE). 45–55. [22] Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419. [23] Mark Harman and Phil McMinn. 2009. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Transactions on Software Engineering 36, 2 (2009), 226–247. [24] René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 international symposium on software testing and analysis. 437–440. [25] Konstantinos Kitsios, Marco Castelluccio, and Alberto Bacchelli. 2025. Automated Generation of Issue-Reproducing Tests by Combining LLMs and Search-Based Testing. arXiv preprint arXiv:2509.01616 (2025). 2 https://github.com/CATGen-repository/CATGen

, Vol. 1, No. 1, Article . Publication date: July 2026.

22

J. Chen, Z. Wang, L. Yang et al.

[26] Divya Kumar and Krishn Kumar Mishra. 2016. The impacts of test automation on software’s cost, quality and time to market. Procedia Computer Science 79 (2016), 8–15. [27] Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 919–931. [28] Runlin Liu, Zhe Zhang, Yunge Hu, Yuhang Lin, Xiang Gao, and Hailong Sun. 2025. LLM-based Unit Test Generation for Dynamically-Typed Programs. arXiv preprint arXiv:2503.14000 (2025). [29] Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2021. Studying the usage of text-to-text transfer transformer to support code-related tasks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 336–347. [30] Chao Ni, Xiaoya Wang, Liushan Chen, Dehai Zhao, Zhengong Cai, Shaohua Wang, and Xiaohu Yang. 2024. CasModaTest: A cascaded and model-agnostic self-directed framework for unit test generation. arXiv preprint arXiv:2406.15743 (2024). [31] Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J Mooney, and Milos Gligoric. 2023. Learning deep semantics for test completion. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). [32] Carlos Pacheco, Shuvendu K Lahiri, Michael D Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In 29th International Conference on Software Engineering (ICSE’07). IEEE, 75–84. [33] Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. 2025. Aster: Natural and multilanguage unit test generation with llms. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 413–424. [34] Qwen. 2024. Qwen2.5-32B. https://huggingface.co/Qwen/Qwen2.5-32B. [35] Qwen. 2024. Qwen2.5-Coder-32B-Instruct. https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct. [36] Qwen. 2024. Qwen2.5-Coder-7B-Instruct. https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct. [37] Per Runeson. 2006. A survey of unit testing practices. IEEE software 23, 4 (2006), 22–29. [38] Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering 50, 1 (2023), 85–105. [39] Davide Spadini, Maurício Aniche, Magiel Bruntink, and Alberto Bacchelli. 2017. To mock or not to mock? an empirical study on mocking practices. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 402–412. [40] Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617 (2020). [41] Dong Wang, Hanmo You, Lingwei Zhu, Kaiwei Lin, Zheng Chen, Chen Yang, Junji Yu, Zan Wang, and Junjie Chen. 2025. A Survey of Reinforcement Learning for Software Engineering. arXiv preprint arXiv:2507.12483 (2025). [42] Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911–936. [43] Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1258–1268. [44] Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On learning meaningful assert statements for unit test cases. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1398–1409. [45] Robert F Woolson. 2007. Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials (2007), 1–3. [46] Xusheng Xiao, Sihan Li, Tao Xie, and Nikolai Tillmann. 2013. Characteristic studies of loop problems for structural test generation via symbolic execution. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 246–256. [47] Chen Yang and Junjie Chen. 2026. Uncovering Business Logic Bugs via Semantics-Driven Unit Test Generation. arXiv:2604.23509 [cs.SE] https://arxiv.org/abs/2604.23509 [48] Chen Yang, Junjie Chen, Bin Lin, Ziqi Wang, and Jianyi Zhou. 2024. Advancing code coverage: Incorporating program analysis with large language models. ACM Transactions on Software Engineering and Methodology (2024). [49] Chen Yang, Ziqi Wang, Yanjie Jiang, Lin Yang, Yuteng Zheng, Jianyi Zhou, and Junjie Chen. 2025. Reflective Unit Test Generation for Precise Type Error Detection with Large Language Models. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 2834–2845. doi:10.1109/ASE63991.2025.00233 [50] Chen Yang, Ziqi Wang, Lin Yang, Dong Wang, Shutao Gao, Yanjie Jiang, and Junjie Chen. 2026. WiseUT: An Intelligent Framework for Unit Test Generation. In 2026 IEEE/ACM 48th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). , Vol. 1, No. 1, Article . Publication date: July 2026.

Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation (Experience Paper)

23

[51] Chen Yang, Lin Yang, Ziqi Wang, Dong Wang, Jianyi Zhou, and Junjie Chen. 2025. Clarifying Semantics of In-Context Examples for Unit Test Generation. arXiv preprint arXiv:2510.01994 (2025). [52] Chen Yang, Lin Yang, Ziqi Wang, Dong Wang, Jianyi Zhou, and Junjie Chen. 2025. Clarifying Semantics of In-Context Examples for Unit Test Generation. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 3046–3057. doi:10.1109/ASE63991.2025.00250 [53] Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, et al. 2024. An Empirical Study of Unit Test Generation with Large Language Models. arXiv preprint arXiv:2406.18181 (2024). [54] Xin Yin, Chao Ni, Xinrui Li, Liushan Chen, Guojun Ma, and Xiaohu Yang. 2025. Enhancing LLM’s Ability to Generate More Repository-Aware Unit Tests Through Precise Contextual Information Injection. arXiv preprint arXiv:2501.07425 (2025). [55] Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and improving chatgpt for unit test generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703–1726. [56] Jia Zhang, Zihao Liu, Yuchen Li, Zhongzhi Chen, Xuefeng Zhang, Shuang Lin, Yuxin Wu, Minghao Xu, Lianjun Wang, Weijie Zhao, Hua Zhou, Jiawei Zhang, Zhiyuan Zhang, Chao Liu, and Jun Guo. 2023. vLLM: High-Performance LLM Inference and Serving. https://arxiv.org/abs/2309.08017 [57] Hong Zhu, Patrick AV Hall, and John HR May. 1997. Software unit test coverage and adequacy. Acm computing surveys (csur) 29, 4 (1997), 366–427. [58] Hengcheng Zhu, Valerio Terragni, Lili Wei, Shing-Chi Cheung, Jiarong Wu, and Yepang Liu. 2025. Understanding and Characterizing Mock Assertions in Unit Tests. Proceedings of the ACM on Software Engineering 2, FSE (2025), 554–575.

, Vol. 1, No. 1, Article . Publication date: July 2026.