Code Health in LLM-Based Test Generation: Effectiveness and Token Efficiency

Code Health in LLM-Based Test Generation: Effectiveness and Token Efficiency Freya Wirdemann

Markus Borg

Nadim Hagatulah

Adam Tornhill

arXiv:2608.18645v1 [cs.SE] 19 Aug 2026

CodeScene Lund University CodeScene and Lund University Heidelberg University Malmö, Sweden Lund, Sweden Malmö, Sweden Heidelberg, Germany is316@stud.uni-heidelberg.de markus.borg@codescene.com nadim.hagatulah@cs.lth.se adam.tornhill@codescene.com

Abstract—Coding agents powered by Large Language Models benchmarking [5], and used operationally as a quality gate in (LLMs) are now prominent in software engineering. Previous CI and code review within the CodeScene product [6], [7]. work has shown that AI tools perform better on high-quality Based on this, we let the following research questions guide source code that is easy to maintain. In this study, we investigate how the effectiveness of LLM-generated unit tests our study: varies across maintainability levels measured by CodeScene’s RQ1 How is CodeHealth associated with the effectiveness of CodeHealth (CH). We assess test effectiveness using traditional LLM-generated test cases? coverage metrics and mutation score across Python, Java, and RQ How does LLM input token usage vary across different 2 C++. Moreover, we study how code with different levels of CH levels of CodeHealth? translates into input tokens using common industrial tokenizers. For RQ1 , we operationalize test effectiveness using both Our results suggest that CH provides a weak but consistent signal of LLM-generated test effectiveness and is negatively standard code coverage metrics and mutation score [8]. Our correlated with input-token count. These findings provide further findings in Sec. IV reveal a weak but consistent relationship evidence for a relationship between maintainability and LLMbetween CH and the effectiveness of the generated test cases, based software development. Index Terms—software engineering, maintainability, software and the results are more pronounced for mutation scores. For RQ2 , we find that lower-CH code generally requires more testing, large language models, token counts

I. I NTRODUCTION Industry observations reported by Thoughtworks indicate that AI tools, just like humans, perform better when operating on well-factored and modular code [1]. They refer to this as “AI-friendly code design.” Our recent work observed this phenomenon in a study of how code smells affect LLMs’ ability to preserve semantic behavior when refactoring competitive programming solutions in Python [2]. While refactoring is an informative proxy for how LLMs process source code with varying levels of maintainability, it captures only a single dimension of software development. In this study, we return to the question of how maintainability influences the performance of AI tools. We revisit the public dataset used in previous work and expand to roughly five times its size by adding C++ and Java solutions, each with varying numbers of code smells. Moreover, we shift the task from refactoring to test case generation, which we argue is another meaningful proxy for software development: we study how capable an LLM is at understanding existing code and generating new code from it. Moreover, we add a discussion of token counts, which have become a first-order cost concern, as illustrated by GitHub Copilot’s shift to usage-based billing on June 1, 2026. In line with the previous study, we characterize different maintainability levels using the CodeHealthTM (CH) metric. CH has been validated as predictive of defects and development effort in previous studies [3], [4], performs well in

input tokens, which persists after controlling for SLOC and is robust across contemporary tokenizer families. We discuss the practical implications and threats to validity in Sec. V. II. BACKGROUND AND R ELATED W ORK Test case generation has been a popular topic in test automation research [9]. Analogous to other research topics, the field has been disrupted by LLMs, and we focus this section accordingly. Jain et al. developed the benchmark TestGenEval, based on SWEBench, to measure LLM-based test generation performance for Python [10]. They evaluated models ranging from 7B to 405B, and reported the best results for OpenAI’s GPT4o: line coverage 35.2% and mutation score 18.8%, while smaller models generally performed worse. We identified three relevant systematic literature reviews on LLMs and testing. Wang et al. had the broadest scope and found that unit test generation was among the most common applications [11]. Chu et al. focused on unit test generation and present a taxonomy based on 115 primary studies [12]. With a similar scope, Zhang et al. presented a review based on 105 primary studies and report that coverage results vary greatly across studies [13]. Both unit-testing reviews conclude that existing research has focused heavily on Python and Java. Ouédraogo et al. present one of the most comprehensive studies of LLM-based unit test generation for Java [14]. Their findings include that only 7.2% of the generated tests compiled, with hallucinated symbols and incorrect API calls among the main causes of compilation failures. Moreover,

they report median line coverage ranging from 3.57% to 100% across their three datasets, and explain this variation by differences in structural complexity and architectural diversity. III. M ETHOD A. Dataset Creation Our starting point is the 5,000 Python files sampled from the CodeContests dataset [15] analyzed in our previous refactoring study [2]. One limitation with that dataset is that it barely contains any examples with CH < 8. To remedy this limitation and allow us to study a broader range of CH scores, we complement the dataset by also sampling C++ and Java solutions, which contain more low-CH files. We first calculated CH for all valid solutions. Following the stratified sampling strategy of the original study [2], we sampled 2,500 solutions with CH ≥ 9 and 2,500 with CH < 9 per language. We then added the lowest-scoring remaining files to improve coverage of the lower CH range, targeting approximately 10,000 files per language. To encourage syntactic diversity, we applied CodeBLEU [16] as a similarity filter in each sampling stage, accepting only candidates below a predefined similarity threshold. The final sets contained 9,795 Java files and 9,825 C++ files. Fig. 1 shows the CH distributions for Python, Java, and C++. Consistent with previous research [17], the distributions are concentrated toward the higher end of the CH scale, with comparatively fewer files in the low-CH range. For Java and C++, our sampling strategy deliberately includes more lowerCH files, making the left-skewed shape more visible. In the remainder of this paper, we discuss seven CH bins. The main threshold is CH = 9, which denotes “healthy” code in CodeScene and has been calibrated over several years to align with human perceptions of maintainability [5]. Our research [2] and experience with customers adopting AI-assisted development suggest that effective LLM-driven development may require even higher CH. Within healthy code (CH ≥ 9.0), we further distinguish ordinary healthy code (9 ≤ CH < 9.5), AI-ready code (9.5 ≤ CH < 9.8), and near-perfect code (CH ≥ 9.8). The remaining bins cover progressively lower ranges: 8–9, 7–8, 6–7, and CH ≤ 6. B. LLM and Prompt Engineering We study test case generation using Qwen3-Coder-30BA3B-Instruct (hereafter, Qwen3) an open-weight code LLM (temp=0.1, max tokens=4,096). The model uses a mixture-ofexperts architecture with approximately 30.5B total parameters and 3.3B activated parameters per token. We selected Qwen3 because it showed promising results in our original study [2] and provided a practical tradeoff between capability and hardware requirements. Previous research has shown that prompts can substantially influence the outcome of LLM-based coding tasks [18], including processing of test code [19]. We therefore iteratively refined language-specific prompt templates for Python, Java, and C++. The goal was to increase the likelihood that Qwen3 generated syntactically valid test code in the expected format,

(a) Python (n=5,000)

(b) Java (n=9,795)

(c) C++ (n=9,825)

Fig. 1: Distribution of CH across the datasets. Dashed vertical lines show bin thresholds. The black line shows the median input tokens per bin (Claude Sonnet 4.6 tokenizer).

so that the outputs could be executed by the corresponding unit test frameworks. All prompts instructed the model to generate a single unit test file, output only raw source code, and avoid markdown fences or explanatory prose. They also specified the expected testing framework for each language: pytest for Python, JUnit 5 for Java, and CppUnit for C++. To reduce invalid outputs, the prompts included language-specific constraints, such as exact test class or module names, required imports, entry-point conventions, and handling of standard input/output. The source code under test was appended directly to the prompt. The exact prompt templates are available in the replication package [20]. C. Experimental Setup The experiments were executed on an NVIDIA A100 Tensor Core GPU with 80GB VRAM. We deployed Qwen3 locally using vLLM as the inference engine. We implemented a batch-processing pipeline that iterates over all input files and generates unit tests in a single-shot setting. For each file, the pipeline prompts Qwen3 using the

corresponding language-specific template and logs the prompt, raw model response, token usage, execution time, and success or failure status. Despite careful prompt instructions, some model outputs contained formatting artifacts. We therefore applied deterministic post-processing before execution. This step removed markdown code fences and prose outside the generated code, validated that responses were non-empty, and wrote test files using deterministic naming schemes. No additional repair or correction was performed. D. Data Analysis We analyze the generated test cases using line coverage, branch coverage, and mutation score. For each outcome, we compute Spearman correlations with CH. Since file size is known to be inversely related to CH [3], we also compute partial Spearman correlations controlling for Source Lines of Code (SLOC). We report significance as ∗ p < .05, ∗∗ p < .01, ∗∗∗ p < .001, and ns for p ≥ .05. 1) Coverage Testing: Coverage was measured separately for each target language using commonly used tooling: coverage.py with branch tracking for Python, JaCoCo with the JUnit 5 console runner for Java, and gcov/lcov with CppUnit for C++. Each generated test file was compiled or executed together with its corresponding source file, and source-test pairs were isolated to avoid interference between files. Finally, we extracted line and branch coverage from the tools’ resulting reports. 2) Mutation Testing: Mutation scores were measured separately for each target language using language-specific tooling: mutmut for Python, PIT with the JUnit 5 plugin for Java, and Mull with CppUnit for C++. For each sourcetest pair, we first executed the generated tests as a baseline and performed mutation testing only when the baseline tests passed. Java source-test pairs were evaluated in temporary Maven projects, Python pairs in temporary directories, and C++ pairs as temporary executables compiled with g++ and the Mull compiler plugin. Across all languages, the mutation score was computed as the ratio of killed mutants to all generated mutants. Test suites that failed to compile, could not be executed, or timed out were recorded as unsuccessful executions and excluded from mutation-score computation. 3) Tokenization: To study input-token counts in relation to RQ2 , we counted source-file tokens using four contemporary model families: Qwen, OpenAI, Claude, and Gemini. Specifically, we used the tokenizer distributed with Qwen, OpenAI’s o200k_base tokenizer, and the official tokencounting APIs for Claude Sonnet 4.6 and Gemini 3.1 Pro Preview. Qwen and OpenAI counts were computed locally using Hugging Face Transformers and tiktoken, respectively. We then calculated Spearman correlations between CH and token counts, including partial correlations controlling for SLOC and character count. IV. R ESULTS AND D ISCUSSION This section analyzes line coverage, branch coverage, and mutation score for the generated test cases across the CH bins.

(a) Line coverage

(b) Branch coverage

(c) Mutation score

Fig. 2: Test effectiveness metrics across different CH bins.

Fig. 2 shows the test effectiveness metrics across different CH bins, and Table I presents further details. Moreover, the section presents an analysis of token counts across tokenizers. A. RQ1: CodeHealth and Test Effectiveness Fig. 2a presents the line coverage achieved by the generated test cases. For Python, the median coverage scores are relatively stable across the CH intervals with at least 10 files, ranging from approximately 30% to 47% (ρ = 0.11). For Java, the medians remain consistently higher, between approximately 66% and 77% across the full CH range, with negligible correlation between CH and coverage (ρ = 0.01, ns). C++ shows substantially lower coverage, especially for files with low CH. However, it also exhibits the clearest increasing trend: the median values increase monotonically across CH intervals, although the correlation remains weak (ρ = 0.15), as presented in Table II. Fig. 2b presents the branch coverage achieved by the generated test cases. Compared with line coverage, branch coverage is generally lower, as expected. For Python, the median values remain modest across the CH intervals, ranging from 8% to

TABLE I: Test case generation results per CH bin, showing median line coverage (%), branch coverage (%), mutation score (%), and median input tokens. Values for Python in the CH<7 bins are very few and presented in italic font. CH

Python (n = 5,000)

Java (n = 9,795)

N SLOC Line Branch Mut. Tokens

N SLOC Line Branch Mut. Tokens

<6 10 28 6–7 7–8 277 2,185 8–9 9–9.5 994 9.5–9.8 942 9.8–10 564

87 84 89 79 78 75 74

19.4 31.3 28.9 40.5 33.3 44.5 46.6

8.6 8.3 15.0 21.9 18.2 26.9 25.0

25.9 1,263.5 237 46.4 1,072.0 893 38.6 1,056.0 4,127 42.6 986.0 2,038 48.3 985.0 1,270 46.8 976.0 959 57.8 933.0 271

130 149 151 142 136 134 119

67.3 68.9 68.3 68.7 66.4 69.4 77.1

TABLE II: CH-test effectiveness correlations. ρS is the partial correlation controlling for SLOC. Language

Line cov. ρ

Python Java C++

ρS

Branch cov. ρ

ρS

Mutation score ρ

ρS

+.11∗∗∗ +.11∗∗∗ +.07∗∗∗ +.07∗∗∗ +.18∗∗ +.12 +.01 −.02∗ +.08∗∗∗ +.06∗∗∗ +.21∗∗∗ +.20∗∗∗ +.15∗∗∗ +.16∗∗∗ +.14∗∗∗ +.14∗∗∗ +.34∗∗∗ +.35∗∗∗

27%, with only a weak positive correlation (ρ = 0.07). Java again achieves the highest coverage levels. Unlike the line coverage results, however, the branch coverage medians show a clearer upward tendency across CH intervals, although the correlation remains very weak (ρ = 0.08). For C++, branch coverage remains low in absolute terms, but increases more consistently with CH than for Python and Java, yielding a weak positive correlation (ρ = 0.14). Fig. 2c shows the mutation scores achieved by the generated test cases. Mutation scores are only reported for files where the generated baseline tests passed, since mutation testing requires an executable test suite. This conditioning may bias the observed CH associations if execution success itself varies with CH, particularly in sparsely populated intervals. The mutation-score boxplots show an upward shift in central tendency, especially for Java and C++. For Python, mutation testing generated results for only 251 files, and the median mutation scores range from approximately 39% to 57%, with a weak positive correlation (ρ = 0.18). For Java, the trend is more pronounced: the median mutation score increases from about 35% in the lowest CH interval to about 95% in the highest CH interval, resulting in a stronger correlation than for line and branch coverage (ρ = 0.21). For C++, the absolute mutation scores remain substantially lower, but they increase steadily across CH intervals, from approximately 3% to 12%. This yields the strongest observed association among the three languages (ρ = 0.34). Across the languages, the boxplots show substantial variability and overlapping distributions across CH intervals. Thus, the observed trends should not be interpreted as deterministic effects at the individual-file level. Instead, they indicate that higher CH is associated with a gradual shift in the distribution toward generating more effective test cases. This pattern is weak for coverage-based metrics, especially for Python and Java, but clearer for mutation score. Actually, within each

36.7 48.5 48.9 52.4 50.0 59.1 70.8

C++ (n = 9,825) 34.8 34.4 43.4 50.0 62.5 62.5 95.1

1,492.0 1,426.0 1,364.0 1,252.0 1,194.5 1,171.0 1,028.0

N SLOC Line Branch Mut. Tokens 479 2,096 2,990 1,760 1,518 822 160

123 140 136 121 120 118 115

3.2 5.2 7.3 14.1 20.9 21.7 35.2

0.9 1.6 2.9 6.1 10.0 12.1 16.7

2.5 4.4 5.7 8.6 8.5 9.6 12.0

1,598.0 1,561.5 1,486.0 1,317.5 1,290.0 1,241.5 1,111.5

TABLE III: CH–token count correlations. ρS and ρC are partial correlations controlling for SLOC and character count, resp. Lang. Python

Java

C++

Tokenizer Qwen OpenAI Claude Gemini Qwen OpenAI Claude Gemini Qwen OpenAI Claude Gemini

ρ ∗∗∗

−.12 −.12∗∗∗ −.11∗∗∗ −.09∗∗∗ −.30∗∗∗ −.30∗∗∗ −.29∗∗∗ −.28∗∗∗ −.28∗∗∗ −.28∗∗∗ −.25∗∗∗ −.25∗∗∗

ρS

ρC

.00 .00 +.01 +.04∗∗ −.34∗∗∗ −.33∗∗∗ −.32∗∗∗ −.30∗∗∗ −.28∗∗∗ −.29∗∗∗ −.25∗∗∗ −.24∗∗∗

−.02 −.02 .00 +.04∗∗ −.28∗∗∗ −.28∗∗∗ −.27∗∗∗ −.25∗∗∗ −.13∗∗∗ −.14∗∗∗ −.05∗∗∗ −.04∗∗∗

language, ρ tends to increase as the metric becomes more semantically demanding, i.e., from standard coverage metrics to mutation scores. This suggests that CH is more strongly related to behavioral test effectiveness than to coverage alone: mutation score captures whether tests detect injected behavioral changes, whereas coverage only captures whether code elements are executed. Partial Spearman correlations controlling for SLOC (ρS ) show that the main pattern remains largely stable. The coverage correlations are essentially unchanged for Python and C++, while the Java coverage associations weaken slightly after the control. For mutation testing, C++ remains the strongest observed association (ρS = 0.35), and Java remains practically unchanged (ρS = 0.20). The main exception is Python, where the mutation-score correlation is no longer statistically significant after controlling for SLOC; however, this result is based on only 251 observations. B. RQ2: CodeHealth and Token Efficiency The input-token columns in Table I report the median number of input tokens per CH interval during the test generation experiment. We find that in the lowest CH interval, the median input-token count is 45.1% higher for Java and 43.8% higher for C++ compared with the near-perfect interval. For Python, the pattern is less clear, which might partly be explained by the dataset containing very few files in the lowest CH intervals. Table III shows detailed correlation analyses from contemporary tokenizers. Values with |ρ| > .20 are shown in bold. Furthermore, Fig. 1 depicts the median input tokens for the Claude Sonnet 4.6 tokenizer across the CH bins.

Across all four tokenizers, lower CH is associated with higher input-token counts for Java and C++ (ρ = −.25 to −.30). The Python subset, which is concentrated in the higher CH intervals, shows only weak correlations (ρ ≈ −.1). For these files, the association essentially disappears after controlling for SLOC. This suggests that lower-CH Python files require more tokens mainly because they contain more source lines. For Java and C++, the negative correlations remain after controlling for SLOC (ρS = −.34 to −.24). Thus, at a comparable number of SLOC, lower-CH Java and C++ files still tend to consume more tokens. The token overhead of lower CH is therefore not merely a line-count effect. When controlling for character count (ρC ), the two languages differ. For Java, the association between CH and token count remains strong, i.e., at a comparable number of characters, lower-CH Java still tends to require more tokens. Low-CH Java appears to fragment into more tokens per character than high-CH Java. For C++, the character-controlled association weakens substantially, especially for the Claude and Gemini tokenizers. This suggests that, in competitive programming, lower-CH C++ files are more token-expensive mainly because they are more text-heavy. A complementary tokensper-character analysis (replication package [20]) supports this reading: CH correlates negatively with tokens per character for Java across all four tokenizers, but for C++ only weakly and only for the Qwen and OpenAI tokenizers. We hypothesize that Java syntax and code conventions might influence the difference in (ρC ). Java code often relies on chained method calls [21] and generic types [22], both of which may contribute to token overhead. Moreover, Java’s camelCase identifiers [23] tend to fragment into several tokens. On the other hand, C++ has compact syntactic mechanisms whose complexity may not expand input tokens as directly as Java idioms do. Examples include pointer use, macros, and operator-heavy expressions. Future work is clearly needed to investigate this in more detail. V. I MPLICATIONS AND L IMITATIONS Coding agents’ ability to analyze and manipulate source code in practical settings does not depend only on the capabilities of their underlying LLMs, agent harnesses, and instructions. The maintainability of the code they operate on is also a critical factor. In our previous work [2], we used the success rate of AI-generated refactoring as a proxy for AI-friendliness [1]. In this study, we extend that perspective to test-case generation by analyzing whether healthier code is associated with more effective generated tests. Our results related to RQ1 add more evidence to the proposition that AI tools, just like humans, perform better in healthy codebases. Median line coverage, branch coverage, and mutation score are highest in the upper CH bins, and the correlation between CH and mutation score in particular suggests a potentially relevant implication for Java and C++. For RQ2 , our findings provide an initial link between software maintainability and LLM economics. Poor maintain-

ability is traditionally understood as a human cost: it makes code harder to understand, test, and evolve. In line with our previous study [2], our analysis suggests an additional AI-era cost: unhealthy code may also consume more LLM context. This matters for AI-assisted development because tokens are the unit on which LLM APIs are billed, affecting prompting costs, code retrieval, context-window usage, and the practical efficiency of agentic workflows. Poor maintainability may therefore create a double penalty, i.e., it challenges human cognition while also making LLM-driven work more costly. This study constitutes a first exploration of test effectiveness and token efficiency. The main threat to the validity of this work is external validity, as the source code under study originates in competitive programming. CodeContests consists of standalone competitive-programming solutions rather than conventional production units, limiting generalizability to industrial unit testing. However, the corpus spans many quality levels, and CH captures local properties (nesting, complexity, etc.) that exist in any code regardless of domain. Also, our effectiveness scores are comparable to those reported in related work [13], [10], thus we argue that the relative differences across bins carry meaning. A second threat to generalizability is that we use a single LLM, Qwen3-Coder-30B. Larger frontier models might generate substantially better test cases, potentially also for lowerCH code. Given the rapid development of AI-assisted coding, future work should study code from additional domains, newer LLMs, and more advanced agentic workflows. VI. C ONCLUSION AND F UTURE W ORK First, CH provides a weak but consistent signal of LLMgenerated test effectiveness. The relationship is clearest for mutation score, suggesting that healthier code is more likely to yield test cases that detect injected behavioral changes, not only tests that execute lines or branches. Second, CH is negatively correlated with input-token count across tokenizers. For Java and C++, the association remains after controlling for SLOC. For Java, it also remains after controlling for character count, suggesting that low-CH Java code incurs token overhead beyond source-text length alone. This preliminary work opens avenues for future research. First, we plan to expand the set of studied LLMs and move beyond the domain of competitive programming. The current setup was useful for identifying differences in how LLMs process code across varying levels of CH. However, future studies should consider more realistic testing workflows based on agentic designs, including test-suite execution and iterative test generation toward coverage targets. Such multi-turn settings would also provide a richer context for studying token economics, including output-token consumption, which was considered out of scope in the present study. ACKNOWLEDGMENT This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP WARAOps) and the Competence Centre NextG2Com (VINNOVA grant 2023-00541).

R EFERENCES [1] Thoughtworks, “Technology Radar: An Opinionated Guide to Today’s Technology Landscape,” Tech. Rep. 32, Apr. 2025. [Online]. Available: https://www.thoughtworks.com/content/dam/thoughtworks/ documents/radar/2025/04/tr technology radar vol 32 en.pdf [2] M. Borg, N. Hagatulah, A. Tornhill, and E. Söderberg, “Code for Machines, Not Just Humans: Quantifying AI-Friendliness with Code Health Metrics,” in Proc. of the 3rd ACM International Conference on AI Foundation Models and Software Engineering, 2026, pp. 51–61. [3] A. Tornhill and M. Borg, “Code Red: The Business Impact of Code Quality - A Quantitative Study of 39 Proprietary Production Codebases,” in Proc. of the 5th International Conference on Technical Debt, 2022, pp. 11–20. [4] M. Borg, I. Pruvost, E. Mones, and A. Tornhill, “Increasing, Not Diminishing: Investigating the Returns of Highly Maintainable Code,” in Proc. of the 7th International Conference on Technical Debt, 2024, pp. 21–30. [5] M. Borg, M. Ezzouhri, and A. Tornhill, “Ghost Echoes Revealed: Benchmarking Maintainability Metrics and Machine Learning Predictions Against Human Assessments,” in Proc. of the 40th International Conference on Software Maintenance and Evolution, 2024, pp. 678–688. [6] T. Mårtensson, “So Much More Than Test Cases - An Industrial Study on Testing of Software Units and Components,” Journal of Systems and Software, vol. 228, p. 112479, 2025. [7] S. Tverdal, P. Nguyen, A. Goknil, A. Martini, M. Astekin, M. Orucevic, M. M. Kruke, and H. Stranden, “Combining Insights from Multiple Tools to Manage Technical Debt in Industrial C# Projects,” in Proc. of the 41st IEEE International Conference on Software Maintenance and Evolution, Sep. 2025, pp. 709–720. [8] A. Siami Namin, J. H. Andrews, and D. J. Murdoch, “Sufficient Mutation Operators for Measuring Test Effectiveness,” in Proc. of the 30th International Conference on Software Engineering, 2008, pp. 351–360. [9] S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, P. McMinn, A. Bertolino, J. Jenny Li, and H. Zhu, “An Orchestrated Survey of Methodologies for Automated Software Test Case Generation,” Journal of Systems and Software, vol. 86, no. 8, pp. 1978–2001, 2013. [10] K. Jain, G. Synnaeve, and B. Roziere, “TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark,” in Proc. of the 13th International Conference on Learning Representations, 2025. [11] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software Testing With Large Language Models: Survey, Landscape, and Vision,” IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 911–936, Apr. 2024. [12] B. Chu, Y. Feng, K. Liu, Z. Guo, Y. Zhang, H. Shi, Z. Nan, and B. Xu, “Large Language Models for Unit Test Generation: Achievements, Challenges, and Opportunities,” Dec. 2025, arXiv:2511.21382. [13] Q. Zhang, C. Fang, S. Gu, Y. Shang, Z. Chen, and L. Xiao, “Large Language Models for Unit Testing: A Systematic Literature Review,” Jun. 2025, arXiv:2506.15227. [14] W. C. Ouedraogo, A. K. Kabore, Y. Li, H. Tian, A. Koyuncu, J. Klein, D. Lo, and T. F. Bissyandé, “Prompt Engineering in LLMs for Automated Unit Test Generation: A Large-scale Study,” Empirical Software Engineering, vol. 31, no. 4, p. 103, Mar. 2026. [15] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. Sutherland Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals, “Competition-Level Code Generation With AlphaCode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022. [16] S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “CodeBLEU: a Method for Automatic Evaluation of Code Synthesis,” Sep. 2020, arXiv:2009.10297. [17] M. Borg, A. Udayakumar, and A. Tornhill, “Industrial Code Quality Benchmarks: Toward Gamification of Software Maintainability,” in Proc. of the 4th Workshop on Gamification in Software Development, Verification, and Validation, 2025, pp. 1–8. [18] A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large Language Models for Software Engineering: Survey and Open Problems,” in Proc. of the International Conference on Software Engineering: Future of Software Engineering, 2023, pp. 31–53.

[19] Y. Gao, X. Hu, X. Yang, and X. Xia, “Automated Unit Test Refactoring,” Proc. of the ACM on Software Engineering, vol. 2, no. FSE, pp. FSE033:713–FSE033:733, 2025. [20] F. Wirdemann, M. Borg, N. Hagatulah, and A. Tornhill, “codesceneresearch/test-generation-scam-2026: Submission,” 2026. [Online]. Available: https://doi.org/10.5281/zenodo.20622202 [21] A. M. Keshk and R. Dyer, “Method Chaining Redux: An Empirical Study of Method Chaining in Java, Kotlin, and Python,” in Proc. of the 20th International Conference on Mining Software Repositories, May 2023, pp. 546–557. [22] C. Parnin, C. Bird, and E. Murphy-Hill, “Adoption and Use of Java Generics,” Empirical Software Engineering, vol. 18, no. 6, pp. 1047– 1089, 2013. [23] J. Zhang, W. Zou, and Z. Huang, “An Empirical Study on the Usage and Evolution of Identifier Styles in Practice,” in Proc. of the 28th AsiaPacific Software Engineering Conference, Dec. 2021, pp. 171–180.

Related documents

• Insecure Coding Preferences in Long-Term Memory: Security Risks for LLM-based Code Generation#386774
• CodePoisonRAG: Knowledge Poisoning Attacks on Retrieval-Augmented Code Generation#924903
• CodePoisonRAG: Knowledge Poisoning Attacks on Retrieval-Augmented Code Generation#920669
• Practical Source Code Recovery from Binary Functions Using Anchor-Based Retrieval and LLM Reasoning#361503
• Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation#821622