Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

softwarearchitecture softwareengineering testing

software engineering, software architecture, testing

arXiv:2606.28229v1 [cs.SE] 26 Jun 2026

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring WENDKÛUNI C. OUÉDRAOGO, University of Luxembourg, Luxembourg YINGHUA LI∗ , Nanjing University of Science and Technology, China XUEQI DANG, University of Luxembourg, Luxembourg PAWEŁ BORSUKIEWICZ, University of Luxembourg, Luxembourg LINGFENG BAO, Zhejiang University, China ANIL KOYUNCU, Bilkent University, Turkey JACQUES KLEIN, University of Luxembourg, Luxembourg DAVID LO, Singapore Management University, Singapore TEGAWENDÉ F. BISSYANDÉ, University of Luxembourg, Luxembourg Search-based test generation tools such as EvoSuite can produce compilable and high-coverage unit tests at scale, yet the resulting suites are often difficult to read and maintain, limiting practical adoption. Large language models (LLMs) can generate more natural tests, but one-shot generation remains brittle, with compilation rates as low as 51–78% in our study. We introduce TestHumanizer, a hybrid SBST+LLM approach that treats LLMs as controlled refactoring layers over compilable SBST suites, improving naming, structure, and developeroriented clarity while preserving test behavior and compilation validity. We evaluate TestHumanizer on 350 classes from Defects4J (147) and SF110 (203). EvoSuite generates 15 suites per class, each refactored under three context configurations (tests-only, code-centric, summary-based) using two LLMs (gpt-4o, mistral-large-2407), yielding 31,500 refactorings. TestHumanizer achieves compilation rates of 88– 98%, approaching EvoSuite’s 100% compilation baseline and clearly outperforming direct LLM generation in both compilability and coverage preservation. Structural coverage is largely preserved (typically within 1–2 percentage points, with median method coverage remaining at 100%), and 86–95% of refactorings remain above a composite faithful-refactoring threshold. Refactored suites exhibit improved predicted readability, reduced control-flow and cognitive complexity, and substantial mitigation of structural smells (e.g., Conditional Logic Test reduced from 36.97% to 5.02% on Defects4J and from 48.31% to 4.37% on SF110 in the summary-based setting). The summary-based configuration provides the most robust trade-off, whereas long code-centric prompts are more prone to hallucination-induced failures and API-level rejections. A developer study on 30 classes (444 test methods) confirms significant gains in perceived readability and willingness to adopt (Wilcoxon, 𝑝 < 0.01, medium-to-large effects), with substantial inter-rater agreement (𝛼 = 0.71–0.85). Overall, LLMs are most effective not as standalone generators but as validation-gated refinement layers over robust ∗ Corresponding author.

Authors’ Contact Information: Wendkûuni C. Ouédraogo, [email protected], University of Luxembourg, Luxembourg; Yinghua Li, [email protected], Nanjing University of Science and Technology, China; Xueqi Dang, [email protected], University of Luxembourg, Luxembourg; Paweł Borsukiewicz, [email protected], University of Luxembourg, Luxembourg; Lingfeng Bao, [email protected], Zhejiang University, China; Anil Koyuncu, anil. [email protected], Bilkent University, Turkey; Jacques Klein, [email protected], University of Luxembourg, Luxembourg; David Lo, [email protected], Singapore Management University, Singapore; Tegawendé F. Bissyandé, [email protected], University of Luxembourg, Luxembourg. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1557-735X/2026/6-ART https://doi.org/10.1145/nnnnnnn.nnnnnnn J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

SBST outputs, motivating agentic and retrieval-augmented refactoring pipelines and evaluation protocols that go beyond surface similarity. CCS Concepts: • Software and its engineering → Software testing and debugging; • Computer systems organization → Neural networks. Additional Key Words and Phrases: Test refactoring, Large Language Models, SBST, Unit Test Generation, Empirical Study ACM Reference Format: Wendkûuni C. Ouédraogo, Yinghua Li, Xueqi Dang, Paweł Borsukiewicz, Lingfeng Bao, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F. Bissyandé. 2026. Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring. J. ACM 1, 1 (June 2026), 37 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

Introduction

Unit tests are essential for detecting defects early, documenting expected behavior, and supporting safe software evolution [9, 59, 61]. Over the last decade, automated test generation has reduced the manual effort of building unit test suites while achieving high structural coverage [18, 49]. Among these tools, [18] has emerged as a state-of-the-art Search-Based Software Testing (SBST) framework and a coverage-oriented baseline validated by large-scale studies [19, 40, 46, 63]. However, despite strong coverage, EvoSuite-generated tests are rarely adopted in practice because developers perceive them as difficult to read, understand, and maintain [3, 40, 50]. This usability gap is largely explained by well-documented readability and maintainability issues in SBST suites—cryptic naming, implementation-driven assertions, and high densities of test smells (e.g., Assertion Roulette, General Fixture, Redundant Assertions)—whose profiles differ markedly from human-written tests and inflate maintenance effort [5, 7, 15, 43, 57]. Optimized for coverage rather than domain intent [18], these suites can obscure scenario semantics, hinder debugging and evolution, and are often rejected or rewritten, limiting their value as executable documentation and their industrial uptake [3, 5, 57]. Large Language Models (LLMs) offer new opportunities for improving test quality [60, 63, 65]. However, empirical evaluations reveal recurring trade-offs: although LLM-generated tests often improve readability, they may fail to compile, hallucinate APIs, or achieve lower coverage in practice [40], and exhibit distinct smell profiles under minimal prompting [43]. Recent work has started to combine SBST and LLM strengths in hybrid pipelines: some approaches use LLMs to enrich test inputs or guide search [31, 46], others improve input consistency [69] or apply postprocessing transformations such as renaming and commenting [11, 16, 23]. Yet these approaches address only a fraction of the coverage–usability gap: input-level and search-guidance methods do not target test readability, while post-processing approaches either restrict transformations to identifier renaming or lack a systematic study of how contextual information affects suite-level refactoring quality, test smells, and developer acceptance. This leaves open the question of whether LLMs can be used as principled, context-aware refactoring layers that improve the full structure of SBST-generated suites without compromising their behavioral correctness. In this paper, we investigate whether LLMs can systematically refactor EvoSuite-generated test suites into suites that developers find easier to read, maintain, and reuse, without altering their behavior. We propose TestHumanizer, an LLM-based framework that refactors EvoSuite-generated test suites to improve developer comprehension and maintainability without changing their behavior. Unlike approaches that focus mainly on renaming identifiers, TestHumanizer operates at the level of entire test suites, preserving EvoSuite-specific structural elements (packages, imports, annotations, and class declarations) while targeting increased readability, maintainability, and modularity through structural changes such as Given–When–Then documentation, grouping of related logic, J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

and descriptive renaming. We examine how the form and scope of contextual information influence refactoring quality, and instantiate three configurations: (i) tests-only, providing only the EvoSuite suite; (ii) code-centric, adding the full source code of the class under test; and (iii) summary-based, providing tests plus an automatically generated natural-language summary of the class—a compact alternative that mitigates the “lost-in-the-middle” effect inherent to long-context prompting [26, 34]. We evaluate TestHumanizer both quantitatively, using metrics that measure refactoring quality across feasibility, semantic preservation, coverage, and structural smells, and qualitatively, via a human study in which professional developers compare EvoSuite tests with TestHumanizer-refactored versions on readability, understandability, and adoption willingness. The main contributions of this work are as follows: (1) TestHumanizer: An LLM-based framework that refactors SBST-generated test suites (e.g., EvoSuite) to improve readability and maintainability while preserving test behavior and coverage. (2) Contextual Refactoring Study: A systematic comparison of three context configurations (tests-only, code-centric, summary-based) to quantify how the form and scope of contextual information affect LLM-driven refactoring quality. (3) Large-Scale Empirical Evaluation: An extensive study over 350 classes (147 Defects4J, 203 SF110), generating 31,500 refactored suites using two LLMs, and assessing readability, understandability, coverage, and test smells. (4) Human Evaluation: A developer study involving four evaluators who compare EvoSuitegenerated suites with their TestHumanizer-refactored counterparts over 30 classes (444 test methods), rating perceived readability, clarity of intent, and willingness to adopt the tests in real-world scenarios. (5) Replication Package: All relevant artifacts are made publicly available to ensure transparency and enable replication: https://anonymous.4open.science/r/TestHumanizer-06FB/ The rest of the paper is organized as follows. Section 2 presents background and related work on automated test generation, test smells, and LLM-based code transformation. Section 3 describes the TestHumanizer framework, including our prompting strategies, context configurations, study design, datasets, and evaluation metrics. Section 4 reports our empirical and human-study results. Section 5 discusses implications and threats to validity. Section 6 reviews prior work, and Section 7 concludes with directions for future research. 2 2.1

Background and Related Work From SBST Tools to LLM-Driven Tests

Automated unit test generation has traditionally relied on SBST tools such as EvoSuite [18] and Pynguin [35], which optimize for high code coverage but often produce tests that lack readability and maintainability [3, 15, 24, 50, 53]. In contrast, LLMs generate more human-readable tests by leveraging natural language and code priors [10, 40, 41, 60, 63, 65, 68]. However, LLM-generated tests generally achieve lower coverage than SBST tools and exhibit higher rates of compilation failures, hallucinated APIs, and brittle assertions [40, 60, 63]. LLMs also face token limitations that make it difficult to handle large classes or complex dependencies, whereas SBST tools can process entire projects or JAR files in a uniform way. To address these trade-offs, recent work has started to combine SBST, static analysis, and LLMs. CodaMOSA [31] uses LLMs to escape EvoSuite-like coverage plateaus, while Gay et al. [23] apply GPT-4 to improve readability by refining test names and comments. Zhou et al. [69] introduce the C3 criterion to align LLM-generated test inputs with source-code usage patterns, and UTGen [16] refines SBST-generated tests but remains sensitive to incomplete context. ASTER [48] pushes this J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

line further by guiding LLM-based test generation with lightweight static analysis and iterative repair: it achieves coverage competitive with, or superior to, EvoSuite and CodaMOSA on Java SE, Java EE, and Python projects, while producing substantially more “natural” tests that 161 professional developers often consider ready to add to regression suites with minor or no changes. These studies show that SBST and LLMs excel at complementary aspects—coverage and robustness on one side, readability and naturalness on the other—and motivate approaches that combine their strengths rather than using LLMs as drop-in replacements. 2.2

Readability vs. Understandability

Readability and understandability are related but distinct dimensions of test quality. Readability concerns how easily developers can visually parse and skim code, influenced by layout, naming, and comments [12, 39]. Understandability captures the cognitive effort required to grasp a test’s intent, control flow, and assertion logic [24]. Buse and Weimer [12] introduced a learned readability model later adapted to unit tests [15], and Scalabrino et al. [54] proposed a comprehensive Java readability model combining structural and lexical features. Recent work shows that modern LLMs can approximate or surpass such models in human alignment [42, 56]. In this study, we adopt Scalabrino’s model as a stable and reproducible large-scale baseline, and complement it with structural metrics and a human evaluation rather than relying on LLMs both for refactoring and for judging readability. Understandability is commonly approximated via structural metrics such as Cyclomatic Complexity [37]. However, generic cognitive complexity measures were designed for production code and overlook test-specific constructs [13, 45]. Structured testing patterns such as Arrange–Act–Assert (AAA) and its Behavior-Driven Development (BDD) variant Given–When–Then (GWT) address this gap: AAA organizes each test into setup, execution, and verification phases, while GWT expresses the same structure in natural-language terms that are accessible to both technical and non-technical stakeholders. Both patterns help developers quickly locate setup, action, and expected outcome within a test, reducing cognitive load when reading test suites. To better reflect the cognitive structure of tests, we combine Cyclomatic Complexity [37] with CCTR, a test-aware cognitive complexity metric tailored to unit tests [45], which accounts for assertion density, annotations, and test structuring patterns. 2.3

Test Smells

Test smells, introduced by Van Deursen et al. [64], denote recurring design flaws in test code that hinder readability and maintainability, analogous to code smells in production systems. Common examples include Assertion Roulette, General Fixture, Eager Test, and Magic Number Test. Empirical studies show that such smells negatively affect comprehension and maintenance effort, and that high coverage alone does not guarantee test quality [7, 8]. Automated test generation is particularly prone to smell diffusion. Prior work reports that EvoSuite-generated suites frequently exhibit smells such as Assertion Roulette and Eager Test [47, 50], with large-scale studies confirming substantial smell prevalence across Defects4J and SF110 [40]. LLM-generated tests exhibit a different but equally non-trivial smell profile, often introducing patterns such as General Fixture, Duplicate Assert, or Lazy Test, with prevalence varying across prompts and context configurations [43]. Overall, both SBST- and LLM-based approaches produce smell-heavy suites, albeit with distinct distributions. This motivates refactoring strategies that target structural clarity and maintainability at the suite level and may indirectly reshape smell profiles as a by-product of improved design. J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

2.4

Code Summaries as LLM Context

Source code summarization aims to produce concise natural-language descriptions of code elements to support comprehension and maintenance [1, 27, 62]. While early approaches relied on neural sequence models [27], recent work leverages LLMs that generate context-aware summaries in zeroor few-shot settings [2, 14]. Summary quality is commonly evaluated using lexical and semantic similarity metrics, as well as code-aware measures such as SIDE [36]. In this study, we use SIDE as a code-aware alignment signal to select summaries that better reflect the functionality of the corresponding classes. These summaries serve as compact, semanticsaware context in our summary-based configuration: instead of providing the full class under test, we supply a condensed behavioral description. This design mitigates known limitations of longcontext prompting—such as the “lost-in-the-middle” effect and position-dependent performance degradation [26, 34]—while preserving essential semantics. By grounding refactoring in highquality summaries rather than raw full-class code, we aim to improve robustness and focus the LLM on the most relevant behavioral aspects. 3 3.1

Experimental Setup Approach Overview

TestHumanizer is designed as a post-processing layer on top of SBST tools such as EvoSuite, incorporating validation gates at each step to ensure that refactored suites remain syntactically correct and semantically faithful. Given an automatically generated test suite for a target class 𝐴, it produces a refactored suite 𝐴′ that aims to be more readable and maintainable while preserving behavior (Figure 1). The EvoSuite-generated suite (A_ESTest.java) and its companion scaffolding file (A_ESTest_scaffolding.java), which contains EvoSuite-specific JUnit runners and configuration annotations, serve as input to TestHumanizer. The pipeline then proceeds as follows. ❶ Parse and Extract. TestHumanizer parses A_ESTest.java and extracts two components: the immutable static elements (package and import declarations, annotations, class declaration) that must remain unchanged throughout refactoring, and the refactorable test body containing the individual test methods. ❷ Code Summarization (summary-based configuration only). The class 𝐴 is summarized by the same LLM used for refactoring. We compute SIDE [36] between the candidate summary and the class source code as a code-aware alignment signal; if SIDE< 50, we regenerate up to three times and retain the highest-scoring candidate, preferring one with SIDE≥ 50 when available. This yields a compact and semantically aligned description of 𝐴. ❸ Test Refactoring. We build a refactoring prompt that includes the immutable static elements and the original test body, and supply one of three context granularities: the tests alone (tests-only), the tests with the full class source code (code-centric), or the tests with the class summary (summary-based). The LLM is instructed to refactor the entire suite for readability, maintainability, and modularity—including Given–When–Then structuring, descriptive renaming, and grouping of related logic—while strictly preserving the static elements and observable test behavior. ❹ Embedding-Based Similarity Check. To discourage drastic semantic drift, we compute cosine similarity between embeddings of 𝐴 and 𝐴′ using CodeBERT. If similarity< 0.8, we retry up to three times and retain the candidate with the highest similarity, even if it remains below the threshold. ❺ Composite Semantic Screening. We compute CTSES [44] as a composite semantic screening signal integrating CodeBLEU, METEOR, and ROUGE-L. If CTSES< 50, we similarly retry up to three times and retain the best-scoring candidate. Overall, TestHumanizer couples SBST robustness with LLM-driven suite-level refactoring. The SIDE-guided summarization, embedding-based similarity check, and CTSES screening together J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al. 2 Code Summarization and Evaluation

Regenerate Java Class Description

≥ 50%

Source Code

Java Class Description Candidate

LLM

Test Suite

< 50% AND 3 regenerations SIDE < 50% AND <3 regenerations

SIDE Score

SIDE Model

Retrieve Best Description

Java Class Description A_ESTest.java

SBST Tool A_ESTest_scaffolding.java

LLM

Raw test

Test Generation

Parse and Extract Test Code Test Suite

3 Test Refactoring

Prompt LLM again

≥ 0.8 < 0.8 AND 3 regenerations

Embeddings and Similarity Calculation

Cosine Similarity

< 0.8 AND <3 regenerations

CodeBERT

Retrieve Best Test

5 Semantics-Driven and Behavior-Preserving Evaluation

≥ 50% < 50% AND 3 regenerations

CTSES Score

< 50% AND <3 regenerations

Retrieve Best Test

Final Test Suite

Fig. 1. Overview of the general workflow of TestHumanizer. form a multi-layered validation pipeline that balances developer-oriented improvements with behavioral faithfulness. 3.2

Research Questions

RQ1 (Feasibility and robustness): How feasible and robust is TestHumanizer in producing syntactically correct and compilable refactored unit test suites? This research question assesses the viability of TestHumanizer as an LLM-based refactoring layer on top of EvoSuite. We distinguish ❶ feasibility, by measuring the proportion of EvoSuite test suites that can be successfully refactored without syntax or compilation errors across Defects4J and SF110, and ❷ robustness, by analyzing how compilability varies across LLMs and context configurations. The goal is to determine whether TestHumanizer can reliably produce behavior-preserving refactorings at scale, rather than fragile or ad hoc transformations. RQ2 (Readability and understandability): To what extent does TestHumanizer improve the readability and understandability of EvoSuite-generated test suites? This research question examines the developer-perceived quality of refactored tests along two dimensions. We quantify ❶ readability using the machine-learned readability model of Scalabrino et al. [54], previously validated as a strong proxy for developer-perceived readability, and ❷ understandability using two complementary structural indicators: Cyclomatic Complexity to capture control-flow complexity, J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

and CCTR, a test-aware cognitive complexity metric specifically designed for unit tests [45]. By comparing EvoSuite tests with their TestHumanizer counterparts, we investigate whether LLMbased refactoring yields test suites that are easier to read and interpret. RQ3 (Structural and semantic similarity): To what extent do TestHumanizer refactorings remain structurally and semantically close to the original EvoSuite tests? Since TestHumanizer is intended to improve readability without altering test intent, this research question examines how closely refactored tests align with their originals at the lexical, structural, and semantic levels. We combine ❶ lexical and structural similarity metrics (CodeBLEU, METEOR, and ROUGE-L) with ❷ embedding-based semantic similarity using CodeBERT, GraphCodeBERT, and OpenAI’s text-embedding-3-small, and ❸ the composite CTSES score [44], which integrates CodeBLEU, METEOR, and ROUGE-L into a human-aligned composite signal specifically designed to evaluate LLM-based test refactorings. Together, these metrics allow us to assess whether structural improvements come at the cost of semantic drift. Note that behavioral preservation in terms of code coverage is examined separately in RQ4, which provides a dynamic, execution-based complement to the static similarity analysis conducted here. RQ4 (Code coverage impact): How does TestHumanizer impact the structural coverage of the original EvoSuite-generated tests? While refactoring should ideally be behavior-preserving, it may inadvertently affect the exercised paths or assertions. This research question compares structural coverage (line, instruction, and method coverage) achieved by EvoSuite suites before and after refactoring with TestHumanizer, across all context configurations. We examine whether LLM-based refactoring maintains, improves, or degrades coverage, and how coverage changes relate to the readability and understandability improvements observed in RQ2. RQ5 (Test smells): Does LLM-based refactoring reduce test smells in EvoSuite-generated test suites? Building on prior evidence that EvoSuite tests often exhibit smells such as Assertion Roulette, General Fixture, and Redundant Assertions, this research question investigates whether TestHumanizer effectively mitigates these issues. We measure ❶ overall smell prevalence and ❷ persmell trends (e.g., reduction, preservation, or introduction of specific smell types) before and after refactoring, using a state-of-the-art smell detector. The results indicate whether TestHumanizer not only restructures tests syntactically but also improves their design quality. RQ6 (Human perception and practical usability): How do developers perceive TestHumanizerrefactored tests compared to original EvoSuite tests in terms of readability and willingness to adopt them in practice? This research question is addressed through a developer study where participants compare EvoSuite and TestHumanizer versions of the same test suites, rating ❶ perceived readability and clarity of intent and ❷ willingness to adopt the tests in real projects. These ratings complement RQ1–RQ5 by grounding our findings in human perception and practical usability. 3.3

Large Language Models and Prompt design

Models and decoding settings. We use gpt-4o as our primary LLM across the full evaluation pipeline (class summarization, suite refactoring, and direct test generation baselines). For RQ1 (feasibility and compilability), we additionally include mistral-large-2407 to contrast provider behavior under the same prompting and pipeline settings. We set temperature to 0.1 and keep the provider-default top_p to favor low-variance outputs while allowing limited lexical variation (e.g., naming and phrasing). This controlled setup isolates the effect of our pipeline and prompt design from decoding-specific variability. J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

Chain-of-Thought prompting. Our prompts follow a Chain-of-Thought (CoT) structure that explicitly decomposes each task into intermediate steps (e.g., identify intent, analyze dependencies, refactor, and verify behavior) rather than requesting a direct output. Prior work shows that such stepwise prompting can improve multi-step reasoning and transformations [66, 67], and it has been adopted in LLM-based test generation/refactoring [16, 21, 23] as well as code summarization [62]. For scalability, we use single-sample CoT (no self-consistency), retaining the step-by-step structure to steer the model toward structured, behavior-preserving edits. Class summarization prompt. For the summary-based configuration of TestHumanizer, we generate a natural-language summary for each class under test. The summarization prompt is inspired by Sun et al. [62] and instructs the model to progressively analyze the class (overall purpose, key methods and fields, relationships between methods, and section-wise descriptions). We report the full template below; placeholders such as {class_code} are instantiated at runtime. Prompt: Class summarization for TestHumanizer You are an expert software engineer with advanced knowledge of Java programming. Your task is to summarize the following Java class, focusing on its functionality, the relationships between methods, and the roles of key variables. Approach the task in a structured way, addressing each element step-by-step. 1. Overall Purpose: Briefly explain the primary function of the class, including its use case within a larger project if identifiable. 2. Key Elements Identification: Identify and describe the purpose of: - Main methods: Explain the role of each significant method, focusing on its purpose, input parameters, and return type. - Important variables: Highlight any essential fields, their roles, and how they interact within the methods. 3. Method Relationships: For methods that are closely related (e.g., helper methods, overloaded methods), group and describe their relationship to provide context on how they collectively support the class’s functionality. 4. Summarize by Sections: - For each logical section of the class, generate a concise, human-readable summary in 2–3 sentences, ensuring clarity on how to use the class effectively. - If the class contains distinct groups of related methods, summarize them together to clarify their combined role in achieving a particular function. Each step is designed to build toward a coherent and context-rich explanation of the class’s functionality. Here is the class to summarize: {class_code} Generate the summary in 2–3 sentences for each method or logical section of the class. If the class contains multiple related methods, group them in the summary to explain their relationship. Please ensure the complete summary is placed between triple backticks “‘ “‘ for easy extraction.

These summaries serve as compact, semantics-aware context for the summary-based refactoring scenario (Section 2.4), mitigating token-limit and “lost-in-the-middle” issues compared to prompting on the full class code. Refactoring prompts for TestHumanizer. TestHumanizer uses three closely related CoT refactoring prompts that differ only in the provided context: tests-only (EvoSuite suite only), code-centric (suite + full class source code), and summary-based (suite + class summary). All variants share the same core design: a Java-testing expert persona, explicit constraints to preserve EvoSuite scaffolding and observable behavior, and objectives to refactor the entire suite for readability, maintainability, and modularity, including Given–When–Then comments to make setup, action, and expected outcomes explicit. We report the full summary-based prompt below; the other variants follow the same template and are available in our replication package. Prompt: Summary-based refactoring with TestHumanizer You are an expert software engineer with advanced knowledge of Java testing and refactoring. The following test suite was generated by EvoSuite and must be refactored for readability, maintainability, and modularity without altering functionality. For each class, you must refactor the entire test suite (all the test methods) and provide the complete refactored suite at the end. The class summary is provided to support contextual refactoring. Leverage this summary to refine the test suite effectively. Class Summary: {source_code_summary} Follow the steps below: Constraints: - Do Not Alter: Retain EvoSuite-specific elements (package/import statements, annotations, and class declaration): {static_part} - Preserve Functionality: Ensure no changes to the test behavior. - Add Given-When-Then Comments: Use structured comments to clarify each test’s context and purpose. Steps:

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

1. Understand Test Intent: Briefly describe the test class’s purpose, focal functions, and relevant context. 2. Analyze Dependencies: Identify dependencies and group related methods to assess opportunities for reducing complexity. 3. Refine Test Methods: - Rename Methods and Variables: Use descriptive names for methods and variables to reflect their role clearly. Add Given-When-Then Comments: Structure comments as follows: - Given: Describe setup/preconditions. - When: Specify the action or method being tested. - Then: Summarize expected results. 4. Verify and Review: Ensure refactoring aligns with EvoSuite constraints and retains functional integrity. Test Suite to Refactor: {test_code} Generate the refactored test code within triple backticks “‘ for easy extraction.

The corresponding tests-only and code-centric prompts differ only in the context block that precedes the steps (no additional context vs. full class source code). Their full templates are included in our replication package. Direct test generation prompts. To establish a direct LLM generation baseline for comparison with TestHumanizer refactorings, we design CoT prompts that treat the model as a professional Java tester, following the prompt design validated in our prior large-scale study on LLM-based test generation [40]. The prompt instructs the model to (i) extract and list public methods, (ii) generate basic JUnit 4 tests for each method, (iii) identify edge cases and exception scenarios from the class source code, and (iv) merge all test cases into a single {class_name}Test.java file, returned between triple backticks. This baseline reflects a realistic one-shot LLM generation scenario and allows us to isolate the added value of the TestHumanizer refactoring pipeline over standalone LLM generation. The full prompt is available in the replication package alongside example instances. 3.4

The SBST tool

For the SBST baseline, we use EvoSuite [18], a widely adopted and empirically validated tool for automated unit test generation in Java. EvoSuite is consistently among the strongest baselines in SBST evaluations and competitions, and it is commonly used as a reference point in large-scale studies and recent SBST–LLM pipelines when benchmarking feasibility and structural coverage [28, 31, 48, 57]. These results motivate our choice of EvoSuite as the baseline generator for TestHumanizer. We run EvoSuite with its standard configuration, including DynaMOSA [49], and allocate three minutes per class. Following prior work [4, 20, 57, 63], we execute 15 independent runs per class to mitigate Genetic Algorithm randomness; each run produces one JUnit suite, so all reported counts and rates are computed at the suite level for consistency with our LLM-based baselines (one suite per model–prompt instance). To characterize raw SBST output and avoid confounding effects, we disable EvoSuite’s optional post-generation optimizations (e.g., minimization and clean-up). This ensures that TestHumanizer operates on unrefined suites and that comparisons reflect the underlying generation strategies rather than tool-specific post-processing. 3.5

Datasets

We evaluate TestHumanizer on two widely used Java benchmarks: Defects4J [29], which contains real faults from open-source projects and is a standard baseline for automated testing research, and SF110 [19], a curated collection of 110 Java projects widely used to study SBST at scale. Both datasets are commonly adopted in prior work on SBST and LLM-based testing [20, 40, 49, 57]. We analyze a subset of production classes selected to span diverse sizes and complexities while remaining within practical LLM context limits: 147 classes from 15 Defects4J projects and 203 classes from 69 SF110 projects. Table 1 summarizes token and LOC statistics and the average number of methods per class, capturing the variability in class granularity that TestHumanizer must handle. 3.6

Implementation and Configuration

All experiments are executed on a dedicated server (AMD EPYC 7552, 48 cores, 640 GB RAM) to ensure that observed timeouts and failures reflect tool or API behavior rather than resource J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

Table 1. Overview of the sampled classes from Defects4J and SF110. Dataset

#Projects

#Classes

Max Tokens

Min Tokens

Average Tokens

Max Min Loc Loc

Average Loc

Total Locs

Average Methods Per Class

Defects4J SF110

15 69

147 203

27896 27968

156 151

1958.19 3462.57

945 2879

169.40 294.14

373530 895664

17.38 27.82

4 4

constraints. EvoSuite is configured as described in Section 3.4. For each selected class (Section ??), we run EvoSuite 15 times with DynaMOSA and a three-minute budget per run, producing 15 independent suites per class. LLM-based components are implemented as Python clients calling gpt-4o with a fixed decoding setup (temperature = 0.1, provider-default top_p; Section ??). We generate direct LLM-based suites from the class source code (15 runs per class) and refactor every EvoSuite suite under the three TestHumanizer context configurations (tests-only, codecentric, summary-based), yielding 15×3 = 45 refactorings per class and 31,500 refactored suites overall across 350 classes. For RQ1 only, we additionally run the same feasibility/compilability pipeline with mistral-large-2407 to contrast provider behavior under identical prompting and configuration. 3.7

Metrics and Evaluation

To assess the impact of TestHumanizer on structural quality, semantic preservation, behavioral correctness, and perceived readability, we employ a comprehensive multi-dimensional evaluation protocol combining static metrics, semantic similarity, dynamic coverage analysis, smell detection, and human assessment. 3.7.1 Cyclomatic and Cognitive Complexity. Cyclomatic Complexity (CC) [37] estimates the number of linearly independent execution paths in a program and is defined as: 𝐶𝐶 = 𝐸 − 𝑁 + 2𝑃

(1)

where 𝐸 is the number of edges in the control-flow graph, 𝑁 the number of nodes, and 𝑃 the number of connected components. In our experiments, we compute cyclomatic complexity using PMD’s1 implementation (aligned with the McCabe definition), as commonly used in static-analysis toolchains. 3.7.2 CCTR: Test-Aware Cognitive Complexity. To better capture test-specific structural complexity, we use CCTR [45], a cognitive complexity metric tailored for unit tests. CCTR is defined as: CCTR = 𝛼𝑁 + 𝛽𝐴 + 𝛾𝑀 + 𝛿𝑇

(2)

where 𝑁 denotes control-flow nesting complexity (as in Sonar Cognitive Complexity2 ), 𝐴 counts as-

sertions and fail() statements, 𝑀 captures mocking-related constructs (e.g., mock(), verify(), when()), and𝑇 represents annotation-based signaling (e.g., @Test, @BeforeEach, @Parameterized Test). Following the original formulation [45], we adopt uniform weights (𝛼 = 𝛽 = 𝛾 = 𝛿 = 1). Overall, CCTR complements traditional control-flow metrics by incorporating test-specific signals that affect comprehension effort. 3.7.3 CTSES: Composite Refactoring Similarity. To quantify semantic and structural preservation between an EvoSuite test suite and its refactored counterpart, we use CTSES [44], a composite similarity score defined as: CTSES = 𝛼 · CodeBLEU + 𝛽 · METEOR + 𝛾 · ROUGE-L, 1 https://pmd.github.io/

2 https://www.sonarsource.com/resources/cognitive-complexity/

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

𝛼 + 𝛽 + 𝛾 = 1.

(3)

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

In our experiments, we report three variants: the uniform baseline CTSESAVG with (𝛼, 𝛽, 𝛾) = ( 13 , 13 , 31 ), as well as the two profiles proposed in [44], namely CTSES_1 (semantic-prioritized, (0.5, 0.3, 0.2)) and CTSES_2 (readability-aware, (0.4, 0.3, 0.3)). CodeBLEU captures syntactic and data-flow similarity, METEOR provides synonym-aware lexical matching, and ROUGE-L measures structural alignment via longest common subsequence. 3.7.4 Embedding-Based Cosine Similarity. To further assess semantic equivalence, we compute cosine similarity between each original test suite (𝐴𝑖 ) and its refactored counterpart (𝐵𝑖 ) using embeddings from three models: CodeBERT, GraphCodeBERT, and OpenAI text-embedding-3-small. Given embedding vectors 𝐴𝑖 and 𝐵𝑖 , cosine similarity is defined as: Í𝑛 𝑖=1 𝐴𝑖 𝐵𝑖 Cosine Similarity = √︃ (4) Í𝑛 2 √︃Í𝑛 2 𝑖=1 𝐴𝑖 × 𝑖=1 𝐵𝑖 High cosine values indicate strong semantic alignment while being robust to renaming and formatting changes. 3.7.5 SIDE for Class Summarization Filtering. For the summary-based configuration, we use SIDE (Summary alIgnment to coDe sEmantics) [36] to assess whether a generated class-level summary is suitable for the code it describes, without requiring any reference description. SIDE is a contrastivelearning metric that returns an alignment score in [−1, 1], where higher values indicate stronger semantic agreement between the summary and the code. For each class, we generate up to three candidate summaries and compute SIDE between each candidate and the class source code. We normalize SIDE to a percentage scale and accept a summary once it reaches 50%; if none does, we retain the highest-scoring candidate. This best-effort selection reduces noisy context while ensuring that the summary-based pipeline always has a usable summary for downstream refactoring. 4 4.1

Experimental Results [RQ1]: Feasibility and compilability of TestHumanizer refactorings.

Experiment Design: For each class sampled from Defects4J and SF110 (Section 3.5), EvoSuite generates 15 test suites (Section 3.4). Each suite is refactored with TestHumanizer under three context configurations (testsonly, code-centric, summary-based) using both gpt-4o and mistral-large-2407, yielding 15 × 3 × 2 refactored suites per class. In parallel, we generate 15 direct LLM-based test suites per class and per model as LLM-only baselines (Section 3.3). For the summary-based configuration, each class is summarized by the same LLM used for refactoring; up to three candidate summaries are generated and the one with the highest SIDE score is retained (Section 3.1). All LLM-produced suites are first parsed with Tree-sitter3 to ensure syntactic validity, then compiled within the corresponding project build environment. The unit of analysis is a single test suite file. For each dataset, model, and configuration, we report: (i) the feasibility rate (percentage of prompts yielding syntactically valid suites), (ii) the compilation rate (percentage of syntactically valid suites that compile successfully), and (iii) the average number of test methods per suite as a structural proxy. Experiment Results: Table 2 summarizes feasibility and compilation rates across paradigms, models, and datasets (detailed statistics in Tables 10 and 11). As expected, EvoSuite achieves 100% feasibility and compilation on both datasets, confirming its robustness as an SBST baseline, though at the cost of comparatively large suites (17.38 test 3 https://github.com/tree-sitter/tree-sitter [Accessed: June 2026]

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

Table 2. Generation and refactoring success rates and suite complexity. Dataset

Paradigm Generation

Approach

Model

Refactoring

Generation

99.77 95.24 100.00

68.48 51.50 100.00

8.58 8.06 17.38

gpt-4o mistral-large-2407 gpt-4o Code-centric mistral-large-2407 gpt-4o Summary-based mistral-large-2407

85.09 85.46 84.49 83.93 95.00 92.94

90.66 90.64 89.91 88.52 97.24 92.92

14.37 15.50 14.29 15.65 14.16 15.54

gpt-4o mistral-large-2407 EvoSuite

98.52 97.87 100.00

78.00 77.83 100.00

8.07 5.52 27.82

gpt-4o mistral-large-2407 gpt-4o Code-centric mistral-large-2407 gpt-4o Summary-based mistral-large-2407

77.09 76.58 75.58 74.87 95.82 93.16

92.41 91.95 90.10 89.02 98.69 96.99

18.63 18.85 18.32 18.60 18.59 18.77

EvoSuite

Raw generation EvoSuite Tests-only

SF110 Refactoring

Test Methods Average

gpt-4o mistral-large-2407 EvoSuite

Raw generation

Tests-only

Defects4J

Feasibility Compilation Rate (%) Rate (%)

methods per class on Defects4J, 27.82 on SF110). Direct LLM-based generation, by contrast, exhibits very high syntactic feasibility but substantially lower compilability. On Defects4J, gpt-4o and mistral-large-2407 produce syntactically valid tests in 99.77% and 95.24% of cases, yet only 68.48% and 51.50% compile successfully. On SF110, feasibility remains high (98.52–97.87%) while compilation drops to 78.00% and 77.83%, respectively. These results confirm that one-shot LLM generation alone does not scale reliably, motivating TestHumanizer’s refactoring-based approach as a more robust alternative. TestHumanizer substantially improves compilation stability over direct LLM generation across all configurations. In the tests-only and code-centric settings, feasibility ranges between 75–85% with compilation rates between 88–92%. The summary-based configuration consistently achieves the best results: with gpt-4o, feasibility reaches 95.00% and compilation 97.24% on Defects4J, and 95.82% and 98.69% on SF110. mistral-large-2407 follows the same trend, exceeding 92–93% feasibility and 96–97% compilation in the summary-based setting. Importantly, refactored suites remain smaller than EvoSuite originals (typically 14–19 methods versus 17–28), showing that structural simplification does not come at the expense of compilability. A practical threat to feasibility in the code-centric configuration with gpt-4o is repeated API-level input rejection (Error code: 400, “issue with repetitive patterns in your prompt”, invalid_prompt).45 Such rejections arise when long prompts contain strongly templated or repetitive content. Despite preprocessing, full-class prompts (class source + structured refactoring instructions) remain the longest and most repetitive inputs, making them most susceptible to this failure mode. This phenomenon was not observed with mistral-large-2407, suggesting provider-specific validation heuristics. Overall, the summary-based configuration is the most robust: compressing class context into a compact semantic representation reduces API-level rejections and achieves the highest compilation stability across datasets and models. Finding 1: TestHumanizer successfully refactors compilable SBST suites in the vast majority of cases, achieving compilation rates of 88–98% across configurations and datasets—substantially higher than direct LLM generation (51–78%). The summary-based configuration is consistently the most feasible and robust across both LLMs and datasets. In the code-centric setting, gpt-4o is additionally affected by API-level prompt rejections (HTTP 400) triggered by repetitive prompt patterns, a failure mode not observed with mistral-large-2407. 4 https://community.openai.com/t/error-code-400-for-repetitive-prompt-patterns/627157 [Accessed: June 2026] 5 https://community.openai.com/t/repetitive-prompt-error-400/716240 [Accessed: June 2026]

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

To characterize compilation failures, we analyzed their error categories across all non-compiling refactorings produced by gpt-4o, focusing on three interpretable error types that directly reflect LLM hallucination patterns: Cannot Find Symbol, Incompatible Types, and Package Does Not Exist. Figure 2 reports their occurrence counts across the three context configurations. (a) Defects4J (b) SF110 Number of compilation error occurrences

140

2500

2,395 2,428

120

2,287

Package Does Not Exist not observed in SF110

118 98

100

2000

1500

60 1000

500

299 308 294

Cannot Find Symbol

Incompatible Types

183 183 183

Package Does Not Exist Tests-only

20 0

Configuration Code-centric

Cannot Find Symbol

Incompatible Types

Summary-based

Fig. 2. Compilation error breakdown across context configurations (gpt-4o). On Defects4J, Cannot Find Symbol is the dominant error category across all three configurations (2,287–2,428 occurrences), followed by Incompatible Types (294–308) and Package Does Not Exist (183, stable across configurations). Crucially, the summary-based configuration consistently yields the lowest counts across all three categories, with Cannot Find Symbol dropping from 2,428 (code-centric) to 2,287, and Incompatible Types from 308 to 294. On SF110, the reduction is more pronounced: Cannot Find Symbol falls from 118 (code-centric) to 61 in the summary-based setting, and Incompatible Types drops from 81 (tests-only) to 25—a threefold decrease. Package Does Not Exist was not observed in SF110. These patterns are consistent with identifier-level LLM hallucinations: Cannot Find Symbol arises when the LLM alters class or constructor identifiers and method calls that are present in the original suite; Incompatible Types reflects hallucinated type signatures or return types; and Package Does Not Exist indicates invented or incorrectly modified import statements. Importantly, providing the full class source in the code-centric configuration does not prevent these failures—some errors involve identifiers explicitly present in the input, suggesting over-aggressive transformation rather than missing context. The consistently lower error counts of the summary-based configuration indicate that a compact, abstract class representation implicitly constrains the LLM’s transformation space, reducing opportunities for identifier-level drift. Finding 2: Compilation failures are predominantly caused by identifier-level LLM hallucinations. Cannot Find Symbol is the dominant error category on Defects4J (2,287–2,428 occurrences across configurations), while on SF110 both Cannot Find Symbol and Incompatible Types drop substantially in the summary-based setting (from 118 to 61 and from 81 to 25, respectively). Providing the full class context (code-centric) does not reduce these failures; the compact summary-based context consistently yields the fewest hallucination-induced errors across both datasets. Beyond compilability, we analyze the quality of the class summaries used in the summary-based configuration through SIDE, a code-aware summary–code alignment metric [36]. As shown in Table 3, gpt-4o achieves average SIDE scores of 0.56 on Defects4J and 0.61 on SF110, whereas mistral-large-2407 reaches lower averages of 0.28 and 0.26, respectively. In our pipeline, each class is summarized up to three times and we retain the highest-scoring candidate, making SIDE J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

Table 3. Summary length statistics and SIDE score distribution across datasets and models. Dataset Defects4J SF110

Model Token Min Token Max SIDE Min SIDE Max SIDE Avg gpt-4o 285 1262 -0.504 0.998 0.563 mistral-large-2407 134 1412 -0.736 0.999 0.282 gpt-4o 321 1208 -0.625 0.990 0.605 mistral-large-2407 187 1964 -0.799 0.983 0.261

a relative selection signal rather than an absolute quality threshold. SIDE was originally trained on short, Javadoc-style method summaries; our setting involves multi-sentence class-level summaries, an out-of-distribution scenario in which we observed occasional instance-level inconsistencies (e.g., similar summaries receiving divergent scores). We therefore interpret SIDE in aggregate as a lightweight alignment filter rather than a ground-truth oracle. This cautious use is consistent with recent findings by Sun et al. [62], who report weak correlations between traditional lexical metrics and human judgments for LLM-generated summaries. The high feasibility and compilation rates of the summary-based configuration confirm that SIDE-guided selection provides sufficiently coherent context for reliable refactoring. Finding 3: gpt-4o produces class summaries more semantically aligned with the source code than mistral-large-2407 (SIDE avg: 0.56–0.61 vs. 0.26–0.28). Used as a relative selection filter rather than an absolute threshold, SIDE-guided summarization provides sufficiently coherent context for the summary-based configuration to remain the most stable and compilable variant of TestHumanizer across both datasets. Our RQ1 findings are consistent with prior evidence that LLM-based post-processing of generated tests can improve their quality, yet still produces a non-negligible fraction of non-compiling outputs requiring explicit safeguards. Biagiola et al. [11] show that even conservative transformations focused on naming can occasionally break compilation, motivating feasibility filters in refactoring pipelines. Deljouyi et al. [16] report similar issues in an SBST+LLM setting and rely on systematic compilation verification with repair/rollback. Our study extends these observations by showing that broader, suite-level refactoring starting from compilable EvoSuite suites can achieve high compilation rates (88–98%)—well above direct LLM generation (51–78%) and approaching EvoSuite’s compilability as an upper-bound reference—while also exposing practical feasibility threats specific to long, code-centric inputs (API-level prompt rejections and hallucination-induced edits). These threats further motivate compact context designs such as our summary-based configuration. Summary of RQ1: EvoSuite achieves 100% feasibility and compilation on both datasets but produces comparatively large suites. Direct one-shot LLM generation remains brittle, with compilation rates as low as 51–78%. TestHumanizer, operating as a refactoring layer over compilable EvoSuite suites, achieves substantially higher compilation stability: refactored suites compile in 88–98% of cases across configurations and datasets, with the summary-based variant consistently the most stable. Two main threats to feasibility are identified: (i) API-level prompt rejections (HTTP 400) affecting long code-centric prompts with gpt-4o (not observed with mistral-large-2407), and (ii) hallucination-induced edits in non-compiling refactorings (e.g., unintended changes to referenced identifiers or imports), which are less frequent in the compact summary-based setting. SIDE-guided summarization yields higher average summary– code alignment for gpt-4o than mistral-large-2407; used as a relative selection filter, it contributes to the robustness of the summary-based configuration in practice. 4.2

[RQ2]: Impact of TestHumanizer on readability and understandability.

Experiment Design: For each EvoSuite suite and its corresponding TestHumanizer-refactored variants, we assess two dimensions of test quality as defined in RQ2 (Readability and Understandability). We use the J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

machine-learned readability model of Scalabrino et al. [54] as a proxy for developer-perceived readability [42, 56]. This model combines structural and lexical features and has been validated as a strong predictor of human readability judgments. We use two complementary structural indicators as proxies for cognitive effort: (i) Cyclomatic Complexity (CC) [37], which captures control-flow complexity, and (ii) CCTR, a test-aware cognitive complexity metric specifically designed for unit tests [45], accounting for assertion density, annotations, and test structuring patterns. Lower values on both indicators reflect suites that place less cognitive demand on developers. We compute all three metrics on: (a) original EvoSuite suites, (b) TestHumanizer-refactored suites for each context configuration (tests-only, code-centric, summary-based) using gpt-4o, and (c) direct gpt-4ogenerated test suites. The unit of analysis is a single test suite file. Experiment Results: Figures 3, 4, and 5 show the distributions of readability, Cyclomatic Complexity (CC), and CCTR across paradigms on Defects4J and SF110. 0.8

0.8

Readability

0.6

0.4

0.2

0 S Evo

uite

gpt-

nly ts-o Tes

sed ntric y-ba e-ce mar Cod Sum

Evo

e Suit

gpt-

Approach

(a) Defects4J

200

Cyclomatic

Approach

(b) SF110

Fig. 3. Readability with Scalabrino.

120

nly se ntric ts-o y-ba e-ce Tes mar Cod Sum

100

150

80 60

100

40 20

50 0

0 uite

S Evo

gpt-

nly sed ntric ts-o y-ba e-ce Tes mar Cod Sum

Evo

e Suit

Approach

(a) Defects4J

gpt-

nly se ntric ts-o y-ba e-ce Tes mar Cod Sum

Approach

Fig. 4. Cyclomatic Complexity.

(b) SF110

All three TestHumanizer configurations with gpt-4o outperform both EvoSuite and direct LLM-generated tests on predicted readability. On Defects4J, the mean Scalabrino score increases from 0.56 (EvoSuite) to 0.67 (direct gpt-4o), and further to 0.74–0.75 for TestHumanizer, with the summary-based variant achieving the highest mean (0.75). A consistent pattern holds on SF110: EvoSuite averages 0.51, direct gpt-4o reaches 0.64, and all TestHumanizer configurations exceed 0.73, again with summary-based slightly ahead. EvoSuite suites exhibit the highest CC and CCTR on both datasets (19.8 CC and 39.5 CCTR on Defects4J; 25.6 CC and 58.6 CCTR on SF110). Direct gpt-4o tests are structurally much simpler (CC ≈ 15–16, CCTR ≈ 26–28), reflecting shorter, less branch-heavy suites that sacrifice structural richness for simplicity. TestHumanizer refactorings strike an intermediate balance: they consistently reduce both CC and CCTR relative to EvoSuite (e.g., summary-based CC ≈ 17.1 vs. J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al. 1600 1400 1200 1000 800 600 400 200 0

500

Cognitive

400 300 200 100 0 Suit

Evo

gpt-

nly sed ntric ts-o y-ba e-ce Tes mar Cod m u S

Approach

(a) Defects4J

Evo

e Suit

gpt-

nly se ntric ts-o y-ba e-ce Tes mar Cod m u S

Approach

(b) SF110

Fig. 5. Test-Aware Cognitive Complexity. 19.8 and CCTR ≈ 34.2 vs. 39.5 on Defects4J; CC ≈ 19.8 vs. 25.6 and CCTR ≈ 46.7 vs. 58.6 on SF110), while remaining more structurally rich than direct LLM outputs. Among the three configurations, summary-based yields the lowest complexity on both datasets. Taken together, these results show that TestHumanizer improves predicted readability while reducing the cognitive load of refactored suites, without collapsing into the overly minimal structure that characterizes direct LLM generation. The summary-based configuration consistently achieves the best trade-off across both dimensions. Finding 4: TestHumanizer systematically improves predicted readability (Scalabrino scores: 0.74–0.75 vs. 0.56 for EvoSuite on Defects4J) and reduces cognitive load (CC and CCTR) relative to EvoSuite across both datasets, while preserving more structural richness than direct LLM generation. The summary-based configuration of gpt-4o achieves the best overall balance between readability gains and understandability improvements. These results complement recent work by Biagiola et al. [11], who target the readability of automatically generated tests but restrict LLM transformations to identifiers and test names. Their study shows that naming-only transformations can reach readability levels comparable to developer written tests. Our findings indicate that allowing the LLM to refactor entire test structures—bodies, control flow, and assertions—yields suites that are substantially more readable and cognitively lighter than EvoSuite originals, while retaining high compilability (RQ1) and strong semantic similarity (RQ3). Summary of RQ2: TestHumanizer consistently improves predicted readability compared to EvoSuite across both datasets, with all three configurations increasing Scalabrino scores substantially and the summary-based variant of gpt-4o achieving the highest mean on both Defects4J and SF110. On understandability, EvoSuite suites exhibit the highest CC and CCTR, while direct LLM-generated tests are the simplest but often overly minimal. TestHumanizer refactorings strike an intermediate balance: they significantly reduce control-flow and cognitive complexity relative to EvoSuite, yet remain structurally richer than direct LLM outputs. The summarybased configuration provides the best trade-off between readability gains and understandability improvements across both datasets. 4.3

[RQ3]: Structural and Semantic Similarity of TestHumanizer Refactorings

Experiment Design: To assess how closely TestHumanizer refactorings align with their original EvoSuite suites at the lexical, structural, and semantic levels, we compare each compilable refactoring with its EvoSuite counterpart using a set of complementary similarity metrics. Following RQ1, we focus J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

on the gpt-4o configuration as the most stable and compilable setting. Note that behavioral preservation in terms of code execution is examined separately in RQ4 through dynamic coverage analysis; the present RQ focuses on static similarity signals. For each EvoSuite–TestHumanizer pair across the three context configurations (tests-only, code-centric, summary-based), we compute three families of similarity signals. First, lexical and structural similarity via CodeBLEU [52], METEOR [6], and ROUGE-L [32, 33], which capture token-level overlap, synonym-aware matching, and longest common subsequence alignment, respectively. Second, embedding-based semantic similarity as cosine similarity using CodeBERT [17], GraphCodeBERT [25], and OpenAI’s text-embedding-3-small6 , which are robust to surface-level renaming and reformatting. Third, CTSES [44], a composite human-aligned similarity metric that integrates CodeBLEU, METEOR, and ROUGE-L into a single score specifically designed to evaluate LLM-based test refactorings; both CTSES variants (semantic-prioritized CTSES_1 and readability-aware CTSES_2) are reported. The unit of analysis is a single suite pair, restricted to successfully compiled outputs (RQ1). For each dataset and configuration, we report distributional statistics (min, quartiles, mean, max) and the proportion of suites above or below key thresholds (e.g., 0.5). These similarity results are interpreted jointly with the readability and understandability findings from RQ2, to examine whether structural improvements come at the cost of semantic drift or whether refactoring remains closely aligned with the original suite. Experiment Results: Table 4 reports cosine similarities between original EvoSuite suites and their TestHumanizer refactorings for gpt-4o across CodeBERT, GraphCodeBERT, and OpenAI embeddings. Across both datasets and all configurations, embedding similarities are near saturation. With CodeBERT, mean similarities range from 0.9948 to 0.9962, with third quartiles above 0.9978 and minima above 0.93–0.95. GraphCodeBERT shows a comparable pattern (means 0.9857–0.9892, Q3 above 0.9922). OpenAI’s text-embedding-3-small yields slightly lower means (≈0.925–0.928) and a small tail of lower-similarity cases (minima ≈0.62–0.79), yet the overall distribution remains strongly concentrated in the high-similarity region. Differences between configurations are negligible: on Defects4J the summary-based variant attains the highest average similarity, while on SF110 codecentric is marginally ahead, but these gaps are minor relative to the overall saturation effect. Taken together, embedding-based similarity confirms that TestHumanizer’s structural and readability improvements (RQ2) do not induce substantial semantic shifts, and that code-aware embeddings alone are not discriminative enough to separate the three refactoring configurations. Finding 5: Across Defects4J and SF110, TestHumanizer refactorings for gpt-4o remain extremely close to the original EvoSuite suites in embedding space (CodeBERT, GraphCodeBERT, OpenAI), with cosine similarity means above 0.92 and often above 0.99. Embedding-based similarity thus indicates strong semantic preservation for all three context configurations, and is not discriminative enough on its own to distinguish between tests-only, code-centric, and summary-based refactorings. Table 5 reports lexical and structural similarity. Across both datasets, METEOR and ROUGE-L remain consistently high (means ≈0.68–0.70; 𝑄1 > 0.59), indicating substantial overlap in structure and wording. CodeBLEU is more conservative (means ≈0.55–0.58), flagging 18% of refactorings below 0.5 on Defects4J and up to 37% on SF110. The composite CTSES metric [44]—which integrates CodeBLEU, METEOR, and ROUGE-L into a score specifically designed for evaluating LLM-based test refactorings—provides a more stable assessment. On Defects4J, CTSES_2 averages ≈0.65, with only 4.6–6.3% of refactorings below 0.5, a threshold defined in CTSES [44] to identify refactorings 6 https://openai.com/index/new-embedding-models-and-api-updates/ [Accessed: June 2026]

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

Table 4. Cosine similarity statistics per embedding model and approach. Dataset

Defects4J

SF110

Approach

Embedding

Mean

Tests-only

CodeBERT GraphCodeBERT OpenAI

0.9496 0.9940 0.8909 0.9819 0.6191 0.9110

Min

0.9948 0.9857 0.9277

0.9978 0.9999 0.9922 0.9999 0.9471 1.0000

Code-centric

CodeBERT GraphCodeBERT OpenAI

0.9377 0.8893 0.6340

0.9955 0.9953 0.9851 0.9871 0.9085 0.9251

0.9982 0.9999 0.9936 0.9999 0.9459 1.0000

CodeBERT Summary-based GraphCodeBERT OpenAI

0.9307 0.8903 0.6250

0.9977 0.9962 0.9939 0.9920 0.9083 0.9249

0.9992 1.0000 0.9976 1.0000 0.9455 1.0000

Tests-only

CodeBERT GraphCodeBERT OpenAI

0.9326 0.9943 0.9045 0.9853 0.7897 0.9123

Code-centric

CodeBERT GraphCodeBERT OpenAI

CodeBERT Summary-based GraphCodeBERT OpenAI

Max

0.9952 0.9880 0.9279

0.9979 1.0000 0.9932 1.0000 0.9472 0.9970

0.9457 0.9003 0.7445

0.9951 0.9958 0.9869 0.9892 0.9104 0.9263

0.9982 1.0000 0.9944 1.0000 0.9463 0.9846

0.9441 0.8885 0.7819

0.9948 0.9957 0.9861 0.9888 0.9094 0.9258

0.9981 1.0000 0.9941 1.0000 0.9461 1.0000

that maintain sufficient structural and lexical alignment with the original. On SF110, CTSES_2 remains around 0.62–0.63, with 86–89% of refactorings above this threshold despite the larger and more complex classes. Differences across configurations are marginal, with summary-based slightly ahead on SF110. Together with near-saturated embedding similarities, these results indicate that TestHumanizer performs non-trivial structural improvements—renaming, reorganization, and control-flow simplification—while remaining closely aligned with the original suites in the vast majority of cases. To better characterize the limits of similarity-based evaluation, we manually inspected refactorings that simultaneously received low scores from CodeBLEU, METEOR, and ROUGE-L—cases where all three lexical signals suggest substantial surface-level divergence. This inspection revealed that low lexical similarity does not always indicate poor refactoring quality: in some instances, the LLM performs a structurally divergent yet beneficial restructuring that preserves test intent while departing strongly from the original at the token level. As a concrete example, one refactoring of the XPathLexer test suite in SF110 received CodeBLEU of 0.21, METEOR of 0.08, and ROUGE-L of 0.14—all below 0.22—because the LLM radically reorganized an original suite of over 100 noisy, duplicated methods by introducing named constants, descriptive method names, and structured comments while eliminating redundancy. The resulting suite is substantially more readable, yet its surface form is too distant from the original for lexical metrics to recognize the alignment. This case illustrates a systematic limitation: even CTSES, which mitigates some of CodeBLEU’s over-pessimism, can still penalize broad-scope refactorings that are beneficial, motivating complementary evaluation dimensions such as naming quality, documentation coverage, and smell reduction. Finding 6: Across Defects4J and SF110, TestHumanizer refactorings maintain high lexical and structural similarity to the original EvoSuite suites. Using CTSES—a composite metric designed for LLM-based test refactoring evaluation—roughly 86–95% of refactorings remain above the 0.5 alignment threshold despite non-trivial structural changes, whereas CodeBLEU alone would flag substantially more refactorings as problematic. Manual inspection further reveals that cases with simultaneously low lexical scores do not always correspond to poor refactorings: structurally divergent but beneficial restructurings can be penalized by surface-level metrics, reinforcing the need to complement similarity signals with additional quality dimensions such as naming, documentation, and smell reduction.

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

Table 5. Metric statistics per dataset and approach. Dataset

Approach

Metric

Tests-only

METEOR ROUGE-L CodeBLEU Average_score CTSES_1 CTSES_2

0.1418 0.1291 0.1900 0.1827 0.1808 0.1834

Max

<0.5%

≥0.5%

0.6192 0.6935 0.7588 0.6259 0.6964 0.7428 0.5139 0.5808 0.6092 0.5913 0.6569 0.6990 0.5715 0.6377 0.6766 0.5841 0.6493 0.6898

1.0000 1.0000 1.0000 1.0000 1.0000 1.0000

4.57 2.40 17.55 4.03 6.07 4.57

95.43 97.60 82.45 95.97 93.93 95.43

METEOR ROUGE-L CodeBLEU Average_score CTSES_1 CTSES_2

0.1457 0.6226 0.1157 0.6286 0.2114 0.5188 0.1953 0.5955 0.1952 0.5796 0.1969 0.5906

0.6841 0.7542 1.0000 0.6910 0.7500 1.0000 0.5786 0.6249 1.0000 0.6512 0.7048 1.0000 0.6327 0.6836 1.0000 0.6440 0.6959 1.0000

5.49 4.45 18.12 5.56 8.01 6.28

94.51 95.55 81.88 94.44 91.99 93.72

Summary-based

METEOR ROUGE-L CodeBLEU Average_score CTSES_1 CTSES_2

0.1693 0.6237 0.1212 0.6338 0.2186 0.5219 0.2121 0.6001 0.2098 0.5822 0.2128 0.5940

0.6875 0.7564 1.0000 0.6933 0.7518 1.0000 0.5846 0.6321 1.0000 0.6551 0.7051 1.0000 0.6372 0.6865 1.0000 0.6481 0.6974 1.0000

5.39 4.33 18.01 5.58 7.39 6.07

94.61 95.67 81.99 94.42 92.61 93.93

Tests-only

METEOR ROUGE-L CodeBLEU Average_score CTSES_1 CTSES_2

0.0757 0.1056 0.1445 0.1109 0.1182 0.1143

0.9761 1.0000 0.9693 0.9756 0.9738 0.9750

9.74 6.48 35.67 10.84 13.28 11.47

90.26 93.52 64.33 89.16 86.72 88.53

Code-centric

METEOR ROUGE-L CodeBLEU Average_score CTSES_1 CTSES_2

0.0474 0.5893 0.0964 0.6111 0.1380 0.4706 0.0979 0.5611 0.1084 0.5376 0.1031 0.5522

0.6551 0.7504 0.9458 0.6804 0.7603 0.9904 0.5553 0.6380 0.9300 0.6303 0.7102 0.9516 0.6103 0.6918 0.9430 0.6228 0.7030 0.9487

12.00 8.00 37.51 12.75 15.85 13.61

88.00 92.00 62.49 87.25 84.15 86.39

Summary-based

METEOR ROUGE-L CodeBLEU Average_score CTSES_1 CTSES_2

0.0584 0.5943 0.1070 0.6129 0.1185 0.4781 0.1102 0.5668 0.1182 0.5434 0.1145 0.5581

0.6623 0.7529 0.9716 0.6838 0.7575 0.9904 0.5628 0.6398 0.9626 0.6363 0.7101 0.9749 0.6169 0.6922 0.9709 0.6290 0.7032 0.9736

10.88 7.65 35.00 11.61 14.18 12.37

89.12 92.35 65.00 88.39 85.82 87.63

Defects4J Code-centric

SF110

Min

Mean

0.5994 0.6626 0.7488 0.6134 0.6760 0.7426 0.4753 0.5527 0.6208 0.5674 0.6305 0.6987 0.5443 0.6104 0.6784 0.5588 0.6227 0.6911

Our analysis relates to recent work on LLM-assisted improvement of generated tests. Biagiola et al. [11] restrict LLMs to renaming identifiers and test names, assessing semantic stability via cosine similarity between improved variants (OpenAI embeddings, average ≈0.9). Deljouyi et al. [16] use CodeBLEU as a safeguard: refactorings scoring below 0.5 trigger re-prompting, and those repeatedly failing are reverted to the original. In contrast, TestHumanizer applies a broader scope of transformation, allowing the LLM to modify not only identifiers and comments but also test bodies, control flow, and assertions, and therefore requires stricter semantic safeguards. We filter first on compilability (RQ1), then rely on a richer set of signals—METEOR, ROUGE-L, CodeBLEU, CTSES, and embedding-based similarity (CodeBERT, GraphCodeBERT, OpenAI)—to identify and discard refactorings with substantial semantic divergence. Despite this broader transformation scope, the vast majority of suites remain above the CTSES alignment threshold (≥90–95% on Defects4J, ≥85–90% on SF110; Table 5) while achieving near-perfect code-aware cosine similarity (Table 4), suggesting that large-scale structural refactoring of SBST tests is feasible in practice when multi-metric safeguards are in place.

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

Summary of RQ3: Embedding-based similarity (CodeBERT, GraphCodeBERT, OpenAI) indicates extremely strong alignment across both datasets, with cosine similarities near saturation and minimal differences between configurations. Lexical and structural metrics confirm this trend: METEOR and ROUGE-L remain consistently high, while CodeBLEU is more conservative. The composite CTSES metric provides a more stable estimate, with 86–95% of refactorings remaining above the 0.5 alignment threshold. Manual inspection reveals that cases with simultaneously low lexical scores do not always correspond to undesirable changes: broad-scope restructurings that preserve intent but diverge strongly at the token level (e.g., the XPathLexer case) can be penalized by surface-level metrics. Overall, TestHumanizer maintains close alignment with the original suites in the vast majority of cases, while highlighting the limits of purely similarity-based evaluation for broad-scope refactorings. 4.4

[RQ4]: Effect of TestHumanizer on code coverage.

100 80 60 40 20 0 Evo

Suit

gpt-

ed ntric -only bas e-ce arym Cod m Su

ts Tes

Instruction Coverage (%)

Experiment Design: For RQ4, we measure the structural coverage achieved by each test suite (original EvoSuite, TestHumanizer refactorings, and direct LLM-generated tests) when executed against the class under test within its project build environment—i.e., the fixed project version in which the class is defined, as provided by Defects4J and SF110. We use JaCoCo7 to collect instruction, line, and method coverage. As in previous RQs, we restrict the analysis to suites that compile successfully (RQ1) and focus on gpt-4o as the best-performing LLM identified in RQ1. For every EvoSuite suite, we measure coverage for (i) the original EvoSuite suite, (ii) the three TestHumanizer variants with gpt-4o (tests-only, code-centric, summary-based), and (iii) the direct gpt-4o test suite generated from the class source code. The unit of analysis is a single test suite file. Experiment Results: Figure 6 shows the distribution of instruction coverage. EvoSuite provides a strong SBST baseline, with tightly clustered high coverage on both Defects4J and SF110. Direct gpt-4o suites exhibit substantially lower and more dispersed coverage, with medians around 80% on Defects4J and close to 60% on SF110. In contrast, all three TestHumanizer configurations closely track EvoSuite, with the summary-based variant showing the strongest overlap and, on Defects4J, slightly exceeding EvoSuite’s median and upper quartile instruction coverage. 100 80 60 40 20 0

uite

S Evo

Approach

(a) Defects4J

gpt-

se ntric -only y-ba e-ce mar Cod Sum

ts Tes

Approach

Fig. 6. Instruction coverage.

(b) SF110

A similar trend emerges for line coverage (Figure 7). The tests-only and code-centric variants incur only modest reductions relative to EvoSuite (typically a few percentage points), while the summary-based configuration almost entirely recovers EvoSuite’s line coverage, with medians above 90% on Defects4J and above 93% on SF110. Coverage regressions below this range are rare 7 https://www.jacoco.org/jacoco/ [Accessed: June 2026]

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

in the summary-based setting, in sharp contrast to the wide dispersion observed for direct LLM generation. 100

Line Coverage (%)

100 80 60 40 20 0 Evo

Suit

gpt-

80 60 40 20 0

sed ntric -only y-ba e-ce mar Cod m u S

uite

S Evo

ts Tes

gpt-

sed ntric -only y-ba e-ce mar Cod m u S

ts Tes

Approach

(a) Defects4J

(b) SF110

Fig. 7. Line coverage.

100

Method Coverage (%)

Figure 8 reports method coverage. EvoSuite achieves near-perfect coverage on both datasets (medians at 100%). Direct gpt-4o tests cover far fewer methods, particularly on SF110 where the distribution centers well below 50%. TestHumanizer almost entirely preserves EvoSuite’s method coverage: all three configurations maintain medians at 100% on both datasets, with mean method coverage above 99.4% on Defects4J and 99.9% on SF110 for the summary-based variant (detailed statistics in Table 12). 80 60 40 20 0 Evo

Suit

gpt-

sed ntric -only y-ba e-ce mar Cod m u S

ts Tes

100 80 60 40 20 0

uite

S Evo

gpt-

sed ntric -only y-ba e-ce mar Cod m u S

ts Tes

Approach

(a) Defects4J

(b) SF110

Fig. 8. Method coverage.

Finding 7: TestHumanizer preserves EvoSuite’s high structural coverage across all three coverage types, with deviations typically within 1–2 percentage points and near-100% method coverage. Direct gpt-4o generation loses 15–40 coverage points on average depending on the metric and dataset. The summary-based configuration delivers the strongest coverage preservation, complementing the readability and understandability improvements observed in RQ2. Our coverage results also position TestHumanizer relative to recent LLM-based enhancements of generated tests. Biagiola et al. [11] apply a strict coverage-preservation regime: transformations are limited to renaming, and any JaCoCo coverage change leads to rejection of the transformed suite. UTGen [16] allows richer changes (improved comments and bodies) and monitors coverage and mutation score, but uses CodeBLEU as the main safeguard without enforcing exact coverage equality. In contrast, TestHumanizer applies broader-scope refactorings—restructuring test bodies, control flow, and comments—yet the summary-based configuration maintains instruction, line, and method coverage essentially at EvoSuite levels on both datasets. This suggests that broadscope structural refactoring of SBST tests can preserve high coverage at least as reliably as more

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

conservative LLM transformations, provided that strong semantic and compilability safeguards (RQ1–RQ3) are in place. Summary of RQ4: EvoSuite provides a strong coverage baseline with consistently high instruction, line, and method coverage across both datasets. Direct gpt-4o generation exhibits substantially lower and more dispersed coverage, often losing 15–40 percentage points depending on the metric and dataset. TestHumanizer refactorings are largely coverage-neutral: instruction and line coverage closely track EvoSuite’s distributions across all three configurations, with only marginal deviations, and method coverage remains near-perfect in the vast majority of cases. The summary-based configuration shows the strongest alignment with EvoSuite, occasionally matching or slightly exceeding its instruction and line coverage. Overall, these results indicate that LLM-based refactoring can improve readability and understandability (RQ2) while preserving the strong coverage achieved by SBST—an advantage that one-shot LLM generation, which sacrifices substantial execution reach, does not offer. Note that LLM-based generation approaches specifically designed to maximize coverage [31, 48] operate under a different paradigm and are not compared here; our claim applies specifically to the one-shot generation setting evaluated in this study. 4.5

[RQ5]: Impact of TestHumanizer on test smells.

Experiment Design: To quantify how TestHumanizer affects the smell profile of EvoSuite-generated tests, we run TsDetect [51] on all original EvoSuite suites and their gpt-4o TestHumanizer refactorings, and include direct gpt-4o generation as an additional baseline. Following recent large-scale analyses [40, 43, 50], we focus on smells that are particularly prevalent or characteristic in SBST- and LLM-based tests: Assertion Roulette (AR), General Fixture (GF), Eager Test (EaT), Lazy Test (LT), Duplicate Assert (DA), Magic Number (MG), and Dependent Test (DpT), together with the remaining TsDetect categories. TsDetect reports smells at the test-method level. We aggregate results by dataset (Defects4J, SF110) and by approach (EvoSuite, direct gpt-4o, and TestHumanizer under tests-only, code-centric, and summary-based), and compute for each (dataset, approach, smell) tuple the percentage of tests affected. We then compare EvoSuite vs. TestHumanizer to assess how refactoring alters the SBST smell distribution, and TestHumanizer vs. direct LLM generation to determine whether refactoring existing SBST suites yields a more favorable smell profile than generating tests from scratch. Since our prompts do not include explicit smell-removal instructions, any observed smell reductions are interpreted as emergent effects of the structural and readability improvements targeted by TestHumanizer. Experiment Results: Table 6 summarizes how the smell profile changes when moving from original EvoSuite suites to TestHumanizer refactorings and to direct gpt-4o-generated tests. Table 6. Percentage of tests affected by each test smell (TsDetect). Dataset

CLT

DfT

ECT

EaT

DpT

Test-only Code-centric Defects4J Summary-based gpt-4o EvoSuite

100.00 100.00 99.85 100.00 100.00

5.87 10.66 5.02 53.76 36.97

1.10 0.89 0.58 4.12 0.00

0.00 0.00 0.00 0.15 0.00

5.46 2.41 2.15 16.77 3.94

95.43 96.11 92.81 28.91 96.06

0.00 4.27 0.00 4.19 0.00 4.11 3.96 1.93 0.00 4.67

0.69 0.32 0.16 0.86 0.00

17.96 17.23 16.80 17.58 19.70

0.00 0.00 0.00 0.91 0.00

63.39 63.83 62.76 48.58 55.41

89.53 89.88 88.38 60.47 80.72

0.53 0.54 0.39 5.95 0.11

5.46 0.09 3.58 2.41 0.03 3.55 2.18 0.00 3.47 21.39 0.00 2.29 3.94 0.00 4.15

100.00 100.00 100.00 99.80 100.00

Test-only Code-centric Summary-based gpt-4o EvoSuite

100.00 100.00 99.82 100.00 100.00

8.11 11.20 4.37 41.05 48.31

1.50 1.38 1.18 1.51 0.00

0.00 0.00 0.00 0.09 0.00

0.11 0.04 0.00 24.49 0.32

99.85 99.72 99.13 14.87 99.59

0.00 0.00 0.00 5.34 0.05

1.93 1.78 1.73 2.58 2.38

0.11 0.17 0.11 1.96 0.00

11.18 11.19 11.12 13.62 9.52

0.00 0.00 0.00 0.27 0.00

29.37 44.04 28.73 43.38 28.69 43.22 52.00 46.75 63.92 84.48

4.88 4.52 4.27 5.16 4.21

0.11 0.04 0.00 32.15 0.37

100.00 100.00 100.00 99.73 100.00

SF110

Approach

0.03 1.93 0.00 1.78 0.00 1.73 0.00 3.21 0.00 2.38

On both datasets, the most pervasive EvoSuite smells remain largely unchanged after refactoring. Assertion Roulette (AR) and Magic Number Test (MT) still affect essentially all tests (≈100%) in every J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

configuration, and General Fixture (GF) stays very high: around 96% for EvoSuite and 93–96% for the three TestHumanizer variants on Defects4J, and ≈99% on SF110. This is expected, as our prompts do not explicitly target assertion redesign, fixture restructuring, or test independence. TestHumanizer does, however, substantially reduce several structural smells characteristic of SBST output. On Defects4J, Conditional Logic Test (CLT) drops from 36.97% in EvoSuite to 5.87– 10.66% in the tests-only and code-centric variants, and further to 5.02% in the summary-based configuration. By contrast, direct gpt-4o generation actually exhibits a higher CLT prevalence (53.76%), showing that TestHumanizer simplifies control flow without introducing the heavy branching that often characterizes tests written from scratch by the LLM. A consistent pattern holds on SF110, where CLT falls from 48.31% (EvoSuite) to 4.37% (summary-based) versus 41.05% for direct generation. For smells related to test focus and duplication, the effect is dataset-dependent. On Defects4J, Lazy Test (LT) and Duplicate Assert (DA) are slightly more frequent after refactoring than in EvoSuite (LT: 55.41% → 62.76%; DA: 80.72% → 88.38% in the summary-based variant), while direct gpt-4o tests exhibit lower DA (60.47%) and LT (48.58%). On SF110, the picture reverses: TestHumanizer clearly mitigates both smells, with LT dropping from 63.92% to ≈29% across all configurations (vs. 52.00% for direct generation) and DA from 84.48% to ≈43–44% (vs. 46.75%). The summary-based configuration consistently performs best on SF110, suggesting that richer contextual guidance helps the LLM reorganize long, repetitive suites into more focused tests. TestHumanizer also avoids LLM-specific smell patterns introduced by direct generation. Exception Catching Test (ECT) and Magic Number (MG) are noticeably more frequent in direct gpt-4ogenerated suites (Defects4J: ECT 16.77%, MG 3.96%) than in any TestHumanizer variant (ECT ≈2–5%, MG = 0%), while TestHumanizer still delivers the readability and understandability improvements observed in RQ2. Overall, TestHumanizer shifts EvoSuite suites toward a smell profile that is less branch-heavy and, on larger systems (SF110), substantially less lazy and less assertion-duplicated, while preserving the structural richness and coverage of SBST. Finding 8: Since our prompts do not include explicit smell-removal instructions, any smell reductions observed in TestHumanizer refactorings emerge as side effects of structural and readability improvements. The dominant EvoSuite smells—Assertion Roulette, General Fixture, and Dependent Test—remain pervasive across all configurations, as they require deep fixture redesign beyond the scope of our prompts. However, TestHumanizer substantially reduces structural smells: Conditional Logic Test drops from 36.97% to 5.02% on Defects4J and from 48.31% to 4.37% on SF110 in the summary-based setting, and Lazy Test and Duplicate Assert are markedly reduced on SF110. Compared to direct LLM generation, TestHumanizer avoids LLM-specific smell patterns such as excessive exception handling and magic numbers. Our smell results can be contextualized against prior analyses of LLM-generated tests. Siddiq et al. [60] show using TsDetect on Defects4J that LLM-generated tests exhibit substantial levels of classic smells such as Assertion Roulette and Magic Number Test, often comparable to or higher than automatically generated and developer-written tests. Ouédraogo et al. [43] extend this picture to Defects4J and SF110, reporting that EvoSuite suites are dominated by Assertion Roulette and Eager Test, while LLM-generated tests introduce relatively more General Fixture, Duplicate Assert, Lazy Test, and Empty Test. In our study, direct gpt-4o-generated suites show a similar smell-heavy profile, with near-universal Magic Number Test, pervasive Assertion Roulette, and non-trivial levels of Lazy Test and Duplicate Assert. TestHumanizer, operating as a refactoring layer on top of EvoSuite, largely preserves the underlying SBST smell profile but avoids the additional smell shifts observed in fully LLM-generated suites, and improves specific dimensions such as Conditional Logic Test and, J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

on SF110, Lazy Test. These results suggest that LLM-based structural refactoring can improve test readability (RQ2) without amplifying smell diffusion beyond what is already present in EvoSuite, and without introducing the LLM-specific smell patterns characteristic of direct generation. Summary of RQ5: Since our prompts do not explicitly target smell removal, all observed smell reductions emerge as side effects of structural and readability improvements. The dominant EvoSuite smells—Assertion Roulette and General Fixture—remain pervasive across all configurations, as they require deep fixture and assertion redesign beyond the scope of the refactoring prompts. TestHumanizer does, however, substantially reduce structural smells: Conditional Logic Test is dramatically reduced in all configurations relative to EvoSuite (from 36.97% to 5.02% on Defects4J and from 48.31% to 4.37% on SF110 in the summary-based setting), while direct LLM generation actually worsens this smell. On larger systems (SF110), TestHumanizer also markedly reduces Lazy Test and Duplicate Assert, suggesting improved test focus and reduced redundancy. Compared to direct LLM generation, TestHumanizer avoids LLM-specific smell patterns such as excessive exception handling and magic numbers, while preserving the structural richness and coverage of SBST. The summary-based configuration offers the most consistent improvements across both datasets. 4.6

[RQ6]: Developer perception and practical usability of TestHumanizer.

Experiment Design: To complement the metric-based analyses in RQ1–RQ5, we conduct a developer study comparing EvoSuite suites with their summary-based gpt-4o TestHumanizer refactorings, selected as the most robust configuration across all prior RQs. We sample 30 classes (15 Defects4J, 15 SF110), stratified by class size into three LOC bins—small (<80 LOC), medium (80–179 LOC), and large (≥180 LOC)—under the constraint that both the original and refactored suites compile successfully. The final evaluation set consists of 15 Defects4J classes (230 EvoSuite test methods) and 15 SF110 classes (214 EvoSuite test methods), for a total of 30 classes and 444 test methods (Table 7). The complete list of selected (dataset, project, class, LOC) tuples is provided in Table 13. For each class, one EvoSuite suite and its matched TestHumanizer refactoring are lightly cleaned of boilerplate while preserving all @Test methods. Four evaluators assess the suites: two senior software developers with industrial experience and two graduate students with advanced testing knowledge. Tasks are split into two balanced sets (15 classes each), with every class evaluated independently by one senior and one graduate evaluator (within-group pairing). For each class, evaluators receive a short natural-language description of the class under test and two anonymized suite versions presented in randomized order. They are asked to rate two dimensions on a 1–5 Likert scale: (i) perceived readability and clarity of intent—how easily they can understand what each test is verifying; and (ii) willingness to adopt—whether they would include the suite in a real project. Evaluators may also provide free-text comments to explain their ratings. We analyze paired ratings using Wilcoxon signed-rank tests with effect sizes (𝑟 ). Inter-annotator agreement is assessed using Krippendorff’s 𝛼 (ordinal) and quadratic weighted Cohen’s 𝜅 for each evaluator group, and reported overall across all four raters. Free-text comments are thematically analyzed to contextualize quantitative findings. Experiment Results: In the following, we refer to the summary-based gpt-4o refactoring simply as TestHumanizer for brevity. Figure 9 and Table 14 show a consistent shift in perceived readability from EvoSuite to TestHumanizer for both evaluator profiles. Graduate evaluators rate EvoSuite at a mean of 2.71 (median 2.5, IQR [2,3]) and TestHumanizer at 3.79 (median 4.0, IQR [3,4]). Senior developers are more critical of EvoSuite (mean 2.14, median 2) but similarly rate TestHumanizer higher (mean 3.50, median J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

Table 7. Overview of the human-evaluation sample by dataset and class size. Dataset

Size bin #Classes

#Test methods

Small Medium Defects4J Large Total

5 5 5 15

30 65 135 230

Small Medium Large Total

5 5 5 15

29 90 95 214

444

SF110

Overall total

Fig. 9. Distributions of readability scores from Human evaluation.

Fig. 10. Distributions of adoption scores from the human evaluation. Same layout as Figure 9. Table 8. Wilcoxon signed-rank test comparing TestHumanizer (summary-based refactoring) and EvoSuite human ratings.

Metric

Evaluator

Median Median Mean diff 𝑊 EvoSuite TestHumanizer (S–E)

𝑝 (two- Effect sided) 𝑟

Readability Advanced graduate student Adopt

2.5 3.0

4.0 4.0

1.07 0.93

30 36

0.0005 0.0008

0.71 0.68

Readability Senior developer Adopt

2.0 2.0

4.0 4.0

1.36 1.11

63 81

0.0022 0.0049

0.59 0.53

4). Overall, EvoSuite suites are judged low to neutral, whereas TestHumanizer consistently shifts ratings toward neutral to high. These gains are statistically significant: graduate evaluators 𝑊 = 30, 𝑝 = 0.0005, 𝑟 = 0.71; senior developers 𝑊 = 63, 𝑝 = 0.0022, 𝑟 = 0.59 (Table 8). A consistent pattern holds for adoption (Figure 10). Graduate evaluators report adoption scores rising from mean 2.57 (median 3.0, IQR [2,3]) to 3.50 (median 4.0, IQR [3,4]). Senior developers rate EvoSuite lower (mean 2.14, median 2) but report substantially higher adoption for TestHumanizer (mean 3.25, median 4.0). EvoSuite suites cluster around low to neutral adoption, whereas TestHumanizer shifts ratings toward neutral to high without lowering medians or quartiles. These gains are significant: graduate evaluators 𝑊 = 36, 𝑝 = 0.0008, 𝑟 = 0.68; senior developers 𝑊 = 81, 𝑝 = 0.0049, 𝑟 = 0.53. Inter-annotator agreement is substantial overall (𝛼 = 0.71–0.85; Table 9), with particularly high agreement for TestHumanizer readability (𝛼 = 0.85). One group exhibited low agreement on EvoSuite readability (𝛼 = −0.15), which is consistent with prior evidence that EvoSuite-style tests elicit divergent judgments due to their mechanical structure [3, 50]; the same group reached J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

Table 9. Inter-annotator agreement (Krippendorff’s 𝛼, ordinal) for human evaluation. Scope

Approach

Readability

Adoption

Group A

EvoSuite TestHumanizer

0.82 0.72

0.79 0.74

Group B

EvoSuite TestHumanizer

-0.15 0.94

0.56 0.94

Overall

EvoSuite TestHumanizer

0.71 0.85

0.80 0.82

near-perfect agreement on the corresponding TestHumanizer condition (𝛼 = 0.94), suggesting that refactored suites yield more stable and consensual judgments. Finding 9: Across both datasets and evaluator profiles, TestHumanizer is consistently preferred over EvoSuite on both dimensions: perceived readability improves from low/neutral to neutral/high (graduate: 𝑟 = 0.71; senior: 𝑟 = 0.59), and willingness to adopt follows the same trend (graduate: 𝑟 = 0.68; senior: 𝑟 = 0.53), with all gains significant at 𝑝 < 0.01 (Wilcoxon). Inter-annotator agreement is substantial overall (𝛼 = 0.71–0.85), and particularly high for TestHumanizer readability (𝛼 = 0.85), confirming that refactored suites elicit more consensual judgments than raw EvoSuite outputs. Free-text comments reinforce the quantitative findings. EvoSuite suites are consistently described as mechanical and auto-generated, with evaluators citing generic test names, arbitrary-looking inputs, and heavy exception scaffolding that obscures behavioral intent—perceptions well-documented in prior work [3, 15, 50]. By contrast, evaluators praise TestHumanizer suites for their explicit scenario structure, descriptive naming, and reduced boilerplate. Several comments specifically highlight the Given–When–Then organization as making tests easier to read, review, and maintain. Senior evaluators in particular emphasize improved reviewability and reduced effort to understand test coverage. A recurring caveat, shared across both evaluator groups, is that improved readability does not substitute for independent coverage and correctness validation: most evaluators state they would still cross-check assertions and edge cases before fully relying on the refactored suite in production. This indicates that TestHumanizer is perceived as a strong aid for comprehension and review, rather than a replacement for rigorous test validation. Finding 10: Qualitative feedback confirms that EvoSuite suites are perceived as mechanical and auto-generated—consistent with prior evidence [3, 50]—whereas TestHumanizer suites are praised for clearer scenario structure, descriptive naming, and Given–When–Then organization. Evaluators across both profiles nonetheless stress that readability gains do not replace independent validation of coverage and correctness, positioning TestHumanizer as a comprehension and review aid rather than a full substitute for rigorous testing. Our RQ6 results complement prior human-centered evaluations of LLM-enhanced tests. Deljouyi et al. [16] assess understandability improvements through a user study linking test quality to downstream bug-fixing performance. Biagiola et al. [11] investigate developer perception under conservative, naming-oriented transformations. In contrast, our study isolates the usability impact of suite-level refactoring on paired Likert judgments of readability and adoption (30 classes, 444 test methods), providing direct evidence that LLM-based post-processing substantially improves developer acceptance of SBST outputs, while independent behavioral validation remains necessary.

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

Summary of RQ6: In a stratified developer study on 30 classes (444 test methods), TestHumanizer significantly improves perceived readability and willingness to adopt compared to EvoSuite for both graduate and senior evaluators (Wilcoxon, 𝑝 < 0.01, medium-to-large effects: 𝑟 = 0.53– 0.71). Inter-annotator agreement is substantial overall (𝛼 = 0.71–0.85), and particularly high for TestHumanizer readability (𝛼 = 0.85). Qualitative feedback aligns with these results: EvoSuite suites are consistently described as mechanical and auto-generated, while TestHumanizer is praised for clearer scenario structure, descriptive naming, and Given–When–Then organization. Evaluators stress that readability gains do not replace independent checks of coverage and correctness, positioning TestHumanizer as a comprehension aid rather than a full validation substitute. 5 5.1

Discussion Why Hallucinations Persist and Implications for Test Refactoring Pipelines

RQ1 shows that hallucination-induced compilation failures are not isolated occurrences but structural risks inherent to LLM-based refactoring. Even under explicit semantics-preservation constraints and full-class context, some refactorings introduce unintended edits to identifiers, method calls, or imports that break compilation. Recent theory frames hallucination as a systemic consequence of current training and evaluation regimes rather than an accidental defect [30]: when abstention is not rewarded, models are incentivized to produce plausible-looking transformations under uncertainty, and refactoring errors reflect overconfident optimization rather than prompt misconfiguration. Crucially, providing more context does not eliminate this risk. The code-centric configuration—which supplies the full class source—did not prevent semantic-breaking edits and was additionally exposed to API-level prompt rejections, whereas the more abstract summary-based configuration achieved higher compilation stability (RQ1). Robustness thus depends less on the volume of context provided than on how that context constrains the LLM’s transformation space. A compact, abstract representation appears to implicitly bound the set of identifiers and structures the model can modify, reducing opportunities for hallucination-induced drift. Consistent with observations in agentic repair workflows [38], hallucination must be treated as a pipeline-level phenomenon rather than a prompt-engineering problem. LLM-based refactoring therefore requires mandatory compilation and validation gates at each transformation step—not as optional safeguards, but as structural requirements of any reliable refactoring pipeline [30, 38]. 5.2

Toward Agentic and Retrieval-Augmented Test Refactoring Pipelines

Our findings indicate that robustness in LLM-based test refactoring is primarily a matter of workflow design rather than prompt engineering. Given the structural hallucination risks identified in RQ1, refactoring should be treated as a staged process that interleaves transformation, validation, and repair—rather than a single-shot LLM call followed by optional checking. Agentic pipelines operationalize this principle through generate–detect–repair loops. Multi-agent systems have been shown to improve smell detection and refactoring reliability, yet they do not fully eliminate erroneous edits [38]. In our setting, similar orchestration could iteratively localize and repair noncompiling transformations, with compilation enforced as a hard acceptance gate at each iteration. The SIDE-based summarization loop and embedding-based similarity checks in TestHumanizer already embody a lightweight version of this principle; extending them with targeted repair steps would further reduce the fraction of non-compiling outputs observed in RQ1. Retrieval-augmented refactoring offers complementary grounding. Injecting project-specific resources—such as API usage patterns, existing test suites, or class-level documentation—constrains the transformation space and reduces unsupported identifier edits, directly addressing the dominant Cannot Find Symbol failures J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

observed in RQ1. Retrieval-enhanced prompting has already been shown to improve coverage in test generation [58]; analogous grounding for refactoring could similarly stabilize compilability and, as a side effect, mitigate structural smells (RQ5). Together, these directions suggest moving from prompt-centric refactoring toward guarded, retrieval-grounded architectures that integrate staged decomposition, external evidence, and mandatory verification gates. Such designs would strengthen the reliability gains of TestHumanizer (RQ1) while further consolidating its readability and structural improvements (RQ2, RQ5)—providing a principled path toward making LLM-based refactoring robust enough for continuous integration pipelines. 5.3

Rethinking Evaluation Beyond Similarity in LLM-Based Test Refactoring

RQ3 shows that similarity-centric evaluation is insufficient for LLM-based refactoring. Embeddingbased similarities saturate across configurations and are too weakly discriminative to distinguish meaningfully between tests-only, code-centric, and summary-based refactorings. Token-level metrics such as CodeBLEU can penalize structurally divergent yet beneficial restructurings, echoing broader evidence that overlap metrics may mis-rank stylistically or structurally different LLM outputs [62]. CTSES mitigates some of this over-pessimism as a composite, human-aligned signal, but our manual inspection shows it can still mis-rank broad-scope refactorings that are genuinely beneficial. These limitations call for multi-signal evaluation that goes beyond surface overlap. Alongside similarity metrics, evaluation frameworks for LLM-based test refactoring should incorporate human-oriented indicators that directly capture the goals of refactoring: intent clarity, naming and documentation quality, scenario structure (e.g., Given–When–Then adherence), and smell reduction. Structural feasibility signals—compilation rates and code coverage—should serve as hard gates rather than optional complements, since a refactoring that improves readability at the cost of compilation or coverage provides no practical value. Developing refactoring-aware metrics that jointly capture these dimensions remains an open challenge, and one our study highlights as critical for progress in this area. A further complication is that readability itself is perception-driven. Developers reach only moderate agreement even under controlled evaluation settings [55], and LLM-based readability evaluators, while increasingly human-aligned, can exhibit systematic biases and stability trade-offs [42]. This means that no single automated metric—whether similarity-based or LLMjudged—can fully substitute for human assessment. Overall, evaluation should treat similarity as a necessary but insufficient signal, and systematically connect automated metrics to developer preference through human studies, as demonstrated in RQ6. 5.4

Implications for Researchers

Our results suggest several research directions to make SBST outputs practically usable through LLM-based refactoring. LLMs should act as post-processing layers rather than SBST replacements. Direct one-shot generation remains brittle and coverage-poor at scale, whereas refactoring compilable SBST suites yields near-baseline compilability while largely preserving coverage (RQ1, RQ4). Future work should explore how this refactoring-layer paradigm generalizes to other SBST tools and testing frameworks beyond EvoSuite. Context granularity is a first-class design choice. Full-code prompting does not prevent hallucination-induced failures and can introduce practical feasibility issues such as API-level rejections, while compact summary-based context is consistently the most stable configuration (RQ1). Context compression and semantic abstraction should be treated as first-order robustness levers in LLM-based refactoring pipelines, not as implementation details. Compilation and dynamic checks are mandatory safeguards. Even with explicit semantics-preservation instructions, some refactorings introduce hallucinated identifier edits that break compilation (RQ1). Compile-time gates and, when needed, repair or rollback loops J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

are not optional additions but structural requirements of any reliable refactoring pipeline. Similarity alone is insufficient for evaluation. Embedding similarities saturate and lexical overlap can misclassify structurally divergent yet beneficial restructurings (RQ3). Composite similarity metrics such as CTSES should be complemented with structural indicators—readability, understandability, and smell profiles (RQ2, RQ5)—and selective manual validation of low-similarity cases. Developing refactoring-aware evaluation frameworks that jointly capture these dimensions is an open and pressing research challenge. Refactoring objectives should be made explicit. Readability and understandability improve consistently as primary objectives, while smell-profile shifts emerge as beneficial side effects (RQ2, RQ5). This suggests that objective-driven refactoring prompts— explicitly targeting smell reduction or modularity alongside readability—may yield further quality gains, and that systematic trade-off analysis between readability, structural stability, and behavioral preservation deserves dedicated study. Developer preference should inform evaluation design. Human ratings confirm that readability gains translate into higher willingness to adopt, while still requiring independent validation of behavioral adequacy (RQ6). Future evaluation frameworks should go beyond automated metrics and connect refactoring quality signals to developer-centered outcomes, including maintenance effort, bug-finding effectiveness, and long-term suite evolution.

5.5

Implications for Developers

Our findings provide actionable guidance for practitioners considering LLM-based refactoring of automatically generated tests. Use LLMs to improve maintainability—not to replace systematic test generation. Direct LLM-generated tests are more readable but exhibit substantially lower and less stable coverage (RQ4). Refactoring SBST suites with TestHumanizer preserves coverage while improving clarity and structure (RQ2, RQ4). In practice, LLMs are best deployed as a refinement layer on top of existing SBST pipelines, not as a replacement for the coverage guarantees that SBST tools provide. Prefer compact contextual guidance over full-code prompting. Providing the entire class under test does not guarantee higher reliability and may introduce instability—including APIlevel prompt rejections and hallucination-induced identifier edits (RQ1). Summary-based context offers a more robust and scalable alternative while maintaining strong compilation and coverage guarantees. For practitioners, this means investing in lightweight class summarization as a preprocessing step rather than passing raw source code directly to the LLM. Keep compilation and coverage checks in the loop. Although most refactorings preserve test behavior, a non-negligible fraction introduce hallucinated changes that break identifier references or alter API usage (RQ1). Automated compile-time verification and coverage regression checks should be integrated into CI pipelines whenever LLM-based refactoring is used, and refactorings that fail these checks should be reverted rather than manually corrected. Expect structural improvements—but not automatic smell elimination. TestHumanizer substantially reduces control-flow-heavy patterns such as Conditional Logic Test, and on larger systems (SF110) also mitigates Lazy Test and Duplicate Assert (RQ5). However, dominant smells such as Assertion Roulette and General Fixture are not automatically resolved, as they require deeper fixture and assertion redesign beyond the scope of readability-oriented refactoring prompts. Developers should treat LLM-based refactoring as structural enhancement, not as a substitute for deliberate test redesign. Readability gains translate into higher developer acceptance—but independent validation remains necessary. Human evaluators consistently prefer TestHumanizer-refactored suites in terms of clarity and willingness to adopt (RQ6). Nevertheless, improved readability does not substitute for independent validation of behavioral correctness and coverage adequacy. LLM-refactored tests should be treated as improved starting points for review, not as fully validated outputs ready for production use. J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

5.6

W. Ouédraogo et al.

Threats to Validity

External Validity. Our study focuses on Java unit tests generated by EvoSuite over Defects4J and SF110, refactored via two LLMs (gpt-4o and mistral-large-2407). Results may not directly generalize to other languages (e.g., Python), testing frameworks, SBST tools, or domains with different API characteristics. We mitigate this threat by using two widely adopted benchmarks covering diverse projects and byanalyzing two distinct model families, but further replication on additional ecosystems and generators remains necessary. For RQ6, the developer study covers 30 classes (444 test methods); while stratified across datasets and LOC bins, it remains a bounded sample and may not capture all industrial test styles or evaluation contexts. Internal Validity. LLM outputs are non-deterministic and can vary with decoding parameters, system prompts, or model updates. We mitigate this by fixing prompting templates and applying the same pipeline systematically across all configurations, though some variance is unavoidable. Feasibility in the code-centric setting for gpt-4o is additionally affected by API-level prompt rejections (HTTP 400) triggered by repetitive prompt patterns, which may depend on provider-side input validation policies and can influence comparative feasibility across models and configurations. A non-negligible fraction of non-compiling refactorings is caused by hallucination-induced edits that break referenced program entities (see Figure 2); we rely on compilation filtering and manual inspection to characterize these cases, but cannot fully eliminate them without automated repair or rollback mechanisms. Our results are also reported over compilable suites for metrics requiring execution (e.g., coverage), which may slightly bias comparisons toward more robust configurations; however, this reflects a realistic deployment constraint in which only compilable suites are actionable. Construct Validity. Readability and maintainability are multi-faceted constructs that resist singlemetric characterization. We mitigate metric bias by combining complementary indicators: a machinelearned readability proxy [54], structural and cognitive complexity measures (Cyclomatic Complexity and CCTR), composite similarity metrics (CTSES and embedding-based cosine similarity), structural coverage, and test smell prevalence. Nevertheless, automated metrics can mischaracterize broad-scope yet beneficial refactorings, as observed in our manual inspection of low-similarity cases (RQ3). For RQ6, we use two Likert-scale questions (readability/clarity of intent and willingness to adopt) supplemented by free-text feedback; while this captures perceived usability across two evaluator profiles, it does not constitute a full longitudinal industrial evaluation. In particular, willingness to adopt as a proxy for practical usefulness should be interpreted cautiously. Future studies with longer-term maintenance tasks, defect-detection outcomes, and larger evaluator pools would provide stronger ecological validity for the human-centered findings reported in RQ6. 6

Related Work

Our work sits at the intersection of three active research threads: LLM-based improvement of automatically generated tests, smell-driven refactoring of unit tests, and analysis-guided LLM test generation. We position TestHumanizer relative to each. 6.1

Improving Automatically Generated Tests with LLMs

Recent work has explored LLMs as post-processing layers to improve the readability and maintainability of automatically generated tests. Biagiola et al. [11] refactor SBST-generated suites through constrained transformations that primarily target identifier renaming, and emphasize compilation checking as a key feasibility guard. Deljouyi et al. [16] integrate LLM-based rewriting within an SBST+LLM workflow, incorporating compilation verification and fallback mechanisms to handle J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

non-compiling outputs and using CodeBLEU as a semantic safeguard. Both approaches apply relatively narrow transformations to individual tests or identifiers. In contrast, TestHumanizer operates at the level of entire test suites and applies a broader scope of structural transformation—including Given–When–Then documentation, grouping of related logic, and control-flow simplification. We additionally study three context granularities and quantify trade-offs across feasibility, semantic similarity, structural coverage, smell profiles, and developer preference, providing a more comprehensive view of the readability–robustness trade-off in LLM-based test refactoring. 6.2

LLM-Driven Refactoring of Human-Written Tests via Smell-Specific Guidance

Beyond automatically generated tests, Gao et al. [22] propose UTRefactor, an LLM-based framework that targets smell removal in human-written unit tests using explicit smell knowledge encoded in a DSL, together with checkpointed prompting and validation. Their study reports substantial smell reduction on real-world Java projects, while also noting that compilation and execution failures can still occur when refactoring rules conflict with project-specific assertion APIs. UTRefactor and TestHumanizer share the goal of improving test quality through LLM-based refactoring, but differ in scope and target. UTRefactor applies rule-guided, smell-specific transformations to developerwritten tests, whereas TestHumanizer takes a holistic approach to “humanizing” SBST-generated suites—improving structure, naming, and documentation—and evaluates outcomes through a multidimensional protocol that includes compilation rates, coverage preservation, similarity metrics, and human judgments. Moreover, any smell reductions in TestHumanizer emerge as side effects of readability-oriented refactoring rather than explicit optimization targets. 6.3

LLM-Based Test Generation with Analysis-Guided Pipelines

A parallel line of work investigates LLMs as test generators augmented with program analysis and verification loops to obtain executable, high-coverage suites. Pan et al. [48] propose ASTER, which combines lightweight static analysis to extract test-relevant context—including mocking opportunities—with post-generation sanitization, compile/execute checks, iterative repair, and coverage-guided augmentation. ASTER yields tests that professional developers perceive as more natural while maintaining coverage competitive with EvoSuite and CodaMOSA. This generationcentric paradigm is fundamentally different from our refactoring-based approach. Where ASTER and similar pipelines start from scratch and use analysis to guide LLM generation, TestHumanizer starts from compilable, high-coverage EvoSuite suites and uses LLMs as a controlled structural improvement layer. This distinction is consequential: by building on SBST outputs, TestHumanizer inherits their coverage guarantees and avoids the compilability fragility of one-shot LLM generation, while still achieving the readability and understandability improvements that generation-centric approaches aim for. Our systematic evaluation of context granularity, semantic preservation, and developer preference provides complementary evidence on how LLMs can be most effectively integrated into test engineering workflows. 7

Conclusion and Future Work

Automatically generated tests provide strong behavioral reach but often lack the readability and maintainability that make them practical for developers. Pure LLM-based generation, while producing more natural tests, remains brittle and coverage-poor at scale. Our results show that these paradigms are complementary: when used as a controlled refactoring layer over compilable SBST suites, LLMs can substantially improve the developer-perceived quality of generated tests without sacrificing structural robustness or coverage. Across 350 classes from Defects4J and SF110, TestHumanizer preserves near-EvoSuite compilation rates (88–98%) and structural coverage while improving predicted readability, reducing control-flow and test-aware cognitive complexity, and J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

substantially mitigating structural smells—most notably, Conditional Logic Test drops from 36.97% to 5.02% on Defects4J and from 48.31% to 4.37% on SF110 in the summary-based configuration. These quality gains translate into significantly higher developer preference and willingness to adopt, as confirmed by our developer study on 30 classes (444 test methods, Wilcoxon 𝑝 < 0.01, medium-tolarge effects). At the same time, hallucination-induced compilation failures, context sensitivity, and the limitations of similarity-based evaluation reveal that LLM-based refactoring must be treated as a guarded transformation process, supported by mandatory compilation gates, coverage checks, and multi-dimensional quality assessment. Beyond TestHumanizer, our findings advocate for hybrid SBST+LLM architectures in which SBST ensures behavioral guarantees and LLMs enhance structure and interpretability under explicit validation constraints. The summary-based configuration consistently emerges as the most robust design choice, demonstrating that compact semantic abstractions can constrain LLM transformations more effectively than full source code—a principle with broader implications for context-aware LLM engineering. Future work should pursue three directions. First, agentic and retrieval-augmented refactoring pipelines with generate–detect–repair loops and project-aware grounding would address the hallucination-induced failures identified in RQ1 through principled repair rather than simple rejection. Second, refactoring-aware evaluation frameworks that move beyond similarity-centric metrics toward developer-aligned signals—capturing intent clarity, naming quality, scenario structure, and smell reduction—would provide more faithful assessments of broad-scope refactorings that current metrics penalize unfairly. Third, objective-driven refactoring strategies that explicitly model trade-offs between readability, smell elimination, coverage, and behavioral stability would enable more targeted and controllable improvements than the emergent effects observed in this study. Overall, LLMs are most effective not as standalone test generators, but as carefully constrained refinement layers over robust automated testing infrastructures—a paradigm that preserves the coverage guarantees of SBST while closing the usability gap that has long limited the practical adoption of automatically generated tests. Acknowledgements This research was funded in whole, or in part, by the Luxembourg National Research Fund (FNR), grant reference AFR PhD bilateral, project reference 17185670. This work was also supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No. 949014). For the purpose of open access, and in fulfilment of the obligations arising from the grant agreement, the author has applied a Creative Commons Attribution 4.0 International (CC BY 4.0) license to any Author Accepted Manuscript version arising from this submission. References [1] Toufique Ahmed and Premkumar Devanbu. 2022. Few-shot training LLMs for project-specific code-summarization. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–5. [2] Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl T Barr. 2024. Automatic semantic augmentation of language model prompts (for code summarization). In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society (2024), 1004–1004. [3] M Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Janis Benefelds. 2017. An industrial evaluation of unit test generation: Finding real faults in a financial application. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). IEEE, 263–272. [4] Andrea Arcuri and Gordon Fraser. 2013. Parameter tuning or default values? An empirical investigation in search-based software engineering. Empirical Software Engineering 18 (2013), 594–623. [5] Alberto Bacchelli, Paolo Ciancarini, and Davide Rossi. 2008. On the effectiveness of manual and automatic unit test generation. In 2008 The Third International Conference on Software Engineering Advances. IEEE, 252–257. [6] Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

translation and/or summarization. 65–72. [7] Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and David Binkley. 2012. An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In 2012 28th IEEE international conference on software maintenance (ICSM). IEEE, 56–65. [8] Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and Dave Binkley. 2015. Are test smells really harmful? an empirical study. Empirical Software Engineering 20 (2015), 1052–1094. [9] Kent Beck. 2000. Extreme programming explained: embrace change. addison-wesley professional. [10] Shreya Bhatia, Tarushi Gandhi, Dhruv Kumar, and Pankaj Jalote. 2024. Unit test generation using generative ai: A comparative performance analysis of autogeneration tools. In Proceedings of the 1st International Workshop on Large Language Models for Code. 54–61. [11] Matteo Biagiola, Gianluca Ghislotti, and Paolo Tonella. 2025. Improving the readability of automatically generated tests using large language models. In 2025 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 162–173. [12] Raymond PL Buse and Westley R Weimer. 2009. Learning a metric for code readability. IEEE Transactions on software engineering 36, 4 (2009), 546–558. [13] G Ann Campbell. 2018. Cognitive complexity: An overview and evaluation. In Proceedings of the 2018 international conference on technical debt. 57–58. [14] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [15] Ermira Daka, José Campos, Gordon Fraser, Jonathan Dorn, and Westley Weimer. 2015. Modeling readability to improve unit tests. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 107–118. [16] Amirhossein Deljouyi, Roham Koohestani, Maliheh Izadi, and Andy Zaidman. 2024. Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests. arXiv preprint arXiv:2408.11710 (2024). [17] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020). [18] Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419. [19] Gordon Fraser and Andrea Arcuri. 2014. A large-scale evaluation of automated unit test generation using evosuite. ACM Transactions on Software Engineering and Methodology (TOSEM) 24, 2 (2014), 1–42. [20] Gordon Fraser, Matt Staats, Phil McMinn, Andrea Arcuri, and Frank Padberg. 2015. Does automated unit test generation really help software testers? a controlled empirical study. ACM Transactions on Software Engineering and Methodology (TOSEM) 24, 4 (2015), 1–49. [21] Yi Gao, Xing Hu, Xiaohu Yang, and Xin Xia. 2024. Context-Enhanced LLM-Based Framework for Automatic Test Refactoring. arXiv preprint arXiv:2409.16739 (2024). [22] Yi Gao, Xing Hu, Xiaohu Yang, and Xin Xia. 2025. Automated Unit Test Refactoring. Proceedings of the ACM on Software Engineering 2, FSE (2025), 713–733. [23] Gregory Gay. 2023. Improving the Readability of Generated Tests Using GPT-4 and ChatGPT Code Interpreter. In International Symposium on Search Based Software Engineering. Springer, 140–146. [24] Giovanni Grano, Simone Scalabrino, Harald C Gall, and Rocco Oliveto. 2018. An empirical investigation on the readability of manual and generated test cases. In Proceedings of the 26th Conference on Program Comprehension. 348–351. [25] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 (2020). [26] Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al. 2024. Found in the middle: Calibrating positional attention bias improves long context utilization. In Findings of the Association for Computational Linguistics: ACL 2024. 14982–14995. [27] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics 2016. Association for Computational Linguistics, 2073–2083. [28] Gunel Jahangirova and Valerio Terragni. 2023. SBFT tool competition 2023-Java test case generation track. In 2023 IEEE/ACM International Workshop on Search-Based and Fuzz Testing (SBFT). IEEE, 61–64. [29] René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 international symposium on software testing and analysis. J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

437–440. [30] Adam Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang. 2025. Why Language Models Hallucinate. https: //doi.org/10.48550/arXiv.2509.04664 [31] Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 919–931. [32] Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81. [33] Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04). 605–612. [34] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics 12 (2024), 157–173. [35] Stephan Lukasczyk and Gordon Fraser. 2022. Pynguin: Automated unit test generation for python. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 168–172. [36] Antonio Mastropaolo, Matteo Ciniselli, Massimiliano Di Penta, and Gabriele Bavota. 2024. Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. [37] Thomas J McCabe. 1976. A complexity measure. IEEE Transactions on software Engineering 4 (1976), 308–320. [38] Rian Melo, Pedro Simões, Rohit Gheyi, Marcelo d’Amorim, Márcio Ribeiro, Gustavo Soares, Eduardo Almeida, and Elvys Soares. 2025. Agentic LMs: Hunting Down Test Smells. IEEE Software (2025). [39] Delano Oliveira, Reyde Bruno, Fernanda Madeiral, Hidehiko Masuhara, and Fernando Castor. 2022. A systematic literature review on the impact of formatting elements on program understandability. Available at SSRN 4182156 (2022). [40] Wendkûuni C Ouédraogo, Kader Kaboré, Haoye Tian, Yewei Song, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2024. Large-scale, Independent and Comprehensive study of the power of LLMs for test case generation. arXiv preprint arXiv:2407.00225 (2024). [41] Wendkuuni C Ouedraogo, Kader Kabore, Haoye Tian, Yewei Song, Anil Koyuncu, Jacques Klein, David Lo, and Tegawende F Bissyande. 2024. Llms and prompting for unit test generation: A large-scale evaluation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 2464–2465. [42] Wendkûuni C Ouédraogo, Yinghua Li, Xueqi Dang, Pawel Borsukiewicz, Xin Zhou, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2025. Human-Aligned Code Readability Assessment with Large Language Models. arXiv preprint arXiv:2510.16579 (2025). [43] Wendkûuni C Ouédraogo, Yinghua Li, Xueqi Dang, Xunzhu Tang, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2024. Test smells in llm-generated unit tests. arXiv preprint arXiv:2410.10628 (2024). [44] Wendkûuni C Ouédraogo, Yinghua Li, Xueqi Dang, Xin Zhou, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2025. Beyond Surface Similarity: Evaluating LLM-Based Test Refactorings with Structural and Semantic Awareness. arXiv preprint arXiv:2506.06767 (2025). [45] Wendkûuni C Ouédraogo, Yinghua Li, Xueqi Dang, Xin Zhou, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2025. Rethinking Cognitive Complexity for Unit Tests: Toward a Readability-Aware Metric Grounded in Developer Perception. In 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 797–802. [46] Wendkûuni C Ouédraogo, Laura Plein, Kader Kabore, Andrew Habib, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2025. Enriching automatic test case generation by extracting relevant test inputs from bug reports. Empirical Software Engineering 30, 3 (2025), 85. [47] Fabio Palomba, Dario Di Nucci, Annibale Panichella, Rocco Oliveto, and Andrea De Lucia. 2016. On the diffusion of test smells in automatically generated test code: An empirical study. In Proceedings of the 9th international workshop on search-based software testing. 5–14. [48] Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. 2025. Aster: Natural and multilanguage unit test generation with llms. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 413–424. [49] Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. 2017. Automated test case generation as a manyobjective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering 44, 2 (2017), 122–158. [50] Annibale Panichella, Sebastiano Panichella, Gordon Fraser, Anand Ashok Sawant, and Vincent J Hellendoorn. 2020. Revisiting test smells in automatically generated tests: limitations, pitfalls, and opportunities. In 2020 IEEE international J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

conference on software maintenance and evolution (ICSME). IEEE, 523–533. [51] Anthony Peruma, Khalid Almalki, Christian D Newman, Mohamed Wiem Mkaouer, Ali Ouni, and Fabio Palomba. 2020. Tsdetect: An open source test smells detection tool. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 1650–1654. [52] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297 (2020). [53] Devjeet Roy, Ziyi Zhang, Maggie Ma, Venera Arnaoudova, Annibale Panichella, Sebastiano Panichella, Danielle Gonzalez, and Mehdi Mirakhorli. 2020. DeepTC-Enhancer: Improving the readability of automatically generated tests. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 287–298. [54] Simone Scalabrino, Mario Linares-Vásquez, Rocco Oliveto, and Denys Poshyvanyk. 2018. A comprehensive model for code readability. Journal of Software: Evolution and Process 30, 6 (2018), e1958. [55] Agnia Sergeyuk, Olga Lvova, Sergey Titov, Anastasiia Serova, Farid Bagirov, and Timofey Bryksin. 2024. Assessing Consensus of Developers’ Views on Code Readability. arXiv preprint arXiv:2407.03790 (2024). [56] Agnia Sergeyuk, Olga Lvova, Sergey Titov, Anastasiia Serova, Farid Bagirov, Evgeniia Kirillova, and Timofey Bryksin. 2024. Reassessing Java Code Readability Models with a Human-Centered Approach. In Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension. 225–235. [57] Sina Shamshiri, René Just, José Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. 2015. Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 201–211. [58] Jiho Shin, Nima Shiri Harzevili, Reem Aleithan, Hadi Hemmati, and Song Wang. 2024. Retrieval-augmented test generation: How far are we? arXiv preprint arXiv:2409.12682 (2024). [59] James Shore and Shane Warden. 2021. The art of agile development. " O’Reilly Media, Inc.". [60] Mohammed Latif Siddiq, Joanna CS Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinícius Carvalho Lopes. 2024. Using Large Language Models to Generate JUnit Tests: An Empirical Study. (2024). [61] Saleem Siddiqui. 2021. Learning Test-Driven Development. " O’Reilly Media, Inc.". [62] Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. 2024. Source Code Summarization in the Era of Large Language Models. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 419–431. [63] Yutian Tang, Zhijie Liu, Zhichao Zhou, and Xiapu Luo. 2024. Chatgpt vs sbst: A comparative assessment of unit test suite generation. IEEE Transactions on Software Engineering (2024). [64] Arie Van Deursen, Leon Moonen, Alex Van Den Bergh, and Gerard Kok. 2001. Refactoring test code. In Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001). Citeseer, 92–95. [65] Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering (2024). [66] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022). [67] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837. [68] Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023. No more manual tests? evaluating and improving chatgpt for unit test generation. arXiv preprint arXiv:2305.04207 (2023). [69] Zhichao Zhou, Yutian Tang, Yun Lin, and Jingzhu He. 2024. An LLM-based readability measurement for unit tests’ context-aware inputs. arXiv preprint arXiv:2407.21369 (2024).

Additional Tables Table 10. Test generation statistics across datasets and tools.

Dataset

Test Generation Compilability Rate (%) Rate (%)

Tokens Max

Tokens Tokens Min Average

LOC Max

LOC LOC Min Average

Test Methods Average

#Projects

#Classes

gpt-4o Defects4J mistral-large-2407 EvoSuite

147

99.77 95.24 100.00

68.48 51.50 100.00

1883 3223 27896

49 48 156

517.33 680.45 1958.19

301 359 945

4 4 4

62.11 71.80 169.40

8.58 8.06 17.38

gpt-4o mistral-large-2407 EvoSuite

203

98.52 97.87 100.00

78.00 77.83 100.00

2211 2821 27968

20 27 151

476.31 461.79 3462.57

316 319 2879

4 4 4

54.11 43.89 294.14

8.07 5.52 27.82

SF110

Model

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

W. Ouédraogo et al.

Table 11. Refactoring effectiveness and structural characteristics of refactored test suites. Dataset

Defects4J

SF110

Refactoring Compilation Rate (%) Rate (%)

Tokens Max

Tokens Min

Tokens Average

LOC LOC Max Min

LOC Average

Test Methods Average

90.66 90.64

13057 12360

156 153

1950.33 2173.74

1024 1440

4 4

183.25 203.67

14.37 15.50

84.49 83.93

89.91 88.52

12193 12568

156 153

1909.53 2050.32

984 1429

4 6

182.77 205.21

14.29 15.65

gpt-4o mistral-large-2407

95.00 92.94

97.24 92.92

11247 13229

160 154

1902.54 2146.21

960 1440

4 6

182.16 204.27

14.16 15.54

Tests-only

gpt-4o mistral-large-2407

77.09 76.58

92.41 91.95

15439 13185

202 148

2471.22 2802.56

1578 1589

5 6

243.07 255.50

18.63 18.85

Code-centric

gpt-4o mistral-large-2407

75.58 74.87

90.10 89.02

14829 12999

209 188

2391.34 2562.74

1580 1607

5 11

239.50 251.40

18.32 18.60

Summary-based

gpt-4o mistral-large-2407

95.82 93.16

98.69 96.99

15653 13253

203 189

2434.80 2731.31

1580 1641

11 11

242.08 253.85

18.59 18.77

Approach

Model

Tests-only

gpt-4o mistral-large-2407

85.09 85.46

Code-centric

gpt-4o mistral-large-2407

Summary-based

Table 12. RQ4 Coverage Statistics per Dataset and Approach. Dataset

Min

Mean

Instruction Coverage EvoSuite 74.93 gpt-4o 2.00 Tests-only 31.33 Code-centric 34.00 Summary-based 66.33

Coverage / Source

91.04 81.50 88.67 87.00 91.00

92.83 79.35 91.59 90.99 93.93

97.33 100.00 88.50 100.00 98.00 100.00 98.00 100.00 100.00 100.00

68.39 0.94 40.71 40.71 60.78

88.24 76.21 83.82 82.99 86.22

91.68 73.76 90.03 89.54 92.29

96.77 85.20 97.25 97.22 100.00

100.00 100.00 100.00 100.00 100.00

78.19 4.17 58.82 57.73 66.25

100.00 100.00 100.00 100.00 100.00

99.60 92.06 98.49 98.63 99.42

100.00 100.00 100.00 100.00 100.00

Instruction Coverage EvoSuite 84.78 gpt-4o 10.00 Tests-only 30.67 Code-centric 76.33 Summary-based 78.00

93.37 60.00 92.00 92.00 93.00

95.35 56.57 94.48 94.68 95.32

98.30 100.00 60.00 100.00 98.00 100.00 98.00 100.00 100.00 100.00

Line Coverage EvoSuite gpt-4o Tests-only Code-centric Summary-based

80.77 7.70 30.63 73.81 72.40

90.82 66.70 88.69 88.90 89.52

93.89 60.29 92.39 92.68 93.61

98.01 66.70 97.52 98.02 100.00

100.00 100.00 100.00 100.00 100.00

Method Coverage EvoSuite gpt-4o Tests-only Code-centric Summary-based

99.82 20.00 33.33 90.95 95.22

100.00 25.00 100.00 100.00 100.00

100.00 35.23 99.88 99.95 99.98

100.00 25.00 100.00 100.00 100.00

100.00 100.00 100.00 100.00 100.00

Line Coverage Defects4J EvoSuite gpt-4o Tests-only Code-centric Summary-based Method Coverage EvoSuite gpt-4o Tests-only Code-centric Summary-based

SF110

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Max

Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

Table 13. Detailed human-evaluation sample (per dataset, project, and class). Dataset

Project

Class

LOC

Size bin

Cli Compress Compress Csv Mockito

WriteableCommandLine DeflateCompressorInputStream UnixStat CSVRecord TypeBasedCandidateFilter

9 24 67 65 10

Small Small Small Small Small

TypeHandler Lang ArchiveStreamFactory X7875_NewUnix SerializationUtils

145 84 118 164 101

Medium Medium Medium Medium Medium

Csv Gson JacksonCore JacksonDatabind Lang

CSVParser ISO8601Utils NumberInput StdDateFormat ExtendedMessageFormat

212 242 247 254 245

Large Large Large Large Large

21_geo-google 41_follow 87_jaw-br commons-math scribe

AddressToUsAddressFunctor TabbedPane Salvar AbstractSimplex Verifier

54 44 16 71 3

Small Small Small Small Small

60_sugar 77_io-project 86_at-robots2-j guava javaml

FSPathExplorer ClientGroup RobotRenderer Suppliers Fold

158 83 114 125 174

Medium Medium Medium Medium Medium

47_dvd-homevideo 69_lhamacaw 85_shop guava guava

Menu DisplayableListPanel JSPredicateForm Monitor Predicates

288 268 218 442 325

Large Large Large Large Large

Cli Codec Defects4J Compress Compress Lang

SF110

Table 14. Descriptive statistics of human evaluation scores (Likert 1–5 scale). Metric Readability

Evaluator

Approach

Min

Median Mean

Max

Advanced graduate student

EvoSuite Summary EvoSuite Summary

2 2 1 1

2 3 1 2

2.5 4 2 4

2.71 3.79 2.14 3.50

3 4 3 4

4 5 4 5

EvoSuite Summary EvoSuite Summary

1 2 1 2

2 3 1 2

3 4 2 4

2.57 3.50 2.14 3.25

3 4 3 4

4 5 4 5

Senior developer Advanced graduate student

Adoption

Senior developer

Table 15. Detailed inter-annotator agreement metrics for RQ6 (per group and approach). Approach

𝛼

QWK

%Agree

𝜌

𝑝

Group A EvoSuite TestHumanizer EvoSuite (Read.) TestHumanizer (Read.)

0.79 0.74 0.82 0.72

0.78 0.75 0.82 0.73

0.71 0.57 0.71 0.57

0.89 0.97 0.94 0.96

1.75 × 10−5 6.09 × 10−9 4.78 × 10−7 3.79 × 10−8

Group B

EvoSuite TestHumanizer EvoSuite (Read.) TestHumanizer (Read.)

0.56 0.94 -0.15 0.94

0.60 0.94 0.17 0.94

0.43 0.93 0.21 0.86

0.80 6.30 × 10−4 0.98 1.10 × 10−9 0.35 0.216 1.00 6.78 × 10−14

Overall

EvoSuite (Adopt.) TestHumanizer (Adopt.) EvoSuite (Read.) TestHumanizer (Read.)

0.80 0.82 0.71 0.85

– – – –

Scope

– – – –

Note: Quadratic weighted kappa, percent agreement, and Spearman’s 𝜌 are reported only for two-rater settings (Groups A and B). For the overall four-rater setting, Krippendorff’s 𝛼 is used as it generalizes to multiple annotators.

J. ACM, Vol. 1, No. 1, Article . Publication date: June 2026.

Record · ID 319728 · SHA-256 c198ebe21eab776e
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.