Finding the Pivots to Repair Broken Source-Test Evolution Using LLMs

arXiv:2611.03197v1 [cs.SE] 2 Nov 2026

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs XIAO CHEN, The Hong Kong University of Science and Technology, China SONGQIANG CHEN, The Hong Kong University of Science and Technology, China JIALUN CAO, The Hong Kong University of Science and Technology, China YUN PENG, Fudan University, China CHUN YONG CHONG, Monash University Malaysia, Malaysia SHING-CHI CHEUNG∗ , The Hong Kong University of Science and Technology, China Repairing obsolete tests in response to source code evolution is critical for test suite reliability and software quality assurance. While large language models (LLMs) have emerged as promising tools for automating many software development tasks, the effective supply of context and the orchestration of repair pipelines in this source–test co-evolution scenario remain poorly understood. Moreover, although reasoning is found helpful for LLMs on some complex coding tasks, it remains unclear which reasoning steps actually contribute to effective test repairs and how the insights learned from LLMs’ behaviors benefit coding agents. In this paper, we present a systematic empirical study of test repair for source–test co-evolution empowered by LLMs, investigating the effective setups and typical LLM behaviors. To support this study, we construct a new benchmark of 500 real-world source–test co-evolution cases from commits submitted between May 2024 and April 2026 across 15 highly-starred open-source Java projects. Our experimental results reveal several key insights for effective LLM-based test repair: the complementary benefits of combining test failure symptoms, source code changes, and supportive code context, as well as the effectiveness of self-repair rounds. We further derive a common reasoning action taxonomy from LLMs’ reasoning traces, which proves instrumental in guiding LLMs to repair tests more effectively while at a lower financial cost. Applying our findings to the Codex agent further improves its repair success rate on challenging cases by 66.7%. CCS Concepts: • Software and its engineering → Software testing and debugging; Software evolution. Additional Key Words and Phrases: Source–Test Co-evolution, Test Repair, Large Language Models ACM Reference Format: Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung. 2026. Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs. 1, 1 (November 2026), 25 pages. https: //doi.org/XXXXXXX.XXXXXXX ∗ Corresponding author.

Authors’ Contact Information: Xiao Chen, xchenfu@cse.ust.hk, The Hong Kong University of Science and Technology, Hong Kong SAR, China; Songqiang Chen, i9s.chen@connect.ust.hk, The Hong Kong University of Science and Technology, Hong Kong SAR, China; Jialun Cao, jcaoap@cse.ust.hk, The Hong Kong University of Science and Technology, Hong Kong SAR, China; Yun Peng, yunp@fudan.edu.cn, Fudan University, Shanghai, China; Chun Yong Chong, Chong.chunyong@monash.edu, Monash University Malaysia, Subang Jaya, Malaysia; Shing-Chi Cheung, scc@cse.ust.hk, The Hong Kong University of Science and Technology, Hong Kong SAR, China. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM XXXX-XXXX/2026/11-ART https://doi.org/XXXXXXX.XXXXXXX , Vol. 1, No. 1, Article . Publication date: November 2026.

2

1

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

Introduction

In the development of software systems, test cases verify the correctness of code functionalities, serving as a standard for software quality assurance [5, 46, 59, 95]. However, during software evolution, tests can become outdated and incompatible with changes introduced to the system under test (SUT) [23, 39, 79, 92], causing compilation or runtime failures. Prior studies reported that broken test cases contribute to about one-fifth of failures in software systems [27, 76]. False alarms raised by these broken tests (i.e., failures caused by outdated test code rather than genuine SUT defects) can hinder the detection of real program faults [28, 38]. Therefore, timely repair of unit tests is essential for software sustainability [23, 28]. Nevertheless, manually updating unit tests in response to SUT changes is time-consuming and error-prone [23, 25, 44, 83], making automated test repair a practically important problem. Automated source–test co-evolution (i.e., updating test code to match source code changes of the SUT) attracts much interest from researchers. Early source–test co-evolution techniques defined rules [9, 10, 35, 40–43, 86] that apply predefined transformation templates for typical types of code changes. To overcome the rigidity and costly build of hand-crafted rules, later researchers built neural models for test repair [21, 90], achieving broader applicability. Recently, large language models (LLMs) emerged as a more effective solution for test repair, showcasing promising performance due to their extensive code knowledge. They leverage context-aware test repair to tackle focal methods’ signature changes [36], utilize runtime feedback [7, 85], and address both test breakage and test enhancement [56, 93]. More recently, MuMuTestUp [73] employs multiple LLM agents for mutation-guided repair of obsolete tests caused by the changes to a single focal method. However, existing LLM-based test repair techniques primarily focus on scenarios involving changes to a single focal method, which is over-simplified compared to real-world co-evolution scenarios. By analyzing over 2,500 source–test co-evolution cases drawn from commits submitted within the past two years in highly-starred open-source Java projects, we characterize real-world co-evolution practices and observe that 70.9% of broken tests invoke multiple changed methods and 77.8% are accompanied by changes to invoked classes. These suggest that real-world source–test co-evolution is substantially more complex than the settings addressed by current techniques. Moreover, each technique proposes and evaluates one specific combination of repair context and pipeline design, leaving it unclear which design choices actually matter in such scenarios. Existing empirical studies only partially answer this question. Prior studies have characterized whether, when, and why production and test code co-evolve [18, 54, 63, 92], but not how broken tests should be repaired. Very recent studies have begun to evaluate LLMs for test evolution, yet they either examine prompt design under single-focal-method changes [37] or focus on the overall performance of coding agents [60, 78]. Therefore, the practical problem of how to enable LLMs in test repair in such complex, realistic scenarios remains underexplored. In particular, several fundamental design choices remain unsettled: what repair context to supply; how the repair pipeline should be orchestrated; whether the explicit reasoning steps taken by the LLM with reasoning mode enabled, which prove helpful in many coding tasks [15, 32, 87, 88], contribute to its superior performance over the LLM with reasoning mode disabled, and which steps are most effective; how LLMs’ repair performance varies across scenarios of increasing complexity; and whether the derived insights can generalize to autonomous coding agents and even further enhance their performance. In this paper, we aim to find the pivots of LLM-based test repair: the key design choices and model behaviors that determine whether a broken test can be successfully repaired in realistic source– test co-evolution scenarios. To answer the above questions, we conduct a systematic empirical study on a new benchmark comprising 500 real-world source–test co-evolution cases from 15

, Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

3

well-maintained Java projects on GitHub [17], covering commits between May 2024 and April 2026, which provides a more up-to-date and realistic evaluation with larger scale and more complex co-evolution scenarios than existing benchmarks. We investigate five research questions (RQs) regarding the key design dimensions of LLM setups for test repair: (RQ1) we assess how three types of repair context (i.e., Symptoms, Cause, and Support) affect test repair performance; (RQ2) we compare the effectiveness of popular pipeline orchestration strategies (i.e., in-context learning (ICL), multi-attempt repair, and iterative refinement) for test repair; (RQ3) we quantify the contribution of explicit reasoning by evaluating the same model with reasoning mode enabled and disabled, and establish a taxonomy of reasoning actions, based on which we further distill the key reasoning steps to help LLMs repair test cost-effectively; (RQ4) we analyze how repair performance scales with repair scenario complexity by partitioning benchmark instances based on the scope of code under test (CUT) changes and the adaptations required in the tests’ golden patch; and (RQ5) we examine whether the preceding findings generalize to agent-based test repair by measuring the improvement yield over a vanilla coding agent baseline. Our experimental results yield several insights. Specifically, we found that combining multiple context types is an effective strategy for LLM-based test repair. We also found that both multiattempt repair and iterative refinement pipelines enhance LLMs’ performance, while feedback from prior attempts can more efficiently guide the repair process, particularly during the initial self-check round. In addition, we derive a taxonomy of 14 common reasoning action patterns organized across four stages, revealing the reasoning behaviors that underlie the superior repair performance of the LLM with reasoning mode enabled over disabled. Based on this taxonomy, we successfully guide LLMs to repair tests effectively at a lower cost. Finally, our empirically validated findings translate into enhanced agent-based repair, achieving a 66.7% relative improvement on challenging cases. To summarize, this paper makes the following contributions: • Empirical Study: We present a systematic empirical study on LLM-based test repair for source– test co-evolution, spanning repair context, pipeline orchestration, reasoning model behavior, scenario complexity, and generalization to the agent, organized by five research questions. • Benchmark: We construct a new benchmark for source–test co-evolution, comprising 500 dynamically validated, recent instances from 15 highly-starred open-source projects, addressing key limitations of existing benchmarks. • Practical Guidelines: We derive actionable guidelines for designing effective LLM-based test repair approaches, grounded in systematic empirical evidence across diverse co-evolution scenarios. We demonstrate that the findings can be translated into tangible improvements for agent-based test repair. 2 2.1

Background and Motivation Test Repair for Source–Test Co-evolution

During software evolution, source code changes can cause test cases to break, leading to compilation or runtime failures. Test repair aims to automatically update broken test cases to restore their correctness and maintain the integrity of the test suite. In this paper, we study test repair in the context of source–test co-evolution, where changes to the source code of a system under test (SUT) may necessitate corresponding updates to the test cases that verify its functionality. Formally, given two versions of a SUT 𝑉 and 𝑉 ′ , a test case 𝑇 that originally passed on 𝑉 may fail on 𝑉 ′ due to source code changes. The goal of test repair is to produce a test case 𝑇 ′ that passes on 𝑉 ′ while preserving the original test intent: TestRepair(𝑇 , 𝑉 , 𝑉 ′ ) → 𝑇 ′

(1)

, Vol. 1, No. 1, Article . Publication date: November 2026.

4

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

1 2 3

@Test public void testDefaultQueryMetricsQuery() { final StubServiceEmitter serviceEmitter = new StubServiceEmitter("", ""); 4 DefaultQueryMetrics<Query<?>> queryMetrics = new DefaultQueryMetrics<>(); 5 ... 6 queryMetrics.query(query); 7 - queryMetrics.reportQueryTime(0) .emit(serviceEmitter); 8 queryMetrics.sqlQueryId("dummy"); 9 queryMetrics.queryId("dummy"); 10 + queryMetrics.reportQueryTime(0) .emit(serviceEmitter); 11 Map<String,Object> actualEvent = serviceEmitter.getEvents() .get(0).toMap(); 12 ... 13 Assert.assertEquals("dummy", actualEvent.get(DruidMetrics.ID)); 14 ... 15 }

1 2 3 4 5 6 7 8 9

[INFO] Running org.apache.druid.query.DefaultQueryMetricsTest [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! -- in org.apache.druid.query.DefaultQueryMetricsTest [ERROR] org.apache.druid.query.DefaultQueryMetricsTest.testDefaultQueryMetricsQuery -- Time elapsed: 0.002 s <<< FAILURE! org.junit.ComparisonFailure: expected:<[dummy]> but was:<[]> at org.junit.Assert.assertEquals(Assert.java:117) at org.junit.Assert.assertEquals(Assert.java:146) at org.apache.druid.query.DefaultQueryMetricsTest .testDefaultQueryMetricsQuery(DefaultQueryMetricsTest.java:88) 10 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 1 public class StubServiceEmitter extends ServiceEmitter implements MetricsVerifier { 2 ... 3 public void emit(Event event) { 4 if (event instanceof ServiceMetricEvent) { 5 ServiceMetricEvent metricEvent = (ServiceMetricEvent) event; 6 metricEvents.computeIfAbsent(metricEvent.getMetric(), 7 name -> new ConcurrentLinkedDeque<>()) 8 .add(metricEvent.copy()); 9 + .add(metricEvent); 10 ...

Fig. 1. Motivating example of test repair in source–test co-evolution. (Left: broken test with the patch submitted in commit 98c63de2cb. Top-Right: failure log of the broken test. Bottom-right: relevant changes in the source code under test.)

2.2

Motivating Example

We motivate the design of our research questions with a real-world source–test co-evolution example from apache/druid, as shown in Figure 1. An originally passing test, testDefaultQuer yMetricsQuery(), fails with an AssertionError after commit 98c63de2cb [3], thus requiring repair. The ground truth test repair patch submitted by the developer in the same commit is shown in the left part of Figure 1. However, generating this patch is non-trivial: it requires selecting the appropriate source code context to understand the failure, potentially refining the patch iteratively, and reasoning step by step to understand the root cause, before determining which statements in the test should be updated. Specifically, synthesizing the patch requires sufficient proper context to facilitate failure diagnosis. Commit 98c63de2cb modified 18 methods in total, in which two methods are directly invoked by the broken test; however, neither of such tightly-related methods is responsible for the test breakage. The failure instead originates in class StubServiceEmitter (invoked transitively at line 3 of the test), which was changed to store the original event reference rather than a copy (metricE vent instead of metricEvent.copy()). To understand this failure, the failure log of the broken test is a helpful indicator. As shown in the right-top subgraph in Figure 1, the log describes the failure Symptom, locating where the test breaks (i.e., line 88 in DefaultQueryMetricsTest.jav a:88) and what went wrong (i.e., an AssertionError reporting a missing dimension value). In addition, the changes in the source code (under test) in the left subgraph explain the Cause of the test breakage regarding what induced the test to fail only if captured at the proper granularity and scope. Note that the root cause of the failure is not in the directly-invoked methods, but rather in a transitive call to StubServiceEmitter, which is not immediately visible from the test case itself. In addition, an unchanged method ServiceEmitter.emit() is also critical in this example to explain how things connect as Support information. Serving as a dispatch mechanism, it connects the test case to the StubServiceEmitter class, which is where the actual failure occurs. Together, these dimensions define a rich design space for repair context, motivating a systematic investigation of how Symptom, Cause (across hunk-, method-, and file-level granularities), and Support context each affect repair performance and how they interact (RQ1). Beyond providing the proper context, the design of test repair pipeline may also facilitate the generation of correct test patches. For example, both iterative refinement and in-context learning (ICL) have been shown to enhance LLM performance in code generation tasks [31, 33, 34]. This motivates an investigation of how another attempt with the same context or feedback from the prior attempt, or additional test code examples can help LLMs produce a correct patch (RQ2). In addition, the example also exposes demands of multiple reasoning steps (e.g., integrating evidence , Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

5

across multiple changed files, understanding indirect runtime dispatch behavior, and inferring which statements in the test need to be updated). This raises our interest to investigate whether the LLM with reasoning mode enabled that produces explicit chain-of-thought reasoning traces before generating code is better equipped for such scenarios than the LLM with reasoning mode disabled; and whether reasoning traces produced by the LLM with reasoning mode enabled can be leveraged to guide the LLM with reasoning mode disabled towards comparable repair performance at lower cost (RQ3). The example also motivates the investigation of repair performance across co-evolution cases with various complexity. For example, although this example involves only seven changed files, commits in real-world software evolution can touch hundreds of files. In our benchmark mined from recent two years’ real-world software evolution, 12.6% of the cases involve more than 100 changed Java files. The growing complexity of co-evolution scenarios motivates us to examine how repair performance varies across different contexts and still continue to hold in complex scenarios (RQ4). Finally, the expanding adoption of coding agents in software development steers us to ask whether these powerful agents can handle the complex cases where LLMs struggle, and whether the context strategies validated in our study can further improve their performance (RQ5). 3 3.1

Study Design Research Questions

To systematically evaluate LLM-driven test repair performance for source–test co-evolution, we conduct experiments to address five research questions (RQs). These RQs investigate the impact of repair context types and pipeline orchestrations on repair performance, analyze the reasoning behaviors of the LLM with reasoning mode enabled, and explore how empirical findings can inform the design of more effective prompting strategies and repair pipelines following the motivating example in Section 2.2. • RQ1. How do different types of repair contexts affect test repair performance? We study three types of repair context as discussed in Section 2.2: (1) Symptom, reflected by the test failure log (i.e., the execution log of the broken test); (2) Cause, i.e., the changes in the code under test (CUT) at different levels of granularity (i.e., hunk-level, method-level, and file-level); and (3) Support, which supplies unchanged but relevant methods and classes identified by LLMs. • RQ2. How do different pipeline orchestrations affect test repair performance? We evaluate three popular pipeline orchestration strategies for LLMs, including in-context learning (ICL) (i.e., prompting with 𝑛-shot example, 𝑛=0, 1, 3, retrieved based on similarity), multi-attempt repair (i.e., pass@𝑘, 𝑘=1, 2, 3), and iterative refinement (i.e., 𝑟 -iteration, 𝑟 =1, 2, 3) to explore their impact on test repair performance. • RQ3. What reasoning behaviors contribute to the superior performance of the LLM with reasoning mode enabled? We first examine the capability of the LLM with reasoning mode enabled for source–test co-evolution by comparing the performance of DeepSeek-V3.2 [11] with reasoning mode enabled versus disabled to understand the extent to which reasoning capabilities contribute to test repair. Then, we analyze the reasoning traces produced by the LLM with reasoning mode enabled to identify recurring actions during test repair and distill these observations into a four-stage reasoning-action taxonomy. We also investigate how the reasoning action taxonomy can inform the design of more cost-effective prompting strategies by regulating the repair pipeline to reduce unnecessary reasoning actions. • RQ4. How does the test repair performance scale with the repair complexity? We categorize complexity along two dimensions, namely the CUT change scale (i.e., the number of changed lines and methods in the CUT) and the test patch adaptation degree (i.e., the number of , Vol. 1, No. 1, Article . Publication date: November 2026.

6

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

Table 1. Details of Projects and Samples in the Benchmark. Project Name

Star Number

shardingsphere [71] druid [66] iotdb [70] spring-ai [77] pinpoint [52] hudi [69] camel [65] hertzbeat [68] spring-ai-alibaba [1] quarkus [57] netty [72] dubbo [67] hazelcast [20] DependencyCheck [50] jenkins [24]

20,714 13,987 6,321 8,674 13,820 6,158 6,199 7,220 9,539 15,657 34,944 41,526 6,568 7,539 25,257

Last Commit #Changed #Repaired Sample Updated Number Tests Tests Number 2026-05-10 2026-05-10 2026-05-11 2026-05-10 2026-05-10 2026-05-10 2026-05-10 2026-05-11 2026-05-11 2026-05-11 2026-05-10 2026-05-10 2026-05-10 2026-05-10 2026-05-10

Total

6,339 1,900 3,091 2,881 1,984 1,992 10,186 1,340 3,137 13,713 1,483 1,126 2,285 1,117 3,340

855 902 1,206 998 238 297 560 124 136 240 104 57 115 39 38

720 574 497 198 155 128 113 72 45 37 60 27 32 16 4

134 107 95 37 29 24 17 13 12 10 9 9 2 1 1

5,909

2,678

500

changed lines and hunks in the golden patch of the broken test). We then analyze the test repair performance for each category to understand how different strategies scale with increasing repair complexity. • RQ5. Can the context configuration strategies studied in the previous RQs benefit agentbased test repair? We apply a SOTA coding agent, Codex [47], to the challenging repair tasks where simply prompting LLMs with test repair instructions (i.e., the setup of RQs1–4) fail and systematically equip the agent with the context found helpful in our experiments of previous RQs. Codex is equipped with a well-crafted harness to guide LLMs to iteratively and autonomously handle coding tasks. The comparison between its performance with and without our context demonstrates the usefulness of our findings in the more powerful agentic setting. 3.2

Benchmark Construction

Existing source–test co-evolution benchmarks suffer from several notable limitations. (1) Simplified source–test co-evolution scenarios. Many benchmarks restrict code changes to overly simplified and aggressively filter out data. For instance, the SOTA benchmark Update4J [93] only includes broken tests caused by a single focal method change. Also, TaRBench [90] only considers test repair patches with a single-hunk change. However, real-world test co-evolution often involves more complex change patterns. In our benchmark, 38.4% of cases have test patches involving more than one changed hunk, which will be further discussed in RQ4 in Section 4. (2) Limited benchmark size. The scale of existing benchmarks is relatively small (e.g., Update4J [93] contains only 165 broken tests), which limits the statistical power and generalizability of experimental results. (3) Test execution reproducibility. Some benchmarks (e.g., CEPROT [21]) do not dynamically validate collected test cases, leaving downstream studies only able to reproduce and dynamically validate 0.44% of the tests, raising concerns about benchmark reliability [7]. (4) Outdated data leading to data contamination. Most commits in TaRBench [90] were submitted before 2020. Even in the most recent benchmark, Update4J [93], a substantial portion of the data predates 2020. Modern LLMs may have learned the correct test patches of these commits in their training data, inducing risks of data contamination. , Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

7

To address these limitations, we construct a new source–test co-evolution benchmark with a larger scale, more realistic code change scenarios, reliable test execution validation, and up-to-date data. The detailed data collection process and benchmark characteristics are described below. Project Selection and Data Filtering. We collected commits from GitHub over a two-year window spanning from May 1, 2024 to April 30, 2026, ensuring that the data reflects recent development practices and partially postdates the training cutoff of modern LLMs, thereby mitigating data contamination. We selected Java projects managed by Maven [64] and instrumented with JaCoCo [12] to support test coverage recording. To focus on well-maintained repositories, we ranked candidates by GitHub stars and retained the top 500. We then removed the projects with fewer than 1,000 commits within the two-year window for insufficient evolution activities and kept 27 candidate projects after project filtering. For each candidate project, we identified genuine test repairs among test method changes following Yaraghi et al. [90], using a three-step dynamic validation procedure executed with Java 17 and Maven 3.9.9. Specifically, for every candidate test evolution instance in each project, we first executed the original test case 𝑇 against the original SUT version 𝑉 and verified that it passed to preserve the instances with test cases that initially pass. Then, we executed 𝑇 against the modified SUT version 𝑉 ′ and checked whether it failed due to incompatibilities introduced by the SUT changes to keep the instances where SUT code changes break the original test Case. Finally, we executed the modified test case 𝑇 ′ against 𝑉 ′ and verified that it passed, confirming a successful source–test co-evolution instance. After this three-step dynamic validation, we obtained 2,678 test cases from 15 projects. From the pool of all dynamically validated co-evolution cases, we randomly sampled 500 instances to construct the final test repair benchmark. To the best of our knowledge, this benchmark is the first to capture large-scale, multi-method and multi-file source–test co-evolution cases with full dynamic validation from very recent software development practices. Detailed benchmark statistics are presented in Table 1. 3.3

Experimental Setup

LLM Selection. We select one commercial LLM (GPT-5.4 [48]) and two open-sourced LLMs (DeepSeek-V3.2, abbr. DS-V3.2 [11] and Qwen3-Coder-480B-Instruct, abbr. Qwen3-Coder [2]) as experimental subjects since they show strong code generation capability and are built by different providers, demonstrating generalizable findings. Note that DeepSeek-V3.2 provides visible reasoning traces for analysis with configurable thinking capabilities when the reasoning mode is enabled. We require high thinking effort in RQ3 to investigate its most extensive reasoning traces. GPT-5.4 also supports configurable thinking efforts but does not provide visible reasoning traces, so we only use it with low thinking effort. We adopt the default configuration with temperature of 0.6, following the recommendation of the respective model vendors, to reflect realistic deployment conditions commonly used by end users [19, 51]. Evaluation Metrics. We adopt two widely-adopted metrics to quantify the effectiveness of test repair for source–test co-evolution following prior work on test repair [36, 93] and test generation [6, 80, 91]: (1) Compilation success rate (comp) measures the proportion of updated tests that compile without errors, capturing syntactic correctness and serving as a prerequisite for execution-based evaluation. (2) Test pass rate (exec) measures the proportion of updated tests that execute and pass against the updated source code, indicating whether the updated tests conform to the current system behavior. Repair Quality Verification. A repaired test may trivially pass by weakening the test oracle (e.g., removing assertions). To detect such degenerate repairs, we employ an LLM-as-a-Judge approach [55, 82, 94] with GPT-5.4 (thinking effort set to high), which compares each repair against , Vol. 1, No. 1, Article . Publication date: November 2026.

8

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

Table 2. Number and ratio of executable (exec) and compilable (comp) test repair generated by different LLMs across various context types over all 500 cases. DeepSeek-V3.2 exec comp Symptom Failure log 88 0.176 149 0.298 Hunk-level 162 0.324 195 0.390 Method-level 163 0.326 194 0.388 Cause File-level 262 0.524 320 0.640 Hunk-level + Failure log 223 0.446 259 0.518 Symptom + Method-level + Failure log 218 0.436 256 0.512 Cause File-level + Failure log 304 0.608 354 0.708 m=1 259 0.518 294 0.588 Hunk-level + m=3 273 0.546 315 0.630 Failure log m=5 274 0.548 315 0.630 m=1 257 0.514 298 0.596 Symptom + Method-level + m=3 258 0.516 290 0.580 Cause + Failure log m=5 266 0.532 301 0.602 Top-𝑚 Support m=1 298 0.596 346 0.692 File-level + m=3 305 0.610 352 0.704 Failure log m=5 311 0.622 357 0.714 Context Type

Setups

Qwen3-Coder exec comp 115 0.230 181 0.362 220 0.440 257 0.514 263 0.526 303 0.606 278 0.556 344 0.688 245 0.490 285 0.570 273 0.546 308 0.616 293 0.586 351 0.702 280 0.560 318 0.636 288 0.576 323 0.646 283 0.566 324 0.648 283 0.566 324 0.648 305 0.610 345 0.690 294 0.588 336 0.672 301 0.602 361 0.722 313 0.626 374 0.748 301 0.602 359 0.718

GPT-5.4 exec comp 157 0.314 226 0.452 260 0.520 293 0.586 276 0.552 307 0.614 296 0.592 339 0.678 275 0.550 304 0.608 283 0.566 316 0.632 321 0.642 364 0.728 302 0.604 338 0.676 324 0.648 356 0.712 320 0.640 356 0.712 308 0.616 351 0.702 317 0.634 354 0.708 315 0.630 359 0.718 341 0.682 386 0.772 339 0.678 385 0.770 345 0.690 391 0.782

the developer-committed test patch. Note that the judge is designed to detect oracle weakening rather than to verify the full semantic correctness of a repair. Across all experimental conditions, we generate a total of 92,054 test repairs. To validate the reliability of the judge, we randomly sample 383 repairs using the standard finite population sample size estimation formula [8] with a 95% confidence level and a margin of error of ±5%. An author independently inspects each sampled repair to determine whether it weakens the test oracle, without access to the judge’s verdict. The human and LLM judgments agree on 98.17% (375 out of 382) of the sampled repairs that only 1.87% (7 out of 375) are oracle-weakening, indicating that the judge reliably identifies oracle-weakening repairs. 4 4.1

Evaluation Results and Analysis RQ1. Impact of Repair Context Types.

In RQ1, we study how Symptom, Cause, and Support contexts contribute to LLMs’ test repair performance. Specifically, as discussed in Section 2.2, we collect failure logs of broken tests as the Symptom context. To represent Cause context, we generate coverage reports using JaCoCo and retrieve hunk-, method-, and file-level differences based on the commit changes tracked by Git [16]. For Support context, we follow recent test update work [7, 93] to prompt a frontier LLM (i.e., DeepSeek-V3.2) to identify the top-𝑚 (𝑚=1, 3, 5) relevant unchanged methods and classes from the SUT based on the broken test, the failure log, and fully qualified names of methods and classes in the CUT changes. We then retrieve their code bodies as context via a language server. We also combine multiple context types to evaluate their complementary effects. Table 2 presents the overall results of different models’ test repair performance on our benchmark across various context configurations. For each of the three LLMs, we present its exec and comp of generated repairs, providing only Symptom (row 1), only Cause (rows 2-4), Symptom and Cause combined (rows 5-7), and combinations of three context types (rows 8-16). We derive two key observations from the results as follows. Three context types complement each other and jointly improve the overall repair performance. As shown in Table 2, incorporating all three context types (Symptom + Cause + Top-𝑚 Support) consistently yields the best performance across all three models. The adoption of both Symptom and Cause (Symptom + Cause) also improves performance compared to using only Symptom or Cause, indicating that each context type provides complementary information for test repair. , Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

9

Specifically, when only the test failure log is provided (Symptom), performance is low (e.g., 23.0% exec and 36.2% comp using Qwen3-Coder). However, combining the failure log with CUT changes (Symptom + Cause) significantly improves the performance, increasing Qwen3-Coder’s exec by 113% with hunk-level CUT changes, 137% with method-level CUT changes, and 155% with file-level CUT changes. Although the performance of all three LLMs with only Cause context is already high, incorporating Symptom further improves exec by 16.0% for DeepSeek-V3.2, 5.40% for Qwen3-Coder, and 8.45% for GPT-5.4. This indicates that the symptom in failure logs, while insufficient on its own, helps the model pinpoint the relevant portion of the production code change. We consider the reason is that each context type captures a distinct aspect of the repair signal: Symptom reveals what and where failed, Cause reveals why it failed, and Support provides additional code context, making them naturally complementary. More context is not always better. Reducing redundancy across code context and matching model capacity are the keys to successful repair. We observe that two factors (i.e., code context’s granularity diversity and model’s capacity) play a critical role in determining the optimal context configuration. First, after incorporating Support context (Symptom + Cause + Top-𝑚 Support), DeepSeek-V3.2 and GPT-5.4 perform worse with method-level CUT changes than with hunk-level, and the advantage of method-level over hunk-level for Qwen3-Coder drops greatly (e.g., from 11.4% to 1.07% in exec with top-1 support). These observations exhibit a general reversal of the granularity trend seen without Support context (Symptom + Cause), likely because the method- and class-level code bodies in Support context overlap with the structural information that already supplied by method-level CUT changes, inducing redundancy that dilutes the model’s focus on the core change. In contrast, hunk-level CUT changes capture a finer-grained code change perspective structurally distinct from Support context, being complementary. This suggests that code context’s granularity diversity across context dimensions, rather than sheer volume, is significant to performance gains. Second, the optimal amount of Support context varies with model capacity. Qwen3-Coder achieves its best exec at top-3 support across all CUT change granularity levels, with top-5 providing no further gain or even a slight regression (Symptom + Cause + Top-𝑚 Support, 𝑚=3, 5). In contrast, GPT-5.4 that has a larger context window continues to benefit from top-5 support (Symptom + Cause + Top-5 Support). This suggests that models with smaller effective context utilization capacity may suffer from information overload when excessive support context is supplied, and that the optimal 𝑚 should be calibrated to the model’s capacity. Answer to RQ1. The three context types are complementary, and their combination offers better performance. Yet inclusion of more context is not always an advantage: code context redundancy and the model’s capacity to utilize context are critical to the optimal context configuration. 4.2

RQ2. Impact of Pipeline Orchestrations

In RQ2, we evaluate LLM performance with three pipeline designs, namely In-context learning (ICL), Multi-attempt repair and Iterative refinement. We evaluate the three LLMs (i.e., DeepSeek-V3.2, Qwen3-Coder, GPT-5.4) under the optimal context configuration (i.e., Symptom + Cause at filelevel + Top-5 Support) together with context configurations with three CUT change granularities (i.e., Symptom + Cause at hunk-/method-/file-level). For ICL, we provide 𝑛-shot examples (𝑛=0, 1, 3) of tests retrieved from version 𝑉 in the prompt to guide the repair, following the common practice for ICL-based coding tasks [31, 33, 34, 45]. We rank candidate tests by similarity to the broken test using three popular combinations of source representation method and comparison method: (1) Lexical, which represents tests as token bags via a PL-based tokenizer and ranks with BM25 [58]; (2) Syntax, which encodes tests as abstract syntax trees (ASTs) and computes similarity via the Jaccard index [45]; and (3) Semantic, which , Vol. 1, No. 1, Article . Publication date: November 2026.

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung lexical similarity syntax similarity semantic similarity

0.6 0.5

0.446

0.4 0.266

0.3 0.258

0.2 n=0

0.284 0.274 0.254

n=1

0.75

0.65 0.6 0.55 0.5

0.490

0.490 0.486

0.45

0.472

0.488 0.480 0.464

Execution Success Rate

Execution Success Rate

0.7

Execution Success Rate

10

0.65 0.6 0.55

0.550

0.5

0.4

n=3

0.7

n=0

n=1

n=0

n=3

0.554 0.544 0.526

0.536 0.540 0.530

n=1

n=3

0.564 0.562 0.552

0.560 0.556 0.548

n=1

n=3

(a1-1) ICL (𝑛-shot) with hunk-level (a1-2) ICL (𝑛-shot) with hunk-level (a1-3) ICL (𝑛-shot) with hunk-level CUT changes and log (DS-V3.2). CUT changes and log (Qwen3- CUT changes and log (GPT-5.4). Coder).

0.55

0.436

0.4 0.294

0.560 0.314 0.292

n=1

n=3

0.296

0.3

0.6

0.292

0.546

0.544 0.522 0.506

0.5

0.560 0.530 0.516

0.65

0.4 n=0

0.7

0.6

0.566

0.55

0.45

0.2

Execution Success Rate

0.5

0.75

0.65

0.6

Execution Success Rate

Execution Success Rate

0.7

0.5 n=0

n=1

n=0

n=3

(a2-1) ICL (𝑛-shot) with method- (a2-2) ICL (𝑛-shot) with method- (a2-3) ICL (𝑛-shot) with methodlevel CUT changes and log (DS- level CUT changes and log (Qwen3- level CUT changes and log (GPTV3.2). Coder). 5.4). 0.6

0.586

0.570

0.55

0.5 0.4 0.3

0.234 0.25 0.246

0.2 n=0

n=1

0.560 0.282

0.576 0.590

0.560 0.578 0.584

0.5

Execution Success Rate

0.6

0.75

0.65 0.608

Execution Success Rate

Execution Success Rate

0.7

0.65 0.6

0.648 0.642 0.640

0.644

n=1

n=3

0.636 0.638

0.5

0.4

0.266

0.642

0.55

0.45 n=0

n=3

0.7

n=1

n=0

n=3

(a3-1) ICL (𝑛-shot) with file-level (a3-2) ICL (𝑛-shot) with file-level (a3-3) ICL (𝑛-shot) with file-level CUT changes and log (DS-V3.2). CUT changes and log (Qwen3- CUT changes and log (GPT-5.4). Coder). 0.6

0.602

0.566

0.55

0.5 0.4

0.324

0.3

0.32

0.2 n=0

n=1

0.304 0.298 0.288

n=3

0.588 0.552

0.560 0.582 0.568

0.5

0.45

Execution Success Rate

0.6

0.75

0.65

0.622

Execution Success Rate

Execution Success Rate

0.7

0.688

0.7 0.65

0.690

0.654

0.672

0.654 0.648 0.660

n=1

n=3

0.6 0.55 0.5

0.4 n=0

n=1

n=3

n=0

(a4-1) ICL (𝑛-shot) with file-level (a4-2) ICL (𝑛-shot) with file-level (a4-3) ICL (𝑛-shot) with file-level CUT changes, log, top-5 support CUT changes, log, top-5 support CUT changes, log, top-5 support (DS-V3.2). (Qwen3-Coder). (GPT-5.4). Fig. 2. Execution success rate (exec) of tests repaired by different LLMs with ICL under different context configurations. Panels (a1-1)-(a1-3), (a2-1)-(a2-3), (a3-1)-(a3-3), and (a4-1)-(a4-3) correspond to Hunk-level + Failure log, Method-level + Failure log, File-level + Failure log, and File-level + Failure log + Top-5 Support respectively.

encodes tests using CodeBERT [13] and ranks by cosine similarity. For Multi-attempt repair, we , Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

0.7 0.6 0.5 0.4

0.656

0.656

0.630 0.518

0.630 0.520

0.436 0.446

0.490

0.490

k=1

k=2

0.622 0.608

0.9

0.9

0.8 0.708 0.672 0.616

0.670

0.7

0.648 0.590

0.602 0.586 0.546

0.6 0.5

0.554

0.572

k=2

k=3

0.490

Execution Success Rate

0.8

Hunk-level + Failure log Method-level + Failure log File-level + Failure log File-level + Failure log + Top-5 support

Execution Success Rate

Execution Success Rate

0.9

11

0.4

0.8 0.7

0.690 0.642

0.6

0.566

0.5

0.724 0.682

0.744 0.694

0.600

0.608 0.596

0.578

0.550

0.4

k=1

k=3

k=1

k=2

k=3

(b1) Repair with 𝑘 attempts (DS- (b2) Repair with 𝑘 attempts (Qwen3- (b3) Repair with 𝑘 attempts (GPTV3.2). Coder) 5.4) Fig. 3. Execution success rate (exec) of tests repaired by different LLMs with multiple attempts under different context configurations. 0.9

0.700

0.7 0.6 0.5 0.4

0.602

0.676

0.574

0.516 0.514

0.742 0.706 0.720

0.730 0.640 0.628

0.592

0.400 0.420

r=0

0.9

0.8 0.7 0.598

0.746 0.722 0.736 0.688 0.706 0.678 0.666 0.720

0.6 0.558 0.5

0.518

0.596

0.564

0.616

0.648

0.484

0.4 r=1

r=2

r=3

Execution Success Rate

0.8

0.766

Execution Success Rate

Execution Success Rate

0.9

0.789

0.8 0.7

0.680

0.734

0.638 0.578 0.6

0.5

0.828 0.784 0.720

0.662

0.850 0.816 0.758

0.704

0.738

0.634

0.550

0.4

r=0

r=1

r=2

r=3

r=0

r=1

r=2

r=3

(c1) Repair with 𝑟 iterative rounds (c2) Repair with 𝑟 iterative rounds (c3) Repair with 𝑟 iterative rounds (DS-V3.2). (Qwen3-Coder). (GPT-5.4). Fig. 4. Execution success rate (exec) of tests repaired by different LLMs with iterative rounds under different context configurations.

generate up to 𝑘 repair attempts (𝑘=1, 2, 3) for each test, and consider the repair successful if any attempt executes successfully. For Iterative refinement, we refine prior repair up to 𝑟 iterations (𝑟 =0, 1, 2, 3) by feeding back the execution result and failure log from each prior attempt, stopping upon a successful repair or when 𝑟 =3. Figures 2, 3, and 4 illustrate the performance of the three LLMs with ICL, multi-attempt repair, and iterative refinement, respectively, across various context configurations. As shown in Figure 2, surprisingly, ICL with few-shot examples does not provide additional benefits over zero-shot repair across all three LLMs and all four context configurations, and sometimes even leads to performance degradation. This suggests that test examples retrieved from version 𝑉 may not serve as effective repair demonstrations, despite sharing similar code features with the broken test. A likely explanation is that in source–test co-evolution, the broken test and its retrieved examples from version 𝑉 share the same outdated patterns: in source–test co-evolution practice, similar test patches are often applied in a batch (i.e., in the same commit submission). Although they may help code generation within the same code version, these examples tend to co-evolve together with the broken test and thus introduce noise rather than useful guidance. Notably, the degradation is particularly severe for DeepSeek-V3.2, whose exec drops sharply once few-shot examples are added (e.g., from 0.608 to around 0.25 with File-level + Failure log), whereas Qwen3-Coder and GPT5.4 remain relatively stable. This might because DeepSeek-V3.2 relies more heavily on in-context examples, imitating the outdated code patterns in the retrieved examples rather than adapting the broken test to the changed CUT. Meanwhile, multi-attempt repair (Figure 3) yields steady improvements across all the three LLMs and all context settings, as more independent attempts increase the chance of a passing repair. , Vol. 1, No. 1, Article . Publication date: November 2026.

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

Execution Success Rate

0.8 0.75 0.7

Hunk-level + Log DeepSeek-v3.2 DeepSeek-V3.2 Hunk-level + Log DeepSeek-v3.2 DeepSeek-V3.2 (thinking) (thinking) 0.656

0.65

0.608

0.6

0.668 0.622

0.55 0.5 0.45 0.4

0.446

0.468

Hunk-level + Failure Log

0.436

0.458

Method-level + Failure Log

File-level + File-level + Failure Failure Log Log + Top-5 Support

(a) Execution success rate (exec).

Compilation Success Rate

12

0.8 0.75 0.7

Hunk-level + Log DeepSeek-v3.2 Hunk-level + Log DeepSeek-v3.2 (thinking) 0.708

0.714

0.714

0.748

0.65 0.6 0.55

0.518

0.53

0.5

0.512

0.52

0.45 0.4

Hunk-level + Failure Log

Method-level + Failure Log

File-level + File-level + Failure Failure Log Log + Top-5 Support

(b) Compilation success rate (comp).

Fig. 5. Comparison of LLM with thinking mode enabled versus disabled on test repair performance.

Iterative refinement achieves larger gains for all the three LLMs, most pronounced in the first iteration, as shown in Figure 4. This suggests that properly collecting and utilizing initial repair attempt’s feedback for further improvement is more effective than simply generating a second independent attempt. Answer to RQ2. ICL with few shots retrieved from SUT 𝑉 does not improve repair performance and may even degrade it, whereas multi-attempt repair and iterative refinement both help. Iterative refinement brings substantial gains in the first round, suggesting that feedback from initial attempts is crucial to repair effectiveness. 4.3

RQ3. Performance and Behavior Investigation for the LLM with Reasoning Mode Enabled

In RQ3, we first compare the performance of DeepSeek-V3.2 with reasoning mode enabled (thinking effort set to high) versus disabled across four context configurations (same as RQ2) to quantify the impact of reasoning capabilities on test repair. We focus on DeepSeek-V3.2 first since it is the only open-source LLM among the three that offers configurable reasoning ability and accessible reasoning traces. Next, we analyze the reasoning traces produced by DeepSeek-V3.2 with reasoning mode enabled during successful repairs to identify recurring behaviors and distill them into a structured taxonomy. Finally, we derive two optimization strategies based on the reasoning behavior taxonomy. In order to evaluate the generalizability, we apply the two optimization strategies to all three LLMs (i.e., DeepSeek-V3.2, Qwen3-Coder, GPT-5.4) in terms of both repair effectiveness and cost efficiency. The model with reasoning mode enabled consistently outperforms the model with reasoning mode disabled across all context configurations. Figure 5 presents the experimental results, with detailed statistics provided in Table 4 (rows of model DS-V3.2 and DS-V3.2 (thinking) with baseline optimization setup). We observe that enabling reasoning mode yields consistent improvements in both exec and comp across all context configurations. It achieves the largest exec gain of 7.89% with File-level + Failure log context, and the smallest exec gain of 4.93% with Hunk-level + Failure log context. Successful repairs by the LLM with reasoning mode enabled follow a four-stage reasoning process comprising 14 common reasoning action patterns. To understand how LLM with reasoning mode enabled succeed at test repair, we manually analyzed reasoning traces of successfully repaired cases using the open coding protocol [26], following Halim et al. [19]. Two annotators with over eight years of programming experience independently labeled fine-grained reasoning behaviors in a random sample of traces, then iteratively grouped similar labels and resolved disagreements through discussion with an arbitrator until consensus was reached. Finally, we develop a reasoning action taxonomy comprising four stages with 14 common reasoning action patterns. Stage 1. Log-guided failure triage. The model begins by diagnosing test failure types and likely reasons from failure logs. Three types of actions are often observed in this stage: (A) Failure , Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

13

Table 3. Statistics of reasoning action taxonomy. A.P. denotes action pattern. S./ F. denote successful/ failure. The numbers in parentheses in the column headers indicate the total number of successful/ failure cases across four context configurations. For each stage and A.P., the table reports the number (ratio in %) of successful cases that include/ failure cases that miss it. Stage A.P.

Hunk-level + Failure log # S. cases include the A.P. (234)

A 1

B C

230 (98.3)

D 2

3

203 (86.8) 165 (70.5)

99 (37.2)

52 (19.5) 12 (4.51) 50 (18.8)

233 (99.6)

0 (0.00)

234 (100) 217

0 (0.00)

# S. cases include the A.P. (229)

224 (97.8)

229

173 (75.5) 192 (83.8) 147 (64.2)

# F. cases miss the A.P. (271)

96 (35.4)

# S. cases include the A.P. (328)

51 (18.8)

317

15 (5.54)

(96.7)

49 (18.1)

229 (100)

1 (0.37)

227 (99.1) 226

1 (0.37)

File-level + Failure log

File-level + Failure log

328

234 (71.3) 287 (87.5) 214 (65.2)

# F. cases miss the A.P. (172)

70 (40.7)

44 (25.6) 17 (9.88) 30 (17.4)

328 (100)

4 (2.33)

327 (99.7) 145

4 (2.33)

+ Top-5 support # S. cases include the A.P. (334)

323 (96.7)

334

216 (64.7) 282 (84.4) 234 (70.1)

# F. cases miss the A.P. (166)

77 (46.4)

54 (32.5) 17 (10.2) 36 (21.7)

333 (99.7)

2 (1.20)

333 (99.7) 143

3 (1.81)

E

234

F

(100) 132 (56.4) (81.6) 83 (31.2) (100) 143 (62.4) (83.4) 89 (32.8) (100) 200 (61.0) (84.3) 71 (41.3) (100) 211 (63.2) (86.1) 75 (45.2)

G

44 (18.8)

215 (80.8)

45 (19.7)

226 (83.4)

68 (20.7)

145 (84.3)

55 (16.5)

H

200 (85.5)

32 (12.0)

200 (87.3)

22 (8.12)

297 (90.6)

28 (16.3)

306 (91.6)

I J

211 (90.2)

K 4

184 (78.6)

# F. cases miss the A.P. (266)

Method-level + Failure log

38 (16.2) 106 (45.3)

248 (93.2)

227 (85.3) 113 (42.5)

(93.9)

33 (14.4) 121 (52.8)

249 (91.9)

229 (84.5) 111 (41.0)

315 (96.0)

22 (6.71) 189 (57.6)

164 (95.3)

153 (89.0) 63 (36.6)

322 (96.4)

34 (10.2) 170 (50.9)

150 (90.4)

22 (13.3) 141 (84.9) 62 (37.3)

228 (97.4)

2 (0.75)

226 (98.7)

2 (0.74)

321 (97.9)

5 (2.91)

328 (98.2)

3 (1.81)

129 (55.1) 255

143 (53.8) 229

108 (47.2) 239

130 (48.0) 327

146 (44.5) 158

84 (48.8) 333

155 (46.4) 151

77 (46.4)

L

234

M

(100) 232 (99.1) (95.9) 3 (1.13)

N

215

141 (84.9)

23 (9.83)

241 (90.6)

(100) 224 (97.8) (88.2) 8 (2.95) 21 (9.17)

(99.7) 314 (95.7) (91.9) 14 (8.14) (99.7) 324 (97.0) (91.0) 13 (7.83)

228 (84.1)

34 (10.4)

149 (86.6)

36 (10.8)

138 (83.1)

type classification: Classify the failure into compilation error or runtime error. (B) Broken element identification: Identify the broken symbols, invocations, assertions, or statements based on the failure log and broken test. (C) Failure summarization: Analyze the failure’s root cause and generate a concise summary before proceeding to repair. Stage 2. CUT delta analysis. The model then reasons about code symbols and program behavior changes from CUT diff, performing the following four typical actions: (D) CUT diff summarization: Summarize symbol (e.g., API) and behavior changes relevant to the test failure in the CUT before editing the test. (E) Old-to-new element mapping construction: Construct a mapping from obsolete test elements to their replacements in the new version. (F) Signature update detection: Detect constructor or method signature changes and identify missing/ changed arguments. (G) New parameter handling: Identify newly required parameters and select appropriate values (e.g., existing variables, null, default or mock values). Stage 3. Repair planning. Next, the model plans concrete repair actions based on the previous failure diagnosis and CUT analysis before editing code, performing three typical actions: (H) Stepwise decomposition: Break the repair into an ordered sequence of edits or plan steps. (I) Referring to relevant usage examples: Refer to relevant code snippets in the codebase to use as replacement idioms for the updated API. (J) Original test intent analysis: Infer the original test’s verification goal to preserve it in the repair. Stage 4. Repair generation. Finally, the model generates the repaired test and verifies output correctness before responding with four typical actions: (K) Test method update: Generate the repaired test method. (L) Oracle update: Update expected values in test assertions to reflect the new behavior. (M) Imports management: Add import statements only when needed and remove obsolete ones. (N) Self-check (reflection): Check the format, syntax, and semantic correctness of the test. This taxonomy reveals that effective test repair by the LLM with reasoning mode enabled is a structured, multi-stage reasoning process that mirrors how an expert developer diagnoses a failure, understands the changed code, plans the fix, and verifies the result. Table 3 presents statistics for the four stages and 14 action patterns across context configurations. We report the number and proportion of successful cases that include each stage or action pattern and the failed cases that miss each stage or action pattern. Overall, all four stages are observed in reasoning traces of over , Vol. 1, No. 1, Article . Publication date: November 2026.

14

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

Table 4. Test repair performance of DeepSeek-V3.2 (abbr. DS-V3.2) with/ without optimized prompting/ pipeline design strategies, and DS-V3.2 with reasoning mode enabled (DS-V3.2 (thinking)). Avg. Token Count per Case Avg. Input Cost Reasoning Output (USD) exec comp cache miss cache hit baseline 223 0.446 259 0.518 4792 \ \ 451 0.00344 single-stage 226 0.452 270 0.540 5233 \ \ 433 0.00366 DS-V3.2 Hunk-level + multi-stage 244 0.488 292 0.584 5729 12247 \ 367 0.00468 Failure log DS-V3.2 (thinking) baseline 234 0.468 265 0.530 4792 \ 2305 448 0.00731 baseline 218 0.436 256 0.512 5873 \ \ 474 0.00408 single-stage 233 0.466 261 0.522 6314 \ \ 435 0.00427 DS-V3.2 Method-level multi-stage 260 0.520 306 0.612 6818 14424 \ 355 0.00542 + Failure log DS-V3.2 (thinking) baseline 229 0.458 260 0.520 5872 \ 2368 471 0.00806 baseline 304 0.608 354 0.708 14875 \ \ 452 0.00909 single-stage 307 0.614 364 0.728 15316 \ \ 435 0.00931 DS-V3.2 File-level + multi-stage 305 0.610 360 0.720 15445 31366 \ 353 0.01144 Failure log DS-V3.2 (thinking) baseline 328 0.656 357 0.714 14543 \ 2298 443 0.01275 baseline 311 0.622 357 0.714 15654 \ \ 447 0.00952 File-level + single-stage 323 0.646 387 0.774 16095 \ \ 432 0.00974 DS-V3.2 Failure log + multi-stage 325 0.650 389 0.778 15518 31547 \ 387 0.01155 Top-5 support DS-V3.2 (thinking) baseline 334 0.668 374 0.748 15341 \ 2372 471 0.01337

Context Setup

Model

Strategy

Performance

90% successful cases. Six action patterns (i.e., B, D, E, H, K, M) appear in more than 70% of successful cases for all context configurations. Meanwhile, action patterns A, C, and F appear in over 60% of successful cases while being missed from about 20%–30% of failed cases. These observations suggest that the identified reasoning stages and action patterns are commonly involved in test repair by the LLMs with reasoning mode enabled; meanwhile, the patterns frequently missing from failed cases may serve as critical instructions to incentivize LLMs’ success. To assess the practical usefulness of the identified taxonomy, we further design the following two reasoning strategies for LLMs based on it: • Optimized single-stage prompting guides LLMs to repair tests in a single query, with the prompt structured to instruct the model through each reasoning stage and action patterns. • Optimized multi-stage prompting guides LLMs to decompose repair into four sequential queries following the divide-and-conquer intuition, one per stage, with each stage’s output passed as context to the next. The two strategies impose the taxonomy at different levels: single-stage prompting describes the process in the prompt at almost no extra cost, while multi-stage prompting enforces it through the pipeline at the price of more queries and longer context. The strategies aim to guide LLMs to perform test repair by following the validated effective process without requiring them to plan the reasoning steps themselves. The prompts are adapted from the SOTA LLM-based test update technique TestUpdater [93] and are available in our replication package in Section 8. We first apply these strategies to DeepSeek-V3.2 with reasoning mode disabled. Surprisingly, our optimized prompting strategies achieve comparable or superior performance to reasoning mode at lower cost. As shown in Table 4, both strategies (rows of DS-V3.2 with single-/multi-stage) consistently improve DeepSeek-V3.2 over the baseline prompting (rows of DS-V3.2 with baseline) across all four context configurations. Moreover, the multi-stage optimization strategy can achieve performance comparable to or even better than DeepSeek-V3.2 with reasoning mode enabled (e.g., it achieves higher exec with hunk- and method-level CUT changes and higher comp under all four context configurations). In addition, both strategies cost less than the original reasoning mode, since they produce no lengthy reasoning tokens and most of the additional input of the multi-stage , Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

15

Table 5. Test repair performance of Qwen3-Coder and GPT-5.4 with/ without optimized prompting/ pipeline design strategies. Avg. Token Count per Case Avg. Cost Input Reasoning Output (USD) exec comp cache miss cache hit baseline 245 0.490 285 0.570 4705 \ \ 401 0.00671 \ \ 393 0.00709 Qwen3-Coder single-stage 266 0.532 311 0.622 5130 multi-stage 229 0.458 285 0.570 5610 12202 \ 436 0.00901 Hunk-level + baseline 275 0.550 304 0.608 4402 \ \ 403 0.01704 Failure log single-stage 296 0.592 328 0.656 4826 \ \ 386 0.01785 GPT-5.4 multi-stage 278 0.556 318 0.636 5303 11267 \ 470 0.02312 baseline 273 0.546 308 0.616 5756 \ \ 400 0.00775 \ \ 390 0.00813 Qwen3-Coder single-stage 302 0.604 342 0.684 6181 multi-stage 250 0.500 304 0.608 6609 14188 \ 429 0.01017 Method-level baseline 283 0.566 316 0.632 5461 \ \ 408 0.01977 + Failure log single-stage 314 0.628 344 0.688 5885 \ \ 386 0.02051 GPT-5.4 multi-stage 301 0.602 336 0.672 6362 13385 \ 466 0.02625 baseline 293 0.586 351 0.702 14100 \ \ 397 0.01608 \ \ 387 0.01646 Qwen3-Coder single-stage 294 0.588 357 0.714 14525 multi-stage 279 0.558 351 0.702 14507 29592 \ 436 0.01965 File-level + baseline 321 0.642 364 0.728 13876 \ \ 420 0.04099 Failure log single-stage 339 0.678 390 0.780 14300 \ \ 387 0.04156 GPT-5.4 multi-stage 353 0.706 405 0.810 14777 30214 \ 466 0.05148 baseline 301 0.602 359 0.718 14823 \ \ 396 0.01680 \ \ 387 0.01718 Qwen3-Coder single-stage 317 0.634 379 0.758 15248 File-level + multi-stage 284 0.568 351 0.702 14672 29953 \ 447 0.01990 Failure log + baseline 345 0.690 391 0.782 14610 \ \ 416 0.04277 Top-5 support single-stage 351 0.702 400 0.800 15034 \ \ 388 0.04340 GPT-5.4 multi-stage 360 0.720 415 0.830 14803 30292 \ 517 0.05233

Context Setup

Model

Strategy

Performance

strategy hits the cache. This suggests that the summarized effective reasoning process avoids unnecessary or even incorrect reasoning steps and paths, thus improving cost-effectiveness. To evaluate generalizability, we further apply both strategies to the other two LLMs (Table 5). We further observe that the single-stage strategy generalizes well to all other LLMs. The single-stage strategy improves both exec and comp of both GPT-5.4 and Qwen3-Coder across all four context configurations, with marginal additional cost (at most 5.7%) (e.g., with Method-level + Failure log, it relatively improves the exec of GPT-5.4 by 11.0% and that of Qwen3-Coder by 10.6%). The multistage strategy also benefits GPT-5.4 under all four context configurations, but slightly decreases Qwen3-Coder’s repair performance. A possible reason is that the four-stage pipeline accumulates intermediate context, to which Qwen3-Coder appears more sensitive, as observed in RQ1. Overall, single-stage prompting is a robust default with consistent gains at marginal cost, while multi-stage prompting can bring larger gains for certain combination of model and context but is costlier and model-dependent. These results suggest that the reasoning patterns distilled from DeepSeek-V3.2 can benefit LLMs beyond it. Answer to RQ3. Explicit reasoning traces improve model performance across all context settings. Reasoning traces from successful repairs reveal a taxonomy of a four-stage process with 14 common action patterns and proves effective for guiding LLMs to repair tests cost-effectively. 4.4

RQ4. Performance across Scenario Complexity

In RQ4, we study how repair performance changes as the complexity of source–test co-evolution scenarios increases. As repair difficulty cannot be measured directly, we approximate the complexity , Vol. 1, No. 1, Article . Publication date: November 2026.

16

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

0.6

0.776

0.7

0.699 0.664 0.661 0.664

0.612 0.571 0.523

0.5

0.494 0.462 0.430

0.4

0.3

0.9

0.796

1-3

4-10

0.440 0.459 0.431 0.440 0.385 0.385

0.714 0.694

0.8

0.653 0.633

0.611 0.582 0.579 0.541 0.512 0.482

0.6 0.5

0.459

0.4

0.486 0.468 0.459 0.413 0.431

Execution Success Rate

Execution Success Rate

0.7

0.8

0.735 0.755 0.735 0.735 0.673 0.694

Execution Success Rate

0.8

0.7

0.878 0.837 0.816 0.757 0.749

0.776 0.714

0.722 0.675 0.599

0.6

0.561 0.553 0.544

0.5

0.367 0.358

0.339 0.312

0.3

>10

1-3

4-10

0.505

0.459 0.468

0.4

>10

Number of changed lines in the test patch

Number of changed lines in the test patch

0.578 0.532 0.514 0.495

1-3

4-10

>10

Number of changed lines in the test patch

(a1) Perf. change with test patch (a2) Perf. change with test patch (a3) Perf. change with test patch adaptation degree by code lines adaptation degree by code lines adaptation degree by code lines (DS-V3.2). (Qwen3-Coder). (GPT-5.4).

0.669

0.6

0.649

0.627

0.620

0.583

0.620

0.578 0.594

0.5 0.4

0.3

0.523 0.497 0.451

0.516 0.474 0.438

0.448 0.411

1

>1

Number of changed hunks in the test patch

Execution Success Rate

Execution Success Rate

0.7

0.8

0.9

0.7

0.8 0.617

0.6 0.5

0.591 0.571 0.568 0.552 0.513

0.578 0.563 0.542 0.479

0.461

0.453

0.4 0.3

1

>1

Number of changed hunks in the test patch

Execution Success Rate

0.8

0.7

0.740

0.724 0.708

0.604 0.568 0.558 0.558

0.5 0.4

0.688

0.661

0.646

0.6

B-prompt hunk +log B-prompt method+log B-prompt file+log B-prompt file+log+support O-prompt hunk +log O-prompt method+log O-prompt file+log O-prompt file+log+support

1

0.635

0.677 0.599 0.563 0.552

0.536

>1

Number of changed hunks in the test patch

(b1) Perf. change with test patch (b2) Perf. change with test patch (b3) Perf. change with test patch adaptation degree by code hunks adaptation degree by code hunks adaptation degree by code hunks (DS-V3.2). (Qwen3-Coder). (GPT-5.4). Fig. 6. Execution success rate (exec) of tests repaired by different LLMs with baseline prompting (abbr., B-prompt) and optimized multi-stage prompting (abbr., O-prompt) under different context configurations (Hunk-/Method-/File-level + Failure log + Top-5 Support) across scenario complexity levels measured by test patch adaptation degree. The complexity is measured by test changed lines in (a1)-(a3) and by test changed hunks in (b1)-(b3).

of a co-evolution scenario by its change scale on both the test and CUT sides, since larger changes generally require LLMs to understand more modifications and produce more edits. Specifically, we measure the complexity along two dimensions. • Test patch adaptation degree, which is quantified by the number of changed code lines and hunks in the broken test’s golden patch, reflecting the adaptation effort needed to align the broken test with the new CUT behavior. • CUT change scale, which is quantified by the number of changed lines and methods in the CUT, as larger-scale changes are more likely to introduce complex, interdependent co-evolution scenarios that require more sophisticated understanding to repair the broken test. We categorize repair cases into different complexity levels and analyze repair performance within each category. Figures 6 and 7 show the repair performance of three LLMs (i.e., DeepSeek-V3.2, , Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs 0.8

0.8

0.6 0.607 0.571

0.660

0.643 0.622 0.592

0.643 0.640

0.520

0.5 0.500

0.3

0.530

0.490

0.464

0.7

0.673

0.480 0.460

0.429

0.447

0.500

0.473

0.459 0.432

0.405 0.338 0.311

0.321

1-3

4-10

11-100

>100

Execution Success Rate

Execution Success Rate

0.643

0.684

0.9 0.714

0.8

0.679 0.643

0.633 0.643 0.622 0.622 0.6 0.602 0.597 0.571 0.571 0.561 0.577 0.551 0.543 0.536 0.531 0.507 0.5 0.500 0.483 0.464 0.437

0.4 0.3

Number of changed lines in the CUT

1-3

4-10

0.617 0.573 0.500 0.446 0.446 0.432 0.405 0.392

11-100

Execution Success Rate

0.704

0.7

0.4

17

0.765 0.745

0.7

0.679 0.643

0.6

0.571

0.735 0.730 0.727 0.680

0.663

0.607 0.607

0.737

0.610

0.592

0.4

0.581

0.571 0.560 0.563 0.582

0.541

0.557

0.5

>100

Number of changed lines in the CUT

0.821

0.514 0.473 0.473 0.446

1-3

4-10

11-100

Number of changed lines in the CUT

>100

(c1) Perf. change with CUT change (c2) Perf. change with CUT change (c3) Perf. change with CUT change scale by code lines (DS-V3.2). scale by code lines (Qwen3-Coder). scale by code lines (GPT-5.4).

0.670 0.614

0.6 0.5 0.4

0.3

0.640 0.633 0.618

0.591

0.651 0.642 0.670 0.642

0.535 0.528 0.512 0.509 0.512 0.504 0.500 0.491 0.472 0.465 0.442 0.431 0.406 0.387

1-3

4-10

>10

Number of changed methods in the CUT

Execution Success Rate

Execution Success Rate

0.7

0.8

0.9

0.7

0.8 0.622

0.6

0.5

0.591 0.598 0.551 0.551 0.528 0.520 0.488

0.577 0.558 0.539 0.506 0.476 0.442

1-3

0.594 0.566

0.557

0.519 0.491 0.462 0.453

0.4

0.3

0.613

0.588

4-10

>10

Number of changed methods in the CUT

Execution Success Rate

0.8

0.7

B-prompt hunk +log B-prompt method+log B-prompt file+log B-prompt file+log+support O-prompt hunk +log O-prompt method+log O-prompt file+log O-prompt file+log+support

0.701

0.730 0.719

0.712 0.717

0.678 0.646 0.607 0.591 0.6 0.583 0.591 0.554 0.575 0.547 0.567 0.547

0.689

0.669

0.5

0.4

1-3

4-10

0.632 0.604 0.594 0.538 0.519

>10

Number of changed methods in the CUT

(d1) Perf. change with CUT change (d2) Perf. change with CUT change (d3) Perf. change with CUT change scale by methods (DS-V3.2). scale by methods (Qwen3-Coder). scale by methods (GPT-5.4). Fig. 7. Execution success rate (exec) of tests repaired by different LLMs with baseline prompting (abbr., B-prompt) and optimized multi-stage prompting (abbr., O-prompt) under different context configurations (Hunk-/Method-/File-level + Failure log + Top-5 Support) across scenario complexity levels measured by CUT change scale. The complexity is measured by CUT changed lines in (c1)-(c3) and by CUT changed methods in (d1)-(d3).

Qwen3-Coder and GPT-5.4) with two strategies (i.e., baseline prompting [93] and multi-stage optimization prompting from RQ3) under four context configurations (i.e., hunk- + failure log, method+ failure log, file- + failure log, and file-level + failure log + top-5 support) across complexity levels, where complexity is measured by the test patch adaptation degree and the CUT change scale, respectively. Higher repair scenario complexity generally leads to lower repair performance, while richer context mitigates this effect. As shown in Figure 6, for all three LLMs, exec declines across all prompting strategies and context configurations as the test patch adaptation degree increases. Taking DeepSeek-V3.2 as an example, the degradation is most notable when the number of changed code lines in the test patch exceeds 10, decreasing by 29.2% on average (Figure 6a1). Regarding the complexity in terms of changed code lines in the CUT (Figure 7c1), six out of eight experimental groups peak at 4–10 changed lines and gradually degrade to their performance bottom beyond 100 changed lines. This trend indicates that when extensive modifications are made to the CUT, repairing becomes more challenging for LLMs. In Figure 7d1, as the number of changed methods , Vol. 1, No. 1, Article . Publication date: November 2026.

18

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

in the CUT increases, method- and hunk-level context degrade in exec while file-level context improves with a decreasing growth rate, suggesting that richer context mitigates degradation when the CUT changes extensively. We can observe that other LLMs exhibit similar trends. Lower-level CUT change context exhibit advantages in providing focused information when the CUT change scale is small. As shown in Figures 7c1–7c3, when only 1–3 lines change in the CUT, hunk-level CUT context outperforms method-level context for all three LLMs with multistage prompting, and for DeepSeek-V3.2 it even surpasses file-level context by 16.6% (Figure 7c1). Similarly, when no more than 3 methods change in the CUT, hunk-level context outperforms method-level context by 10.1% for DeepSeek-V3.2 with baseline prompting (Figure 7d1). This might because hunk-level CUT changes offer more focused and precise information for test repair when the CUT change scale is small, exhibiting advantages in simpler co-evolution scenarios. However, this advantage is model-dependent and diminishes when the CUT change scale is measured by the number of changed methods (Figures 7d1–7d3). Answer to RQ4. Higher scenario complexity generally leads to lower repair success rates while richer context can mitigate this to some extent. In simple co-evolution scenarios, lower-level CUT changes provide more focused information and perform competitively. 4.5

RQ5. Generalization to Agent-Based Test Repair

In RQ5, we examine whether the helpful test repair context identified in RQ1 also benefits the autonomous LLM agents, which are equipped with powerful coding harnesses that orchestrate iterative LLM actions. We adopt Codex [47], a SOTA coding agent that integrates well with GPT-5.4, which performs best in our experiments. Specifically, we compare a baseline run and a contextenhanced run of Codex, both using the same Codex version (0.142.0-alpha.1), model (GPT-5.4 with high thinking effort), and harness. The only difference is that the context-enhanced run is explicitly provided with the three contexts. To be more specific, for each repair, we provide the agent with two workspaces: the project at version 𝑉 and the project at version 𝑉 ′ . In the 𝑉 ′ workspace, every test file modified by the commit, including the file containing the broken test, is reverted to its version-𝑉 content, so that no developer-written test patch from the same commit is visible to the agent. Test utility classes and test resources changed by the commit are treated as part of the code under test and kept at version 𝑉 ′ , consistent with the Cause context in RQ1. To keep dynamic validation possible, the other reverted test files are excluded from test compilation. We remove all Git-related information from both workspaces so that the agent cannot access the repository history or the ground-truth repair. The agent is only allowed to modify the specified test file, while retaining the tools needed for dynamic validation (including shell, Python, third-party libraries, and Maven), and is instructed to produce a single final repair. Given the time and budget cost of running the agent at scale, we apply it to a challenging subset of our benchmark to study whether the guidance derived from our empirical study can benefit the powerful agents. Specifically, we identify the 155 test repair tasks that GPT-5.4 failed under the best-performing context configuration (i.e., Symptom + Cause at file-level + Top-5 Support) in RQ1. We further focus on cases involving large-scale CUT changes by selecting co-evolution cases introduced by commits modifying at least 100 Java source files, yielding 27 challenging cases from 6 projects. These cases represent the challenging end in our benchmark, where LLMs with only test repair instructions struggle. The experimental results show that directly applying the agent Codex to these 27 cases yields 12 (44.4%) correct repairs, while providing the three types of context information improves the agent’s performance to 20 (74.1%) correct repairs (i.e., +29.6 percentage points, 95% CI: [+11.1, , Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

19

+48.1] pp by paired bootstrapping over cases with 10,000 resamples), corresponding to a 66.7% relative improvement (95% CI: [+16.7%, +175.0%]). This suggests that the guidance summarized from our study can also effectively enhance the test repair performance of powerful agents. To understand how the provided context helps the agent, we manually analyze the cases where it fails to repair the broken tests without context while successfully repairing them with context. We found that although the agent can search across an entire codebase and compare changes between two versions, it may not always correctly locate the critical information for test repair. We observe that providing clear Symptom, Cause, and Support contexts helps the agent in three important ways. • (𝑊 1) Context directs the agent to repair the correct API adaptation point instead of rewriting downstream assertions. • (𝑊 2) Context helps the agent preserve the original test intent, preventing the agent from drifting to an unrelated execution path without producing a passing patch. • (𝑊 3) The three types of context complement each other in enabling the agent to locate the actual relevant CUT change behind the observed failure behavior. We present two cases to illustrate how Codex benefits from these contexts. After source code changes made by commit 8805b3e3d6 [4] in project apache/iotdb, without the three types of context provided by us, Codex’s repair attempt modified the result-checking logic (𝑊 1), incorrectly assuming that the spill-reader behavior had changed. The actual issue lies in the test-fixture code that no longer matched the new API (i.e., DiskSpiller had become abstract and SortBufferMana ger now required constructor arguments). In this case, the Symptom context exposed the exact compilation mismatch error, the Cause context localized the change to DiskSpiller and SortBuf ferManager and the Support context exposed their new API shape (𝑊 3). With these cues, Codex recognized this API mismatch and repaired the fixture setup directly while keeping the original assertions unchanged (𝑊 1). Another example is about the broken test caused by commit cda0888352 [53] in project pinpointapm/pinpoint. Without context provided, Codex bypassed the original PluginTestAgent bootstrap path and rewrote the test to instantiate DefaultApplicationContext, changing the program behavior that was intended to be validated by the original test while still not leading to a successful repair (𝑊 2). In this case, the Symptom context revealed the incompatibility around AgentOptio n, the Cause context localized the relevant CUT change to PluginTestAgent and OverrideMod uleFactory, and the Support context exposed the updated constructor shape of PluginTestAg ent (𝑊 3). With such guidance, Codex preserved the original bootstrap scenario and repaired it by supplying the missing agentPath (𝑊 2), which was the minimal change needed to satisfy the updated startup path. Answer to RQ5. Providing the context that proves helpful for LLMs in previous RQs effectively improves Codex’s performance in test repair by 66.7% on challenging cases. Case studies show that such contexts help the agent choose the correct repair target, avoid drifting away from the original behavior, and locate the actual relevant CUT change. 5

Threats to Validity

Internal validity. LLM outputs can vary with temperature [49]. To observe representative model behaviors and obtain meaningful conclusions, we set the temperature to 0.6 per vendor recommendations. Besides, our multi-attempt repair experiments in RQ2 show that each additional independent attempt yields a steady, consistent performance gain, confirming that individual results are not artifacts of random variance. Second, the quality of Support context may affect , Vol. 1, No. 1, Article . Publication date: November 2026.

20

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

repair performance. To ensure its quality, we employ a frontier LLM, DeepSeek-V3.2, to select the relevant unchanged methods and classes. To ensure a fair comparison, we supply identical Support context to all three LLMs. Third, flaky tests and our unified build environment (Java 17, Maven 3.9.9) may cause spurious test failures or passes. To mitigate this, we executed each instance three times during benchmark construction and retained only those with consistent outcomes. External validity. Our study subjects target JUnit tests in Java projects managed by Maven, which may not represent other ecosystem settings. But Maven and JUnit are among the most widely adopted Java toolchains, and the co-evolution patterns and investigation dimensions we study are not inherently language-specific. We also evaluate three LLMs from diverse providers to reduce model-specific bias. Moreover, the reasoning action taxonomy is derived from the traces of DeepSeek-V3.2, and the effectiveness of context types is evaluated on our benchmark; these findings may not fully generalize to other models or benchmarks. We partially mitigate this by applying the taxonomy-derived strategies to all three LLMs (RQ3) and transferring the context findings to the Codex agent (RQ5). Finally, our benchmark covers commits from May 2024 to April 2026, partially post-dating the training cutoffs of modern LLMs, which reduces but does not eliminate the risk of data contamination. Construct validity. Test pass rate (exec) does not guarantee semantic equivalence with developer intent, as a repaired test may pass by weakening assertions. This metric is nonetheless standard in prior test repair work [36, 93], and we mitigate this by verifying repair quality against developercommitted patches via both LLM-as-a-Judge and human inspection. The LLM judge and independent human auditor reach strong agreement that only very few cases weakened the test oracle on the sampled repairs (Section 3). Nevertheless, the judge only detects degenerate oracle weakening (e.g., assertion removal) rather than verifying full semantic correctness, so subtler deviations from developer intent may remain undetected. 6

Related Work

Empirical Studies on Source–Test Co-evolution. A long line of empirical work has characterized how production and test code co-evolve. Early repository-mining studies examined whether and when test code changes together with production code, using co-change analysis [92], association rules [38], and fine-grained change patterns [29, 39]. Pinto et al. [54] studied real-world test-suite evolution and found that repairing broken tests is only one driver of test changes, alongside the more frequent test additions, deletions, and refactorings. Subsequent studies characterized code-totest co-evolution patterns [62, 79] and leveraged them to identify obsolete tests [79] or recommend test updates [61]. Sun et al. [63] revisited how co-evolution samples are mined, revealing that the widely adopted co-change assumption introduces pervasive noise, and proposed CHOSEN to identify genuine production–test co-evolution samples; Huang et al. [22] similarly predict whether test code should co-change with production code. Guo et al. [18] studied the technical debt incurred by unbalanced production–test co-evolution in 15 open-source projects and proposed an approach to identify and quantify it. Closer to the use of LLMs, Lemner et al. [28] explored which industrial test maintenance activities LLMs can support. These studies focus on understanding whether, when, and why tests co-evolve with production code, or on detecting tests that need to be updated, but do not investigate how broken tests should be repaired. Meanwhile, existing evaluations of LLM-based test repair are conducted as part of proposing new techniques, each fixing a particular choice of repair context and pipeline. Very recent studies have begun to evaluate LLMs for test evolution, yet they either examine prompt design under single-focal-method changes [37] or focus on the overall performance of coding agents [60, 78]. Different from existing work, our work is the first to systematically and empirically investigate the design space of LLM-based test repair, (i.e., , Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

21

repair context, pipeline orchestration, explicit reasoning, scenario complexity, and generalization to coding agents), on realistic co-evolution scenarios involving multiple changed methods and classes. Test Repair. Early rule-driven techniques [9, 10, 35, 40–43, 86] fixed obsolete test cases using handcrafted rules or predefined patterns, limiting their applicability to diverse repair scenarios. Deep learning later enabled learned repair models such as CEPROT [21] and TaRGET [90]. More recently, LLM-based approaches have emerged: SYNTER [36] was the first to address test failures caused by changes in focal method signatures. REACCEPT [7] leverages dynamic validation feedback to improve repair performance. UTFix [56] prompts LLMs with static and dynamic code slices to repair tests and TestUpdater [93] uses LLMs to both repair and enhance test cases in response to focal method modifications. MuMuTestUp [73] is a very recent multi-agent technique that updates tests with mutation-guided assertion refinement. Among these techniques, TaRGET focuses on single-hunk test patch generation and the others repair obsolete tests caused by a single focal method change. MuMuTestUp relaxes this assumption by also considering changes along the call chain, but still centers its analysis on a single focal method. Different from existing work, our work addresses more complex and practical test repair scenarios. Test Generation. Automated test generation reduces the manual burden of constructing test cases. Conventional tools like EvoSuite [14] and Symstra [84] are based on evolutionary strategies, random testing, or symbolic execution, while learning-based approaches such as AthenaTest [74] (built on BART [30]), Atlas [81] and Tufano et al. [75] target test or assertion generation, though often at the cost of syntactic validity or executability [89]. Recent LLM-based approaches, including ChatTester [91] which is the first LLM-based generator, ChatUniTest [6] adopting a generate–verify–repair workflow to enhance test accuracy and coverage, HITS [80] that employs code slicing to decompose complex methods into simpler components. Unlike these methods, which generate new tests for individual methods, our work focuses on repairing and enhancing existing tests during software evolution. 7

Conclusion

This paper presented the first systematic empirical study of LLM-based test repair for source–test co-evolution, grounded in a new benchmark of 500 dynamically validated, real-world instances from 15 highly-starred open-source Java projects. Across five research questions, we investigated key design dimensions including repair context, pipeline orchestration, LLM reasoning actions, repair scenario complexity, and agent-based generalization. Our study reveals that test failure symptoms, source code changes, and supportive code context complementarily contribute to effective test repair, and that iterative self-repair rounds further benefit LLMs. We developed a reasoning action taxonomy from LLM reasoning traces on successful repairs. It helps incentivize cost-effective LLMbased test repairs. We also demonstrate that our empirically derived insights translate into tangible improvements for coding agents. Our findings provide empirical evidence and offer actionable guidelines for effective LLM-based test repair. 8

Data Availability

The replication package is available at https://anonymous.4open.science/r/Source-Test-Co-evolutionStudy-F4F4. References [1] Alibaba. 2026. Spring AI Alibaba - A production-ready framework for building Agentic, Workflow, and Multi-agent applications. https://github.com/alibaba/spring-ai-alibaba [2] Alibaba Qwen Team. 2025. Qwen3-Coder: Code-Specialized Large Language Model. https://github.com/QwenLM/ Qwen3-Coder , Vol. 1, No. 1, Article . Publication date: November 2026.

22

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

[3] Apache. 2025. Apache Druid Commit 98c63de2cb. https://github.com/apache/druid/commit/98c63de2cb [4] Apache. 2025. Apache IOTDB Commit 8805b3e3d6. https://github.com/apache/iotdb/commit/8805b3e3d6 [5] Xiao Chen, Hengcheng Zhu, Jialun Cao, Ming Wen, and Shing-Chi Cheung. 2025. SemBIC: Semantic-Aware Identification of Bug-Inducing Commits. Proc. ACM Softw. Eng. 2, FSE (2025), 1363–1385. doi:10.1145/3715781 [6] Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, Porto de Galinhas, Brazil, July 15-19, 2024. ACM, 572–576. doi:10.1145/3663529.3663801 [7] Jianlei Chi, Xiaotian Wang, Yuhan Huang, Lechen Yu, Di Cui, Jianguo Sun, and Jun Sun. 2025. REACCEPT: Automated Co-evolution of Production and Test Code Based on Dynamic Validation and Large Language Models. Proc. ACM Softw. Eng. 2, ISSTA (2025), 1234–1256. doi:10.1145/3728930 [8] William G. Cochran. 1977. Sampling Techniques (3rd ed.). Wiley. [9] Brett Daniel, Tihomir Gvero, and Darko Marinov. 2010. On test repair using symbolic execution. In Proceedings of the Nineteenth International Symposium on Software Testing and Analysis, ISSTA 2010, Trento, Italy, July 12-16, 2010. ACM, 207–218. doi:10.1145/1831708.1831734 [10] Brett Daniel, Vilas Jagannath, Danny Dig, and Darko Marinov. 2009. ReAssert: Suggesting Repairs for Broken Unit Tests. In ASE 2009, 24th IEEE/ACM International Conference on Automated Software Engineering, Auckland, New Zealand, November 16-20, 2009. IEEE Computer Society, 433–444. doi:10.1109/ASE.2009.17 [11] DeepSeek AI. 2025. DeepSeek-V3: A Strong Mixture-of-Experts Language Model. https://github.com/deepseekai/DeepSeek-V3 [12] EclEmma. 2024. JaCoCo - Java Code Coverage Library. https://www.jacoco.org/jacoco/ [13] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 (Findings of ACL, Vol. EMNLP 2020), Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1536–1547. doi:10.18653/V1/2020.FINDINGS-EMNLP.139 [14] Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In SIGSOFT/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hungary, September 5-9, 2011. ACM, 416–419. doi:10.1145/2025113.2025179 [15] Shuzheng Gao, Chaozheng Wang, Cuiyun Gao, and Michael R Lyu. 2025. SEER: Enhancing Chain-of-Thought Code Generation through Self-Exploring Deep Reasoning. arXiv preprint arXiv:2510.17130 (2025). [16] Git Contributors. 2024. Git. https://git-scm.com [17] GitHub, Inc. 2024. GitHub. https://github.com [18] Yimeng Guo, Zhifei Chen, Lu Xiao, Lin Chen, Yanhui Li, and Yuming Zhou. 2025. Understanding and Identifying Technical Debt in the Co-Evolution of Production and Test Code. IEEE Trans. Software Eng. 51, 5 (2025), 1415–1436. doi:10.1109/TSE.2025.3553112 [19] Kevin Halim, Sin G. Teo, Ruitao Feng, Zhenpeng Chen, Yang Gu, Chong Wang, and Yang Liu. 2025. A Study on Thinking Patterns of Large Reasoning Models in Code Generation. arXiv:2509.13758 [cs.SE] https://arxiv.org/abs/2509.13758 [20] Hazelcast. 2026. Hazelcast - Open Source Real-time Data Platform. https://github.com/hazelcast/hazelcast [21] Xing Hu, Zhuang Liu, Xin Xia, Zhongxin Liu, Tongtong Xu, and Xiaohu Yang. 2023. Identify and Update Test Cases When Production Code Changes: A Transformer-Based Approach. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, 1111–1122. doi:10.1109/ASE56229. 2023.00165 [22] Yuan Huang, Zhicao Tang, Xiangping Chen, and Xiaocong Zhou. 2024. Towards automatically identifying the co-change of production and test code. Softw. Test. Verification Reliab. 34, 3 (2024). doi:10.1002/STVR.1870 [23] Victor Hurdugaci and Andy Zaidman. 2012. Aiding Software Developers to Maintain Developer Tests. In 16th European Conference on Software Maintenance and Reengineering, CSMR 2012, Szeged, Hungary, March 27-30, 2012. IEEE Computer Society, 11–20. doi:10.1109/CSMR.2012.12 [24] Jenkins Community. 2026. Jenkins - Open Source Automation Server. https://github.com/jenkinsci/jenkins [25] Jussi Kasurinen, Ossi Taipale, and Kari Smolander. 2010. Software Test Automation in Practice: Empirical Observations. Adv. Softw. Eng. 2010 (2010), 620836:1–620836:18. doi:10.1155/2010/620836 [26] Shahedul Huq Khandkar. [n. d.]. Open coding. ([n. d.]). [27] Adriaan Labuschagne, Laura Inozemtseva, and Reid Holmes. 2017. Measuring the cost of regression testing in practice: a study of Java projects using continuous integration. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017. ACM, 821–830. doi:10.1145/3106237. 3106288

, Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

23

[28] Ludvig Lemner, Linnea Wahlgren, Gregory Gay, Nasser Mohammadiha, Jingxiong Liu, and Joakim Wennerberg. 2024. Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes. CoRR abs/2409.06416 (2024). arXiv:2409.06416 doi:10.48550/ARXIV.2409.06416 [29] Stanislav Levin and Amiram Yehudai. 2017. The Co-evolution of Test Maintenance and Code Maintenance through the Lens of Fine-Grained Semantic Changes. In 2017 IEEE International Conference on Software Maintenance and Evolution, ICSME 2017, Shanghai, China, September 17-22, 2017. IEEE Computer Society, 35–46. doi:10.1109/ICSME.2017.9 [30] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. Association for Computational Linguistics, 7871–7880. doi:10.18653/V1/ 2020.ACL-MAIN.703 [31] Dongze Li, Songqiang Chen, Jialun Cao, and Shing-Chi Cheung. 2025. What Builds Effective In-Context Examples for Code Generation? CoRR abs/2508.06414 (2025). arXiv:2508.06414 doi:10.48550/ARXIV.2508.06414 [32] Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2025. Structured chain-of-thought prompting for code generation. ACM Transactions on Software Engineering and Methodology 34, 2 (2025), 1–23. [33] Jia Li, Chongyang Tao, Jia Li, Ge Li, Zhi Jin, Huangzhao Zhang, Zheng Fang, and Fang Liu. 2025. Large Language ModelAware In-Context Learning for Code Generation. ACM Trans. Softw. Eng. Methodol. (Feb. 2025). doi:10.1145/3715908 [34] Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2024. AceCoder: An Effective Prompting Technique Specialized in Code Generation. ACM Trans. Softw. Eng. Methodol. 33, 8, Article 204 (Nov. 2024), 26 pages. doi:10.1145/3675395 [35] Xiangyu Li, Marcelo d’Amorim, and Alessandro Orso. 2019. Intent-Preserving Test Repair. In 12th IEEE Conference on Software Testing, Validation and Verification, ICST 2019, Xi’an, China, April 22-27, 2019. IEEE, 217–227. doi:10.1109/ICST. 2019.00030 [36] Jun Liu, Jiwei Yan, Yuanyuan Xie, Jun Yan, and Jian Zhang. 2024. Fix the Tests: Augmenting LLMs to Repair Test Cases with Static Collector and Neural Reranker. In 35th IEEE International Symposium on Software Reliability Engineering, ISSRE 2024, Tsukuba, Japan, October 28-31, 2024. IEEE, 367–378. doi:10.1109/ISSRE62328.2024.00043 [37] Weichang Liu, Junwei Zhang, Yuqing Niu, and Bo Zhou. 2026. On the Evaluation of Large Language Models in Unit Test Evolution (Experience Paper). Proc. ACM Softw. Eng. 3, ISSTA, Article ISSTA191 (Oct. 2026), 24 pages. doi:10.1145/3832282 [38] Zeeger Lubsen, Andy Zaidman, and Martin Pinzger. 2009. Using association rules to study the co-evolution of production & test code. In Proceedings of the 6th International Working Conference on Mining Software Repositories, MSR 2009 (Co-located with ICSE), Vancouver, BC, Canada, May 16-17, 2009, Proceedings. IEEE Computer Society, 151–154. doi:10.1109/MSR.2009.5069493 [39] Cosmin Marsavina, Daniele Romano, and Andy Zaidman. 2014. Studying Fine-Grained Co-evolution Patterns of Production and Test Code. In 14th IEEE International Working Conference on Source Code Analysis and Manipulation, SCAM 2014, Victoria, BC, Canada, September 28-29, 2014. IEEE Computer Society, 195–204. doi:10.1109/SCAM.2014.28 [40] Mehdi MirzaAghaei. 2011. Automatic test suite evolution. In SIGSOFT/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hungary, September 5-9, 2011. ACM, 396–399. doi:10.1145/2025113.2025172 [41] Mehdi MirzaAghaei, Fabrizio Pastore, and Mauro Pezzè. 2010. Automatically repairing test cases for evolving method declarations. In 26th IEEE International Conference on Software Maintenance (ICSM 2010), September 12-18, 2010, Timisoara, Romania. IEEE Computer Society, 1–5. doi:10.1109/ICSM.2010.5609549 [42] Mehdi MirzaAghaei, Fabrizio Pastore, and Mauro Pezzè. 2012. Supporting Test Suite Evolution through Test Case Adaptation. In Fifth IEEE International Conference on Software Testing, Verification and Validation, ICST 2012, Montreal, QC, Canada, April 17-21, 2012. IEEE Computer Society, 231–240. doi:10.1109/ICST.2012.103 [43] Mehdi MirzaAghaei, Fabrizio Pastore, and Mauro Pezzè. 2014. Automatic test case evolution. Softw. Test. Verification Reliab. 24, 5 (2014), 386–411. doi:10.1002/STVR.1527 [44] Leon Moonen, Arie van Deursen, Andy Zaidman, and Magiel Bruntink. 2008. On the Interplay Between Software Testing and Evolution and its Effect on Program Comprehension. In Software Evolution. Springer, 173–202. doi:10.1007/978-3540-76440-3_8 [45] Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023. Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2450–2462. doi:10.1109/ICSE48619.2023.00205 [46] Michael Olan. 2003. Unit testing: test early, test often. J. Comput. Sci. Coll. 19, 2 (2003), 319–328. doi:10.5555/948785. 948830 [47] OpenAI. 2025. Codex. https://chatgpt.com/codex/ [48] OpenAI. 2025. GPT-5. https://openai.com

, Vol. 1, No. 1, Article . Publication date: November 2026.

24

Xiao Chen, Songqiang Chen, Jialun Cao, Yun Peng, Chun Yong Chong, and Shing-Chi Cheung

[49] Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. 2025. An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Trans. Softw. Eng. Methodol. 34, 2 (2025), 42:1–42:28. doi:10.1145/3697010 [50] OWASP Foundation. 2026. OWASP Dependency-Check - Software Composition Analysis. https://github.com/ dependency-check/DependencyCheck [51] Zhiyuan Pan, Xing Hu, Xin Xia, and Xiaohu Yang. [n. d.]. CATCODER: Repository-Level Code Generation with Relevant Code and Type Context. ACM Transactions on Software Engineering and Methodology ([n. d.]). [52] Pinpoint. 2026. Pinpoint - Application Performance Management Tool. https://github.com/pinpoint-apm/pinpoint [53] pinpoint-apm. 2025. pinpoint-apm pinpoint Commit cda0888352. https://github.com/pinpoint-apm/pinpoint/commit/ cda0888352 [54] Leandro Sales Pinto, Saurabh Sinha, and Alessandro Orso. 2012. Understanding myths and realities of test-suite evolution. In 20th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-20), SIGSOFT/FSE’12, Cary, NC, USA - November 11 - 16, 2012, Will Tracz, Martin P. Robillard, and Tevfik Bultan (Eds.). ACM, 33. doi:10.1145/ 2393596.2393634 [55] Michael Pradel. 2025. Testora: Using Natural Language Intent to Detect Behavioral Regressions. arXiv preprint arXiv:2503.18597 (2025). [56] Shanto Rahman, Sachit Kuhar, Berk Çirisci, Pranav Garg, Shiqi Wang, Xiaofei Ma, Anoop Deoras, and Baishakhi Ray. 2025. UTFix: Change Aware Unit Test Repairing using LLM. Proc. ACM Program. Lang. 9, OOPSLA1 (2025), 143–168. doi:10.1145/3720419 [57] Red Hat. 2026. Quarkus - Supersonic Subatomic Java. https://github.com/quarkusio/quarkus [58] Stephen E. Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr. 3, 4 (2009), 333–389. doi:10.1561/1500000019 [59] Per Runeson. 2006. A Survey of Unit Testing Practices. IEEE Softw. 23, 4 (2006), 22–29. doi:10.1109/MS.2006.91 [60] Ye Shang, Quanjun Zhang, Haichuan Hu, Chunrong Fang, Liang Xiao, and Zhenyu Chen. 2026. Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution. CoRR abs/2605.06125 (2026). arXiv:2605.06125 doi:10.48550/arXiv.2605.06125 [61] Samiha Shimmi and Mona Rahimi. 2022. Leveraging Code-Test Co-evolution Patterns for Automated Test Case Recommendation. In IEEE/ACM International Conference on Automation of Software Test, AST@ICSE 2022, Pittsburgh, PA, USA, May 21-22, 2022. ACM/IEEE, 65–76. doi:10.1145/3524481.3527222 [62] Samiha Shimmi and Mona Rahimi. 2022. Patterns of Code-to-Test Co-evolution for Automated Test Suite Maintenance. In 15th IEEE Conference on Software Testing, Verification and Validation, ICST 2022, Valencia, Spain, April 4-14, 2022. IEEE, 116–127. doi:10.1109/ICST53961.2022.00023 [63] Weifeng Sun, Meng Yan, Zhongxin Liu, Xin Xia, Yan Lei, and David Lo. 2023. Revisiting the Identification of the Coevolution of Production and Test Code. ACM Trans. Softw. Eng. Methodol. 32, 6 (2023), 152:1–152:37. doi:10.1145/3607183 [64] The Apache Software Foundation. 2024. Maven. https://maven.apache.org/ [65] The Apache Software Foundation. 2026. Apache Camel - Open Source Integration Framework. https://github.com/ apache/camel [66] The Apache Software Foundation. 2026. Apache Druid - High Performance Real-time Analytics Database. https: //github.com/apache/druid [67] The Apache Software Foundation. 2026. Apache Dubbo - High-performance RPC Framework. https://github.com/ apache/dubbo [68] The Apache Software Foundation. 2026. Apache HertzBeat - Real-time Monitoring System. https://github.com/apache/ hertzbeat [69] The Apache Software Foundation. 2026. Apache Hudi - Open Data Lakehouse Platform. https://github.com/apache/hudi [70] The Apache Software Foundation. 2026. Apache IoTDB - Internet of Things Database. https://github.com/apache/iotdb [71] The Apache Software Foundation. 2026. Apache ShardingSphere - Enterprise Distributed Database Ecosystem. https://github.com/apache/shardingsphere [72] The Netty Project. 2026. Netty - Asynchronous Event-driven Network Application Framework. https://github.com/ netty/netty [73] Dawei Tian, Jiakun Liu, Yun Peng, Yichen Zhang, Jianlei Chi, Jun Sun, and Xiaohong Su. 2026. MuMuTestUp: Mutation-Based Multi-agent Test Case Update. Proc. ACM Softw. Eng. 3, ISSTA (2026), 2883–2904. doi:10.1145/3832217 [74] Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2021. Unit Test Case Generation with Transformers and Focal Context. arXiv:2009.05617 [cs.SE] https://arxiv.org/abs/2009.05617 [75] Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, and Neel Sundaresan. 2022. Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers. In IEEE/ACM International Conference on Automation of Software Test, AST@ICSE 2022, Pittsburgh, PA, USA, May 21-22, 2022. ACM/IEEE, 54–64. doi:10.1145/3524481.3527220 [76] Arash Vahabzadeh, Amin Milani Fard, and Ali Mesbah. 2015. An empirical study of bugs in test code. In 2015 IEEE International Conference on Software Maintenance and Evolution, ICSME 2015, Bremen, Germany, September 29 - October

, Vol. 1, No. 1, Article . Publication date: November 2026.

Finding the Pivots to Repair Broken Source–Test Evolution Using LLMs

25

1, 2015. IEEE Computer Society, 101–110. doi:10.1109/ICSM.2015.7332456 [77] VMware Tanzu. 2026. Spring AI - Application Framework for AI Engineering. https://github.com/spring-projects/ spring-ai [78] Jiale Amber Wang, Kaiyuan Wang, and Pengyu Nie. 2026. TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution. CoRR abs/2607.02469 (2026). arXiv:2607.02469 doi:10.48550/arXiv.2607.02469 [79] Sinan Wang, Ming Wen, Yepang Liu, Ying Wang, and Rongxin Wu. 2021. Understanding and Facilitating the CoEvolution of Production and Test Code. In 28th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2021, Honolulu, HI, USA, March 9-12, 2021. IEEE, 272–283. doi:10.1109/SANER50967.2021.00033 [80] Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024. ACM, 1258–1268. doi:10.1145/3691620.3695501 [81] Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On learning meaningful assert statements for unit test cases. In ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020. ACM, 1398–1409. doi:10.1145/3377811.3380429 [82] Martin Weyssow, Aton Kamanda, Xin Zhou, and Houari Sahraoui. 2026. Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences. ACM Transactions on Software Engineering and Methodology 35, 3 (2026), 1–36. [83] David Gray Widder, Michael Hilton, Christian Kästner, and Bogdan Vasilescu. 2019. A conceptual replication of continuous integration pain points in the context of Travis CI. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, August 26-30, 2019. ACM, 647–658. doi:10.1145/3338906.3338922 [84] Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. 2005. Symstra: A Framework for Generating ObjectOriented Unit Tests Using Symbolic Execution. In Tools and Algorithms for the Construction and Analysis of Systems, 11th International Conference, TACAS 2005, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2005, Edinburgh, UK, April 4-8, 2005, Proceedings (Lecture Notes in Computer Science, Vol. 3440). Springer, 365–381. doi:10.1007/978-3-540-31980-1_24 [85] Tangzhi Xu, Jianhan Liu, Yuan Yao, Cong Li, Feng Xu, and Xiaoxing Ma. 2025. Comprehend, Imitate, and then Update: Unleashing the Power of LLMs in Test Suite Evolution. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 469–481. doi:10.1109/ASE63991.2025.00046 [86] Yong Xu, Bo Huang, Guoqing Wu, and Mengting Yuan. 2014. Using Genetic Algorithms to Repair JUnit Test Cases. In 21st Asia-Pacific Software Engineering Conference, APSEC 2014, Jeju, South Korea, December 1-4, 2014. Volume 1: Research Papers. IEEE Computer Society, 287–294. doi:10.1109/APSEC.2014.51 [87] Chengran Yang, Zhensu Sun, Hong Jin Kang, Jieke Shi, and David Lo. 2025. Think Like Human Developers: Harnessing Community Knowledge for Structured Code Reasoning. arXiv preprint arXiv:2503.14838 (2025). [88] Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Terry Yue Zhuo, and Taolue Chen. 2024. Chain-of-thought in neural code generation: From and for lightweight language models. IEEE Transactions on Software Engineering 50, 9 (2024), 2437–2457. [89] Yanming Yang, Xin Xia, David Lo, and John C. Grundy. 2022. A Survey on Deep Learning for Software Engineering. ACM Comput. Surv. 54, 10s (2022), 206:1–206:73. doi:10.1145/3505243 [90] Ahmadreza Saboor Yaraghi, Darren Holden, Nafiseh Kahani, and Lionel C. Briand. 2025. Automated Test Case Repair Using Language Models. IEEE Trans. Software Eng. 51, 4 (2025), 1104–1133. doi:10.1109/TSE.2025.3541166 [91] Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation. Proc. ACM Softw. Eng. 1, FSE (2024), 1703–1726. doi:10.1145/3660783 [92] Andy Zaidman, Bart Van Rompaey, Serge Demeyer, and Arie van Deursen. 2008. Mining Software Repositories to Study Co-Evolution of Production & Test Code. In First International Conference on Software Testing, Verification, and Validation, ICST 2008, Lillehammer, Norway, April 9-11, 2008. IEEE Computer Society, 220–229. doi:10.1109/ICST.2008.47 [93] Yuanhe Zhang, Zhiquan Yang, Shengyi Pan, and Zhongxin Liu. 2025. Unit Test Update through LLM-Driven Context Collection and Error-Type-Aware Refinement. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Republic of Korea, November 16-20, 2025. IEEE, 2504–2515. doi:10.1109/ASE63991.2025.00206 [94] Xin Zhou, Kisub Kim, Ting Zhang, Martin Weyssow, Luís F Gomes, Guang Yang, Kui Liu, Xin Xia, and David Lo. 2025. SE-Jury: An LLM-as-Ensemble-Judge Metric for Narrowing the Gap with Human Evaluation in SE. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2606–2618. [95] Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software Unit Test Coverage and Adequacy. ACM Comput. Surv. 29, 4 (1997), 366–427. doi:10.1145/267580.267590

, Vol. 1, No. 1, Article . Publication date: November 2026.

Related documents

• Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report#1036232
• Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report#1006923
• If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs#997036
• Large Language Model Guided Self-Debugging Code Generation#1535989
• Pinning Decisions Before Failure: Executable Records of Underspecified Choices in AI-Assisted Code Generation#1187915