Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution
Ye Shang 1 Quanjun Zhang 2 Haichuan Hu 2 Chunrong Fang 1 Liang Xiao 2 Zhenyu Chen 1
arXiv:2605.06125v1 [cs.SE] 7 May 2026
Abstract
Analysis of execution trajectories reveals a reactive “execute-fail-fix” loop that succeeds for breaking tests but structurally cannot address stale or missing tests. TEBench is publicly available at https://github.com/iSEngLab/ TEBench, with a continuously updated leaderboard at https://tebench-leadership. vercel.app.
As production code evolves, the associated test suite must co-evolve to remain effective. Existing benchmarks for test evolution operate at methodlevel granularity with pre-paired inputs, bypassing the critical task of locating affected tests from the full project and excluding the need for new tests entirely. We present TEBench, the first project-level benchmark for test evolution. Given a project repository and a code-changing commit, TEBench requires systems to autonomously identify tests requiring modification, determine where new tests are needed, and produce the corresponding test patch. We construct TEBench through a four-stage filtering pipeline over projects from the Defects4J ecosystem, curating 314 task instances from 10 projects with developer-written ground truth. Each instance is annotated with one or more of three evolution types: Test-Breaking (tests that fail), Test-Stale (tests that pass but no longer meaningfully validate the updated behavior), and Test-Missing (new tests needed for introduced behavior). We evaluate seven configurations spanning three industrial agent frameworks (Claude Code, Codex CLI, and OpenCode) and six base models, alongside a heuristic baseline. All seven configurations converge on an identification F1 of 45.7% to 49.4%, revealing a shared performance ceiling that holds across both agent frameworks and base models. Test-Stale is the most challenging type, with an average F1 of approximately 36%, since configurations rely on execution failure signals and lack proactive semantic reasoning. On the update task, configurations produce highly executable test modifications whose surface form nonetheless diverges substantially from developer-written ground truth.
Software systems evolve continuously, with production code undergoing frequent modifications to fix bugs, add features, and refactor implementations. As production code changes, the associated test suite should co-evolve to remain effective. This challenge, known as test evolution, is pervasive in practice, yet developers often struggle to systematically identify all tests affected by a given change across a project. Some tests begin to fail due to changed interfaces or updated output formats; others continue to pass but silently lose their ability to validate the behavior they were designed to check; still others are simply absent, as newly introduced functionality lacks any corresponding test. Left unaddressed, these issues lead to gradual test suite degradation that silently undermines software quality. A growing body of research has addressed the problem of test evolution (Hu et al., 2023; Chi et al., 2025; Sun et al., 2023; Zhang et al., 2025). Due to the limited context window and reasoning capability of earlier techniques such as fine-tuned CodeT5 models (Hu et al., 2023), existing approaches adopt a method-level input formulation ⟨m, m′ , t⟩ that pairs the original and updated production methods with an associated test method. This design presupposes that an obsolete test t has already been selected, structurally bypassing the identification step and restricting the problem scope to two categories of obsolete tests: tests that fail after the code change, and tests that still pass but whose coverage of the changed code has degraded. The possibility of missing tests, where new behavior lacks any corresponding test, is excluded entirely. With the rapid advancement of large language models and autonomous coding agents, benchmarks should evolve to better reflect real-world development scenarios. This paradigm shift has already occurred in adjacent fields: SWE-bench (Jimenez
1 State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China 2 Nanjing University of Science and Technology, Nanjing, China. Correspondence to: Zhenyu Chen <[email protected]>.
Preprint. May 8, 2026.
1
Benchmarking Coding Agents on Project-Level Test Evolution
et al., 2024) and SWT-bench (Mündler et al., 2024) elevated the task from patching isolated functions and generating method-level tests (as in Defects4J (Just et al., 2014)) to resolving GitHub issues and reproducing bugs across entire repositories, catalyzing rapid progress in coding agent development. To bridge this gap in test evolution, we lift the input from method-level to project-level and refine the problem taxonomy accordingly. We subdivide obsolete tests into Test-Breaking, where the test fails after the change, and Test-Stale, where the test still passes but no longer meaningfully validates the updated behavior. We further introduce Test-Missing to capture the need for new tests that cover behavior introduced by the change. Together, these three types constitute a more complete characterization of test evolution that aligns with real-world practice.
The main contributions of this paper are as follows: • New Dimension. We propose the project-level test evolution task, which requires systems to autonomously identify tests requiring modification and determine where new tests are needed from the full project context, extending beyond the method-level paired formulations of prior work. • Benchmark. We construct TEBench, to the best of our knowledge the first benchmark for project-level test evolution, comprising 314 high-quality task instances with developer-written ground truth from 10 real-world Java projects, covering three evolution types (Test-Breaking, Test-Stale, and Test-Missing), accompanied by a two-dimensional evaluation framework for identification and update quality.
In this paper, we present TEBench (Test Evolution Benchmark), to the best of our knowledge, the first projectlevel benchmark for test evolution. TEBench defines a new task formulation: given a project repository and a commit that modifies production code, the system must autonomously identify tests requiring modification and determine where new tests are needed across the entire project. We construct TEBench through a four-stage filtering pipeline over projects from the Defects4J (Just et al., 2014) ecosystem, ultimately curating 314 high-quality task instances with developer-written ground truth from 10 realworld open-source Java projects. Each task instance is classified into one or more of the three evolution types defined above, and evaluated through a two-dimensional metric framework that separately measures identification accuracy and update quality.
• Evaluation Study. We conduct the first systematic evaluation of seven LLM-based configurations spanning three industrial agent frameworks and six base models, revealing a shared performance ceiling, typespecific difficulty patterns, and limitations in proactive semantic reasoning.
1. Motivation and Task Definition 1.1. Motivating Example We illustrate the complexity of real-world test evolution through a concrete example. Figure 1 shows a commit from JSOUP, a widely used Java HTML parsing library with over 11K stars on GitHub. This commit is also included as a task instance in TEBench.
Using TEBench, we conduct the first systematic evaluation of LLM-based systems on the test evolution task. We evaluate seven configurations spanning three industrial agent frameworks (Claude Code, Codex CLI, and OpenCode) and six base models, alongside a heuristic baseline. All seven configurations converge on an identification F1 of 45.7% to 49.4%, with less than four percentage points separating them, and the convergence holds across both agent frameworks and base models, indicating that the bottleneck lies in the inherent task difficulty rather than in any specific configuration. Test-Stale emerges as the most challenging evolution type, with an average F1 of approximately 36%, since configurations rely almost entirely on execution failure signals and lack the ability to proactively reason about semantic test adequacy. On the update task, configurations produce highly executable test modifications whose surface form nonetheless diverges substantially from developer-written ground truth, indicating that executability is far from a sufficient proxy for update quality. Even exhaustive structural dependency analysis achieves only 66% Recall, leaving roughly one-third of affected tests undetectable through direct dependency tracing alone.
The commit message states: “Wrap first inline elements in block, ignoring preceding whitespace.” This change fixes the pretty-printing logic in isInlineable() method of Element.java file. In the original implementation, the decision of whether an inline element should be wrapped and indented inside a block element was governed by a simple boolean condition. The revised implementation introduces an additional check for a previously overlooked edge case: when the first inline element in a block is preceded by a blank text node, it should still be treated as the block’s first child and therefore be indented, rather than rendered inline. Although the code change is localized to a single production method, its impact on the test suite is both broad and heterogeneous, spanning multiple test files in different packages. Impact 1: Test failure. Some existing tests fail immediately after the change. For example, a test such as nestedAnchorElements01() now observes different pretty-printing output: anchor elements that were previously rendered inline are instead wrapped onto indented new lines 2
Benchmarking Coding Agents on Project-Level Test Evolution Test Method Diff src/test/java/org/jsoup/parser/HtmlTreeBuilderStateTest.java
Example: jsoup a349582236a7e69ce341f3b4ee913cb8cd19b955 committed on Mar 9, 2023 Commit Message: Wrap first inline elements in block, ignoring preceding whitespace Breaking Production Method Diff src/main/java/org/jsoup/nodes/Element.java private boolean isInlineable(Document.OutputSettings out) { return tag().isInline() && (parent() == null || parent().isBlock()) && previousSibling() != null + if (!tag.isInline()) + return false; + + final Node prev = previousSibling(); + boolean isFirst = siblingIndex == 0; + if (siblingIndex == 1 && prev instanceof TextNode + && (((TextNode) prev).isBlank())) + isFirst = true; + return (parent() == null || parent().isBlock()) + && !isFirst && !out.outline(); }
@Test public void nestedAnchorElements01() { ... // Original test fails on updated code assertEquals("<html>\n" + " <head></head>\n" + " <body><a href=\"#1\"> </a>\n" + + " <body>\n" + + " <a href=\"#1\"> </a>\n" + " <div>\n" + ...
src/test/java/org/jsoup/select/ElementsTest.java
Stale
+ + +
@Test public void unwrap() { // Original test passes on updated code ... assertEquals("<div>One <a href=\"/\">Two</a></div>", TextUtil.stripNewlines(doc.body().html())); assertEquals("<div>\n" + " One <a href=\"/\">Two</a>\n" + "</div>", doc.body().html()); }
src/test/java/org/jsoup/nodes/ElementTest.java
Missing
......
+ @Test void inlineInBlockShouldIndent() { + String html = "<div>One <span>Hello</span><span>!</span></div>" + + "<div>\n<span>There</span></div>" // Not in original test suite + + "<div> <span>Now</span></div>"; + Document doc = Jsoup.parse(html); + assertEquals( + “<div>\n” + " One <span>Hello</span><span>!</span>\n" + + “</div>\n” + “<div>\n” + " <span>There</span>\n" + + “</div>\n” + “<div>\n” + " <span>Now</span>\n" + + “</div>”, doc.body().html()); + }
Figure 1. Motivating example from JSOUP: a change to isInlineable() in Element.java impacts multiple test files across different packages, exhibiting three types of test evolution. Table 1. Comparison of test evolution benchmarks.
when preceded by blank text nodes. The same failure pattern also appears in the other two tests across different packages.
Benchmark
Impact 2: Silent quality degradation. Other tests continue to pass, yet become semantically outdated. In particular, the test unwrap() still succeeds because it applies TextUtil.stripNewlines() before comparing outputs, which masks the formatting change introduced by the updated logic. As a result, the test remains executable but no longer validates the intended pretty-printing behavior. The developer subsequently revised this test to compare directly against the formatted output. Impact 3: Uncovered new behavior. The developer also added a new test, inlineInBlockShouldIndent(), to cover scenarios that were previously untested. This test verifies that inline elements inside a block are consistently indented across several representative inputs, filling a gap in the original test suite.
Granularity
Input
Identification
Update
Output
SITAR (Wang et al., 2021) CHOSEN (Sun et al., 2023) CEPROT (Hu et al., 2023) REACCEPT (Chi et al., 2025) TaRBench (Yaraghi et al., 2025) Synter (Liu et al., 2024) Tool-Bench (Rahman et al., 2025) Updates4J (Zhang et al., 2025)
Method Method Method Method Method Method Method Method
⟨m, m′ , t⟩ ⟨m, m′ , t, t′ ⟩ ⟨m, m′ , t⟩ ⟨m, m′ , t⟩ ⟨P, m, m′ , t⟩ ⟨P, m, m′ , t⟩ ⟨P, m, m′ , t⟩ ⟨P, m, m′ , t⟩
Paired Paired Paired Paired Assumed Assumed Assumed Assumed
No No B+S B+S B B B+S B+S
Binary label Binary label Updated t′ Updated t′ Repaired t′ Repaired t′ Repaired t′ Updated t′
SWE-bench (Jimenez et al., 2024) SWT-bench (Mündler et al., 2024)
Project Project
⟨P, Issue⟩ ⟨P, Issue⟩
N/A N/A
N/A N/A
Code patch Repro. test
TEBench
Project
⟨P, ∆⟩
Autonomous
B+S+M
Test patch
1.2. Limitations of Existing Benchmarks Table 1 compares TEBench with existing test evolution benchmarks. In the identification dimension, existing benchmarks either adopt a Paired setting, where code–test associations are pre-paired for classification, or an Assumed setting, where the affected test is directly given as input. By contrast, TEBench uses an Autonomous setting, requiring the system to identify affected tests independently at the project level. In the update dimension, prior benchmarks mainly cover Breaking and, in some cases, Stale tests, while TEBench further includes Missing tests. We identify three limitations that prevent current benchmarks from capturing the complexity illustrated above.
This example highlights a key characteristic of real-world test evolution: even a small and localized code change can induce diverse and non-obvious impacts on tests distributed across the project. In practice, a single commit may modify multiple production methods across several files, each affecting a different subset of tests, thereby creating a complex many-to-many relationship between code changes and test impacts. This naturally raises the question: can existing benchmarks capture such complexity?
Limitation 1: Method-level granularity. Existing test evolution benchmarks are formulated at the method level, typically taking as input a tuple such as ⟨m, m′ , t⟩, where m and m′ denote the original and updated production method, and t denotes the associated test method. This formulation 3
Benchmarking Coding Agents on Project-Level Test Evolution
reduces the task to a bounded and pre-identified one-to-one setting (one production method change to one test method). While such formulations are suitable for studying localized test co-evolution, they abstract away the project-level reasoning required in realistic development settings. In the JSOUP example, a change to a single production method affects tests across three different packages, requiring cross-file and cross-module reasoning that method-level benchmarks cannot assess.
code–test association is given as part of the input, our task requires the system to navigate the codebase autonomously, locate affected tests, and generate appropriate updates or additions. As illustrated by the motivating example, we categorize test evolution instances into three types according to how existing tests behave after the code change and how the developer responds. Test-Breaking. An existing test t ∈ T fails to compile or execute after ∆ is applied, and the developer modifies t in the ground truth (GT) to restore correctness. In the motivating example, this corresponds to Impact 1: several tests fail because their expected output strings no longer match the updated formatting behavior.
Limitation 2: Identification is bypassed. Because the code–test association is already provided, existing benchmarks structurally bypass the test identification task. As shown in Table 1, prior work either provides pre-paired associations for classification (Paired) or directly gives the affected test for repair (Assumed). In practice, however, identifying which tests among hundreds of files require attention is the first, and often one of the hardest, steps after a code change.
Test-Stale. An existing test t ∈ T still passes after ∆ is applied, but the developer nonetheless updates t in the GT so that it better reflects the revised semantics of the code. In the motivating example, this corresponds to Impact 2: a test remains executable but no longer meaningfully validates the formatting behavior because its comparison logic masks the change.
Limitation 3: Incomplete coverage of evolution types. Because the input always includes an existing test t, the output is inherently restricted to a modified version t′ , thereby excluding the possibility of generating entirely new tests. Consequently, existing benchmarks cover at most tests that fail after a code change and tests that still pass but nevertheless require updates. They do not account for cases in which new behavior is introduced or exposed, but no corresponding test yet exists. In practice, however, adding new tests in response to a code change is not an independent test generation task; it is contextually grounded in the commit itself, motivated by the need to cover behavior specifically introduced by the commit change. In this sense, such tests addition constitutes a natural form of test evolution rather than general-purpose test generation. By restricting the output to modifications of a given t, prior benchmarks artificially narrow the scope of test evolution, excluding a response pattern that developers regularly employ, as illustrated by Impact 3 in the motivating example.
Test-Missing. The developer adds a new test method tnew ∈ / T in the GT to cover behavior introduced or exposed by ∆. In the motivating example, this corresponds to Impact 3: a new test is added to verify consistent indentation behavior across several representative scenarios. The first two types, Test-Breaking and Test-Stale, correspond to what prior work broadly refers to as obsolete tests (Hu et al., 2023; Chi et al., 2025; Sun et al., 2023). We further distinguish them according to whether the test fails on the updated code. The third type, Test-Missing, extends beyond the scope of prior work by capturing the need for entirely new tests.
2. Benchmark Construction 2.1. Task Construction Pipeline We designed a multi-stage filtering pipeline to extract highquality test evolution task instances from real-world opensource Java projects. Figure 2 illustrates the overall process.
1.3. Task Definition To address these limitations, we formulate the task of project-level test evolution as follows. Definition 1.1 (Project-Level Test Evolution). Given a project repository P and a commit change ∆, the system must:
2.1.1. P ROJECT S OURCE . TEBench draws its projects from the Defects4J (Just et al., 2014) ecosystem, a widely-used benchmark repository in software engineering research that curates real-world Java projects with high-quality test suites. Starting from 17 Java open-source projects in Defects4J, we excluded 3 projects that do not use Maven as their build system, since our automated test execution and coverage analysis pipeline relies on the Maven Surefire Plugin and JaCoCo. The remaining 14 Maven-based projects span diverse functional domains, including data parsing, text processing, encoding,
1. Identify: determine which existing tests require modification and whether additional tests should be created; 2. Update: produce the corresponding test patch, including modifications to obsolete tests and any newly generated tests. Unlike prior method-level formulations, where the relevant 4
Benchmarking Coding Agents on Project-Level Test Evolution Project Source GitHub
Maven Build Required
67,670 commits from 14 projects
No Method Body Changes
6,169 commits from 14 projects
17 Defects4J Projects
Static Filtering Date Filter
Without Co-Modification
Execution-Based Validation Build Failure
Non-functional Changes
Unrelated Changes
561 commits from 12 projects
Deduplication
314 commits from 10 projects
Quality Filtering Exclude Merge Commits
Size Constraints
ifications have no measurable impact on test outcomes or code coverage, indicating non-functional changes such as test reorganization, comment edits, or stylistic adjustments, which account for approximately half of the exclusions; and (3) commits where the test changes lack a verifiable causal relationship with the production code changes. After execution-based validation, 561 commits remained across 12 projects. 2.1.4. Q UALITY F ILTERING
Figure 2. Task construction pipeline. Numbers indicate the remaining commits after each stage.
We applied final quality controls to ensure each task instance is suitable for benchmarking. Merge commits were excluded as they represent branch integration rather than individual code evolution; the actual evolution occurs in the constituent commits that we already analyze independently. We constrained test change size to 5–200 lines, a range determined through manual inspection of a stratified sample of candidate commits: commits with fewer than 5 lines of test changes consistently involved superficial modifications such as single-assertion tweaks or comment additions that do not constitute meaningful evolution instances, while those exceeding 200 lines typically involved large-scale refactoring that obscures the causal relationship between specific code changes and test updates. Finally, we performed method-level deduplication using (project, ClassName.methodName) as a composite key, retaining only the earliest commit when the same test method appears in multiple commits. This strategy was adopted after manual review revealed that later commits modifying the same test method predominantly represent iterative refinements rather than independent evolution scenarios, which would otherwise introduce redundancy and inflate task counts artificially. After quality filtering, 314 task instances from 10 projects constitute the final TEBench dataset. Four projects were excluded as they yielded insufficient valid task instances after the full filtering process.
compression, mathematical computation, chart rendering, and general-purpose language utilities, yielding a total of 67,670 commits. 2.1.2. S TATIC F ILTERING We applied a series of static filters to narrow the candidate set without requiring code execution. First, we restricted the time range to commits after January 2019 to ensure Java 8+ syntax compatibility and relevance to modern coding practices. For projects with limited recent history (e.g., commons-math), we relaxed the cutoff to 2016 to maintain a sufficient sample size. Next, we performed file-level scanning to retain only commits that simultaneously modify both production code (under src/main/) and test code (under src/test/), as co-modification is a necessary signal for test evolution. Finally, we used the javalang AST parser to extract method-level change information from each candidate commit, filtering out commits whose modifications are limited to imports, annotations, or comments rather than substantive method body changes. After static filtering, 6,169 commits remained across 14 projects. 2.1.3. E XECUTION -BASED VALIDATION
2.2. Task Formulation and Evaluation Protocol
For each remaining candidate, we constructed isolated execution environments using git worktree to validate the relationship between code changes and test modifications. Specifically, for each commit we built two versions: one with the full commit applied (both production and test changes), and one with only the non-test changes applied while retaining the original test suite. We executed the test suite on both versions and collected line and branch coverage via JaCoCo, enabling us to determine whether the test modifications address actual test failures or contribute to coverage improvements. The detailed version structure is formalized in Section 2.2. Based on the execution results, we excluded commits in three categories: (1) commits where the project fails to compile on the historical version, or where pre-existing test failures unrelated to the commit’s code changes are observed; (2) commits where the test mod-
2.2.1. V ERSION D ESIGN Each task instance in TEBench is built around a threeversion structure, as illustrated in Figure 3. V−1 represents the project state at the parent commit, before any changes are applied. V−0.5 is constructed by applying all changes from the commit except modifications to test files. This includes production code changes, build configuration updates, and resource file modifications, simulating the real-world scenario where a developer has committed code changes but has not yet updated the corresponding tests. V0 represents the full commit state including the developer’s actual test modifications, serving as the GT. This version structure also serves as the basis for classifying task instances into Breaking, Stale, and Missing, as defined in Section 1.3. During 5
Benchmarking Coding Agents on Project-Level Test Evolution Example: jsoup a349582236a7e69ce341f3b4ee913cb8cd19b955 committed on Mar 9, 2023 Changes Without Test File
V1 Version Test Diff
V0.5 Agent Input) TUBench Commit
Source Code Changes Only]
+12 -3 lines changed
CHANGES 3 | 0 src/main/java/org/jsoup/nodes Element.java 9 | 3
LLM Agent Test Evolution
+X -X lines changed
Agent's test modifications X | X
Executability measures whether the agent’s modifications produce valid, runnable tests. We compile and execute the union of test methods modified by the agent and those in the GT. This serves two purposes: executing the agent’s modifications verifies whether they introduce compilation or runtime errors, while executing GT methods that the agent did not modify reveals whether the agent missed broken tests that require repair. We assign a three-level score:
Full Changes
V0 Ground Truth) Ground Truth Commit
+39 -7 lines changed
CHANGES 3 | 0 src/main/java/org/jsoup/nodes Element.java 9 | 3 src/test/java/org/jsoup nodes ElementTest.java 20 | 1 parser HtmlTreeBuilderStateTest.java 4 | 2 select ElementsTest.java 3 | 1
0, sexec = 0.5, 1,
Identification / Update Metrics
Figure 3. Three-version structure and its dual role in classification and evaluation.
if compilation fails if compilation succeeds but tests fail if all tests pass (1)
Coverage Overlap measures how well the agent’s tests align with the developer’s testing intent for the changed code. We execute both the agent’s and the GT test suites, and collect line coverage and branch coverage restricted to the production methods modified by ∆. Rather than measuring absolute coverage improvement, we compute the overlap between the agent’s coverage and the GT coverage:
evaluation, V−0.5 is the project state presented to the coding agent. The agent is informed via its prompt that the most recent commit modified the production code without updating the test suite, and is tasked with identifying and updating any affected tests. The agent can access all project-level information, including commit history, commit messages, and code structure, through its standard tooling. The GT for evaluation is the developer’s actual test modifications in V0 . 2.2.2. I DENTIFICATION M ETRICS
sline =
The identification stage evaluates whether the agent correctly locates the tests that require attention. We extract the set of affected tests from the GT and compare it against the set of tests actually modified or added by the agent, computing Precision, Recall, and F1-score. We adopt different granularities for different change types. Modified and deleted test methods are evaluated at method-level granularity: a true positive requires the agent to modify or delete the same test method as in the GT. Newly added test methods are evaluated at file-level granularity: a true positive requires the agent to add at least one new test method in the same test file where the GT adds new methods. For modified tests, the agent should precisely identify which existing methods need changes; for new tests, it is unreasonable to expect the agent to predict the exact method names or count chosen by the developer, but it should recognize where new tests are needed.
agent gt |Cline ∩ Cline | , gt |Cline |
sbranch =
agent gt |Cbranch ∩ Cbranch | gt |Cbranch | (2)
where C agent and C gt denote the sets of lines or branches covered by the agent’s and GT tests, respectively. This design reflects the fact that the goal of test evolution is not to maximize coverage indiscriminately, but to ensure the test suite evolves in alignment with the developer’s intent regarding the specific code change. Modification Similarity measures how closely the agent’s test changes resemble the GT, capturing whether the agent makes precise, targeted modifications rather than excessive rewrites. We compute the token-level Jaccard similarity between the agent’s and GT test modifications:
smod =
|tokensagent ∩ tokensgt | |tokensagent ∪ tokensgt |
(3)
2.2.3. U PDATE M ETRICS The goal of test evolution is not merely to produce passing tests or to maximize coverage, but to align the test suite with the intent behind the code change. We therefore design our update metrics around the developer-written GT as a reference for evolution intent, evaluating agents across three dimensions: executability, coverage alignment, and modification similarity.
Composite Score. We combine the three dimensions into a single update score. Executability serves as a gate: if the agent’s modifications do not compile, the entire score is zero. When the GT produces no coverage change over the original tests (e.g., the change only updates assertion values), the modification similarity receives full weight: 6
Benchmarking Coding Agents on Project-Level Test Evolution Table 3. Task complexity statistics.
0.3 sline + 0.3 sbranch supdate = sexec × + 0.4 smod , smod ,
if |C gt | > 0 gt
if |C | = 0 (4)
Metric
Mean
P25
Median
P75
Max
Source lines changed Test lines changed Total files changed Test files per task Test methods per task
74.8 48.6 4.9 1.8 4.6
13 14 3 1 2
34 32 4 1 3
78 72 6 2 5
732 200 20 10 74
2.3. Dataset Statistics Table 2 provides an overview of the 10 projects and 314 task instances in TEBench. Source lines of code (Src LOC) count production code only, excluding test files. B, S, and M denote Breaking, Stale, and Missing, respectively.
that assume a one-to-one mapping between code changes and test modifications. Temporal Distribution. TEBench spans commits from 2016 to 2025. The majority of tasks (77.4%) originate from 2020 or later, with 2024–2025 contributing 125 tasks (39.8%), ensuring that the dataset reflects contemporary development practices and coding conventions.
Table 2. Overview of TEBench. Project
Tasks
Src LOC
Test Files
B
S
M
commons-cli commons-codec commons-collections commons-compress commons-csv commons-lang commons-math gson jfreechart jsoup
18 19 23 86 31 69 8 1 3 56
9,716 25,102 80,241 92,057 6,295 101,573 142,903 22,329 211,097 27,390
51 84 300 260 43 275 403 139 361 84
8 12 10 34 22 28 8 1 1 48
12 11 14 58 18 46 3 0 3 42
9 12 15 53 16 40 2 1 1 50
Total
314
718,703
2,000
172
207
199
3. Experimental Setup 3.1. Evaluated Systems We evaluate eight systems organized along two axes: a heuristic baseline and seven LLM-based configurations spanning three coding agent frameworks and six base models. Table 4 summarizes all evaluated systems. Table 4. Evaluated systems.
Label Distribution. The three evolution types are wellrepresented across the dataset: Breaking appears in 172 tasks (54.8%), Stale in 207 (65.9%), and Missing in 199 (63.4%). Notably, 219 tasks (69.7%) carry multiple labels, and 45 tasks (14.3%) exhibit all three types simultaneously. The most frequent combination is Stale + Missing (105 tasks, 33.4%), suggesting that when developers recognize quality degradation in existing tests, they often supplement new tests in the same commit. Only 95 tasks (30.3%) involve a single evolution type, confirming that real-world test evolution is predominantly multi-faceted.
Agent Framework
Base Model
Version
Heuristic Baseline
—
—
Claude Code Codex CLI OpenCode
Claude Sonnet 4.6 ChatGPT 5.3 Codex Claude Sonnet 4.6
v2.1.45 v0.114.0 v1.2.16
OpenCode OpenCode OpenCode OpenCode
Qwen3.5 GLM-5 Kimi-K2.5 DeepSeek-V3.2
v1.2.16 v1.2.16 v1.2.16 v1.2.16
3.1.1. H EURISTIC BASELINE .
Task Complexity. Table 3 summarizes the complexity characteristics of the task instances. The median task involves 4 changed files, 34 lines of source code changes, and 32 lines of test changes, indicating moderate complexity that is challenging yet tractable for current coding agents. The distribution exhibits a long tail: the most complex task spans 20 files with 732 lines of source changes.
To establish a lower bound on what structural analysis alone can achieve, we implement a static dependency baseline that operates in three steps. First, it extracts changed classes and methods from the source code diff using AST-level analysis via javalang. Second, it scans all @Test-annotated methods in the project, retaining those whose enclosing file imports a changed class and whose method body invokes a changed symbol. Third, it validates candidates by executing them with Maven’s Surefire plugin, filtering out methods that cannot be located at runtime. This baseline is designed exclusively for the identification subtask and does not perform test updates, serving as a reference for evaluating how far structural analysis alone can reach in locating affected tests.
Project-Level Characteristics. A key motivation for TEBench is that test evolution requires project-level reasoning. Our statistics confirm this: 114 tasks (36.4%) involve modifications to more than one test file, 63 tasks (20.1%) span multiple test packages, and 236 tasks (75.2%) require changes to more than one test method. These numbers demonstrate that a substantial portion of test evolution tasks cannot be adequately captured by method-level benchmarks 7
Benchmarking Coding Agents on Project-Level Test Evolution You are working on a test evolution task for a Java Maven project.
3.1.2. C ODING AGENTS AND BASE M ODELS .
## Context
The source code has already been updated in the current HEAD, while test code may now be outdated. Your task is to identify outdated tests and update them so they reflect current source behavior.
We evaluate three widely-adopted industrial coding agent frameworks: Claude Code (Anthropic, 2025) (Anthropic, closed-source), Codex CLI (OpenAI, 2025) (OpenAI, closed-source), and OpenCode (OpenCode Contributors, 2025) (open-source). These frameworks are paired with six base models that span closed-source flagships and open-weight families: Claude Sonnet 4.6, ChatGPT 5.3 Codex, Qwen3.5 (Team, 2025), GLM-5 (GLM, 2026), KimiK2.5 (Team, 2026), and DeepSeek-V3.2 (DeepSeek-AI, 2025). Claude Code and Codex CLI are evaluated under their respective default backbones (Claude Sonnet 4.6 and ChatGPT 5.3 Codex). OpenCode is evaluated with five backbones: Claude Sonnet 4.6, Qwen3.5, GLM-5, Kimi-K2.5, and DeepSeek-V3.2, yielding five distinct configurations under a single framework.
## Allowed and Forbidden Changes
- Allowed: - Test files (for example `src/test/**`) - Test resources/config files (for example `src/test/resources/**`) - Maven/build configuration needed to execute or align tests (for example `pom.xml`, module poms, surefire/failsafe config) - Forbidden: - Any production source changes under `src/main/**` - Any "fix" that makes tests pass by changing production logic instead of updating tests/build setup
## Workflow
1. Inspect source changes with `git diff HEAD1 -- src/main/`. 2. Inspect relevant tests and test configs. 3. Run a baseline verification to reproduce current behavior (prefer targeted tests first, then full test run if needed). 4. Suggestion: use JaCoCo (for example `target/site/jacoco/jacoco.xml`) to check coverage of changed production code. 5. Apply minimal, test-side/build-side changes. 6. Re-run verification and coverage checks; iterate only when there is a concrete next fix.
## Termination Conditions
Stop when any of these holds: - Relevant tests now pass AND coverage requirements below are satisfied, OR - Remaining failures are clearly unrelated/pre-existing and can‘t be resolved without editing `src/main/**`, OR - A new verification run shows no actionable new signal compared with the previous run. Coverage requirements (pass-only is NOT enough): - Passing tests alone is insufficient if changed production behavior is still weakly tested. - Suggest using JaCoCo coverage results to confirm changed production code is adequately covered.
## Output Requirements
- Do not commit changes. - Keep modifications minimal and explainable. - Before finishing, provide a concise summary: - files changed - why they changed - final verification command(s) and outcomes - coverage evidence JaCoCo - unresolved blockers (if any)
All configurations run with default agent settings and are given full access to the project workspace within an isolated environment (Section 3.3). We adopt a natural-run evaluation mode: each configuration receives the unified task prompt and is allowed to freely explore the project, execute tests, inspect coverage reports, and iteratively refine its modifications without any artificial constraints on its problem-solving strategy. Upon completion, we extract the actual modifications produced by each configuration to infer its identification decisions. Specifically, test methods that are modified or deleted are treated as identifications of obsolete tests, while newly added test methods are treated as identifications of missing tests.
Figure 4. Unified task prompt provided to all configurations.
JaCoCo coverage analysis to assess whether changed production code is adequately exercised, since passing tests alone may mask insufficient coverage of newly introduced behavior. The Termination Conditions component defines explicit stopping criteria to prevent configurations from entering infinite repair loops, allowing them to stop when tests pass with adequate coverage, when remaining failures are clearly unrelated to the current commit, or when successive verification runs yield no additional actionable signal. Finally, the Output Requirements component instructs configurations to keep modifications minimal and explainable, and to provide a concise summary of changed files, rationale, verification outcomes, and coverage evidence before finishing, which facilitates subsequent automated evaluation.
3.2. Task Prompt All configurations receive an identical, commit-typeagnostic task prompt regardless of whether a task instance involves Test-Breaking, Test-Stale, or Test-Missing changes, ensuring that each configuration must independently determine the nature and extent of required updates. The full prompt is presented in Figure 4.
3.3. Execution Environment For each task instance, we construct an isolated execution environment based on the V−0.5 version defined in Section 2.2. Specifically, we use Git’s worktree mechanism to create a dedicated working directory for each task: a new branch is created at the V−0.5 state, which contains the updated source code with original tests, and a separate worktree is attached to this branch. This approach provides full filesystem isolation between tasks, as each configuration operates in its own independent copy of the project at the correct historical state, free from interference by other versions or concurrent executions.
The prompt is structured around five components that together define the task boundaries. The Context component informs each configuration that source code has already been updated while tests may be outdated. The Allowed and Forbidden Changes component restricts modifications to test files, test resources, and build configuration, while strictly prohibiting any changes to production source code under src/main/, which mirrors the real-world constraint that the source change has already been committed and the task is solely to bring tests in line. Each configuration is also restricted from inspecting any commits beyond the current HEAD, which prevents it from accessing ground-truth test modifications in subsequent commits. The Workflow component suggests a recommended sequence of inspection, verification, and iteration steps, including the use of
Compared to provisioning a separate Docker container per project, the worktree-based approach is significantly more lightweight while achieving equivalent isolation for our purposes, since each task has its own independent source tree, 8
Benchmarking Coding Agents on Project-Level Test Evolution
build artifacts, and test execution context while sharing only the read-only Git object store with the main repository. To support reproducibility, we provide a Docker image that bundles all project environments and evaluation scripts in our replication package, allowing other researchers to replicate our experiments without configuring individual project dependencies.
17.8 percentage points (mean of 13.7). No configuration deviates from this pattern, which indicates a shared inductive bias toward over-prediction rather than an idiosyncratic property of any single backbone. Together, these observations suggest that the performance bottleneck lies not in any specific framework or backbone, but in the inherent difficulty of project-level test identification. The three evolution types exhibit substantially different difficulty, and the relative ordering is preserved across all seven configurations. Test-Breaking is the easiest, with an average F1 of 59.9% and a tight spread of 2.1 points across configurations, since explicit execution failure signals are available to locate affected tests. Test-Missing occupies an intermediate position with an average F1 of 52.9%, where relatively high Precision is paired with lower Recall, indicating that configurations recognize some scenarios requiring new tests but miss over half of them. Test-Stale is by far the hardest, with an average F1 of 35.8% and both Precision and Recall substantially depressed. Because stale tests still pass on the updated code, no execution signal indicates that updates are needed, and configurations must rely entirely on proactive semantic reasoning, a capability that the seven evaluated systems lack in roughly equal measure.
4. Results and Analysis We organize our evaluation around four research questions: • RQ1 (Identification): How effectively can current configurations identify obsolete tests and missing tests in evolving projects? • RQ2 (Update): How effectively can current configurations update obsolete tests and generate missing tests? • RQ3 (Task Characteristics): How do task characteristics influence configuration performance? • RQ4 (Failure Analysis): What are the typical failure modes of configurations on test evolution tasks?
The heuristic baseline provides an informative reference point. Its Recall of 66.1% surpasses every LLM-based configuration, while its Precision is only 2.0% with over 39,000 false positives. The contrast is particularly striking on TestStale, where the heuristic’s Recall of 59.1% exceeds the seven-configuration average of 42.2% by 16.9 percentage points. Even this exhaustive one-hop dependency analysis fails to reach 100% Recall on any type, with approximately one-third of truly affected tests remaining undetected, which indicates that a substantial portion of test-code dependencies operate through indirect channels such as multi-hop call chains, shared state, or implicit semantic coupling that structural analysis cannot capture.
4.1. RQ1: Identification Effectiveness Table 5 presents the identification results across all 314 task instances, reported both overall and per evolution type. Table 5. Identification results (Precision / Recall / F1, %). Best F1 per type column among the seven LLM-based configurations is bolded. The heuristic baseline participates only in identification and is reported separately for reference. Overall
Breaking
Stale
Missing
Configuration
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Heuristic Baseline
2.0
66.1
4.0
1.7
73.8
3.3
1.6
59.1
3.0
1.0
47.0
2.0
Claude Code Codex CLI OpenCode (Sonnet)
40.6 43.4 44.2
56.1 57.4 53.3
47.1 49.4 48.3
53.2 53.3 52.6
67.6 69.4 66.8
59.6 60.3 58.8
29.5 32.7 33.3
43.1 43.6 38.3
35.0 37.4 35.6
65.1 64.4 62.1
46.3 46.5 43.5
54.1 54.0 51.2
OpenCode (Qwen) OpenCode (GLM) OpenCode (Kimi) OpenCode (DeepSeek)
40.9 43.2 44.1 39.3
58.7 57.4 53.9 54.7
48.2 49.3 48.5 45.7
53.0 53.0 56.1 52.9
70.4 71.0 66.2 65.8
60.5 60.6 60.7 58.6
29.8 32.6 32.1 27.8
45.4 43.1 40.4 41.8
36.0 37.1 35.8 33.4
62.1 63.6 67.7 59.6
48.2 46.7 43.0 43.0
54.3 53.9 52.6 50.0
Answer to RQ1: Across seven configurations spanning three agent frameworks and six base models, identification F1 remains within 45.7% to 49.4%, with backbone variation contributing only 3.6 F1 points and framework variation only 1.2 points. All configurations exhibit a systematic Recall-over-Precision imbalance, with the gap ranging from 9.1 to 17.8 percentage points, which reveals a shared bias toward over-prediction. Test-Stale is the hardest type, with an average F1 of approximately 36%, because it requires proactive semantic reasoning without execution signals, and even exhaustive structural analysis misses about one-third of affected tests.
The seven LLM-based configurations achieve remarkably similar overall F1 scores, ranging from 45.7% to 49.4%, with less than four percentage points separating the strongest from the weakest. This convergence holds across closedsource flagships and open-weight backbones, as well as across proprietary and open-source agent frameworks. When the framework is held constant, five backbones evaluated under OpenCode span only 3.6 F1 points; when the backbone is held constant, the Claude Code and OpenCode configurations sharing Claude Sonnet 4.6 differ by 1.2 F1 points. Beneath this aggregate convergence, all seven configurations exhibit a systematic Recall-over-Precision imbalance, with Recall exceeding Precision by between 9.1 and
4.2. RQ2: Update Effectiveness Table 6 presents the update quality metrics for the seven LLM-based configurations. Overall composite scores clus9
Benchmarking Coding Agents on Project-Level Test Evolution Table 6. Update results (%). Exec: executability score; Cov: coverage overlap score; Mod: modification similarity; OA: overall composite score. Best OA per type column is bolded. Overall
Breaking
Stale
Missing
Configuration
Exec
Cov
Mod
OA
Exec
Cov
Mod
OA
Exec
Cov
Mod
OA
Exec
Cov
Mod
OA
Claude Code Codex CLI OpenCode (Sonnet)
97.0 99.2 96.2
90.2 79.2 77.3
51.0 50.3 47.6
70.5 72.3 68.9
94.8 98.5 93.6
90.7 71.4 69.9
65.0 63.3 62.5
73.2 76.6 73.8
97.3 99.3 95.9
90.7 79.0 76.8
45.8 45.8 41.0
68.5 70.8 65.4
96.7 99.5 96.0
88.5 73.2 71.8
39.1 38.1 36.9
63.8 65.7 62.6
OpenCode (Qwen) OpenCode (GLM) OpenCode (Kimi) OpenCode (DeepSeek)
94.9 95.5 87.7 91.4
83.1 87.8 80.1 80.4
48.6 47.8 54.0 50.3
67.0 69.3 63.6 64.5
94.8 94.2 90.7 91.0
83.9 88.2 84.0 82.6
64.6 59.7 70.9 65.4
73.3 73.0 68.3 70.4
94.2 95.4 84.8 90.3
82.2 88.5 79.2 80.7
41.7 41.7 46.3 43.6
62.8 66.7 59.8 60.9
93.5 94.2 85.9 88.9
80.0 86.2 77.8 76.5
37.0 36.4 42.1 38.4
59.1 62.2 56.0 55.4
ter within a band of 8.8 percentage points, ranging from 63.6% to 72.3%, with Codex CLI achieving the highest score on every type column.
whereas identification difficulty ranks Test-Breaking, TestMissing, and Test-Stale. This crossover indicates that TestStale is hardest to identify but not hardest to update, since stale tests require only targeted assertion changes once located, while Test-Missing is easier to identify but harder to update because it demands generating entirely new code that naturally produces lower similarity to GT and lower coverage overlap.
Configurations achieve high executability scores, ranging from 87.7% to 99.2%, indicating that producing compilable and runnable test modifications is largely tractable. Coverage overlap differs more substantially across configurations, with Claude Code attaining 90.2% and OpenCode with the GLM backbone attaining 87.8%, both well above the 77% to 84% range of the remaining five configurations. This advantage in coverage does not translate directly into higher composite scores, as Codex CLI leads at 72.3% despite a coverage overlap of only 79.2%, while Claude Code lags at 70.5% with the highest coverage. The pattern reflects a recurring dimensional trade-off: Claude Code applies more aggressive modifications that improve coverage at the cost of marginally lower executability of 97.0% relative to Codex CLI’s 99.2%, whereas OpenCode with the Kimi backbone pursues higher modification fidelity at 54.0%, the highest across configurations, at the cost of pronounced executability degradation to 87.7%, the lowest across configurations, which depresses its composite score to 63.6%.
Answer to RQ2: The seven configurations cluster within an 8.8-point band on the composite update score, ranging from 63.6% to 72.3%. Executability remains consistently high (87.7% to 99.2%) yet exceeds modification similarity by 33.7 to 48.9 percentage points within each configuration, indicating that producing executable tests is far easier than producing tests aligned with developer intent. The type-wise difficulty ranking flips between the two subtasks: Test-Stale is hardest to identify but not to update, whereas Test-Missing exhibits the inverse pattern. 4.3. RQ3: Impact of Task Characteristics To understand what makes test evolution tasks difficult, we analyze configuration performance, averaged across the seven LLM-based configurations, along three task characteristic dimensions: evolution type composition, source change scale, and test change scope. Table 7 summarizes the results.
The modification similarity score is the lowest sub-metric across all configurations and evolution types, ranging from 36.4% to 70.9%, and falls 33.7 to 48.9 percentage points below the corresponding executability score within each configuration. This systematic gap indicates that current configurations can produce executable test modifications whose surface form diverges substantially from how developers actually update tests. The gap widens further on Test-Missing, where modification similarity drops to between 36.4% and 42.1%, reflecting the inherently larger implementation space when generating new tests rather than revising existing assertions. The pattern argues against treating executability as a sufficient proxy for update quality, since high executability can mask substantial divergence from developer intent.
The type composition dimension reveals how the cooccurrence of different evolution types shapes task difficulty. Single-type tasks serve as instructive baselines: Breakingonly tasks achieve the highest update score of 84.2% with solid identification F1 of 62.0%, while Stale-only tasks are substantially harder to identify with F1 of 33.1% but still yield high update scores of 78.4% once the correct tests are located, which confirms that stale tests are difficult to find but require only targeted modifications. Missing-only tasks are too few in number to support robust conclusions, although their low update score of 41.7% hints at the difficulty of generating new tests from scratch. Among multi-type combinations, Breaking + Missing achieves the highest identification F1 of 74.3%, since breaking tests provide explicit failure signals that anchor the search process. Once Test-
The type-wise difficulty ranking on the update task differs from that on identification. Update difficulty follows the order Test-Breaking with an average composite score of 72.7%, Test-Stale at 65.0%, and Test-Missing at 60.7%, 10
Benchmarking Coding Agents on Project-Level Test Evolution Table 7. Impact of task characteristics on configuration performance, averaged across the seven LLM-based configurations. IdF1: identification F1 (%); Up-OA: update overall score (%). Dimension
Group
N
Id-F1
Up-OA
Breaking-only Stale-only Missing-only
58 33 4
62.0 33.1 68.8
84.2 78.4 41.7
Breaking + Missing Breaking + Stale + Missing Breaking + Stale Stale + Missing
45 45 24 105
74.3 64.8 29.8 34.8
63.5 65.5 75.4 58.2
Source Change Scale
Small (≤19 lines) Medium (20–55 lines) Large (≥56 lines)
102 100 112
46.0 52.1 46.1
71.4 68.4 64.6
Test Change Scope
Small (1 method) Medium (2–3 methods) Large (≥4 methods)
81 102 131
22.7 46.6 53.2
61.9 70.4 69.9
Type Composition
exhibit a consistent tendency to modify a similar number of test methods regardless of task size: on single-method tasks, configurations predict approximately 3.6 methods on average, well above the affected count, producing massive over-prediction that collapses Precision to 13.6%. On large tasks, the prediction volume aligns more naturally with the GT, yielding higher Precision of 53.2%. This finding points to a fundamental limitation, namely that current configurations lack the ability to calibrate their modification scope to the actual task requirements, applying a roughly constant effort budget regardless of whether the task demands touching one method or ten. Answer to RQ3: Three factors shape task difficulty. First, Test-Stale acts as a poison factor that sharply reduces identification performance in mixed-type tasks, although the presence of explicit Missing signals partially compensates for this effect. Second, medium-scale source changes are easier to identify than both small and large ones, forming an inverted-U pattern. Third, configurations lack scope calibration, causing severe over-prediction on small tasks.
Stale enters the combination, identification F1 drops sharply regardless of what other types are present, with Breaking + Stale falling to 29.8% and Stale + Missing falling to 34.8%. This pattern suggests that Test-Stale acts as a “poison factor” in identification, since the absence of any execution signal undermines the systematic location of all affected tests. A noteworthy exception arises when all three types co-occur: Breaking + Stale + Missing recovers to an identification F1 of 64.8%, well above Breaking + Stale alone, suggesting that the explicit signals from Missing components partially compensate for the disorientation introduced by Stale. An interesting contrast emerges in the update dimension, where Breaking + Stale tasks achieve the highest multi-type update score of 75.4% despite having among the lowest identification F1, indicating that once located, the required updates for breaking and stale types are relatively straightforward compared to generating missing tests.
4.4. RQ4: Failure Analysis We select the jsoup motivating example from Section 1 for in-depth analysis, as it is the most representative task in TEBench: it simultaneously involves all three evolution types, requires updates to five test methods across three files, and exhibits the “small change, wide impact” characteristic, with only 12 source-code lines changed. Figure 5 compares the GT modifications with those produced by the three industrial agent frameworks (Claude Code, Codex CLI, and OpenCode under the Sonnet backbone), which serve as a representative cross-section of the seven configurations evaluated in TEBench.
The source change scale dimension reveals a non-monotonic pattern in identification performance. Medium-scale changes between 20 and 55 lines yield the highest identification F1 of 52.1%, while both small-scale changes at 46.0% and large-scale changes at 46.1% are more difficult. Small source diffs provide insufficient contextual information for configurations to infer the scope of test impact, while large diffs present excessive information that complicates focusing on the most relevant changes. Update quality, in contrast, decreases monotonically from 71.4% for small changes to 64.6% for large changes, consistent with the intuition that larger source changes require more extensive and complex test modifications.
All three depicted configurations successfully fix the three Breaking tests, correctly updating the expected HTML strings. Claude Code additionally adds a semantic comment to divAInlineable, explaining the root cause of the change. However, none of these configurations updates the Stale test unwrap. The four OpenCode configurations using open-weight backbones exhibit qualitatively similar patterns, none of which extends materially beyond the crosssection depicted in Figure 5. For Test-Missing, Codex CLI partially addresses the missing coverage by generating a new test. This test is not entirely misaligned with the GT, since it captures one representative scenario also covered by inlineInBlockShouldIndent, namely the case where an inline element follows non-blank preceding text. However, the GT test is broader and behavior-oriented, systematically verifying consistent indentation across three distinct input variants, including non-blank text, a newline,
The test change scope dimension reveals a counterintuitive pattern in which identification F1 increases from 22.7% for single-method tasks to 53.2% for tasks involving four or more methods. This is not because single-method tasks are inherently harder to understand. Rather, configurations 11
Benchmarking Coding Agents on Project-Level Test Evolution Breaking Test
Ground Truth Changes Diff CHANGE Element.java
ElementTest.java
Stale Test
Missing Test
HtmlTreeBuilderStateTest.java
ElementsTest.java
nestedAnchorElements01() nestedAnchorElements02()
unwrap()
inlineInBlockShouldIndent() divAInlineable()
Not implemented
Partially Implemented
Claude Code Changes Diff
Codex Changes Diff
OpenCode Changes Diff
@Test void divAInlineable() { + // a preceding blank text node means the <a> is treated as the first element in the block, not inlineable String html = "<body><div> <a>Text</a>"; Document doc = Jsoup.parse(html); - assertEquals("<div><a>Text</a>\n</div>", doc.body().html()); + assertEquals("<div>\n <a>Text</a>\n</div>", doc.body().html()); // Fixing divAInlineable() : Pass }
@Test void divAInlineable() { String html = "<body><div> <a>Text</a>"; Document doc = Jsoup.parse(html); - assertEquals("<div><a>Text</a>\n</div>", doc.body().html()); + assertEquals("<div>\n <a>Text</a>\n</div>", doc.body().html()); // Fixing divAInlineable() : Pass + } + + @Test void divAInlineableAfterNonBlankText() { + Document doc = Jsoup.parse("<body><div>x<a>Text</a>"); + assertEquals("<div>\n x<a>Text</a>\n</div>", doc.body().html()); // Adding divAInlineableAfterNonBlankText() : Pass }
@Test void divAInlineable() { String html = "<body><div> <a>Text</a>"; Document doc = Jsoup.parse(html); - assertEquals("<div><a>Text</a>\n</div>", doc.body().html()); + assertEquals("<div>\n <a>Text</a>\n</div>", doc.body().html()); // Fixing divAInlineable() : Pass }
" <head></head>\n" + - " <body><a href=\"#1\"> </a>\n" + + " <body>\n" + + " <a href=\"#1\"> </a>\n" + " <div>\n" + // Fixing nestedAnchorElements01 : Pass " <head></head>\n" + - " <body><a href=\"#1\"> </a>\n" + + " <body>\n" + + " <a href=\"#1\"> </a>\n" + " <div>\n" + // Fixing nestedAnchorElements02 : Pass
" <head></head>\n" + - " <body><a href=\"#1\"> </a>\n" + + " <body>\n" + + " <a href=\"#1\"> </a>\n" + " <div>\n" + // Fixing nestedAnchorElements01 : Pass " <head></head>\n" + - " <body><a href=\"#1\"> </a>\n" + + " <body>\n" + + " <a href=\"#1\"> </a>\n" + " <div>\n" + // Fixing nestedAnchorElements02 : Pass
" <head></head>\n" + - " <body><a href=\"#1\"> </a>\n" + + " <body>\n" + + " <a href=\"#1\"> </a>\n" + " <div>\n" + // Fixing nestedAnchorElements01 : Pass
" <head></head>\n" + - " <body><a href=\"#1\"> </a>\n" + + " <body>\n" + + " <a href=\"#1\"> </a>\n" + " <div>\n" + // Fixing nestedAnchorElements02 : Pass
Figure 5. Case study on Task 293 (jsoup): GT changes (top) versus actual modifications produced by each configuration (bottom).
and a blank space before the inline element. In contrast, Codex CLI covers only one of these scenarios, leaving the broader behavioral consistency unchecked. This suggests that Codex CLI identifies part of the newly exposed behavior but does not recover the full semantic scope reflected in the developer-written test.
this semantic gap. Answer to RQ4: Current configurations operate in a reactive “execute-fail-fix” loop that succeeds for TestBreaking but cannot address Test-Stale, since no failure signal is available, nor can they fully cover Test-Missing, since the loop encourages branch-level patching rather than behavior-level test design.
Since Codex CLI is the only configuration that both fixes all Breaking tests and attempts to generate a new test, we examine its execution trajectory to understand the underlying problem-solving strategy. Codex CLI begins by inspecting the source diff and searching for related test keywords, then runs ElementTest in isolation, where only divAInlineable fails. It fixes this assertion and adds its new test, but the new test’s assertion is initially incorrect, requiring an additional fix-rerun cycle. Only when Codex CLI later runs the full test suite does it discover the two failures in HtmlTreeBuilderStateTest, which resides in a different package. After patching these, all tests pass with adequate JaCoCo coverage, and Codex CLI terminates. This trajectory exposes two fundamental limitations. First, the configuration’s identification strategy is entirely executiondriven, since it discovers affected tests through test failures rather than through proactive reasoning about change impact: the HtmlTreeBuilderStateTest failures were found by running the full suite rather than by analyzing cross-package dependencies. Second, the configuration’s termination is governed by the joint condition of all tests passing and adequate coverage being met, which structurally prevents it from detecting stale tests. The unwrap test passes because stripNewlines() masks the formatting change, and no execution signal alerts the configuration to
5. Related Work Test Generation and Test Evolution. Automated test generation has been extensively studied. Traditional searchbased tools such as EvoSuite (Fraser & Arcuri, 2011) and constraint-based approaches (Lukasczyk & Fraser, 2022) maximize structural coverage through evolutionary algorithms. With the rise of LLMs, recent work has shifted toward more natural test generation: ChatUniTest (Chen et al., 2024) introduced a generation-validation-repair loop, HITS (Wang et al., 2024a) decomposes methods via slicing for incremental generation, and CoverUp (Altmayer Pizzorno & Berger, 2025) iteratively targets uncovered lines. Other notable approaches further improve coverage and quality through program analysis, execution path guidance, and multi-agent collaboration (Lemieux et al., 2023; Schäfer et al., 2023; Ryan et al., 2024; Yang et al., 2024a; Yuan et al., 2024; Pan et al., 2025; Wang et al., 2024b; Jain et al., 2025; Wang et al., 2025a). However, as LLM-based coding agents become increasingly embedded in development workflows, test evolution—updating existing tests in response to code changes— emerges as a more common and practical need than generat12
Benchmarking Coding Agents on Project-Level Test Evolution
ing tests from scratch. Researchers have studied this challenge from both empirical (Zaidman et al., 2011; Marsavina et al., 2014) and automation perspectives. SITAR (Wang et al., 2021) identified factors influencing test updates through a large-scale empirical study. CEPROT (Hu et al., 2023) proposed a Transformer-based approach for identifying and updating obsolete tests given method-level code changes. REACCEPT (Chi et al., 2025) integrated LLMs with dynamic validation to automate both identification and updating. Other recent work has further advanced test repair and update techniques (Sun et al., 2023; Liu et al., 2024; Yaraghi et al., 2025; Zhang et al., 2025). Several benchmarks have been proposed alongside these methods, such as Updates4J (Zhang et al., 2025) with 195 samples, but they all adopt a method-level paired formulation where the mapping between changed code and affected tests is pre-given. TEBench is the first to define a project-level task requiring the system to autonomously identify tests requiring modification and determine where new tests are needed from the full project context.
External Validity. TEBench covers 10 Java projects from the Defects4J ecosystem, which may limit generalizability to other languages or project types. The uneven distribution across projects reflects natural variation in test maintenance activity rather than sampling bias. We mitigate this by reporting per-project results alongside aggregate metrics, and note that extending TEBench to additional languages and projects is a natural direction for future work.
Coding Agents and SE Benchmarks. LLM-based coding agents such as SWE-agent (Yang et al., 2024b), OpenHands (Wang et al., 2025b), Claude Code (Anthropic, 2025), Codex CLI (OpenAI, 2025), and OpenCode (OpenCode Contributors, 2025) have demonstrated strong capabilities on code understanding and repair (Xia et al., 2025), yet none have been systematically evaluated on test evolution tasks. On the benchmarking side, SWE-bench (Jimenez et al., 2024) established the paradigm of repository-level issue resolution, inspiring extensions toward long-horizon evolution and enterprise complexity (Chowdhury et al., 2024; Thai et al., 2025; Deng et al., 2025). SWT-bench (Mündler et al., 2024) shifted focus to test generation for known bugs (Ahmed et al., 2024). TEBench targets a complementary dimension: evolving existing test suites alongside production code changes, bridging the gap between code repair benchmarks and test generation benchmarks.
7. Conclusion
Construct Validity. Our evaluation measures coverage overlap with the GT rather than absolute coverage improvement, based on the rationale that test evolution should align with developer intent for the specific change rather than maximize coverage indiscriminately. Our unified task prompt was designed for fairness across all configurations; configuration-specific prompt tuning could yield higher absolute performance but would introduce confounds that undermine cross-configuration comparability. Our setup therefore prioritizes standardization and reproducibility over per-configuration optimization.
We presented TEBench, the first project-level benchmark for test evolution, which requires systems to autonomously identify tests requiring modification and determine where new tests are needed given a project repository and a codechanging commit. TEBench comprises 314 task instances from 10 real-world Java projects, covering three evolution types: Test-Breaking, Test-Stale, and Test-Missing. Our evaluation of seven configurations spanning three industrial agent frameworks and six base models reveals that current systems achieve an identification F1 of only 45.7% to 49.4%, with Test-Stale posing the greatest challenge at an average F1 of approximately 36% due to the absence of execution failure signals, which fundamentally limits the ability of current systems to detect semantically outdated tests or proactively generate missing ones. These findings point to several directions for future work. First, integrating static dependency analysis with LLMbased semantic reasoning could combine the high recall of structural approaches with the precision of language understanding. Second, developing systems that reason about testing intent beyond execution signals, for instance by analyzing coverage gaps, inferring behavioral contracts from code changes, or aligning modifications more closely with developer intent, could address both the Test-Stale bottleneck and the systematic divergence between executable and developer-aligned test updates. Third, extending TEBench to additional programming languages and larger-scale industrial projects would further validate the generalizability of our findings. We hope that TEBench serves as a catalyst for advancing test evolution capabilities in coding agents.
6. Threats to Validity Internal Validity. The GT is derived from developerwritten test modifications in real commits, which constitutes a faithful record of actual developer intent rather than an arbitrary gold standard, even though it may not represent the unique correct solution. Functionally equivalent updates with different implementations could be penalized by our modification similarity metric. To mitigate this, we employ multiple complementary metrics: executability is entirely objective; coverage overlap remains meaningful regardless of implementation differences; and modification similarity is interpreted alongside the other dimensions rather than in isolation, ensuring that semantically valid alternatives are not disproportionately penalized.
13
Benchmarking Coding Agents on Project-Level Test Evolution
References
Jain, K., Synnaeve, G., and Rozière, B. TestGenEval: A real world unit test generation and test completion benchmark. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Ahmed, T., Hirzel, M., Pan, R., Shinnar, A., and Sinha, S. TDD-Bench verified: Can LLMs generate tests for issues before they get resolved? arXiv preprint arXiv:2412.02883, 2024.
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE-Bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/ forum?id=VTF8yNQM66.
Altmayer Pizzorno, J. and Berger, E. D. CoverUp: Effective high coverage test generation for Python. Proceedings of the ACM on Software Engineering (PACMSE), 2(FSE): 2897–2919, 2025. Anthropic. Claude code. https://docs.anthropic. com/en/docs/claude-code, 2025. URL https://docs.anthropic.com/en/docs/ claude-code. Accessed: June 2025. Chen, Y., Hu, Z., Zhi, C., Han, J., Deng, S., and Yin, J. ChatUniTest: A framework for LLM-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE-Companion), pp. 572–576, 2024.
Just, R., Jalali, D., and Ernst, M. D. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA), pp. 437–440, 2014. Lemieux, C., Inala, J. P., Lahiri, S. K., and Sen, S. CodaMOSA: Escaping coverage plateaus in test generation with pre-trained large language models. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE), pp. 919–931, 2023.
Chi, J., Wang, X., Huang, Y., Yu, L., Cui, D., Sun, J., and Sun, J. REACCEPT: Automated co-evolution of production and test code based on dynamic validation and large language models. Proceedings of the ACM on Software Engineering (PACMSE), 2(ISSTA):1234–1256, 2025.
Liu, J., Yan, J., Xie, Y., Yan, J., and Zhang, J. Fix the tests: Augmenting LLMs to repair test cases with static collector and neural reranker. In Proceedings of the 35th IEEE International Symposium on Software Reliability Engineering (ISSRE), pp. 367–378, 2024.
Chowdhury, N., Aider, J., Cassano, F., Zhuo, J., Liu, Q., Jimenez, C. E., Narasimhan, K., and Press, O. SWEBench verified: A stricter evaluation for AI software engineering. arXiv preprint arXiv:2406.12952, 2024.
Lukasczyk, S. and Fraser, G. Pynguin: Automated unit test generation for Python. In Proceedings of the 44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 168–172, 2022.
DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large language models. CoRR, abs/2512.02556, 2025. Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., et al. SWEBench pro: Can AI agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025.
Marsavina, C., Romano, D., and Zaidman, A. Studying finegrained co-evolution patterns of production and test code. In Proceedings of the 14th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM), pp. 195–204, 2014.
Fraser, G. and Arcuri, A. EvoSuite: Automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE), pp. 416–419, 2011.
Mündler, N., Müller, M. N., He, J., and Vechev, M. SWTBench: Testing and validating real-world bug-fixes with code agents. Advances in Neural Information Processing Systems (NeurIPS), 37:81857–81887, 2024.
GLM. GLM-5: from vibe coding to agentic engineering. CoRR, abs/2602.15763, 2026.
OpenAI. Codex CLI. https://github.com/ openai/codex, 2025. URL https://github. com/openai/codex. Accessed: June 2025.
Hu, X., Liu, Z., Xia, X., Liu, Z., Xu, T., and Yang, X. Identify and update test cases when production code changes: A transformer-based approach. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1111–1122, 2023.
OpenCode Contributors. OpenCode: An opensource coding agent. https://github.com/ opencode-ai/opencode, 2025. URL https: //github.com/opencode-ai/opencode. Accessed: June 2025. 14
Benchmarking Coding Agents on Project-Level Test Evolution
Pan, R., Kim, M., Krishna, R., Pavuluri, R., and Sinha, S. ASTER: Natural and multi-language unit test generation with LLMs. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 413–424, 2025.
Wang, Z., Liu, K., Li, G., and Jin, Z. HITS: High-coverage LLM-based unit test generation via method slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1258–1268, 2024a.
Rahman, S., Kuhar, S., Cirisci, B., Garg, P., Wang, S., Ma, X., Deoras, A., and Ray, B. UTFix: Change aware unit test repairing using LLM. Proceedings of the ACM on Programming Languages (PACMPL), 9(OOPSLA1):143– 168, 2025.
Wang, Z., Liu, M., Chu, Z., Wang, W., Song, D., and Ma, L. TestAgent: An LLM-based multi-agent system for automated unit test generation. arXiv preprint arXiv:2401.01602, 2024b. Xia, C. S., Deng, Y., Dunn, S., and Zhang, L. Demystifying LLM-based software engineering agents. Proceedings of the ACM on Software Engineering (PACMSE), 2(FSE): 801–824, 2025.
Ryan, G., Jain, S., Shang, M., Wang, S., Ma, X., Ramanathan, M. K., and Ray, B. Code-aware prompting: A study of coverage-guided test generation in regression setting using LLM. Proceedings of the ACM on Software Engineering (PACMSE), 1(FSE):951–971, 2024.
Yang, C., Chen, J., Lin, B., Zhou, J., and Wang, Z. Enhancing LLM-based test generation for hard-to-cover branches via program analysis. arXiv preprint arXiv:2404.04966, 2024a.
Schäfer, M., Nadi, S., Eghbali, A., and Tip, F. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering (TSE), 50(1):85–105, 2023.
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. SWE-Agent: Agentcomputer interfaces enable automated software engineering. Advances in Neural Information Processing Systems (NeurIPS), 37:50528–50652, 2024b.
Sun, W., Yan, M., Liu, Z., Xia, X., Lei, Y., and Lo, D. Revisiting the identification of the co-evolution of production and test code. ACM Transactions on Software Engineering and Methodology (TOSEM), 32(6):1–37, 2023.
Yaraghi, A. S., Holden, D., Kahani, N., and Briand, L. C. Automated test case repair using language models. IEEE Transactions on Software Engineering (TSE), 51(4):1104– 1133, 2025.
Team, K. Kimi K2.5: visual agentic intelligence. CoRR, abs/2602.02276, 2026.
Yuan, Z., Liu, M., Ding, S., Wang, K., Chen, Y., Peng, X., and Lou, Y. Evaluating and improving ChatGPT for unit test generation. Proceedings of the ACM on Software Engineering (PACMSE), 1(FSE):1703–1726, 2024.
Team, Q. Qwen3 technical report. CoRR, abs/2505.09388, 2025. Thai, M. V., Le, T., Manh, D. N., Nhat, H. P., and Bui, N. D. SWE-Evo: Benchmarking coding agents in longhorizon software evolution scenarios. arXiv preprint arXiv:2512.18470, 2025.
Zaidman, A., Van Rompaey, B., Van Deursen, A., and Demeyer, S. Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empirical Software Engineering, 16(3):325–364, 2011.
Wang, S., Wen, M., Liu, Y., Wang, Y., and Wu, R. Understanding and facilitating the co-evolution of production and test code. In Proceedings of the 28th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 272–283, 2021.
Zhang, Y., Yang, Z., Pan, S., and Liu, Z. Unit test update through LLM-driven context collection and errortype-aware refinement. arXiv preprint arXiv:2509.24419, 2025.
Wang, W., Yang, C., Wang, Z., Huang, Y., Chu, Z., Song, D., Zhang, L., Chen, A. R., and Ma, L. TestEval: Benchmarking large language models for test case generation. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 3547–3562, 2025a. Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. OpenHands: An open platform for AI software developers as generalist agents. In The Thirteenth International Conference on Learning Representations (ICLR), 2025b. 15