arXiv:2605.04320v1 [cs.SE] 5 May 2026
Reproduction Test Generation for Java SWE Issues Toufique Ahmed
Jatin Ganhotra
IBM Yorktown Heights, New York, USA [email protected]
IBM Yorktown Heights, New York, USA [email protected]
Avraham Shinnar
Martin Hirzel
IBM Yorktown Heights, New York, USA [email protected]
IBM Yorktown Heights, New York, USA [email protected]
Abstract Given an issue on a software repository, a reproduction test confirms its presence in the code before it gets fixed and its absence after. Reproduction tests provide crucial execution-based feedback for diagnosis and validation during software development. Unfortunately, they are usually missing. Therefore, recent work has introduced both benchmarks and a thriving literature on solutions for reproduction test generation from issues. However, that work has focused on Python and neglected other languages such as Java, which is important for enterprise software. This paper introduces both a benchmark and a solution for Java repository-level reproduction test generation. The benchmark, TDD-Bench-Java, is the first to model this problem and comprises 250 instances sourced from popular open-source repositories. The solution, e-Otter++ for Java, adapts a state-of-the-art reproduction test generator for Python to yield high performance on Java. To evaluate in an industry setting, besides empirical results with TDD-Bench-Java, this paper also presents results with a contamination-free proprietary dataset. Overall, we hope that this paper contributes to bringing better diagnosis and validation to Java software development.
Keywords LLMs, SWE Patches, Reproduction Tests ACM Reference Format: Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel. 2026. Reproduction Test Generation for Java SWE Issues. In . ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
Recent advances in AI coding assistants combine the promise of creating more code faster with the burden of checking that code for correctness. The most effective approach to checking code correctness at scale remains testing, and in the case of new bugs, this necessitates reproduction tests. A reproduction test is a test that Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
fails on the current code-base to confirm the presence of an issue and passes in the new code after the issue has been addressed to confirm its absence. Reproduction test generation differs from traditional test generation since it must start from an issue and since the new code on which a reproduction test should pass does not yet exist at test generation time. Unfortunately, while there has been good progress towards reproduction test generation for Python [2, 18, 23, 28], this is less true for other languages. For example, Java is widely used in industry thanks to its enterprise software ecosystem, but reproduction test generation for Java lags behind that for Python. Because of Java’s static type system and object-oriented nature, workflows and models optimized for Python tend to perform less well for Java [24, 32]. This is exacerbated by the fact that bugs in enterprise Java applications do not follow the same distribution as bugs in open-source Python repositories. On the positive side, while Python benchmarks are struggling with saturation and overfitting [20], this problem is less severe with Java benchmarks, making them a more genuine yardstick for innovation. Benchmarks and solutions for Java reproduction testing are few. Libro [17] is an early reproduction test generator for Java but has been outperformed by later approaches [23]. BRT Agent [9] was evaluated on 80 bugs across 7 languages including some Java bugs, but that evaluation data is small and not publicly available. Both Libro and BRT Agent have an important architectural omission: they lack fault localization and thus only solve a simplified variant of the reproduction test generation problem. The OmniCode [25] benchmark includes Java instances and a test-generation task, but does not evaluate whether tests reproduce issues on existing (prepatch) code. Furthermore, being public, the datasets for Libro and OmniCode are susceptible to contamination, raising the question how evaluated solutions perform on industry datasets. To address these gaps, this paper evaluates reproduction test generation for Java both on a public dataset (to facilitate comparisons) and on a proprietary dataset (to avoid contamination and explore industry relevance). Our public dataset has 250 instances, large enough for meaningful results; we have released it as a benchmark. Our proprietary dataset has 150 instances; while we cannot make it publicly available, we share insights useful for the research community. Furthermore, we report our experiences adapting a state-of-the-art Python reproduction test generator [3] to Java, which involved various changes to improve its effectiveness. By doing so, we end up with the (to our knowledge) strongest solution for this task to-date.
Conference’17, July 2017, Washington, DC, USA
Test generator (e.g. Otter for Java) 𝑥 = 𝑑 generate test
Overall, we hope that by introducing TDD-Bench-Java, this paper spurs progress towards better and more reliable AI-based coding assistance for Java.
2
Problem Statement and Background
This section discusses the problem statement and briefly introduces prior work we build upon: e-Otter [3], SWT-bench [23], and TDDBench-Verified [4]. These works focus on Python rather than Java.
2.1
Problem Statement
This paper tackles the problem of generating reproduction tests directly from issue descriptions. Figure 1 illustrates the evaluation harness. The input 𝑥 to repository-level reproduction test generation comprises the issue description 𝑑 issue and the original code 𝑐 old on which the issue was reported. The issue description is typically written in natural language and may include code snippets or stack traces. A reproduction test generator, such as Otter for Java, must generate as output 𝑦 a test 𝑡 gen that the benchmark driver then evaluates. Crucially, the driver does not reveal the new code 𝑐 new that resolves the issue to the test generator, reflecting real-world scenarios where fixes are not yet available. The test generator must work without access to 𝑐 new , which is only used in the evaluation harness. The evaluation harness executes the generated test 𝑡 gen twice, once each on the code before (𝑐 old ) and after (𝑐 new ) issue resolution. To get credit for generating a good reproduction test, that test must be fail-to-pass. The TDD-Bench-Java benchmark contains 250 instances, each of which has its own issue description 𝑑 issue and code 𝑐 old, 𝑐 new . Repeating the above procedure for all of them enables the benchmark driver to calculate a fail-to-pass rate. Formally, the goal is to design a function genTests that takes 𝑥 = ⟨𝑑 issue, 𝑐 old ⟩ as input and outputs the generated test 𝑦 = 𝑡 gen . 1 https://github.com/IBM/TDD-Bench-Verified/tree/main/TDD-Bench-Java
,𝑐
Evaluation harness
Execution environment
𝑦 = 𝒕𝐠𝐞𝐧 𝑐
,𝑐
, 𝒕𝐠𝐞𝐧 𝑐
𝑟 𝑟
, 𝒕𝐠𝐞𝐧 execute test
𝑟 𝑐
, 𝒕𝐠𝐞𝐧 𝑟
execute test
= fail and = pass
tally result
(1) TDD-Bench-Java1 is a new benchmark for generating reproduction tests given an issue and a source code repository. We have already open-sourced the benchmark. (2) e-Otter++ for Java, a strong reproduction-test generator, adapted from earlier work on the same task for Python. e-Otter++ achieves 43.6% and 46.4% fail-to-pass rate with Claude-Sonnet-4.5 and GPT-5.2, respectively. (3) Empirical insights from experiments on both the public TDDBench-Java and a proprietary industry dataset.
TDD-Bench-Java benchmark driver
evaluate test
The starting point for our Java reproduction test generator is e-Otter++, an LLM-based workflow [3]. Unlike recent coding agents that give the AI unfettered access to terminal commands [28, 29], e-Otter++ carefully restricts available actions. This prevents dangerous side-effects that would be too risky for an industry setting [14, 22]. Our solution starts from localizing which code to test and where to put the new test, followed by generating a test, and using a feedback-driven refinement loop to improve it. Furthermore, we use inference scaling to generate multiple reproduction tests candidates, and finally select one test to submit. Unlike most work on inference scaling that increases candidate diversity using the temperature hyperparameter [13, 30], our approach uses heterogeneous prompting [3] instead. This paper makes the following contributions:
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel
Figure 1: Evaluation harness for bug reproduction test. First, run the test generator to obtain a test 𝑡 gen . Second, execute test 𝑡 gen on the old code 𝑐 old to obtain an execution result 𝑟 old . Third, execute test 𝑡 gen on the new code 𝑐 new to obtain an execution result 𝑟 new . Finally, tally whether the test was failto-pass, which means 𝑟 old is fail and 𝑟 new is pass.
2.2
Reproduction Test Generation Benchmarks
SWT-bench [23] and TDD-Bench-Verified [4] are both benchmarks designed to evaluate the ability of models to generate reproduction tests for software engineering issues, and both are derived from SWE-bench datasets. They assess whether a generated test correctly captures an issue by checking the fail-to-pass (𝐹→𝑃) behavior—i.e., the test should fail on the original code and pass after applying the developer’s fix. These benchmarks provide standardized evaluation harnesses consistent with Figure 1 that simulate real-world development scenarios where only the issue description 𝑑 issue and pre-fix code 𝑐 old are available to the test generator. While both benchmarks share the same goal and evaluation principle, they differ slightly in implementation details and dataset construction. TDD-Bench-Verified focuses on high-quality, filtered instances where contributing tests are explicitly validated, whereas SWT-bench includes broader settings and variations such as Lite and Verified subsets, and may execute entire test files instead of only contributing tests. Despite these differences, prior work shows that model performance is generally consistent across both benchmarks, making them complementary tools for evaluating reproduction test generation systems. Both benchmarks support evaluation for Python, not Java.
2.3
Reproduction Test Generation for Python
Reproduction test generation plays a critical role in resolving software engineering issues. Prior approaches either rely on zero-shot prompting with LLMs or incorporate existing tests as contextual input [4, 23]. However, recent studies have demonstrated that zeroshot methods underperform compared to approaches (e.g., Otter [2], e-Otter [3], AEGIS [28], AssertFlip [18]) that leverage relevant portions of the codebase. Our prior work, Otter, is an LLM-based system that combines localization, a self-reflective action planner, and test
Reproduction Test Generation for Java SWE Issues
Conference’17, July 2017, Washington, DC, USA
Table 1: TDD-Bench-Java benchmark dataset attributes.
Average Word Count in Issue Description
Fraction of Dataset (in %)
trinodb/trino fasterxml/jackson-databind apache/rocketmq apache/dubbo google/gson fasterxml/jackson-core alibaba/fastjson2 apolloconfig/apollo mockito/mockito elastic/logstash fasterxml/jackson-dataformat-xml googlecontainertools/jib google/guava
44 42 41 36 34 18 6 6 6 5 5 5 2
17.6 16.8 16.4 14.4 13.6 7.2 2.4 2.4 2.4 2.0 2.0 2.0 0.8
Maven Maven Maven Maven Maven Maven Maven Maven Gradle Gradle Maven Gradle Maven
101.2 39.6 61.1 59.8 67.5 38.7 12.8 117.8 101.8 64.0 105.0 21.4 59.0
82.8 63.2 82.8 159.8 108.6 68.0 31.8 62.8 99.5 39.8 65.8 26.2 92.5
98.5 198.2 190.6 181.2 198.0 115.7 89.2 113.0 160.7 294.8 200.8 235.0 94.5
Average
19.5
7.8
NA
86.3
87.2
199.4
generation. It identifies relevant files and functions, plans actions (read/write/modify), and generates tests while using rule-based validation and repair. e-Otter is an improved version of Otter, where we apply execution feedback on the old codebase (𝑐 old ) along with an LLM-based critic to refine the tests. We also increase test candidate diversity through issue morphing, where an LLM rewrites the issue description. Experiments show that both execution feedback and issue morphing significantly improve performance. Finally, we select the best test based on execution feedback from candidate test patches, which we refer to as e-Otter++. In this paper, we modify our pipeline to make it work for Java, and we discuss the approach in Section 4.
3
Average # of Line Deleted and Added on Code On Tests
# of Instances
Project
TDD-Bench Java Benchmark
Resolving an issue involves modifying code to create a patch that fixes the problem and is commonly used to evaluate LLM-based coding agents. However, existing benchmarks such as SWE-bench [15] are largely limited to Python, making them insufficient for assessing model performance across diverse software ecosystems. To address this limitation, recent work introduces two complementary multi-language benchmarks: Multi-SWE-bench [32] and SWEPolyBench [24]. Multi-SWE-bench expands evaluation to seven programming languages (Java, TypeScript, JavaScript, Go, Rust, C, and C++), comprising 1,632 carefully curated and human-validated instances. In parallel, SWE-PolyBench provides 2,110 instances across four languages (Python, Java, JavaScript, and TypeScript), covering a broader range of tasks, including bug fixes, feature additions, and refactoring. It also introduces novel evaluation metrics based on syntax tree analysis to better capture code understanding and localization capabilities. Both are repository-level, executionbased benchmark. Together, they enable comprehensive evaluation of state-of-the-art coding agents across languages, task types, and
Build Tools
complexity levels. However, neither benchmark is designed to evaluate the quality of reproduction tests. To address this gap, we use Java samples from both benchmarks and propose a new benchmark for evaluating reproduction tests.
3.1
Data Filtering
There are 128 and 165 Java samples in Multi-SWE-bench and SWEPolyBench, respectively. We start with these samples and attempt to reproduce the issues using golden developer-written tests. We find that, for some instances, the tests pass on both 𝑐 old and 𝑐 new , and for others, the contributing tests (updated or modified in the golden test patch) do not exhibit fail-to-pass behavior. There are two instances that appear in both benchmarks. After filtering for failto-pass behavior and deduplication, we end up with 250 instances. Table 1 presents the attributes of the benchmark instances. In SWEPolyBench, all projects use Maven as the build tool, whereas in Multi-SWE-bench, three projects use Gradle.
3.2
Evaluation Harness and Metric
Figure 1 illustrates how the evaluation harness works in this setup. We first execute the reproduction test on 𝑐 old , where the test should fail to reproduce the issue before the fix. After applying the developerwritten code patch, we execute the test again; this time, it should pass, confirming that the issue has been resolved. Both SWE-PolyBench and Multi-SWE-bench provide pre-built Docker environments for test execution. However, they are configured to run the entire test suite rather than a specific test. We primarily modify the SWEPolyBench framework to support evaluation of reproduction tests. We design our harness to be as flexible as possible, allowing multiple ways to execute reproduction tests. Users can provide a classfunction name pair or just a class name. The harness can also automatically identify the test file and function from test patch if none are specified.
Conference’17, July 2017, Washington, DC, USA
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel
input x
output y old code 𝑐
issue description 𝑑
6 candidate tests
e-Otter++ test 𝑡
Select final test
for each of 6 morphs m: Reformulate with morph 𝑚
morphed issue 𝑑
Localize files from names
Pick package
Localize functions from names
Propose imports
Localize functions from bodies
Choose file path
Localizer
Contextualizer Initial test generator
Generate initial test
Legend: data
LLM call
e-Otter test 𝑡 for at most 10 iterations: test to refine 𝑡 Execute test 𝑡 on old code cold
Refiner
Call read actions
Criticize and repair
Otter test 𝑡
Rule-based
Workflow
Figure 2: Overview of test generation pipeline. We use the fail-to-pass rate as the primary evaluation metric, following prior work [4, 23]. The fail-to-pass rate is the percentage of generated tests that fail on the original code (𝑐 old ) and pass after applying the fix (𝑐 new ), indicating successful reproduction and validation of the issue. SWT-Bench also hosts a leaderboard2 for Python reproduction test generators, where ranking is based on the fail-to-pass rate. One key difference between Java and Python is that Java requires compilation and build steps, unlike Python. If a Java test attempts to access a class or function that has not yet been defined or will be introduced in the code patch, it results in a build error instead of a test failure. In our benchmark, we treat such build errors as equivalent to test failures, whereas in Python, similar issues would typically result in runtime errors.
4
Methodology
Figure 2 illustrates our reproduction test generation pipeline. The pipeline uses an execution-free test generation approach (Otter) to generate the initial test, then uses execution feedback to refine the test (e-Otter), and finally uses inference scaling to further increase test quality (e-Otter++). In other words, the Otter and e-Otter tests can be viewed as ablations of the final e-Otter++ test; they are named after and inspired by our prior work with Python [2, 3]. Due to the properties of the TDD-Bench-Java dataset, model advances, and insights from our proprietary data, we made some modifications to the Python pipeline, discussed below.
4.1
Test Generator
This section discusses the core of our test generation pipeline, which has four components: localizer, contextualizer, initial test generator, and refiner. 2 https://swtbench.com/?results=verified
Localizer: To generate a fail-to-pass reproduction test, it helps to first localize focal functions, which are the functions under test that will likely be updated as part of fixing the issue. In addition, it also helps to localize related regression tests functions, which usually do not reproduce the issue but cover the focal functions. The localizer makes three LLM calls. The first call presents the issue description and the names of all Java files in the repository to the LLM, and the LLM selects 50 relevant files from the list (prioritizing recall over precision). In most cases, the issue description contains enough information to help LLMs select the files where the changes will be made. The second call presents those file names along with the names of functions defined in them and asks the LLM to choose the relevant functions from the list. This yields a set of functionfile pairs that can be used as context for test generation. So far, the LLM has not seen any function bodies to the LLM, making its selections based solely on the file and function names. The final call presents the function body and asks the LLM to filter out unnecessary functions. This final filtered set of file-function pairs serves as input for the contextualizer. Our original Otter work for Python [2] had two separate localizations for focals and tests, whereas for Java, we merge them. This change was motivated by dataset characteristics. In TDD-BenchVerified for Python, new test files were written for less than 1% of instances. Instead, tests were primarily written or modified in existing test files. Localizing tests is very important in that setup, and we had a separate test localizer to identify relevant tests. In Java, about 15% of instances involve tests written in new files in open-source projects. In our proprietary data, we have observed that the majority of tests are written in new files, and for some instances, there are fewer than five existing test files. Reading irrelevant tests may hurt the performance of the test generator. By providing a single merged localization step, we effectively leave it up to the LLM to skip test localization if it determines that no existing tests are relevant. Contextualizer: As discussed earlier, in Java, new test files are frequently written, and relevant test files are missing for many instances. Unlike Python, writing a test in a file with an incompatible package can cause build errors. To address this challenge and avoid irrelevant test context, we decide to write one test file for each instance. A Java test file has different components (e.g., package name and relevant imports). Therefore, the contextualizer starts with the localized functions from the localizer step and tries to construct these components before writing the full test file. For the package name, we gather all existing package statements from the repository and then ask the LLM to choose from them, given the issue description and localized functions. We frame this as a selection problem to avoid hallucination by the model. However, the model will have the opportunity to fix an incorrect package based on execution feedback in the refiner step. Similar to the package name, we also determine the imports necessary for the test file. We collect all imports from the initially selected 50 files and ask the LLM to choose the required imports from them to provide better context for writing the test. Another challenge in writing a new test file is determining its location in the repository. To address this, we collect all paths to existing directories under the test directory and, similar to package
Reproduction Test Generation for Java SWE Issues
selection, ask the LLM to choose the appropriate location for the new test file. All in all, the contextualizer makes three LLM calls, and each call uses the output from the previous call as additional context. Initial Test Generator: The initial test generator generates the 𝑚 as a single new function within a new class in a new Otter test 𝑡 ot file. It does so via a single LLM call whose input is the issue description and the information gathered by the preceding localizer and contextualizer steps, prompted to generate a fail-to-pass reproduction test. As discussed above, generating a new file is easier for the LLM and consistent with the characteristics of our proprietary dataset. We ask the LLM to generate a test class name that ends with “OtterTest” to avoid name conflicts and clearly mark which tests were added by the LLM. 𝑚 from the Refiner: The refiner starts by setting the Otter test 𝑡 ot 𝑚 previous step as the initial test to refine 𝑡 ref . It iteratively refines it using execution feedback. To get that execution feedback, it 𝑚 on the current code 𝑐 executes test 𝑡 ref old in the repository. Next, it presents the LLMs with the issue description and execution logs on 𝑐 old , and asks whether the test is failing for the right reason: the problem described in the issue description. If the LLM-based critic determines that the test is failing for the right reason, the loop immediately terminates. Otherwise, the LLM rewrites the test, notes the changes made, and requests any function or class information the model needs; this all happens in the same single LLM call that also implements the critic. The refiner gathers this additional information for future attempts when the critic determines that the test is still not failing for the right reason. The refiner loop iterates at most ten times if the critic is not satisfied. We cumulatively collect the changes and present them to the LLM to prevent it from generating same buggy code. For better context, we try to retrieve the functions and classes proposed by the LLM. In the class information, we include the package name, imports, and caller–callee functions, which may help the LLM per𝑚, form better. After the loop is done, the refiner returns the final 𝑡 ref 𝑚 which we refer to as the e-Otter test 𝑡 eot .
We share all the prompts used for test generation in the supplementary material3 .
4.2
Heterogeneous prompting
Inference scaling is beneficial for reproduction test generation [2, 3]. Our prior work demonstrated that mask- and morph-based inference scaling perform better than traditional temperature-based scaling [3]. It also demonstrated that morphing yielded relatively better results than masking. However, with newer advanced models, we did not observe any benefit from masking on the Java dataset. After some initial experiments, we decided to move forward with morphing only. In morph-based inference scaling, the LLM rewrites the issue description to increase the diversity of test candidates. Our previous work contains an extensive discussion of morphing including relevant prompts [3]. For completeness, we briefly discuss them below. As shown in Figure 2, each morph 𝑚 yields one rewritten issue description 𝑑𝑚 issue . 3 https://drive.google.com/drive/folders/1i6Tp2AvlN7x7lWcGw_npeZIZvGGWleWl?
usp=sharing
Conference’17, July 2017, Washington, DC, USA
standard: This morphing technique asks the LLM to rewrite the issue description in a standard format. A well-written issue description includes several components, such as a title, description, steps to reproduce, and expected behavior. These components may help improve the generation of reproduction tests. simple: Some issue descriptions contain code artifacts and use project-specific jargon, which can make it difficult to understand the problem. This morph asks the LLM to simplify the issue description so that it becomes easier for the model to perform better. dropCode: Some issue descriptions contain misleading or irrelevant code snippets that can confuse the model. For example, an issue may include code using a library that is not actually part of the project, leading the model to generate incorrect tests. Removing such code (dropCode morph) helps the model focus on the core problem and avoid making invalid assumptions. initTest: In the initTest morph, the LLM is asked to generate an initial fail-to-pass test and incorporate it into the issue description without additional context. This provides a starting point for the test generation process, allowing subsequent stages to refine and improve the test. initPatch: In the initPatch morph, the LLM is prompted to propose an initial solution or code patch for the issue and include it in the description. This helps guide the model by providing insight into the expected behavior or potential fix, which can improve the quality of generated tests. default: Our test generation pipeline and experiments also includes one setting with the original issue description, without any LLMbased issue rewriting.
4.3
Test Selector
In real development settings, developers typically prefer a single test. Therefore, after inference scaling, we need to select the best test from a pool of candidates. To address this, the test selector 𝑚 for the six morphs 𝑚, prompts an LLM with six test patches 𝑡 eot along with their execution logs on 𝑐 old . The LLM then selects the best test from the pool based on these logs. We instruct the LLM to critically analyze and compare the tests before choosing one. The prompt is provided in the supplementary material. We refer to this single final generated test 𝑡 gen as the e-Otter++ test.
5
Results on TDD-Bench-Java
This section reports results on our new public benchmark TDDBench-Java (see Section 3) using our approach e-Otter++ (see Section 4). It also drills down into the impact of key components of our approach, execution feedback and heterogeneous prompting.
5.1
Experimental Setup
We used two models, Claude-Sonnet-4.5 and GPT-5.2, for our experiments. The number of LLM calls is evident from Figure 2: seven to 𝑚 plus up to 10 for refinement to generate generate each initial test 𝑡 ot 𝑚 each test 𝑡 eot . Inference scaling adds one more call per non-default morph, and there is a final selection call to choose 𝑡 gen .
Conference’17, July 2017, Washington, DC, USA
50
Otter e-Otter e-Otter++
Fail-to-Pass Rate (%)
46.4%
43.6% 40.0%
40
30
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel
Table 2: Performance of Otter and e-Otter with different morphs and models.
39.6%
Claude-Sonnet-4.5
default standard simple dropCode initTest initPatch
76 64 71 68 71 80
30.4 25.6 28.4 27.2 28.4 32.0
100 99 97 99 99 101
40.0 39.6 38.8 39.6 39.6 40.4
31.6 54.7 36.6 45.6 39.4 26.2
GPT-5.2
default standard simple dropCode initTest initPatch
65 63 68 56 66 67
26.0 25.2 27.2 22.4 26.4 26.8
99 102 98 92 95 104
39.6 40.8 39.2 36.8 38.0 41.6
52.3 61.9 44.1 64.3 43.9 55.2
30.4% 26.0%
10
Claude
GPT-5.2
Figure 3: Performance of Otter, e-Otter, and e-Otter++ on TDD-Bench Java.
5.2
Table 3: Impact of heterogeneous prompts. The ‘All’ rows evaluate 𝐹→𝑃 @ 6, whereas rows of the form ‘All - 𝑑’ evaluate 𝐹→𝑃 @ 5, since they ablate morph 𝑑 from the candidate pool.
Model
Prompt
𝐹→𝑃 @ N
in %
Change in %
Claude-Sonnet-4.5
All All - default All - standard All - simple All - dropCode All - initTest All - initPatch
137 136 134 136 133 135 136
54.8 54.4 53.6 54.4 53.2 54.0 54.4
NA -0.7 -2.2 -0.7 -2.9 -1.5 -0.7
GPT-5.2
All All - default All - standard All - simple All - dropCode All - initTest All - initPatch
137 133 133 137 132 135 134
54.8 53.2 53.2 54.8 52.8 54.0 53.6
NA -2.9 -2.9 0.0 -3.6 -1.5 -2.2
Finding 1. We observe a 43.6% and 46.4% fail-to-pass rate with Claude-Sonnet-4.5 and GPT-5.2, respectively, using e-Otter++.
Impact of Refiner
Table 2 presents the results for both Otter and e-Otter across all six test candidates based on different morphs 𝑚. The performance of e-Otter is similar for both Claude-Sonnet-4.5 and GPT-5.2. The fail-to-pass rate ranges from 38.8% to 40.4% for Claude, while for GPT-5.2 it ranges from 36.8% to 41.6%. However, the relative performance improvement for GPT-5.2 (43.9%–64.3%) is higher than that of the Claude model (26.2%–54.7%) across all candidates. We also performed McNemar’s test to assess statistical significance and observed that the improvement is significant (𝑝 < 0.01) for all model–candidate pairs. Finding 2. Using refinement, the relative improvement from Otter to e-Otter ranges from 43.9%–64.3% for GPT-5.2 and 26.2%–54.7% for Claude across all candidates.
5.4
Change in %
Performance of Test Generators
Figure 3 presents the performance of Otter, e-Otter, and e-Otter++ using two models: Claude-Sonnet-4.5 and GPT-5.2. We observe that Claude-Sonnet-4.5 performs better than GPT-5.2 with Otter and e-Otter, while GPT-5.2 outperforms Claude with e-Otter++. The improvement from refinement is higher for GPT-5.2 (13.6 points vs. 9.4 points). To measure statistical significance, we perform McNemar’s test [21]. McNemar’s test is applicable to this setup because it evaluates statistical significance on paired binary outcomes (failto-pass vs. not), which aligns with comparing model performance on the same set of instances. We observe statistical significance (𝑝 < 0.01) for both models when comparing Otter and e-Otter. For e-Otter and e-Otter++, we only achieve statistical significance for GPT-5.2, but not for the Claude model.
5.3
e-Otter 𝐹→𝑃 in %
Prompt
20
0
Otter 𝐹→𝑃 in %
Model
Impact of Heterogeneous Prompting
In our prior work [3], we demonstrated that heterogeneous prompting helps increase test candidate diversity for Python. Does this hold for Java as well? Table 2 shows that with a single test variant, we
can achieve at most 101 and 104 fail-to-pass cases with Claude-4.5Sonnet and GPT-5.2, respectively. However, Table 3 shows that the fail-to-pass @N (N=6) reaches 54.8% (137) for both models. This indicates that heterogeneous prompting increases the fail-to-pass rate by more than 32%. Fail-to-pass @N is the proportion of instances for which at least one out of N generated test candidates fails on the old code and passes on the new code. Table 3 also presents ablation results showing the contribution of each test variant. We observe that each variant generates 0–5 fail-to-pass tests that are not resolved by other candidates. Although the unique contribution of each variant is relatively small, it still benefits the test selector by providing more fail-to-pass tests to choose from.
Reproduction Test Generation for Java SWE Issues
Finding 3. Heterogeneous prompting increases the fail-to-pass rate @N by more than 32% for both Claude-Sonnet-4.5 and GPT5.2 models.
Conference’17, July 2017, Washington, DC, USA
issue description 𝑑 issue 1 2 3 4
5.5
An Illustrative Example
Figure 4 shows an example of an e-Otter++ generated test for “forfasterxml__jackson-core-183”. We also present the developerwritten test. Both tests reproduce the issue correctly but differ significantly in nature. The issue asks for consistent behavior. The developer test ensures this by comparing lengths, whereas the Ottergenerated test expects the same object instance, which is too strict compared to the developer-written test. Although the two tests are qualitatively different, they both successfully reproduce the bug. A deeper qualitative comparison between model-generated tests and developer-written tests is beyond the scope of this paper and is left for future work.
6
Reproduction Test Generation on Proprietary Projects
One of the challenges of applying LLMs to open-source data is model contamination. The model may have already seen the data, which may impact the overall findings. Some benchmarks address this problem by filtering issues that appear after a certain model cutoff date [6]. However, even for those benchmarks, the evaluation may not be bias-free compared to proprietary data, since the model may have seen prior versions of the open-source project and acquired background knowledge of the repository that can help it perform better. At the same time, businesses seek to benefit from LLMs on their own proprietary code. To this end, this section shares our findings on industry code.
5
Inconsistent TextBuffer#getTextBuffer behavior Hi, I'm using 2.4.2. While I'm working on CBORParser, I noticed that CBORParser#getTextCharacters() returns sometimes `null` sometimes `[]` (empty array) when it's parsing empty string `""`.
6 7 8
While debugging, I noticed that TextBuffer#getTextBuffer behaves inconsistently.
9 10 11 12 13 14 15 16
``` TextBuffer buffer = new TextBuffer(new BufferRecycler()); buffer.resetWithEmpty(); buffer.getTextBuffer(); // returns null buffer.contentsAsString(); // returns empty string "" buffer.getTextBuffer(); // returns empty array [] ```
17 18 19
I think getTextBuffer should return the same value. Not sure which (`null` or `[]`) is expected though.
Developer Written Test 1 2 3 4 5 6 7
public void testEmpty() { TextBuffer tb = new TextBuffer(new BufferRecycler()); tb.resetWithEmpty(); assertTrue(tb.getTextBuffer().length == 0); tb.contentsAsString(); assertTrue(tb.getTextBuffer().length == 0); }
Otter Generated Test 1
package com.fasterxml.jackson.core.util;
2 3 4 5 6
import com.fasterxml.jackson.core.util.BufferRecycler; import com.fasterxml.jackson.core.util.TextBuffer; import static org.junit.Assert.*; import org.junit.Test;
7
6.1
Characteristics of Proprietary Data
We collected 150 issues from an actively-used and important IBMinternal Java project and applied our proposed reproduction test generator. In our public benchmark, all instances include test patches; however, in our proprietary dataset, only 28% of instances have developer-written golden code patches. This suggests that developers may not always write reproduction tests. From our initial observations, issue descriptions in our proprietary data are much shorter than those in our public benchmark and often lack detailed information. Figure 6 shows that the average word count for issue descriptions 𝑑 issue in the public benchmark is 163.7, whereas for our proprietary dataset it is 94.1. Additionally, the proprietary issue descriptions rarely contain code snippets or stack traces, making them more difficult to interpret compared to issues in TDD-Bench-Java. Comparing code patches from public vs. proprietary data: We compare open-source and proprietary code patches to estimate the difficulty of the issues. Figure 5a shows the number of lines changed in code patches; the average number of lines changed is much higher for proprietary code patches (362.2 vs. 54.8). We also compare the number of diff-hunks in Figure 5b, total number of files (modified + new) in Figure 5c, and newly added files in Figure 5d. On average, 2.6 new files are added in our proprietary data, whereas there are hardly any new files added in public TDD-Bench-Java instances. New files imply that new classes and functions are introduced, which are not
8
public class TextBufferConsistencyOtterTest {
9 10 11 12 13
@Test public void testGetTextBufferConsistency() { TextBuffer buffer = new TextBuffer(new BufferRecycler()); buffer.resetWithEmpty();
14 15 16
// First call to getTextBuffer char[] firstResult = buffer.getTextBuffer();
17 18 19 20
// Call contentsAsString which may modify internal state String str = buffer.contentsAsString(); assertEquals("", str);
21 22 23 24
// Second call to getTextBuffer should return the same // as first call char[] secondResult = buffer.getTextBuffer();
25 26 27 28 29 30 31 32 33 34 35
// Both should be non-null and consistent assertNotNull("First␣getTextBuffer() ␣␣␣␣␣␣␣␣␣␣␣␣␣␣call␣should␣not␣return␣null", firstResult); assertNotNull("Second␣getTextBuffer() ␣␣␣␣␣␣␣␣␣␣␣␣␣␣call␣should␣not␣return␣null", secondResult); assertSame("getTextBuffer()␣should␣return␣the ␣␣␣␣␣␣␣␣␣␣␣␣␣␣same␣reference␣on␣consecutive␣calls", firstResult, secondResult); } }
Figure 4: e-Otter++ generated test forfasterxml__jacksoncore-183.
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel
25
Mean: 3.1
20
50
40
00
10
TDD-Bench-Java
Proprietary
(b) Number of diff-hunks
5 0
Mean: 2.6
15
20
50
Proprietary
0
TDD-Bench-Java
(a) Number of changed lines
Added New Files
50
Mean: 21.4
35 30
60
00
000 500 000 0
500
Mean: 54.8
Mean: 10.0
80
Mean: 8.9
Total Files (Changed + New)
Number of1 Code Hunks 1 2 2 3
500
Number of1 Lines2 Changed 1 2 3
000
Mean: 362.2
00
Conference’17, July 2017, Washington, DC, USA
TDD-Bench-Java
Proprietary
0
(c) Total files (modified + new)
Mean: 0.1
TDD-Bench-Java
Proprietary
(d) Added new files
Word Count in Issue Descriptions
Figure 5: Comparing the code patches in open-source and proprietary data.
1200
Mean: 167.3
1000 800 600
Mean: 94.1
400 200
the issue descriptions. Similar to the bug/feature categorization, we present the LLM with the issue description and corresponding code patch and ask whether the patch addresses the issue. Table 4 shows that, for TDD-Bench-Java, the code patches align with the issue descriptions in 98.4% of instances, whereas for our proprietary data, this alignment is observed in 81.3% of cases. This is in part due to the quality filters of public benchmarks and further highlights the increased difficulty and challenges associated with proprietary data. Finding 4. Our proprietary code patches are significantly larger than those in our public TDD-Bench-Java, but the issue descriptions themselves are shorter and under-specified.
0 TDD-Bench-Java
Proprietary
6.2 Figure 6: Comparing the word count of issue descriptions in public TDD-Bench-Java vs. proprietary data.
available in the current repository 𝑐 old . Such newly-added classes or functions are difficult to anticipate for models, making reproduction test generation more challenging. However, this raises the question of why code patches are significantly larger in proprietary projects compared to open-source projects. To answer this, we attempt to categorize the instances below. Issue types: bug fix or feature request: Developers write issues primarily for two reasons: (i) fixing a bug in the repository and (ii) requesting a new feature. Adding a new feature typically involves more code changes compared to fixing bugs. LLMs have been shown to be effective at annotating software engineering artifacts [1, 27]. We present the LLM with the issue description and corresponding code patch and ask it to categorize the issue as either a bug fix or a feature request. Table 4 shows that 70.8% of issue descriptions in TDD-Bench-Java are related to bug fixing, whereas in our proprietary data, only 16.0% are bug fixes. In contrast, 29.2% of issues in TDD-Bench-Java are feature requests, which is much lower than in our proprietary project (84%). Since most issue descriptions in our proprietary data are feature requests, this helps explain the larger code patches. Alignment between issue descriptions and code patches: We are also interested in examining whether the code patches address
Performance of e-Otter on Proprietary Data
We ran e-Otter on our proprietary data and were able to generate fail-to-pass tests for only 4% (6 out of 150) of the instances (Figure 7), which is significantly lower than what we observed on TDD-Bench-Java. As discussed in Section 6.1, our proprietary data often introduces new files and classes for each issue, making it difficult for the model to predict the correct class or function names, leading to syntactical errors. Even with execution feedback, we were unable to generate a sufficient number of fail-to-pass tests. Due to organizational policy, we used Claude-Sonnet-4.5 for these experiments. One workaround for this problem is to add identifier hints [11] to the issue description, including new class and function names. In real development settings, developers can provide such hints. In this paper, we present the code patch to the LLM and ask it to generate these hints. We then reran e-Otter and were able to fix 20% of the instances (30 out of 150). Since the default issue descriptions are relatively short and performance is highly dependent on the hints, morphing primarily affects the hints rather than the issue description. Therefore, we did not apply inference scaling in this setup. However, we believe that improving the quality of hints can significantly enhance performance. Finding 5. With the default issue description, e-Otter achieved only 4% fail-to-pass rate on proprietary projects. However, adding hints with newly introduced class names and function signatures increased the performance to 20%.
Reproduction Test Generation for Java SWE Issues
Conference’17, July 2017, Washington, DC, USA
Table 4: Performance of e-Otter on open-source and proprietary data across different categories
Type
Bug Feature
Aligned Misaligned
Source
Total Count in %
Fail-to-pass Test Count in %
Open-source Proprietary Open-source Proprietary
177 24 73 126
70.8 16.0 29.2 84.0
78 11 22 19
44.1 45.8 30.1 15.1
Open-source Proprietary Open-source Proprietary
246 122 4 28
98.4 81.3 1.6 18.7
100 26 0 4
40.7 21.3 0.0 14.3
Table 5: Impact of Iteration & Prior Change Model
Component
𝐹→𝑃
in %
Change in %
Claude-Sonnet-4.5
e-Otter e-Otter - iteration e-Otter - change
100 85 88
40.0 34.0 35.2
NA -15.0 -12.0
GPT-5.2
e-Otter e-Otter - iteration e-Otter - change
99 84 90
39.6 33.6 36.0
NA -15.2 -9.1
Table 6: Impact of Test Selector
Model
Group
# of Sample
Claude-Sonnet-4.5
All (6) 𝐹→𝑃 None (0) 𝐹→𝑃 Some (1-5) 𝐹→𝑃
59 113 78
All (6) 𝐹→𝑃 None (0) 𝐹→𝑃 Some (1-5) 𝐹→𝑃
57 113 80
50
Fail-to-Pass Rate (%)
40
40.0% (100/250)
30
20.0% (30/150)
20
GPT-5.2
Selector’s Success Count in % NA NA 50
64.1 NA NA
59
73.8
10
4.0% (6/150) 0
-Java
ench TDD-B
ts)
Propri
/o hin etary (w
Benchmark
/ hints) etary (w
Propri
Figure 7: e-Otter performance on Open- and Closed-sourced Projects using Claude-Sonnet-4.5 model.
6.3
Lession Learned and Future Direction
When moving from open-source to proprietary projects, we encountered new challenges that can guide the design of better approaches in the future. First, the specifications or issue descriptions in proprietary projects are often not well written. Including hints can help in this scenario; however, several new concerns arise, such as how to generate better hints or improve the specifications. We can consider designing automated processes or incorporating humanin-the-loop approaches. Improving hints or specifications is an important problem, which we leave for future research. Table 4 shows how the performance of e-Otter changes between our public TDD-Bench-Java and our proprietary dataset across different categories of issue descriptions. Interestingly, for bug fixes, performance differs little between our public and private datasets (with hints): for both, the fail-to-pass rate is around 45%. However, we should interpret these results with caution, as the number of bug-fixing issues is relatively small in our proprietary dataset. As expected, the fail-to-pass rate is higher for aligned samples. Finding 6. Improving the issue description or specification using automated approaches or human-in-the-loop methods can be a promising direction for future work.
7 Discussion and Further Ablation 7.1 Impact of Iteration and Prior Change The refiner iteratively uses execution feedback and feeds the changes made in prior attempts to repair the test. Table 5 shows that if we stop after the first execution feedback and do not try again, the performance drops by 15.0% and 15.2% for the Claude-Sonnet-4.7 and GPT-5.2 models. Similarly, if we do not use the changes to prevent the model’s repetitive errors, the performance drops by 12.0% and 9.1%. Therefore, both iteration and the summary of changes from prior attempts help improve performance.
7.2
Performance of Test Selector
The test selector calls an LLM to choose the best candidate from the pool of test candidates. We divide the instances into three groups: (i) all six tests go from fail-to-pass, (ii) none of the tests go from fail-topass, and (iii) at least one test goes from fail-to-pass (but not all six). In the first two groups, it does not matter what the selector chooses: in the first group, the judge will always find a fail-to-pass test, and in the second group, the selector will never find one. Only in the third group can the judge make a meaningful contribution. Table 6 shows the distribution of samples across these three groups and the selector’s performance. On group (iii) where some but not all tests are 𝐹→𝑃, we observe 64.1% and 73.8% accuracy for the selector for the Claude-Sonnet-4.5 and GPT-5.2 models, respectively.
7.3
Cost Estimation
In e-Otter++, we generate six test candidates and use a selector at the end to choose the best ones. It costs us approximately $2.5 and $1.5 for each instance with the Claude-Sonnet-4.5 and GPT-5.2 models, respectively. Thus, for each candidate, it costs $0.42 and
Conference’17, July 2017, Washington, DC, USA
$0.30. We can reduce or adjust the number of candidates to control overall spending.
8
Threats to Validity
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel
However, none of the above solutions cover Java. Repository-level agents and workflows that cover Java include InfCode [19], Prometheus [8], and iSWE [14]. Unfortunately, the task they solve is not reproduction test generation. There are very few solutions for Java reproduction test generation: Libro [17] is a workflow of two LLM calls, and BRT Agent [9] is an agentic solution. However, neither of these is evaluated in a setting where they perform their own localization. Thus, Otter for Java is the first complete solution for repository-level Java reproduction test generation.
In prior work, coverage has been considered as an additional evaluation metric. We observe that fail-to-pass tests tend to have higher coverage on average than non-fail-to-pass tests. However, many non-fail-to-pass tests exhibit similar coverage levels. It makes coverage an unreliable indicator of the fail-to-pass property of the test. As a result, we do not consider coverage as a primary metric for reproduction test evaluation. Additionally, while Python allows lightweight coverage measurement without modifying repositories, Java tools such as JaCoCo require bytecode instrumentation and additional configuration. Since our primary objective is to evaluate fail-to-pass behavior, we avoid incorporating JaCoCo to keep the evaluation harness lightweight, reduce complexity, and prevent potential side effects. This also ensures consistent evaluation across repositories with different build systems and configurations. One of the key concerns with this line of generative work is the contamination or memorization problem of the model [6, 20]. However, in this paper, we evaluated e-Otter++ on proprietary data, which gives us a contamination-free evaluation. From our experiments, it is evident that models struggle with unseen repositories, but we also observe challenges specific to the proprietary issues. Therefore, it is not conclusive how much model performance is influenced by model contamination versus issue difficulty. We find that resolving issues in proprietary projects is more difficult than in open-source projects. However, our findings are limited to our proprietary dataset of 150 issues and may not generalize to other proprietary projects within our organization or elsewhere.
Industry experience for Java reproduction test generation. A serious threat to validity pervading the research literature on repositorylevel software engineering is that open-source repositories may not be representative of industry code. One concern is that large language models tend to be trained on huge datasets that include all available open-source software, which means that their training data is (at least to some extent) contaminated [7]. This has given rise to benchmarks such as SWE-rebench [6], which resists contamination by releasing new instances predated by LLM training data cut-offs, and SWE-Bench Pro [12], which resists contamination by using data whose license prohibits use for LLM training. Unfortunately, neither of these benchmarks covers Java, and neither of them evaluates the reproduction test generation task. Since our new TDD-Bench-Java benchmark leverages public data, this paper additionally also reports insights based on a proprietary industry dataset. In doing so, it takes inspiration from recent papers on BRT Agent with insights on Google-internal code [9]; HULA with insights on Atlassian-internal code [26]; and AutoCodeRover with insights on SonarSource-internal code [22]. Of these, only BRT Agent focuses on reproduction test generation, and the number of Java instances involved is much smaller than in our experiments.
9
10
Related Work
Benchmarks for Java reproduction test generation. Our work on TDD-Bench-Java is inspired by recent repository-level software engineering benchmarks such as SWE-bench [15] and SWE-bench Verified [10]. Those initial benchmarks focused on the patch generation task for Python, and were soon followed by benchmarks for the repository-level reproduction test generation task, specifically, SWT-bench [23], TDD-Bench-Verified [4], and later USEBench [5]. However, none of the above benchmarks cover Java. Repositorylevel software engineering benchmarks that cover Java include Defects4J [16], SWE-PolyBench [24], and Multi-SWE-bench [32]. Unfortunately, the task in these benchmarks is not reproduction test generation. A recent benchmark, OmniCode [25], includes repository-level Java instances and a “test generation” task. However, that task does not refer to reproduction tests: it does not evaluate the test on the old pre-patch code 𝑐 old , which would be required to reproduce the issue. Thus, TDD-Bench-Java is the first repository-level Java benchmark for reproduction test generation. Solutions for Java reproduction test generation. Our work on Otter for Java is inspired by recent agents and workflows for repositorylevel software engineering tasks such as SWE-Agent [31], Agentless [30], and OpenHands [29]. Those initial solutions focused on the patch generation task for Python, and were soon followed by agents and workflows for repository-level reproduction test generation, such as Otter [2], Aegis [28], e-Otter [3], and AssertFlip [18].
Conclusion
Issue reproduction tests are tests that demonstrate the presence of an open issue before and its resolution after it gets fixed. These tests serve several core software engineering needs including requirements clarification, issue resolution, and quality assurance via continuous integration. Unfortunately, while there have been some benchmarks and solutions for reproduction test generation for Python, they have been missing for Java. This paper takes steps to remedy that situation by introducing a new public benchmark (TDD-Bench-Java), and by describing and evaluating an LLMbased workflow for it (Otter for Java). Since results on carefully curated public benchmarks do not always generalize to an industrial setting, this paper complements the experiments on public data with experiments on proprietary data. Not surprisingly, that setting is difficult and requires more attention and innovation. Data Availability Statement: The code for the reproduction test generators is proprietary. We have open-sourced TDD-Bench-Java at https://github.com/IBM/TDD-Bench-Verified/tree/main/TDDBench-Java and shared our prompts at https://drive.google.com/ drive/folders/1i6Tp2AvlN7x7lWcGw_npeZIZvGGWleWl?usp=sharing.
Reproduction Test Generation for Java SWE Issues
References [1] Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs replace manual annotation of software engineering artifacts?. In Conference on Mining Software Repositories (MSR). 526–538. [2] Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. 2025. Otter: Generating Tests from Issues to Validate SWE Patches. In International Conference on Machine Learning (ICML). [3] Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel. 2026. Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection. In International Conference on Software Engineering(ICSE). https://arxiv.org/abs/2508.06365 [4] Toufique Ahmed, Martin Hirzel, Rangeet Pan, Avraham Shinnar, and Saurabh Sinha. 2024. TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved? https://arxiv.org/abs/2412.02883 [5] Leonhard Applis, Yuntong Zhang, Shanchao Liang, Nan Jiang, Lin Tan, and Abhik Roychoudhury. 2026. Unified Software Engineering agent as AI Software Engineer. In International Conference on Software Engineering (ICSE). https: //arxiv.org/abs/2506.14683 [6] Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. 2025. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. https://arxiv. org/abs/2505.20411 [7] Zhi Chen, Wei Ma, and Lingxiao Jiang. 2025. Unveiling Pitfalls: Understanding Why AI-driven Code Agents Fail at GitHub Issue Resolution. https://arxiv.org/ abs/2503.12374 [8] Zimin Chen, Yue Pan, Siyu Lu, Jiayi Xu, Claire Le Goues, Martin Monperrus, and He Ye. 2025. Prometheus: Unified Knowledge Graphs for Issue Resolution in Multilingual Codebases. https://arxiv.org/abs/2507.19942 [9] Runxiang Cheng, Michele Tufano, Jürgen Cito, José Cambronero, Pat Rondon, Renyao Wei, Aaron Sun, and Satish Chandra. 2025. Agentic Bug Reproduction for Effective Automated Program Repair at Google. https://arxiv.org/abs/2502.01821 [10] Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. 2024. Introducing SWEbench Verified. https://openai.com/index/introducing-swe-bench-verified/ [11] Le Deng, Zhonghao Jiang, Jialun Cao, Michael Pradel, and Zhongxin Liu. 2025. NoCode-bench: A Benchmark for Evaluating Natural Language-Driven Feature Addition. https://arxiv.org/abs/2507.18130 [12] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 2025. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? https://arxiv.org/ abs/2509.16941 [13] Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Re, and Azalia Mirhoseini. 2025. CodeMonkeys: Scaling Test-Time Compute for Software Engineering. https://arxiv.org/abs/2501.14723 [14] Jatin Ganhotra, Sami Serhan, Antonio Abu Nassar, Avraham Shinnar, Ziv Nevo, and Martin Hirzel. 2026. Resolving Java Code Repository Issues with iSWE Agent. https://arxiv.org/abs/2603.11356 [15] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve RealWorld GitHub Issues?. In International Conference on Learning Representations (ICLR). [16] René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In International Symposium on Software Testing and Analysis (ISSTA). 437–440. https://doi.org/10.1145/2610384.2628055 [17] Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large Language Models are FewShot Testers: Exploring LLM-Based General Bug Reproduction. In International Conference on Software Engineering (ICSE). 2312–2323. https://doi.org/10.1109/
Conference’17, July 2017, Washington, DC, USA
ICSE48619.2023.00194 [18] Lara Khatib, Noble Saji Mathews, and Meiyappan Nagappan. 2026. AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests. In International Conference on Software Engineering (ICSE). [19] KeFan Li, Mengfei Wang, Hengzhi Zhang, Zhichao Li, Yuan Yuan, Mu Li, Xiang Gao, Hailong Sun, Chunming Hu, and Weifeng Lv. 2025. InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolution. https://arxiv.org/abs/2511.16004 [20] Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. 2025. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. https://arxiv.org/abs/2506.12286 [21] Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 2 (1947), 153–157. [22] Martin Mirchev, Ridwan Shariffdeen, Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2026. AutoCodeRover: Agentic Program Repair for SonarQube Issues. In Industry paper at Symposium on the Foundations of Software Engineering (FSE-Industry). [23] Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. 2024. SWTBench: Testing and Validating Real-World Bug-Fixes with Code Agents. In Conference on Neural Information Processing Systems (NeurIPS). [24] Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buccholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, and Laurent Callot. 2025. SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents. https://arxiv.org/abs/2504.08703 [25] Atharv Sonwane, Eng-Shen Tu, Wei-Chung Lu, Claas Beger, Carter Larsen, Debjit Dhar, Simon Alford, Rachel Chen, Ronit Pattanayak, Tuan Anh Dang, Guohao Chen, Gloria Geng, Kevin Ellis, and Saikat Dutta. 2026. OmniCode: A Benchmark for Evaluating Software Engineering Agents. https://arxiv.org/abs/2602.02262 [26] Wannita Takerngsaksiri, Jirat Pasuksmit, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Ruixiong Zhang, Fan Jiang, Jing Li, Evan Cook, Kun Chen, and Ming Wu. 2025. Human-In-The-Loop Software Development Agents. In International Conference on Software Engineering: Software Engineering in Practice track (ICSE-SEIP). 342–352. https://doi.org/10.1109/ICSE-SEIP66354.2025.00036 [27] Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can LLMs Replace Human Evaluators? An Empirical Study of LLM-asa-Judge in Software Engineering. In International Symposium on Software Testing and Analysis (ISSTA). https://doi.org/10.1145/3728963 [28] Xinchen Wang, Pengfei Gao, Xiangxin Meng, Chao Peng, Ruida Hu, Yun Lin, and Cuiyun Gao. 2024. AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptions. https://arxiv.org/abs/2411.18015 [29] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=OJd3ayDDoF [30] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-based Software Engineering Agents. In Symposium on the Foundations of Software Engineering (FSE). 801–824. https://doi.org/10.1145/3715754 [31] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-Agent: Agent-computer Interfaces Enable Automated Software Engineering. In Conference on Neural Information Processing Systems (NeurIPS). https://proceedings.neurips.cc/ paper_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-AbstractConference.html [32] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Aoyan Li, Lu Chen, Xiaojian Zhong, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Ming Ding, and Liang Xiang. 2025. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. In NeurIPS Datasets and Benchmarks Track. https://openreview.net/ forum?id=MhBZzkz4h9