JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java Khashayar Etemadi
Zhendong Su
[email protected] ETH Zurich Switzerland
[email protected] ETH Zurich Switzerland
arXiv:2606.31767v1 [cs.SE] 30 Jun 2026
Abstract Automated fixing of performance issues is gaining increasing attention. However, existing benchmarks of execution time improvement patches are fixed datasets that target Python, C++, or .NET and cannot be extended to new patches according to user-defined configurations. In this paper, we present JETO-Mine, the first configurable and reusable tool for automatically creating reproducible benchmarks of execution time improvement patches (ETIPs) in real-world Java projects. We are particularly interested in Java, as its reliance on just-in-time compilation and garbage collection makes execution measurements volatile and poses particular challenges. JETO-Mine employs a three-phase pipeline: a static analysis phase that crawls GitHub repositories and identifies ETIPs using user-defined filters and an LLM-based issue classifier, a dynamic analysis phase that wraps the identified ETIPs in Docker images for fully reproducible execution and performs statistical testing to find objective evidence of execution time improvement, and an evaluation harness that enables quantitative assessment of both generated patches and generated tests. Unlike existing benchmarks, JETO-Mine is designed as a reusable tool that allows researchers continuously collect new benchmarks with their own desired filters and statistical rigor levels. We use JETO-Mine to build JETO-Bench, a benchmark of 660 identified ETIPs and 91 manually verified executable ETIPs collected from 174 open-source Java repositories. To build JETO-Bench, JETO-Mine scans 11 years of open-source development history and nearly 1.8 million commits. We run OpenHands, a leading open-source coding agent, on the 91 manually verified executable ETIPs in JETO-Bench and find that it correctly fixes 14.3% (13/91) of the issues, aligning with results reported by similar studies on other programming languages. Our results also reveal that open-source Java projects largely lack tests that demonstrate execution time improvements, presenting an opportunity for future research in test generation. JETO-Mine and JETO-Bench are publicly available at https://github.com/khesoem/JETO-Bench.
Keywords Execution Time Optimization, Automated Program Repair, Benchmark, Java, Coding Agents, Reproducibility
1
Introduction
During the past decade, automated program repair (APR) has made remarkable progress, further accelerated by the advent of agentic coding frameworks [39–41]. Benchmarks such as Defects4J [24], QuixBugs [25], and SWE-Bench [23] have been instrumental in driving these advances, providing standardized tasks and reproducible environments for evaluating APR techniques. However, these benchmarks are almost exclusively focused on functional
bugs, leaving non-functional issues, and performance issues in particular, underrepresented. Recently, researchers have turned their attention to automated fixing of performance issues [10, 16, 32, 42]. This new trend has led to the creation of benchmarks of execution time improvement patches, such as PIE [28], SWE-Perf [20], GSO [36], and PerfBench [15]. These benchmarks target languages such as Python, C++, and .NET. Despite being one of the most widely used programming languages, Java remains underrepresented in this area. Note that Java’s reliance on just-in-time compilation and garbage collection makes execution time measurements volatile, demanding more rigorous benchmarking methodology than is needed for languages like Python or C++. The only existing benchmark of execution time improvement patches in real-world Java projects is recently proposed by Yi et al. [44], which includes 65 tasks from four repositories. This dataset presents a fixed set of patches without containerized environments, making it difficult to reproduce results as project dependencies evolve over time. Beyond the language gap, to the best of our knowledge, existing performance improvement benchmarks have a common limitation: these datasets cannot be extended to collect new real-world patches according to user-defined configurations. In this paper, we present JETO-Mine, the first configurable and reusable tool for creating reproducible benchmarks of real-world execution time improvement patches (ETIPs) in Java. JETO-Mine provides a fully automated pipeline that researchers can run with their own desired filters and configurations to continuously collect new benchmarks. The pipeline consists of three phases. First, a static analysis phase crawls GitHub repositories and identifies commits and PRs that address execution time issues, using user-defined filters and an LLM-based issue classifier. Second, a dynamic analysis phase wraps the identified ETIPs in Docker images, building and executing the project tests on both original and patched versions to create fully reproducible environments. In this phase, JETO-Mine also runs the existing tests in the project multiple times and conducts statistical testing with user configurable parameters to find objective evidence for improvement of execution time. Third, an evaluation harness enables the quantitative assessment of generated patches and tests by running them inside Docker images and performing statistical testing on execution times. Using JETO-Mine, we build JETO-Bench, a benchmark of 660 identified ETIPs and 91 manually verified executable ETIPs collected from 174 open-source Java repositories spanning 11 years of development history. To demonstrate the usefulness of JETO-Bench, we run OpenHands [39], a leading open-source coding agent, on the 91 verified executable ETIPs. OpenHands correctly fixes 14.3% (13/91)
Conference’17, July 2017, Washington, DC, USA
of execution time issues, aligning with the results reported by similar studies on other programming languages [15, 36]. We also find that the evaluation harness automatically detects most incorrect patches, confirming the value of execution-based feedback for patch assessment. Furthermore, our analysis reveals that open-source Java projects largely lack tests that quantitatively demonstrate execution time improvements, highlighting a significant gap in current testing practices and a promising opportunity for future research. In summary, this paper makes the following contributions: • JETO-Mine, the first configurable and reusable tool for automatically creating reproducible benchmarks of execution time improvement patches in Java. JETO-Mine consists of a three-phase pipeline with user-defined configurations and statistical testing. • JETO-Bench, a benchmark of 660 identified and 91 manually verified executable ETIPs collected from 174 real-world Java repositories, accompanied by Docker images for full reproducibility. • An empirical evaluation of OpenHands on JETO-Bench. This empirical investigation provides insight into the current capabilities and limitations of advanced coding agents on Java execution time improvement tasks, and reveals the scarcity of execution time improvement tests in open-source Java projects.
2
Execution Time Improvement Patches
With recent developments in automated bug fixing, researchers are creating new techniques to fix non-functional bugs, such as memory bugs [12, 18], vulnerabilities [11], code quality deficiencies [8], and execution time issues [35]. To enable further research in this area, we propose JETO-Mine, the first configurable and reusable tool for creating benchmarks of real-world execution time improvement patches in Java. Execution time improvement patches differ from functional bug fixes in an important way: a correct patch preserves the functionality of the original version while improving its performance w.r.t. the input under which it is tested. These nuances require precise definitions, which we provide below. Definition 1 (Functional Equivalence). Two versions of a program, original and patched, are functionally equivalent if and only if both produce the same output for any expected input. Definition 2 (Execution Time Improvement Patch (ETIP)). A patch that modifies the original version of a program into the patched version is an execution time improvement patch (ETIP) if and only if the two versions are functionally equivalent and the patched version completes its execution significantly faster than the original given a certain expected input. In this case, we consider the original version to have an execution time bug that is fixed by the patch. Definition 3 (ETIP Detector Test). A test t is an ETIP detector test if and only if there is an ETIP that modifies an original version P to a patched version Q, t runs successfully on both P and Q, and the execution of t on Q is significantly faster than on P. Note that in our definitions of both ETIP and ETIP detector test, we only consider significant improvements of execution time. We emphasize significance in order to dismiss flaky changes due to randomness or execution environment variability. JETO-Mine lets
Khashayar Etemadi and Zhendong Su
users determine which level of difference is significant by choosing the number of times a test is run, the minimum execution time improvement, and the p-value that entails statistical significance. We explain this user-driven configuration further in Section 3.3. Benchmarking execution time in Java is particularly challenging due to the characteristics of the Java Virtual Machine (JVM). The JVM’s just-in-time (JIT) compiler progressively optimizes frequently executed code paths at runtime, meaning that the first several executions of a method can be significantly slower than subsequent ones [17]. Garbage collection (GC) introduces nondeterministic pauses that can distort timing measurements, especially for short running tests [2]. Class loading and bytecode verification also add overhead to early executions. These effects make measurements unreliable for detecting genuine execution time improvements. In contract with benchmarks targeting Python or C++, where execution time is more directly determined by the source code, a Java ETIP benchmark must explicitly account for JVM warm-up, GC interference, and JIT compilation variability. These JVM-specific concerns motivate several design decisions in JETO-Mine, which we describe in Section 3.3. JETO-Mine identifies ETIPs in open-source Java projects, creates Docker images that enable reproducible execution of these ETIPs, collects test execution time to help quantitative comparison, and provides tools for evaluating generated patches and tests. This framework facilitates future research in the area of execution time improvement in Java programs.
3 System Design 3.1 Overview Figure 1 illustrates an overview of how JETO-Mine works. The workflow consists of three main phases: 1) Static analysis: JETOMine goes over GitHub commits and PRs and identifies ETIPs based on the filters provided by the user, 2) Dynamic analysis: JETO-Mine wraps the identified ETIPs in docker images and creates a fully reproducible environment for their execution, and 3) Evaluation harness: using the reproducible environments generated in the second phase, JETO-Mine provides the tools for the quantitative evaluation of generated patches and tests. In the remainder of this section, we explain each phase in detail.
3.2
Static Analysis
Figure 2 shows how the static analysis phase of JETO-Mine works. The inputs are the user-defined filters that determine which patches should be collected and a GitHub token that enables access to public repositories. After going through patches in the form of both commits and PRs, JETO-Mine identifies and outputs ETIPs and their associated issues. This output provides the necessary data for a performance improvement patch generation task: the original version of the project, the issue that describes the performance problem, and the ground-truth patched version that resolves it. Static analysis starts by crawling repositories (step 1) that meet four criteria: 1) it is a Java project that uses Java 8 or newer and builds with mvnw, 2) the repository has been updated as recently as the user specified date, 3) it has at least the number of stars determined by the user, and 4) the project is not archived. These
JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java
Conference’17, July 2017, Washington, DC, USA
Figure 1: An overview of JETO-Mine workflow.
we deduplicate the set of all commits and PRs based on their code diff, as there can be multiple commits on the default branch with the same diff depending on the PR merge strategy used in the project. The result of static analysis is the set of identified ETIPs, each accompanied by an issue describing the execution time problem resolved by the patch. Each ETIP provides three essential pieces of information: an original version of the program with an execution time defect, a GitHub issue describing that defect, and a groundtruth patch showing how the issue can be resolved. Traditionally, such data has been successfully used for the evaluation of patch generation tools [38], which confirms the usability of the identified ETIPs. Identified ETIPs can be used for patch generation evaluation by checking if a generated patch exactly matches the ground truth [34, 38]. However, a generated patch can still correctly improve execution time without exactly matching the ground truth, e.g. by applying the same algorithmic optimization with minor differences in boundary checks or operation ordering. To provide more finegrained assessment beyond syntactic similarity, JETO-Mine makes ETIPs executable in a fully reproducible manner. By executing a generated patch, we can determine if it compiles, passes tests, or improves test execution times. Such feedback enables more precise evaluation and can also serve as a signal to iteratively improve the generated patch [43]. To guarantee executability, the identified ETIPs are passed to the dynamic analysis phase of JETO-Mine. Figure 2: The static analysis workflow in JETO-Mine.
criteria ensure that the project is active and popular according to user definition, and uses a build tool that enables reproducibility. After the desired repositories are selected, JETO-Mine checks the commit diffs on the default branch over the timeline specified by the user (step 2). A commit diff is considered valid if it 1) only changes .java files, 2) does not change test files, and 3) modifies fewer files than the user specified limit. The commits that meet these requirements are then checked to find their associated issues (step 3). JETO-Mine then prompts an LLM with the issue title, description, and commit message to determine if the issue is primarily focused on improving execution time (step 4). Only valid commits linked to execution time improvement issues are kept. In the fifth and final step, JETO-Mine cleans up the filtered commit-issue pairs. First, for commits that are part of a PR, we replace the commit with the corresponding PR to capture the complete change intended by the developer. This ensures that the considered code change fully addresses the execution time issue. We also verify that the PR’s changed files pass all the considered criteria. Second,
3.3
Dynamic Analysis
The dynamic analysis phase of JETO-Mine, as shown in Figure 3, takes the identified ETIPs and user configurations as input and outputs executable ETIPs in the form of Docker images. The resulting images contain the environment needed for building the project and running relevant tests, including all required dependencies, enabling full reproducibility for running generated patches or tests. The user can configure three parameters that determine whether a patch shows significant execution time improvement: the number of executions 𝑛𝑢𝑚_𝑒𝑥𝑒𝑐, the minimum improvement percentage 𝑚𝑖𝑛_𝑖𝑚𝑝𝑟 , and the target p-value 𝑝_𝑣𝑎𝑙. These parameters let the user control the statistical rigor of the results. The dynamic analysis phase of JETO-Mine is designed to address the JVM-specific challenges described in Section 2. To mitigate the effect of JIT compilation warm-up, JETO-Mine executes the test suite once as a dedicated warm-up round before beginning timed executions. The results of the warm-up round are discarded. To reduce the impact of garbage collection pauses and other sources of timing variation, JETO-Mine runs each test suite 𝑛𝑢𝑚_𝑒𝑥𝑒𝑐 times (30 by default) and applies statistical testing. Moreover, JETO-Mine runs original and patched versions inside the same docker image
Conference’17, July 2017, Washington, DC, USA
with identical JVM versions and configurations, ensuring that JIT compilation behavior, GC settings, and class loading conditions are the same for both versions. These measures are essential for producing reliable execution time comparisons on the JVM. Given the inputs, the dynamic analysis starts by analyzing the code diff between the original and patched versions to identify the modified files and modules (step 1). Next, JETO-Mine creates a Docker image with the correct Java version and builds and runs the JUnit tests of the modified modules on both versions (step 2). The tests are executed using Maven Wrapper multiple times according to the user given configuration. The build and test logs are saved in the image for later investigation and analyzed to ensure that across all executions of both versions: 1) the project successfully builds, 2) the modified modules contain tests, and 3) all tests pass. If any of these requirements is not met, the ETIP is excluded from the final set of executable ETIPs. This ensures that the tests can later be used to assess the functional correctness of generated patches. After confirming that both versions pass all tests, JETO-Mine extracts test execution times from the logs (step 3). The extracted data consists of times 𝑇𝑖 𝑗 𝑣 , where 1 ≤ 𝑖 ≤ 𝑛 refers to the 𝑖th test class in the modified modules, 1 ≤ 𝑗 ≤ 𝑛𝑢𝑚_𝑒𝑥𝑒𝑐 indicates the execution round, and 𝑣 ∈ {𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙, 𝑝𝑎𝑡𝑐ℎ𝑒𝑑 } indicates the version. At step 4, JETO-Mine performs statistical testing on the collected times. For each test class 1 ≤ 𝑖 0 ≤ 𝑛, JETO-Mine conducts a paired one-sided binomial test between the lists of 𝑇𝑖 0 ,𝑗,𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙 and 𝑇𝑖 0 ,𝑗,𝑝𝑎𝑡𝑐ℎ𝑒𝑑 for 1 ≤ 𝑗 ≤ 𝑛𝑢𝑚_𝑒𝑥𝑒𝑐. If the test shows that one version is 𝑚𝑖𝑛_𝑖𝑚𝑝𝑟 percent faster than the other with a p-value below 𝑝_𝑣𝑎𝑙, JETO-Mine considers the 𝑖 0 th test class an ETIP detector test. Note that for a given ETIP, some tests may favor the original while others favor the patched version. Finally, JETO-Mine summarizes the execution results in a report (step 5). For each ETIP, the report indicates whether it successfully builds and passes tests, which test classes detect a significant execution time difference, and whether there is a significant difference in overall execution time. JETO-Mine considers an ETIP executable and creates a Docker image for it if both versions build and pass tests in all 𝑛𝑢𝑚_𝑒𝑥𝑒𝑐 rounds. Even if no existing test detects an execution time improvement, the Docker-based executability still provides valuable feedback on build and test success that patch generation tools can leverage.
3.4
Evaluation Harness
The third and final phase of JETO-Mine provides an evaluation harness. This harness takes the ID of an executable ETIP, a generated patch, and an execution configuration as input, and evaluates whether any test class in the modified modules detects a significant execution time improvement by the generated patch. The evaluation harness works in five steps. First, the Docker image corresponding to the selected ETIP is pulled if not already present and a container is created. Second, the generated patch is applied to the original version and saved in a separate directory inside the container. Third, the tests are executed on both original and patched versions as many times as specified in the configuration. Fourth, the logs are analyzed to verify that both versions pass all tests in every execution round and to extract test execution times. Finally, a statistical test is performed to determine if any test class
Khashayar Etemadi and Zhendong Su
Figure 3: The dynamic analysis phase of JETO-Mine.
detects a statistically significant difference between the execution times of the two versions. The logs and statistical test results are presented to the user as the final report. In addition to patch evaluation, JETO-Mine also supports the evaluation of test generation tools. For this purpose, JETO-Mine takes a generated test class and runs it on the original and groundtruth patched versions, then conducts statistical testing to determine if the generated test is an ETIP detector test. This enables research in a rarely studied area: the generation of tests that detect execution time improvement.
3.5
Implementation
We implement JETO-Mine in Python. In its default configuration, the static analysis phase targets projects with at least 20 stars and commits that change at most 20 files since 2015, and uses gpt-5.1-codex-mini for the LLM-based issue filter. In the dynamic analysis phase, the tests are executed once for warm-up and 30 times for collecting execution times. Following previous work [20], JETO-Mine considers an improvement of 5% with a p-value of 10% significant. JETO-Mine, JETO-Bench, and all data relevant to this paper are publicly available [9].
4 Experimental Methodology 4.1 Building JETO-Bench We start our experiments by running JETO-Mine to build JETOBench, the first benchmark of execution time improvement patches in diverse real-world Java projects. We build JETO-Bench by running JETO-Mine with its default configuration (see Section 3.5), collecting commits and PRs until 2025-11-28. JETO-Mine checks 3,686 repositories and 1,769,958 commits spanning 11 years of open-source development history.
JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java
Conference’17, July 2017, Washington, DC, USA
Metric Min Q1 Median Q3 Max Commits 1 11 58 288 55,824 Stars 20 29 53 139 93,448 Table 1: The number of commits and stars across the considered repositories.
work [20], with 𝑝_𝑣𝑎𝑙 = 0.1 and 𝑚𝑖𝑛_𝑖𝑚𝑝𝑟 = 0.05, and 2) a conservative configuration with 𝑝_𝑣𝑎𝑙 = 0.05 and 𝑚𝑖𝑛_𝑖𝑚𝑝𝑟 = 0.1. For each configuration, we check how many executable ETIPs have a test that detects a significant execution time improvement. Cases where no such test exists indicate that the open-source project lacks tests demonstrating execution time improvements, highlighting an opportunity for future research in test generation.
This shows the large scale of JETO-Bench and our study, which provides unprecedented insight into execution time improvement in open-source Java projects. Table 1 reports the distribution of commits and stars across the considered repositories. The number of commits per repository ranges from 1 to 55,824 with a median of 58, while stars range from 20 to 93,448 with a median of 53, showing notable diversity in both activity and popularity. The result of running JETO-Mine on this large and diverse set of repositories and commits is JETO-Bench. JETO-Bench includes a set of ETIPs identified by the static analysis phase and a set of executable ETIPs in the form of Docker images built by the dynamic analysis phase. We conduct experiments to study the characteristics of JETO-Bench, the effectiveness of advanced coding agents on JETO-Bench, and the performance of JETO-Mine in building Java ETIP benchmarks.
4.4
4.2
Research Questions
Our experiments study the following research questions: • RQ1 (JETO-Bench characteristics): What are the characteristics of the ETIPs collected by JETO-Mine in terms of diversity, scope, and test coverage? We assess the characteristics of the identified and executable ETIPs in JETO-Bench to show the diversity of real-world execution time improvements collected by our approach. • RQ2 (using JETO-Bench in practice): How effectively can advanced coding agents fix execution time issues in JETOBench? We run OpenHands, a leading open-source coding agent, on the executable ETIPs in JETO-Bench to evaluate the current state of automated ETIP generation. • RQ3 (JETO-Mine performance): How precise and practical is JETO-Mine for building Java ETIP benchmarks? We investigate the precision and accuracy of JETO-Mine and measure the cost of benchmark building in terms of time, space, and financial expenses.
4.3
Protocol for RQ1 (JETO-Bench characteristics)
To answer RQ1, we count the number of identified ETIPs, executable ETIPs, and the repositories from which they are collected. These numbers indicate the overall applicability of JETO-Mine. We then characterize the ETIPs along four dimensions: the number of files changed, the number of modules modified, the year of creation, and the popularity of the corresponding repository. A diverse set of ETIPs enables the evaluation of various aspects of patch generation tools. We also examine ETIP detector tests using two statistical testing configurations: 1) the default configuration, following previous
Protocol for RQ2 (using JETO-Bench in practice)
To answer RQ2, we use OpenHands to generate patches for the execution time issues related to in executable ETIPs. OpenHands provides isolation via Docker sandboxing; we leverage this by letting OpenHands use the Docker images created by JETO-Mine as its sandbox environment with all the dependencies and settings needed to run the project. Based on preliminary experiments, we opt for gpt-5-mini, which provides strong effectiveness at reasonable cost compared to both older and larger models. To ensure a high-quality evaluation set, we manually filter the executable ETIPs according to two criteria: 1) the linked issue is descriptive enough for a developer to implement the fix, and 2) the issue is genuinely focused on execution time improvement. This yields a manually verified set of executable ETIPs that we recommend for future research. After running OpenHands, we manually classify each generated patch into four categories: 1) exact match: the AST is identical to the ground truth, 2) semantically equivalent: functionally equivalent to the ground truth and applying similar optimization but syntactically different, 3) correct location: modifies the right code location but differs semantically from the ground truth, and 4) wrong location: modifies an incorrect location. The first two categories constitute correct fixes. Although correct location patches fall short of correctness, they demonstrate accurate fault localization, a key step in patch generation [45]. We also track multi-file and multimodule patches in each category to assess performance on issues that span beyond a single file. We use the evaluation harness to build and run tests on all generated patches. The harness automatically identifies patches that have incorrect format, do not compile, or fail tests, allowing them to be dismissed without manual assessment. We verify that all correct and semantically equivalent patches pass the tests, which reaffirms the soundness of the harness. With this experiment, we check that the evaluation harness of JETO-Mine successfully employs the executable ETIPs in JETO-Bench to provide execution-based feedback on generated patches. Note that the goal of this experiment is not to build a state-ofthe-art ETIP generation tool. The goal is to investigate whether JETO-Bench and the evaluation harness can be successfully used to evaluate patch generation tools, and to gain insight into the capabilities of an advanced off-the-shelf agent on Java execution time issues.
4.5
Protocol for RQ3 (JETO-Mine performance)
To answer RQ3, we randomly select and manually analyze 50 identified ETIPs to evaluate the precision of ETIP identification, checking whether each is linked to an issue that focuses on execution time
Conference’17, July 2017, Washington, DC, USA
Khashayar Etemadi and Zhendong Su
Modules Files
asy
Identified ETIPs Executable ETIPs 0
20
40
60
Number of ETIPs
80
Number of ETIPs
200
sp
rin s d g-d pri ol at ng ph cl shar nc ne h op ttp liq a-mo-datainschoudsding k o4 j-o tr uni eng -cli uiba dub ngo -ne edu imp sph nequar eyclo gm ino ffle rok en se bo db o4 le lus ere tty kus ak t j r
Table 2: Number of modules and files modified by patches.
150
improvement. To assess overall accuracy, we separately sample 50 issues passed to the LLM-based Issue Filter (see Figure 2) and manually check whether this filter correctly labels them, regardless of whether they are execution time related or not. We also measure the time needed for identifying ETIPs, the cost of LLM-based issue filtering, the time the dynamic analysis phase takes to build Docker images, and the size of the resulting images. These numbers indicate the feasibility of using JETO-Mine on a large corpus of open-source commits.
4.6
Experiment Resources
We run all experiments on a machine with one AMD EPYC 7742 processor (64 cores, up to 2.25 GHz) and 256 GB DDR4 RAM (3200 MT/s). In the dynamic analysis phase, we dedicate 32 CPU cores and 80 GB RAM for building each Docker image.
5 Experimental Results 5.1 Results for RQ1 (JETO-Bench characteristics) JETO-Bench contains 660 identified ETIPs and 102 executable ETIPs1 . Compared to similar benchmarks for other programming languages [15, 20], 102 executable patches provide sufficient data for evaluating advanced patch generation techniques. Additionally, 21 ETIPs build successfully but are excluded from the executable set because their modified modules contain no tests, showing that the dynamic analysis effectively filters out patches lacking functional correctness validation. The identified ETIPs are collected from 174 distinct repositories, while the executable ETIPs come from 40. Figure 4 shows the distribution across the top 15 repositories, which cover topics such as distributed systems, data management, cloud-native frameworks, and software infrastructure, demonstrating the topical diversity of JETO-Bench. 1 Of the 102 executable ETIPs, 91 pass our manual verification criteria (see Section 4.4
and Section 5.2).
Executable ETIPs (102)
Min
Avg
Max
Min
Avg
Max
1 1
1.2 2.1
13 20
1 1
1.1 2.1
3 17
Non-executable ETIPs Executable ETIPs
100 50 0
Figure 4: The number of identified ETIPs and executable ETIPs from the most frequent repositories in JETO-Bench.
Identified ETIPs (660)
49 20-
9 9 9 99 .9k 9.9k k-99.9k -24 -49 -99 501k-4 100 250 500 5k-1 20 Repository Stars
Figure 5: The number stars of the repositories from which the ETIPs in JETO-Bench are collected.
Table 2 presents the modification scope of ETIPs in JETO-Bench. While most patches modify a single module (average close to 1), the benchmark includes patches spanning up to 13 modules and 20 files. Notably, 270 identified ETIPs and 40 executable ETIPs modify multiple files, making JETO-Bench suitable for evaluating repolevel patch generation tools [23]. Figure 5 shows that most ETIPs come from repositories with at least 10,000 stars, indicating maturity and reliability. Figure 6 shows a steady distribution of ETIPs from 2019 onward, with fewer from earlier years due to the Java 8 minimum requirement. This temporal diversity ensures the benchmark is not biased toward a particular development period. Finally, we check if existing JUnit tests serve as ETIP detector tests. With the default configuration (𝑝_𝑣𝑎𝑙 = 0.1, 𝑚𝑖𝑛_𝑖𝑚𝑝𝑟 = 0.05), only 19 ETIPs have a test class showing statistically significant improvement. With the conservative configuration (𝑝_𝑣𝑎𝑙 = 0.05, 𝑚𝑖𝑛_𝑖𝑚𝑝𝑟 = 0.1), only 8 do. Our manual assessment reveals that many of these tests show significant improvement due to noise in time collection rather than actual performance testing. Our takeaway is that open-source Java projects largely lack tests that verify execution time improvements, presenting an opportunity for research on automated ETIP detector test generation. Answer to RQ1: What are the characteristics of the ETIPs collected by JETO-Mine in terms of diversity, scope, and test coverage? JETO-Bench includes 660 identified ETIPs from 174 repositories and 102 executable ETIPs from 40 repositories, spanning 11 years of development history and repositories with 20 to 93,448 stars. The benchmark covers diverse project types and includes 270 multi-file ETIPs suitable for evaluating repository-level patch generation tools.
JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java
Number of ETIPs
80
Non-executable ETIPs Executable ETIPs
60
20 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 Commit Year
Figure 6: The year in which the ETIPs in JETO-Bench are created.
5.2
1 Original Version: 2 private void readClassFileNames() throws IOException { 3 URL jarUrl = jar.toUri().toURL(); 4 URLClassLoader classLoaderFromJar = new URLClassLoader(new URL[] {jarUrl},
null); ClassPath classPath = ClassPath.from(classLoaderFromJar); ImmutableSet<ClassInfo> allClasses = classPath.getAllClasses(); classFileNames = allClasses.stream().map(ClassInfo::getName).collect(toImmutableSet());
5 6 7
40
0
Conference’17, July 2017, Washington, DC, USA
Results for RQ2 (using JETO-Bench in practice)
As explained in Section 4.4, before running OpenHands on JETOBench, we first manually check the executable ETIPs in JETOBench and their corresponding issues. We only keep the issues that are focused on execution time improvement and have informative description. The curated set contains 91 issues and their executable ETIPs. We run OpenHands on these 91 cases and prompt it to generate patches to fix the execution time issue in the original version. We then manually assess the generated patches to gain a deep understanding of OpenHands performance on JETO-Bench. Table 3 shows the results of fixing the 91 manually verified execution time issues, broken down by issue scope (single-file vs. multi-file and single-module vs. multi-module). Each generated patch is classified as a correct fix (exact match or semantically equivalent), a modification at the correct location but with semantic differences, a modification at the wrong location, or not generated. The column “Test Success” reports how many generated patches successfully build and pass the test suite according to the evaluation harness of JETO-Mine. As presented in Table 3, OpenHands generates a correct fix for 14.3% (13/91) of the issues. This aligns with state-of-the-art studies on fixing performance issues in other programming languages [15]. This suggests that JETO-Bench is a reliable benchmark for evaluating advanced patch generation tools on Java execution time issues. We also see that 10 of the correct fixes exactly match the ground-truth, while 3 of them are semantically equivalent to the ground-truth but do not match it. Listing 1 is an example of a semantically equivalent patch generated by OpenHands for issue #1320 of “cloud-opensource”. This listing shows the original code, the ground-truth patched version and the patched version generated by OpenHands. To save space, we show the ground-truth and OpenHands patches in a single snippet as a diff. The issue states that in the original version all the classes are loaded from the Jar file and then the class names are taken from the loaded classes. In the patched version, the class names are directly read from the Jar file. The difference between the ground-truth and generated patches appears at lines 19-21, where they build the class name. Both versions read the filename, remove .class from the end of it, and replace / with ., but do this in a different order. Also, the ground-truth uses 6 as a literal, while OpenHands computes this literal with ".class".length(). This example shows how a
8 } 9 10 Ground-truth Patched Version and Its Diff with OpenHands Patched Version: 11 private void readClassFileNames() throws IOException { 12 try (JarFile jarFile = new JarFile(jar.toFile())) { 13 ImmutableSet.Builder<String> classNames = ImmutableSet.builder(); 14 Enumeration<JarEntry> entries = jarFile.entries(); 15 while (entries.hasMoreElements()) { 16 JarEntry entry = entries.nextElement(); 17 String name = entry.getName(); 18 if (name.endsWith(".class")) { 19 +
String className = name.replace('/', '.').substring(0,name.length()-6);
20 +
String className = name.substring(0, name.length() - ".class".length())
21 +
.replace('/', '.');
22 classNames.add(className); 23 } 24 } 25 } 26 this.classFileNames = classNames.build(); 27 }
Listing 1: The original code, the ground-truth patch and the OpenHands generated patch for issue #1320 of “cloudopensource”. The patches are semantically equivalent, but do not exactly match syntactically.
generated patch can apply the same improvement as the groundtruth with a different syntactical modification. Consequently, we cannot solely rely on statically checking the changes to the AST. This demonstrates the importance of having more fine-grained and execution-based feedback on generated patches. The JETO-Mine evaluation harness gives such feedback by running generated patch in the docker-based reproducible environment of executable ETIPs. The column “Test Success” of Table 3 shows that only for 22% (20/91) of issues the generated patch successfully applies, compiles, and passes the tests. This means that most incorrect generated patches can be automatically labeled as incorrect using the evaluation harness of JETO-Mine, which reaffirms the significance of the execution-based feedback provided by the harness. We also find that for 36.3% (33/91) of issues, OpenHands does not generate a patch. Looking at OpenHands logs, we find that in most of these cases OpenHands fails to generate a reasonable patch because it cannot understand the context of the issue and find the exact location of the fault. This indicates the importance of fault localization for fixing execution time issues. In Table 3, we see that in 27.5% (25/91) of cases, OpenHands fails to generate a correct patch, however it localizes the fault and modifies the correct location. This is a promising result that shows the feasibility of improving the effectiveness of OpenHands on JETO-Bench. The comparison between issues with different scopes in Table 3 shows that OpenHands can fix issues at all scopes: 18.5% of single-file issues, 8.1% of multi-file issues, 14.3% of single-module issues, and 14.3% of multi-module patches. This indicates the ability of OpenHands to fix repository level bugs and the usefulness of JETO-Bench in evaluating repository level bug fixing, which is becoming the focus of patch generation research [23]. Examining
Conference’17, July 2017, Washington, DC, USA
Khashayar Etemadi and Zhendong Su
Table 3: Results of OpenHands on generating execution time improvement patches. A patch is deemed a Correct Fix if it is either an Exact Match (its AST is identical to the ground truth) or a Sem. Equiv. (semantically equivalent but syntactically different). Corr. Loc. indicates a patch that modifies the correct location but differs semantically, Wron. Loc. indicates a patch that modifies an incorrect location, and Not Gen. indicates that the agent failed to produce a patch. Test Success indicates that the patched project compiles and passes the test suite. Correct Fix
0
20
Issue Scope
N
Exact Match
Sem. Equiv.
Total
Corr. Loc.
Wron. Loc.
Not Gen.
Test Success
Single-file Multi-file
54 37
7 3
3 0
10 (18.5%) 3 (8.1%)
19 (35.2%) 6 (16.2%)
7 (13.0%) 13 (35.1%)
18 (33.3%) 15 (40.5%)
14 (25.9%) 6 (16.2%)
Single-module Multi-module
77 14
8 2
3 0
11 (14.3%) 2 (14.3%)
24 (31.2%) 1 (7.1%)
14 (18.2%) 6 (42.9%)
29 (37.7%) 4 (28.6%)
16 (20.8%) 4 (28.6%)
Overall
91
10
3
13 (14.3%)
25 (27.5%)
20 (22.0%)
33 (36.3%)
20 (22.0%)
40
60
80 Duration (min)
100
120
140
160
(a) Distribution of time spent to build the images (min.).
500
1000
1500 Size (MiB)
2000
2500
3000
(b) Distribution of the size of images (MiB).
Figure 7: Characteristics of Docker images built by JETOMine.
the two multi-module correct fixes, we find that both involve small, repetitive changes across modules (e.g. changing a constant from RefreshPolicy.WAIT_UNTIL to RefreshPolicy.IMMEDIATE), suggesting that complex multi-module performance issues remain an open challenge. Answer to RQ2: How effectively can advanced coding agents fix execution time issues in JETO-Bench? OpenHands correctly fixes 14.3% (13/91) of the manually verified execution time issues in JETO-Bench, aligning with results reported for other programming languages. The evaluation harness automatically dismisses most incorrect patches, with only 20 out of 91 generated patches passing all tests. OpenHands also generates correct fixes for 3 multi-file and 2 multi-module issues, demonstrating the applicability of JETOBench for evaluating repository-level patch generation tools.
5.3
Results for RQ3 (JETO-Mine performance)
As explained in Section 4.5, to assess the precision of JETO-Mine, we manually analyze 50 randomly selected identified ETIPs. Note that the LLM-based filter of JETO-Mine considers all issues corresponding to these ETIPs to be focused on execution time. Our manual analysis confirms that 48 of these issues are focused on execution time improvement. This means that according to this
investigation, the precision of JETO-Mine is 96% (48/50). Compared to previous study on issue classification [1], this is a high precision that makes JETO-Mine a reliable tool for collecting ETIPs. A more detailed analysis reveals that beyond the 2 non-relevant issues, 5 additional issues focus on execution time but also require new functionality, which falls outside our ETIP definition (Section 2). We exclude these from our manually verified set of 91 executable ETIPs to ensure all patches preserve functional equivalence. To assess overall accuracy, we also sample 50 commits passed to the “LLM-based Issue Filter” step, of which 3 are manually confirmed as execution-time-related and 47 are not. JETO-Mine correctly labels 48 of 50 issues (96% accuracy), confirming that the filter is reliable across both positive and negative cases. Measuring the time dedicated to running the static analysis phase of JETO-Mine on the 1,769,958 target commits, we find that JETOMine takes 478 hours to identify the 660 ETIPs. This shows the value of JETO-Bench for future research on fixing execution time issues, as it provides filtered data that requires almost 20 days of commit inspection. The static analysis of the commits also spends $5.33 to filter the issues with the help of LLMs, which shows the affordability of our approach. The dynamic analysis phase of JETO-Mine takes 152 hours to process the 660 identified ETIPs and create 102 executable ETIPs in form of Docker images. This means that processing each identified ETIP takes only 13.8 minutes, which indicates the feasibility of building large benchmarks with JETO-Mine. Figure 7 shows the characteristics of the Docker images built by JETO-Mine. The median time it takes to process an identified ETIP and build the Docker image is 34 minutes. Also, the median size of built docker images is 790 MB. Again, this shows that JETO-Mine can be used in practice to create Docker images for reproducible execution of ETIPs.
JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java
Answer to RQ3: How precise and practical is JETO-Mine for building Java ETIP benchmarks? JETO-Mine achieves 96% precision in identifying execution time issues, based on manual analysis of 50 randomly selected identified ETIPs. The static analysis phase processes 1.7 million commits in 478 hours at a cost of $5.33, while the dynamic analysis phase builds Docker images with a median time of 34 minutes and a median size of 790 MB. These results indicate that JETO-Mine is both reliable and practical for building large scale ETIP benchmarks.
6 Discussion 6.1 JMH Tests
Conference’17, July 2017, Washington, DC, USA
detector test generation will likely need to combine code coverage techniques with workload synthesis to produce inputs that stress the performance-critical paths identified by the patch.
6.3
Threats to Validity
6.3.1 Threat to External Validity. JETO-Mine collects ETIPs from projects that use Maven Wrapper to build. This means Java projects that use other build tools, such as Gradle, are not considered. Hence, we cannot be certain that our approach generalizes to Java projects that do not use Maven Wrapper. However, as reported in Section 5.1, the number of ETIPs identified by JETO-Mine aligns with similar benchmarks for other programming languages [15, 20], which indicates the usefulness of our tool for evaluating advanced patch generation tools.
As explained in Section 3.3, JETO-Mine runs the JUnit tests multiple times and performs statistical testing to see if any test detects a significant execution time improvement. Our experiments show that such JUnit tests are rare (see Section 5.1). This leads us to another question: Is there a framework that open-source Java projects commonly use to demonstrate execution time improvements? The standard and most widely used framework for performance testing in Java is the Java Microbenchmark Harness (JMH)2 [44]. We analyze the 91 manually verified executable ETIPs in JETO-Bench to assess the use of JMH in open-source Java projects. We find that only 15% (14/91) of the executable ETIPs have JMH benchmarks in their code. This shows that even considering JMH benchmarks does not solve the lack of tests that demonstrate execution time improvements. Therefore, we emphasize the importance of adding more ETIP detector tests in open-source Java projects.
6.3.2 Threat to Internal Validity. The dynamic analysis and evaluation harness of JETO-Mine run project tests to determine if they detect an execution time improvement. Since the execution environment can be volatile, the results of this step may not be fully robust and reproducible. To address this concern, we run the tests multiple times and use statistical testing to report how confidently we can claim a test detects an improvement. However, in our experiments, we noticed that the order of running tests can affect execution time. This is an issue because by default, Maven does not guarantee a deterministic order for running tests. We resolve this with two techniques. First, we always configure Maven to run the tests in alphabetical order on both versions. Second, over the 𝑛𝑢𝑚_𝑒𝑥𝑒𝑐 rounds of executing tests on original and patched versions, at odd rounds we first run the patched version and at even rounds we first run the original version. These two techniques minimize the effects of test execution order and environment volatility to maximize confidence in our statistical test results.
6.2
7
Test Generation
Given the lack of tests that show execution time differences in open-source projects, automated ETIP detector test generation is a valuable task for future research. As explained in Section 3.4, the evaluation harness of JETO-Mine enables the evaluation of tools that generate ETIP detector tests. In a preliminary study, we employ OpenHands to generate tests that show a statistically significant difference between the execution times of original and patched versions for our 91 manually verified executable ETIPs. We find that OpenHands is able to generate an ETIP detector test for only one of the 91 ETIPs (the generated test and the corresponding ETIP are available in our replication package [9]). This indicates the difficulty of generating ETIP detector tests and presents a promising opportunity for future studies. The challenge of generating ETIP detector tests lies in the fact that such tests must not only exercise the modified code paths, but must do so with inputs large or representative enough to reveal measurable performance differences. A test that calls the improved code with a small input may pass on both versions with little timing difference, even if the patch provides orders of magnitude improvement on realistic workloads. This contrasts with functional test generation, where a single well-chosen input can definitively distinguish correct from incorrect behavior. Future work on ETIP 2 https://github.com/openjdk/jmh
Related Work
Recently, many researchers are working on automated repair of various non-functional bugs, including performance issues [14, 16, 19, 32, 37, 42]. This line of research heavily depends on having benchmarks for the evaluation of patches generated by repair tools. Consequently, researchers are also working on new performance improvement benchmarks and tools to collect such benchmarks [15, 20, 36]. In this section, we review the literature on performance issue benchmarks, automated fixing of performance issues, and studies on the performance optimization capabilities of AI agents, which are the closest research areas to our work.
7.1
Performance Issue Benchmarks
There are established bug benchmarks, such as Defects4J [24], Bugs.jar [33], IntroClassJava [7], QuixBugs [25], and Bears [29], which have been used for evaluating automated program repair (APR) tools for more than a decade. However, these benchmarks are particularly focused on functional bugs and rarely updated. In a more recent effort, Jimenez et al. [23] have proposed SWE-Bench, which has become one of the most prominent benchmarks for APR evaluation. SWE-Bench provides pairs of GitHub issues and their resolving PRs as well as Docker containers for reproducing the issues and executing generated patches. In contrast to JETO-Bench, SWE-Bench is focused on Python and functional bugs.
Conference’17, July 2017, Washington, DC, USA
After years of progress in automatically fixing functional bugs, researchers have recently paid more attention to building APR tools for performance bugs [27]. Consequently, new benchmarks of performance bugs are also created to support this effort [3]. Several benchmarks are collected from code competition submissions, including PIE [28] (77K C++ modifications), EffiBench [22], EvalPerf [26], and Mercury [6]. While competition submissions are widely used for evaluating the performance of generated code [4], these programs are usually small and different from real-world projects. To address this limitation, researchers have created more realistic benchmarks from real-world repositories: SWE-Perf [20] (140 tasks from 12 Python repositories), SWE-fficiency [27] (performance PRs from nine data-science and HPC repositories), GSO [36] (102 tasks from 10 codebases in Python, C, and SIMD), PeacExec [32] (146 tasks from 47 Python repositories), and PerfBench [15] (81 tasks from .NET repositories with an evaluation harness for test generation). Evaluations on these benchmarks consistently show the difficulty of automatically fixing performance issues, with OpenHands achieving success rates of 4.9% on GSO and 3% on PerfBench. Automated performance improvement tools have also been tested on more specialized programs, such as GPU kernels [30] and compilers [5]. Compared to these benchmarks, JETO-Bench is the first benchmark of execution time improvement patches specifically targeting Java programs. While the existing benchmarks primarily focus on Python [20, 27, 32], C++ [28], .NET [15], or multiple languages [36], Java remains underrepresented despite being one of the most widely used programming languages. The most closely related benchmark that focuses on Java is the dataset used by Yi and Leitner [44], which contains 65 tasks from four open-source Java projects. However, their dataset does not provide an automated pipeline for collecting new tasks or Docker-based environments for reproducible evaluation. JETO-Mine provides a fully automated pipeline for creating new benchmarks with user-defined filters, Docker-based reproducible environments, and an evaluation harness that supports both patch and test generation evaluation.
7.2
Fixing Performance Issues
The existing benchmarks have enabled many studies on automated fixing of performance issues. Several tools target function-level optimizations: RAPGen [16] uses retrieval-augmented prompt generation, DeepDev-PERF [14] fine-tunes a transformer model for repairing performance bugs, PerfCoder [42] fine-tunes an LLM on curated real-world optimizations, SBLLM [13] combines LLMs with evolutionary search, and EFFI-LEARNER [21] uses execution overhead profiles to improve efficiency of LLM-generated code. At the repository level, Ren et al. propose PEACE [32], a hybrid framework for code efficiency optimization through automatic code editing, achieving a 69.2% correctness rate on PeacExec. Researchers have also evaluated the performance of patches generated by agentic frameworks, such as OpenHands [39], Agentless [40], SWE-Agent [41], and AutoCodeRover [46]. Fan et al. [10] introduce new metrics to compare the performance of programs generated by these frameworks, while Peng et al. [31] present an empirical study comparing agent-authored and human-authored
Khashayar Etemadi and Zhendong Su
optimizations, finding that the optimization patterns largely overlap. The work of Yi and Leitner [44] presents an investigation on LLM-based performance improvements using their dataset of 65 real-world tasks mined from four open-source Java programs. They generate patches using two leading LLMs under four prompt variations and use JMH benchmarks to compare the results with humanauthored solutions. Their study demonstrates that human developers outperform LLM fixes by a statistically significant margin. Our evaluation of OpenHands on the 91 manually verified executable ETIPs in JETO-Bench aligns with the findings reported by these studies. Similarly to previous work, we observe that stateof-the-art agents achieve limited success rates on performance optimization tasks, motivating the need for dedicated benchmarks like JETO-Bench.
8
Conclusion
In this paper, we present JETO-Mine, the first configurable and reusable tool for creating reproducible benchmarks of execution time improvement patches (ETIPs) in real-world Java projects. JETOMine combines three phases: 1) static analysis for identifying ETIPs from GitHub repositories, 2) dynamic analysis for building Dockerbased reproducible environments, and 3) evaluation harness for quantitative assessment of generated patches and tests. Using JETO-Mine, we built JETO-Bench, a benchmark of 660 identified ETIPs and 91 manually verified executable ETIPs collected from 174 open-source Java repositories. Our manual analysis shows that JETO-Mine identifies ETIPs with a precision of 96%. Running OpenHands on the 91 verified executable ETIPs, we find that it correctly fixes 14.3% (13/91) of the issues, aligning with results reported by similar studies on other programming languages. The evaluation harness automatically detected most incorrect patches, confirming the value of its execution-based feedback. Our analysis also reveals that open-source Java projects largely lack tests that demonstrate execution time improvements: only 15% of verified executable ETIPs contain JMH benchmarks, and existing JUnit tests rarely detect statistically significant execution time differences. Therefore, the evaluation harness of JETO-Mine presents an opportunity for future research on automated fixing of execution time issues and automated generation of ETIP detector tests.
Acknowledgements We used Cursor as a coding assistant during the development of JETO-Mine. We also used Claude to help polish the text and figures of this paper.
Data Availability Statement JETO-Mine, JETO-Bench, and all data and scripts needed to reproduce the results reported in this paper are publicly available at: https://github.com/khesoem/JETO-Bench The replication package includes the source code of JETO-Mine, the full list of 660 identified ETIPs and their associated issues, the Docker images for the executable ETIPs, and manually verified set of 91 executable ETIPs. The data and scripts needed for reproducing the tables and charts are also presented.
JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java
References [1] Gabriel Aracena, Kyle Luster, Fabio Santos, Igor Steinmacher, and Marco A Gerosa. 2025. Applying large language models to issue classification: Revisiting with extended data and new models. Science of Computer Programming 246 (2025), 103333. [2] Stephen M Blackburn, Robin Garner, Chris Hoffmann, Asjad M Khang, Kathryn S McKinley, Rotem Bentzur, Amer Diwan, Daniel Feinberg, Daniel Frampton, Samuel Z Guyer, et al. 2006. The DaCapo benchmarks: Java benchmarking development and analysis. In Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications. 169–190. [3] Aymeric Blot and Justyna Petke. 2025. A comprehensive survey of benchmarks for improvement of software’s non-functional properties. Comput. Surveys 57, 7 (2025), 1–36. [4] Tristan Coignion, Clément Quinton, and Romain Rouvoy. 2024. A performance study of llm-generated code on leetcode. In Proceedings of the 28th international conference on evaluation and assessment in software engineering. 79–89. [5] Chris Cummins, Volker Seeker, Dejan Grubisic, Mostafa Elhoushi, Youwei Liang, Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Kim Hazelwood, Gabriel Synnaeve, et al. 2023. Large language models for compiler optimization. arXiv preprint arXiv:2309.07062 (2023). [6] Mingzhe Du, Luu A Tuan, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. Mercury: A code efficiency benchmark for code large language models. Advances in Neural Information Processing Systems 37 (2024), 16601–16622. [7] Thomas Durieux and Martin Monperrus. 2016. IntroClassJava: A benchmark of 297 small and buggy Java programs. Ph. D. Dissertation. Universite Lille 1. [8] Khashayar Etemadi, Nicolas Harrand, Simon Larsen, Haris Adzemovic, Henry Luong Phu, Ashutosh Verma, Fernanda Madeiral, Douglas Wikström, and Martin Monperrus. 2022. Sorald: Automatic patch suggestions for sonarqube static analysis violations. IEEE Transactions on Dependable and Secure Computing 20, 4 (2022), 2794–2810. [9] Khashayar Etemadi and Zhendong Su. 2026. Replication package. https://github. com/khesoem/JETO-Bench [10] Zhiyuan Fan, Kirill Vasilevski, Dayi Lin, Boyuan Chen, Yihao Chen, Zhiqing Zhong, Jie M. Zhang, Pinjia He, and Ahmed E. Hassan. 2025. SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints. ArXiv abs/2509.09853 (2025). https://api.semanticscholar.org/CorpusID: 281310248 [11] Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dinh Phung. 2022. Vulrepair: a t5-based automated software vulnerability repair. In Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering. 935–947. [12] Qing Gao, Yingfei Xiong, Yaqing Mi, Lu Zhang, Weikun Yang, Zhaoping Zhou, Bing Xie, and Hong Mei. 2015. Safe memory-leak fixing for c programs. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 459–470. [13] Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael R. Lyu. 2025. SearchBased LLMs for Code Optimization. IEEE Press, 578–590. https://doi.org/10.1109/ ICSE55347.2025.00021 [14] Spandan Garg, Roshanak Zilouchian Moghaddam, Colin B Clement, Neel Sundaresan, and Chen Wu. 2022. Deepdev-perf: a deep learning-based approach for improving software performance. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 948–958. [15] Spandan Garg, Roshanak Zilouchian Moghaddam, and Neel Sundaresan. 2025. PerfBench: Can Agents Resolve Real-World Performance Bugs? arXiv preprint arXiv:2509.24091 (2025). [16] Spandan Garg, Roshanak Zilouchian Moghaddam, and Neel Sundaresan. 2025. Rapgen: An approach for fixing code inefficiencies in zero-shot. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 124–135. [17] Andy Georges, Dries Buytaert, and Lieven Eeckhout. 2007. Statistically rigorous java performance evaluation. ACM SIGPLAN Notices 42, 10 (2007), 57–76. [18] Mohammadreza Ghanavati, Diego Costa, Janos Seboek, David Lo, and Artur Andrzejak. 2020. Memory and resource leak defects and their repairs in Java projects. Empirical Software Engineering 25, 1 (2020), 678–718. [19] Jingzhi Gong, Vardan Voskanyan, Paul Brookes, Fan Wu, Wei Jie, Jie Xu, Rafail Giavrimis, Mike Basios, Leslie Kanthan, and Zheng Wang. 2025. Language models for code optimization: Survey, challenges and future directions. arXiv preprint arXiv:2501.01277 (2025). [20] Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. 2025. Swe-perf: Can language models optimize code performance on real-world repositories? arXiv preprint arXiv:2507.12415 (2025). [21] Dong HUANG, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao QING, Heming Cui, Zhijiang Guo, and Jie Zhang. 2024. EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=KhwOuB0fs9 [22] Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M Zhang. 2024. Effibench: Benchmarking the efficiency of automatically generated code. Advances
Conference’17, July 2017, Washington, DC, USA
in Neural Information Processing Systems 37 (2024), 11506–11544. [23] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. SWE-bench: Can Language Models Resolve RealWorld GitHub Issues? ArXiv abs/2310.06770 (2023). https://api.semanticscholar. org/CorpusID:263829697 [24] René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 international symposium on software testing and analysis. 437–440. [25] Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017. QuixBugs: A multi-lingual program repair benchmark set based on the Quixey Challenge. In Proceedings Companion of the 2017 ACM SIGPLAN international conference on systems, programming, languages, and applications: software for humanity. 55–56. [26] Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024. Evaluating language models for efficient code generation. arXiv preprint arXiv:2408.06450 (2024). [27] Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, and Parthasarathy Ranganathan. 2025. SWEfficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? arXiv preprint arXiv:2511.06090 (2025). [28] Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. 2023. Learning Performance-Improving Code Edits. ArXiv abs/2302.07867 (2023). https: //api.semanticscholar.org/CorpusID:256868633 [29] Fernanda Madeiral, Simon Urli, Marcelo Maia, and Martin Monperrus. 2019. Bears: An extensible java bug benchmark for automatic program repair studies. In 2019 IEEE 26th international conference on software analysis, evolution and reengineering (SANER). IEEE, 468–478. [30] Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Re, and Azalia Mirhoseini. 2025. KernelBench: Can LLMs Write Efficient GPU Kernels?. In Forty-second International Conference on Machine Learning. https: //openreview.net/forum?id=yeoN1iQT1x [31] Huiyun Peng, Antonio Zhong, Ricardo Andrés Calvo Méndez, Kelechi G Kalu, and James C Davis. 2025. How Do Agents Perform Code Optimization? An Empirical Study. arXiv preprint arXiv:2512.21757 (2025). [32] Xiaoxue Ren, Jun Wan, Yun Peng, Zhongxin Liu, Ming Liang, Dajun Chen, Wei Jiang, and Yong Li. 2025. PEACE: Towards Efficient Project-Level Efficiency Optimization via Hybrid Code Editing. arXiv preprint arXiv:2510.17142 (2025). [33] Ripon K Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Mukul R Prasad. 2018. Bugs. jar: A large-scale, diverse dataset of real-world java bugs. In Proceedings of the 15th international conference on mining software repositories. 10–13. [34] Shaunak Samant. 2025. Syntax Is Not Enough: An Empirical Study of Small Transformer Models for Neural Code Repair. arXiv preprint arXiv:2512.22216 (2025). [35] Marija Selakovic and Michael Pradel. 2015. Poster: Automatically fixing realworld JavaScript performance bugs. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. IEEE, 811–812. [36] Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, and Ion Stoica. 2025. Gso: Challenging software optimization tasks for evaluating swe-agents. arXiv preprint arXiv:2505.23671 (2025). [37] Manav Singhal, Tushar Aggarwal, Abhijeet Awasthi, Nagarajan Natarajan, and Aditya Kanade. 2024. Nofuneval: Funny how code lms falter on requirements beyond functional correctness. arXiv preprint arXiv:2401.15963 (2024). [38] Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM) 28, 4 (2019), 1–29. [39] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=OJd3ayDDoF [40] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents. Proc. ACM Softw. Eng. 2, FSE, Article FSE037 (June 2025), 24 pages. doi:10.1145/3715754 [41] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652. [42] Jiuding Yang, Shengyao Lu, Hongxuan Liu, Shayan Shirahmad Gale Bagi, Zahra Fazel, Tomasz Czajkowski, and Di Niu. 2025. PerfCoder: Large Language Models for Interpretable Code Performance Optimization. arXiv preprint arXiv:2512.14018 (2025). [43] He Ye, Matias Martinez, and Martin Monperrus. 2022. Neural program repair with execution-based backpropagation. In Proceedings of the 44th international conference on software engineering. 1506–1518.
Conference’17, July 2017, Washington, DC, USA
[44] Lirong Yi, Gregory Gay, and Philipp Leitner. 2025. An Experimental Study of RealLife LLM-Proposed Performance Improvements. arXiv preprint arXiv:2510.15494 (2025). [45] Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen. 2023. A survey of learning-based automated program repair. ACM Transactions
Khashayar Etemadi and Zhendong Su
on Software Engineering and Methodology 33, 2 (2023), 1–69. [46] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604.