Rethinking Code Performance Benchmarks for LLMs

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Rethinking Code Performance Benchmarks for LLMs

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

softwarearchitecture softwareengineering testing

software engineering, software architecture, testing

Noname manuscript No. (will be inserted by the editor)

Rethinking Code Performance Benchmarks for LLMs Nhat Minh Le · Yisen Xu · Zhijie Wang · Tse-Hsun (Peter) Chen

arXiv:2607.07619v1 [cs.SE] 8 Jul 2026

Received: date / Accepted: date

Abstract Many function-level performance benchmarks have been proposed to evaluate whether large language models (LLMs) can generate efficient programs. However, results on these benchmarks often show that LLM-generated implementations have little or no execution-time difference from canonical solutions. This observation motivates us to revisit these benchmarks and examine whether they are suitable for performance evaluation. In this paper, we revisit four popular benchmarks: EffiBench, Enamel, EvalPerf, and Mercury. We evaluate 1,538 tasks under more rigorous setting by running each task 30 times and assessing the runtime differences between the canonical solutions and benchmark-provided performant implementations with statistical testing. With the benchmark-provided test suites, only 6.11% of the performant implementations are significantly faster than the canonical solutions. In a manual analysis of 308 non-significant tasks, 99 performant implementations contain no meaningful performance change, while 209 contain potential performance improvements that are not exposed by the original tests. Nhat Minh Le Software Performance, Analysis, and Reliability (SPEAR) Lab, Concordia University, Montreal, Quebec, Canada E-mail: [email protected] Yisen Xu Software Performance, Analysis, and Reliability (SPEAR) Lab, Concordia University, Montreal, Quebec, Canada E-mail: [email protected] Zhijie Wang Concordia University, Montreal, Quebec, Canada E-mail: [email protected] Tse-Hsun (Peter) Chen Software Performance, Analysis, and Reliability (SPEAR) Lab, Concordia University, Montreal, Quebec, Canada E-mail: [email protected]

INTRODUCTION

These results suggest that the main limitation is not only the evaluation method, but also the limited sufficiency of the benchmark-provided performance tests. To address this limitation, we propose an LLM-based multi-agent framework to generate performance-oriented tests that expose runtime differences more effectively than the original tests. The framework uses three separate agents to generate, diagnose, and repair deterministic tests that preserve functional correctness while better exposing performance differences. Across 1,345 benchmark tasks for which the original tests found no significant performance difference, tests generated by our framework with DeepSeek-v3.1 and GPT-4o reveal statistically significant improvements in 24.01% and 25.43% of the tasks, respectively, outperforming the SOTA LLM-based performance test generation method. Finally, we discuss the implications for future performance benchmark for LLM-generated code. In addition to using repeated execution and statistical testing to improve rigor, future work should consider selecting problems with meaningful opportunities for performance optimization rather than relying on overly simple tasks, constructing sufficiently challenging test cases that make runtime differences between implementations observable, and extending this line of evaluation from isolated function-level tasks to class-level or repository-level settings where performance bottlenecks may arise from interactions among multiple components. Keywords Performance Benchmark, Large Language Models, Code Generation 1 Introduction Software performance has a long research history in the software engineering community (Woodside et al., 2007). Decades of work have studied how to measure, diagnose (Jin et al., 2012; Baltes et al., 2015), and improve runtime behavior through performance testing (Vokolos and Weyuker, 1998; Weyuker and Vokolos, 2000), workload characterization (Avritzer et al., 2002), performance debugging, benchmarking (Mytkowicz et al., 2009; Kalibera and Jones, 2013), and statistically rigorous performance (Georges et al., 2007). As software systems increasingly rely on automated code generation, this long-standing problem has re-emerged in a new setting: whether code produced by large language models (LLMs) is not only functionally correct but also computationally efficient. A series of benchmarks for evaluating the correctness of LLM-generated code have been proposed at the function level (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021; Lai et al., 2023; Li et al., 2022a; Jain et al., 2025; Zhuo et al., 2025), class level (Du et al., 2023; Cao et al., 2024; Rahman et al., 2025; Chen et al., 2026), or repository level (Jimenez et al., 2024; Liu et al., 2024b; Li et al., 2024b; Le Hai et al., 2025; Li et al., 2025). Notably, function-level Python code generation benchmarks, such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), have become the dominant evaluation setting because individual functions naturally serve as self-contained,

executable units with well-defined input-output specifications. Compared with class or repository-level generation, function-level tasks largely avoid confounding factors such as project organization, dependency management, or build configuration, allowing model performance to be attributed more directly to code synthesis ability. Furthermore, the generated functions can be reliably evaluated using unit tests, enabling scalable and reproducible comparison across models. Nevertheless, correctness alone does not tell us whether generated code is computationally efficient. To address this gap, recent work has developed function-level Python program efficiency benchmarks, such as EffiBench (Huang et al., 2024b), Enamel (Qiu et al., 2025), EvalPerf (Liu et al., 2024a), and Mercury (Du et al., 2024). Each task includes a canonical solution and a benchmark-provided performant implementation. The canonical solution serves as the baseline for evaluating LLM-generated implementations, whereas the performant implementation demonstrates that the task admits a more efficient solution. During evaluation, an LLM-generated implementation is first validated for correctness and, if correct, its execution time is compared against the canonical solution using the benchmark-provided test cases. Yet, evaluations using these benchmarks often find that LLM-generated implementations show little runtime difference from, or remain slower than, the corresponding canonical implementations. For example, a recent study (Islam et al., 2025) shows that even strong LLMs remain slower than human-written canonical reference solutions: on 298 commonly solved LeetCode tasks solved by all 20 models, the canonical reference code runs in 74.16 ms on average, while LLMgenerated solutions range from 75.64 ms for DeepSeek-V3 (DeepSeek, 2025b) to 147.95 ms for GPT-4 Turbo (OpenAI, 2023). Before attributing such results solely to model limitations, we first ask whether the benchmarks can expose intended optimizations at all. Surprisingly, even benchmark-provided performant implementations often show little or no runtime improvement over the corresponding canonical solutions under the current evaluation settings. This suggests that the evaluation settings of these benchmarks may not fully capture meaningful execution-time differences. Two factors can contribute to this problem. First, many benchmarks do not adopt statistically rigorous measurement procedures. Execution times are often collected from only a single run (e.g., Effibench (Huang et al., 2024b) and Mercury (Du et al., 2024)) or a small number of runs (e.g., 6 times in Enamel (Qiu et al., 2025)), without repeated measurements and statistical significance testing to determine whether observed differences are reliable rather than the result of measurement noise. It remains unclear whether the reported performance differences between LLM-generated implementations and reference implementations reflect genuine efficiency improvements. Second, many performance benchmarks are derived from existing code-generation benchmarks such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), where test cases are usually designed to assess functional correctness rather than execution efficiency. As a result, the benchmark-provided test inputs may not be sufficiently challenging to reveal meaningful execution-time differences

INTRODUCTION

between implementations. Implementations with substantially different algorithmic or optimization characteristics may appear to perform similarly under the existing benchmark’s evaluation setting. These observations raise a fundamental question: do current benchmarks reliably distinguish genuinely efficient implementations from merely functionally correct ones? In this paper, we revisit four function-level Python performance benchmarks using a more statistically rigorous performance evaluation. We study 1,538 tasks from EffiBench (Huang et al., 2024b), Enamel (Qiu et al., 2025), EvalPerf (Liu et al., 2024a), and Mercury (Du et al., 2024). For each task, we execute both the canonical solution and the performant implementation 30 times on the benchmark-provided tests, providing sufficient observations for stable statistical analysis while maintaining a practical computational cost (Arcuri and Briand, 2011; Zeng et al., 2019; Su et al., 2019; Liao et al., 2025; Jangali et al., 2023). Our re-evaluation shows that most benchmark-provided performant implementations do not exhibit statistically significant (p ≥ 0.05) execution-time improvements under the original test suites. Across 1,538 tasks, only 94 performant implementations (6.11%) are significantly faster than their corresponding canonical solutions, while 1,444 implementations (93.89%) are not statistically distinguishable from the canonical solutions. To understand why these implementations remain statistically indistinguishable, we analyze the corresponding benchmark tasks. We sampled 308 tasks from the 1,444 non-significant tasks across the four benchmarks for manual analysis. The sample provides a 95% confidence level with a 5% margin of error. Two of the authors performed open coding (final Cohen’s Kappa: 0.93, indicating almost perfect agreement) and found two main reasons. First, 99 out of 308 performant implementations contain no meaningful performance change, such as refactoring or no code changes. The remaining 209 tasks contain algorithmic or data-structure changes that should plausibly improve runtime performance. However, the original test inputs do not stress the code paths where these changes matter. We further extended our analysis with LLM-as-a-Judge (Ahmed et al., 2025; Li et al., 2024a; Zhang et al., 2024b) over the remaining 1,136 non-significant tasks. We first confirm substantial agreement between human labelers and two LLMs (DeepSeek-V3.1 and GPT40-2024-08-06) on the 308 manually labeled tasks, with Cohen’s Kappa scores of 0.72 and 0.75, respectively. We observe the same pattern at larger scale: DeepSeek-V3.1 and GPT-4o-2024-08-06 label 76.06% and 73.94% of these tasks, respectively, as changes with potential performance impact. To generate test suites that can effectively reveal the performance differences, we develop a multi-agent test-generation framework that produces tests intended to preserve correctness while increasing the computational load on performance-critical paths. Our framework adopts a batch-based multiagent workflow that iteratively generates, executes, diagnoses, and repairs performance-oriented tests before using the validated suite for performance evaluation. With the tests generated by our framework, more performant implementations become statistically distinguishable from their canonical solu-

tions: among 1,345 previously non-significant tasks, DeepSeek-v3.1 and GPT4o generated tests reveal statistically significant improvements in 323 (24.01%) and 342 (25.43%) tasks, respectively. When we re-evaluate implementations generated by GPT-4o-mini (OpenAI, 2024), Claude-Sonnet-4.5 (Anthropic, 2025b), and Gemini-2.5-Flash (Google, 2025), stronger tests reveal statistically significant performance improvements in 22.19% of the evaluated cases. Overall, our findings suggest that conclusions drawn from existing functionlevel Python performance benchmarks should be interpreted with caution. Reliable evaluation of LLM-generated code efficiency requires both statistically rigorous execution measurement and test suites that sufficiently stress performance-critical behavior. Our contributions are summarized as follows: – We revisit four function-level Python performance benchmarks using repeated execution, statistical testing, and effect-size analysis. – We investigate why benchmark-provided performant implementations are often not distinguishable from canonical solutions, combining manual analysis with a validated LLM-as-a-Judge procedure. – We develop a multi-agent framework that generates, diagnoses, and repairs performance-oriented tests to better stress performance-critical behavior. – We demonstrate that stronger performance-oriented test suites can substantially alter conclusions about the efficiency of LLM-generated code. Paper Organization. Section 2 introduces the background and motivation. Section 3 describes the study design and evaluation metrics. Section 4 presents the results for each research question. Section 5 discusses the findings and potential future research directions. Section 6 reviews related work. Section 7 examines threats to validity. Finally, Section 8 concludes the paper. 2 Background and Motivation In this section, we first introduce performance benchmark for LLM’s code generation. Then we discuss the challenges that motivate our study. 2.1 Performance Benchmark for LLM’s Code Generation A function-level performance benchmark evaluates whether an LLM can generate code that is both functionally correct and computationally efficient. Let a benchmark be a collection of tasks B = {τ1 , τ2 , . . . , τn }. Each task τ can be represented as τ = (dτ , Tτ , cτ , pτ ), where dτ is the natural-language problem description, Tτ is the benchmarkprovided set of executable test cases, cτ is the canonical solution, and pτ is the benchmark-provided performant implementation. Each test case in Tτ specifies the inputs and expected outputs used to verify functional correctness. The

BACKGROUND AND MOTIVATION

canonical solution serves as the baseline implementation for evaluating the runtime efficiency of LLM-generated code. The benchmark-provided performant implementation plays a different role: it is a reference artifact indicating that the task can be optimized and showing one possible optimized implementation. Let M denote the LLM-based code generator being evaluated. For each task, the benchmark asks M to synthesize an implementation gτ = M (dτ ). Only implementations that pass all the test cases in Tτ are considered for performance evaluation. Let Tb(f, τ ) denote the runtime reported by the benchmark after executing implementation f on Tτ . For LLM-generated code, the runtime comparison is made against the canonical solution. After a generated implementation gτ passes the functional tests, its runtime Tb(gτ , τ ) is compared with the canonical runtime Tb(cτ , τ ). The generated implementation is considered faster when its measured runtime is lower than the canonical runtime. The benchmarkprovided performant implementation pτ is not the target that generated code must strictly outperform. Instead, pτ is an optimized reference solution. Its role is to show that task τ has an intended optimization opportunity and to illustrate how the canonical solution can be improved. Therefore, the measured runtime difference between pτ and cτ should reflect a meaningful efficiency difference, with pτ running faster than cτ . If even this comparison does not show a reliable difference, then the benchmark may not be able to reveal performance improvements in LLM-generated code either.

2.2 Challenges in Building Performance Benchmarks for LLM-Generated Code We argue two major challenges exist when building reliable performance benchmarks for LLM-generated code. Challenge 1: Runtime Measurement Noise. Runtime measurements are inherently noisy. Even identical code executed on the same hardware can produce different runtimes across runs due to operating system scheduling, CPU cache state, and background process interference (Chen and Revels, 2016; Mogul and Borg, 1991; Tsafrir et al., 2005). The performance engineering community has long established that single-run comparisons are unreliable and commonly recommends at least 30 executions when comparing runtime performance (Arcuri and Briand, 2011). Nevertheless, benchmarks such as EffiBench and Mercury execute each implementation only once per task, making their evaluations vulnerable to measurement noise. Enamel and EvalPerf repeat ex-

2.2

Challenges in Building Performance Benchmarks for LLM-Generated Code

ecutions 6 and 5 times, respectively, which reduces measurement variability but still provides very limited support for rigorous statistical inference. Canonical solution

Performant implementation

def fib(self, n: int) -> int: if n == 1: return 1 if n <= 0: return 0 return self.fib(n-1) + self.fib( n-2)

def fib(self, n: int) -> int: if n == 0: return 0 elif n == 1: return 1 return self.fib(n-1) + self.fib( n-2)

Example 1: Changes that do not have performance impact. On the other hand, benchmark conclusions can be misleading in the presence of measurement noise. Example 1 shows a task (Task ID: 509) from Mercury. The code snippet at the right is labeled by the benchmark as the more efficient implementation. However, it only reorders conditional statements and uses elif, without changing the algorithm or its asymptotic runtime complexity. Consequently, neither implementation should be expected to exhibit a meaningful runtime advantage. The reported performance difference is therefore likely attributable to measurement noise rather than genuine efficiency gains. Indeed, when we executed both the canonical solution and the performant implementation 30 times, we observed no statistically significant runtime difference (Mann-Whitney U test p-value: 0.18, effect size: 0.1). Challenge 2: Insufficient Test Inputs. Even with nearly noise-free measurements, runtime comparisons are meaningful only when test inputs impose sufficient computational load to expose algorithmic differences. This is a fundamental challenge for current performance benchmarks, because many are derived from correctness-oriented code generation benchmarks such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). As a result, their test suites are designed to verify functional correctness rather than evaluate runtime efficiency. For example, suppose one implementation replaces an O(n3 ) algorithm with an O(n2 ) algorithm. When evaluated on a very small input (e.g., n = 4), both implementations complete within microseconds, making their runtime difference negligible despite the asymptotic improvement. In such cases, the benchmark cannot expose the optimization because the test inputs never exercise the performance-critical behavior. As a result, genuinely faster implementations may appear no different from their baseline counterparts. This is not because the optimizations are ineffective, but because the benchmark test inputs are insufficiently demanding. These challenges motivate us to first examine the reliability of existing function-level Python performance benchmarks. Before using these benchmarks to evaluate LLM-generated code, we ask whether the benchmark-provided performant implementations are distinguishable from the corresponding canonical solutions under the benchmark’s own test cases and a more rigorous measurement procedure. If they are not, the benchmark cannot meaningfully assess whether generated implementations are truly more efficient. Instead, the observed results may be driven by measurement noise or insufficiently demanding

STUDY DESIGN

test inputs rather than genuine performance differences, undermining conclusions about both benchmark quality and LLMs’ ability to generate efficient code.

3 Study Design 3.1 Benchmarks and Data We collect tasks from four performance-oriented function-level code generation benchmarks: EffiBench (Huang et al., 2024b), Enamel (Qiu et al., 2025), EvalPerf (Liu et al., 2024a), and Mercury (Du et al., 2024). We collected a total of 1,538 tasks. Table 1 shows the distribution. For each task, we obtain two implementations: the canonical solution and the benchmark-provided performant implementation.

Table 1: An overview of the benchmarks. Reported Runs shows the number of repeated executions per task reported by the corresponding paper. Benchmark

Task Source

#Tasks

Reported Runs

EffiBench Enamel EvalPerf Mercury

LeetCode HumanEval / HumanEval+ HumanEval / MBPP LeetCode

1,000 164 118 256

1 6 5 1

Total

–

1,538

–

The canonical solutions are provided by each benchmark’s task source. For example, the canonical solutions in Enamel (Qiu et al., 2025) and EvalPerf (Liu et al., 2024a) come from existing correctness-oriented datasets such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). EffiBench derives canonical solutions from the corresponding LeetCode discussion forum, consisting of user-submitted solutions that have been accepted by the platform. In contrast, Mercury constructs canonical solutions from historical accepted submissions for each task. In terms of collecting the performant implementations, for EffiBench, the performant implementations are generated by GPT 3.5 Turbo in their paper. For Mercury (Du et al., 2024), performant implementations are derived by aggregating historical LeetCode submissions to construct a runtime distribution and identify faster solutions. In Enamel (Qiu et al., 2025) and EvalPerf (Liu et al., 2024a), the performant implementations are human-optimized versions of the corresponding canonical solutions.

3.2

Execution Procedure and Statistical Analysis

3.2 Execution Procedure and Statistical Analysis For each task, we execute both the canonical solution and the benchmarkprovided performant implementation using the test inputs supplied by the benchmark. During each run, we record the execution time of the program. To reduce the impact of transient system noise, such as operating system scheduling, CPU cache effects, and background processes (Chen and Revels, 2016; Mogul and Borg, 1991; Tsafrir et al., 2005), each implementation is executed 30 times while keeping inputs and environmental conditions constant. This follows prior performance measurement research that relies on repeated executions to reduce unreliable single-run comparisons (Arcuri and Briand, 2011; Zeng et al., 2019; Su et al., 2019; Liao et al., 2025; Jangali et al., 2023). These repeated runs produce independent runtime samples for the canonical and performant implementations, allowing statistical analysis to assess whether observed runtime differences are statistically significant and not attributable to measurement noise. Statistical Hypothesis Test. Following the stability-oriented practice used in prior software performance research (Arcuri and Briand, 2011), we do not rely on aggregate execution-time summaries alone to decide whether an implementation is faster. Instead, for each task, we compare the full runtime samples collected from repeated executions of the canonical solution and the benchmark-provided performant implementation. Before applying any statistical test, we first check functional correctness and execution validity. If the benchmark-provided performant implementation fails to pass at least the same test cases as the canonical solution, or if any execution times out, we treat the task as not statistically significant for performance improvement. In other words, such tasks are assigned to the non-significant category, equivalent to failing the p < 0.05 decision criterion. For tasks with valid executions, We use the Mann-Whitney U test (Mann and Whitney, 1947), a non-parametric test that does not assume normally distributed runtimes and is suitable for comparing two independent runtime samples. The null hypothesis is that the performant implementation is not faster than the canonical solution, while the alternative hypothesis is that its runtime distribution is shifted toward lower execution times. We consider a task statistically significant only when the test yields p < 0.05. To analyze performance differences across tasks at the benchmark level, we compute one average runtime for each implementation on each task and apply a one-sided Wilcoxon signed-rank test (Wilcoxon, 1945; Sidney, 1957) to these paired pertask averages. This benchmark-level test is non-parametric and is well suited for paired measurements across tasks (Demšar, 2006), allowing us to examine whether performant implementations are consistently faster than canonical solutions within each benchmark. Effect Size. To avoid overinterpreting statistically significant but practically negligible differences (Kampenes et al., 2007), we measure effect size using Cliff’s delta (Cliff, 1993). We interpret the magnitude of δ using standard

RESULTS

thresholds: negligible (δ < 0.147), small (0.147 ≤ δ < 0.33), medium (0.33 ≤ δ < 0.474), and large (δ ≥ 0.474). 3.3 Experimental Environment All experiments were conducted on a desktop machine equipped with an Intel® Core™ i5-8500 CPU @ 3.00 GHz (6 physical cores, 6 logical threads, 9 MB L3 cache). To reduce runtime measurement noise, we ran only one benchmark task at a time on the machine and ensured that no other programs or processes were running during test execution. We used the same execution timeout for all evaluated implementations: each test run was terminated if it exceeded 300 seconds. For LLM-as-a-Judge in RQ2 and test generation in RQ3, we used DeepSeek-V3.1 (DeepSeek, 2025a) and GPT-4o (gpt-4o-202408-06) (OpenAI, 2025). These models are selected for their strong reasoning capabilities and accessible APIs. For all LLM calls, we set the temperature to 0 to reduce the randomness and improve replicability. 4 Results 4.1 RQ1: Can Existing Benchmarks Reliably Expose Performance Differences? Motivation. The reliability of existing benchmarks remains unclear. This research question aims to establish a fair and robust runtime performance measurement framework and rigorously evaluate whether the benchmark-provided performant implementations are significantly faster than the corresponding canonical solutions. Approach. For each of the 1,538 tasks, we execute both the canonical solution and the performant implementation 30 times using the benchmark-provided test suites. We record the runtime of each run and compute the mean for both the canonical solution and the performant implementation. For each task, we further apply Mann-Whitney U test and compute Cliff’s delta on the runtime measurements. Results. Table 2 summarizes the results across all four benchmarks. Across all 1,538 tasks, only 94 (6.11%) tasks’ performant implementations are significantly faster than the corresponding canonical solutions, while the remaining 1,444 (93.89%) tasks show no statistically significant difference. This pattern is consistent across EffiBench, Enamel, and Mercury where over 93% of tasks in each benchmark show no statistically significant difference. Among the tasks with p < 0.05, most exhibit large effect sizes (92 cases, 5.98%), with only 1 small (0.07%) and 1 medium (0.07%) case, and no negligible effects observed. Three out of four benchmarks (i.e., EffiBench, Enamel, and Mercury) have fewer than 10% tasks where the performance differences are statistically significant. EvalPerf has the highest rate of detectable performance improvement

4.2

RQ2: Do Code Changes in Performant Implementations Have Real Impact?

Table 2: Mann-Whitney U test results across the four benchmarks under 30run execution. Tasks with p ≥ 0.05 are considered non-significant. For tasks with p < 0.05, we further categorize the effect sizes using Cliff’s delta (δ): negligible (δ < 0.147), small (0.147 ≤ δ < 0.33), medium (0.33 ≤ δ < 0.474), and large (δ ≥ 0.474). # Tasks

p ≥ 0.05 (# %)

Neg.

Small

EffiBench Enamel EvalPerf Mercury

1,000 164 118 256

944 (94.4%) 153 (93.29%) 102 (86.44%) 245 (95.7%)

0 0 0 0

0 1 (0.61%) 0 0

0 56 (5.6%) 1 (0.61%) 9 (5.49%) 0 16 (13.56%) 0 11 (4.3%)

Total

1,538

1,444 (93.89%)

1 (0.07%)

Benchmark

Effect Size δ (p < 0.05) Medium

Large

92 (5.98%)

(13.56%). This may be attributed to the fact that EvalPerf was specifically designed to select performance-challenging tasks and generate computationally expensive inputs. However, a large portion of EvalPerf’s tasks (86.44%) still remain non-statistically significant. We also compare the average mean runtime differences across tasks for each benchmark and apply the Wilcoxon signed-rank test to examine statistical differences. Across all benchmarks, over 93% of tasks do not exhibit statistically significant differences between performant implementations and canonical solutions, with p-values often close to 1. When restricting the analysis to the small subset of statistically significant tasks (p < 0.05), the performant implementations are on average faster than the canonical solutions, with mean improvements ranging from +6.6% to +27.1% and p-values below 0.001 with medium to large effect sizes. RQ1-Takeaway. Under repeated execution (30 runs) and statistical testing, over 93% of tasks show no statistically significant performance differences between performant implementations and canonical solutions. Only a small fraction (6.11%) exhibit measurable improvements, indicating that most benchmark-provided performant implementations do not yield actual performance gains under existing test suites.

4.2 RQ2: Do Code Changes in Performant Implementations Have Real Impact? Motivation. RQ1 establishes that 93.89% of benchmark tasks’ performant implementations are not statistically significantly faster than the corresponding canonical solutions. This raises a fundamental question: do benchmarkprovided performant implementations, change throughout implementations actually involve code changes with performance impact?

RESULTS

Approach. To investigate this research question, we first performed open coding on a subset sampled from the 1,444 non-significant tasks identified RQ1 to identify the code change patterns between the canonical solution and the corresponding performant implementation. We used stratified sampling (Neyman, 1992; Baltes and Ralph, 2022) to preserve representativeness across benchmarks and sampled 308 tasks. The sample size is statistically significant, with a 95% confidence level with a 5% margin of error. Before performing open coding, two authors met and reviewed several pairs of the canonical solution and the performant implementation together. At this stage, we found that code change patterns could be grouped into three categories: (1) changes without performance impact, (2) changes with performance impact, and (3) no changes (i.e., the canonical solution and the performant implementation are identical). For each sampled task, two authors then independently compared the canonical solutions and the performant implementations. They were asked to classify each task into one of the three aforementioned categories. In addition, they were tasked with summarizing and documenting the code change patterns in greater detail. These two authors held multiple meetings to develop a codebook and resolve labeling inconsistencies. Eventually, in addition to the group of no changes, two themes were identified for changes without performance impact, and two themes were identified for changes with performance impact. The final Cohen’s Kappa (Cohen, 1960) is 0.93, indicating an almost perfect agreement (Landis and Koch, 1977). The whole labeling process took around 1200 person hours. LLM-as-a-Judge. After establishing the root-cause categories through manual analysis, we use LLM-as-a-Judge as a secondary validation step. Its role is not to replace the manual findings, but rather to confirm whether the discovered categories can be reliably recognized by strong language models and to extend the analysis to the remaining 1,136 non-significant tasks across the four benchmarks. Recent studies (Ahmed et al., 2025; Li et al., 2024a; Zhang et al., 2024b) suggest that LLM-based evaluators can achieve agreement levels comparable to human annotators in structured classification tasks. Motivated by this, we design a Chain-of-Thought prompting strategy that guides the model through a structured comparison of canonical solutions and benchmark-provided performant implementations. For each task, the model is asked to reason about the differences (if any) between the two implementations in terms of algorithmic strategy, data structure choices, implementation-level optimizations, and input-dependent behavior, and then group the task into one of the themes we identified through manual analysis (art, 2026). We evaluated two judge models, gpt-4o-2024-08-06 (OpenAI, 2025) and deepseek-v3.1 (DeepSeek, 2025a), with the temperature set to zero in all runs to minimize randomness and improve consistency. To assess the reliability of LLM-as-a-Judge, we first applied LLMs to classify the 308 manually labeled tasks. Then we computed Cohen’s Kappa (Cohen, 1960) to measure the interrater agreement between the LLM and the human labelers. The scores are

4.2

RQ2: Do Code Changes in Performant Implementations Have Real Impact?

Table 3: Code change patterns between the canonical solutions and benchmark-provided performant implementations. Change pattern Subcategory

Manual (308)

LLM-as-a-Judge (1,136) DeepSeekv3.1

GPT-4o2024-08-06

Changes without performance impact

84 (27.27%) 204 (17.96%) 218 (19.19%)

Built-in function substitution Refactoring

4 (1.3%) 80 (25.97%)

Changes with performance impact Data structure replacement

Algorithm strategy change

No changes

Array to Hash Map / Set Array to Heap / Priority Queue Sorting-Based Optimization Dynamic Programming Graph Traversal (BFS/DFS) Binary Search Mathematical Optimization Prefix Sum Sliding Window / Two Pointers Control-Flow Restructuring Slice-Based Vectorization Bit Manipulation Stack-Based Approach Other

0 (0%) 204 (17.96%)

1 (0.09%) 217 (19.1%)

209 (67.86%) 864 (76.06%) 840 (73.94%) 24 (7.79%)

39 (3.43%)

56 (4.93%)

2 (0.65%)

6 (0.53%)

8 (0.7%)

22 (7.14%)

148 (13.03%)

132 (11.62%)

34 (11.04%)

121 (10.65%)

99 (8.71%)

26 (8.44%)

92 (8.1%)

119 (10.48%)

11 (3.57%)

36 (3.17%)

43 (3.79%)

19 (6.17%)

105 (9.24%)

95 (8.36%)

4 (1.3%)

37 (3.26%)

29 (2.55%)

12 (3.9%)

99 (8.71%)

73 (6.43%)

22 (7.14%)

23 (2.02%)

34 (3%)

12 (3.9%)

28 (2.46%)

42 (3.7%)

11 (3.57%)

54 (4.75%)

49 (4.31%)

4 (1.3%)

40 (3.52%)

28 (2.46%)

6 (1.95%)

36 (3.17%)

33 (2.9%)

15 (4.87%)

68 (6%)

78 (6.9%)

0.75 and 0.72 with GPT-4o-2024-08-06 and DeepSeek-v3.1, respectively, indicating substantial agreement (Landis and Koch, 1977). These results indicate that LLM-as-a-Judge can reliably reproduce manual labeling. We then applied LLM-as-a-Judge to label the remaining 1,136 tasks. Results. Table 3 presents the code change patterns we identified through our manual analysis and the distribution across the dataset. We elaborate on four code change patterns we identified. Changes without performance impact. 84 out of the 308 tasks were classified with changes that do not likely to have impact on runtime performance. We identified two specific patterns:

RESULTS

– Built-in function substitution replaces a manual implementation or library call with an equivalent built-in function that provides the same functionality without changing performance behavior. For example, replacing the manual loops with built-in sum, max, count for arrays. 4 out of the 308 tasks belong to this group. In these cases, both program semantics and asymptotic complexity are preserved, the runtime performance should not have significant differences. – Refactoring include changes that improve readability or organization (e.g., renaming, reordering conditions, and restructuring control flow) without modifying semantics or performance behavior. For example, renaming variables such as balance to b or max reachable to mx, or rewriting loops from for i in range(len(nums)) to for x in nums. We identified 80 out of the 308 tasks involves such refactoring. These changes have no performance impact, as complexity remains unchanged. Changes with performance impact. 209 out of the 308 tasks involves changes that may have impact on runtime performance. Specifically, the following two patterns were found: – Algorithm strategy change includes changes that replace one algorithmic approach with another that significantly changes execution behavior or reduces computational complexity. For example, a brute-force enumeration may be replaced with a sortingbased approach that avoids repeatedly exploring the same search space. In our manual analysis, 183 out of the 308 tasks (59.42%) include algorithm strategy changes. The detailed subcategories in Table 3 show that these changes cover a range of optimization strategies. Dynamic programming and graph traversal are the two most common subcategories, appearing in 34 tasks (11.04%) and 26 tasks (8.44%), respectively. The remaining cases are distributed across sorting-based optimization and control-flow restructuring, each with 22 tasks (7.14%); mathematical optimization, with 19 tasks (6.17%); sliding window or two-pointer techniques and slice-based vectorization, each with 12 tasks (3.9%); binary search and bit manipulation, each with 11 tasks (3.57%); and prefix sums and stack-based approaches, each with 4 tasks (1.3%). The Other subcategory contains 6 tasks (1.95%) whose algorithmic changes do not belong to any of the listed categories. Although these changes may have performance impact, the benchmark-provided test cases with inputs are often not effective enough to distinguish the runtime differences between the canonical and performant implementations. – Data structure replacement includes changes where the performant implementation uses a different data structure to reduce the cost of repeated operations, such as replacing linear scans over arrays with hash-based lookup using maps or sets, or using heaps or priority queues to manage ordered elements more efficiently. 26 out of the 308 manually analyzed tasks (8.44%) include data structure replacements. Among these tasks, replacing arrays with hash maps or sets is substantially more common than replacing arrays with heaps or priority

4.2

RQ2: Do Code Changes in Performant Implementations Have Real Impact?

queues: we observe 24 array-to-hash-map/set replacements (7.79%) and 2 array-to-heap/priority-queue replacements (0.65%). However, their runtime benefits can remain hidden when the benchmark-provided test cases with inputs are not large or complex enough for these operations to dominate execution time. Surprisingly, we also found 15 tasks where the canonical solution and the performant implementation are identical. These results indicate that further data cleaning could be essential for existing benchmarks. Our findings are further supported by the expanded analysis using LLM-asa-Judge on all remaining 1,136 non-significant tasks. 17.96% and 19.19% of the tasks are classified as changes without performance impact by DeepSeek-v3.1 and GPT-4o-2024-08-06, respectively. Refactoring remains the most frequent change pattern that is unlikely to have a performance impact, identified in 17.96% of tasks by DeepSeek-v3.1 and 19.19% by GPT-4o-2024-08-06, confirming that this pattern is pervasive across the full dataset. In terms of tasks involving changes that may have performance impact, 72.1% and 68.31% of the tasks are identified as involving algorithm strategy changes by DeepSeekv3.1 and GPT-4o-2024-08-06, respectively. 3.96% and 5.63% of the tasks are identified as involving data structure replacements, respectively. These results indicate that the performance implementations for the majority of tasks in existing benchmarks include changes with potential performance impact. However, these changes do not result in statistically significant runtime differences under the current evaluation settings RQ2-Takeaway 1. Both manual analysis and LLM-as-a-Judge analysis confirm that about 21.47% of the tasks in existing benchmarks do not include meaningful performance changes compared to the canonical solutions, while about 72.62% involve potential performance improvements that do not result in statistically significant runtime differences under the current evaluation settings. Canonical solution O(n3 )

Performant implementation O(n2 )

def triples_sum_to_zero(l): for i in range(len(l)): for j in range(i + 1, len(l)): for k in range(j + 1, len(l)): if l[i] + l[j] + l[k] == 0: return True return False

def triples_sum_to_zero(l): for i, x in enumerate(l[:-2]): buf = set() for y in l[i + 1:]: if y in buf: return True buf.add(-x - y) return False

Example 2: Algorithm strategy changes improve time complexity from O(n3 ) to O(n2 ) To understand why current evaluation settings cannot reliably expose performance differences between performant implementations and canonical solutions, we reviewed the corresponding test suites for these tasks. We found

RESULTS

that the benchmark-provided test suites are often too small. For instance, the performant implementation of Task 40 in the Enamel benchmark includes an algorithm strategy change (Example 2). A set-based approach (O(n2 )) is used in the performant implementation instead of a triple-loop approach (O(n3 )) in the canonical solution. However, the length of the longest input array in the benchmark-provided test suite is 6, which is insufficient to distinguish the performance differences between the two implementations. Canonical solution O(n2 )

Performant implementation O(n)

def frequency_lists(list): list = [item for sublist in list for item in sublist] return { x: list.count(x) for x in list }

def frequency_lists(nested_lists): flat_list = list(chain(*nested_lists) ) return dict( Counter(flat list) )

Example 3: Replacing inefficient list.count with Counter The performant implementation of Mbpp/97 in the EvalPerf benchmark demonstrates a data structure change (Example 3). The canonical solution repeatedly counts element frequencies using list.count, resulting in quadratic time complexity (O(n2 )). In contrast, the performant implementation replaces this approach with a hash-based aggregation using Counter, reducing the complexity to linear time (O(n)). However, the benchmark-provided test suite contains only small nested lists, which are insufficient to expose the performance gap between the two implementations. RQ2-Takeaway 2. A large portion (more than 60%) of the tasks’ performant implementations introduce substantive algorithmic or data structure changes that should yield runtime gains. However, these changes do not result in statistically significant runtime differences because the benchmarkprovided test suites do not include input sizes at which such changes affect execution time. These findings indicate that the test suites of existing benchmarks should be improved to cover input sizes that make the expected runtime differences measurable

4.3 RQ3: Can Multi-Agent-Based Test Generation Reveal Hidden Performance Differences? Motivation. In RQ2, we found that the test suites of existing benchmarks may be further improved. In this RQ, we investigate whether and to what extent a multi-agent-based approach can enhance the test suites to better detect such performance gains. Approach. We develop a multi-agent framework to automatically generate performance-oriented test suites, as illustrated in Figure 1. The framework

4.3

RQ3: Can Multi-Agent-Based Test Generation Reveal Hidden Performance Differences? 17

Problem Description

TEST GENERATION AGENT

EXECUTE

Generate candidate test in batches

Candidate Tests

Canonical Solution

No Yes No

Failed Tests

Failed Logs

Pass?

Yes

All batches?

Pass?

Yes PERFORMANT TESTS

No BATCH REFINEMENT LOOP (MAX 3 ITERATIONS)

EXECUTE

VALIDATION LOOP (UP TO 5 ITERATIONS)

TEST REPAIR AGENT

TEST DIAGNOSIS AGENT

Fixed Tests STRUCTURE DIAGNOSTIC

Fig. 1: Overview of our multi-agent framework for generating and repairing performance-oriented test suites.

contains three agents: The Test Generation Agent, The Diagnosis Agent, and The Repair Agent. The Test Generation Agent receives the problem description and the code under test, then creates deterministic tests that preserve the intended behavior while using inputs that can expose runtime differences. The Test Diagnosis Agent is used when generated tests fail. Its role is to reason over the error log and convert the full traceback into a compact structured root-cause diagnostic, including the error type and message, the root cause of the failure, and the relevant line information. This diagnostic reduces the log text included in the LLM prompt for the Test Repair Agent and provides clearer evidence for fixing the failed tests. The Test Repair Agent then uses the code, failed-test identifiers, and diagnostics to repair the failed tests. Both DeepSeek-V3.1 (DeepSeek, 2025a) and GPT-4o-2024-08-06 (OpenAI, 2025) have a context window of 128,000 tokens. Therefore, we generate ten tests for each task in two batches of five rather than in a single request. This design bounds the amount of generated code, failure logs, and repair information included in each prompt while preserving sufficient space for the model’s response. For each batch, the Test Generation Agent produces five candidate tests, which are then executed against the canonical implementation. If some tests fail, the framework sends only the failing tests from that batch, the code under test, and the corresponding error logs to the Test Diagnosis Agent. This agent analyzes each failure to identify the error type and message, determine its root cause, and locate the relevant failing line in the generated test. It

RESULTS

Prompt Template: Test Generation (Simplified) Task: Generate exactly y standalone performance-oriented test functions for the current batch. Instructions: 1. Read the problem description and the code. 2. Create deterministic inputs that preserve the intended behavior but make inefficient code run longer. 3. Reason about the expected correct behavior independently. 4. Avoid timing assertions, I/O, randomness inside loops, external libraries, comments, and explanations. Output: JSON with a testcase field containing the generated Python test functions.

Prompt Template: Test Diagnosis (Simplified) Task: Analyze the failure of a generated test. Instructions: 1. Read the failed test function, the code under test, and the error log. 2. Identify the error type and error message from the log. 3. Identify the root cause in the generated test using the evidence from the failure. 4. Reason about the traceback to identify the failing line number and line content from the test function. 5. Do not generate code and do not output the full traceback. Output: JSON with error, root cause, line number, and line content.

Prompt Template: Test Repair (Simplified) Task: Repair or regenerate only the failed generated tests. Instructions: 1. Read the code under test, failed tests, and structured diagnostics. 2. Infer the intended behavior from the code and use the diagnostics to locate each test problem. 3. Fix invalid expected values, invalid inputs, nondeterminism, or timeout-prone inputs. 4. Preserve the original failed test names. 5. Do not modify the implementation code; avoid classes, external libraries, comments, explanations, stack traces, and large diffs. Output: JSON with a testcase field containing the repaired Python test functions.

Fig. 2: Simplified prompt templates used by the Test Generation, Test Diagnosis, and Test Repair Agents.

then converts this information into a compact structured diagnostic. The Test Repair Agent uses these diagnostics to correct the expected outputs, inputs, or input sizes of the failed tests. Restricting diagnosis and repair to the failed tests further controls prompt length and avoids repeatedly including the full generated suite. After repair, the batch is re-executed. This diagnose-andrepair loop is repeated for up to three refinement iterations per batch and stops early when all five tests pass. After both batches are processed, the framework executes the combined suite of ten generated tests. If any test still fails, it repeats the batch-level generation and refinement process for only the failing tests, again providing the LLM with summarized diagnostics rather than the complete failure history. This full-suite validation loop is allowed up to five iterations and stops

4.3

RQ3: Can Multi-Agent-Based Test Generation Reveal Hidden Performance Differences? 19

when all generated tests pass or the iteration limit is reached. If the iteration limit is reached while some tests still fail, the framework filters out the failing tests and keeps only those that pass validation. The final output for each task is therefore a validated set of executable, deterministic, performance-oriented tests that can be used to re-measure the canonical and performant implementations under the same statistical procedure as RQ1. Our simplified prompt templates can be found in Fig. 2 We evaluate our framework against COFFE (Peng et al., 2025b), a SOTA LLM-based performance test generation for function-level Python programs in two stages. First, we apply both approaches to the 209 tasks from the manual analysis that were confirmed to contain code changes with performance impact but yielded no significant results under the benchmark-provided test suites. This stage directly measures how many previously hidden performance gains each approach can reveal through its generated performance-oriented tests. We then extend the evaluation to the remaining 1,136 non-significant tasks to assess their effectiveness across the dataset. Table 4: Two-stage evaluation of generated test suites on tasks that were non-significant under the benchmark-provided test suites, broken down by approach and benchmark. Stage 1 evaluates the 209 manually verified tasks whose code changes were confirmed to have performance impact. Stage 2 evaluates the remaining 1,136 non-significant tasks. We report the number of tasks that become statistically significant (p < 0.05) under each generated test suite. All detected significant cases correspond to large effect sizes (δ ≥ 0.474). Stage

Benchmark #Tasks

Manual Verified

EffiBench Enamel EvalPerf Mercury Total

Our approach DeepSeek

138 18 19 34 209

GPT-4o

COFFE DeepSeek

GPT-4o

51 (36.96%) 63 (45.65%) 3 (2.17%) 0 (0%) 14 (77.78%) 13 (72.22%) 0 (0%) 0 (0%) 14 (73.68%) 12 (63.16%) 0 (0%) 1 (5.26%) 15 (44.12%) 14 (41.18%) 1 (2.94%) 0 (0%) 94 (44.98%) 102 (48.80%) 4 (1.91%) 1 (0.48%)

EffiBench Enamel Remaining EvalPerf Mercury Total

744 108 (14.52%) 115 (15.46%) 15 (2.01%) 0 (0%) 120 72 (60%) 69 (57.5%) 0 (0%) 0 (0%) 78 23 (29.49%) 26 (33.33%) 0 (0%) 5 (6.41%) 194 26 (13.40%) 30 (15.46%) 8 (4.12%) 0 (0%) 1,136 229 (20.16%) 240 (21.13%) 23 (2.02%) 5 (0.44%)

Overall

1,345 323 (24.01%) 342 (25.43%)

Total

27 (2%)

6 (0.45%)

Results. Table 4 shows that our framework consistently enables more tasks to exhibit statistically significant performance differences than COFFE. On the 209 manually verified tasks, our DeepSeek-v3.1 and GPT-4o test suites yield statistically significant performance differences for 94 (44.98%) and 102 (48.80%) tasks, respectively. Under the same models, COFFE achieves this outcome for only 4 (1.91%) and 1 (0.48%) tasks. Thus, for the subset in which

RESULTS

the code changes are known to affect performance, our framework reveals substantially more of the performance differences that were not observable with the benchmark-provided test suites. The same pattern holds for the remaining 1,136 tasks. Our framework yields statistically significant performance differences for 229 (20.16%) tasks with DeepSeek-v3.1 and 240 (21.13%) with GPT-4o, compared with 23 (2.02%) and 5 (0.44%) for COFFE. Overall, across all 1,345 tasks, our framework achieves statistically significant results for 323 (24.01%) and 342 (25.43%) tasks, whereas COFFE does so for only 27 (2.00%) and 6 (0.45%), respectively. All statistically significant cases have large effect sizes (δ ≥ 0.474). These results indicate that our multi-agent generation, diagnosis, and repair process produces performance-oriented test suites that expose performance differences substantially more effectively than COFFE in our Python function-level setting. Canonical solution

Performant implementation

def lengthOfLongestSubstring(s): ss, i, ans = set(), 0, 0 for j, c in enumerate(s): while c in ss: ss.remove(s[i]); i += 1 ss.add(c) ans = max(ans, j - i + 1) return ans

def lengthOfLongestSubstring(s): idx, start, ans = {}, 0, 0 for i, char in enumerate(s): if char in idx: if idx[char] >= start: start = idx[char] + 1 idx[char] = i ans = max(ans, i - start + 1) return ans

# Original tests - small inputs assert lengthOfLongestSubstring('uIxsZwqW2u') == 9 # Generated tests - large inputs assert lengthOfLongestSubstring('ab' * 500000) == 2 assert lengthOfLongestSubstring('a' * 1000000) == 1

Example 4: Our generated tests effectively expose performance differences with larger inputs Task 3 of EffiBench illustrates how our generated tests can reveal a performance difference that is not statistically significant under the benchmarkprovided tests. The performant implementation uses index jumps instead of incremental window shrinking (Example 4). However, the benchmark inputs are too short (the maximum input-string length is 10), causing the two implementations to exhibit similar execution times. In contrast, our generated tests use much larger inputs (e.g., a string of length 500,000), making the performance advantage of index jumps observable. A key limitation of our approach arises when handling programs with multiple nested functions, where the agent may misidentify the target function for testing. As shown in Example 5, the canonical solution defines sum Of product as the entry-point, which calls binomial Coeff. However, the agent incorrectly targets binomial Coeff during test generation, as it appears to contain the core logic. This leads to a mismatch when evaluating the performant implementation, where the computation is implemented directly within sum Of product

4.4

RQ4: How do performance-oriented test suites affect the benchmark results?

and binomial Coeff no longer exists. As a result, the generated tests become incompatible, preventing meaningful performance comparison. This issue highlights the agent’s limitations in handling functional interfaces and suggests the need for improved function-level reasoning to correctly identify and target the intended entry-point. Canonical solution O(n2 )

Performant implementation O(n)

def binomial_Coeff(n, k): C = [0] * (k + 1) C[0] = 1 for i in range(1, n + 1): for j in range(min(i, k), 0, -1): C[j] = C[j] + C[j - 1] return C[k]

def sum Of product(n): binomial, binomial[0] = [0] * (n + 1) , 1 total = 0 for i in range(1, n + 1): binomial[i] = binomial[i-1] * (n +1-i) // i total += binomial[i-1] * binomial[ i] return total

def sum Of product(n): return binomial_Coeff(2*n, n-1) # Original test cases assert sum Of product(3) == 15 # Generated test cases - wrong function expected = 8965199470901314966...0932000 assert binomial Coeff(200, 99) == expected # Fixed generated tests assert sum_Of_product(100) == expected

Example 5: Our generated tests fail to expose performance differences

RQ3-Takeaway. Our generated tests uncover 44.98% and 48.80% of previously missed performance differences on the manually verified subset with performance impact, and 24.01% and 25.43% on the overall dataset, using DeepSeek-v3.1 and GPT-4o, respectively. These results demonstrate the effectiveness of our approach in generating performance-oriented test suites.

4.4 RQ4: How do performance-oriented test suites affect the benchmark results? Motivation. In RQ3, we investigated whether a multi-agent approach can generate test suites under which more canonical and performant implementations exhibit statistically significant runtime differences. In this RQ, we examine how these performance-oriented test suites influence the evaluation results of LLM-generated code on the benchmarks. Approach. To assess how stronger test suites affect benchmark evaluation results, we compare LLM generated implementations with the original implementations using both the benchmark provided test suites and the performanceoriented test suites introduced in RQ3. For each task, we consider performant

Benchmark #Tasks

1,000 164 118 256 1,538 1,000 164 118 256 1,538 1,000 164 118 256 1,538

EffiBench Enamel EvalPerf Mercury Total

Model

GPT-4o-mini

EffiBench Enamel Claude-Sonnet-4.5 EvalPerf Mercury Total

EffiBench Enamel Gemini-2.5-Flash EvalPerf Mercury Total

961 154 103 240 1,458

916 154 106 212 1,388

947 157 108 230 1,442

p ≥ 0.05

0 0 0 0 0

0 0 0 1 1

39 10 15 16 80

84 10 12 44 150

53 7 10 25 95

Neg. Small Med. Large

Effect size (p < 0.05)

Original Tests

882 108 70 204 1,264

555 95 66 142 858

777 111 81 173 1,142

p ≥ 0.05

0 0 0 0 0

0 0 1 0 1

0 0 0 0 0

2 1 0 0 3

118 56 48 52 274

445 69 52 114 680

221 52 36 83 392

Neg. Small Med. Large

Effect size (p < 0.05)

DeepSeek-v3.1 Tests

888 111 73 204 1,276

546 92 70 132 840

772 114 80 170 1,136

p ≥ 0.05

0 0 0 0 0

1 0 0 0 1

112 53 45 52 262

454 72 48 124 698

227 50 38 86 401

Neg. Small Med. Large

Effect size (p < 0.05)

GPT-4o Tests

Table 5: Statistical analysis results across four benchmarks under repeated execution (30 runs) for three LLM-generated code models. For each task, we apply the Mann–Whitney U test. Tasks with p ≥ 0.05 are considered not statistically significant. For tasks with p < 0.05, effect sizes are categorized using Cliff’s delta (δ): negligible (δ < 0.147), small (0.147 ≤ δ < 0.33), medium (0.33 ≤ δ < 0.474), and large (δ ≥ 0.474).

22 4 RESULTS

implementations generated by three LLMs: GPT-4o-mini (OpenAI, 2024), Claude-Sonnet-4.5 (Anthropic, 2025b), and Gemini-2.5-Flash (Google, 2025), and evaluate them alongside canonical solutions under these test suites. Following the statistical evaluation protocol used in RQ3, we run each implementation 30 times on each test suite and collect runtime measurements across runs. We then apply the same statistical testing method to determine whether the LLM generated implementation shows a statistically significant performance improvement over the original implementation. By comparing the results obtained from the benchmark tests and the performance-oriented tests across all tasks, we analyze how stronger test suites influence the measured efficiency improvements and the overall benchmark evaluation outcomes. Results. Table 5 summarizes the results across all four benchmarks for the three LLM-generated models under different test suites. Under the benchmarkprovided test suites (Original), all three models show a similar pattern where most tasks do not exhibit statistically significant performance differences. This includes 1,442 (93.76%) tasks for GPT-4o-mini, 1,388 (90.25%) for ClaudeSonnet-4.5 and 1,458 (94.8%) for Gemini-2.5-Flash. Among the significant cases (p < 0.05), almost all exhibit large effect sizes while negligible and small effects are nearly absent. When evaluated with stronger test suites, the number of statistically distinguishable performance improvements increases substantially. Under DeepSeekv3.1-generated tests, the number of significant cases increases to 396 (25.75%), 680 (44.21%), and 274 (17.82%) for GPT-4o-mini, Claude-Sonnet-4.5, and Gemini-2.5-Flash, respectively. Under GPT-4o–generated tests, the number of significant cases increases to 402 (26.14%), 698 (45.38%), and 262 (17.03%), respectively. Notably, Claude-Sonnet-4.5 shows a larger improvement compared to the other models. This may be attributed to the fact that Claude-Sonnet-4.5 usually has a longer and more complete reasoning process, which could enable it to generate more effective optimizations that are better exposed by stronger test suites. However, we also notice that many tasks still remain non-significant across all models, indicating the generation of performant implementations remain challenging for LLMs. RQ4-Takeaway. Modern LLMs are capable of generating performant implementations. Yet, their performance improvements could be ignored by the weak test suites provided by existing benchmarks. With our enhanced test suites generated in RQ3, an average of 22.19% performance improvements can be further confirmed as statistically distinguishable.

5 Discussion and Future Work In this section, we discuss the implications of our empirical findings on existing performance benchmarks and outline future research directions motivated by the limitations observed in our study.

DISCUSSION AND FUTURE WORK

Performance-Aware Benchmark Design. Our findings in RQ1 suggest that many tasks in existing efficiency benchmarks fail to reliably distinguish performance differences between canonical and performant implementations. Although these benchmarks aim to evaluate efficiency, most reported performance improvements are not statistically significant under repeated execution. This may be largely attributed to the fact that tasks of these benchmarks are adopted directly from datasets that prioritize functional correctness rather than exploring algorithm strategy’s efficiency. Our study highlights several principles that should be considered when designing benchmarks for evaluating the performance of LLM-generated code. First, input scaling and diversity should be explicitly designed to expose algorithmic differences. Tasks should include sufficiently large and varied inputs so that implementations with different time or space complexities exhibit measurable performance gaps. Second, evaluation methods should incorporate robust measurement practices, such as repeated executions, statistical significance testing, and controlled runtime environments, to reduce measurement noise. Third, benchmarks should consider multiple performance dimensions beyond runtime, including memory consumption, scalability across increasing input sizes, and the stability of performance across runs. Finally, task design itself may also emphasize performance-sensitive workloads, such as scenarios in which different algorithmic strategies lead to substantially different asymptotic or constant-factor behavior. While our work later demonstrates that enhanced test suites can reveal relatively more hidden efficiency differences, designing benchmarks that systematically incorporate these principles remains an open challenge. Without such advances, current benchmarks risk continuing to misrepresent the true performance of LLM-generated code. Strengthening Test Suites Beyond Correctness. Our findings in RQ2 show that existing benchmark test suites are primarily designed to validate functional correctness, with input sizes that are often too small to reveal meaningful performance differences. As a result, many performance improvements remain hidden under the default tests. This highlights the need for test generation approaches that explicitly target performance-critical execution paths. Our investigation in RQ3 further supports this finding. By incorporating more powerful test suites that are generated to stress the program, we can reveal an average of 32.66% more performance differences between canonical and performant implementations. While our multi-agent approach improves test diversity, it is limited by using LLM-based agents. In some cases, the generated tests may fail to exercise worst-case execution behavior due to the non-deterministic nature of LLMs, and may also misidentify the main function in programs with nested functions, leading to correctness issues. Future work may focus on guided input generation that targets specific complexity patterns, such as inputs that trigger worst-case behavior or workloads that scale progressively to reveal asymptotic differences (Lemieux et al.,

2018; Petsios et al., 2017). In addition, agents could incorporate profiling feedback to identify hot execution paths and generate inputs that repeatedly exercise those paths. (Ye et al., 2025) Another promising direction is to adaptively refine test inputs, where the agent iteratively increases input size or structural complexity based on observed runtime differences until statistically significant performance gaps emerge. Finally, integrating static analysis or complexity estimation could help agents identify code regions that are likely to produce performance divergence and generate targeted stress tests accordingly. These directions are essential for enabling test suites that can reliably expose meaningful performance differences and support trustworthy evaluation of code efficiency. Beyond Function-Level Python Benchmarks. Our study focuses on Pythonbased, function-level benchmark tasks, which provide a controlled setting for comparing implementations but may not fully capture real-world performance scenarios. In practice, performance issues often arise from interactions across multiple components, such as object lifecycles and repeated method calls within larger systems. As a result, class-level or repository-level benchmarks can provide a more realistic setting for evaluating performance, since they involve richer program structures, larger codebases, and more complex execution contexts where performance differences accumulate over time (Jimenez et al., 2024; Zhang et al., 2023; Patil et al., 2024; Liu et al., 2024b). In addition, real-world performance is often influenced by factors that are rarely represented in function-level tasks. For example, many applications are I/O-bound, where performance depends on file operations, database queries, or network communication rather than pure algorithmic computation (Zhao et al., 2022; Li et al., 2019). Similarly, concurrent or asynchronous programs introduce scheduling and synchronization overheads that can significantly affect performance behavior (Selakovic and Pradel, 2016; Zhang et al., 2019). Another important scenario involves long-running workloads, such as batch data processing or server-side services, where small per-call efficiency improvements may accumulate into substantial runtime differences (He et al., 2025; Zhang et al., 2024a; Eismann et al., 2022). Future benchmarks should therefore incorporate these more realistic workloads and system-level interactions to better evaluate the effectiveness of optimization techniques and LLM-generated performance improvements in practical software systems. 6 Related Work Benchmarks for Code Generation Code-generation benchmarks evaluate LLMs at different levels of program scope. At the function level, benchmarks such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), APPS (Hendrycks et al., 2021), DS-1000 (Lai et al., 2023), CodeContests (Li et al., 2022a), LiveCodeBench (Jain et al., 2025), and BigCodeBench (Zhuo et al., 2025) assess whether a model can synthesize a function or program that satisfies an input-output specification and passes executable tests. Class-level

RELATED WORK

benchmarks, including ClassEval (Du et al., 2023), JavaBench (Cao et al., 2024), and subsequent class-generation benchmarks (Rahman et al., 2025; Chen et al., 2026), extend evaluation to multiple methods, class state, and interactions among class members. Repository-level benchmarks further evaluate code generation and software-engineering capabilities in the context of project structure, dependencies, and existing source files (Jimenez et al., 2024; Liu et al., 2024b; Li et al., 2024b; Le Hai et al., 2025; Li et al., 2025). Across these levels, evaluation primarily emphasizes functional correctness. However, functional correctness does not establish whether generated code is computationally efficient. Performance-oriented benchmarks such as EffiBench (Huang et al., 2024b), Enamel (Qiu et al., 2025), EvalPerf (Liu et al., 2024a), and Mercury (Du et al., 2024) evaluate runtime efficiency for Python functions. EffiBench-X (Qing et al., 2026) extends EffiBench to five additional languages: C++, Java, JavaScript, Ruby, and Golang. These benchmarks provide a canonical solution and a performant implementation for each task. The performant implementation serves as the runtime reference for evaluating LLM-generated code using executable tests and runtime measurements. In this paper, our study examines the four Python function-level performance benchmarks: EffiBench, Enamel, EvalPerf, and Mercury. Empirical Studies on Code Efficiency A number of studies have explored methodologies for evaluating code efficiency (Stoico et al., 2025; Guimaraes et al., 2025). These include executing code in local environments (Singhal et al., 2024; Cheng et al., 2023), using containerized execution for reproducibility (Khan et al., 2024), and designing performance-oriented test cases to stress input sizes and expose efficiency differences (Qiu et al., 2025; Liu et al., 2024a; Du et al., 2024; Peng et al., 2025b). Other studies directly evaluate the efficiency of LLM-generated code. Runtime-based evaluations on HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and LeetCodeEval show that functional correctness does not necessarily correlate with execution efficiency (Niu et al., 2024). Further empirical studies compare LLM-generated solutions with human-written implementations across runtime performance, memory usage, and energy consumption (Coignion et al., 2024; Islam et al., 2025; Solovyeva et al., 2025). These studies show growing interest in evaluating the computational efficiency of generated code. However, they mainly focus on assessing LLM-generated implementations, whereas our work re-examines the benchmarks themselves by testing whether benchmark-provided performant reference implementations are statistically distinguishable from their corresponding canonical solutions. Automated Test Generation with LLMs Beyond code generation, recent work has investigated the use of large language models (LLMs) for automated test generation. A major line of work focuses on functional unit test generation, where LLMs are used to produce executable tests for a given method, function, class, or API under test (Schäfer et al., 2023; Yuan et al., 2024; Chen et al., 2024). These studies primarily aim to improve functional correctness, compilability, and code coverage. Subsequent work further improves

LLM-generated tests by incorporating execution feedback, coverage feedback, search-based exploration, or mutation testing signals (Lemieux et al., 2023; Alagarsamy et al., 2024; Altmayer Pizzorno and Berger, 2025; Pan et al., 2025). In parallel, other studies examine supporting tasks for test generation, including assertion and oracle generation, test repair, test-suite augmentation, and model adaptation for unit testing (Rao et al., 2023; Shin et al., 2024; Lukasczyk and Fraser, 2022; Wang et al., 2025; Hossain and Dwyer, 2025). Despite these advances, most LLM-based test generation techniques focus on correctness-oriented testing rather than performance-oriented testing. Recent work on performance-oriented test generation, such as WEDGE (Yang et al., 2025) and COFFE (Peng et al., 2025b), has begun to address this gap by generating inputs that expose inefficient program behavior. Among them, COFFE is most closely aligned with our setting, as it studies stressful test generation for evaluating the efficiency of LLM-generated code at the Python function level. Our work complements this direction by examining how test-suite strength affects the reliability of performance evaluation. LLM-based Code Generation Recent advances in large language models (LLMs), such as GPT (OpenAI, 2023), LLaMA (Llama, 2025), Gemini (Reid et al., 2024), Claude (Anthropic, 2025a), DeepSeek (DeepSeek, 2025a), and Mixtral (Jiang et al., 2024), have substantially improved automated code generation. Most existing work only focus on generating functionally correct code, where models are assessed by whether their generated programs pass the provided test cases or benchmark oracles (Lin et al., 2025; Wei et al., 2024; Wang et al., 2023; Li et al., 2022b). Beyond correctness, recent studies have begun to investigate whether LLMs can generate computationally efficient code, focusing on runtime-aware prompting, optimization-oriented fine-tuning, selfimprovement, and efficiency-guided code generation (Huang et al., 2024a; Ye et al., 2025; Feng et al., 2025; Peng et al., 2025a; Waghjale et al., 2024).

7 Threats to Validity Internal Validity. One potential threat arises from the variability of runtime measurements in RQ1, RQ3 and RQ4, as execution time can be affected by system noise (e.g., OS scheduling and caching). To mitigate this, we execute each implementation 30 times and apply the Mann–Whitney U test to ensure that observed differences are statistically significant. Another potential internal threat is the subjectivity in the manual analysis in RQ2. Since open coding relies on human judgment, bias may affect how code changes are interpreted and categorized. We mitigate this by having two authors independently label each task, resolving disagreements through discussion, and achieving a high inter-rater agreement (Cohen’s Kappa: 0.93). External Validity. Our study focuses on four performance benchmarks: EffiBench (Huang et al., 2024b), Enamel (Qiu et al., 2025), EvalPerf (Liu et al., 2024a), and Mercury (Du et al., 2024) which primarily consist of Python based

CONCLUSION

programming tasks at the function level. As a result, our findings may not directly generalize to other programming languages, large scale software systems, or performance scenarios involving input-output operations, concurrency, or system level interactions. However the qualitative insights from running code multiple times with statistical testing and performance test cases are expected to generalize to similar benchmark studies. Researchers should thus consider the potential biases highlighted by our study when exploring similar datasets. Construct Validity. A construct threat concerns how we operationalize “performance” in Python function-level tasks. Our measurements focus on execution time, which fits the benchmark setting studied in this paper. However, in larger software systems, performance may also involve memory consumption, energy usage, throughput, latency, and interactions among components. Thus, our findings should be interpreted as evidence about functionlevel execution-time efficiency rather than a complete assessment of all performance dimensions. Future studies that move beyond function-level benchmarks should consider these additional performance dimensions when evaluating LLM-generated code.

8 Conclusion Existing Python function-level performance benchmarks are widely used to evaluate whether LLMs can generate efficient code. However, when these benchmarks report little or no execution-time difference between generated implementations and canonical solutions, it is unclear whether the result reflects a limitation of the LLMs or a limitation of the benchmark setting. In this paper, we revisit four benchmarks, EffiBench, Enamel, EvalPerf, and Mercury, using repeated execution and statistical testing. Across 1,538 tasks, only 6.11% of benchmark-provided performant implementations are significantly faster than their corresponding canonical solutions under the original test suites. This result shows that the original benchmark settings often do not provide enough evidence to support reliable performance comparisons. Our analysis further shows that non-significant results arise from two main causes. Some benchmark-provided performant implementations contain no meaningful performance change, such as refactoring or no code changes. Others contain algorithmic or data-structure changes that may improve runtime, but the original tests do not sufficiently stress the performance-critical paths where those changes matter. To address this limitation, we develop a multiagent framework for generating deterministic performance-oriented tests. With stronger tests, more benchmark-provided performant implementations become statistically distinguishable from their canonical solutions, and LLM-generated implementations show statistically significant performance improvements in 22.19% of the evaluated cases. Overall, our findings show that function-level Python performance benchmarks need to be evaluated carefully before they are used to judge whether performant code is truly more efficient than a canonical solution. The eval-

uation should rely on multiple executions and statistical testing, rather than single-run or small-sample runtime comparisons. This direction is important for future benchmark design: researchers who build performance benchmarks should explicitly account for runtime noise and include statistical testing in the evaluation protocol.

9 Declarations Funding This research received no external funding.

Ethical Approval Not applicable.

Informed Consent Not applicable.

Author Contributions The contributions of the authors are as follows. Nhat Minh Le: Conceptualization, Methodology, Analysis, Investigation, Experiments, Writing - original draft, Writing - review and editing. Yisen Xu: Analysis, Validation, Investigation, Writing - review and editing. Zhijie Wang: Analysis, Validation, Investigation, Writing - review and editing. Tse-Hsun (Peter) Chen: Conceptualization, Supervision, Funding acquisition, Validation, Writing - review and editing.

Conflict of Interest The authors declare that they have no conflict of interest.

Data and Code Availability Statement Our artifacts, including the replication package, all generated code, and measured code performance data, are publicly available online (art, 2026).

DECLARATIONS

References (2026) Replication package. https://doi.org/10.5281/zenodo.21227455 Ahmed T, Devanbu PT, et al. (2025) Can llms replace manual annotation of software engineering artifacts? In: 22nd IEEE/ACM International Conference on Mining Software Repositories, MSR@ICSE 2025, Ottawa, ON, Canada, April 28-29, 2025, IEEE, pp 526–538, DOI 10.1109/MSR66628. 2025.00086, URL https://doi.org/10.1109/MSR66628.2025.00086 Alagarsamy S, et al. (2024) A3test: Assertion-augmented automated test case generation. Information and Software Technology 176:107565 Altmayer Pizzorno J, Berger ED (2025) Coverup: Effective high coverage test generation for python. Proceedings of the ACM on Software Engineering 2(FSE):2897–2919 Anthropic (2025a) Anthropic. https://www.anthropic.com/news/ claude-opus-4-5 Anthropic (2025b) claudesonnet45. https://www.anthropic.com/news/ claude-sonnet-4-5 Arcuri A, Briand LC (2011) A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Taylor RN, Gall HC, Medvidovic N (eds) Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu , HI, USA, May 21-28, 2011, ACM, pp 1–10, DOI 10.1145/1985793.1985795, URL https: //doi.org/10.1145/1985793.1985795 Austin J, et al. (2021) Program synthesis with large language models. CoRR abs/2108.07732, URL https://arxiv.org/abs/2108.07732, 2108.07732 Avritzer A, Kondek J, Liu D, Weyuker EJ (2002) Software performance testing based on workload characterization. In: Proceedings of the 3rd International Workshop on Software and Performance, pp 17–24 Baltes S, Ralph P (2022) Sampling in software engineering research: a critical review and guidelines. Empir Softw Eng 27(4):94, DOI 10.1007/S10664-021-10072-8, URL https://doi.org/10.1007/ s10664-021-10072-8 Baltes S, Moseler O, Beck F, Diehl S (2015) Navigate, understand, communicate: How developers locate performance bugs. In: 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, pp 1–10 Cao J, Chen Z, Wu J, Cheung SC, Xu C (2024) Javabench: A benchmark of object-oriented code generation for evaluating large language models. In: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pp 870–882 Chen J, Revels J (2016) Robust benchmarking in noisy environments. CoRR abs/1608.04295, URL http://arxiv.org/abs/1608.04295, 1608.04295 Chen M, et al. (2021) Evaluating large language models trained on code. CoRR abs/2107.03374, URL https://arxiv.org/abs/2107.03374, 2107.03374 Chen Y, Hu Z, Zhi C, Han J, Deng S, Yin J (2024) Chatunitest: A framework for llm-based test generation. In: Companion Proceedings of the 32nd ACM

International Conference on the Foundations of Software Engineering, pp 572–576 Chen Y, Xie C, Shi Y, Zeng W, Wang Y, Zhang H, Gu X (2026) Classeval-pro: A cross-domain benchmark for class-level code generation. arXiv preprint arXiv:260426923 Cheng W, et al. (2023) Revisiting knowledge-based inference of python runtime environments: A realistic and adaptive approach. IEEE Transactions on Software Engineering 50(2):258–279 Cliff N (1993) Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological bulletin 114(3):494 Cohen J (1960) A coefficient of agreement for nominal scales. Educational and psychological measurement 20(1):37–46 Coignion T, Quinton C, Rouvoy R (2024) A performance study of llmgenerated code on leetcode. In: Proceedings of the 28th international conference on evaluation and assessment in software engineering, pp 79–89 DeepSeek (2025a) Deepseek chat. https://api-docs.deepseek.com/ DeepSeek (2025b) Deepseek model. https://api-docs.deepseek.com/ Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research 7(Jan):1–30 Du M, et al. (2024) Mercury: A code efficiency benchmark for code large language models. In: Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, URL http://papers.nips.cc/paper_files/paper/2024/ hash/1df1df43b58845650b8dada00fca9772-Abstract-Datasets_and_ Benchmarks_Track.html Du X, Liu M, Wang K, Wang H, Liu J, Chen Y, Feng J, Sha C, Peng X, Lou Y (2023) Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:230801861 Eismann S, et al. (2022) A case study on the stability of performance tests for serverless applications. Journal of Systems and Software 189:111294 Feng Y, Xu Y, et al. (2025) Towards better correctness and efficiency in code generation. CoRR abs/2508.20124, DOI 10.48550/ARXIV.2508.20124, URL https://doi.org/10.48550/arXiv.2508.20124, 2508.20124 Georges A, et al. (2007) Statistically rigorous java performance evaluation. ACM SIGPLAN Notices 42(10):57–76 Google (2025) Gemini25. https://ai.google.dev/gemini-api/docs/ models/gemini-2.5-flash Guimaraes E, et al. (2025) Analyzing prominent llms: An empirical study of performance and complexity in solving leetcode problems. In: Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, pp 949–958 He X, et al. (2025) Swe-perf: Can language models optimize code performance on real-world repositories? CoRR abs/2507.12415, DOI 10.48550/ARXIV. 2507.12415, URL https://doi.org/10.48550/arXiv.2507.12415, 2507. 12415

DECLARATIONS

Hendrycks D, et al. (2021) Measuring coding challenge competence with APPS. In: Vanschoren J, Yeung S (eds) Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/ hash/c24cd76e1ce41366a4bbe8a49b02a028-Abstract-round2.html Hossain SB, Dwyer MB (2025) Togll: Correct and strong test oracle generation with llms. In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), IEEE, pp 1475–1487 Huang D, Dai J, et al. (2024a) Effilearner: Enhancing efficiency of generated code via self-optimization. In: Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, URL http://papers.nips.cc/paper_files/paper/2024/hash/ 99c66755871ae101a4cef87c67fb29e8-Abstract-Conference.html Huang D, et al. (2024b) Effibench: Benchmarking the efficiency of automatically generated code. In: Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, URL http://papers.nips.cc/paper_files/paper/2024/ hash/15807b6e09d691fe5e96cdecde6d7b80-Abstract-Datasets_and_ Benchmarks_Track.html Islam MA, Jonnala DV, Rekhi R, Pokharel P, Cilamkoti S, Imran A, Kosar T, Turkkan B (2025) Evaluating the energy-efficiency of the code generated by llms. arXiv preprint arXiv:250520324 Jain N, Gu A, Li WD, Yan F, Zhang T, Wang S, Solar-Lezama A, Sen K, Stoica I (2025) Livecodebench: Holistic and contamination free evaluation of large language models for code. In: International Conference on Learning Representations, vol 2025, pp 58791–58831 Jangali M, et al. (2023) Automated generation and evaluation of JMH microbenchmark suites from unit tests. IEEE Trans Software Eng 49(4):1704– 1725, DOI 10.1109/TSE.2022.3188005, URL https://doi.org/10.1109/ TSE.2022.3188005 Jiang AQ, et al. (2024) Mixtral of experts. CoRR abs/2401.04088, DOI 10.48550/ARXIV.2401.04088, URL https://doi.org/10.48550/arXiv. 2401.04088, 2401.04088 Jimenez CE, et al. (2024) Swe-bench: Can language models resolve real-world github issues? In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, URL https://openreview.net/forum?id=VTF8yNQM66 Jin G, Song L, Shi X, Scherpelz J, Lu S (2012) Understanding and detecting real-world performance bugs. ACM SIGPLAN Notices 47(6):77–88 Kalibera T, Jones R (2013) Rigorous benchmarking in reasonable time. In: Proceedings of the 2013 international symposium on memory management, pp 63–74

Kampenes VB, et al. (2007) A systematic review of effect size in software engineering experiments. Information and Software Technology 49(11-12):1073– 1086 Khan MAM, et al. (2024) Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 6766–6805 Lai Y, et al. (2023) DS-1000: A natural and reliable benchmark for data science code generation. In: International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, PMLR, Proceedings of Machine Learning Research, vol 202, pp 18319–18345, URL https://proceedings.mlr.press/v202/lai23b.html Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. biometrics pp 159–174 Le Hai N, Nguyen DM, Bui ND (2025) On the impacts of contexts on repository-level code generation. In: Findings of the Association for Computational Linguistics: NAACL 2025, pp 1496–1524 Lemieux C, et al. (2018) Perffuzz: Automatically generating pathological inputs. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, pp 254–265 Lemieux C, et al. (2023) Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), IEEE, pp 919– 931 Li D, Jiang B, et al. (2024a) From generation to judgment: Opportunities and challenges of llm-as-a-judge. CoRR abs/2411.16594, DOI 10.48550/ARXIV. 2411.16594, URL https://doi.org/10.48550/arXiv.2411.16594, 2411. 16594 Li J, Li G, Zhao Y, Li Y, Liu H, Zhu H, Wang L, Liu K, Fang Z, Wang L, et al. (2024b) Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories. In: Findings of the Association for Computational Linguistics: ACL 2024, pp 3603–3614 Li S, et al. (2019) Detecting performance bottlenecks guided by resource usage. IEEE Access 7:117839–117849 Li W, Zhang X, Guo Z, Mao S, Luo W, Peng G, Huang Y, Wang H, Li S (2025) Fea-bench: A benchmark for evaluating repository-level code generation for feature implementation. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 17160–17176 Li Y, Choi D, Chung J, Kushman N, Schrittwieser J, Leblond R, Eccles T, Keeling J, Gimeno F, Dal Lago A, et al. (2022a) Competition-level code generation with alphacode. Science 378(6624):1092–1097 Li Y, et al. (2022b) Competition-level code generation with alphacode. CoRR abs/2203.07814, DOI 10.48550/ARXIV.2203.07814, URL https://doi. org/10.48550/arXiv.2203.07814, 2203.07814

DECLARATIONS

Liao L, et al. (2025) Early detection of performance regressions by bridging local performance data and architectural models. In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), IEEE, pp 2841– 2853 Lin F, et al. (2025) SOEN-101: code generation by emulating software process models using large language model agents. In: 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025, IEEE, pp 1527–1539, DOI 10.1109/ICSE55347.2025.00140, URL https://doi.org/10.1109/ ICSE55347.2025.00140 Liu J, et al. (2024a) Evaluating language models for efficient code generation. CoRR abs/2408.06450, DOI 10.48550/ARXIV.2408.06450, URL https:// doi.org/10.48550/arXiv.2408.06450, 2408.06450 Liu T, et al. (2024b) Repobench: Benchmarking repository-level code autocompletion systems. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, URL https://openreview.net/forum?id=pPjZIOuQuF Llama (2025) Llama. https://ai.meta.com/blog/meta-llama-3/ Lukasczyk S, Fraser G (2022) Pynguin: Automated unit test generation for python. In: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pp 168–172 Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics pp 50–60 Mogul JC, Borg A (1991) The effect of context switches on cache performance. In: ASPLOS-IV Proceedings - Forth International Conference on Architectural Support for Programming Languages and Operating Systems, Santa Clara, California, USA, April 8-11, 1991, ACM Press, pp 75–84, DOI 10.1145/106972.106982, URL https://doi.org/10.1145/106972.106982 Mytkowicz T, et al. (2009) Producing wrong data without doing anything obviously wrong! ACM Sigplan Notices 44(3):265–276 Neyman J (1992) On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection. In: Breakthroughs in statistics: Methodology and distribution, Springer, pp 123–150 Niu C, et al. (2024) On evaluating the efficiency of source code generated by llms. In: Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering, pp 103–107 OpenAI (2023) GPT-4 technical report. CoRR abs/2303.08774, DOI 10.48550/ARXIV.2303.08774, URL https://doi.org/10.48550/arXiv. 2303.08774, 2303.08774 OpenAI (2024) Chatgpt. https://developers.openai.com/api/docs/ models/gpt-4o-mini OpenAI (2025) Chatgpt. https://platform.openai.com/docs/models/ gpt-4o

Pan R, Kim M, Krishna R, Pavuluri R, Sinha S (2025) Aster: Natural and multi-language unit test generation with llms. In: 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), IEEE, pp 413–424 Patil SG, et al. (2024) Gorilla: Large language model connected with massive apis. In: Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, URL http://papers.nips.cc/paper_files/paper/2024/hash/ e4c61f578ff07830f5c37378dd3ecb0d-Abstract-Conference.html Peng Y, Gotmare AD, Lyu MR, Xiong C, Savarese S, Sahoo D (2025a) Perfcodegen: Improving performance of llm generated code with execution feedback. In: 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge), IEEE, pp 1–13 Peng Y, Wan J, et al. (2025b) COFFE: A code efficiency benchmark for code generation. Proc ACM Softw Eng 2(FSE):242–265, DOI 10.1145/3715727, URL https://doi.org/10.1145/3715727 Petsios T, et al. (2017) Slowfuzz: Automated domain-independent detection of algorithmic complexity vulnerabilities. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp 2155– 2168 Qing Y, et al. (2026) Effibench-x: A multi-language benchmark for measuring efficiency of llm-generated code. Advances in Neural Information Processing Systems 38 Qiu R, et al. (2025) How efficient is llm-generated code? A rigorous & highstandard benchmark. In: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, URL https://openreview.net/forum?id=suz4utPr9Y Rahman M, Khatoonabadi S, Shihab E (2025) Beyond synthetic benchmarks: Evaluating llm performance on real-world class-level code generation. arXiv preprint arXiv:251026130 Rao N, et al. (2023) Cat-lm training language models on aligned code and tests. In: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, pp 409–420 Reid M, et al. (2024) Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR abs/2403.05530, DOI 10.48550/ARXIV. 2403.05530, URL https://doi.org/10.48550/arXiv.2403.05530, 2403. 05530 Schäfer M, et al. (2023) An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering 50(1):85–105 Selakovic M, Pradel M (2016) Performance issues and optimizations in javascript: an empirical study. In: Proceedings of the 38th international conference on software engineering, pp 61–72 Shin J, et al. (2024) Domain adaptation for code model-based unit test case generation. In: Proceedings of the 33rd ACM SIGSOFT International Sym-

DECLARATIONS

posium on Software Testing and Analysis, pp 1211–1222 Sidney S (1957) Nonparametric statistics for the behavioral sciences. The Journal of Nervous and Mental Disease 125(3):497 Singhal M, Aggarwal T, et al. (2024) Nofuneval: Funny how code lms falter on requirements beyond functional correctness. CoRR abs/2401.15963, DOI 10.48550/ARXIV.2401.15963, URL https://doi.org/10.48550/arXiv. 2401.15963, 2401.15963 Solovyeva L, Weidmann S, Castor F (2025) Ai-powered, but power-hungry? energy efficiency of llm-generated code. In: 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge), IEEE, pp 49–60 Stoico V, et al. (2025) An empirical study on the performance and energy usage of compiled python code. In: Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, pp 46– 56 Su P, et al. (2019) Pinpointing performance inefficiencies in java. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 818–829 Tsafrir D, Etsion Y, et al. (2005) System noise, OS clock ticks, and finegrained parallel applications. In: Proceedings of the 19th Annual International Conference on Supercomputing, ICS 2005, Cambridge, Massachusetts, USA, June 20-22, 2005, ACM, pp 303–312, DOI 10.1145/1088149.1088190, URL https://doi.org/10.1145/1088149.1088190 Vokolos FI, Weyuker EJ (1998) Performance testing of software systems. In: Proceedings of the 1st International Workshop on Software and Performance, pp 80–87 Waghjale S, Veerendranath V, Wang Z, Fried D (2024) Ecco: Can we improve model-generated code efficiency without sacrificing functional correctness? In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp 15362–15376 Wang W, et al. (2025) TESTEVAL: benchmarking large language models for test case generation. In: Chiruzzo L, Ritter A, Wang L (eds) Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, Association for Computational Linguistics, Findings of ACL, pp 3547–3562, DOI 10.18653/ V1/2025.FINDINGS-NAACL.197, URL https://doi.org/10.18653/v1/ 2025.findings-naacl.197 Wang Y, et al. (2023) Codet5+: Open code large language models for code understanding and generation. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Association for Computational Linguistics, pp 1069–1088, DOI 10.18653/V1/2023.EMNLP-MAIN.68, URL https://doi. org/10.18653/v1/2023.emnlp-main.68 Wei Y, Cassano F, et al. (2024) Selfcodealign: Self-alignment for code generation. In: Advances in Neural Information Processing Sys-

tems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, URL http://papers.nips.cc/paper_files/paper/2024/hash/ 72da102da91a8042a0b2aa968429a9f9-Abstract-Conference.html Weyuker EJ, Vokolos FI (2000) Experience with performance testing of software systems: issues, an approach, and case study. IEEE transactions on software engineering 26(12):1147–1156 Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics bulletin 1(6):80–83 Woodside M, Franks G, Petriu DC (2007) The future of software performance engineering. In: Future of Software Engineering (FOSE’07), IEEE, pp 171– 187 Yang J, et al. (2025) Synthesizing performance constraints for evaluating and improving code efficiency. CoRR abs/2505.23471, DOI 10.48550/ARXIV. 2505.23471, URL https://doi.org/10.48550/arXiv.2505.23471, 2505. 23471 Ye T, Huang W, et al. (2025) LLM4EFFI: leveraging large language models to enhance code efficiency and correctness. CoRR abs/2502.18489, DOI 10.48550/ARXIV.2502.18489, URL https://doi.org/10.48550/arXiv. 2502.18489, 2502.18489 Yuan Z, et al. (2024) Evaluating and improving chatgpt for unit test generation. Proceedings of the ACM on Software Engineering 1(FSE):1703–1726 Zeng Y, et al. (2019) Studying the characteristics of logging practices in mobile apps: a case study on f-droid. Empirical Software Engineering 24(6):3394– 3434 Zhang C, et al. (2019) Understanding and statically detecting synchronization performance bugs in distributed cloud systems. IEEE Access 7:99123–99135 Zhang C, et al. (2024a) Trace-based multi-dimensional root cause localization of performance issues in microservice systems. In: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp 1– 12 Zhang F, et al. (2023) Repocoder: Repository-level code completion through iterative retrieval and generation. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Association for Computational Linguistics, pp 2471–2484, DOI 10.18653/V1/2023.EMNLP-MAIN.151, URL https: //doi.org/10.18653/v1/2023.emnlp-main.151 Zhang X, Li Y, et al. (2024b) Large language models as evaluators for recommendation explanations. In: Proceedings of the 18th ACM Conference on Recommender Systems, RecSys 2024, Bari, Italy, October 14-18, 2024, ACM, pp 33–42, DOI 10.1145/3640457.3688075, URL https://doi.org/ 10.1145/3640457.3688075 Zhao Y, et al. (2022) A large-scale empirical study of real-life performance issues in open source projects. IEEE Transactions on Software Engineering 49(2):924–946

DECLARATIONS

Zhuo TY, Vu MC, Chim J, Hu H, Yu W, Widyasari R, Yusuf INB, Zhan H, He J, Paul I, et al. (2025) Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In: International Conference on Learning Representations, vol 2025, pp 66602–66656

Record · ID 349685 · SHA-256 3993724a283c82a8
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.