DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation

DSEffi-Bench: Demystifying Large Language Models’ Capability in Efficient Data Science Code Generation Zhihao Gong1 * , Junzhe Yu2 * , Dong Huang4 , Zeyu Sun3 , Jie M. Zhang5 , Dan Hao1,2† 1 Key Lab of HCST (PKU), MOE; SCS, Peking University, China 2 School of Electronic and Computer Engineering, PKU Shenzhen Graduate School, China 3 Institute of Software, Chinese Academy of Sciences, Beijing, China 4 National University of Singapore, Singapore 5 King’s College London, London, United Kingdom zhihaogong@stu.pku.edu.cn, haodan@pku.edu.cn

Current data science (DS) code generation benchmarks equate correctness with quality, overlooking execution time differences that span orders of magnitude between correct solutions. We introduce DSEffi-Bench, the first benchmark specifically targeting execution efficiency in LLM-generated DS code, comprising 1,000 instances across 10+ DS libraries with stress-testing harnesses and human-validated references. Evaluating 16 models across 3 tiers, we find that correctness alone fails to characterize efficiency: GPT-5.4 leads in correctness (Pass, 66.9%) but its efficiency score (B|P, 71.7%) nearly matches GPT-5.4-mini (71.6%), which solves 47 fewer tasks; Kimi-K2.5 ranks lowest in correctness among frontier models (40.2%) yet achieves the highest efficiency score (73.6%) across all 16 models. A humanannotated five-category taxonomy reveals that 79.1% of efficiency deficits extend beyond algorithmic complexity to domain-specific root causes, with distinct failure profiles across model tiers and libraries. Two exploratory experiments provide initial evidence that these diagnostics can guide improvement, yielding up to +14.7% efficiency gains via taxonomyguided optimization and approaching ClaudeOpus-4.6 Best@3 in efficiency at 13.0× lower cost via library-conditioned routing. Code and data are available at https://github.com/ Albert-Gong/DSEffi-Bench.

1

Introduction

In data science (DS) coding (Lai et al., 2023), programmers write code to solve DS problems such as statistical modeling, numerical optimization, and deep learning. As LLM-assisted code generation (Chen et al., 2021; Jain et al., 2025) advances, DS coding has emerged as one of its most active application domains (Sun et al., 2026). * Equal contribution. †

Corresponding author.

Efficiency (Beyond|Pass, %)

arXiv:2608.30248v1 [cs.SE] 31 Aug 2026

Abstract

80 B=10 75 70 65 60 55 50 45 40 10

B=20

B=30

B=40

Kimi-K2.5 Qwen3.5-397B/A17B

B=50

GPT-5.4-mini GPT-5.4 Claude-Opus-4.6 Claude-Sonnet-4.6 DeepSeek-V4-Flash

MiniMax-M2.5 Qwen3-Coder-Next DeepSeek-V3.2 Devstral-2-123B

Qwen2.5-Coder-32B

Codestral-22B DeepSeek-Coder-33B Qwen2.5-Coder-14B DeepSeek-Coder-V2-Lite

20

30

40

50

Correctness (Pass, %)

Closed Frontier Open Frontier Open Coder

60

70

Figure 1: Functional correctness (Pass, %) vs. execution efficiency (B|P, %) for 16 models on DSEffi-Bench.

DS coding by nature differs from generalpurpose programming. The domain knowledge is extensive and encapsulated in highly optimized library APIs (Huang et al., 2022; Wang et al., 2026). Writing authentic DS code demands selecting the right primitives, not merely implementing the logic from scratch. Prior benchmarks such as ExeDS (Huang et al., 2022), DS-1000 (Lai et al., 2023) and DSCodeBench (Ouyang et al., 2026) curate realistic DS tasks to evaluate LLM capability. All of them reveal uneven performance across libraries and observe that larger models tend to outperform smaller ones in correctness. However, current benchmarks evaluate functional correctness exclusively and leave execution efficiency (Niu et al., 2024; Gong et al., 2026) unexamined. In DS coding, efficiency depends critically on API selection: functionally equivalent paths can differ by orders of magnitude in execution time. For instance, df.groupby(’k’).cummax() and df.groupby(’k’).apply(lambda g: g.cummax()) produce identical output yet differ by an order of magnitude because the latter incurs Python-level groupwise callback overhead. LLMs are particularly susceptible to such gaps (Xue et al., 2025), as they default to general-purpose patterns over libraryspecific optimized primitives. Correctness-only evaluation cannot surface this systematic weakness.

To address this gap, we introduce DSEffi-Bench, the first execution efficiency benchmark for evaluating LLM-based DS code generation. DSEffi-Bench treats correctness and efficiency as two distinct capabilities and exposes their interplay through a dedicated evaluation protocol. Concretely, we source efficiency-oriented Stack Overflow (SO) posts and apply a quality-gated four-stage pipeline that progressively filters 17,257 candidates down to 1,000 human-validated instances. Each instance contains a problem description, an evaluation harness covering both correctness and stress test cases, and a human-validated reference solution. Using DSEffi-Bench, we evaluate 16 models spanning 3 tiers. First, correctness and efficiency are not aligned. GPT-5.4 achieves the highest correctness (Pass, 66.9%), yet its efficiency on solved tasks (B|P, 71.7%) is virtually identical to GPT-5.4mini (71.6%), which solves 47 fewer tasks. Conversely, Kimi-K2.5 solves the fewest tasks among frontier models (40.2%) yet attains the highest B|P (73.6%) across all 16 models. Second, our fivecategory taxonomy shows that 79.1% of efficiency failures extend beyond algorithmic complexity to domain-specific root causes. These failures are tierdependent: loop-to-vectorization gaps rise from 7.3% in Closed Frontier LLMs to 29.6% in Open Coder LLMs, while overhead-aware stack matching (OSM) declines from 29.3% to 9.3%. Third, two exploratory experiments show that these diagnostics can guide improvement: taxonomy-guided optimization prompting raises efficiency by up to +14.7%, and library-conditioned routing among open models approaches Claude-Opus-4.6 Best@3 in efficiency at 13.0× lower cost. This work makes the following contributions: 1) Benchmark. We propose DSEffi-Bench, the first execution efficiency benchmark for DS code generation, comprising 1,000 human-validated instances across 10+ libraries. 2) Diagnosis. We conduct a systematic evaluation and develop a five-category taxonomy that localizes 79.1% of efficiency deficits to domain-specific root causes. 3) Remedy. We explore taxonomy-guided optimization and library-aware routing and show their potential to improve efficiency.

2

Related Work

2.1

Efficiency Evaluation in Code Generation

Existing efforts to evaluate the execution efficiency of LLM-generated code have largely focused on

algorithmic or implementation-level optimization, leaving the DS domain underexplored. In the competitive programming domain (Puri et al., 2021), numerous benchmarks have been proposed to assess algorithmic efficiency, such as EffiBench(X) (Huang et al., 2024b; Qing et al., 2026), Mercury (Du et al., 2024), COFFE (Peng et al., 2025), ENAMEL (Qiu et al., 2025), and EvalPerf (Liu et al., 2024b). At the repository level (Jimenez et al., 2024), SWE-Perf (He et al., 2025) and GSO (Shetty et al., 2026) evaluate whether models can generate patches that optimize code within the repository context. In scientific computing, KernelBench (Ouyang et al., 2025) and AlgoTune (Press et al., 2026) target CUDA kernel generation and numerical algorithm redesign, respectively. 2.2

DS Code Generation and Evaluation

Existing DS code generation benchmarks have advanced in both evaluation harness and task scope. ExeDS (Huang et al., 2022) introduces executionbased evaluation for DS notebook cells, replacing surface-form metrics. DS-1000 (Lai et al., 2023) curates 1,000 problems from Stack Overflow across seven libraries and combines test cases with surface-form constraints. DSCodeBench (Ouyang et al., 2026) extends evaluation to GitHub-sourced tasks with stronger test suites across ten libraries. DA-Code (Huang et al., 2024d) and DataSciBench (Zhang et al., 2025) further scale to endto-end LLM pipeline evaluation on multi-step DS tasks. Despite these advances, all existing work treats correctness as the sole capability proxy, leaving execution efficiency unmeasured. Position. DSEffi-Bench is the first benchmark targeting execution efficiency of LLM-generated DS code. This dimension is orthogonal to correctness: models that scale well in correctness do not necessarily scale in efficiency, and the efficiency deficits in DS coding extend beyond algorithmic complexity to domain-specific patterns (e.g., loop-to-vectorization), distinguishing them from those studied in competitive-programming benchmarks (Huang et al., 2024b,c). Our evaluation confirms both points: correctness scaling (Ouyang et al., 2026) does not extend to efficiency (§5.1), and a human-annotated taxonomy localizes 79.1% of deficits to library-specific root causes (§5.2).

3

Benchmark Construction

DSEffi-Bench comprises 1,000 instances sourced

Raw Stack-Overflow Posts Collection

Benchmark Instance � Problem Description

Stack Overflow # 69881959

Is it possible to compute nested sums using pandas and groupby?

task_description

Views: 1.2k | Votes: 3 | Answers: 1

pandas pandas- groupby

"Compute i ,j ρ(τ i ,τ j )x i x j for each category group..."

input_specification

Question

Accepted Answer

Suppose I have a DataFrame with columns τ, x, cat. How do I compute i ,j ρ(τ i ,τ j )x i x j for each group? Currently looping ... wondering a more elegant way.

Pivot the df by τ, stack ρ, then use reindex ... : df = dftest.pivot('cat', 'tau', 'x') s = rho.stack() out = df1.mul(df2).mul(s).sum(axis=1)

output_specification

{"df": DataFrame, "rho": matrix}

pd.Series indexed by cat, float per group

input_example

output_example

"df": DataFrame(6 rows), "rho": 3x3

Series({ "a": 193.4, "b": 496.4})

� Evaluation Harness

Quality-Gated Construction Pipeline 1. Post Collection

17,257

Input: Raw SO Posts from January 2021 to December 2025 Process: 21 DS-related Tags × 21 Efficiency-related Keywords Output: Candidate SO Posts

3. Instance Transcription

2,301

Input: Relevant and Feasible SO Posts Process: Problem Description & Evaluation Harness Generation Output: Candidate Benchmark Instances

input_domain ["correctness-small", ..., "stress-large"]

2. Relevance & Feasibility Filter

4,360

Input: Candidate SO Posts Process: Question-level Relevance & Answer-level Feasibility Output: Relevant and Feasible SO Posts

4. Reference Solution Validation Input: Candidate Benchmark Instances Process: Reference Solution Annotation & Validation Output: Validated Benchmark Instances

1,000

input_generator

output_validator

def input_generator(seed, domain): if "small" in domain: k, N = 3, 12 elif "large" in domain: k, N = 300, 50000

def output_validator(pred, ref): assert_allclose( to_series(pred), to_series(ref), rtol=1e-4)

� ��� Human-validated Reference Solution def solution(input_data): V = df.groupby("cat","tau").sum().unstack() return np.einsum("ij, jk, ik->i", V, R, V)

Figure 2: Overview of the four-stage benchmark construction pipeline. Upper left: raw SO posts. Lower left: the construction stage. Right: the resulting benchmark instance.

from Stack Overflow (SO), which captures everyday performance pain points that DS programmers encounter. Rather than synthesizing tasks from scratch (Wu et al., 2026; Zhan et al., 2026; Wang et al., 2023), we derive every instance from a real SO post, preserving authentic optimization intent. To ensure that instances are both DS-related and efficiency-oriented, we design a four-stage quality-gated construction pipeline (Figure 2): (1) Efficiency-oriented Post Collection (§3.1), (2) Relevance and Feasibility Filtering (§3.2), (3) Instance Transcription (§3.3), and (4) Reference Solution Validation (§3.4). Benchmark Schema. Each instance has three components. (1) Problem description X, comprising five fields: task description, input specification, output specification, input example, and output example, serving as the prompt to the model under evaluation. (2) Evaluation harness T , containing a testing domain list (input_domain) that defines both correctness and stress-scale conditions, a fuzzstyle input generator (input_generator) for deterministic test input sampling, and a correctness checker (output_validator) that compares candidate output against the reference. (3) Humanvalidated reference solution Sref to anchor both correctness judgments and efficiency scoring. The detailed schema is provided in Appendix A.2. 3.1

Efficiency-oriented Post Collection

Given the volume of SO, we first employ keywordbased retrieval via the SO Advanced Search API (Stack Exchange Inc., 2024) to collect posts that jointly concern a DS library and an explicit efficiency goal. We design two keyword sets targeting the two aspects: 21 DS-related library tags

(e.g., Pandas, NumPy), and 21 efficiency-oriented keywords covering terms such as vectorize and performance. Both sets are derived from an extensive survey of prior DS benchmarks (Ouyang et al., 2026) and code efficiency studies (Huang et al., 2024b). Each retrieval query requires the co-occurrence of one library tag and one efficiency keyword in the same post, over January 2021 to December 2025. We retain only posts with at least 1 vote, 50 views, and an accepted answer, then deduplicate by post id, yielding 17,257 seed posts. The full keyword lists are in Appendix A.3. 3.2

Relevance and Feasibility Filtering

Keyword-based retrieval matches surface tokens rather than semantics (Chen et al., 2024): a post may mention query terms, yet neither pose a genuine efficiency problem nor offer a reproducible solution. We adopt LLM-as-Judge (Li et al., 2024) to further assess post quality, reducing the seed set to 4,360 through two sequential filters: (1) questionlevel relevance, and (2) answer-level feasibility. We use Claude-Sonnet-4.6 as the judge for both filters; each returns a structured JSON decision. The question-level relevance filter takes as input the post’s title and body, and asks the judge to verify that the post (i) concerns a DS coding scenario, (ii) targets efficiency improvement, and (iii) provides sufficient detail for a self-contained instance; posts that fail any criterion are discarded (17,257 → 5,850). The answer-level feasibility filter takes as input the full post (question and accepted answer), and checks whether the answer provides a concrete, reproducible optimization using DS libraries, rather than a conceptual explanation alone (5,850 → 4,360). Full prompts are provided in

Appendix A.4. To calibrate LLM-as-Judge filters, we draw a random sample of N =385 per stage (95% CI, ±5% margin) and have two authors independently label each instance as accept or reject. Cohen’s κ is 0.93 (question-level) and 0.71 (answer-level), indicating almost-perfect and substantial agreement, respectively (Landis and Koch, 1977). 3.3

Statistic Benchmark Profile Instances SO time span Problem description (words) DS libraries Evaluation Protocol Evaluated models Reference solution (lines) Test cases (per instance) Spectrum size (mean / median) Slowest/fastest ratio (median)

Instance Transcription

This stage takes the 4,360 filtered posts and transcribes them into candidate benchmark instances. Specifically, we (i) generate the problem description X, (ii) generate the evaluation harness T , and (iii) apply an LLM-driven quality scorer to discard under-specified transcriptions. We use Claude-Sonnet-4.6 to read each post’s question body and produce the problem description X. We explicitly formulate the unified function signature def solution(input_data): in the prompt for ease of downstream model evaluation. Given X and the original SO accepted answer as implementation context, we use the same model to generate the evaluation harness T , following fuzz-style test generation (Peng et al., 2025; Liu et al., 2024b), i.e., deterministically sampling diverse inputs from specified domains. To filter low-quality cases, we apply an LLM-asJudge that rates each instance on four dimensions (1–3 scale; 1=low quality, 2=adequate, 3=high quality): (i) task description clarity, (ii) I/O specification precision, (iii) input generator correctness, and (iv) output validator robustness. Instances where any dimension receives a score of 1 are discarded. To mitigate model self-preference bias (Wataoka et al., 2024), we select a judge from a different model family (GPT-5.4). This yields 2,301 candidate instances. All transcription prompts are provided in Appendix A.5. 3.4

Table 1: Benchmark statistics and evaluation protocol.

Human-Validated Reference Solution

This stage augments the LLM-generated artifacts with human-validated reference solutions. To balance human effort with benchmark quality, we rank candidates by their mean quality score from the previous judging process and annotate from the top down until 1,000 instances are obtained. Two authors, each with over five years of DS coding experience, serve as annotators. Both annotators independently study the problem description, the I/O specification and example, and the original SO accepted answer. Each then

Value 1,000 2021.01–2025.12 358.3 10+ 16 (3 tiers) 27.5 38.8 29.4 / 31.0 14.5×

writes a reference solution conforming to the def solution(input_data): signature. Annotators may choose to implement the accepted answer’s optimization suggestion faithfully or further improve it when a more efficient approach is apparent. During annotation, 50.3% of reference solutions adopt a fundamentally different strategy, 32.2% further refine the answer’s approach, and the remaining 17.5% faithfully reproduce it. Afterwards, the two solutions are compared; when they differ, annotators discuss and select the more efficient one, or compose a hybrid that combines the best elements. Annotators also revise the problem description, input generator, and output validator when any artifact was ambiguous or incomplete, ensuring that the final instances are quality-gated and not simple reproductions of any single LLM’s output. In total, we annotated 1,247 candidates and obtained 1,000 validated instances; the remaining 247 were discarded due to quality reasons such as ambiguous problem statements, or tasks where the reference solution could not establish a clear efficiency advantage over naive approaches. Detailed annotation statistics are in Appendix A.6. Benchmark Statistics. As summarized in Table 1, the benchmark spans five years of SO posts (2021–2025) and covers over 10 DS libraries, including representative libraries such as NumPy, Pandas, PyTorch, Polars, and SciPy, along with Numba, TensorFlow, and other long-tail libraries. This distribution reflects the natural frequency of efficiency-oriented questions on SO, where NumPy and Pandas dominate everyday DS workloads.

4

Evaluation Protocol

Given a problem description X, a model generates a candidate solution Sc . We evaluate each Sc along two dimensions using the evaluation harness T and the human-validated reference Sref : functional

correctness and execution efficiency. Table 1 summarizes the evaluation protocol. 4.1

Evaluation Metrics

Functional correctness (Pass). For each task, we use input_generator to produce multiple test inputs {xi }.
Sc is correct if and only if V Sc (xi ), Sref (xi ) = T RUE for every xi , where V is the output_validator. Solutions that fail or exceed the 5-second timeout on any test case receive a score of 0. Pass is the fraction of tasks where the model passed all the tests. Execution efficiency (Beyond, Beyond|Pass). Following Mercury (Du et al., 2024), we evaluate efficiency via a relative score. For each task, we construct an efficiency spectrum from a pool of correct solutions sorted by execution time. Each solution receives a normalized score Beyond(Sc ) = (log tmax − log tc ) / (log tmax − log tmin ), where tc is the execution time of Sc , and tmin , tmax are the fastest and slowest in the spectrum. The fastest solution scores 1; slowest and incorrect solutions receive 0. Note that the original Beyond metric in Mercury uses linear normalization. We find that execution time spans in our benchmark are wide (median slowest-to-fastest ratio 14.5×, 90th percentile 522.0×), causing 46.6% of correct solutions to score above 0.95 under linear normalization, which compresses meaningful differences into a narrow band. We therefore normalize in log-space, which spreads the median score to 0.74 and preserves discriminative range. We report three model-level metrics: Pass (fraction of tasks solved), Beyond (mean spectrum score, 0 for unsolved), and Beyond|Pass (B|P) (mean spectrum score restricted to solved tasks, isolating efficiency from correctness). 4.2

Efficiency Spectrum

We evaluate 16 models grouped into 3 tiers by accessibility and specialization: Closed Frontier (proprietary general-purpose, e.g., GPT-5.4, Claude-Opus-4.6), Open Frontier (open-weight general-purpose, e.g., DeepSeek-V4-Flash, KimiK2.5), and Open Coder (open-weight codespecialized, e.g., Qwen3-Coder-Next, Devstral-2123B). The full model list is in Appendix C. The efficiency spectrum is constructed from all correct solutions produced by these 16 models plus the human reference. Each model generates one greedy sample (T=0.0) and three temperature samples (T=0.8, following Liu et al. (2024b)) per task,

contributing up to four correct solutions per model per task. This yields an average spectrum size of 29.4 per task (median 31.0), with a median slowestto-fastest ratio of 14.5× (90th percentile: 522.0×), providing sufficient dynamic range for efficiency discrimination. 4.3

Measurement Procedure

The input_generator (described in §3.3) produces on average 38.8 test cases per task (26.8 correctness, 12.0 stress). For correct solutions, we measure execution time via Python’s time.perf_counter using a repeated-execution protocol: 1 warm-up run followed by 3 timed runs, reporting the mean elapsed time. A solution’s execution time for a task is the average across tests. The full prompts and the rationale for the measurement parameters are in Appendix C.

5

Evaluation

This section presents evaluation results of 16 LLMs along correctness and efficiency. 5.1

Overall Evaluation

Table 2 and Figure 1 summarize the main results. Correctness alone fails to characterize efficiency. Table 2 shows a partial decoupling between correctness and efficiency. GPT-5.4 leads in correctness (Pass, 66.9%), yet its efficiency score (B|P, 71.7%) is nearly tied with GPT-5.4-mini (71.6%), which solves 47 fewer tasks. The gap widens for open models. Kimi-K2.5 ranks lowest (40.2%) in Pass among the frontier models, yet achieves the highest B|P (73.6%) across all models. DeepSeek-V3.2 presents the mirror image, with Pass 56.1% but B|P only 54.2%, trailing Kimi by 19.4%. These rank inversions contrast with the scaling observation reported by correctness-only DS benchmarks (Lai et al., 2023; Ouyang et al., 2026). Moreover, code-specialized training does not close this gap. Open Coder models average B|P 50.2%, trailing Open Frontier generalists (64.3%) despite explicit code-corpus training. Meanwhile, the human reference achieves B|P = 86.6%, outperforming the best LLM by 13.0%, suggesting that generating efficient DS code remains far from saturated for current LLMs. Head-to-head comparison sharpens the picture. To move beyond aggregate metrics, we compute pairwise efficiency win rates on models’ common-correct subsets. For each model

Table 2: Model performance on DSEffi-Bench (T=0.0). Pass (abbr. P): tasks solved. Beyond (abbr. B): normalized efficiency score. B|P: Beyond restricted to passed tasks. Bold = best in tier; underline = second. Overall Model

NumPy

Pandas

P B|P

P B|P

PyTorch P B|P

SciPy

Polars

Others

P B|P

P B|P

P B|P

Pass

Beyond

B|P

66.9 62.2 61.3 59.7

48.0 44.6 42.4 39.4

71.7 71.6 69.2 66.1

76.8 71.4 64.9 67.3

78.9 77.2 73.4 68.3

65.3 58.0 59.6 57.7

71.2 71.9 69.6 69.6

64.7 73.5 67.6 55.9

53.6 60.9 49.4 48.2

64.6 70.8 75.0 62.5

82.3 67.3 80.3 74.6

72.6 62.9 72.6 67.7

59.7 57.4 66.9 57.4

37.6 32.5 38.5 36.8

48.6 62.0 57.1 55.8

57.7 56.1 40.2 42.8 57.6

37.5 30.4 29.6 26.0 39.1

65.0 54.2 73.6 60.6 67.9

65.5 66.0 44.6 47.9 67.0

70.7 59.9 79.6 64.4 70.2

54.9 52.7 41.0 43.5 57.1

63.2 52.3 74.0 60.3 72.2

58.8 57.4 41.2 30.9 52.9

40.4 46.2 59.1 35.5 51.0

68.8 58.3 27.1 47.9 43.8

67.6 64.2 87.6 64.9 69.8

46.8 51.6 53.2 40.3 58.1

72.1 41.1 58.1 56.0 62.0

40.2 33.3 21.4 29.9 35.9

55.5 36.8 60.0 57.4 53.3

Qwen3-Coder-Next Devstral-2-123B Qwen2.5-Coder-32B Qwen2.5-Coder-14B Codestral-22B DeepSeek-Coder-33B DeepSeek-Coder-V2-Lite

50.5 43.0 34.1 32.8 27.4 17.5 17.1

28.7 23.5 18.7 15.2 13.8 8.0 7.3

56.8 54.6 54.7 46.4 50.5 45.5 42.6

59.5 52.3 40.7 39.9 34.5 19.3 20.1

58.9 57.8 56.5 44.0 49.5 47.2 32.5

46.7 39.7 31.5 29.7 23.0 16.1 12.3

57.1 55.8 52.2 50.6 53.5 45.2 36.3

55.9 35.3 35.3 33.8 39.7 26.5 27.9

53.2 40.2 73.3 53.2 44.0 42.4 84.2

52.1 37.5 41.7 43.8 29.2 22.9 20.8

54.5 64.1 47.4 39.9 61.1 61.7 31.1

38.7 35.5 14.5 11.3 12.9 11.3 12.9

56.8 51.0 69.9 61.0 55.2 39.5 50.5

33.3 31.6 25.6 23.9 15.4 11.1 14.5

48.4 40.3 39.3 41.9 44.4 30.9 60.3

Human Reference

100.0

86.6

86.6

100.0 88.5 100.0 85.8 100.0 78.7 100.0 94.2 100.0 78.1 100.0 88.3

Closed Frontier GPT-5.4 GPT-5.4-mini Claude-Opus-4.6 Claude-Sonnet-4.6 Open Frontier DeepSeek-V4-Flash DeepSeek-V3.2 Kimi-K2.5 MiniMax-M2.5 Qwen3.5-397B/A17B Open Coder

M1. GPT-5.4

Avg Lose

34.1% 43.7% 27.7% 54.0% 56.2%

42.9% 25.2%

38.9% 29.6% 50.5% 53.8%

40.3% 28.0%

17.4% 41.9% 42.9%

30.0% 33.5%

52.7% 53.9%

39.5% 23.1%

36.4%

25.1% 46.7%

M6. 18.5% 20.1% 21.0% 18.7% 33.3% Devstral-2-123B (352) (329) (338) (230) (324) M1. M2. M3. M4. M5. M6.

22.4% 48.4%

(507)

M2. 31.6% Claude-Opus-4.6 (507)

(478)

(458)

M3. 23.0% 26.9% DeepSeek-V4-Flash (478) (458)

(346)

(318)

(328)

M4. 30.1% 33.0% 34.8% Kimi-K2.5 (346) (318) (328) M5. 21.7% 23.8% 24.2% 20.1% Qwen3-Coder-Next (411) (390) (384) (273)

(411)

(390)

(384)

(273)

(352)

(329)

(338)

(230)

(324)

55 50 45

Win Rate (%)

Avg Win

40 35 30 25 20 15

Figure 3: Pairwise efficiency comparison of six representative models.

pair (A, B), we restrict to tasks that both solve and define A as more efficient (winning) if (tB − tA )/tB ≥ 0.10 following Shypula et al. (2023). Figure 3 shows results for six representative LLMs. GPT-5.4 has the broadest win margin (Avg Win 42.9%, Avg Lose 25.2%), making it the most consistently fast model. Kimi-K2.5 follows closely in win rate (39.5%) with the lowest loss rate (23.1%), and is rarely the slower solver across opponents, indicating that its high B|P reflects genuine efficiency rather than a favorable task mix. DeepSeek-V4Flash falls on the other side, losing more matchups than it wins (33.5% vs. 30.0%) despite passing more tasks than Kimi. On the 131 tasks where all six models produce correct solutions, the ranking holds, with Kimi-K2.5 (B|P, 76.7%), GPT-5.4 (73.5%), Claude-Opus-4.6 (70.9%), DeepSeek-V4Flash (69.3%), Qwen3-Coder-Next (58.5%), and Devstral-2-123B (55.1%), ruling out correctness-

conditioned selection bias. These patterns consistently hold on the full 16model matrix (detailed in Appendix B.2), where GPT-5.4 maintains Avg Win 45.9% and Kimi-K2.5 the lowest Avg Lose (22.2%). Efficiency leadership rotates across libraries. We reveal where each model wins via per-library breakdown. As shown in Table 2, no single model dominates uniformly. Kimi-K2.5 leads B|P on 3 of 5 primary libraries (NumPy 79.6%, Pandas 74.0%, SciPy 87.6%); GPT-5.4-mini leads PyTorch among frontier models (60.9%); DeepSeek-V4-Flash leads on Polars (72.1%). Even within the same family, advantages do not transfer: GPT-5.4 outperforms GPT-5.4-mini on SciPy by 15.0% (82.3% vs. 67.3%) yet trails by 7.3% on PyTorch (53.6% vs. 60.9%). These rotating leads and intra-family reversals indicate that efficiency knowledge is library-specific rather than a general capability. Full per-library breakdowns appear in Appendix B.1. 5.2

Efficiency Gap Diagnosis

For each task, we collect all correct solutions across 16 models and select the maximum-ratio slow–fast pair. We retain only tasks whose slowdown ratio exceeds 1.5×, yielding 868 annotatable pairs. Two authors independently annotate the pairs and iteratively consolidate labels via a snowball procedure, converging on 5 Level-1 categories and 19 Level-2 sub-patterns (Appendix B.3.1). Cohen’s

Category

% Med. P75

Optimized Primitive Utilization(OPU) Complexity-Reducing Computation(CRC) Loop-to-Vectorization(L2V) Data Layout Optimization(DLO) Overhead-Aware Stack Matching(OSM)

23.3 5.1 18.9 20.9 11.6 64.6 19.2 10.6 36.6 20.2 2.9 7.6 16.5 14.4 98.4

Proportion (%)

Table 3: Five-category efficiency gap taxonomy.

% Med. OPU CRC L2V DLO OSM

27.2 13.6 7.3 22.5 29.3

3.0 7.4 4.9 2.5 6.2

P75

% Med.

P75

7.6 23.4 20.1 4.4 21.0

24.4 4.2 19.0 6.7 12.5 8.3 26.2 2.9 17.9 27.2

33.8 24.4 21.1 11.8 279.7

% Med.

CRC

10.8

11.2

80

17.5

25.3

60 40 20

25.3

32.8

21.4

32.8

26.4

14.7

20.0

27.4

6.9 10.3 17.2

L2V

DLO

14.0 4.7 11.6

29.1

30.2

14.5 10.9 16.4

39.5

OSM 46.3 16.4 23.9

29.1

13.4

0 NumPy Pandas PyTorch SciPy Polars Others

Table 4: Category distribution by slow-side model tier. Category Closed Frontier(191) Open Frontier(279) Open Coder(398)

OPU

100

Figure 4: Optimization gap distribution (%) by library.

P75

20.6 8.8 25.2 25.6 22.1 100.7 29.6 12.5 42.6 14.8 4.8 7.8 9.3 15.5 99.0

κ is 0.82 (almost perfect agreement (Landis and Koch, 1977)), after which remaining disagreements are resolved through discussion. LLM efficiency deficits are structured. The five categories are shown in Table 3: (1) Optimized Primitive Utilization (OPU), using a library’s existing optimized primitive instead of reimplementation; (2) Complexity-Reducing Computation (CRC), reducing computational complexity through algorithmic redesign; (3) Loop-toVectorization (L2V), replacing Python-level iteration with bulk array or table operations; (4) Data Layout Optimization (DLO), selecting data representations that match the computation and avoid unnecessary intermediates; and (5) OverheadAware Stack Matching (OSM), avoiding unnecessary setup, dispatch, or transfer costs by choosing a lighter computation layer. We observe that the five categories differ sharply both in frequency and severity. OPU is the most frequent (23.3%), with a moderate median slowdown (5.1×) because a single API substitution typically suffices (scipy.spatial.distance.cdist replacing a triple-nested loop). OSM is the least frequent (16.5%) but the most severe (median 14.4×), as unnecessary overhead can dominate execution time by orders of magnitude. CRC and L2V occupy the middle band (median 11.6× and 10.6×), requiring algorithmic restructuring or loop elimination; DLO is the mildest (median 2.9×). Model tiers exhibit distinct blind spots. Table 4 groups pairs by the slow-side model’s tier. Two categories show clear gradients: L2V rises from 7.3% (Closed Frontier) to 29.6% (Open Coder), while OSM drops from 29.3% to 9.3%. Open Coder models still write explicit loops where vectorized calls exist; Closed Frontier models have

mastered vectorization but default to heavier computation stacks than the task requires. Perhaps more surprisingly, Closed Frontier models also show a compositional bias in OPU (27.2% of their slow-side pairs): they favor constructing solutions from basic operations rather than reusing a library’s dedicated primitive, producing correct but suboptimal code. This build-over-reuse tendency aligns with observations that frontier coding agents prefer custom implementations over adopting existing tools (Ong and Vikati, 2026); our results locate the same pattern at the granularity of DS library APIs and quantify its performance cost. Each library demands a different optimization skill mix. Figure 4 reveals library-specific profiles. NumPy concentrates in CRC (26.4%) and L2V (25.3%), reflecting the library’s emphasis on algorithmic computation and arraylevel vectorization. Pandas shifts toward OPU (27.4%) and DLO (25.3%): models miss builtin methods (e.g., groupby.cummax instead of apply(lambda. . . )) and create unnecessary DataFrame copies. PyTorch and Polars share elevated OSM (32.8% and 29.1%), suggesting that models pay framework-dispatch costs disproportionate to the computation. SciPy is the most OPUconcentrated library (39.5%), as its rich function inventory means the fastest path is almost always a single specialized call. These divergent profiles, combined with the tierspecific blind spots above, account for the rotating efficiency rankings in §5.1: no model dominates across libraries because each library foregrounds a different optimization category, and each tier struggles with a different subset of those categories. 5.3

From Diagnosis to Improvement

We probe the value of our diagnostics via two experiments: taxonomy-guided optimization, and library-conditioned model routing (Lib-Routing).

Table 5: Exploration of efficiency-aware improvement. Strategy

OPT10

OPT50

Regr.

DeepSeek-V4-Flash (Ncorrect =577) 57.7 37.5 65.0 Basic EL 57.3 41.2 72.0 +7.0 EL w/ Rx 56.6 44.1 77.9 +12.9

– 22.7 35.5

– 11.4 20.8

– 3.8 6.1

Kimi-K2.5 (Ncorrect =402) Basic 40.2 29.6 EL 39.4 33.7 39.3 34.7 EL w/ Rx

– 37.6 54.5

– 19.4 27.6

– 3.2 4.5

5.3.1

Pass Beyond B|P

73.6 85.5 +11.9 88.3 +14.7

Taxonomy-Guided Optimization

Setup. We test whether the taxonomy from §5.2 can serve as actionable guidance for inferencetime improvement. We adopt Effi-Learner (Huang et al., 2024a) (EL), a state-of-the-art optimization method: given a correct solution and its line-level execution profile, EL iteratively prompts the model to produce a faster version. We extend EL with a taxonomy-derived prescription (EL w/ Rx, where Rx denotes prescription): the system prompt is augmented with a library-specific optimization guide listing the five L1 categories, each with its L2 sub-patterns and representative code examples (Appendix B.5.1, Figure 22). We treat the correctness tests as public tests exposed to each method for three rounds of iterative optimization, using them to assess both correctness and efficiency when selecting candidates. Final evaluation uses the full suite of correctness and stress tests. We apply both variants to DeepSeek-V4-Flash (Ncorrect =577) and Kimi-K2.5 (Ncorrect =402) across 1,000 tasks; tasks that fail under Basic prompting retain their original submission. OPT10 and OPT50 measure the fraction of originally correct tasks whose optimized solutions remain correct and reduce execution time by at least 10% and 50% relative to their Basic-prompting baselines, respectively. Higher values indicate that optimization benefits more tasks. Regression (Regr.) measures the fraction of originally correct tasks whose optimized solutions become incorrect or run at least 10% slower than their baselines. Taxonomy-guided optimization yields substantial efficiency gains. Table 5 reports the results. Vanilla EL raises B|P by +7.0% (DeepSeek) and +11.9% (Kimi) at negligible Pass cost (≤0.8%), confirming that profile-guided reprompting can improve efficiency without retraining. Injecting taxonomy-guided optimization amplifies the gain: EL w/ Rx reaches +12.9% and +14.7% in B|P, producing ≥2× speedups on 20.8% and 27.6% of originally correct tasks while regression stays within 6.1%. The per-library breakdown (Appendix B.5.1)

Table 6: Lib-Routing vs. closed-frontier baselines on the temporal test set (N =304). Configuration

Pass Beyond B|P $/task

Claude-Opus-4.6 Claude-Opus-4.6 Best@3 GPT-5.4 GPT-5.4 Best@3

63.5 72.0 63.5 73.4

43.7 53.8 46.4 58.9

68.9 74.6 73.0 80.3

0.018 0.052 0.014 0.043

Lib-Routing (K=3)

69.4

51.4

74.1

0.004

shows that EL w/ Rx outperforms vanilla EL across all six library groups for DeepSeek-V4-Flash and five of six for Kimi-K2.5, indicating that the gains are not concentrated in a single library within DSEffi-Bench. 5.3.2

Library-Conditioned Routing

Setup. Since efficiency knowledge is libraryspecific, a natural complement to per-model optimization is routing each task to the open model best suited for its library. We split the 1,000 tasks into training (60%), validation (10%), and test (30%) sets for each library in temporal order. The training set is used to rank 12 open models and select the top K=3 experts per library. The validation set selects α for the composite Pass and B|P ranking score, yielding α=0.35. At test time, each task is routed to its library’s three experts, and the fastest correct solution is submitted. We compare this fixed configuration against closed-frontier baselines on the unchanged temporal test set (selection and scoring details in Appendix B.5.2). Library-conditioned routing approaches closed-frontier quality at a fraction of cost. Table 6 shows that Lib-Routing outperforms both closed-frontier Best@1 baselines in Pass and B|P. It approaches Claude-Opus-4.6 Best@3 in B|P (74.1 vs. 74.6). At $0.004/task, Lib-Routing is 13.0× cheaper than Claude-Opus-4.6 Best@3. This result is consistent with the library-level complementarity observed in §5.2, where different models exhibit strengths across different libraries.

6

Discussion

Failure-Type Breakdown of Pass. Pass treats both functional errors and solutions exceeding the 5second timeout as failures, potentially conflating correctness with runtime behavior. We therefore break down the Pass outcomes by failure type. Concretely, at T=0.0, we classify solutions from all evaluated models as Correct, wrong answer (WA), runtime error (RE), or time limit exceeded (TLE). Each failed solution is labeled by its first failing

test, following competitive-judging practice. Correctness tests run first, and only passing solutions proceed to stress tests. The resulting distribution is 45.4% Correct, 32.9% WA, 20.4% RE, and 1.3% TLE. The 1.3% TLE share consists of 0.8% on correctness tests and 0.5% on stress tests. These results show that Pass failures primarily arise from WA and RE rather than timeouts. Future Scoring Protocol for Beyond. Beyond is computed relative to the current task-level solution pool, so adding a new model may change the fastest or slowest anchor and rescale existing scores. To evaluate a new model, researchers generate outputs under the same protocol and run the task-specific tests against the references. Correct solutions are then time-profiled and added to the corresponding task-level spectra. Researchers then recompute Beyond for both the new and existing models using the expanded spectra. Licensing and Attribution. All SO posts used in this work were published in or after 2021 and are licensed under CC BY-SA 4.0. Our benchmark retains source links for attribution and distributes SO-derived content under the same license. Societal Impact. This work encourages the development of models that generate more efficient DS code, potentially reducing computational costs and energy use in DS workflows. Our benchmark and taxonomy may support future research on efficiency-aware prompt engineering, model routing, and post-training.

7

Conclusion

We present DSEffi-Bench, a 1,000-instance benchmark that jointly evaluates correctness and execution efficiency of LLM-generated DS code. Evaluating 16 LLMs across 3 tiers, we find that correctness is not sufficient to characterize efficiency, with notable rank inversions where the models strongest in correctness are not those strongest in efficiency. A five-category taxonomy traces these deficits to library- and tier-dependent root causes. The taxonomy further shows potential for guiding singlemodel optimization and lower-cost cross-model routing within the benchmark. Together, our benchmark, taxonomy, and diagnostic experiments offer a foundation for future work on improving LLMs’ ability to generate efficient DS code.

Limitations Programming Language Scope. This work fo-

cuses on Python-based DS coding. Other languages, such as R and MATLAB, are outside our evaluation scope. LLM-generated code in these languages may exhibit different efficiency patterns and bottlenecks. Environment Dependence. Our evaluation protocol relies on test execution, making runtime measurements dependent on hardware resources and software versions. We therefore evaluate all candidate solutions in the same controlled environment. Data Provenance and Contamination. DSEffiBench is derived from public Stack Overflow posts. Evaluated models may have encountered the source content during training, and some recent answers may contain AI-generated or AI-assisted content. We mitigate these risks through task reformulation and human validation.

Acknowledgments This work is supported by the National Natural Science Foundation of China under Grant Nos. 62232001 and 62402482. Jie M. Zhang is supported by the GENIUS project (grant no. 23026).

References Anthropic. 2026. Claude Opus 4.6 and Claude Sonnet 4.6. https://www.anthropic.com/. Accessed: 2026-05-23. Ruisheng Cao, Mouxiang Chen, Jiawei Chen, Zeyu Cui, Yunlong Feng, Binyuan Hui, Yuheng Jing, Kaixin Li, Mingze Li, Junyang Lin, and 1 others. 2026. Qwen3-coder-next technical report. arXiv preprint arXiv:2603.00729. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 4(5). Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. DeepSeek AI. 2026. DeepSeek V4 Preview Release. https://api-docs.deepseek.com/news/ news260424. Accessed: 2026-05-23. Mingzhe Du, Luu A Tuan, Bin Ji, Qian Liu, and SeeKiong Ng. 2024. Mercury: A code efficiency benchmark for code large language models. Advances in Neural Information Processing Systems, 37:16601– 16622. Zhihao Gong, Zeyu Sun, Dong Huang, Qingyuan Liang, Jie M Zhang, and Dan Hao. 2026. Trace: Evaluating execution efficiency of llm-based code translation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3089–3117. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yifan Wu, YK Li, and 1 others. 2024. Deepseekcoder: when the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Yin Zheng, Zejian Yuan, and Zejun Ma. 2025. Swe-perf: Can language models optimize code performance on real-world repositories? arXiv preprint arXiv:2507.12415. Dong Huang, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao Qing, Heming Cui, Zhijiang Guo, and Jie M Zhang. 2024a. Effilearner: Enhancing efficiency of generated code via self-optimization. Advances in Neural Information Processing Systems, 37:84482– 84522. Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M Zhang. 2024b. Effibench: Benchmarking the efficiency of automatically generated code. Advances in Neural Information Processing Systems, 37:11506–11544.

Dong Huang, Guangtao Zeng, Jianbo Dai, Meng Luo, Han Weng, Yuhao Qing, Heming Cui, Zhijiang Guo, and Jie M Zhang. 2024c. Efficoder: Enhancing code generation in large language models through efficiency-aware fine-tuning. arXiv preprint arXiv:2410.10209. Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, and Nan Duan. 2022. Execution-based evaluation for data science code generation models. In Proceedings of the Fourth Workshop on Data Science with Humanin-the-Loop (Language Advances), pages 28–36. Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and 1 others. 2024d. Da-code: Agent data science code generation benchmark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13487–13521. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2.5-coder technical report. arXiv preprint arXiv:2409.12186. Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, volume 2025, pages 58791–58831. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157. Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. 2024. Dsbench: How far are data science agents from becoming data science experts? arXiv preprint arXiv:2409.07703. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. PMLR. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024b. Evaluating language models for efficient code generation. arXiv preprint arXiv:2408.06450. MiniMax. 2026. MiniMax M2.5: Built for RealWorld Productivity. https://www.minimax.io/ news/minimax-m25. Accessed: 2026-05-23. Mistral AI. 2024. Codestral. https://mistral.ai/ news/codestral. Accessed: 2026-05-23. Mistral AI. 2025. Devstral 2. https://docs.mistral. ai/models/model-cards/devstral-2-25-12. Accessed: 2026-05-23. Changan Niu, Ting Zhang, Chuanyi Li, Bin Luo, and Vincent Ng. 2024. On evaluating the efficiency of source code generated by llms. In Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering, pages 103–107. Edwin Ong and Alex Vikati. 2026. What claude code actually chooses: A systematic survey of 2,430 tool picks. V1.0. 3 models, 4 repos, 20 categories. OpenAI. 2026. GPT-5.4 and GPT-5.4 mini. https: //openai.com/. Accessed: 2026-05-23. Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. 2025. Kernelbench: Can llms write efficient gpu kernels? arXiv preprint arXiv:2502.10517. Shuyin Ouyang, Dong Huang, Jingwen Guo, Zeyu Sun, Qihao Zhu, and Jie M Zhang. 2026. Dscodebench: A realistic benchmark for data science code generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 32628–32636. Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. Coffe: A code efficiency benchmark for code generation. Proceedings of the ACM on Software Engineering, 2(FSE):242–265. Ori Press, Brandon Amos, Haoyu Zhao, Yikai Wu, Samuel Ainsworth, Dominik Krupke, Patrick Kidger, Touqir Sajed, Bartolomeo Stellato, Jisun Park, and 1 others. 2026. Algotune: Can language models speed up general-purpose numerical programs? Advances in Neural Information Processing Systems, 38. Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, and 1 others. 2021. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655.

Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie Zhang, Heming Cui, Siu Ming Yiu, Dong Huang, and 1 others. 2026. Effibench-x: A multi-language benchmark for measuring efficiency of llm-generated code. Advances in Neural Information Processing Systems, 38. Ruizhong Qiu, Weiliang Zeng, James Ezick, Christopher Lott, and Hanghang Tong. 2025. How efficient is llm-generated code? a rigorous & high-standard benchmark. In International Conference on Learning Representations, volume 2025, pages 2233–2261. Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, and Ion Stoica. 2026. Gso: Challenging software optimization tasks for evaluating swe-agents. Advances in Neural Information Processing Systems, 38. Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. 2023. Learning performance-improving code edits. arXiv preprint arXiv:2302.07867. Stack Exchange Inc. 2024. Stack exchange api: Advanced search. https://api.stackexchange. com/docs/advanced-search. Accessed: 2026-0520. Maojun Sun, Yifei Xie, Yue Wu, Ruijian Han, Binyan Jiang, Defeng Sun, Yancheng Yuan, and Jian Huang. 2026. Dsaeval: Evaluating data science agents on a wide range of real-world data science problems. arXiv preprint arXiv:2601.13591. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, and 1 others. 2026. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276. Siddhant Waghjale, Vishruth Veerendranath, Zhiruo Wang, and Daniel Fried. 2024. Ecco: Can we improve model-generated code efficiency without sacrificing functional correctness? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15362–15376. Yangsen Wang, Yizhou Chen, Zhihao Gong, and Dan Hao. 2026. Lamda: Leveraging large language models for better api misuse detection via multi-agent collaboration. ACM Transactions on Software Engineering and Methodology. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 13484–13508. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2024. Self-preference bias in llm-as-a-judge. arXiv preprint arXiv:2410.21819.

Jie Wu, Haoling Li, Xin Zhang, Jiani Guo, Jane Luo, Steven Liu, Yangyu Huang, Ruihang Chu, Scarlett Li, and Yujiu Yang. 2026. X-coder: Advancing competitive programming with fully synthetic tasks, solutions, and tests. arXiv preprint arXiv:2601.06953. Pengyu Xue, Kunwu Zheng, Zhen Yang, Yifei Pei, Linhao Wu, Jiahui Dong, Xiapu Luo, Yan Xiao, Fei Liu, Yuxuan Zhang, and 1 others. 2025. Translibeval: Demystify large language models’ capability in thirdparty library-targeted code translation. arXiv preprint arXiv:2509.12087. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Shaoxiong Zhan, Yanlin Lai, Ziyu Lu, Dahua Lin, Ziqing Yang, and Fei Tan. 2026. Mathsmith: Towards extremely hard mathematical reasoning by forging synthetic problems with a reinforced policy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34602–34610. Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, and Yisong Yue. 2025. Datascibench: An llm agent benchmark for data science. arXiv preprint arXiv:2502.13897. Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, and 1 others. 2024. Deepseekcoder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931.

A

Benchmark Details

This section provides supplementary details of the benchmark construction described in §3. A.1

Comparison with Existing Benchmarks

Table 7 positions DSEffi-Bench among 12 related benchmarks along 6 dimensions. Competitiveprogramming benchmarks (EffiBench(-X), Mercury, COFFE, ENAMEL, EvalPerf) (Huang et al., 2024b; Qing et al., 2026; Du et al., 2024; Peng et al., 2025; Qiu et al., 2025; Liu et al., 2024b) evaluate efficiency on algorithmic problem-solving, anchored to human-curated or model-generated reference solutions. Repository-level benchmarks (SWE-Perf, GSO) (He et al., 2025; Shetty et al., 2026) target implementation-level optimization in real-world software projects. Scientific-computing benchmarks (KernelBench, AlgoTune) (Ouyang et al., 2025; Press et al., 2026) focus on kernel or numerical solver optimization. None of these address DS library-conditioned efficiency. Existing DS benchmarks (ExeDS, DS-1000, DSCodeBench) (Huang et al., 2022; Lai et al., 2023; Ouyang et al., 2026) assess only functional correctness without measuring execution efficiency. DSEffi-Bench fills this gap with 1,000 human-validated, efficiency-oriented instances spanning 10+ DS libraries, grounded in a multi-model efficiency spectrum plus a humanvalidated reference for efficiency scoring.

Question-Level Relevance Filter. Given the post title and body, the judge determines whether the post genuinely poses a DS performance optimization problem and returns a structured {is_relevant, reason} decision (Figure 10). Answer-Level Feasibility Filter. Given the full post (question and accepted answer), the judge determines whether the answer provides a reproducible optimization under our evaluation environment, applying four Feasible criteria and six NonFeasible exclusions (Figure 11). A.5

Instance Transcription

Table 8 lists the fields that constitute a complete instance, divided into the problem description X (provided to models), the evaluation harness T (used for correctness and efficiency measurement), and the reference solution Sref .

This subsection provides the full prompts for the three transcription steps in §3.3: Problem Description Generation (X, Figure 12), Evaluation Harness Generation (T , Figure 13), and MultiDimensional Scoring (Figure 14). Problem Description Generation. Given the original SO question body, the model produces five structured fields (task_description, input_specification, output_specification, input_example, and output_example) under the unified function signature def solution (input_data = None), with type/shape annotations for all I/O fields. Evaluation Harness Generation. Given the five X-fields and the original SO accepted answer, the model generates the three T components: input_domain, input_generator, and output_validator. Multi-Dimensional Scoring. The quality scorer belongs to a different model family than the transcription model to avoid self-evaluation bias (§3.3). It receives the complete instance and rates it on four dimensions.

A.3

A.6

A.2

Benchmark Instance Schema

Efficiency-oriented Post Collection

We adopt 21 DS-related tags and 21 efficiencyoriented keywords shown in Figure 9, surveyed from prior DS benchmarks (Lai et al., 2023; Ouyang et al., 2026; Jing et al., 2024; Zhang et al., 2025) and code efficiency studies (Qing et al., 2026; Qiu et al., 2025; Waghjale et al., 2024; Peng et al., 2025). The tag set spans the dominant tools in everyday DS workloads; alias merges (e.g., pandas/dataframe) consolidate semantically equivalent SO tags into a single retrieval channel. A.4

Relevance and Feasibility Filtering

This subsection presents the full prompts for the two-stage LLM-as-Judge filtering (§3.2).

Human-Validated Reference Solution

Reference Solution vs. Accepted Answer Analysis. During annotation, we used the SO accepted answer as a starting point and categorized each resulting reference solution into one of three relationships: (1) Faithful Adoption (we directly implement the accepted answer’s optimization strategy), (2) Refinement (we follow the same high-level idea but improve implementation details), and (3) Alternative Strategy (we determine that a fundamentally different approach is more efficient). Figure 5 shows the distribution. About 17.5% of reference solutions are faithful adoptions of the accepted answer; 32.2% refine the answer’s strategy with implementation improvements; and 50.3%

Table 7: Comparison with related benchmarks. The last 3 columns: 1) whether a benchmark evaluates execution efficiency, 2) what type of bottleneck it targets, and 3) what reference anchor grounds the evaluation. Benchmark

Task Form

Effi.

Effi. Domain

Efficiency Anchor

Competitive Programming EffiBench 1,000 LeetCode Mercury 1,889 LeetCode COFFE 756 HumanEval/MBPP/APPS ENAMEL 142 HumanEval EvalPerf 121 HumanEval/MBPP

Scale

Code Generation Code Generation Code Generation Code Generation Code Generation

✓ ✓ ✓ ✓ ✓

Algorithmic Algorithmic Algorithmic Algorithmic Algorithmic

1 human-optimized solution Multiple human solutions 1 ground-truth solution 1 expert-designed solution Multi-model solution clusters

Repository-Level Software SWE-Perf 140 GitHub PRs GSO 102 GitHub commits

Patch Generation Patch Generation

✓ ✓

Impl.-level Impl.-level

1 expert-optimized PR patch 1 expert-optimized commit

Kernels & Scientific Computing KernelBench 250 PyTorch ops AlgoTune 154 Domain experts

Code Optimization Code Optimization

✓ ✓

Hardware Algorithmic

1 PyTorch reference impl. 1 human-authored reference solution

Data-Science Code Generation ExeDS 534 Jupyter notebooks DS-1000 1,000 Stack Overflow DSCodeBench 1,000 GitHub

Code Completion Code Completion Code Generation

✗ ✗ ✗

— — —

— — —

DSEffi-Bench

Code Generation

✓

Library idiom

Multiple model solutions + 1 human-validated reference

1,000

Task Source

Stack Overflow

Table 8: Benchmark instance schema.

X

T

Sref

Field

Description

task_description input_specification output_specification input_example output_example

str. Problem statement preserving the original SO post’s optimization intent str. Input types, shapes, and constraints str. Return type, shape, and semantics str. Concrete input case str. Expected output for the input example

input_domain input_generator output_validator

List[str]. Testing conditions (correctness and stress sub-domains) input_generator(seed, input_domain). Deterministic generator producing valid inputs output_validator(pred_output, ref_output). Correctness checker

reference_solution

def solution(input_data=None). Human-validated solution

Faithful Adoption 17.5% (175)

0

Alternative Strategy 50.3% (503)

Refinement 32.2% (322)

20

40

60

Percentage of Benchmark Instances (%)

80

100

Figure 5: Distribution of reference solution strategies relative to the SO accepted answer.

employ an entirely different optimization approach. This distribution reveals several properties of our benchmark: (1) Genuine human effort. Over 80% of instances required annotators to go beyond transcribing the SO answer, either improving its implementation or devising a superior strategy within their expertise. (2) Grounded construction. The accepted answer served as a meaningful starting point: nearly half of all instances (Faithful Adoption and Refinement combined, 49.7%) build on the answer’s core idea, ensuring the reference solutions are anchored in real practitioner knowledge. (3) Reduced dependence on accepted-answer strategies. Because 82.5% of references refine or replace the accepted answer’s strategy, most are not direct

implementations of the source answer. We illustrate each category with a representative example (Figures 6–8). The original SO posts are lightly reformatted (whitespace and HTML markup) for readability. Example 1 (Faithful Adoption, Figure 6). Here the reference directly adopts the accepted answer’s strategy: extract coordinate index arrays and call np.maximum.at per channel. The only differences are function-signature adaptation (solution(input_data)) and grid-shape parameterization; the core optimization is identical. Example 2 (Refinement, Figure 7). Both solutions use the same mathematical transformation (prefix product plus cumulative sum) to eliminate the Python loop, but the reference streamlines the implementation. The accepted answer constructs an auxiliary array via np.insert(x[1:], 0, 0) and special-cases x[0]; the reference builds P directly (P[0]=1, P[1:]=cumprod(d[1:])), yielding a single expression P * cumsum(x / P) that avoids the intermediate allocation and branch. Example 3 (Alternative Strategy, Fig-

ure 8). The accepted answer uses Pandas’ expanding().apply() with a custom lambda, invoking a Python callback at every row that scans all prior elements, resulting in O(n2 ) total work with per-invocation interpreter overhead. After discussion, the annotators adopted a fundamentally different strategy: maintaining a sorted list of prior values and using binary search (bisect) to count elements exceeding the current value in O(log k) per step (where k is the number of prior elements), while insertion remains O(k) via C-level memmove. The total complexity is still O(n2 ), but the elimination of Python-level callbacks and pandas overhead yields a substantial constant-factor speedup. Scope of the reference. The human-validated reference solution is not necessarily the fastest implementation for every task. As our evaluation results confirm (§5.1), some LLM-generated solutions outperform the reference on individual instances (the human reference achieves Beyond = 86.6%, not 100.0%). However, the reference serves as a grounded correctness and efficiency baseline, informed by both the SO community’s domain knowledge and the annotators’ optimization expertise. A.7

Metric Robustness Analysis

We evaluate whether model rankings are sensitive to the choice of normalization strategy for the spectrum score. Three alternatives are considered: • Rank-based (ordinal): score = (n − r)/(n − 1), where r is the 1-indexed rank by execution time. Ignores execution time magnitudes. • Linear min-max (Mercury-style): score = (tmax − tk )/(tmax − tmin ). Uses raw execution time values. • Log-scale min-max (adopted): score = (log tmax − log tk )/(log tmax − log tmin ). Normalizes in log-space. Table 9 reports B|P under all three strategies. Rankings are robust: the Spearman correlation between log-scale and linear is ρ = 0.994 (p < 10−14 ); between log-scale and rank-based is ρ = 0.961 (p < 10−8 ). Kendall τ shows only 2 discordant pairs (out of 120) for linear vs. log-scale and 8 for rank-based vs. log-scale, all concentrated among models with B|P differences below 2%. Why log-scale. Per-task execution time spans are wide: the median max/min ratio across tasks with

Table 9: B|P (%) under three normalization strategies. Normalization Strategy Model

Rank-based Linear-based Log-based

Closed Frontier 59.5 60.2 60.9 52.1

83.3 83.7 80.2 79.0

71.7 71.6 69.2 66.1

49.3 42.5 63.6 50.3 57.3

78.0 67.9 84.5 74.5 81.1

65.0 54.2 73.6 60.6 67.9

Devstral-2-123B Qwen3-Coder-Next Qwen2.5-Coder-32B Qwen2.5-Coder-14B Codestral-22B DeepSeek-Coder-33B DeepSeek-Coder-V2-Lite

43.2 43.5 46.5 38.8 44.0 38.8 35.7

70.8 72.5 71.0 63.9 66.2 62.1 57.8

54.6 56.8 54.7 46.4 50.5 45.5 42.6

Human Reference

79.0

91.9

86.6

GPT-5.4 GPT-5.4-mini Claude-Opus-4.6 Claude-Sonnet-4.6 Open Frontier DeepSeek-V4-Flash DeepSeek-V3.2 Kimi-K2.5 MiniMax-M2.5 Qwen3.5-397B/A17B Open Coder

multiple correct solutions is 14.5×, and the 90th percentile reaches 522.0×. Under linear normalization, the median spectrum score among all correct LLM solutions is 0.93 (46.6% above 0.95), compressing meaningful efficiency differences into a narrow range. Log-scale normalization yields a median of 0.74, preserving discriminative range while assigning equal score intervals to equal multiplicative speedups. This matches how efficiency gains are conventionally interpreted: a 2.0× speedup carries the same engineering significance whether from 1 ms to 0.5 ms or from 1 s to 0.5 s. A.8

Solution Generation Prompt

Figure 15 shows the prompt used for all 16 models to generate solutions in DSEffi-Bench. The prompt is intentionally neutral: it asks for a correct solution without explicitly requesting optimization, so that the resulting code reflects each model’s natural coding style rather than a prompted optimization effort.

Example 1 — Faithful Adoption (SO #78920216) Question: Optimizing Point Cloud to Voxel Grid with Max Sampling in NumPy I have two arrays that represent the point coordinates and values respectively. To max sample from this point cloud, I am initializing a grid with the desired size, and looping over each point to assign the max values: N = 1000000 coords = np.random.randint(0, 256, size=(N, 3)) vals = np.random.rand(N, 3) grid = np.zeros((3, 256, 256, 256), dtype=np.float16) for i, pt in enumerate(coords): x, y, z = pt grid[0, x, y, z] = max(grid[0, x, y, z], vals[i, 0]) grid[1, x, y, z] = max(grid[1, x, y, z], vals[i, 1]) grid[2, x, y, z] = max(grid[2, x, y, z], vals[i, 2]) Is there a way I can do this through NumPy without the for loop (which is very slow)? Accepted Answer: np.maximum.at can be used to perform the max operation in place at specific indices: indices = coords[:, 0], coords[:, 1], coords[:, 2] np.maximum.at(grid[0], indices, vals[:, 0]) np.maximum.at(grid[1], indices, vals[:, 1]) np.maximum.at(grid[2], indices, vals[:, 2]) Reference Solution: import numpy as np def solution(input_data=None): coords = input_data['coords'] vals = input_data['vals'] grid_shape = input_data.get('grid_shape', (3, 256, 256, 256)) C, X, Y, Z = grid_shape grid = np.zeros(grid_shape, dtype=np.float64) x = coords[:, 0] y = coords[:, 1] z = coords[:, 2] for c in range(C): # same strategy np.maximum.at(grid[c], (x, y, z), vals[:, c]) return grid

Figure 6: Example 1: Faithful Adoption — reference directly implements the accepted answer’s strategy.

Example 2 — Refinement (SO #77752955) Question: Dynamically Discounted Cumulative Sum in NumPy I have two arrays of the same length: one with values and one with dynamic decay factors; and wish to calculate a vector of the decayed cumulative sum at each position. Using a Python loop to express the desired recurrence: c = np.empty_like(x) c[0] = x[0] for i in range(1, len(x)): c[i] = c[i-1] * d[i] + x[i] The Python code is very clear but slows things down significantly. Is there a more “Numpythonic” way to express this without sacrificing clarity or efficiency? Accepted Answer: This is a first-order non-homogeneous recurrence with variable coefficients. Note that d[0] is supposed to be equal to 1. The values in c do not depend on previous values in c: g = np.insert(x[1:], 0, 0) # auxiliary shifted array pd = np.cumprod(d) # full prefix product c = pd * (x[0] + np.cumsum(g / pd)) # x[0] handled separately Reference Solution: import numpy as np def solution(input_data=None): x = np.asarray(input_data['x'], dtype=np.float64) d = np.asarray(input_data['d'], dtype=np.float64) n = x.shape[0] if n == 0: return np.array([], dtype=np.float64) P = np.empty(n, dtype=np.float64) P[0] = 1.0 if n > 1: np.cumprod(d[1:], out=P[1:]) # direct construction, no np.insert return P * np.cumsum(x / P) # unified expression, no x[0] special case

Figure 7: Example 2: Refinement — reference streamlines the accepted answer’s approach.

Example 3 — Alternative Strategy (SO #68057308) Question: Compute Frequency of Numbers with Greater Values Before Current Date I’m trying to count the frequency of the number of occurrences a value has been higher than today’s value for all dates in the past. Below is a working but inefficient approach: def freq_greater_than(r): smaller_date = df[df.date < r.date] larger_num = smaller_date[(smaller_date.num > r.num)] return round(len(larger_num) / len(smaller_date) * 100, 2) Given a DataFrame with dates and numeric values, I need to compute for each row the percentage of previous dates where the numeric value was greater than the current row’s value. Accepted Answer: Try with expanding apply + mean over gt comparison: df['freq_greater_than'] = ( df['num'].expanding() .apply(lambda s: s.iloc[:-1] .gt(s.iloc[-1]).mean() * 100) )

# O(n^2): Python callback at every row

Reference Solution: import numpy as np import pandas as pd from bisect import bisect_left, bisect_right, insort_left def solution(input_data=None): dates = input_data['dates'] values = input_data['values'] n = len(values) result = np.full(n, np.nan, dtype=np.float64) sorted_prev = [] # <-- different strategy: for i in range(n): # maintain sorted list if i > 0: pos_right = bisect_right(sorted_prev, values[i]) count_greater = len(sorted_prev) - pos_right # O(log n) per step result[i] = round(count_greater / len(sorted_prev) * 100, 2) insort_left(sorted_prev, values[i]) # insert in O(n), but C-level memmove is fast df = pd.DataFrame({ 'date': pd.to_datetime(dates), 'num': values, 'freq_greater_than': result}) return df

Figure 8: Example 3: Alternative Strategy — reference departs from the accepted answer entirely.

DS Library Tags (with aliases) Numerical / scientific: numpy, scipy, cvxpy DataFrame / out-of-core: pandas, dataframe, polars, python-polars, dask Classical ML: scikit-learn, sklearn, lightgbm, xgboost, catboost Deep learning: pytorch, tensorflow, keras, huggingface Visualization: matplotlib, seaborn Compiled acceleration: cython, numba Efficiency-Related Keywords General performance: performance, optimization, optimize, efficiency, efficient, inefficient Speed: speed, slow, fast Profiling: profile, profiling, bottleneck Memory: memory, out of memory Hardware / parallel: gpu, cuda, parallel Code patterns: vectorize, vectorization, loop, overhead

Figure 9: DS-related tags and efficiency-oriented keywords used for SO post retrieval.

Question-Level Relevance Filter (System) You are a Data Science (DS) expert specialized in performance optimization with Python's DS ,→ libraries. Given a Stack Overflow (SO) post, determine if it is relevant to code performance ,→ optimization in the DS programming context. Question-Level Relevance Filter (User) Evaluate if the following SO post is RELEVANT to Data Science Code Performance Optimization. **Title**: {title} **Body**: {body} ### RELEVANT Criteria (Must satisfy ALL): 1. Context: Python code using DS libraries (NumPy, Pandas, Scikit-learn, PyTorch, SciPy, etc.). 2. Goal: Code Performance Optimization (speed, throughput, vectorization, etc. ). 3. Clarity: The post clearly describes the functional scenario, the specific performance ,→ optimization requirement, and the input/output data structure. ### IRRELEVANT Criteria (Filter out if ANY applies): 1. Non-Python (R, Julia, Java, C++, SQL, shell). 2. Pure correctness/debugging (unless performance-related). 3. Basics/concept/theory without code optimization. 4. External system/configuration (cluster setup, cloud config). 5. Non-standard input (e.g., images or screenshots). 6. Web/App development (Django, Flask, React, mobile). ### Output Format {"is_relevant": True or False, "reason": "..."}

Figure 10: Prompt for question-level relevance filtering.

Answer-Level Feasibility Filter (System) You are a Data Science (DS) expert specialized in performance optimization with Python's DS ,→ libraries. Given a Stack Overflow (SO) post (Question + Answer), determine if the Answer ,→ provides a feasible code performance optimization solution. Answer-Level Feasibility Filter (User) Evaluate if the following SO Answer is a FEASIBLE SOLUTION for DS Code Performance Optimization. **Question Title**: {title} **Question Body**: {q_body} **Answer Body**: {a_body} **Environment**: Linux Ubuntu 22.04, 96 CPU cores, 512 GB RAM, 4 * H200 GPUs, Python 3.12, standard ,→ DS libraries. ### Feasibility Criteria (Must satisfy ALL): 1. Reproducibility: concrete solution (code or precise description) that solves the problem. 2. DS Coding: uses Python DS libraries (Numpy, Pandas, etc. ). 3. Optimization: explicitly addresses performance (vectorization, faster API path, algorithmic ,→ improvement, etc. ). 4. Clarity: I/O specification can be adopted from the context. ### Non-Feasibility Criteria (Filter out if ANY applies): 1. Negative/impossible ("Cannot be done", "Switch to C++"). 2. Non-code solution (change environment, use larger cluster). 3. External dependencies (non-standard setup, external files). 4. External references as primary content (links to docs/tutorials). 5. Non-optimization (only fixes syntax/bug without performance). 6. Irrelevant or off-topic. ### Output Format {"is_feasible": True or False, "reason": "..."}

Figure 11: Prompt for answer-level feasibility filtering.

Problem Description Generation (System) You are an expert Data Science (DS) coding task designer. Given a seed Stack Overflow (SO) post, ,→ transform it into a standard coding task for evaluation. Problem Description Generation (User) Analyze the SO post and reformulate it into a standard coding task with: Problem Description, I/O ,→ Specification, I/O Examples. **Question Title**: {title} **Question Body**: {body} ### Objectives 1. task_description: Transform the SO question into a clear problem in Markdown. Preserve the ,→ original questioner's writing style. Minimal edits; keep original intent and constraints. 2. input_specification, output_specification: Assume the solution signature is: def solution(input_data): Define input as a dictionary with typed keys (e.g., 'x': numpy array of float64 with shape (N,)). ,→ Define output with precise type, shape, and semantics. 3. input_example, output_example: Provide a structurally complete dictionary input matching the ,→ spec, and the corresponding expected output. ### Output Format (Strict JSON) { "task_description": "...", "input_specification": "...", "input_example": "...", "output_specification": "...", "output_example": "..." }

Figure 12: Prompt for problem description generation.

Evaluation Harness Generation (System) You are an expert Data Science (DS) coding task designer. You are tasked with designing tests for a ,→ Python DS coding problem. Evaluation Harness Generation (User) ### Task Context **Problem Description**: {task_desc} **Input Spec**: {input_spec} **Output Spec**: {output_spec} **Input Example**: {input_example} **Output Example**: {output_example} **Solution Signature**: def solution(input_data): ... ### Components to Generate 1. input_domain: List of strings. Include correctness tests (prefix correctness-) and stress tests ,→ (prefix stress-). - correctness-small, correctness-medium, correctness-corner-case - stress-medium, stress-large 2. input_generator: def input_generator(seed, input_domain): - Set seed via random.seed(seed) and np.random.seed(seed) - Branch on input_domain to produce one valid input_data dict - Same (seed, domain) -> identical data; different seeds -> different data - Yield a single tuple (case_name, input_data) - stress inputs must complete within 5s on 96 cores / 512GB RAM 3. output_validator: def output_validator(pred_output, ref_output): - Return (True, "") or (False, "reason") - Use numpy.testing or np.allclose with tolerance ~1e-4 ### Output Format (JSON) { "input_domain": [...], "input_generator": "...", "output_validator": "..." }

Figure 13: Prompt for evaluation harness generation.

Multi-Dimensional Scoring (System) You are DS Code Benchmark Quality Judge. You will receive fields defining ONE benchmark instance ,→ for Python DS performance optimization. Evaluate quality and assign scores. Multi-Dimensional Scoring (User) ## Evaluation Environment Linux, 96-core CPU, 4x H200, 512GB RAM, no network access. ## Instance Fields - original_post: {original_post} - question: {question} - input_spec: {input_spec} - output_spec: {output_spec} - input_example: {input_example} - output_example: {output_example} - input_domain: {input_domain} - input_generator: {input_generator} - output_validator: {output_validator} ## Hard DISCARD Rules (any triggers -> DISCARD): A) Not Python DS / not about performance optimization B) Requires network, special OS features, or non-standard deps C) Code performs file I/O or modifies environment variables ## Scoring Dimensions (score 1-3) 1) question_clarity: Does the task faithfully reflect the original SO post's intent and ,→ DS/performance optimization nature? 2) i_o_clarity: Are input/output specs explicit about types, shapes, constraints? Are examples ,→ valid and consistent? 3) input_generator_quality: Does the generator produce valid, diverse inputs with edge cases and ,→ sufficient scale? 4) output_validator_quality: Does the validator compare outputs correctly with appropriate ,→ numerical tolerances? Score scale: 3=High quality, 2=Adequate, 1=Low quality (DISCARD) ## Output (JSON only, no markdown) { "analysis": { "question_clarity": "...", "i_o_clarity": "...", "input_generator_quality": "...", "output_validator_quality": "..." }, "scores": { "question_clarity": N, "i_o_clarity": N, "input_generator_quality": N, "output_validator_quality": N }, "overall_score": N, "overall_decision": "KEEP|FIX|DISCARD" }

Figure 14: Prompt for multi-dimensional quality scoring.

Solution Generation (System) You are an expert Data Science (DS) programmer proficient in Python and its ecosystem of DS ,→ libraries (NumPy, Pandas, SciPy, PyTorch, etc.). Given a DS coding problem, produce a correct ,→ Python solution. Solution Generation (User) Solve the following DS coding problem. ## Task **Task Description**: {task_description} **Input Specification**: {input_specification} **Output Specification**: {output_specification} **Input Example**: {input_example} **Output Example**: {output_example} ## Constraints 1. Define the entry function strictly as: `def solution(input_data=None):` 2. The input is a dictionary; access fields via input_data['key']. 3. Return the result directly (do not print). 4. You may import any standard DS library. ## Output Format Enclose your solution in a single Python code block: ```python import ... def solution(input_data=None): # Your implementation return result ```

Figure 15: Prompt for DS solution generation.

B

Additional Experiment Results

B.1

Library Breakdown Analysis

Table 10 consolidates Pass and B|P for all 16 models across libraries (T=0.0). NumPy (N =388) most closely tracks the aggregate results: GPT-5.4 leads in Pass (76.8%), while Kimi-K2.5 leads B|P (79.6%) despite solving fewer than half the tasks. Pandas (N =317) follows a similar pattern, with Kimi-K2.5 leading B|P (74.0%) and Qwen3.5-397B second (72.2%). PyTorch (N =68) stands out: GPT-5.4-mini leads all models in Pass (73.5%). Polars (N =62) presents the sharpest efficiency divergence: DeepSeek-V4Flash leads B|P at 72.1%, while DeepSeek-V3.2 scores only 41.1%. SciPy (N =48) exhibits particularly strong efficiency, with closed-frontier models performing well (GPT-5.4 82.3%, Claude-Opus4.6 80.3%). For less-frequent libraries, Numba (N =25) tasks explicitly require JIT compilation; Claude-Opus-4.6 and DeepSeek-V3.2 achieve the highest Pass (68.0%), whereas the highest B|P comes from models that solve fewer tasks, making efficiency comparisons on this small subset more sensitive to the composition of the solved tasks. B.2

Pairwise Efficiency Comparison

To isolate efficiency from correctness, we compute pairwise win rates on common-correct subsets. For each ordered pair (A, B), we identify tasks both solve and count the fraction where A is ≥10% faster. This threshold follows the OPT% metric from PIE (Shypula et al., 2023). Figure 16 visualizes the full 16×16 matrix. GPT5.4 achieves the highest Avg Win (45.9%) with the second-lowest Avg Lose (23.7%), confirming broad efficiency advantage. Kimi-K2.5 follows closely (43.6% Win, 22.2% Lose), matching its strong B|P (73.6% in Table 2). At the other extreme, DeepSeek-Coder-V2-Lite loses 54.6% of matchups, and DeepSeek-V3.2 loses 41.9% despite Pass 56.1%, reinforcing the correctness–efficiency decoupling in §5.1. Across tiers, all four closedfrontier models achieve Avg Win >36% and Avg Lose <31%; all seven open-coder models show the opposite pattern. The tier-level efficiency gap thus holds consistently at the task level. B.3

Efficiency Gap Taxonomy

This section supplements §5.2 with the complete two-level taxonomy, category definitions, per-library distribution, and representative exam-

ples. We derive five Level-1 categories and 19 Level-2 sub-patterns from 868 slow–fast code pairs, each naming the optimization strategy the fast solution applies. Among the five categories, CRC (Complexity-Reducing Computation) captures algorithmic optimizations (e.g., early exit, closed-form substitution, index-based lookup) that parallel the strategies targeted by competitiveprogramming efficiency benchmarks; the remaining four categories (OPU, L2V, DLO, OSM) require DS library-specific knowledge that those benchmarks do not assess. B.3.1 Category Definitions Table 11 reports the full two-level distribution with median and 75th-percentile slowdowns. Optimized Primitive Utilization (OPU, 202 pairs). The fast solution calls a library’s existing optimized primitive instead of reimplementing equivalent logic: (1) Direct Primitive Call, where an optimized function already exists (e.g., np.clip, pd.factorize); (2) Lightweight API Path, using a faster option within the same function family (e.g., transform("min") vs. transform(lambda. . . )); (3) Thin Dispatch Selection, choosing a lighter dispatch path (e.g., scipy.spatial.cKDTree vs. KDTree). Complexity-Reducing Computation (CRC, 181 pairs). The fast solution reduces computational complexity through algorithmic redesign or redundant-work elimination: (1) Pass Fusion / Early Exit, fusing multiple passes or terminating early; (2) Closed-Form Substitution, replacing a generic numerical routine with an algebraic identity; (3) Index-Based Lookup, replacing bruteforce scans with binary search or prefix sums; (4) Purpose-Built Index, introducing a KD-tree, heap, or hash map; (5) Scatter / Segment Reduction, replacing per-group filtering with bincount or reduceat. Loop-to-Vectorization (L2V, 167 pairs). The fast solution replaces Python-level iteration (for/while loops, iterrows, apply) with bulk array or table operations in compiled backends: (1) Bulk Array Operation, element-wise NumPy/PyTorch vectorization; (2) Table-Native Method, replacing row-wise Pandas/Polars loops with built-in vectorized methods; (3) Shape Broadcasting, leveraging broadcasting or einsum instead of per-element scalar math. Data Layout Optimization (DLO, 175 pairs). The fast solution selects data representations that

Table 10: Per-library results for all 16 models (T=0.0). NumPy (388) Model

P B|P

Pandas (317) P B|P

PyTorch (68) P B|P

SciPy (48)

Polars (62)

P B|P

P B|P

Numba (25)

TF (22)

P B|P

P B|P

Sklearn (8)

Dask (10)

Others (52)

P B|P

P B|P

P B|P

Closed Frontier GPT-5.4 GPT-5.4-mini Claude-Opus-4.6 Claude-Sonnet-4.6

76.8 71.4 64.9 67.3

78.9 77.2 73.4 68.3

65.3 58.0 59.6 57.7

71.2 71.9 69.6 69.6

64.7 73.5 67.6 55.9

53.6 60.9 49.4 48.2

64.6 70.8 75.0 62.5

82.3 67.3 80.3 74.6

72.6 62.9 72.6 67.7

59.7 57.4 66.9 57.4

48.0 40.0 68.0 60.0

31.6 43.3 44.4 50.2

9.1 13.6 9.1 9.1

43.8 55.6 43.7 92.5

25.0 44.2 0.0 – 12.5 39.6 0.0 –

20.0 0.0 0.0 – 10.0 30.0 20.0 6.1

50.0 48.1 46.2 46.2

60.9 70.3 69.0 60.4

65.5 66.0 44.6 47.9 67.0

70.7 59.9 79.6 64.4 70.2

54.9 52.7 41.0 43.5 57.1

63.2 52.3 74.0 60.3 72.2

58.8 57.4 41.2 30.9 52.9

40.4 46.2 59.1 35.5 51.0

68.8 58.3 27.1 47.9 43.8

67.6 64.2 87.6 64.9 69.8

46.8 51.6 53.2 40.3 58.1

72.1 41.1 58.1 56.0 62.0

60.0 68.0 24.0 32.0 56.0

24.8 16.9 45.8 34.9 34.7

13.6 9.1 9.1 13.6 9.1

87.8 18.8 93.3 96.1 94.6

0.0 – 0.0 – 0.0 – 0.0 – 12.5 40.0

10.0 30.7 0.0 – 0.0 – 0.0 – 10.0 68.5

53.8 38.5 32.7 46.2 46.2

69.3 55.5 61.1 60.0 60.6

Qwen3-Coder-Next Devstral-2-123B Qwen2.5-Coder-32B Qwen2.5-Coder-14B Codestral-22B DS-Coder-33B DS-Coder-V2-Lite

59.5 52.3 40.7 39.9 34.5 19.3 20.1

58.9 57.8 56.5 44.0 49.5 47.2 32.5

46.7 39.7 31.5 29.7 23.0 16.1 12.3

57.1 55.8 52.2 50.6 53.5 45.2 36.3

55.9 35.3 35.3 33.8 39.7 26.5 27.9

53.2 40.2 73.3 53.2 44.0 42.4 84.2

52.1 37.5 41.7 43.8 29.2 22.9 20.8

54.5 64.1 47.4 39.9 61.1 61.7 31.1

38.7 35.5 14.5 11.3 12.9 11.3 12.9

56.8 51.0 69.9 61.0 55.2 39.5 50.5

48.0 56.0 52.0 48.0 40.0 24.0 28.0

22.5 5.6 23.6 29.5 28.1 1.8 56.1

9.1 13.6 9.1 0.0 0.0 4.5 4.5

93.8 89.7 90.1 – – 73.9 83.6

0.0 0.0 0.0 0.0 0.0 0.0 0.0

10.0 0.0 0.0 0.0 0.0 0.0 0.0

46.2 38.5 28.8 30.8 15.4 11.5 17.3

59.5 57.2 46.3 51.2 64.8 52.8 61.0

Human Ref.

100.0 88.5 100.0 85.8 100.0 78.7 100.0 94.2 100.0 78.1 100.0 83.8 100.0 94.4 100.0 78.0 100.0 83.5 100.0 90.4

Open Frontier DS-V4-Flash DS-V3.2 Kimi-K2.5 MiniMax-M2.5 Qwen3.5-397B Open Coder – – – – – – –

4.0 – – – – – –

match the computation, avoiding unnecessary copies or intermediates: (1) In-Place / View Reuse, reusing buffers, views, or in-place operations; (2) Fit-for-Purpose Container, matching the container to the task (e.g., NumPy array instead of DataFrame for pure array work); (3) Precise Dtype Selection, reducing dtype overhead; (4) Lazy / Chunked Expansion, deferring materialization; (5) Contiguous Layout Alignment, ensuring memorycontiguous layout for cache-friendly access. Overhead-Aware Stack Matching (OSM, 143 pairs). The fast solution avoids unnecessary setup, dispatch, or transfer costs by choosing a lighter computation layer: (1) Minimal-Path Execution, removing unnecessary validation, copying, or wrapper logic; (2) Native Python for Small Scale, using Python builtins instead of heavy frameworks for small data; (3) Interpreted Sufficiency, avoiding Numba/JIT overhead when NumPy already handles the operation efficiently.

Table 11: Two-level taxonomy (868 pairs). Slowdown = slow / fast execution time.

B.3.2 Per-Library Distribution Table 12 extends the main-paper top-5 cross-tab to all library categories.

solution, revealing where it most consistently underperforms. The most striking pattern is an optimizationweakness migration across capability levels: the dominant weakness shifts from Loop-toVectorization (48.3% for DS-Coder-V2-Lite) toward Overhead-Aware Stack Matching (45.5% for GPT-5.4). Frontier models have largely mastered basic vectorization but develop a new blind spot: over-engineering. GPT-5.4, the best model by Pass rate, has the lowest median slowdown (3.0×), yet

B.3.3 Model-Level Optimization Profiles Individual models exhibit distinct optimization biases. Table 13 profiles per-model weaknesses (categories where the model most often appears as the slow solution) and Table 14 profiles strengths. Weakness profiles (Table 13). Each row shows the Level-1 distribution when the model is the slow

Level-1 / Level-2

Count

%

Median

p75

OPU Direct Primitive Call Lightweight API Path Thin Dispatch Selection

202 127 54 21

23.3 14.6 6.2 2.4

5.1× 5.5× 4.6× 5.5×

18.9× 21.0× 11.4× 12.0×

CRC Pass Fusion / Early Exit Closed-Form Substitution Index-Based Lookup Purpose-Built Index Scatter / Segment Reduction

181 79 33 33 24 12

20.9 9.1 3.8 3.8 2.8 1.4

11.6× 5.1× 46.7× 33.5× 23.8× 23.2×

64.6× 16.8× 94.2× 144.7× 43.9× 90.1×

L2V Bulk Array Operation Table-Native Method Shape Broadcasting

167 109 42 16

19.2 12.6 4.8 1.8

10.6× 12.7× 7.0× 23.0×

36.6× 45.2× 13.3× 67.9×

DLO In-Place / View Reuse Fit-for-Purpose Container Precise Dtype Selection Lazy / Chunked Expansion Contiguous Layout Alignment

175 83 62 17 7 6

20.2 9.6 7.1 2.0 0.8 0.7

2.9× 2.4× 4.7× 2.4× 1.8× 10.4×

7.6× 6.6× 15.1× 6.0× 4.6× 27.6×

OSM Minimal-Path Execution Native Python for Small Scale Interpreted Sufficiency

143 77 50 16

16.5 8.9 5.8 1.8

14.4× 14.1× 8.3× 3,924.4×

98.4× 52.4× 20.1× 14,371.9×

28.3

GPT-5.4

34.1

42.5

43.7

55.5

27.7

47.6

38.2

54.0

56.2

55.1

67.3

58.4

62.4

64.4

45.9 23.7

35.5

39.2

39.2

52.4

24.5

46.6

38.2

52.0

51.2

50.6

63.8

58.2

57.0

63.6

43.5 24.5

33.7

38.9

53.3

29.6

46.6

33.1

50.5

53.8

53.7

62.6

51.4

60.7

63.5

43.1 26.5

31.0

44.7

23.6

40.1

29.6

46.6

45.9

46.0

56.3

48.4

56.1

60.9

36.9 30.9

38.9

17.4

35.1

24.4

41.9

42.9

42.7

49.8

44.8

51.4

55.2

33.9 30.7

17.4

32.8

22.7

34.3

39.2

38.3

46.7

44.3

45.5

52.8

29.2 41.9

46.8

38.4

52.7

53.9

53.2

61.0

48.7

54.7

66.0

43.6 22.2

19.7

38.8

42.1

37.4

51.7

43.8

46.2

57.4

31.5 37.1

44.0

46.8

46.4

57.1

46.5

62.2

62.3

37.0 27.6

36.4

38.7

46.7

38.1

50.0

49.6

29.2 41.6

34.7

41.2

35.5

39.8

47.8

26.8 42.6

34.7

32.9

47.7

43.6

30.4 41.5

25.1

36.3

36.7

23.7 51.0

24.5

34.8

25.1 44.1

44.3

22.8 49.7

GPT-5.4-mini

26.0

Claude-Opus-4.6

31.6

31.6

Claude-Sonnet-4.6

26.6

27.8

23.4

DeepSeek-V4-Flash

23.0

26.0

26.9

30.6

DeepSeek-V3.2

19.2

17.5

22.2

25.1

26.4

Kimi-K2.5

30.1

34.4

33.0

39.5

34.8

53.2

MiniMax-M2.5

20.7

24.4

23.8

28.9

25.6

38.6

16.7

Qwen3.5-397B

25.5

24.4

28.0

29.6

31.4

45.3

23.6

36.4

Qwen3-Coder-Next

21.7

19.4

23.8

25.6

24.2

34.0

20.1

29.9

21.5

Devstral-2-123B

18.5

20.3

20.1

26.1

21.0

29.4

18.7

28.4

18.9

33.3

Qwen2.5-Coder-32B

25.0

25.9

28.5

26.8

28.1

37.5

22.0

33.5

23.8

32.8

31.4

Qwen2.5-Coder-14B

16.0

16.9

21.0

25.8

23.4

26.8

22.5

26.3

21.1

26.7

23.2

23.3

Codestral-22B

17.3

15.5

23.9

23.0

28.5

26.9

21.1

30.8

22.8

27.8

28.0

24.4

34.9

DS-Coder-33B

17.4

20.1

15.6

22.7

18.1

30.6

18.9

26.4

20.5

21.0

26.8

18.0

29.2

26.6

DS-Coder-V2-Lite

26.7

22.1

22.6

21.1

21.0

24.0

22.3

26.6

21.7

29.6

28.7

31.7

29.4

35.9

Model (row)

Avg Lose

60

50

Win Rate (%)

Avg Win

20

25.6 54.6

4-m

ini e-O pu s -4. ud 6 e-S on De ne t-4 ep Se .6 ek -V4 -Fl a De sh ep Se ek -V3 .2 Kim i-K 2 Mi niM .5 ax -M 2.5 Qw en 3 . 5-3 Qw 97 en B 3-C od erNe De xt vst ral Qw -212 en 3 2.5 -Co B Qw de r en -32 2.5 -Co B de r-1 4B Co de str al22 DS B -Co de r-3 DS 3B -Co de r-V 2-L ite

27.9

30

T-5 .

Cla

Cla

ud

GP

GP

T-5 .

4

40

Opponent (column)

Figure 16: Pairwise efficiency win rate (%) among all 16 models (T=0.0). Cell (A, B): fraction of common-correct tasks where A is ≥10% faster. Right columns: task-level average win and lose rates across 15 opponents. Table 12: Category distribution (%) by library. Library

N

OPU

CRC

L2V

DLO

OSM

NumPy Pandas PyTorch Polars SciPy Numba Others TF Matplotlib Sklearn Dask

360 285 58 55 43 22 34 3 4 2 2

20.0 27.4 17.2 29.1 39.5 22.7 8.8 0.0 25.0 0.0 0.0

26.4 14.7 10.3 16.4 30.2 13.6 32.4 33.3 0.0 50.0 0.0

25.3 21.4 6.9 10.9 11.6 0.0 0.0 0.0 0.0 0.0 0.0

17.5 25.3 32.8 14.5 4.7 9.1 17.6 66.7 25.0 0.0 0.0

10.8 11.2 32.8 29.1 14.0 54.5 41.2 0.0 50.0 50.0 100.0

45.5% of its slow pairs are OSM—importing heavy frameworks, adding unnecessary validation, or wrapping already-vectorized code in defensive scaffolding. DLO emerges as the dominant weakness for several otherwise strong models: DeepSeek-V4Flash (47.3%), Qwen3-Coder-Next (31.8%), and Qwen3.5-397B (31.4%). This suggests that data layout awareness is a distinct skill from algorithmic or API knowledge. Strength profiles (Table 14). When a model appears as the fast solution, the category distribution reveals which optimization strategies it success-

Table 13: Level-1 category distribution (%) when the model is the slow solution. Model

N

Med.

OPU

CRC

L2V

DLO

OSM

55 33 47 56

3.0× 5.0× 3.4× 5.2×

21.8 42.4 23.4 26.8

5.5 27.3 12.8 14.3

7.3 3.0 8.5 8.9

20.0 18.2 29.8 21.4

45.5 9.1 25.5 28.6

55 110 15 64 35

4.3× 6.6× 6.8× 11.8× 4.4×

14.5 32.7 6.7 21.9 25.7

14.5 19.1 33.3 17.2 22.9

5.5 20.9 0.0 9.4 8.6

47.3 14.5 26.7 25.0 31.4

18.2 12.7 33.3 26.6 11.4

88 55 48 76 39 34 58

6.6× 11.9× 7.7× 18.9× 7.9× 13.9× 25.6×

21.6 16.4 20.8 22.4 25.6 14.7 20.7

22.7 34.5 31.2 23.7 25.6 32.4 15.5

18.2 18.2 31.2 36.8 23.1 35.3 48.3

31.8 16.4 6.2 6.6 15.4 11.8 6.9

5.7 14.5 10.4 10.5 10.3 5.9 8.6

Closed Frontier GPT-5.4 GPT-5.4-mini Claude-Opus-4.6 Claude-Sonnet-4.6 Open Frontier DS-V4-Flash DS-V3.2 Kimi-K2.5 MiniMax-M2.5 Qwen3.5-397B Open Coder Qwen3-Coder-Next Devstral-2-123B Qwen2.5-Coder-32B Qwen2.5-Coder-14B Codestral-22B DS-Coder-33B DS-Coder-V2-Lite

fully applies. The fast-side profiles complement the weakness view. Closed Frontier models spread their optimization strength broadly, with GPT-5.4 strongest in CRC (30.8%) and Claude-Opus-4.6 similarly CRC-leaning (24.8%), reflecting proficiency in al-

Table 14: Level-1 category distribution (%) when the model is the fast solution. Models with N <20 are omitted. Model

N

Med.

OPU

CRC

L2V

DLO

OSM

130 8.7× 105 6.2× 105 10.8× 64 5.6×

21.5 19.0 21.9 25.0

30.8 21.9 24.8 21.9

23.1 20.0 18.1 18.8

13.8 22.9 20.0 25.0

10.8 16.2 15.2 9.4

78 71 50 45 42

6.8× 5.9× 5.8× 8.4× 6.5×

29.5 23.9 14.0 33.3 31.0

25.6 15.5 14.0 17.8 14.3

21.8 28.2 24.0 17.8 19.0

17.9 19.7 26.0 11.1 16.7

5.1 12.7 22.0 20.0 19.0

43 24 42 26

5.7× 4.7× 4.4× 9.0×

18.6 37.5 19.0 23.1

23.3 8.3 14.3 11.5

11.6 16.7 7.1 11.5

18.6 20.8 21.4 26.9

27.9 16.7 38.1 26.9

Closed Frontier GPT-5.4 GPT-5.4-mini Claude-Opus-4.6 Claude-Sonnet-4.6 Open Frontier Kimi-K2.5 Qwen3.5-397B DS-V3.2 DS-V4-Flash MiniMax-M2.5 Open Coder Qwen3-Coder-Next Devstral-2-123B Qwen2.5-Coder-32B Qwen2.5-Coder-14B

gorithmic redesign. Kimi-K2.5’s fast-side profile is OPU-heavy (29.5%) and has the lowest OSM share (5.1%), consistent with its preference for direct, lightweight code paths. Open Coder models, when they do appear on the fast side, concentrate in OSM (Qwen2.5-Coder-32B at 38.1%, Qwen3Coder-Next at 27.9%): their “wins” often come from writing simpler code that avoids overhead, rather than applying sophisticated optimizations. Orthogonal to the per-category profiles, JIT compilation overhead is a recurring cross-model pattern. Of the 868 pairs, 49 (5.6%) contain Numba or JIT decorators in the slow code; their median slowdown is 1,131.0× versus 6.5× for non-JIT pairs. DeepSeek-V3.2 uses Numba in 14.4% of its slow pairs, DeepSeek-V4-Flash in 12.7%. On the small-to-medium data sizes typical of DS tasks, JIT overhead can exceed the computation it was meant to accelerate. B.3.4 Case Study Below we present one slow–fast pair per Level-1 category. Task descriptions and I/O specifications are condensed for brevity. Optimized Primitive: triple nested loop vs. scipy.spatial.distance.cdist (389.4×). DeepSeek-V3.2 writes a triple-nested Python loop over points and dimensions, producing O(N M D) interpreter iterations. Claude-Sonnet-4.6 recognizes that scipy.spatial.distance.cdist performs the identical computation in compiled C, eliminating all Python-level iteration (Figure 17). Complexity-Reducing: per-matrix np.linalg.det loop vs. closed-form ad−bc (183.8×). Devstral-2-123B calls np.linalg.det

in a Python list comprehension over N matrices, invoking LAPACK LU decomposition per element. Claude-Opus-4.6 exploits the 2×2 structure: det = ad − bc is four element-wise array operations, fully vectorized across the batch (Figure 18). Loop-to-Vec: nested loop with per-cell np.sum vs. nine-slice vectorization (167.4×). The nested Python loop calls np.sum on a 3×3 slice per pixel, incurring function-call overhead proportional to the image area. The vectorized version sums nine shifted array views in a single expression, eliminating Python iteration entirely in favor of compiled NumPy (Figure 19). Data Layout: DataFrame-based row replication vs. pre-allocated NumPy arrays (95.5×). GPT-5.4-mini stays in the DataFrame layer throughout: index.repeat expands the index, then pd.concat materializes n·R copies of the row, and a boolean mask overwrites positions, triggering multiple internal copies. DeepSeek-V4-Flash drops to NumPy arrays: pre-allocated buffers filled with the repeated value, then original rows scattered into place via computed indices, constructing the DataFrame only once at the end (Figure 20). Stack Matching: function-level import and dead-code branches (218.4×). Both models write the same core computation: (A * B).sum(dim=1). The 218.4× gap comes entirely from GPT-5.4’s defensive scaffolding: import torch inside the function body (re-executing module lookup on every call) and a dead if input_data is None branch that constructs dummy tensors. The fast code hoists the import to module scope and removes the dead branch. This is a pure overhead example: the algorithm is identical, but the “production-ready” wrapper dominates execution time (Figure 21). B.4

Memorization Analysis

To assess potential memorization, we split the 1,000 tasks by whether the source Stack Overflow question was created before or after June 2024, following the temporal split methodology of LiveCodeBench (Jain et al., 2025). This yields 904 pre-cutoff and 96 post-cutoff tasks (Table 15). Because the post-cutoff subset is limited in size, we interpret this analysis as a sensitivity check rather than conclusive evidence against memorization. Pass rates drop slightly on post-cutoff tasks (mean −3.5%). B|P does not decline systematically: 8 of 16 models score higher on post-cutoff

Table 15: Pre- vs. post-cutoff results. Cutoff: June 2024. Pre Model

Post

P

B

B|P

P

B

B|P

67.6 62.1 61.4 59.3

48.2 44.7 42.2 39.2

71.4 72.0 68.7 66.1

60.4 63.5 60.4 63.5

45.7 43.1 44.4 42.0

75.7 67.9 73.6 66.1

57.9 56.1 40.5 42.5 58.0

37.4 30.7 30.2 25.7 39.4

64.7 54.7 74.5 60.5 68.0

56.2 56.2 37.5 45.8 54.2

38.0 27.7 24.2 28.3 36.2

67.5 49.3 64.6 61.7 66.8

Qwen3-Coder-Next Devstral-2-123B Qwen2.5-Coder-32B Qwen2.5-Coder-14B Codestral-22B DS-Coder-33B DS-Coder-V2-Lite

51.8 43.6 35.5 33.6 27.5 17.5 17.5

29.4 23.9 19.4 15.5 13.8 7.9 7.7

56.8 54.8 54.6 46.1 50.2 45.0 43.8

38.5 37.5 20.8 25.0 26.0 17.7 13.5

21.9 19.7 11.8 12.9 13.9 8.9 3.9

56.7 52.6 56.7 51.4 53.2 50.0 28.7

Human Reference

100.0 86.8 86.8 100.0 84.1 84.1

Closed Frontier GPT-5.4 GPT-5.4-mini Claude-Opus-4.6 Claude-Sonnet-4.6 Open Frontier DS-V4-Flash DS-V3.2 Kimi-K2.5 MiniMax-M2.5 Qwen3.5-397B Open Coder

Table 16: Per-library B|P (%) on Basic-correct tasks. N : tasks correct under Basic in each library. DeepSeek-V4-Flash N

N

EL

∆

Rx

∆

NumPy 254 77.8 +7.1 82.6 +12.0 Pandas 174 74.2 +11.0 76.9 +13.7 PyTorch 40 47.3 +6.9 51.8 +11.4 SciPy 33 76.8 +9.2 85.6 +18.0 Polars 29 75.6 +3.5 78.1 +6.0 Others 47 62.6 +7.1 78.4 +23.0

173 130 28 13 33 25

88.9 90.6 73.0 93.5 78.1 82.9

+9.3 +16.6 +13.9 +5.9 +20.0 +22.9

90.4 92.8 71.3 97.2 83.2 88.6

+10.8 +18.8 +12.2 +9.6 +25.1 +28.6

All

402 87.4 +13.8 89.5 +15.9

EL

577 73.2

∆

Kimi-K2.5 ∆

Library

Rx

+8.3 78.5 +13.5

tasks, and rankings are well-preserved (Spearman ρ = 0.87 for B|P, 0.95 for Pass). These results are consistent with the main correctness–efficiency decoupling, but do not eliminate contamination concerns because all source posts remain public and the post-cutoff subset is relatively small. B.5

From Diagnosis to Remedy: Experimental Details

This section supplements §5.3 with method details, experimental protocols, and breakdowns for both experiments. B.5.1 Taxonomy-Guided Optimization Method. Effi-Learner (EL) operates in an iterative feedback loop. In each round, the system (1) executes the current best code on the benchmark’s correctness-test inputs and collects a line-level execution time profile, (2) constructs a prompt containing the original task description, the current code, and the profile highlighting hotspot lines, and (3) sends the prompt to the same base model requesting a faster implementation. If the returned

code passes all correctness tests and runs faster than the current best on those same tests, it replaces it; otherwise the previous version is carried forward. Each successive round receives the output of the previous round (not the original Basic code) along with an updated profile, enabling incremental optimization. EL w/ Rx augments the EL prompt with a taxonomy-derived optimization prescription listing the five L1 categories, each with its L2 subpatterns and one-line code examples. This prescription is prepended so the model can match profiled hotspots to known optimization strategies before generating a replacement; the iteration mechanism is otherwise identical to vanilla EL (Figure 22). Experimental protocol. We apply both EL and EL w/ Rx to DeepSeek-V4-Flash (Ncorrect =577) and Kimi-K2.5 (Ncorrect =402), prompting at T=0.0. Only Basic-correct tasks are submitted for optimization; Basic-failed tasks retain their original submission and are never reprompted. We set the iteration budget to 3 rounds per task; the per-round breakdown (Table 17) confirms this is sufficient, with accepted improvements concentrated in the first round and diminishing returns after. The final submission is evaluated on the full test suite; if it fails correctness or degrades execution time by ≥10%, it counts as a regression. Per-library results. Table 16 reports B|P on the Basic-correct subset per library for both models. Unlike Table 5, which reports full-benchmark outcomes after optimization, this breakdown fixes the Basic-correct subset to isolate efficiency changes within each library. EL w/ Rx delivers uniformly larger gains than vanilla EL across nearly all library groups and both models. The variation across libraries is substantial and model-dependent. For DeepSeek-V4-Flash, Others sees the largest Rx gain (+23.0%) and SciPy follows (+18.0%), consistent with their OSMand OPU-concentrated taxonomy profiles where the prescription directly guides the model toward lighter execution stacks or specialized primitives. For Kimi-K2.5, Others (+28.6%) and Polars (+25.1%) benefit most, reflecting Kimi’s stronger baseline algorithmic choices that leave frameworklevel overhead as the primary remaining gap. PyTorch is the only library where Rx occasionally underperforms EL (Kimi: +12.2 vs. +13.9), suggesting that knowledge-augmented reprompting can destabilize solutions in frameworks with complex dispatch semantics. Across both models, EL retains ≥98% of Basic-

Table 17: New optimizations accepted per round. Model

Method

R-1

R-2

R-3

DS-V4-Flash

EL EL w/ Rx

155 201

46 60

25 32

Kimi-K2.5

EL EL w/ Rx

159 198

42 51

10 17

correct tasks (DeepSeek: 573/577; Kimi: 394/402), and EL w/ Rx retains ≥97% (DeepSeek: 566/577; Kimi: 393/402), confirming that efficiency-focused reprompting rarely breaks correctness. Per-round results. Table 17 reports how many tasks are newly optimized in each round. R1 captures 69–75% of all accepted improvements; by R3, at most 32 tasks are newly accepted in any setting, justifying the three-round budget. B.5.2 Library-Conditioned Routing Method. We split the 1,000 benchmark instances into training (60%), validation (10%), and test (30%) sets via a per-library temporal split. Within each library, instances are sorted by SO creation date. The earliest 60% form the training set, the next 10% form the validation set, and the latest 30% form the test set. This yields 597 training, 99 validation, and 304 test instances, while preserving the original temporal test set. Expert and hyperparameter selection. On the training set, we compute per-library Pass and B|P (T=0.0) for all 12 open-weight models (5 Open Frontier + 7 Open Coder). Models are ranked independently by Pass and B|P within each library. We combine the rankings using α · rankPass + (1−α) · rankB|P and designate the top K=3 models as experts. We then sweep α ∈ [0, 1] at 0.05 increments on the validation set. The validation set selects α=0.35, which achieves the highest B|P (77.4), with Beyond used to break ties. After fixing α and the corresponding experts, we evaluate the routing configuration on the test set. Per-library expert assignment. Table 18 shows the experts selected at α=0.35. NumPy, Pandas, Polars, and Others share the same three experts, although their ranking differs. PyTorch and SciPy select different experts, reflecting variation in model strengths across libraries.

Table 18: Per-library experts selected from the training set using the validation-selected α=0.35 (K=3). Library Expert 1

Expert 2

Expert 3

NumPy Qwen3.5-397B DS-V4-Flash Kimi-K2.5 Pandas Qwen3.5-397B Kimi-K2.5 DS-V4-Flash PyTorch Qwen3.5-397B Qwen3-C-Next Qwen2.5-C-32B SciPy DS-V4-Flash DS-V3.2 MiniMax-M2.5 Polars Kimi-K2.5 DS-V4-Flash Qwen3.5-397B Others Qwen3.5-397B DS-V4-Flash Kimi-K2.5

Q#69531729 · NumPy · 389.4× slowdown Task. Compute pairwise Euclidean distances between two point sets. Input. ’XA’: float64 array (N, D); ’XB’: float64 array (M, D). Output. Float64 array (N, M ) where element (i, j) is the Euclidean distance between XA[i] and XB[j]. Slow (DeepSeek-V3.2) import numpy as np def solution(input_data=None): XA = input_data['XA'] XB = input_data['XB'] N, D = XA.shape M, _ = XB.shape distances = np.zeros((N, M), dtype=np.float64) for i in range(N): for j in range(M): sq_dist = 0.0 for k in range(D): diff = XA[i, k] - XB[j, k] sq_dist += diff * diff distances[i, j] = np.sqrt(sq_dist) return distances Fast (Claude-Sonnet-4.6) import numpy as np from scipy.spatial.distance import cdist def solution(input_data=None): XA = np.array(input_data['XA'], dtype=np.float64) XB = np.array(input_data['XB'], dtype=np.float64) return cdist(XA, XB, metric='euclidean')

Figure 17: Optimized Primitive Utilization: triple nested loop vs. cdist (389.4×).

Q#78368046 · NumPy · 183.8× slowdown Task. Compute determinants of a batch of 2×2 matrices. Input. ’matrix_batch’: float64 array (N, 2, 2). Output. Float64 array (N, ) where element i is the determinant of matrix_batch[i]. Slow (Devstral-2-123B) import numpy as np def solution(input_data=None): matrix_batch = input_data['matrix_batch'] determinants = np.array( [np.linalg.det(matrix) for matrix in matrix_batch]) return determinants Fast (Claude-Opus-4.6) import numpy as np def solution(input_data=None): matrix_batch = input_data['matrix_batch'] return (matrix_batch[:, 0, 0] * matrix_batch[:, 1, 1] - matrix_batch[:, 0, 1] * matrix_batch[:, 1, 0])

Figure 18: Complexity-Reducing Computation: np.linalg.det loop vs. closed-form (183.8×).

Q#72153725 · NumPy · 167.4× slowdown Task. Sum all 3×3 neighborhoods in a 2D array. Input. ’array’: 2D int array (rows, cols) with rows, cols ≥ 3. Output. 2D float array (rows−2, cols−2) of neighborhood sums. Slow (Qwen2.5-Coder-14B) import numpy as np def solution(input_data=None): array = input_data['array'] rows, cols = array.shape result = np.zeros((rows-2, cols-2), dtype=array.dtype) for i in range(rows-2): for j in range(cols-2): result[i, j] = np.sum(array[i:i+3, j:j+3]) return result Fast (Qwen3.5-397B) import numpy as np def solution(input_data=None): array = input_data['array'] result = (array[:-2, :-2] + array[:-2, 1:-1] + array[:-2, 2:] + array[1:-1, :-2] + array[1:-1, 1:-1] + array[1:-1, 2:] + array[2:, :-2] + array[2:, 1:-1] + array[2:, 2:]) return result.astype(float)

Figure 19: Loop-to-Vectorization: per-cell np.sum loop vs. nine-slice vectorization (167.4×).

Q#68592930 · Pandas · 95.5× slowdown Task. Insert n copies of a fixed row after every row in a DataFrame. Input. ’df’: DataFrame (R, 3); ’myrow’: single-row DataFrame; ’n_repeats’: integer. Output. DataFrame (R · (1+n), 3) interleaving originals and copies. Slow (GPT-5.4-mini) import pandas as pd import numpy as np def solution(input_data=None): df = input_data["df"] myrow = input_data["myrow"] n_repeats = int(input_data["n_repeats"]) n = len(df) labels = np.tile(np.arange(n_repeats + 1), n) df_repeated = df.loc[ df.index.repeat(n_repeats + 1) ].reset_index(drop=True) myrow_block = pd.concat( [myrow] * (n * n_repeats), ignore_index=True) repeated_positions = labels != 0 df_repeated.loc[ repeated_positions, df.columns ] = myrow_block.to_numpy() return df_repeated Fast (DeepSeek-V4-Flash) import numpy as np import pandas as pd def solution(input_data=None): df = input_data['df'] myrow = input_data['myrow'] n = input_data['n_repeats'] N = len(df) total = N * (1 + n) cols = df.columns.tolist() data = {} for col in cols: data[col] = np.full( total, myrow[col].iloc[0], dtype=df[col].dtype) orig_indices = np.arange(N) * (1 + n) for col in cols: data[col][orig_indices] = df[col].values return pd.DataFrame(data)

Figure 20: Data Layout Optimization: DataFrame replication vs. pre-allocated NumPy arrays (95.5×).

Q#69702937 · PyTorch · 218.4× slowdown Task. Compute the row-wise dot product of two square matrices. Input. ’A’: torch.Tensor (N, N ); ’B’: torch.Tensor P (N, N ). Output. torch.Tensor (N, ) where element i = j A[i, j] · B[i, j]. Slow (GPT-5.4) def solution(input_data=None): import torch if input_data is None: A = torch.tensor([[1, 2], [3, 4]]) B = torch.tensor([[1, 1], [1, 1]]) else: A = input_data["A"] B = input_data["B"] if not isinstance(A, torch.Tensor): A = torch.tensor(A) if not isinstance(B, torch.Tensor): B = torch.tensor(B) result = (A * B).sum(dim=1) return result Fast (Qwen3.5-397B) import torch def solution(input_data=None): A = input_data['A'] B = input_data['B'] if not isinstance(A, torch.Tensor): A = torch.tensor(A, dtype=torch.float32) if not isinstance(B, torch.Tensor): B = torch.tensor(B, dtype=torch.float32) result = (A * B).sum(dim=1) return result

Figure 21: Overhead-Aware Stack Matching: function-level import and dead-code branches (218.4×).

EL w/ Rx System Prompt ## Role You are an expert in data-science (DS) code performance optimization. You will receive a correct but potentially slow Python solution along with its execution time profile. Your task is to produce an optimized version that is functionally equivalent but faster. --## Optimization Guide: Below is a reference optimization guide. Use it as a checklist when analyzing the profile, but always verify that any change preserves correctness. ### 1. Loop-to-Vectorization (L2V) When to apply: The profile shows hot Python for/while loops, iterrows(), apply(axis=1), or per-element indexing. Key sub-patterns: - Bulk Array Operation: Replace element-wise Python loops with NumPy/PyTorch vectorized ops. - Table-Native Method: Replace iterrows/apply row loops with pandas/polars native methods. - Shape Broadcasting: Replace nested scalar math with broadcasting or einsum. ### 2. Optimized Primitive Utilization (OPU) When to apply: The code manually reimplements logic that a library already provides, or uses a slow API path. Key sub-patterns: - Direct Primitive Call: Replace hand-written logic with a single optimized library function. - Lightweight API Path: Switch to a faster option within the same API family. - Thin Dispatch Selection: Use the lighter backend variant (e.g., cKDTree over KDTree). ### 3. Data Layout Optimization (DLO) When to apply: The code creates large unnecessary intermediate copies or uses a heavier container than needed. Key sub-patterns: - In-Place / View Reuse: Avoid allocating large temporaries; use views, inplace=True, or buffer reuse. - Fit-for-Purpose Container: Match the container to the task (ndarray for numeric, dict for small lookups). - Precise Dtype Selection: If suitable, use float32 over float64, Categorical for repeated strings. ### 4. Complexity-Reducing Computation (CRC) When to apply: The code uses brute-force scans or generic solvers when a specialized algorithm exists. Key sub-patterns: - Pass Fusion / Early Exit: Merge multiple data passes into one, or terminate early. - Index-Based Lookup: Replace linear scans with np.searchsorted, prefix sums, or binary search. - Closed-Form Substitution: Replace iterative routines with algebraic identities. ### 5. Overhead-Aware Stack Matching (OSM) When to apply: The code invokes a heavy framework for a task too small or simple to benefit from the overhead. Key sub-patterns: - Minimal-Path Execution: Remove unnecessary validation, defensive wrappers, or fallback logic. - Native Python for Small Scale: Use dict/list/set instead of DataFrame/groupby when data is small. - Interpreted Sufficiency: Skip Numba/JIT when NumPy vectorized ops already suffice. --Important: Always prioritize correctness over efficiency.

EL w/ Rx User Prompt Here is a correct Python solution and its line-level execution time profile. Your goal is to produce an optimized version that is functionally equivalent and significantly faster. ## Current Solution ```python {code} ``` ## Execution Time Profile ``` {profile} ``` ## Your Task Step 1 — Understand: Read the solution and its profile. Identify what the code does. Step 2 — Diagnose: Locate the performance bottleneck(s) from the profile. Step 3 — Plan: Based on your diagnosis, decide on an optimization strategy using the patterns above. Step 4 — Optimize: Rewrite the solution. Ensure the same output, same signature: def solution(input_data=None). Output your optimized Python code inside a single ```python``` code block.

Figure 22: Full prompt for EL w/ Rx.

Coverage (%)

100 98 96 94 92 90 88 86

Table 19: Mean CV (%) of reported mean execution time across (W, M ) configurations.

N=5 95%

W =1 W =3 W =5

Line Coverage Branch Coverage 1

3

5

7

10

Num of Sampled Test Cases per Domain (N) Figure 23: Line and branch coverage of the reference solution. Both metrics exceed 95% at N =5.

C

Evaluation Protocol Details

C.1

Evaluation Protocol

Execution Environment. All correctness and efficiency measurements are conducted on a dedicated server with the following specifications: dual Intel Xeon Platinum 8468V processors (96 physical cores, 192 threads, base 2.0 GHz / turbo 3.8 GHz), 512 GB DDR5 RAM, and 4 NVIDIA H200 NVL GPUs (140 GB each), running Ubuntu 22.04 LTS. The evaluation environment uses Python 3.12 with the following library versions: NumPy 2.2, Pandas 2.3, PyTorch 2.10, SciPy 1.17, Polars 1.37, Numba 0.61, TensorFlow 2.20, and Scikit-learn 1.8. The server is used exclusively for evaluation with no concurrent workloads, ensuring stable execution time measurements. Test Case Generation. Each task contains on average 5.4 correctness sub-domains and 2.4 stress sub-domains. We set N =5 unique test cases per domain, yielding approximately 26.8 correctness cases and 12.0 stress cases per task (38.8 total). Concretely, the input_generator(seed, input_domain) takes a random seed and a domain identifier and returns a single test input. We iterate over all domains, starting from seed 42 and incrementing, collecting up to N =5 unique inputs per domain (duplicates are discarded). To validate this choice, we measure line and branch coverage of the reference solution as a function of N on 100 randomly sampled tasks (Figure 23). Coverage improves sharply from N =1 to N =5 (+7.3% line, +8.9% branch), then plateaus: N =5 to N =7 gains only +0.9%/+1.2%. At N =5, both metrics exceed 95%, balancing coverage with evaluation cost across 16 models and 1,000 tasks. Execution Time Measurement Configuration. Each test case is subject to a 5-second timeout; exceeding it is treated as a correctness failure. We

M =1

M =2

M =3

M =4

M =5

7.16% 6.73% 5.82%

4.92% 4.62% 3.98%

3.92% 3.67% 3.15%

3.34% 3.12% 2.67%

2.93% 2.74% 2.33%

adopt a repeated-execution scheme parameterized by W (untimed warm-up runs) and M (timed measurement runs, whose mean is reported). To select W and M , we conduct a calibration experiment on 100 randomly sampled tasks: for each, we execute the reference solution under every (W, M ) combination with W ∈ {1, 3, 5} and M ∈ {1, 2, 3, 4, 5}, and compute the coefficient of variation (CV) of the reported mean across 10 independent repetitions. Table 19 yields two observations that guide our choice. First, increasing M yields the dominant variance reduction: from M =1 to M =3, CV drops by 3.24% at W =1, whereas each subsequent increment gains less than 0.6%. Second, increasing W provides only marginal benefit: at fixed M =3, moving from W =1 to W =5 reduces CV by 0.77% at the cost of four additional executions per test. We select W =1, M =3 to balance measurement stability with cost. At our scale (16 models × 2 settings × 1,000 tasks × 38.8 cases), each additional execution per case adds ∼1.2M runs. A CV of 3.92% is well within the discriminative range. Domain-Level Execution Time Statistics. We summarize the execution time distribution of the reference solution across the two domain groups. On correctness domains, the reference solution runs in 1.80 ms at the median, 20.84 ms on average, and 40.24 ms at the 90th percentile (P90). On stress domains it runs substantially longer, at 25.12 ms median, 132.56 ms mean, and 199.44 ms P90. The median stress-domain execution time of 25.12 ms provides sufficient headroom for discriminating efficient from inefficient candidate solutions. C.2

Evaluated Models

We evaluate 16 models grouped into three accessibility tiers. Closed Frontier models are accessed via proprietary APIs; Open Frontier and Open Coder models are open-weight and served via vLLM. Below we list each model by its display name followed by the official model identifier used in our experiments. The Closed Frontier tier includes two models from OpenAI (OpenAI, 2026), GPT-5.4 (gpt-5.4) and GPT-5.4-mini (gpt-5.4-mini), and two from Anthropic (Anthropic, 2026), Claude-Opus-4.6

(claude-opus-4-6) and Claude-Sonnet-4.6 (claudesonnet-4-6). The Open Frontier tier spans five generalpurpose models: DeepSeek-V4-Flash (deepseekai/DeepSeek-V4-Flash) (DeepSeek AI, 2026) and DeepSeek-V3.2 (deepseek-ai/DeepSeekV3.2) (Liu et al., 2024a) from DeepSeek, Kimi-K2.5 (moonshotai/Kimi-K2.5) (Team et al., 2026) from Moonshot AI, MiniMax-M2.5 (MiniMaxAI/MiniMax-M2.5) (MiniMax, 2026) from MiniMax, and Qwen3.5-397B/A17B (Qwen/Qwen3.5-397B-A17B) (Yang et al., 2025) from Alibaba Qwen. The Open Coder tier covers seven codespecialized models: Qwen3-Coder-Next (Qwen/Qwen3-Coder-Next) (Cao et al., 2026), Qwen2.5-Coder-32B (Qwen/Qwen2.5Coder-32B-Instruct), and Qwen2.5-Coder-14B (Qwen/Qwen2.5-Coder-14B-Instruct) (Hui et al., 2024) from Alibaba Qwen; Devstral-2-123B (mistralai/Devstral-2-123B-Instruct-2512) (Mistral AI, 2025) and Codestral-22B (mistralai/Codestral22B-v0.1) (Mistral AI, 2024) from Mistral AI; and DeepSeek-Coder-33B (deepseek-ai/deepseekcoder-33b-instruct) (Guo et al., 2024) and DeepSeek-Coder-V2-Lite (deepseek-ai/DeepSeekCoder-V2-Lite-Instruct) (Zhu et al., 2024) from DeepSeek.

Related documents

• DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation#799464
• DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation#798041
• From If-Statements to ML Pipelines: Revisiting Bias in Code-Generation#126582
• CodeT5+: Open Code Large Language Models for Code Understanding and Generation#16601
• PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization#394496