arXiv CSopen access

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

Published as a conference paper at ICLR 2026

M ULTI -LCB: E XTENDING L IVE C ODE B ENCH TO M ULTIPLE P ROGRAMMING L ANGUAGES Maria Ivanova∗ 2

Pavel Zadorozhny∗ 1

Pavel Adamenko 1

Ivan Lopatin 1

1

GigaCode

2

Rodion Levichev∗ 1 Alexey Kutalev 1

Ivan Petrov 1 Dmitrii Babaev 1

Yandex School of Data Analysis, Applied AI Institute

arXiv:2606.20517v1 [cs.AI] 18 Jun 2026

https://github.com/Multi-LCB/Multi-LCB

A BSTRACT LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB’s contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB’s primary limitation and exposing critical gaps in current LLM capabilities.

1

I NTRODUCTION

Large language models (LLMs) have recently demonstrated impressive capabilities in code-related tasks (Ridnik et al., 2024; Lozhkov et al., 2024; Roziere et al., 2023; Li et al., 2022; Nijkamp et al., 2022), powering applications such as AI-assisted programming, automated debugging, and code translation. To measure these abilities, benchmarks such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and APPS (Hendrycks et al., 2021) have been widely adopted. However, these datasets suffer from well-documented limitations, including contamination from training corpora, narrow task scope, and weak correlation with human judgment. LiveCodeBench (LCB) (Jain et al., 2024) addresses these shortcomings by continuously curating competitive-programming problems, filtering them by release date, and enabling contamination-aware, continuously updatable evaluation. As a result, LCB has quickly become a standard benchmark for evaluating LLMs on codegeneration tasks (Google DeepMind, 2025; DeepSeek, 2025). Despite these strengths, LCB (Jain et al., 2024) evaluates only Python. While convenient, this limitation overlooks a central reality of software engineering: developers routinely work across diverse programming languages, each with its own syntax, semantics, and idiomatic practices. An LLM capable of solving problems exclusively in Python may perform poorly when C++ is required ∗

Equal contribution. Correspondence to Dmitrii Babaev: [email protected]

1

Published as a conference paper at ICLR 2026

for systems programming, Java for enterprise software, or JavaScript for web development. Current evaluations therefore leave open a critical question: can LLMs generalize coding competence across multiple programming languages, or are they overfitted to Python? In this work, we introduce Multi-LCB, an extension of LCB (Jain et al., 2024) to twelve programming languages while preserving its contamination controls and evaluation protocol. Multi-LCB replicates every LCB task across all supported languages, enabling direct comparison of model performance on identical problems in different programming languages and updating automatically as LCB evolves. We evaluate 24 reasoning- and instruction-oriented LLMs on Multi-LCB and uncover key findings: 1. Python is not always a reliable proxy for individual non-Python languages. Our results reveal substantial and practically meaningful performance gaps across languages. In several cases, models that are stronger on Python do not retain their advantage in other languages. 2. Python overfitting. Models that perform strongly in Python often degrade sharply in other languages. 3. Language-specific contamination. Evidence of data leakage varies by programming language, reflecting uneven distribution in pretraining corpora. 4. Substantial multi-programming-language disparities. Models show large performance gaps across languages, with weaker results in statically typed or less prevalent languages. Our main contributions are: 1. We extend LCB (Jain et al., 2024) to 12 programming languages without task loss, enabling direct comparison of LLM abilities to solve identical problems across different languages. 2. We provide a comprehensive evaluation of 24 instruction- and reasoning-oriented LLMs across these languages, revealing systematic multi programming languages performance gaps and evidence of language-specific contamination. 3. We publicly release all prompts, source code and experimental configurations to facilitate reproducibility and future research. These results establish Multi-LCB as a rigorous benchmark for multi-programming-language code evaluation, directly addressing LCB’s Python-only limitation and providing a foundation for developing more robust, programming language agnostic coding models.

2

R ELATED W ORK

Single-language code benchmarks. Early code-generation benchmarks evaluate functional correctness almost exclusively in Python. HumanEval (Chen et al., 2021) contains 164 hand-written problems, each defined by a natural language prompt, a fixed function signature, and hidden unit tests; tasks are short, single-function programs created specifically for evaluation rather than drawn from programming contests. MBPP (Austin et al., 2021) likewise offers small Python exercises aimed at introductory programming and interview practice. Subsequent datasets expanded scale and difficulty: APPS (Hendrycks et al., 2021) aggregates competition and interview style problems with hidden test suites, CodeContests (Li et al., 2022) compiles algorithmic contest tasks with official judge input/output data, and CodeXGLUE (Lu et al., 2021) provides a broad suite of generation, translation, and retrieval tasks. Despite their influence, these resources are static snapshots, lack release date filtering to prevent training set contamination and are therefore largely saturated, remain heavily Python centric, and do not enforce a unified STDIN/STDOUT protocol. Multi-programming-language benchmarks. Several datasets extend code generation evaluation beyond Python. MBXP (Athiwaratkun et al., 2022) translates functional-format Python problems (e.g., HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021)) by rewriting function signatures and regenerating unit tests for each language. Even a simple Python assertion like: assert binomial coeff(5, 2) == 10 must be expanded into multi-line Java test code. This translation must be repeated separately for every language and is sensitive to syntax and runtime differences. Concurrent work MultiPL-E (Cassano et al., 2023) similarly performs translation of HumanEval and MBPP (including their unit tests) 2

Published as a conference paper at ICLR 2026

LLM Codegeneration: generation: Code Code generation: STDIN/STDOUT STDIN/STDOUT STDIN/STDOUT format format format

AtCoder, Codeforces Problems Tests

Execution

> g++ main.cpp > rustc main.rs

Tests

LCB / LCB PRO LiveCodeBench LiveCodeBench LiveCodeBench Problemsample: sample: Problem Problem sample: Question Question Question - Natural Natural language language descripdescrip-- Natural language description tion tion -Test Testcases casesexamples examples --Test cases examples Tests Tests Tests

Question

Prompts Naturallanguage languagedescription description Natural Natural language description """ C++ C++ """ """ C++ CODE HERE # YOUR YOUR # CODE HERE # """ YOUR CODE HERE """ """ Testcases casesexamples examples Test Test cases examples

> python main.p

LeetCode Problems Tests Converter

[[1,2,3], [4,5,6]]

=⇒

2 1 2 3 4 5 6

Pass@1

Figure 1: Multi-LCB overview. Top: LCB natural-language problem descriptions are wrapped into prompts specifying the target programming language and passed to the LLM for STDIN/STDOUT code generation. AtCoder and Codeforces problem tests are passed directly to the execution stage. Bottom: LeetCode problem tests are transformed through a dedicated test converter to produce equivalent STDIN/STDOUT inputs. The generated code is compiled or executed in the target programming language and evaluated using Pass@1.

into 19 programming languages. HumanEval-XL (Peng et al., 2024) similarly expands HumanEval to additional languages and provides a standardized execution harness while preserving the functional, unit-test format. Multi-LCB avoids this by keeping only the natural-language description and converting hidden tests into a language-agnostic STDIN/STDOUT format, for example: Input: 52 Output: 10 Other projects broaden language coverage in different ways. Ag-LiveCodeBench-X (BoruchGruszecki et al., 2025) reuses a subset of LiveCodeBench tasks already in STDIN/STDOUT format and adds rarer targets such as Lua, R, Julia, OCaml, and Fortran. xCodeEval (Khan et al., 2023) likewise provides a unified multilingual execution framework and resembles our approach, but it draws exclusively from Codeforces problems and is not continuously updated. McEval (Chai et al., 2024) and BigCodeBench (Zhuo et al., 2024) once offered broad language coverage, but both are static and evaluate different task sets per language, hindering direct cross language comparison. Contamination-aware evaluation. LiveCodeBench (LCB) (Jain et al., 2024) introduced release date filtering and continuous collection of Python problems from three major competitive programming platforms: LeetCode, AtCoder, and Codeforces (see Appendix D.1 for task statistics). By harvesting new tasks and filtering them by post-training release dates, LCB enables live, contamination aware evaluation of LLMs and has become a de-facto standard for robust single language (Python) code assessment (Comanici et al., 2025; Yang et al., 2025; Liu et al., 2024). A related effort, EvoCodeBench (Li et al., 2024), followed a similar evolving design but was not actively maintained and remained limited to Python. Multi-LCB builds directly on this foundation. It reuses the entire LCB (Jain et al., 2024) task pool and inherits its contamination controls.

3

B ENCHMARK D ESIGN

This section describes the approach, used to construct the Multi-LCB benchmark. Figure 1 illustrates the full pipeline. Please note, that although Multi-LCB is built on LCB, the same approach can be applied to any dataset with a comparable structure. Data Source. Earlier versions of LCB supported several evaluation scenarios beyond code generation: self-repair, code execution, and test output prediction. But the latest releases (v5-v6) focus exclusively on code generation, the most widely benchmarked capability of modern LLMs. In this setting, a model receives a natural language problem statement with sample input/output pairs and must synthesize a program that passes all hidden test cases. 3

Published as a conference paper at ICLR 2026

To construct Multi-LCB, we load the desired version of the LCB code generation dataset from Hugging Face, retrieving Python problems and their metadata. We convert every release of LCB code generation dataset without modification, preserving all tasks from three competitive-programming platforms: LeetCode, AtCoder, and Codeforces. Each task includes a natural language description, input/output examples, and contest release date for contamination-aware filtering. Test conversion is applied only to LeetCode’s functional format tasks to ensure unified STDIN/STDOUT evaluation. Details about platforms and temporal distribution appear in the Appendix D.1. Conversion of functional format. LCB supports two native task formats: STDIN/STDOUT (as in AtCoder and Codeforces), where a program reads from standard input and writes to standard output, and Functional (as in LeetCode), where a specific function is implemented and invoked by the evaluation system. Directly extending the functional format to a multi-programming language benchmark is challenging. Each LeetCode task provides Python starter code tightly coupled to its own testing harness. Producing equivalent starter code and call signatures for many target languages would require custom templates for every language, leading to an unsustainable and error-prone process. To overcome this limitation, we designed an automatic conversion pipeline that rewrites every Functional task into a unified STDIN/STDOUT format. This pipeline consists of two components: (1) prompt adaptation that reformats problem statements and examples for model input, and (2) test conversion that transforms all test cases for automated evaluation. The pipeline first parses examples from the problem statement and reformats them into STDIN/STDOUT format for inclusion in model prompts. (see Appendix C.1). Separately, it converts all test cases (both public and hidden) from the original format to enable unified automated evaluation. This unification allows a single evaluation harness to handle both the original STDIN/STDOUT problems and the adapted functional tasks across all supported languages. Since the original benchmark is based on Python, tasks involving Python-specific behavior could theoretically appear. However, tasks on LeetCode, AtCoder, and Codeforces are authored by human experts and are intentionally designed to avoid language-specific ambiguities, as these platforms support many programming languages. Consequently, Multi-LCB requires no language-specific rewriting, and the tasks remain inherently language-agnostic. Moreover, in our manual inspection of approximately 500 tasks, we did not find any cases in which language-dependent features introduced inconsistencies. Note that tasks unsuitable for strict input/output grading, such as those admitting multiple valid answers or requiring explicit data structure construction, are already excluded in the official LCB dataset that we load, so Multi-LCB inherits this filtering without any additional intervention. The remaining tasks are grouped by I/O structure: Scalar: inputs and outputs are single, scalar values (e.g. integers, floats, booleans, or simple strings); One-Dimensional: involve one-dimensional arrays (lists) as input or output; Two-Dimensional: include exactly one two-dimensional array (matrix or jagged array) in the I/O. As a result, all functional tasks, including their examples and hidden tests, are consistently converted to STDIN/STDOUT format: lists are space-separated, and for 2D arrays the first line specifies the number of rows, followed by row-wise space-separated values. This conversion applies to both the examples shown to the models and all test cases used for evaluation. Code generation. We adopt a zero-shot prompting strategy that follows the original LiveCodeBench protocol. For each task, the benchmark constructs a prompt with three components: 1. a system message instructs the model to act as an expert programmer in the target language (e.g., ‘‘You are an expert Python programmer...’’); 2. a user message provides the complete natural language problem statement with explicit STDIN/STDOUT specifications and input/output sample cases already provided in the original problem descriptions; 3. a code-block placeholder indicates where the solution must be written: """ python # YOUR CODE HERE """ ### Answer: (use the provided format with backticks) The code-block header is set to the target language (e.g., cpp, java, python) to ensure correct syntax highlighting and parsing. 4

Published as a conference paper at ICLR 2026

Models are required to output only the complete program source that reads from the standard input and writes to the standard output. High-level zero-shot template prompts for both native AtCoder and CodeForces tasks and adapted LeetCode problems are included in the Appendix C for reference. Automatic Testing and Evaluation. Correctness is assessed against a hidden suite of official test cases provided by the original contests. A program is marked correct only if it passes all tests without runtime errors or timeouts. For quantitative comparison we report Pass@1, the fraction of tasks for which the model’s first generated solution passes every public and hidden test. Together, these stages create a fully automated pipeline: a model receives a problem prompt, emits a candidate solution, the code is securely compiled and executed, and the output is graded against hidden tests – all without human intervention. This process preserves LCB’s rigorous contamination controls while enabling direct, language-agnostic evaluation of code generation across the diverse set of languages supported by Multi-LCB. Note, that the same set of tasks is used across evaluations on different programming languages, hence task difference does not hinder the comparison of the multi-language model capabilities. 3.1

L ANGUAGE SET AND MOTIVATION

This study evaluates multilingual code generation across major programming languages: C++, C#, Python, Java, Rust, Go, TypeScript, JavaScript, Ruby, PHP, Kotlin and Scala. The selection balances three criteria: (1) popularity based on Github, StackOverFlow, RedMonk and TIOBE rankings, (2) stable infrastructure support through package managers like Conda for reproducible execution, and (3) paradigmatic diversity across compilation strategies, type systems, and memory management models. For detailed programming language rankings across multiple sources, as well as the runtime characteristics information, please see Appendix E.

4

E XPERIMENT S ETUP

Here we describe the experimental configuration used to evaluate LLMs on the Multi-LCB benchmark. Models We evaluate a diverse set of 24 publicly available large language models (LLMs) spanning from 7B to 685B parameters and covering both general-purpose and code-specialized domains. The pool includes instruction-tuned and reasoning-augmented variants from the Qwen3, DeepSeek, OlympicCoder, OpenReasoning, and OpenCoder families, among others. Representative examples include GPT-OSS-120B∗ (Medium), Qwen3-235B-A22B-Thinking-2507∗ , DeepSeek-R1-0528∗ and OpenReasoning-Nemotron-32B∗ . We intentionally selected models to capture a wide variety of training paradigms (pure code pretraining, mixed-domain training, instruction tuning, reasoning-enhanced fine-tuning). Appendix F.1 lists all checkpoints with their estimated training cut-off dates. Hardware and Environment. All experiments were run on a cluster of 16 NVIDIA H100 80 GB GPUs with CUDA 12.3 and Python 3.11 inside Conda environments. Each programming language is executed inside an isolated sandbox container that bundles its corresponding compiler or interpreter (e.g., GCC 13 for C++, Rust 1.79, OpenJDK 21, .NET 8, CPython 3.11, Node.js 20). The sandbox enforces strict resource limits: 6 s wall-time per test case, 4 GB memory, and no external network access. This ensures deterministic, secure, and language-agnostic execution. Inference Protocol. Following the original LiveCodeBench protocol, we adopt a zero-shot prompting strategy. For each problem, we generate a model-specific number of tokens (set according to its configuration) using nucleus sampling with temperature = 0.2 and top-p = 0.95, applying a triplebacktick stop sequence to capture the complete code block. Models are served with vLLM (Kwon et al., 2023) or SGLang (Zheng et al., 2024) for efficient batched decoding. Evaluation Metric We report Pass@1 (%) averaged on 10 runs as the primary metric, which measures the fraction of problems solved correctly by the first generated solution. A solution is marked correct only if it compiles/interprets successfully and passes all hidden official test cases without runtime errors or timeouts. 5

Published as a conference paper at ICLR 2026

Table 1: Performance results on Multi-LCB for the tasks from February 2025 till May 2025. Scores represent the Pass@1 (%) metric averaged on 10 runs. Higher is better, bold is best, italic is the second best. Temperature t=0.2 (* - reasoning mode) Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

GPT-OSS-120B* (Medium) Qwen3-235B-A22B-Thk-2507* DeepSeek-R1-0528* GPT-OSS-20B* (Medium) Qwen3-30B-A3B-Thk-2507* GPT-OSS-120B* (Low) Qwen3-235B-A22B* Qwen3-32B* Qwen3-30B-A3B* GPT-OSS-20B* (Low) Qwen3-14B* Qwen3-235B-A22B-Instr-2507 Qwen3-8B* Qwen3-Coder-30B-A3B-Instr Qwen3-30B-A3B-Instr-2507 Qwen2.5-Coder-32B-Instr Seed-Coder-8B-Instr OpenRsn-Nmt-32B* DeepSeek-R1-Distill-Qwen-32B* Devstral-Small-2505* Qwen2.5-Coder-14B-Instr OpenCodeRsn-Nmt-1.1-32B* DeepSeek-R1-Distill-Qwen-14B* Deepseek-Coder-33B-Instr

71.1 ± 2.1 74.0 ± 3.7 66.3 ± 2.0 63.6 ± 2.5 64.0 ± 2.6 56.0 ± 3.1 58.9 ± 2.8 57.6 ± 4.0 55.0 ± 3.6 46.2 ± 3.0 53.5 ± 5.3 43.8 ± 2.8 46.3 ± 5.9 36.6 ± 2.5 38.9 ± 2.5 27.5 ± 0.8 22.1 ± 0.8 64.4 ± 3.6 39.4 ± 7.3 23.2 ± 1.0 22.0 ± 0.6 56.0 ± 12.4 41.8 ± 5.5 17.2 ± 0.7

72.3± 1.9 75.8 ± 2.4 68.0 ± 1.6 65.7 ± 4.0 65.7 ± 4.0 55.4 ± 2.8 58.3 ± 2.7 55.3 ± 3.4 51.5 ± 3.2 47.9 ± 2.4 47.2 ± 4.1 42.7 ± 2.4 39.7 ± 5.0 31.1 ± 2.9 35.6 ± 2.2 26.9 ± 0.7 23.4 ± 0.7 44.2 ± 5.2 22.2 ± 4.6 22.6 ± 0.9 21.3 ± 0.3 37.3 ± 8.0 16.3 ± 1.8 16.2 ± 0.5

70.4± 3.0 73.9 ± 2.0 67.8 ± 1.8 62.7 ± 2.7 62.4 ± 3.2 56.8 ± 2.0 55.0 ± 4.2 56.0 ± 4.5 50.6 ± 2.6 46.3 ± 1.8 47.2 ± 2.8 45.5 ± 2.4 36.7 ± 5.5 35.3 ± 2.8 37.2 ± 2.0 30.5 ± 0.9 26.0 ± 1.5 40.8 ± 3.0 33.2 ± 6.5 22.8 ± 0.7 23.9 ± 0.8 33.1 ± 4.2 24.9 ± 2.7 18.5 ± 0.8

69.9 ± 3.0 56.7 ± 2.0 55.0 ± 3.0 59.9 ± 3.4 44.1 ± 1.9 51.8 ± 2.2 48.7 ± 3.5 42.1 ± 2.6 36.9 ± 1.8 42.6 ± 1.4 32.4 ± 3.9 35.0 ± 1.4 25.8 ± 4.4 25.8 ± 2.2 22.4 ± 1.9 23.9 ± 0.7 22.1 ± 1.5 11.5 ± 4.2 11.9 ± 2.8 16.1 ± 2.2 19.2 ± 0.6 9.9 ± 2.6 10.8 ± 2.1 12.4 ± 0.5

70.5 ± 1.9 67.0 ± 3.5 64.6 ± 2.8 61.9 ± 3.4 51.9 ± 4.3 55.9 ± 2.9 50.0 ± 2.8 49.6 ± 2.6 49.9 ± 4.0 45.1 ± 2.0 45.0 ± 3.0 26.4 ± 1.3 36.5 ± 4.9 28.4 ± 1.5 20.8 ± 1.8 6.3 ± 1.2 23.3 ± 2.3 10.8 ± 6.9 16.2 ± 3.9 22.7 ± 1.0 22.6 ± 0.8 8.2 ± 3.7 10.2 ± 3.4 8.5 ± 1.7

70.3 ± 3.8 62.5 ± 2.9 58.9 ± 3.5 61.8 ± 2.3 46.5 ± 2.3 55.6 ± 1.9 48.8 ± 3.1 49.3 ± 3.8 48.2 ± 4.9 42.7 ± 1.9 46.0 ± 5.2 19.5 ± 2.7 38.8 ± 4.8 28.0 ± 1.4 18.2 ± 1.7 28.8 ± 0.6 23.1 ± 1.6 10.5 ± 5.3 11.6 ± 3.4 24.7 ± 1.4 17.5 ± 0.8 4.9 ± 2.0 11.5 ± 3.0 7.4 ± 2.3

57.3± 2.7 66.5 ± 2.2 61.6 ± 2.8 52.4 ± 2.5 56.5 ± 3.8 45.7 ± 2.3 51.0 ± 4.0 49.1 ± 4.1 43.9 ± 2.8 41.2 ± 2.1 43.3 ± 2.8 44.1 ± 1.8 36.3 ± 4.3 34.7 ± 2.3 36.5 ± 1.1 28.5 ± 1.3 27.0 ± 0.8 29.9 ± 3.8 29.3 ± 4.3 24.1 ± 1.4 23.3 ± 0.5 25.5 ± 3.4 29.2 ± 4.0 17.1 ± 0.7

70.5 ± 2.5 47.7 ± 2.8 63.1 ± 2.3 61.9 ± 2.3 51.7 ± 4.0 56.0 ± 1.7 40.7 ± 3.7 40.1 ± 4.1 38.4 ± 2.9 42.0 ± 1.7 31.5 ± 2.7 39.5 ± 1.0 20.5 ± 4.1 34.3 ± 2.3 32.1 ± 2.7 24.7 ± 0.6 21.8 ± 1.4 2.8 ± 1.5 20.2 ± 4.6 19.9 ± 1.2 16.7 ± 0.6 1.1 ± 0.6 3.7 ± 1.4 2.6 ± 0.6

70.2 ± 2.0 49.4 ± 3.2 62.4 ± 1.5 61.7 ± 2.1 42.1 ± 2.6 53.0 ± 2.8 46.6 ± 2.6 52.1 ± 2.7 46.8 ± 3.1 44.7 ± 1.6 45.3 ± 5.0 41.5 ± 1.4 39.5 ± 5.8 34.7 ± 2.2 34.7 ± 2.2 24.6 ± 0.8 21.6 ± 0.9 18.3 ± 3.4 40.1 ± 5.9 19.9 ± 2.0 22.7 ± 0.8 23.4 ± 3.1 34.5 ± 4.2 15.2 ± 0.7

66.1 ± 2.8 69.0 ± 3.7 61.6 ± 2.2 60.5 ± 2.5 58.8 ± 2.9 53.4 ± 2.3 48.4 ± 3.8 50.0 ± 3.1 48.0 ± 3.0 45.8 ± 1.6 45.5 ± 2.9 42.4 ± 2.2 36.0 ± 2.2 31.8 ± 1.3 34.8 ± 1.9 27.3 ± 0.6 20.4 ± 1.3 15.8 ± 3.2 12.9 ± 2.6 20.8 ± 1.3 22.7 ± 0.6 19.3 ± 4.3 3.8 ± 1.8 16.5 ± 0.7

71.0 ± 2.5 67.7 ± 3.0 66.0 ± 2.8 62.4 ± 2.2 50.6 ± 2.7 55.8 ± 2.8 47.5 ± 3.9 46.4 ± 2.7 44.3 ± 3.7 46.3 ± 2.3 39.2 ± 3.1 41.1 ± 2.0 24.0 ± 3.3 35.7 ± 2.3 35.9 ± 1.9 26.6 ± 0.8 23.4 ± 1.4 17.3 ± 3.5 20.5 ± 4.7 21.2 ± 1.0 18.1 ± 0.4 12.3 ± 3.2 11.5 ± 3.6 16.0 ± 0.5

54.1 ± 3.0 57.6 ± 3.0 62.3 ± 2.2 43.1 ± 2.9 43.6 ± 2.8 42.2 ± 4.2 33.6 ± 3.4 35.6 ± 3.4 32.2 ± 2.7 29.2 ± 2.7 32.4 ± 3.0 28.1 ± 1.9 27.0 ± 2.7 20.2 ± 1.8 25.7 ± 1.6 24.5 ± 0.6 21.8 ± 1.0 6.0 ± 1.3 7.0 ± 1.4 17.2 ± 1.4 20.4 ± 0.6 7.0 ± 2.2 3.3 ± 1.3 12.2 ± 1.0

67.8 ± 5.9 64.0 ± 9.4 63.1 ± 3.8 59.8 ± 6.1 53.2 ± 8.3 53.1 ± 4.6 48.9 ± 7.0 48.6 ± 6.7 45.5 ± 6.7 43.3 ± 4.9 42.4 ± 7.0 37.5 ± 8.4 33.9 ± 7.7 31.4 ± 4.9 31.1 ± 7.2 25.0 ± 6.2 23.0 ± 1.9 22.7 ± 18.5 22.0 ± 11.2 21.3 ± 2.7 20.9 ± 2.4 19.8 ± 16.1 16.8 ± 12.8 13.3 ± 4.9

5

E XPERIMENTS AND R ESULTS

We evaluate a suite of frontier large language models on Multi-LCB, spanning 12 programming languages and reporting Pass@1 averaged on 10 runs as the primary metric (Table 1). This section presents a detailed analysis of model performance on latest Dataset v6 (Feb 2025 – May 2025) (Section 5.1), compares findings with single-language LiveCodeBench (LCB) results (Section 5.2), and investigates contamination signals (Section 5.3). For additional performance results at various sampling temperatures, Pass@5 and Pass@10 metrics and other dataset releases (July 2024-May 2025 and the full 1,055-task benchmark) see appendix F. 5.1

E XPERIMENTS R ESULTS ON M ULTI -LCB

We study performance variations in models released more recently. Particularly, we evaluate 24 recent large language models on Multi-LCB, restricting tasks to those released after 2025-02-01 to ensure live, post-cutoff evaluation and minimize any risk of training-data leakage. Model approximate cutoff dates are listed in Appendix F.1 Table 4. Table 1 summarizes Pass@1 averaged on 10 runs with temperature t = 0.2 performance across twelve programming languages on Dataset v6 (Feb 2025 – May 2025), while Figure 2 highlights the results for the 10 bestperforming models. Our results reveal substantial and practically meaningful performance gaps across languages. For example, GPT-OSS-120B∗ (Medium) outperforms Qwen3-235B-A22B-Thk-2507∗ on Go, Javascript, Typescript, Rust, Ruby and Kotlin, and DeepSeek-R1-0528∗ outperforms Qwen3- 235B-A22B-Thk-2507∗ on Rust, Ruby and Scala, despite Qwen3-235B-A22B-Thk-2507∗ being consistently stronger on Python. This is precisely why strong Python ability is not always a reliable proxy for true cross-lingual code generation competence and evaluation must consider performance in the target languages rather than relying on Python alone.

Figure 2: Top-10 models by Pass@1

6

Published as a conference paper at ICLR 2026

Figure 3 plots per-model Pass@1 averaged on 10 runs with sampling temperature t = 0.2 scores on Python against the cross programming language average on Dataset v6 (Feb 2025 – May 2025). Almost every point lies above the x = y diagonal, demonstrating a consistent bias toward Python. Models without explicit multi programming languages training, such as OpenRsn-Nmt-32B∗ and OpenCodeRsn-Nmt-1.1-32B∗ , show the starkest gap, exceeding 60% on Python while remaining below 30% across other languages. Even the largest reasoning-augmented models, including Qwen3-235B-Thk and DeepSeek-R1, retain a measurable positive bias toward Python, though the disparity is less pronounced.

Figure 3: Scatter of Python vs. Average Pass@1

These results confirm that strong Python ability is not necessarily a reliable proxy for true crosslingual code generation competence. The most strongest models, GPT-OSS-120B∗ (Medium), Qwen3-235B-A22B-Thk-2507∗ and DeepSeek-R1-0528∗ establish a strong yet far-from-saturated frontier, while the next tier of high-performing models, such as Qwen3-30B-A3B-Thk-2507∗ , illustrates that only a handful of reasoning-augmented variants can exceed the 50% mark. Most of the evaluated models remain below 40%, underscoring the benchmark’s challenge of achieving robust multi programming language code generation correctness. Figure 4 plots Pass@1 distribution across 12 languages on with sampling temperature t = 0.2 on Dataset v6 (Feb 2025 – May 2025). Boxes show the interquartile range with the horizontal line marking the median and the red diamond indicating the mean. This reveals a clear difficulty gradient. Python achieves the highest mean Pass@1 of 0.482, with Java and C++ close behind at about 0.44. C#, Ruby, PHP, Go, Rust, Kotlin and JavaScript/TypeScript form a middle tier with means near 0.33-0.39, while Scala consistently trail at means below 0.29. These gaps persist across the top-performing models, reflecting structural challenges such as compilation complexity, ownership semantics, and smaller ecosystem resources.

Figure 4: Pass@1 distribution across 12 languages

We observe that Python consistently outperforms other languages on Multi-LCB. This suggests that current LLMs are substantially more trained on Python code, especially for reasoning-mode training, and that cross-language knowledge transfer remains only partial. We suppose that model performance could be improved by increasing training coverage of non-Python programming languages. 5.2

C OMPARISON WITH L IVE C ODE B ENCH

To verify that our multilingual extensions preserve the fidelity of the original LiveCodeBench (LCB), we compare Pass@1 scores on the Python subset of Multi-LCB against the official results reported 7

Published as a conference paper at ICLR 2026

for LCB versions v4-v6. Table 2 reports original leaderboard results (ORIG) and our reproduced scores (OUR), with ∆ representing the absolute difference. Table 2: Comparison of reasoning/code models on Python across benchmark versions (v4–v6). Original leaderboard values (ORIG, %) are contrasted with our reproduced scores (OUR, %). Difference is computed as ∆ = OUR − ORIG. Model

Benchmark version (range)

ORIG (%)

OUR (%)

∆ (%)

Source

Qwen3-235B-A22B-Thinking-2507 DeepSeek R1 0528 Qwen3-30B-A3B-Thinking-2507 OpenReasoning-Nemotron-32B OpenCodeReasoning-Nemotron-1.1-32B Qwen3-30B-A3B* Qwen3-235B-A22B Qwen3-235B-A22B-Instruct-2507

v6 [2502–2505] v6 [2502–2505] v6 [2502–2505] v6 [2502–2505] v6 [2502–2505] v6 [2502–2505] v6 [2502–2505] v6 [2502–2505]

74.1 68.7 66.0 65.6 61.4 57.4 55.7 51.8

74.0 66.3 64.0 64.4 56.0 55.0 58.9 43.8

-0.1 -2.4 -2.0 -1.2 -5.4 -2.4 3.2 -8.0

Hugging Face LCB leaderboard Hugging Face LCB leaderboard LCB leaderboard Hugging Face LCB leaderboard Hugging Face

Qwen3-32B* Qwen3-14B* Qwen3-30B-A3B* Qwen3-8B* Seed-Coder-8B-Instruct

v5 [2410–2502] v5 [2410–2502] v5 [2410–2502] v5 [2410–2502] v5 [2410–2502]

65.7 63.5 62.6 57.5 24.7

64.3 56.7 61.0 49.1 19.8

-1.4 -6.8 -1.6 -8.6 -4.9

Qwen3 Tech report Qwen3 Tech report Qwen3 Tech report Qwen3 Tech report Hugging Face

v4–v5 [2408–2502] v4–v5 [2408–2502] v4–v5 [2408–2502] v4–v5 [2408–2502]

69.9 54.5 40.7 28.3

65.3 52.3 35.6 27.6

-4.6 -2.2 -5.1 -0.7

Hugging Face Hugging Face Hugging Face Hugging Face

OpenCodeReasoning-Nemotron-1.1-32B OlympicCoder-32B OlympicCoder-7B Qwen2.5-Coder-32B-Instruct

Overall, reproduction is strong: differences are typically within a few percentage points, with a mean absolute deviation of only about 3%. For example, Qwen3-235B-A22B-Thinking-2507 achieves 74.0% Pass@1 in our evaluation versus 74.1% on the original v6 leaderboard (∆ = −0.1), while DeepSeek-R1-0528 records 66.3% compared to 68.7% (∆ = −2.4). Even for models with larger gaps, such as Qwen3-235B-A22B-Ins-2507 (∆ = −8.0) or Qwen3-8B* (∆ = −8.6), the rank ordering across models remains consistent. These close alignments confirm that Multi-LCB’s multilingual transformations introduce no artificial difficulty for Python tasks. Performance differences instead reflect natural leaderboard variance and underscore that the multilingual benchmark faithfully reproduces the single-language LCB setting, ensuring that any additional challenges arise from genuine cross-language generalization rather than implementation artifacts. 5.3

C ONTAMINATION ON M ULTI -LCB

A core design goal of Multi-LCB is contamination-aware evaluation via release-date filtering. Nevertheless, time-wise analysis reveals clear evidence of residual contamination on older (precutoff) problems. Figure 5 shows monthly Pass@1 trends for the top-10 models averaged across all programming languages : scores are systematically higher on earlier months and exhibit step-like drops when the evaluation window crosses model cutoffs, followed by sustained lower performance on post–cutoff problems. Our main comparisons in Section 5 restrict evaluation to tasks released on or after 2025-02-01, ensuring live, post-cutoff measurement. Under this setting, perfor- Figure 5: Monthly Pass@1 trends averaged across all mance drops to a level that better reflects programming languages for top-10 models. true generalization, whereas inflated scores on older windows are explained by pretraining exposure rather than genuine zero-contamination generalization. 8

Published as a conference paper at ICLR 2026

6

L IMITATIONS AND T HREATS TO VALIDITY

Language Coverage and Selection. Multi-LCB covers 12 programming languages but does not include some important languages such as Swift, Haskell, R, and others. The language selection is based on popularity rankings in 2025, which may not reflect specialized domains or emerging languages. Additionally, some languages have various dialects and versions that are not accounted for in our evaluation framework. Task Complexity and Domain. While the selected programming languages span different domains (systems programming, web development, data science), the tasks themselves remain rooted in competitive programming. Although algorithmic problem-solving has indirect relevance to industrial coding capabilities, Multi-LCB does not directly assess real-world software engineering scenarios such as API integration, debugging legacy code, or collaborative development workflows. Evaluation Protocol Constraints. The strict STDIN/STDOUT format may introduce performance degradation not only due to algorithmic reasoning limitations but also due to syntax unfamiliarity, difficulty parsing input formats, or failure to follow output specifications. Models may fail tasks due to format compliance issues rather than core problem-solving deficits, potentially confounding our assessment of true multilingual coding competence. Model Selection Bias. Our evaluation focuses exclusively on publicly available models, excluding proprietary systems that may represent the current state-of-the-art. This limitation means our results reflect only a subset of available models and may not accurately represent the real-world leaderboard of multilingual code generation capabilities. Construct Validity. The automatic conversion from functional format to STDIN/STDOUT may alter task complexity differently across programming languages. Some languages may be more naturally suited for certain problem types, potentially creating unequal evaluation conditions that affect cross-language comparisons. Internal Validity. Despite date-based filtering, hidden forms of contamination may persist through similar problem patterns or solution templates present in training data. Additionally, models may exhibit temporal bias based on varying exposure to different programming languages during their training periods.

7

F UTURE W ORK

Multi-LCB’s modular design enables straightforward language expansion. We plan to add Swift, Haskell, R, and Julia by defining their compilation commands and runtime environments. We will evaluate proprietary models (GPT-4, Claude, Gemini) to establish comprehensive multilingual leaderboards reflecting current state-of-the-art performance. The STDIN/STDOUT framework directly supports LCB-Pro (Zheng et al., 2025) and other benchmarks requiring format conversion, enabling broader contamination-aware multilingual evaluation without additional infrastructure changes.

8

C ONCLUSIONS

We introduced Multi-LCB, a contamination aware benchmark for evaluating large language models on multilingual code generation. Multi-LCB provides an extensible framework spanning twelve programming languages and continuously updates with newly released problems. The conversion methodology extends beyond LCB to other Python benchmarks (e. g. LCB Pro (Zheng et al., 2025)), offering a general approach for multilingual code evaluation. By inheriting LiveCodeBench’s live evaluation protocol and unified STDIN/STDOUT execution, it enables rigorous, cross programming language assessment and mitigates data contamination that affects static benchmarks. Our experiments expose programming language specific contamination, evidence of Python overfitting, and significant performance gaps across programming languages. We hope Multi-LCB will serve as a durable resource for advancing the evaluation of code-oriented LLMs and guiding future research in multilingual program synthesis. 9

Published as a conference paper at ICLR 2026

R EFERENCES Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868, 2022. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Aleksander Boruch-Gruszecki, Yangtian Zi, Zixuan Wu, Tejas Oberoi, Carolyn Jane Anderson, Joydeep Biswas, and Arjun Guha. Agnostics: Learning to code in any programming language via reinforcement with a universal learning environment. arXiv preprint arXiv:2508.04865, 2025. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multiple: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 49(7):3675–3691, 2023. Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436, 2024. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. DeepSeek. Deepseek-r1-0528 release, May 2025. URL https://api-docs.deepseek. com/news/news250528. DeepSeek News release. Google DeepMind. Gemini 2.5: Our most intelligent ai model, March 2025. URL https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025. Google DeepMind Blogpost. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv preprint arXiv:2303.03004, 2023. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp. 611–626, 2023. Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. arXiv preprint arXiv:2404.00599, 2024. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022. 10

Published as a conference paper at ICLR 2026

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022. Qiwei Peng, Yekun Chai, and Xuhong Li. Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization. arXiv preprint arXiv:2402.16694, 2024. Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500, 2024. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37: 62557–62583, 2024. Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, et al. Livecodebench pro: How do olympiad medalists judge llms in competitive programming? arXiv preprint arXiv:2506.11928, 2025. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024.

11

Published as a conference paper at ICLR 2026

Appendix C ONTENTS A Legal Compliance and License

13

B UI of Multi-LCB

13

C Prompt Examples

13

C.1 AtCoder/CodeForces Example (native STDIN/STDOUT) . . . . . . . . . . . . . .

14

C.2 LeetCode Example (adapted into STDIN/STDOUT) . . . . . . . . . . . . . . . . .

14

D Tasks Distribution

15

D.1 Task Distribution by Difficulty and Platform . . . . . . . . . . . . . . . . . . . . .

15

D.2 Task Distribution by I/O Data Dimensionality (LeetCode Functional Format) . . .

16

E Programming language rankings and runtime characteristics

17

F Experiments

18

F.1

Models overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

F.2

Performance on the Multi-LCB (Feb-May 2025 Subset) Across Sampling Temperatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

F.3

Performance on the Multi-LCB (Jul 2024-May 2025 Subset) . . . . . . . . . . . .

21

F.4

Performance on the Complete Multi-LCB Benchmark . . . . . . . . . . . . . . . .

21

G Computation time

22

H Languages and Compiler Versions

22

I

Platform Analysis

24

J

Difficulty Analysis

27

K Temporal Analysis

30

L Languages errors type

32

12

Published as a conference paper at ICLR 2026

A

L EGAL C OMPLIANCE AND L ICENSE

The Multi-LCB benchmark contains no personally identifiable information, offensive content, or proprietary code. It is derived entirely from the publicly released LiveCodeBench (LCB) dataset, which itself sources only publicly accessible contest problems, reference solutions, and test cases from LeetCode, AtCoder, and Codeforces. Our redistribution and multi programming language transformation of LCB fall under Fair Use (§107, U.S. Copyright Act): the benchmark is provided solely for non-commercial academic research, reproduces only the material necessary for evaluation, and does not diminish the market value of the original platforms or LCB. Multi-LCB is strictly an evaluation resource, no models are trained on these tasks, and is released under a CC BY-NC 4.0 license to ensure non-commercial use.

B

UI OF M ULTI -LCB

Figure 6 presents the web interface of Multi-LCB, displaying a subset of tasks released between January 2024 and December 2024. A time-range scroller at the top allows users to interactively select different time windows to filter tasks and monitor model performance on newly released problems. This interactive design highlights the live and continuously updated nature of the benchmark, enabling researchers to track progress as fresh contest tasks are incorporated.

Figure 6: Multi-LCB web interface showing tasks released between January 2024 to December 2024, with an interactive time-range scroller for filtering and visualization.

C

P ROMPT E XAMPLES

This appendix shows example prompts from Multi-LCB. We distinguish the original problem text as it appears on the source platform and the additional instructions that we add in order to unify everything into the STDIN/STDOUT format. Original parts are placed in blue boxes, while added parts are placed in red boxes. 13

Published as a conference paper at ICLR 2026

C.1

AT C ODER /C ODE F ORCES E XAMPLE ( NATIVE STDIN/STDOUT) Original ### Question: Find the number of positive integers not greater than N that have exactly 9 positive divisors. Input: N Output: Print the answer. Constraints: 1 ≤ N ≤ 4 × 1012 Sample Input 1: 200 Sample Output 1: 3 Added ### Format: Read the inputs from STDIN solve the problem and write the answer to STDOUT (do not directly test on the sample inputs). Enclose your code within delimiters as follows. Ensure that when the python program runs, it reads the inputs, runs the algorithm and writes output to STDOUT. """ python # YOUR CODE HERE """ ### Answer: (use the provided format with triple quotes)

C.2

L EET C ODE E XAMPLE ( ADAPTED INTO STDIN/STDOUT) Original ### Question: You are given an integer array enemyEnergies and an integer currentEnergy... (original description) Example 1: Input: enemyEnergies = [3, 2, 2], currentEnergy = 2 Output: 3 Explanation: Several operations lead to a maximum of 3 points (see original problem description). Example 2: Input: enemyEnergies = [2], currentEnergy = 10 Output: 5 Explanation: Performing the first operation 5 times on enemy 0 yields the maximum number of points. Constraints: • 1 ≤ enemyEnergies.length ≤ 105 • 1 ≤ enemyEnergies[i] ≤ 109 • 0 ≤ currentEnergy ≤ 109

14

Published as a conference paper at ICLR 2026

Added ### Format: Read the inputs from STDIN solve the problem and write the answer to STDOUT (do not directly test on the sample inputs). Enclose your code within delimiters as follows. Ensure that when the python program runs, it reads the inputs, runs the algorithm and writes output to STDOUT. For 2D arrays, the first line indicates the number of rows, followed by newline-separated rows. Sample Input 1: 3 2 2 2 Sample Output 1: 3 """ python # YOUR CODE HERE """ ### Answer: (use the provided format with triple quotes) For non-Python settings, only the header of the code block is replaced (e.g., """ cpp, """ java). The rest of the prompt structure remains identical.

D

TASKS D ISTRIBUTION

D.1

TASK D ISTRIBUTION BY D IFFICULTY AND P LATFORM

LiveCodeBench (LCB) continuously aggregates competitive programming problems in Python from three major platforms: LeetCode, AtCoder, and Codeforces. Figure 7 shows the monthly distribution of tasks by difficulty, and Figure 8 presents the monthly distribution by source platform. Together, these figures highlight the steady inflow of new problems and the live, contamination-aware nature of LCB, and, by extension Multi-LCB.

Figure 7: Monthly distribution of Tasks by Difficulty.

15

Published as a conference paper at ICLR 2026

Figure 8: Monthly distribution of LCB tasks by platform.

Each platform hosts frequent contests whose tasks provide a natural language description of a problem, example input/output pairs, and hidden tests, ensuring that solutions must be fully correct to receive credit. Because every contest attracts thousands of participants and receives official editorial review, the problems are inherently vetted for clarity and correctness. Across the full lifetime of the dataset, the platform composition is as follows: Codeforces: Competitive-programming problems known for a wide range of difficulty and algorithmic focus, almost exclusively in STDIN/STDOUT format. LeetCode: Interview oriented challenges emphasizing data structures and algorithms, originally in a Functional format. AtCoder: Algorithmically rich STDIN/STDOUT input/output. D.2

competitive

programming

problems,

typically

using

TASK D ISTRIBUTION BY I/O DATA D IMENSIONALITY (L EET C ODE F UNCTIONAL F ORMAT )

Figure 9: Monthly task distribution by I/O data dimensionality (LeetCode Functional format). Figure 9 presents the temporal distribution of LeetCode tasks grouped by the I/O data dimensionality of their Functional format. The plot highlights how problems with different input/output structures, 16

Published as a conference paper at ICLR 2026

such as scalar values, one-dimensional arrays, and two-dimensional arrays, have entered the benchmark over time, illustrating the variety of functional tasks inherited from LeetCode within the LCB dataset.

E

P ROGRAMMING LANGUAGE RANKINGS AND RUNTIME CHARACTERISTICS

This study evaluates multilingual code generation across major programming languages selected for their 2025 popularity and broad industrial relevance. Programming language rankings across multiple sources presented in Table 3. Table 3: Programming language rankings across multiple sources (dates in footnotes) Language

TIOBE1

GitHub2

Stack Overflow3

Python C++ Java C# JavaScript TypeScript Go Rust Ruby PHP Kotlin Scala

1 (26.98%) 2 (9.80%) 4 (8.76%) 5 (4.87%) 6 (3.36%) 37 (0.28%) 7 (2.04%) 18 (1.01%) 23 (0.76%) 14 (1.28%) 20 (0.90%) 34 (0.41%)

1 5 2 10 4 6 3 13 8 7 15 14

4 (57%) 9 (23%) 7 (29%) 8 (27%) 1 (66%) 6 (43%) 13 (∼2%) 14 (∼2%) 18 (∼1.5%) 12 (∼15%) 15 (∼3%) 29 (∼1%)

RedMonk4 2 7 3 5 1 6 12 19 9 4 14 14

These languages span a wide range of paradigms and runtime characteristics, capturing the diversity of real-world software development: • Compilation model: – Compiled/JIT — C++, Rust, Go, Java, C#, Scala, Kotlin – Interpreted — Python, Ruby, PHP – Transpiled — TypeScript → JavaScript • Type system: – Static — C++, Rust, Go, Java, C#, Scala, Kotlin, TypeScript – Dynamic — Python, JavaScript, Ruby, PHP • Memory management: – RAII/manual — C++ – Ownership/borrowing — Rust – Garbage collection — Java, C#, Go, Scala, Kotlin, PHP, Ruby, JavaScript/TypeScript • Runtime platforms: – Native — C++, Rust, Go – JVM — Java, Scala, Kotlin – .NET CLR — C# – Interpreters/VMs — Python, Ruby, PHP – JavaScript engines — JavaScript, TypeScript • Domain ecosystems: – Systems/performance — C++, Rust, Go – Enterprise/JVM and .NET — Java, C#, Scala, Kotlin – Web/backend and scripting — JavaScript, TypeScript, PHP, Ruby – Data/AI glue — Python 17

Published as a conference paper at ICLR 2026

F

E XPERIMENTS

F.1

M ODELS OVERVIEW

We provide details for all models included in our study in Table 4. Table 4: Overview of Large Language Models (* denotes reasoning mode) Model

Short Name

openai/gpt-oss-120b openai/gpt-oss-20b Qwen/Qwen3-235B-A22B-Thinking-2507 deepseek-ai/DeepSeek-R1-0528 Qwen/Qwen3-30B-A3B-Instruct-2507 Qwen/Qwen3-32B Qwen/Qwen3-235B-A22B Qwen/Qwen3-30B-A3B Qwen/Qwen3-14B open-r1/OlympicCoder-32B Qwen/Qwen3-235B-A22B-Instruct-2507 Qwen/Qwen3-8B Qwen/Qwen3-Coder-30B-A3B-Instruct Qwen/Qwen3-30B-A3B-Instruct-2507 open-r1/OlympicCoder-7B Qwen/Qwen2.5-Coder-32B-Instruct nvidia/OpenCodeReasoning-Nemotron-1.1-32B ByteDance-Seed/Seed-Coder-8B-Instruct Qwen/Qwen2.5-Coder-14B-Instruct mistralai/Devstral-Small-2505 nvidia/OpenReasoning-Nemotron-32B deepseek-ai/deepseek-coder-33b-instruct

Gpt-oss-120B* (Medium/Low) Gpt-oss-20B* (Medium/Low) Qwen3-235B-A22B-Thk* DeepSeek-R1-0528* Qwen3-30b-A3b-Thk-2507* Qwen3-32B* Qwen3-235B-A22B* Qwen3-30B-A3B* Qwen3-14B* OlympicCoder-32B* Qwen3-235b-A22b-Instr-2507 Qwen3-8B* Qwen3-Coder-30B-A3B-Instr Qwen3-30B-A3B-Instr-2507 OlympicCoder-7B* Qwen2.5-Coder-32B-Instr OpenRsn-Nmt-32B* Seed-Coder-8B-Instr Qwen2.5-Coder-14B-Instr Devstral-Small-2505 OpenRsn-Nmt-32B DeepSeek-Coder-33B-Instr

F.2 F.2.1

Approximate Cutoff Date

Link

08/05/2025 08/05/2025 10/31/2024 11/29/2024 10/31/2024 10/31/2024 10/31/2024 10/31/2024 10/31/2024 10/31/2024 10/31/2024 10/31/2024 03/23/2024 03/23/2024 11/22/2024 08/30/2023

gpt-oss gpt-oss qwen deepseek-ai qwen qwen qwen qwen qwen open-r1 qwen qwen qwen qwen open-r1 qwen nvidia bytedance-seed qwen mistralai nvidia deepseek-ai

P ERFORMANCE ON THE M ULTI -LCB (F EB -M AY 2025 S UBSET ) ACROSS S AMPLING T EMPERATURES PASS @1 AVERAGED OVER 10 RUNS PERFORMANCE AT VARIOUS SAMPLING TEMPERATURES

Table 5 report Pass@1 scores averaged over 10 runs at sampling temperature t = 0.6. Each score indicates the percentage of problems solved correctly on the first attempt, with higher values reflecting better performance. Table 5: Performance results at temperature t = 0.6 Scores represent the Pass@1 (%) metric averaged on 10 runs. Higher is better, bold is best, italic is the second best. (* - reasoning mode) Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

Gpt-oss-120B* (Medium) Qwen3-235B-A22B-Thk-2507* DeepSeek-R1-0528* Gpt-oss-20B* (Medium) Qwen3-30B-A3B-Thk-2507* Gpt-oss-120B* (Low) Qwen3-235B-A22B* Qwen3-32B* Qwen3-30B-A3B* Qwen3-14B* Gpt-oss-20B* (Low) Qwen3-235B-A22B-Instr-2507 Qwen3-8B* Qwen3-30B-A3B-Instr-2507 Qwen3-Coder-30B-A3B-Instr DeepSeek-R1-Distill-Qwen-32B* Qwen2.5-Coder-32B-Instr OpenRsn-Nmt-32B* Seed-Coder-8B-Instr Qwen2.5-Coder-14B-Instr OpenCodeRsn-Nmt-1.1-32B* Devstral-Small-2505* DeepSeek-R1-Distill-Qwen-14B* DeepSeek-Coder-33B-Instr

69.9 ± 1.8 74.0 ± 2.5 66.6 ± 2.6 62.3 ± 2.7 65.2 ± 3.0 57.6 ± 2.5 58.2 ± 1.7 58.6 ± 2.7 55.3 ± 3.2 55.9 ± 3.5 45.7 ± 2.0 44.9 ± 2.6 50.5 ± 2.5 41.3 ± 2.5 36.0 ± 2.0 45.9 ± 2.8 27.4 ± 2.6 66.0 ± 2.9 22.6 ± 1.2 22.3 ± 1.8 62.8 ± 3.6 22.1 ± 1.6 45.7 ± 3.2 18.6 ± 1.5

72.6 ± 2.1 75.3 ± 2.6 68.4 ± 2.8 65.5 ± 3.3 66.0 ± 3.6 56.6 ± 2.4 58.6 ± 3.1 56.2 ± 2.5 53.5 ± 3.4 49.9 ± 2.2 47.5 ± 2.1 42.8 ± 2.7 43.7 ± 2.8 36.7 ± 1.5 33.1 ± 1.8 25.8 ± 1.6 25.3 ± 2.4 44.8 ± 4.7 22.9 ± 1.7 21.8 ± 1.7 41.2 ± 2.6 22.0 ± 1.1 18.5 ± 2.3 18.2 ± 0.9

70.0 ± 1.9 74.8 ± 2.9 67.2 ± 2.3 62.5 ± 2.0 63.9 ± 2.9 57.2 ± 3.0 56.1 ± 2.6 54.2 ± 2.7 51.0 ± 3.4 50.6 ± 3.2 45.6 ± 2.4 45.5 ± 2.9 42.5 ± 1.8 37.1 ± 2.5 35.5 ± 3.1 38.8 ± 2.9 27.6 ± 2.7 41.3 ± 1.7 24.4 ± 1.7 24.6 ± 1.4 31.8 ± 4.8 22.9 ± 0.9 24.6 ± 4.1 20.8 ± 1.3

69.9 ± 2.4 57.7 ± 2.4 54.1 ± 2.4 59.2 ± 0.4 44.5 ± 2.0 53.6 ± 2.6 48.6 ± 3.6 42.5 ± 4.0 37.1 ± 3.1 34.6 ± 1.8 41.8 ± 1.9 36.1 ± 2.2 29.2 ± 3.7 23.4 ± 3.6 25.2 ± 1.8 12.9 ± 3.3 25.0 ± 2.1 10.8 ± 3.8 19.2 ± 1.9 18.2 ± 1.6 8.6 ± 2.1 16.8 ± 2.4 8.3 ± 1.3 14.0 ± 1.4

70.5 ± 2.7 68.6 ± 2.5 64.9 ± 2.8 63.6 ± 2.9 53.4 ± 1.9 54.8 ± 2.1 49.9 ± 2.9 50.5 ± 3.5 50.1 ± 1.8 48.1 ± 3.1 43.5 ± 1.8 27.9 ± 3.0 41.3 ± 2.5 21.4 ± 1.3 28.6 ± 1.5 20.4 ± 3.4 8.1 ± 1.9 12.2 ± 7.2 23.4 ± 1.6 22.5 ± 2.5 9.9 ± 5.6 22.3 ± 1.3 9.8 ± 1.7 10.8 ± 3.6

71.9 ± 2.0 63.4 ± 2.1 58.6 ± 4.6 63.9 ± 2.0 50.6 ± 3.0 54.6 ± 2.3 46.6 ± 2.5 50.8 ± 3.2 49.9 ± 2.9 47.7 ± 3.8 44.1 ± 2.4 21.8 ± 1.8 41.8 ± 2.6 20.2 ± 3.5 26.3 ± 2.1 15.9 ± 1.9 25.0 ± 1.3 11.0 ± 6.6 22.8 ± 1.8 19.8 ± 1.9 6.8 ± 3.5 24.1 ± 1.5 10.5 ± 2.3 9.0 ± 3.8

59.8 ± 2.9 65.8 ± 2.6 62.1 ± 2.2 50.2 ± 2.3 56.2 ± 3.0 46.4 ± 1.4 51.5 ± 2.8 51.1 ± 2.4 42.9 ± 2.8 44.8 ± 1.5 39.9 ± 2.9 43.7 ± 2.5 39.8 ± 1.7 35.3 ± 2.5 34.5 ± 2.2 34.2 ± 3.2 28.6 ± 2.1 31.5 ± 2.0 22.8 ± 1.8 23.7 ± 2.3 25.0 ± 2.9 22.3 ± 1.6 30.5 ± 3.2 18.6 ± 1.8

70.1 ± 2.4 51.5 ± 3.2 62.5 ± 3.8 61.5 ± 2.1 51.2 ± 3.2 55.8 ± 2.3 43.2 ± 2.9 39.1 ± 2.1 38.4 ± 3.0 32.0 ± 3.3 43.1 ± 2.8 40.2 ± 2.3 22.4 ± 2.0 31.8 ± 2.5 33.7 ± 2.2 21.2 ± 2.2 24.2 ± 2.3 3.1 ± 2.6 21.8 ± 1.3 17.6 ± 1.3 1.2 ± 0.8 19.9 ± 1.2 3.7 ± 1.5 2.9 ± 1.5

69.6 ± 3.1 48.9 ± 2.4 62.4 ± 2.0 61.4 ± 2.8 43.1 ± 3.5 53.8 ± 3.2 48.4 ± 1.8 52.0 ± 2.3 47.3 ± 2.6 46.2 ± 2.9 44.0 ± 3.0 41.5 ± 2.9 42.5 ± 3.2 33.7 ± 1.4 33.8 ± 1.7 43.0 ± 2.3 23.5 ± 2.1 17.6 ± 4.7 21.5 ± 2.6 22.4 ± 1.7 25.7 ± 3.4 20.2 ± 2.5 36.6 ± 3.5 17.0 ± 2.0

67.3 ± 1.8 67.5 ± 2.5 61.3 ± 1.9 60.6 ± 3.1 57.0 ± 3.6 53.4 ± 2.5 48.9 ± 3.0 51.9 ± 1.8 48.2 ± 2.0 44.5 ± 1.3 44.7 ± 2.7 42.6 ± 2.5 38.7 ± 2.6 35.3 ± 1.4 31.5 ± 1.2 13.8 ± 1.5 25.3 ± 2.3 13.9 ± 2.9 19.9 ± 2.0 21.0 ± 2.7 21.1 ± 3.3 21.1 ± 2.1 4.0 ± 1.9 16.2 ± 1.5

70.2 ± 2.8 68.6 ± 2.8 66.4 ± 3.0 62.1 ± 2.6 52.2 ± 3.3 54.9 ± 2.6 47.7 ± 4.0 45.2 ± 2.6 43.9 ± 3.1 39.5 ± 5.3 45.0 ± 1.6 41.5 ± 1.5 25.4 ± 2.8 36.1 ± 1.9 35.0 ± 2.1 21.1 ± 2.4 26.6 ± 2.1 17.2 ± 2.9 23.5 ± 1.9 22.3 ± 0.7 13.6 ± 2.4 21.6 ± 1.8 13.6 ± 3.4 17.0 ± 1.7

54.1 ± 3.6 59.2 ± 2.8 61.2 ± 3.5 42.4 ± 3.3 40.5 ± 2.4 40.8 ± 3.2 34.4 ± 3.5 38.1 ± 2.1 33.7 ± 1.7 31.2 ± 2.7 32.5 ± 4.1 28.8 ± 3.2 29.5 ± 2.5 26.4 ± 2.1 20.9 ± 1.7 8.7 ± 1.3 23.5 ± 1.8 7.4 ± 1.9 21.5 ± 0.7 19.8 ± 2.3 7.1 ± 3.2 16.3 ± 1.3 3.5 ± 1.0 12.1 ± 2.5

68.0 ± 5.5 63.7 ± 8.6 63.0 ± 4.1 59.6 ± 6.6 53.6 ± 8.5 53.3 ± 4.9 49.3 ± 6.7 49.2 ± 6.5 46.0 ± 6.8 43.7 ± 7.8 43.1 ± 3.9 38.1 ± 7.8 37.3 ± 8.5 31.6 ± 6.9 31.2 ± 4.8 25.1 ± 12.5 24.2 ± 5.3 23.1 ± 18.9 22.2 ± 1.5 21.3 ± 2.1 21.2 ± 17.6 20.9 ± 2.4 17.4 ± 14.0 14.6 ± 5.1

Table 6 reports Pass@1 scores averaged over 10 runs at sampling temperature t = 1.0. Each score indicates the percentage of problems solved correctly on the first attempt, with higher values reflecting better performance. 18

Published as a conference paper at ICLR 2026

Table 6: Performance results at temperature t = 1.0. Scores represent the Pass@1 (%) metric averaged on 10 runs. Higher is better, bold is best, italic is the second best. (* - reasoning mode) Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

Gpt-oss-120B* (Medium) Qwen3-235B-A22B-Thk-2507* Gpt-oss-20B* (Medium) DeepSeek-R1-0528* Gpt-oss-120B* (Low) Qwen3-30B-A3B-Thk-2507* Qwen3-32B* Qwen3-235B-A22B* Qwen3-30B-A3B* Qwen3-14B* Gpt-oss-20B* (Low) Qwen3-235B-A22B-Instr-2507 Qwen3-8B* Qwen3-30B-A3B-Instr-2507 Qwen3-Coder-30B-A3B-Instr DeepSeek-R1-Distill-Qwen-32B* Qwen2.5-Coder-32B-Instr OpenRsn-Nmt-32B* Seed-Coder-8B-Instr Qwen2.5-Coder-14B-Instr OpenCodeRsn-Nmt-1.1-32B* Devstral-Small-2505* DeepSeek-R1-Distill-Qwen-14B* Deepseek-Coder-33B-Instr

69.1 ± 2.0 73.7 ± 2.6 64.2 ± 2.9 59.8 ± 2.7 58.2 ± 3.2 63.9 ± 3.9 59.0 ± 3.2 59.2 ± 1.7 57.4 ± 2.7 55.3 ± 2.1 47.3 ± 2.4 44.3 ± 1.6 50.8 ± 3.1 40.5 ± 2.0 36.6 ± 2.1 47.6 ± 2.3 26.9 ± 2.2 66.1 ± 3.0 21.2 ± 2.4 22.1 ± 2.2 63.5 ± 2.8 23.3 ± 2.0 45.0 ± 2.8 16.9 ± 1.6

72.3 ± 1.9 75.0 ± 2.7 65.0 ± 2.1 62.3 ± 2.4 56.6 ± 3.0 65.9 ± 3.2 56.8 ± 2.6 58.5 ± 2.1 52.2 ± 2.3 50.6 ± 2.0 45.2 ± 2.1 42.8 ± 3.6 44.5 ± 2.3 35.6 ± 1.5 32.8 ± 1.9 24.9 ± 2.4 25.2 ± 1.4 44.1 ± 2.1 21.8 ± 2.3 22.5 ± 1.8 41.7 ± 3.6 21.2 ± 2.7 18.1 ± 2.6 17.9 ± 1.7

70.0 ± 3.5 73.7 ± 3.2 63.1 ± 3.2 62.4 ± 1.1 55.3 ± 3.3 63.8 ± 3.7 55.8 ± 2.4 56.6 ± 2.2 52.1 ± 2.5 50.5 ± 2.2 42.9 ± 2.9 45.7 ± 3.2 42.7 ± 3.3 37.4 ± 1.8 35.0 ± 2.8 38.7 ± 2.2 27.5 ± 2.4 38.4 ± 3.1 21.9 ± 2.2 23.0 ± 1.5 30.5 ± 3.7 22.4 ± 1.3 22.7 ± 3.8 18.9 ± 1.4

67.9 ± 2.8 57.0 ± 3.2 58.9 ± 2.5 50.5 ± 4.6 53.0 ± 2.4 45.2 ± 4.1 42.7 ± 2.5 48.0 ± 3.4 36.7 ± 3.2 34.7 ± 4.4 41.4 ± 2.9 36.7 ± 2.0 30.2 ± 4.3 26.0 ± 1.9 25.4 ± 1.9 14.7 ± 4.0 24.2 ± 1.8 11.6 ± 3.0 18.2 ± 1.1 17.5 ± 2.1 8.2 ± 4.6 13.6 ± 2.8 9.2 ± 2.0 12.9 ± 1.7

71.8 ± 3.1 69.0 ± 3.5 62.1 ± 3.1 59.9 ± 1.7 54.4 ± 2.2 49.5 ± 3.2 51.8 ± 2.0 51.5 ± 2.6 50.1 ± 3.2 50.5 ± 1.9 42.6 ± 2.2 27.3 ± 2.5 41.6 ± 3.1 20.8 ± 1.5 26.3 ± 3.2 19.6 ± 2.0 10.0 ± 2.5 11.4 ± 6.2 20.9 ± 2.3 21.4 ± 1.9 8.5 ± 6.1 20.8 ± 1.8 9.5 ± 2.6 10.7 ± 3.1

70.5 ± 2.1 63.5 ± 2.2 61.5 ± 2.5 55.3 ± 2.7 54.7 ± 3.6 45.0 ± 3.9 51.2 ± 3.1 48.6 ± 2.3 50.4 ± 2.5 50.2 ± 1.8 42.1 ± 3.1 23.9 ± 4.5 42.0 ± 3.0 20.6 ± 1.8 27.1 ± 3.2 15.9 ± 3.6 25.3 ± 1.7 8.2 ± 4.7 19.8 ± 2.3 18.5 ± 2.6 5.5 ± 5.0 20.5 ± 2.2 10.0 ± 1.5 8.7 ± 2.5

58.8 ± 4.8 67.4 ± 2.5 52.6 ± 2.8 58.4 ± 2.5 47.3 ± 2.7 57.6 ± 3.5 50.6 ± 3.2 50.0 ± 2.8 44.7 ± 2.0 45.4 ± 1.2 38.5 ± 2.9 43.5 ± 2.7 40.8 ± 2.4 36.6 ± 2.4 34.3 ± 2.1 33.4 ± 4.4 28.1 ± 1.7 29.7 ± 1.9 20.6 ± 2.0 22.9 ± 1.9 22.7 ± 3.5 20.8 ± 1.6 28.2 ± 1.9 16.8 ± 2.4

70.5 ± 3.1 54.2 ± 3.5 59.8 ± 2.1 57.4 ± 2.3 54.6 ± 2.5 51.2 ± 3.3 38.8 ± 2.5 43.4 ± 2.3 39.4 ± 2.7 31.5 ± 2.0 43.0 ± 2.4 38.5 ± 3.0 24.1 ± 1.9 31.3 ± 2.1 34.1 ± 2.5 22.7 ± 3.3 24.7 ± 1.1 2.4 ± 2.2 20.2 ± 3.0 16.8 ± 1.7 1.9 ± 0.7 18.9 ± 1.4 4.5 ± 1.4 2.3 ± 1.3

69.9 ± 1.9 49.2 ± 2.4 61.7 ± 3.0 57.3 ± 2.6 53.7 ± 1.8 42.4 ± 3.4 51.0 ± 2.8 47.3 ± 3.0 47.0 ± 2.3 47.2 ± 2.5 43.4 ± 2.2 42.4 ± 2.1 42.8 ± 1.8 34.5 ± 2.9 31.9 ± 3.3 44.3 ± 3.5 24.6 ± 2.5 17.6 ± 3.8 21.7 ± 1.9 21.7 ± 1.5 23.2 ± 3.2 18.9 ± 1.9 34.8 ± 3.0 14.4 ± 1.4

68.1 ± 2.2 68.7 ± 1.8 61.5 ± 3.3 57.4 ± 1.4 53.1 ± 1.8 57.3 ± 2.9 51.7 ± 4.8 47.9 ± 1.8 48.9 ± 4.6 46.3 ± 3.0 44.3 ± 2.0 43.4 ± 3.1 35.9 ± 2.4 34.7 ± 2.1 29.5 ± 2.2 13.7 ± 2.0 23.7 ± 2.4 14.9 ± 1.7 19.1 ± 2.8 19.3 ± 2.6 17.7 ± 3.4 20.2 ± 1.8 3.8 ± 1.0 12.8 ± 1.3

71.2 ± 1.8 68.6 ± 3.4 62.9 ± 3.4 57.7 ± 2.2 56.0 ± 2.7 51.6 ± 3.6 46.5 ± 3.7 46.6 ± 2.8 44.9 ± 3.2 37.8 ± 1.9 42.9 ± 2.2 42.9 ± 2.1 27.7 ± 2.8 34.8 ± 1.7 34.7 ± 1.8 23.1 ± 2.4 25.4 ± 2.9 16.0 ± 2.3 21.8 ± 1.7 22.6 ± 1.7 12.7 ± 1.9 21.4 ± 1.4 12.8 ± 3.4 16.1 ± 1.5

51.1 ± 3.4 58.5 ± 2.8 40.4 ± 2.3 57.4 ± 2.8 40.1 ± 2.8 40.5 ± 1.5 35.7 ± 3.6 34.2 ± 2.6 33.8 ± 1.3 32.4 ± 2.5 31.4 ± 2.3 31.2 ± 2.7 29.9 ± 2.6 25.9 ± 3.2 19.2 ± 2.0 9.2 ± 1.3 23.1 ± 2.5 6.6 ± 2.4 21.2 ± 2.3 18.8 ± 1.8 5.7 ± 2.6 15.2 ± 1.9 3.8 ± 1.8 7.8 ± 1.7

67.6 ± 6.3 64.9 ± 8.4 59.5 ± 6.8 58.0 ± 3.2 53.1 ± 4.9 52.8 ± 8.8 49.3 ± 7.1 49.3 ± 6.9 46.5 ± 6.9 44.4 ± 8.1 42.1 ± 4.0 38.5 ± 7.3 37.8 ± 8.1 31.6 ± 6.6 30.6 ± 5.1 25.6 ± 12.6 24.1 ± 4.7 22.3 ± 18.8 20.7 ± 1.2 20.6 ± 2.3 20.2 ± 18.0 19.8 ± 2.8 16.9 ± 13.3 13.0 ± 4.9

F.2.2

PASS @5 PERFORMANCE AT DIFFERENT SAMPLING TEMPERATURES

Table 7, Table 8 and Table 9 reports Pass@5 scores at sampling temperatures t = 0.2, t = 0.6 and t = 1.0 respectively. Each score indicates the percentage of problems solved correctly on the 5th attempt, with higher values reflecting better performance. Table 7: Performance results at temperature t = 0.2. Scores represent the Pass@5 (%) metric. Higher is better, bold is best, italic is the second best. (* - reasoning mode) Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

Gpt-oss-120B* (Medium) Qwen3-235B-A22B-Thk-2507* DeepSeek-R1-0528* Gpt-oss-20B* (Medium) Qwen3-30B-A3B-Thk-2507* Gpt-oss-120B* (Low) Qwen3-235B-A22B* Qwen3-32B* Qwen3-30B-A3B* Qwen3-14B* Gpt-oss-20B* (Low) Qwen3-8B* Qwen3-235B-A22B-Instr-2507 OpenRsn-Nmt-32B* OlympicCoder-7B* Qwen3-30B-A3B-Instr-2507 Qwen3-Coder-30B-A3B-Instr DeepSeek-R1-Distill-Qwen-32B* OpenCodeRsn-Nmt-1.1-32B* Qwen2.5-Coder-32B-Instr DeepSeek-R1-Distill-Qwen-14B* Seed-Coder-8B-Instr Devstral-Small-2505* Qwen2.5-Coder-14B-Instr DeepSeek-Coder-33B-Instr

83.7 83.6 78.3 77.3 77.5 69.1 69.2 68.8 66.9 66.4 59.6 56.0 53.0 78.4 49.6 49.0 43.0 51.4 74.5 33.0 51.4 27.8 27.3 22.9 21.5

83.7 86.2 79.7 80.4 79.5 69.4 70.4 68.7 64.5 59.3 61.1 51.8 53.9 69.2 49.1 44.9 39.5 35.5 59.8 31.3 30.3 26.0 26.9 25.9 22.3

85.1 85.8 79.9 78.1 77.4 70.3 70.1 70.5 63.2 61.7 59.3 51.0 58.5 66.2 45.9 46.9 42.1 47.1 58.7 36.1 42.0 31.2 25.3 28.7 24.1

83.8 78.8 72.9 78.9 65.4 65.9 66.4 61.9 56.3 54.4 55.8 44.6 45.4 30.1 38.9 31.0 34.0 27.8 24.5 31.6 21.0 26.6 24.0 25.8 19.0

85.2 80.0 77.3 77.7 73.4 70.3 69.0 67.3 62.8 60.4 56.2 52.0 36.8 33.8 44.0 32.4 36.9 39.3 26.6 13.4 28.0 29.3 27.0 26.5 14.7

82.6 80.4 78.2 78.8 72.4 70.5 69.5 67.3 62.1 62.9 54.2 52.9 31.6 30.1 43.6 27.5 34.3 32.2 18.2 31.2 29.6 27.5 29.2 24.5 13.6

77.6 81.8 77.5 73.1 71.9 62.6 66.9 65.4 59.1 57.7 53.8 49.3 52.7 54.8 43.3 47.0 43.2 43.7 47.7 34.5 44.1 31.7 27.0 28.0 22.9

85.0 75.3 77.9 76.0 72.5 68.2 66.8 61.8 54.1 48.0 54.6 36.3 50.1 10.9 29.1 42.2 39.9 38.7 3.8 31.3 10.9 26.6 24.8 22.4 8.8

85.2 65.0 75.7 75.9 57.3 65.5 58.1 64.7 58.0 59.7 55.4 51.2 48.0 40.2 44.5 43.6 42.1 49.2 44.4 30.6 47.2 26.6 26.3 29.0 23.2

80.4 81.1 77.0 74.9 73.9 65.8 66.5 63.8 61.5 58.9 57.8 48.3 52.1 42.1 39.1 42.9 38.3 33.5 42.5 31.0 11.1 26.5 26.8 26.6 23.5

85.0 84.5 79.1 79.3 67.6 69.1 65.8 67.6 59.8 60.9 59.8 38.7 49.7 39.5 41.9 45.5 43.4 40.8 29.7 33.0 28.8 26.7 25.7 25.4 22.7

78.7 79.1 78.4 66.6 62.0 63.6 55.1 57.7 47.0 47.0 48.5 39.0 39.1 20.2 32.2 34.4 29.0 19.6 21.3 28.9 10.5 26.2 22.1 24.6 18.8

83.0 80.1 77.7 76.4 70.9 67.5 66.2 65.5 59.6 58.1 56.3 47.6 47.6 43.0 41.8 40.6 38.8 38.2 37.7 30.5 29.6 27.7 26.0 25.8 19.6

Table 8: Performance results at temperature t = 0.6. Scores represent the Pass@5 (%) metric. Higher is better, bold is best, italic is the second best. (* - reasoning mode) Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

Gpt-oss-120B* (Medium) Qwen3-235B-A22B-Thk-2507* Gpt-oss-20B* (Medium) DeepSeek-R1-0528* Qwen3-30B-A3B-Thk-2507* Gpt-oss-120B* (Low) Qwen3-235B-A22B* Qwen3-32B* Qwen3-14B* Qwen3-30B-A3B* Gpt-oss-20B* (Low) Qwen3-8B* Qwen3-235B-A22B-Instr-2507 Qwen3-30B-A3B-Instr-2507 OpenRsn-Nmt-32B* DeepSeek-R1-Distill-Qwen-32B* OlympicCoder-7B* OpenCodeRsn-Nmt-1.1-32B* Qwen3-Coder-30B-A3B-Instr Qwen2.5-Coder-32B-Instr DeepSeek-R1-Distill-Qwen-14B* Seed-Coder-8B-Instr Devstral-Small-2505* Qwen2.5-Coder-14B-Instr DeepSeek-Coder-33B-Instr

84.7 85.0 79.2 79.6 78.3 70.5 71.1 70.1 68.4 64.4 59.9 61.0 54.6 50.3 77.8 58.3 51.5 75.6 42.4 34.3 55.5 32.7 30.8 29.0 26.7

84.9 85.7 81.5 79.1 79.3 68.6 69.4 68.7 63.9 65.0 61.1 55.4 54.4 47.1 71.6 42.8 49.5 63.4 40.3 33.0 35.1 29.4 29.6 27.9 25.6

84.8 86.4 80.1 78.8 78.5 70.5 70.7 69.5 63.6 64.8 59.1 57.4 58.9 49.0 67.4 53.4 44.8 60.4 41.3 36.3 39.8 30.3 29.1 29.9 27.0

85.1 78.0 79.5 72.4 67.6 68.5 66.2 65.3 57.0 56.9 57.3 51.0 50.3 39.0 28.3 31.8 39.7 26.1 34.0 33.5 19.4 28.3 28.5 26.9 21.8

84.1 80.8 80.7 77.5 75.2 68.1 70.3 67.4 63.6 63.9 56.6 56.2 44.2 36.8 35.3 45.1 46.3 32.7 38.9 17.8 28.1 32.0 30.8 28.9 20.8

85.2 81.3 82.5 76.0 73.0 68.7 67.7 69.8 62.8 62.4 56.8 56.6 39.5 38.5 36.3 36.2 43.2 25.4 37.7 34.0 28.0 31.2 33.3 27.8 22.4

80.7 81.4 75.5 78.1 72.2 63.4 67.1 66.0 60.4 58.1 54.6 51.8 53.5 48.4 57.1 54.4 43.4 49.2 42.4 37.7 46.0 31.1 29.6 31.6 26.8

84.1 78.4 79.8 79.1 69.9 71.5 67.0 61.1 50.1 53.7 58.1 39.3 51.0 44.6 10.7 41.4 31.2 5.6 41.7 32.2 13.2 32.1 29.1 25.8 7.4

83.3 68.4 79.1 76.3 62.4 69.6 60.6 65.8 60.4 59.9 58.4 55.4 49.0 42.8 41.2 53.4 40.0 46.0 41.2 32.9 50.6 31.1 29.5 29.4 26.2

82.1 79.3 77.9 75.9 71.8 65.7 67.0 67.1 58.7 60.1 55.7 49.3 55.2 45.5 39.8 40.2 40.0 46.6 43.0 34.5 15.6 28.8 29.5 28.3 24.9

83.7 82.7 79.8 79.5 70.3 68.1 66.9 68.8 61.3 59.1 58.4 41.4 51.2 47.9 36.9 43.8 38.0 32.3 43.5 35.9 31.6 31.3 29.4 30.0 23.5

80.2 79.6 65.8 77.6 59.1 65.0 57.1 58.5 47.6 48.4 53.3 43.5 43.0 38.0 23.5 24.4 29.4 22.7 30.1 32.3 11.2 27.8 25.2 27.3 21.8

83.6 80.6 78.4 77.5 71.5 68.2 66.8 66.5 59.8 59.7 57.4 51.5 50.4 44.0 43.8 43.8 41.4 40.5 39.7 32.9 31.2 30.5 29.5 28.6 22.9

19

Published as a conference paper at ICLR 2026

Table 9: Performance results at temperature t = 1.0. Scores represent the Pass@5 (%) metric. Higher is better, bold is best, italic is the second best. (* - reasoning mode)

F.2.3

Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

Gpt-oss-120B* (Medium) Qwen3-235B-A22B-Thk-2507* Gpt-oss-20B (Medium)* DeepSeek-R1-0528* Qwen3-30B-A3B-Thk-2507* Gpt-oss-120B* (Low) Qwen3-32B* Qwen3-235B-A22B* Qwen3-14B* Qwen3-30B-A3B* Gpt-oss-20B* (Low) Qwen3-8B* Qwen3-235B-A22B-Instr-2507 DeepSeek-R1-Distill-Qwen-32B* Qwen3-30B-A3B-Instr-2507 OpenRsn-Nmt-32B* OlympicCoder-7B* Qwen3-Coder-30B-A3B-Instr OpenCodeRsn-Nmt-1.1-32B* Qwen2.5-Coder-32B-Instr DeepSeek-R1-Distill-Qwen-14B* Seed-Coder-8B-Instr Qwen2.5-Coder-14B-Instr Devstral-Small-2505* Deepseek-Coder-33B-Instr

82.8 85.2 79.4 72.3 76.7 71.8 73.4 70.0 68.4 66.3 60.0 63.1 54.8 59.6 52.0 78.8 47.6 44.5 77.3 34.7 56.4 32.0 29.7 33.4 25.7

83.8 85.2 80.5 74.3 79.9 71.7 70.1 71.4 63.5 63.7 59.3 56.7 56.1 47.0 48.8 69.6 47.1 40.9 66.1 34.0 35.2 29.3 30.2 28.5 27.2

85.9 85.9 81.0 73.9 77.8 71.0 70.3 70.0 66.1 64.8 59.7 58.7 59.6 55.4 49.4 64.1 46.7 43.3 60.6 39.8 41.9 30.6 30.8 30.1 27.8

84.1 77.4 77.5 67.6 68.5 67.8 64.1 65.8 57.9 59.1 54.9 52.8 51.1 35.5 40.2 29.3 36.7 34.4 24.4 33.6 21.5 28.0 28.0 26.2 23.3

84.6 81.6 79.2 72.5 72.9 68.7 70.9 69.7 67.5 64.7 55.6 59.3 44.1 45.6 38.1 35.2 43.1 37.6 29.3 26.1 29.7 30.7 29.8 32.8 22.3

83.9 82.4 80.5 71.8 73.3 69.7 69.4 70.5 64.6 63.8 55.6 57.5 46.1 43.0 39.3 27.2 42.2 39.3 21.1 36.7 31.0 29.6 29.0 30.6 21.3

78.0 82.4 75.2 73.2 73.1 65.3 66.2 66.0 59.5 59.0 55.6 56.3 53.7 53.2 47.5 53.3 43.3 43.0 44.8 36.6 44.1 31.7 31.7 30.8 26.0

83.5 82.0 77.5 70.8 70.0 68.7 61.8 66.6 50.7 57.7 58.6 42.6 49.4 44.9 44.9 9.5 28.2 42.1 8.3 33.6 14.7 30.8 28.6 27.7 8.8

83.6 67.8 78.2 69.4 60.0 67.7 65.3 61.7 60.3 57.7 58.0 56.8 52.3 56.6 45.5 41.9 42.3 40.0 44.1 36.3 50.0 31.9 31.8 28.1 24.4

84.0 79.8 77.4 72.3 71.9 65.1 66.5 65.4 61.2 62.6 56.8 50.9 55.9 39.1 48.8 42.9 39.1 40.0 44.5 34.4 15.5 31.8 29.6 30.7 20.5

85.2 84.0 82.6 68.7 68.5 69.6 67.5 66.3 60.4 61.4 57.7 44.8 52.9 47.7 47.0 39.7 37.8 44.3 31.5 35.6 32.5 30.6 31.0 30.4 23.8

78.1 79.0 66.9 71.7 60.5 63.4 58.9 54.9 50.7 49.8 51.4 43.1 44.9 25.6 39.7 22.4 29.2 32.6 19.4 33.1 13.6 31.9 28.5 25.8 17.9

83.1 81.1 78.0 71.5 71.1 68.4 67.0 66.5 60.9 60.9 56.9 53.5 51.8 46.1 45.1 42.8 40.3 40.2 39.3 34.5 32.2 30.7 29.9 29.6 22.4

PASS @10 PERFORMANCE AT DIFFERENT SAMPLING TEMPERATURES

Table 10, Table 11 and Table 12 reports Pass@10 scores at sampling temperatures t = 0.2, t = 0.6 and t = 1.0 respectively. Each score indicates the percentage of problems solved correctly on the 10th attempt, with higher values reflecting better performance. Table 10: Performance results at temperature t = 0.2. Scores represent the Pass@10 (% ) metric. Higher is better, bold is best, italic is the second best. (* - Rsn mode) Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

Gpt-oss-120B* (Medium) Qwen3-235B-A22B-Thinking-2507* Gpt-oss-20B* (Medium) DeepSeek-R1-0528* Qwen3-30B-A3B-Thinking-2507* Gpt-oss-120B* (Low) Qwen3-235B-A22B* Qwen3-32B* Qwen3-30B-A3B* Qwen3-14B* Gpt-oss-20B* (Low) OpenRsn-Nmt-32B* Qwen3-8B* Qwen3-235B-A22B-Instr-2507 OlympicCoder-7B* OpenCodeRsn-Nmt-1.1-32B* Qwen3-30B-A3B-Instr-2507 DeepSeek-R1-Distill-Qwen-32B* Qwen3-Coder-30B-A3B-Instr DeepSeek-R1-Distill-Qwen-14B* Qwen2.5-Coder-32B-Instr Seed-Coder-8B-Instr Devstral-Small-2505* Qwen2.5-Coder-14B-Instr DeepSeek-Coder-33B-Instr

87.0 87.0 80.2 80.9 80.9 73.3 72.5 72.5 69.5 71.0 64.1 84.0 60.3 55.7 53.4 78.6 51.9 55.0 45.0 54.2 36.6 29.8 29.0 23.7 22.9

86.3 88.6 84.0 82.4 82.4 72.5 74.8 73.3 67.2 63.4 64.9 76.3 56.5 58.0 53.4 67.9 48.9 38.9 42.0 35.9 32.8 26.7 27.5 27.5 23.7

87.8 88.6 82.4 83.2 79.4 74.1 73.3 74.1 66.4 66.4 63.4 74.1 55.7 63.4 51.2 65.7 51.2 48.9 43.5 45.8 40.5 32.8 26.0 29.8 24.4

86.3 84.0 84.0 78.6 68.7 70.2 69.5 67.2 61.1 60.3 60.3 39.7 51.2 48.1 44.3 32.1 33.6 35.1 36.6 26.0 35.1 27.5 26.7 27.5 20.6

87.0 84.0 81.7 80.2 77.9 75.6 71.8 72.5 65.7 64.9 60.3 45.8 56.5 41.2 48.9 37.4 37.4 48.1 38.9 35.9 14.5 32.1 28.2 27.5 17.6

84.7 85.5 84.7 83.2 77.9 74.8 74.8 73.3 64.9 67.9 58.8 40.5 56.5 37.4 47.3 26.7 32.8 42.8 36.6 38.2 32.8 29.0 30.5 26.0 16.0

84.0 85.5 77.9 80.9 74.8 68.7 70.2 69.5 63.4 63.4 57.3 65.7 52.7 55.0 48.1 58.0 51.9 46.6 45.8 48.9 35.9 33.6 27.5 29.8 25.2

87.8 82.4 80.9 80.9 77.9 71.8 72.5 67.9 58.8 54.2 58.8 18.3 41.2 52.7 37.4 5.3 46.6 44.3 41.2 15.3 34.4 28.2 27.5 24.4 10.7

88.6 71.8 81.7 80.9 62.6 68.7 61.1 67.9 62.6 64.9 59.5 49.6 56.5 51.2 50.4 54.2 45.8 52.7 43.5 51.9 32.8 28.2 29.8 30.5 25.2

84.0 84.7 79.4 80.9 79.4 70.2 72.5 68.7 64.9 63.4 61.1 55.0 51.9 55.7 45.0 53.4 46.6 41.2 39.7 15.3 33.6 28.2 28.2 29.0 26.0

87.0 87.8 83.2 81.7 73.3 72.5 71.0 73.3 63.4 65.7 64.9 49.6 43.5 52.7 45.8 37.4 49.6 47.3 45.8 35.9 36.6 27.5 27.5 27.5 23.7

84.0 84.0 76.3 81.7 67.2 67.9 61.8 63.4 51.9 51.2 55.7 30.5 42.8 42.8 38.9 28.2 37.4 26.7 33.6 14.5 31.3 28.2 24.4 25.2 21.4

86.2 84.5 81.4 81.3 75.2 71.7 70.5 70.3 63.3 63.0 60.8 52.4 52.1 51.2 47.0 45.4 44.5 44.0 41.0 34.8 33.1 29.3 27.7 27.4 21.4

Table 11: Performance results at temperature t = 0.6. Scores represent the Pass@10 (%) metric. Higher is better, bold is best, italic is the second best. (* - Rsn mode) Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

Gpt-oss-120B* (Medium) Qwen3-235B-A22B-Thinking-2507* Gpt-oss-20B* (Medium) DeepSeek-R1-0528* Qwen3-30B-A3B-Thinking-2507* Gpt-oss-120B* (Low) Qwen3-32B* Qwen3-235B-A22B* Qwen3-14B* Qwen3-30B-A3B* Gpt-oss-20B* (Low) Qwen3-8B* Qwen3-235B-A22B-Instr-2507 OpenRsn-Nmt-32B* DeepSeek-R1-Distill-Qwen-32B* OpenCodeRsn-Nmt-1.1-32B* Qwen3-30B-A3B-Instr-2507 OlympicCoder-7B* Qwen3-Coder-30B-A3B-Instr DeepSeek-R1-Distill-Qwen-14B* Qwen2.5-Coder-32B-Instr Seed-Coder-8B-Instr Devstral-Small-2505* Qwen2.5-Coder-14B-Instr DeepSeek-Coder-33B-Instr

86.7 84.5 83.0 81.2 76.1 72.7 71.7 71.3 65.0 63.6 61.9 56.2 54.6 52.5 50.5 49.1 48.4 46.6 42.1 37.3 36.3 33.5 32.5 30.9 25.7

87.8 87.0 83.2 83.2 80.9 74.1 75.6 75.6 72.5 66.4 63.4 64.1 58.0 80.9 63.4 78.6 52.7 56.5 44.3 59.5 37.4 36.6 35.9 30.5 29.8

84.7 86.3 83.2 81.7 77.1 69.5 71.0 70.2 65.7 62.6 59.5 55.7 56.5 67.9 59.5 59.5 52.7 47.3 45.0 51.9 40.5 32.1 32.8 34.4 29.0

87.8 87.8 84.0 80.9 82.4 72.5 73.3 71.8 68.7 66.4 64.9 60.3 58.8 79.4 48.1 69.5 51.2 54.2 42.8 41.2 35.1 32.8 31.3 30.5 27.5

87.8 84.0 83.2 79.4 73.3 72.5 71.8 71.0 64.1 62.6 61.8 57.3 54.2 38.2 40.5 37.4 45.0 45.0 37.4 24.4 36.6 32.1 32.1 29.0 25.2

87.8 90.1 83.2 81.7 81.7 74.1 74.8 74.8 66.4 69.5 61.8 62.6 64.1 74.1 55.0 68.7 54.2 49.6 42.0 42.8 39.7 31.3 31.3 31.3 29.0

87.0 84.7 84.7 81.7 79.4 71.8 72.5 76.3 66.4 67.9 61.8 60.3 50.4 47.3 54.2 46.6 42.8 50.4 41.2 36.6 21.4 35.9 34.4 30.5 24.4

86.3 84.7 83.2 83.2 77.1 71.8 74.8 72.5 67.2 61.8 61.8 47.3 55.0 44.3 51.2 41.2 51.9 44.3 45.8 39.7 38.9 35.1 32.1 32.1 26.0

86.3 82.4 81.7 80.9 75.6 69.5 72.5 71.0 65.7 63.4 61.1 51.9 59.5 51.9 50.4 55.0 48.9 45.0 45.8 24.4 37.4 31.3 31.3 31.3 28.2

85.5 74.1 85.5 80.2 69.5 74.8 70.2 64.1 64.9 64.1 62.6 60.3 51.2 50.4 57.3 51.2 45.8 43.5 43.5 55.0 38.9 35.1 32.1 32.1 28.2

86.3 84.0 84.0 83.2 74.8 75.6 64.9 73.3 56.5 58.8 62.6 44.3 54.2 15.3 48.1 10.7 48.9 39.7 43.5 19.9 35.1 35.9 33.6 28.2 9.9

85.5 84.7 73.3 80.2 65.7 72.5 64.1 64.1 55.0 53.4 59.5 48.9 47.3 32.1 33.6 31.3 41.2 35.1 32.8 16.0 36.6 29.8 28.2 30.5 23.7

87.8 84.0 87.0 78.6 75.6 74.1 74.8 71.0 67.2 65.7 61.8 61.1 46.6 48.9 44.3 39.7 45.0 48.1 41.2 35.9 37.4 33.6 35.1 30.5 27.5

20

Published as a conference paper at ICLR 2026

Table 12: Performance results at temperature t = 1.0. Scores represent the Pass@10 (%) metric. Higher is better, bold is best, italic is the second best. (* - Rsn mode)

F.3

Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

Gpt-oss-120B* (Medium) Qwen3-235B-A22B-Thk-2507* Gpt-oss-20B* (Medium) DeepSeek-R1-0528* Qwen3-30B-A3B-Thk-2507* Gpt-oss-120B* (Low) Qwen3-32B* Qwen3-235B-A22B* Qwen3-14B* Qwen3-30B-A3B* Gpt-oss-20B* (Low) Qwen3-8B* Qwen3-235B-A22B-Instr-2507 DeepSeek-R1-Distill-Qwen-32B* OpenRsn-Nmt-32B* Qwen3-30B-A3B-Instr-2507 OpenCodeRsn-Nmt-1.1-32B* OlympicCoder-7B* Qwen3-Coder-30B-A3B-Instr DeepSeek-R1-Distill-Qwen-14B* Qwen2.5-Coder-32B-Instr Seed-Coder-8B-Instr Qwen2.5-Coder-14B-Instr Devstral-Small-2505* DeepSeek-Coder-33B-Instr

86.3 84.8 82.6 75.9 75.4 73.1 71.9 70.8 65.7 64.5 61.2 58.6 55.8 53.2 51.8 49.7 47.5 46.1 42.8 39.5 38.2 33.8 33.2 33.1 26.0

85.5 90.1 82.4 77.9 78.6 74.8 76.3 72.5 72.5 68.7 64.1 67.2 58.0 64.9 82.4 55.7 81.7 52.7 48.1 60.3 38.2 35.1 32.8 38.2 29.0

80.9 85.5 81.7 77.9 75.6 71.0 71.0 71.0 64.9 61.1 60.3 60.3 56.5 58.0 61.1 51.2 51.2 50.4 45.8 48.9 39.7 35.1 34.4 34.4 29.0

87.0 87.8 84.7 79.4 83.2 76.3 74.8 74.8 67.2 65.7 63.4 60.3 60.3 55.0 77.1 54.2 71.8 51.9 44.3 42.0 38.2 31.3 33.6 30.5 31.3

87.0 80.2 82.4 74.8 74.1 72.5 71.0 71.0 64.1 64.1 60.3 59.5 55.7 44.3 38.2 43.5 35.1 42.8 37.4 27.5 35.9 30.5 31.3 30.5 27.5

89.3 90.1 85.5 77.9 81.7 75.6 74.8 74.1 71.0 67.2 64.9 64.9 64.1 59.5 71.8 54.2 71.0 51.9 45.0 48.9 43.5 32.8 34.4 33.6 31.3

87.8 85.5 84.7 76.3 77.1 73.3 76.3 74.1 71.8 68.7 58.8 64.1 48.9 54.2 46.6 44.3 41.2 47.3 39.7 42.8 33.6 33.6 32.8 36.6 26.0

87.0 87.0 86.3 71.8 73.3 74.8 71.8 71.0 65.7 64.9 61.1 51.2 55.7 53.4 51.9 51.9 38.2 42.8 45.8 41.2 38.2 32.8 34.4 34.4 26.7

87.0 82.4 80.2 76.3 74.8 68.7 71.0 68.7 66.4 66.4 61.1 56.5 59.5 48.9 52.7 53.4 54.2 46.6 43.5 24.4 37.4 35.1 33.6 34.4 23.7

86.3 74.1 81.7 73.3 65.7 71.8 70.2 66.4 64.1 61.8 61.1 61.1 56.5 60.3 52.7 48.9 53.4 48.1 41.2 55.0 40.5 35.1 35.1 32.1 27.5

86.3 85.5 80.9 73.3 74.1 73.3 67.2 71.0 56.5 62.6 64.1 48.9 51.9 51.9 15.3 49.6 13.7 35.1 44.3 21.4 35.9 35.1 32.1 29.8 13.7

84.0 84.0 75.6 76.3 68.7 69.5 64.9 60.3 55.7 54.2 55.7 47.3 49.6 33.6 32.8 43.5 26.7 35.9 36.6 20.6 37.4 35.1 32.8 29.0 20.6

87.0 85.5 84.7 75.6 77.9 75.6 73.3 74.8 68.7 68.7 59.5 61.8 52.7 54.2 38.9 45.8 31.3 48.1 41.2 41.2 40.5 33.6 31.3 33.6 25.2

P ERFORMANCE ON THE M ULTI -LCB (J UL 2024-M AY 2025 S UBSET )

Table 13 reports Pass@1 scores at sampling temperature t = 0.2 for all evaluated models on the Multi-LCB subset containing tasks from July 2024 to May 2025. Each score reflects the percentage of problems solved correctly on the first attempt, with higher values indicating better performance. Table 13: Performance results on Multi-LCB tasks from July 2024 till May 2025. Scores represent the Pass@1 (%) metric (higher is better). (* - reasoning mode)

F.4

Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

Qwen3-235B-A22B-Thk-2507* Qwen3-30B-A3B-Thk-2507* Qwen3-235B-A22B* Qwen3-32B* Qwen3-30B-A3B* Qwen3-14B* Qwen3-235B-A22B-Instr-2507 OlympicCoder-32B* Qwen3-8B* Qwen3-30B-A3B-Instr-2507 Qwen3-Coder-30B-A3B-Instr OlympicCoder-7B* OpenRsn-Nemotron-32B* Qwen2.5-Coder-32B-Instr Seed-Coder-8B-Instr Qwen2.5-Coder-14B-Instr Devstral-Small-2505 OpenCodeRsn-Nemotron-1.1-32B* DeepSeek-Coder-33B-Instr

76.7 69.4 65.8 63.4 60.8 57.5 49.5 51.3 49.9 40.6 35.2 36.0 69.8 27.4 20.7 21.3 25.4 62.8 17.5

78.3 68.2 63.4 63.6 56.5 54.7 47.9 51.9 44.3 36.4 30.0 35.6 49.5 27.4 21.7 21.1 20.9 38.0 15.9

78.3 66.6 59.4 59.6 56.3 51.7 47.9 47.9 42.5 37.8 33.0 31.4 41.9 30.2 21.9 23.5 23.7 30.6 18.9

59.6 44.5 49.7 43.7 37.0 38.4 38.6 34.6 27.2 22.1 21.1 24.5 12.9 24.7 17.7 20.5 19.1 8.5 12.1

71.4 52.1 56.3 51.9 49.9 50.7 29.4 36.0 39.8 23.7 27.2 25.6 13.7 6.0 22.1 23.1 22.9 8.9 8.7

62.2 48.5 51.9 54.7 53.1 49.9 22.7 34.4 41.7 22.5 26.0 24.1 10.5 28.0 21.7 18.9 22.7 8.0 5.8

69.6 59.8 57.5 53.5 50.5 48.5 46.5 42.9 38.0 37.4 34.0 28.9 33.0 27.8 22.5 22.5 21.1 27.0 17.7

50.9 52.5 45.1 41.4 43.1 33.4 42.9 34.0 24.7 32.4 31.9 13.3 1.8 25.4 20.7 17.3 19.5 1.4 2.8

54.1 45.1 51.9 53.9 50.5 50.1 41.6 45.3 42.3 35.0 34.2 26.6 20.7 24.1 22.3 21.7 19.1 23.5 14.7

70.0 59.0 52.7 52.5 50.7 46.9 43.7 43.3 33.6 36.8 31.0 24.7 20.3 27.0 21.5 22.7 21.3 24.7 15.7

64.6 46.7 48.7 42.3 39.2 37.6 42.5 34.6 18.1 26.8 33.6 24.7 16.5 26.8 21.5 21.3 20.7 12.5 16.9

56.9 41.0 32.8 35.6 30.4 30.2 29.8 25.6 24.9 19.7 18.3 13.3 7.8 22.7 19.3 18.9 15.1 5.6 12.9

66.0 54.4 52.9 51.3 48.2 45.8 40.2 40.1 35.6 31.0 29.6 25.7 24.9 24.8 21.2 21.1 21.0 21.0 13.3

P ERFORMANCE ON THE C OMPLETE M ULTI -LCB B ENCHMARK

Table 14: Performance results on Multi-LCB (n=1055 per language). Scores represent the Pass@1 (%) metric (higher is better). (* - reasoning mode) Model

Python

C++

Java

Go

JS

TS

C#

Rust

Ruby

PHP

Kotlin

Scala

Avg

Qwen3-235B-A22B-Thk-2507* Qwen3-30B-A3B-Thk-2507* Qwen3-235B-A22B* Qwen3-32B* Qwen3-30B-A3B* Qwen3-14B* Qwen3-235B-A22B-Instr-2507 OlympicCoder-32B* Qwen3-8B* Qwen3-30B-A3B-Instr-2507 Qwen3-Coder-30B-A3B-Instr Qwen2.5-Coder-32B-Instr OlympicCoder-7B* Qwen2.5-Coder-14B-Instr OpenRsn-Nmt-32B* Seed-Coder-8B-Instr OpenCodeRsn-Nmt-1.1-32B* Devstral-Small-2505 DeepSeek-Coder-33B-Instr

85.6 80.6 77.5 77.5 74.8 73.4 59.8 61.4 65.6 52.5 47.8 40.5 45.0 33.6 80.2 28.3 72.4 30.3 22.4

86.6 80.2 72.0 70.2 67.9 67.2 59.4 57.3 54.7 48.4 41.3 36.8 43.0 28.0 56.9 27.8 47.2 23.9 18.9

85.9 78.5 72.0 73.3 68.2 65.9 59.3 58.7 54.5 51.5 43.5 41.6 40.7 35.1 50.5 28.8 39.1 27.9 22.7

60.0 50.2 60.2 51.3 43.9 47.8 45.0 41.0 36.3 31.6 26.1 33.6 30.0 28.3 13.8 22.7 8.5 21.1 15.5

80.8 62.5 64.6 64.6 62.5 63.5 34.3 44.8 51.4 33.6 36.6 5.7 32.8 32.2 16.5 30.1 11.8 27.5 12.0

73.7 57.7 63.3 64.3 63.2 62.9 27.0 41.7 52.9 30.8 34.1 38.7 31.9 23.0 12.5 29.0 10.0 26.8 8.5

79.1 73.5 68.5 67.1 61.4 60.7 56.4 52.9 51.8 49.2 46.1 39.1 36.3 30.9 39.9 29.9 30.9 25.7 22.3

58.8 63.2 56.2 51.4 52.8 41.7 53.5 41.1 31.9 43.2 42.3 36.4 18.6 31.6 4.3 25.6 2.1 24.4 3.1

65.0 58.7 65.2 68.4 64.3 64.8 53.5 55.0 55.6 46.4 44.6 38.0 34.5 26.6 25.1 28.5 30.3 26.4 17.9

78.8 70.0 62.4 64.0 60.8 60.0 54.9 53.1 47.5 47.6 41.7 38.0 28.5 26.3 21.5 27.0 28.3 26.0 18.8

75.7 60.5 53.7 54.5 47.1 41.7 46.3 41.0 23.6 33.8 33.5 33.0 28.5 26.6 17.4 27.7 13.7 24.0 18.8

66.1 50.6 40.0 48.1 39.9 39.0 35.4 31.2 34.4 28.1 22.3 30.3 19.8 26.3 10.1 23.8 7.8 17.9 15.4

74.7 65.5 62.9 62.9 58.9 57.4 48.7 48.3 46.7 41.4 38.3 34.1 32.7 29.5 29.1 27.4 25.2 24.9 16.4

Table 14 reports Pass@1 scores at sampling temperature t = 0.2, for all 19 evaluated models on the complete Multi-LCB benchmark, which contains 1,055 tasks per programming language. Each 21

Published as a conference paper at ICLR 2026

score reflects the percentage of problems solved correctly on the first attempt, with higher values indicating better performance. Models marked with an asterisk (*) are reasoning-enhanced variants.

G

C OMPUTATION TIME

Table 15 reports the average compilation and execution time required to evaluate one full Multi-LCB run (1,050 tasks per language) across 90 parallel CPUs. On average, each language requires about 8 min 50 s (≈ 530 s) per model, with a total wall-clock time of roughly 106 hours when aggregated over all twelve languages. Execution cost varies noticeably by language. Ruby shows the highest mean time at 17 min 37 s, followed by Go and Python, each exceeding 11 minutes on average. In contrast, Kotlin, PHP, and JavaScript complete evaluation in under 4 minutes. These differences primarily reflect compilation overheads and runtime performance of each language’s toolchain, and they guide resource planning for future large-scale model evaluations. Table 15: Evaluation times (compilation + execution on tests + matching) across programming languages. Runs were executed in parallel on 90 CPUs over 1050 tasks (v1–v6). Averages and standard deviations are computed across the measured models.

H

Language

Avg. Time (mm:ss)

Std. Dev. (mm:ss)

Avg. Time (s)

Std. Dev. (s)

C# C++ Go Java JavaScript Kotlin PHP Python Ruby Rust Scala TypeScript

9:10 10:25 12:36 10:44 3:44 3:14 3:29 11:38 17:37 7:41 7:25 8:17

3:08 2:54 2:36 2:55 1:05 1:46 0:49 2:56 3:27 3:50 1:16 0:55

550.15 625.72 756.13 644.07 224.73 194.87 209.82 698.71 1057.42 461.04 445.24 497.59

188.83 174.17 156.71 175.96 65.14 106.23 49.78 176.07 207.45 230.24 76.41 55.37

Average Total (sum)

8:50 106:05

4:12 —

530.46 6365.48

252.51 —

L ANGUAGES AND C OMPILER V ERSIONS

All experiments were conducted in a controlled environment using the following language runtimes and compiler versions to ensure consistency and reproducibility across all tasks in Multi-LCB: • C++: gcc 14.3.0 • Java: OpenJDK 8.0.412 • Python: 3.12.11 • Rust: 1.88.0 • Go: 1.22.12 • Ruby: 3.3.6 • JavaScript (Node.js): 20.19.4 • TypeScript (Deno): 2.3.4 • C# (Mono): 6.12.0.199 • Compilers (general): 1.11.0 • PHP: 8.1.0 • Kotlin: 2.2.0 • Scala: 2.11.8 • pip: 25.2 22

Published as a conference paper at ICLR 2026

These versions were used consistently for compilation, execution, and evaluation to guarantee reproducibility of all Multi-LCB results.

23

Published as a conference paper at ICLR 2026

I

P LATFORM A NALYSIS

Figures 10, 11, and 12 show performance comparison between LeetCode and AtCoder platforms across different programming languages. Models demonstrate varying capabilities depending on the platform, with some excelling on LeetCode’s interview-style problems while others perform better on AtCoder’s competitive programming tasks.

Figure 10: Code generation performance heatmap by platform for Python, C++, Java, and C#. Shows overall performance and platform-specific results (LeetCode vs AtCoder) across different models. Values represent Pass@1 scores (%).

24

Published as a conference paper at ICLR 2026

Figure 11: Code generation performance heatmap by platform for Ruby, PHP, Kotlin, and JavaScript. Shows overall performance and platform-specific results (LeetCode vs AtCoder) across different models. Values represent Pass@1 scores (%).

25

Published as a conference paper at ICLR 2026

Figure 12: Code generation performance heatmap by platform for TypeScript, Go, Rust, and Scala. Shows overall performance and platform-specific results (LeetCode vs AtCoder) across different models. Values represent Pass@1 scores (%).

26

Published as a conference paper at ICLR 2026

J

D IFFICULTY A NALYSIS

Figures 13, 14, and 15 present performance breakdown by difficulty levels (Easy, Medium, Hard) across programming languages. The results reveal significant performance degradation as problem complexity increases, with Hard problems showing the largest performance gaps between models.

Figure 13: Code generation performance heatmap by difficulty level for Python, C++, Java, and C#. Shows overall performance and difficulty-specific results (Easy, Medium, Hard) across different models. Values represent Pass@1 scores (%).

27

Published as a conference paper at ICLR 2026

Figure 14: Code generation performance heatmap by difficulty level for Ruby, PHP, Kotlin, and JavaScript. Shows overall performance and difficulty-specific results (Easy, Medium, Hard) across different models. Values represent Pass@1 scores (%).

28

Published as a conference paper at ICLR 2026

Figure 15: Code generation performance heatmap by difficulty level for TypeScript, Go, Rust, and Scala. Shows overall performance and difficulty-specific results (Easy, Medium, Hard) across different models. Values represent Pass@1 scores (%).

29

Published as a conference paper at ICLR 2026

K

T EMPORAL A NALYSIS

Figures 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, and 27 illustrate monthly performance trends from 2023 to 2025 across different programming languages. A notable declining trend is observed across all models and languages, with top-performing models dropping from approximately 80% to 60% Pass@1 scores over time. This consistent degradation pattern appears universally across programming languages, suggesting systematic factors rather than language-specific issues. The decline may be attributed to two primary factors: (1) data contamination effects, where models perform better on older, potentially seen problems, and (2) increasing problem complexity over time as benchmark creators develop more challenging tasks to maintain discriminative power.

Figure 16: Monthly Pass@1 trends for Python.

Figure 17: Monthly Pass@1 trends for C++.

Figure 18: Monthly Pass@1 trends for C#.

Figure 19: Monthly Pass@1 trends for Java.

Figure 21: JavaScript.

Figure 20: Monthly Pass@1 trends for Go.

30

Monthly Pass@1 trends for

Published as a conference paper at ICLR 2026

Figure 22: Monthly Pass@1 trends for Kotlin.

Figure 23: Monthly Pass@1 trends for PHP.

Figure 24: Monthly Pass@1 trends for Ruby.

Figure 25: Monthly Pass@1 trends for Rust.

Figure 26: Monthly Pass@1 trends for Scala.

Figure 27: Monthly Pass@1 trends for TypeScript.

31

Published as a conference paper at ICLR 2026

L

L ANGUAGES ERRORS TYPE

Figures 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42 and 43 illustrate detailed error breakdowns across different programming languages and models. Several consistent patterns emerge: 1. Wrong-answer (WA) errors dominate across almost all languages and models. For every model, WA is the largest source of failure in both Python and non-Python languages, indicating that the primary bottleneck remains algorithmic correctness rather than compilation or parsing. 2. Compiled languages show substantially more compiler- and type-related errors. Languages such as C++, Java, Rust, and Go exhibit significantly higher rates of compilation errors (e.g., missing imports, type mismatches, incorrect signatures) compared to Python. This pattern is consistent across all models and reflects the challenge of generating syntactically valid and type-correct code when strict compilation pipelines are enforced. 3. Runtime exceptions increase in languages that require explicit input parsing. In languages like Java, C#, and Go, runtime errors (e.g., NullPointerException, IndexError, ValueError) are far more frequent than in Python. This supports the hypothesis that the STDIN/STDOUT format, while uniform across languages, exposes weaknesses in model robustness to input handling and data conversion. 4. Timeout and resource-related failures appear more often in slower languages and for reasoning-tuned models. Java, Rust, and Go show noticeably more TimeoutExpired cases, likely because models occasionally generate inefficient implementations. Reasoning-heavy models (e.g., R1-0528, Nemotron-32B) are more prone to long-running solutions when they attempt more complex multi-step logic. 5. Empty-code and trivial-syntax errors are rare but nonzero. These errors appear mostly in smaller models (e.g., 7B-14B) and are nearly absent for 30B+ models. This indicates that larger models rarely fail at the initial code-structuring stage, with most errors occurring deeper in the execution pipeline. 6. Cross-model consistency in error profiles. Despite architectural and training differences, the overall error distributions are remarkably stable across models, demonstrating that: Python remains the least error-prone language, Compiled languages introduce predictable error modes, and Languages with verbose input/output handling (Java, C#, Go) amplify runtime failures. 7. Error distributions reinforce the observed performance gaps. The breakdowns offer a mechanistic explanation for the Pass@1 disparities reported in the main results. For example, models underperforming in Rust and C++ do so not because they fail to produce solutions, but because syntactic and type-level correctness is significantly harder to achieve in those languages.

Figure 28: deepseek-coder-33b-instruct

Figure 29: DeepSeek-R1-0528*

32

Published as a conference paper at ICLR 2026

Figure 30: Devstral-Small-2505

Figure 31: OlympicCoder-7B*

Figure 33: 1.1-32B*

Figure 32: OlympicCoder-32B*

Figure 34: OpenReasoning-Nemotron-32B*

OpenCodeReasoning-Nemotron-

Figure 35: Qwen2.5-Coder-14B-Instruct

33

Published as a conference paper at ICLR 2026

Figure 36: Qwen2.5-Coder-32B-Instruct

Figure 37: Qwen3-8B*

Figure 38: Qwen3-14B*

Figure 39: Qwen3-30B-A3B*

Figure 41: 2507*

Figure 40: Qwen3-30B-A3B-Instruct-2507

34

Qwen3-235B-A22B-Thinking-

Published as a conference paper at ICLR 2026

Figure 42: Qwen3-Coder-30B-A3B-Instruct

Figure 43: Seed-Coder-8B-Instruct

35