Official Review · ICLR 2026

This paper introduces DevBench, a new benchmark for evaluating code generation models, specifically on code completion tasks. The benchmark's primary contribution is its "telemetry-driven" design, where the task categories are derived from an analysis of over one billion real-world developer interactions, aiming for high "ecological validity". To avoid data contamination and protect privacy, the benchmark consists of 1,800 synthetically generated instances that were subsequently validated by human experts. DevBench spans six programming languages (Python, JS, TS, Java, C++, C#) and six task categories (e.g., API Usage, Code Purpose Understanding, Pattern Matching). The paper evaluates 9 different models using a combination of functional correctness (Pass@1), similarity metrics, and LLM-judge scoring. * Focus on Ecological Validity and Contamination Resistance: The paper's core motivation—to create a benchmark grounded in "observed developer behavior" rather than arbitrary open-source scrapes—is a significant strength. The use of a synthetic generation pipeline based on telemetry-derived patterns, rather than using the telemetry data directly, is a clever approach to avoiding privacy issues and, crucially, training data contamination. * Human-in-the-Loop Validation: The inclusion of a rigorous, multi-annotator human review process to validate the realism, usefulness, and complexity of each generated instance is a high-quality step that many benchmarks lack . * Detailed Diagnostics: The benchmark's structure, spanning 6 languages and 6 categories, provides a fine-grained framework for diagnosing model strengths and weaknesses (e.g., the case study on DeepSeek-V3) beyond a single aggregate score. 1. Contradiction in Benchmark Difficulty: The primary weakness is the conflict between the paper's claims of high complexity (high cyclomatic complexity in Table 3) and the high Pass@1 scores in Table 5. A benchmark with an 84.8% Pass@1 for the best model (and 90.3% in one category ) is not a challenging benchmark. This high pass rate suggests the benchmark fails in its primary goal of rigorously evaluating and differentiating SOTA models. 2. Lack of Transparency in Data Curation: The process of moving from "one billion developer interactions" to 1,800 instances is a black box. The paper uses vague terms like "sampling" and "discussions" without providing a systematic, reproducible methodology. This opaqueness undermines the central claim of being "telemetry-driven," as the selection of "common challenging scenarios" appears subjective and unverifiable. 3. Insufficient Model Evaluation: The paper fails to evaluate a sufficient range of models to demonstrate its discriminative power. The evaluation primarily includes a few top-tier proprietary models and one small 3B model, with a large gap in between. The lack of evaluation on widely-used open-source models (e.g., Code Llama 7B/13B/34B, or other intermediate-sized models) makes it impossible t