arXiv:2604.24712v1 [cs.SE] 27 Apr 2026
When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation Amal Akli
Mike Papadakis
University of Luxembourg Luxembourg [email protected]
University of Luxembourg Luxembourg [email protected]
Maxime Cordy
Yves Le Traon
University of Luxembourg Luxembourg [email protected]
University of Luxembourg Luxembourg [email protected]
Abstract Large language models (LLMs) are increasingly used for code generation, yet the correctness of their outputs depends not only on model capability but also on how tasks are specified. Prior studies demonstrate that small changes in natural language prompts, particularly under-specification can substantially reduce code correctness; however, these findings are largely based on minimalspecification benchmarks such as HumanEval and MBPP, where limited structural redundancy may exaggerate sensitivity. In this exploratory study, we investigate how prompt structure, task complexity, and specification richness interact with LLM robustness to prompt mutations. We evaluate 10 models, ranging from small (6–7B) to large open-source (15–34B) and cutting-edge reasoning models, across HumanEval and the structurally richer LiveCodeBench. Our results reveal that robustness is not a fixed property of LLMs but is highly dependent on prompt structure: the same under-specification mutations that degrade performance on HumanEval have near-zero net effect on LiveCodeBench due to redundancy across descriptions, constraints, examples, and I/O conventions. Surprisingly, we also find that prompt mutations can improve correctness. In LiveCodeBench, under-specification often breaks misleading lexical or structural cues that trigger incorrect retrieval-based solution strategies, leading to correctness improvements that counterbalance degradations. Manual analysis identifies consistent mechanisms behind these improvements, including the disruption of over-fitted terminology, removal of misleading constraints, and elimination of spurious identifier triggers. Overall, our study shows that structurally rich task descriptions can substantially mitigate the negative effects of under-specification and, in some cases, even enhance correctness. We outline categories of prompt modifications that positively influence the behavior of LLM code-generation, offering practical insights into the construction of robust coding prompts.
1
Introduction
Large language models (LLMs) such as CodeLlama [33], DeepSeekCoder [13], Qwen2.5Coder [15], and proprietary API-based models [1, 28] are now widely used for code generation in real-world development workflows [19]. Despite these advances, the correctness of generated code remains highly variable, and understanding
the factors that influence this variability is critical for both practitioners and researchers. A key factor is the natural-language specification used to describe the task. In contrast to traditional program synthesis, where specifications are formal and precise, LLM-based code generation relies on prompts that are often incomplete, ambiguous, or inconsistently structured. Prior work shows that even minor changes in wording can significantly affect correctness [6, 24, 30, 34, 43], and that under-specification (i.e., missing or weakened requirements) can substantially degrade performance [21]. These findings have led to a prevailing assumption that more precise and more complete prompts are necessary for reliable code generation. However, existing evidence is almost exclusively derived from minimal-specification benchmarks such as HumanEval and MBPP. These benchmarks express tasks through short, single-docstring descriptions with limited structural redundancy, making them highly sensitive to perturbations. They also exhibit contamination issues, which may result in fallacious experimental results [41]. In contrast, real-world programming tasks typically include multiple complementary specification layers [39], including descriptions, constraints, input/output formats, and examples. These components often encode overlapping information, suggesting that robustness to prompt imperfections may depend not only on wording, but also on the structure and richness of the specification. This observation motivates us to study a fundamental question: how prompt wording and structure relate to code correctness in LLMbased code generation? Addressing this question requires moving beyond minimal benchmarks and considering both the structure of prompts and the nature of the words and specificity they contain. In this paper, we conduct an exploratory study of how prompt wording, structure, and task complexity interact with code correctness in LLM-based code generation. We systematically introduce prompt mutations by altering original task descriptions, in two benchmark of contrasting complexities: HumanEval (HE – simple task descriptions) and LiveCodeBench (LCB – detailed, multilayered descriptions). We evaluate 10 models spanning small and large open-source models as well as state-of-the-art proprietary models. We measure the effect of under specification based on aggregate performance (Pass@1) and finer analysis of test result transitions (Pass→Fail and Fail→Pass). We also performed a manual
Akli et al.
LiveCodeBench/abc361_e — Minimum Tree Traversal Distance Original prompt (excerpt): Road i connects cities A_i and B_i , length C_i . Find the minimum travel distance to visit all N cities . Constraints : 1 <= C_i <= 10^9 <-- DELETED ( see " Beware overflow ." in sample ) Constraint removed: - 1 <= C_i <= 10^9
Original × WA (expected 11, got 14) total_weight = 0 max_weight = 0 for a , b , c in edges : total_weight += c max_weight = max ( max_weight , c) answer = 2 * total_weight - max_weight print ( answer )
Mutated ✓ Passes end1 , _ = bfs ( graph , root =1) end2 , diameter = bfs ( graph , start = end1 ) print (2 * total_weight - diameter )
Why: The correct formula is 2× total − diameter, where diameter is the longest root-to-leaf path found by two BFS passes. The C_i ≤ 10ˆ9 bound alongside the “Beware overflow” note primed the model toward substituting max_weight for diameter (only correct for star graphs). On the sample, max_weight = 4 but diameter = 7, giving 14 instead of 11. Removing the bound eliminated the overflow cue; the model then applied the general two-BFS algorithm.
Figure 1: A LiveCodeBench example where removing a constraint improves generated code: the constraint primed the model toward an incorrect algorithm; its removal forced reasoning from problem structure. root-cause analysis of cases where prompt mutations surprisingly lead to improvements. Our results reveal a more nuanced picture of LLM robustness than previously reported, via three main findings. Finding 1. Robustness to under-specification is governed by prompt structure rather than by model size or architecture. Mutations that cause substantial performance drops on HE have near-zero net impact on LCB, where multi-layered specifications provide redundancy across descriptions, constraints, examples, and I/O formats. Attention patterns further shows that models rely heavily on these auxiliary components, which act as fallback signals when parts of the prompt are degraded. Finding 2. Under-specification both harms and improves correctness, with these effects canceling out in rich prompts. On LCB, improvements and degradations occur in nearly equal proportion (Fail→Pass/Pass→Fail = 0.99), whereas degradations dominate on HE (0.40). We identify 69 LCB tasks that consistently improve pass rate under prompt mutations across multiple models, indicating that certain elements of the original specification can hinder correct code generation. Finding 3. Improvements arise when under-specification disrupts misleading retrieval cues. Specific words, identifiers, or constraint formulations can anchor models to incorrect, memorized solution patterns. Removing or weakening these cues reduces overfitting to surface forms and encourages reasoning from the underlying problem structure, often leading to simpler and more accurate solutions. Overall, our findings challenge the assumption that more detailed specifications necessarily yield better code generation. Instead, they highlight the importance of prompt structure and the complex role of natural-language cues in shaping model behavior. By identifying both harmful and beneficial prompt characteristics, this work provides actionable insights for constructing robust prompts and calls for a re-evaluation of how code generation systems are benchmarked and used in practice.
2
Exploratory study objectives
We seek to understand how prompt structure, complexity and wording characteristics relate to code correctness. We make a manual exploratory study that looks into the prompt pairs of under-specified and well-specified task descriptions and related them with the generated code correctness. In particular, we are trying to understand what is causing the LLM to generate correct and incorrect code. Figure 1 shows an example pair of a well-formed and underspecified prompt that leads to incorrect and correct LLM-generated code. It is surprising to see that “less is more,” the removal of an important constraint leads to correct code. Through our exploratory study we hypothesize that this is caused by the specific wording used that wrongly associates the sought solution with a wrong concept, thereby preventing the model from reasoning about the whole information provided. We exploratory come to our explanations and observations by identifying the same patterns across multiple different prompt pairs. As such our study is exploratory by nature and aims to increase our understanding of what could improve generated code correctness, leading to the formulation of concrete observations that could help the research community make a better use of existing tools and could lead stronger code generation tools. In particular, we summarize our objectives-questions: What is the relation between code correctness and task complexity and specification richness in the presence and absence of prompt mutations? We evaluate 10 code generation LLMs, including 2 reasoning models (GPT-5-mini and ClaudeSonnet-4), on HumanEval (easy tasks ) and LiveCodeBench (complex tasks). We consider 3 mutant types (LV, US, SF) and measure Pass@1 changes to determine whether the effect of the mutations relate to the complexity and richness of the underlying tasks. Does prompt under-specification uniformly degrade code generation performance ? The near-zero net effect observed on LiveCodeBench in RQ1 could reflect either insensitivity to mutations or the cancellation of two opposing forces. To distinguish between these explanations, we analyze Pass@1 changes and transitions, solutions that passed before mutation and fail after (P→F), and solutions that failed before and pass after (F→P), and check if the observed balance is systematic.
When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation
What are the key characteristics of prompts that lead to correct code as opposed to those that lead to incorrect code for the same tasks? Having established that mutations cause a consistent correctness improvement on LCB, we investigate what is causing it. We manually analyze F→P transitions on LiveCodeBench and construct a root-cause taxonomy, identifying how specific elements of the original specification (vocabulary, identifier names, and constraint lines) steer models toward incorrect solutions and how their removal or softening steers them toward correct ones.
3
Prompts used
To study how under-specification affect code correctness, we systematically mutate task descriptions using three types of transformations. The resulting 3,651 mutated instances are validated by an independent LLM judge, with a stratified random sample being reviewed by the authors of the paper.
3.1
Selected Benchmarks
We select two benchmarks that are structurally contrastive along the dimensions most relevant to prompt sensitivity: specification richness and task complexity. HumanEval [9] consists of 164 hand-written Python programming problems. Each problem communicates its specification exactly through a function signature with a short docstring ( 231 characters average), validated by unit tests. There are no separate constraints sections, no structured sample I/O, no input format specifications. Previous studies report that HumanEval suffers from benchmark contamination, with 8 to 18% direct overlap with training datasets [41], suggesting that many HumanEval problems can be memorized from publicly available repositories. LiveCodeBench (LCB) [17] collects problems from three competitive programming platforms (LeetCode, AtCoder, Codeforces), each time stamped at publication. follow a full competitive programming format, each problem contains (1) full prose description, (2) explicit constraints section with value ranges, (3) multiple sample I/O pairs with explanations, (4) input/output format specification. These channels encode overlapping information. This benchmark is contamination-free by design. Problems are time-stamped from weekly contests on LeetCode, Codeforces, and AtCoder. We sample latest 1,055 problems that range from easy to hard difficulty.
3.2
Mutation of original benchmarks
We create under-specified prompts by mutating original prompts. Thus, each description yields three mutated variants, one per mutation type. In cases where the original descriptions are too short or have no constraint to delete, no mutation is produced. All mutations are applied exclusively at the prompt level; reference solutions, test cases, and evaluation harnesses remain untouched — ensuring that any Pass@1 change can be attributed solely to the modified description. 3.2.1 Mutation Type Selection. Our three mutation types are grounded in prior empirical findings. Token-level perturbations and lexical ambiguity have been shown to introduce substantial variance in code generation correctness [6, 21, 37]. Incomplete or missing requirements, even when the description remains grammatically wellformed, steer models toward solutions that satisfy the syntax but
miss the intended semantics [11, 14, 18, 27]. Formatting noise, despite being purely surface-level, can produce measurable correctness swings in open-source models [34, 37]. Each class targets a distinct channel through which a natural-language specification can fail, making the three types complementary rather than redundant. Table 1 provides representative examples of each mutation type applied to problems from the two benchmarks. 3.2.2
Mutation Rules.
Lexical Vagueness (LV). LV mutations reduce the specificity of a task by replacing precise, task-relevant terms with semantically broader or less informative alternatives, while preserving all explicit constraints. This type of transformation is closely related to paraphrasing and synonym-substitution perturbations studied by Wang et al. [37] and Chen et al. [6]. In practice, an LV mutation can: replace precise terms or verbs with broader synonyms (e.g. “sort” becomes “arrange”); rename function and parameter identifiers into less informative names (e.g., “delimiter” becomes “filler”); generalize or remove type annotations. Under-Specification (US). US mutations remove a single explicit constraint from the prompt, resulting in a description that remains grammatically well-formed but becomes semantically underspecified. This design is motivated by empirical evidence showing that incomplete requirements are a primary source of intent mismatch in LLM-based code generation [21]. To ensure consistent under-specification, we remove only: bounds/threshold values; ordering rules; input preconditions; output edge-cases description; formal constraints; error and exception handling requirements; output type and format constraints. Syntax and Formatting (SF). SF mutations introduce surface-level noise into the prompt without altering its semantic content, targeting the susceptibility of LLMs to low-level textual corruption [34]. Sclar et al. [34] demonstrated that format variations alone can have a significant impact, motivating this mutation type. Each SF mutation can create noise at different levels: tokens; delimiters, brackets and colons; indentation and whitespaces; example-block formatting. 3.2.3 Mutants Generation. Mutations are generated using gpt-5-mini via the OpenAI Batch API [28]. A structured prompt is used for each mutation type, specifying transformation rules and constraining the model output. The generation process is iterated, with rule refinements applied as needed, until the automated judge (described below) assigns compliance scores above 85% for all types of mutations and benchmarks1 .
3.3
Validation of Mutated Benchmarks
We validate mutations using an independent LLM judge followed by manual annotation of a stratified sample. 3.3.1 Automated Quality Assessment. Judge model. We use Qwen2.5Coder-32B-Instruct [16] as the judge. It is architecturally distinct from the mutation generator (gpt-5-mini), limiting collusion risk [42]; 1 All generation prompts are available in the replication package. https://github.com/ Amal-AK/PromptAnalysis
Akli et al.
Table 1: Examples of mutation strategies applied to HumanEval and LiveCodeBench. US (Under-Specification) removes a constraint from the docstring; LV (Lexical Vagueness) paraphrases using different vocabulary and renames identifiers; SF (Syntax/Formatting) introduces typographical errors and syntax noise into both the signature and docstring. Element
Mutation type
Original prompt and its mutations
Original
“You are given two integers, 𝑛 and 𝑘.” “An array of distinct positive integers is called a k-avoiding array if there does not exist any pair of distinct elements that sum to 𝑘. Return the minimum possible sum of a k-avoiding array of length 𝑛.” Input: n=5, k=4 ⇒ Output: 18 Input: n=2, k=6 ⇒ Output: 3
US
“You are given two integers, 𝑛 and 𝑘.” “An array of integers is called a k-avoiding array if there does not exist any pair of distinct elements that sum to 𝑘. Return the minimum possible sum of a k-avoiding array of length 𝑛.” Input: n=5, k=4 ⇒ Output: 18 Input: n=2, k=6 ⇒ Output: 3
LV
“A list of different positive numbers is considered a k-avoiding list if there aren’t any two different members that add up to 𝑘. Return the smallest possible total of a k-avoiding list containing 𝑛 entries.” Input: n=5, k=4 ⇒ Output: 18 Input: n=2, k=6 ⇒ Output: 3
SF
INPUT;{"specific_prompt":"You are given two integres, 𝑛 and 𝑘. “An array of distinct positive integers is called a k-avoiding array if there does not exist any pair of distinct elements that sum to. Return the minimum possible sum of a Return the minimum possible sum . . . ” Input: n=5, k=4 ⇒ Output: 18 Constraints [no colon] . . . **INVALID****OCUMENT**}**
Original
def intersperse(numbers: List[int], delimeter: int) -> List[int]: “Insert a number ‘delimeter’ between every two consecutive elements of input list ‘numbers’.” »> intersperse([1, 2, 3], 4) ⇒ [1, 4, 2, 4, 3]
US
def intersperse(numbers: List[int], delimeter: int) -> List[int]: “Insert a number ‘delimeter’ into the input list ‘numbers’.” »> intersperse([1, 2, 3], 4) ⇒ [1, 4, 2, 4, 3]
LV
def place_between(items, filler): “Put a value ‘filler’ between neighboring entries of the provided sequence.” »> place_between([1, 2, 3], 4) ⇒ [1, 4, 2, 4, 3]
SF
def intersperse(numbers: List[int), delimter: int) -> List[int) ”’ Insert a number ‘delimter’ between every two consecutive elements of input list ‘numbers’ . . .
LCB/2811
HE/5
it runs locally, ensuring full reproducibility; and its instructionfollowing capability is sufficient for our binary-judgment protocol. The model runs in float16 with greedy decoding, outputting a single JSON object ("score": 0 or "score": 1). Evaluation criteria. Each mutant is assessed for compliance and naturalness, both framed as binary yes/no questions. Compliance is type-specific: for LV, the judge asks whether only vocabulary or identifiers were made less precise without altering the task; for SF, whether only typographical or formatting noise was introduced; for US, whether exactly one requirement was removed and nothing else changed. Naturalness asks whether the resulting description reads as something a real developer might plausibly write. The scores reported are the proportion of "yes" answers across all mutants of a given benchmark-mutation pair. Evaluation results. LV and SF achieve near-perfect compliance (0.99–1.00) and high naturalness (0.96–0.99) across both benchmarks. US compliance is slightly lower (0.89 on HumanEval, 0.85 on LiveCodeBench), as structural constraints are stricter in LCB and
removing them can produce unusually terse problem statements, reflected in a naturalness drop to 0.61 on LCB for US mutations. 3.3.2 Manual Inspection. To ground the automated scores in human judgment, three researchers independently annotated a stratified random sample of 100 mutants covering all mutation types and benchmarks, approximately 11 instances per benchmark-mutation pair, using the same binary compliance and naturalness criteria as the judge. Disagreements were resolved by majority vote. Annotators agreed with the judge on 97% of compliance assessments and 86% of naturalness assessments, confirming that the automated scores are a reliable proxy for human evaluation and that the reported quality figures can be trusted
4
Experimental Setup
We describe below our general experimental setup used throughout our study. All scripts and data are publicly available. 2 2 https://anonymous.4open.science/r/PromptAnalysis/
When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation
4.1
Models Under Study
We evaluate a diverse set of code-generation models to ensure our findings are not artifacts of a single architecture or training regime. The models are grouped into three levels. Small open-source models (∼7B parameters): Qwen2.5-Coder7B [15], DeepSeek-Coder-6.7B [13], and CodeLlama-7B [33]. These models are representative of deployable, resource-constrained settings and are widely used as baselines in the literature. Large open-source models (15B–34B parameters): Qwen2.5Coder-32B [15], DeepSeek-Coder-33B [13], CodeLlama-34B [33], Codestral-22B [26], and StarCoder2-15B [23]. This tier allows us to examine whether larger capacity yields greater robustness to prompt mutations, independent of proprietary training data or alignment procedures. Closed-source API models: GPT-5 mini [28], and Claude Sonnet 4 [1]. These represent the current state-of-the-art in code generation and serve as upper-bound references. Their inclusion enables a direct comparison between reasoning models [38] and open-source alternatives under identical evaluation conditions. Collectively, this selection spans five model families, two provider categories (open-source and proprietary), and two scale regimes, enabling controlled cross-family and cross-scale comparisons. We use greedy decoding (temperature = 0) to ensure deterministic model outputs. Each model produces exactly one outcome per problem, and re-running the same experiment yields identical results. API models are queried via the OpenAI and Anthropic batch APIs, while open-source models are served locally on NVIDIA A100 GPUs in float16 precision. Generated solutions are extracted via markdown fenced code block parsing, with fallback to modelspecific delimiters (e.g., CodeLlama’s [PYTHON] tags). Each solution is executed in an isolated process with a 20-second timeout to safely handle infinite loops and runtime errors in generated code.
4.2
Metrics
Pass@1 [7] is used as the primary correctness metric, as it reflects the real-world scenario where a developer accepts the model’s first suggestion. This metric is the de facto standard in code generation evaluation [2, 7] and enables direct comparison with prior work. Additionally, we complement Pass@1 with a transition-based analysis that examines test outcomes (pass or fail) before and after mutation. Thus, we track (i) Pass→Fail transitions, where a previously correct solution becomes incorrect after mutating the prompt, and (ii) Fail→Pass transitions, where a previously incorrect solution becomes correct. This finer-grained analysis allows us to distinguish between uniform degradation and more nuanced effects where improvements and regressions coexist and potentially cancel out at the aggregate level.
5 Experimental Results 5.1 Impact of mutations Table 2 reveals a consistent asymmetry among the results of HumanEval and LiveCodeBench; HumanEval is systematically degraded by all three types of mutations used, while LiveCodeBench remains near-neutral, with the gap continuously widening from SF to LV to US across all model sizes.
The three mutant types induce markedly different levels of degradation on HumanEval, yet remains largely ineffective on LiveCodeBench. US causes the largest drops on HumanEval (avg. −11.8%), consistent with prior findings that missing constraints are the most consequential source of intent mismatch [18, 27], while the same mutant type produces only −0.9% on LCB. LV produces moderate degradation on HumanEval (avg. −7.1%), compared to a near-zero average of −0.2% on LCB, and several models record marginal improvements. Table 3 decomposes this effect into docstring paraphrasing (Δvocab ) and identifier renaming (Δname ): both contribute independently, but Δname dominates in smaller models (DeepSeek-6.7B: −9.1% vs. −3.1% for vocabulary alone), suggesting that these models are disproportionately anchored to function signatures as retrieval cues, a pattern consistent with memorisation-driven processing [4] rather than specification understanding. SF represents a practical lower bound on both benchmarks (avg. −1.5% on HumanEval, −0.4% on LCB) [? ], with its effect on the LCB sign inconsistent across models. Execution rates in Table 2 exhibit smaller declines across models and mutation types compared to the larger drops observed in Pass@1. This suggests that our mutations mainly affect functional correctness rather than causing syntactic or runtime failures. Model scale provides no consistent protection. As Table 2 shows, the degradation is identical across small, large, and API model groups on HumanEval, with API level reasoning models equally affected (GPT-5-mini and Claude Sonnet 4 both losing ∼9.7 % under US). The divergence between benchmarks is not explained by mutation magnitude but by specification structure. Figure 2 shows that in HumanEval, drops below LV are uniformly negative across all edit distance bins with no correlation between change magnitude and severity, pointing to memorized surface form dependence rather than noise sensitivity. Figure 3 provides the structural explanation: on HumanEval, models concentrate 60–86% of attention on the description, the only available specification, whereas on LiveCodeBench, 30–50% of attention falls on I/O format and sample I/O regions that remain intact on mutated prompts. Takeaway. Sensitivity to prompt wording is an artifact of specification structure: prompts encoding information across multiple independent regions tend to be robust, suggesting that robustness results reported by previous studies overestimate the fragility of code LLMs when prompts combine descriptions, constraints, and examples.
5.2
Non-uniform mutation effects
Given that LiveCodeBench shows near-zero net sensitivity in RQ1, is this because mutations have no effect, or because something more nuanced is occurring? Table 4 break down aggregate Pass@1 deltas into per-example transitions: problems that fail on the original prompt but pass on the mutated one (F→P), and the reverse (P→F), revealing that the near-zero net effect on LCB results from two opposing trends of comparable magnitude that cancel each other. On HumanEval, mutations produce strongly asymmetric flip distributions dominated by degradation. As shown in Table 4, under LV, the average F→P/P→F ratio is 0.40, meaning that
Akli et al.
Table 2: Pass@1 and execution rate results across benchmarks and mutation types. P@1: pass@1 (%). Ex: execution rate (%). For each mutation (US, LV, SF), Pass@1 deltas relative to the original are shown in red (↓ drop) and green (↑ gain). HumanEval Orig
US
LiveCodeBench
LV
SF
Orig
US
LV
SF
Model
P@1
Ex
P@1
Ex
P@1
Ex
P@1
Ex
P@1
Ex
P@1
Ex
P@1
Ex
P@1
Ex
Small models (≤7B) CodeLlama-7B DeepSeek-Coder-6.7B Qwen2.5-Coder-7B
37.2 72.6 82.3
84.8 97.0 97.6
29.3 ↓7.9 57.3 ↓15.3 67.7 ↓14.6
90.9 93.9 97.0
29.3 ↓7.9 60.4 ↓12.2 75.6 ↓6.7
81.7 95.7 95.1
37.2 = 68.9 ↓3.7 81.7 ↓0.6
87.8 93.9 98.2
8.1 16.3 20.5
8.2 16.4 20.6
8.4 ↑0.3 16.0 ↓0.3 20.3 ↓0.2
8.5 16.1 20.4
8.2 ↑0.1 14.8 ↓1.5 22.2 ↑1.7
8.3 14.9 22.3
7.8 ↓0.3 15.2 ↓1.1 20.5 =
7.9 15.3 20.6
Large models (15B–34B) StarCoder2-15B Codestral-22B CodeLlama-34B DeepSeek-Coder-33B Qwen2.5-Coder-32B
65.9 72.6 51.2 72.0 85.4
97.6 97.0 89.6 97.6 98.8
51.2 ↓14.7 58.5 ↓14.1 42.1 ↓9.1 61.6 ↓10.4 73.2 ↓12.2
97.0 97.6 89.0 97.0 98.2
54.3 ↓11.6 65.2 ↓7.4 45.1 ↓6.1 68.3 ↓3.7 84.1 ↓1.3
95.1 96.3 89.6 97.0 97.0
62.8 ↓3.1 70.7 ↓1.9 50.0 ↓1.2 73.8 ↑1.8 86.6 ↑1.2
93.9 96.3 92.7 97.6 99.4
6.8 23.0 13.6 20.3 32.0
6.9 23.1 13.7 20.4 32.1
8.0 ↑1.2 22.5 ↓0.5 11.6 ↓2.0 19.4 ↓0.7 30.6 ↓1.4
8.1 22.6 11.7 19.5 30.7
7.3 ↑0.5 22.7 ↓0.3 13.1 ↓0.5 19.9 ↓0.4 31.7 ↓0.3
7.4 22.8 13.2 20.0 31.8
7.6 ↑0.8 22.6 ↓0.4 12.4 ↓1.2 18.8 ↓1.5 31.2 ↓0.8
7.7 22.7 12.5 18.9 31.3
Reasoning models (API) GPT-5-mini Claude Sonnet 4
96.3 95.7
97.6 98.2
86.6 ↓9.7 86.0 ↓9.7
95.7 97.6
89.0 ↓7.3 89.0 ↓6.7
91.5 97.6
93.3 ↓3.0 95.7 =
96.3 97.6
52.5 51.1
52.6 51.2
48.5 ↓4.0 49.6 ↓1.5
48.6 49.7
52.5 = 49.9 ↓1.2
52.6 50.0
53.2 ↑0.7 50.7 ↓0.4
53.3 50.8
Table 3: Pass@1 (%) on HumanEval;LVname preserves the original function name while paraphrasing the docstring; LV applies the full paraphrasing. Δvocab = LVname − Orig (effect of docstring paraphrasing); Δname = LV − LVname (additional effect of identifier renaming).
Figure 2: Pass@1 delta under LV mutations on HumanEval as a function of edit distance between original and mutated prompts (quintile bins). Each cell reports the mean Pass@1 change per model; red indicates degradation. for every example a mutation leads to fixes, it breaks 2.5 others. Under US 4, this ratio drops further to 0.20, one improvement for every five regressions. Across all models, P→F counts consistently and substantially exceed F→P counts, confirming that mutations act almost exclusively as a source of degradation on single-specification prompts. On LCB, the same mutations produce balanced flip distributions, with improvements offsetting degradations. Under LV 4, the average F→P/P→F ratio is 0.99, near perfect balance, with four models recording a positive net (StarCoder2-15B: +5, Qwen2.5-Coder-7B: +17, CodeLlama 7B: +1). Under US 4, the ratio is 0.89, still substantially higher than on HumanEval. The near-zero aggregate effect on LCB is therefore not the absence of signal but the result of two competing forces canceling each other out. A subset of LCB problems improves consistently and exclusively across models. Table 5 identifies tasks where F→P flips
Model
Orig
LVname
LV
Δvocab
Δname
Small models Qwen2.5-Coder-7B DeepSeek-Coder-6.7B CodeLlama-7B
82.3 72.6 37.2
78.0 69.5 34.8
75.6 60.4 29.3
−4.3 −3.1 −2.4
−2.4 −9.1 −5.5
Large models Qwen2.5-Coder-32B DeepSeek-Coder-33B Codestral-22B StarCoder2-15B CodeLlama-34B
85.4 72.0 72.6 65.9 51.2
84.1 73.2 68.9 54.9 41.5
84.1 68.3 65.2 54.3 45.1
−1.3 −3.7 −3.7 −11.0 −9.7
0.0 +1.2 −3.7 −0.6 +3.6
API models GPT-5-mini Claude Sonnet 4
96.3 95.7
90.9 96.3
89.0 89.0
−5.4 +0.6
−1.9 −7.3
occur in at least two models with no accompanying P→F flips elsewhere, denoted clean tasks. Under LV, 33 such tasks are identified; under US, 14. On these tasks, the F→P/P→F ratio reaches 3.08 and 3.56 respectively, with net gains of +104 and +41 model-task pairs. The consistency of these improvements across models, rather than being isolated to a single model, suggests that the original prompts contain systematic properties that hinder correct code generation, independent of model-specific behavior. To rule out the possibility that these passing solutions are artifacts of insufficient test coverage, Table 6 reports branch and line coverage of F→P solutions: mean line coverage reaches 91.2% and mean branch coverage 83.4%, both comparable to P→F solutions (91.9% and 84.5% respectively), confirming that the test suite
When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation
Figure 3: Normalized last-layer last-token attention distributed across prompt regions for HumanEval (Signature, Description, Examples) and LiveCodeBench (Description, I/O Format, Sample I/O), averaged over all attention heads. Table 4: Per-example flip analysis under LV and US mutations on LCB and HumanEval. Fail→Pass counts examples failing on the original prompt but passing on the mutated prompt; Pass→Fail counts the reverse. LCB Model
F→P
P→F
HumanEval
Table 5: Cross-model consistent Fail→Pass flip statistics on LCB under LV (lexical vagueness) and US (underspecification). A flip is “consistent” if it occurs in at least 2 models. “Clean” tasks are those where no model flipped P→F. Statistic
Ratio
F→P
P→F
Ratio
Lexical Vagueness (LV) mutations Qwen2.5-Coder-32B CodeLlama-34B DeepSeek-33B StarCoder2-15B Codestral-22B Qwen2.5-Coder-7B CodeLlama-7B DeepSeek-6.7B GPT-4o-mini Claude-Sonnet-4
45 36 36 33 45 52 18 32 62 52
48 41 40 28 48 35 17 47 62 64
0.94 0.88 0.90 1.18 0.94 1.49 1.06 0.68 1.00 0.81
7 16 15 7 9 5 11 7 1 1
9 26 21 26 21 16 24 27 13 12
0.78 0.62 0.71 0.27 0.43 0.31 0.46 0.26 0.08 0.08
Average
41.1
43.0
0.99
7.9
19.5
0.40
Under-Specification (US) mutations Qwen2.5-Coder-32B CodeLlama-34B DeepSeek-33B StarCoder2-15B Codestral-22B Qwen2.5-Coder-7B CodeLlama-7B DeepSeek-6.7B GPT-4o-mini Claude-Sonnet-4
17 10 20 24 28 27 17 32 65 54
31 30 29 12 33 29 14 35 105 69
0.55 0.33 0.69 2.00 0.85 0.93 1.21 0.91 0.62 0.78
4 6 8 3 4 4 7 7 1 3
24 21 25 27 27 28 20 32 17 19
0.17 0.29 0.32 0.11 0.15 0.14 0.35 0.22 0.06 0.16
Average
29.4
38.7
0.89
4.7
24.0
0.20
exercises the generated code thoroughly and that the observed improvements reflect correctness gains.
LV
US
69 33 36 15 1
27 14 13 3 0
Task-level counts Tasks with F→P in ≥2 models Clean (P→F = 0) Mixed (P→F ≥ 1) Tasks with F→P in ≥3 models Tasks with F→P in ≥4 models
Model–task pair counts (consistent tasks only) Total F→P pairs Total P→F pairs Net F→P / P→F ratio
154 50 +104 3.08
57 16 +41 3.56
Dominant change/constraint type (tasks, % of section) Terminology / variable rename Task description rephrasing I/O description change Domain noun substitution Value range removed Size bound removed Input format constraint removed Uniqueness / ordering / other
33 (48%) 27 (39%) 6 (9%) 2 (3%) — — — —
— — — — 12 (44%) 8 (30%) 4 (15%) 3 (11%)
Takeaway. Under specified prompts do not uniformly degrade code generation. On LiveCodeBench, 47 problems improve consistently and exclusively across multiple models when vagueness or under-specification is introduced, revealing that for a non-trivial subset of real-world problems, over-specification silently misleads models in ways not apparent from the prompt itself.
Akli et al.
Table 6: Test-suite coverage of passing solutions for F→P and P→F flip cases on LiveCodeBench (10 models, US/LV mutations). Line coverage: fraction of executable statements reached by at least one test. Branch coverage: fraction of conditional branch outcomes (true/false arms) exercised. LV
US
All
Metric
F→P
P→F
F→P
P→F
F→P
P→F
𝑁 Mean line cov. (%) Median line cov. (%) Mean branch cov. (%)
411 91.8 95.9 83.7
429 92.7 95.8 84.9
294 90.8 95.7 83.0
387 90.8 94.4 83.6
1001 91.2 95.7 83.4
1154 91.9 95.5 84.5
5.3
Causes of improvement
Having established that a consistent subset of LCB problems improves under vagueness and under-specification, we now ask what linguistic mechanisms drive these improvements and what they reveal about how models process specifications. Under lexical vagueness, improvements are driven primarily by disrupting over-fitted retrieval cues. Table 7 shows that the two dominant mechanisms in LCB under LV are variable renaming that breaks overfitted retrieval (27%) and terminology generalization that activates better algorithmic patterns (27%). In both cases, the original prompt contains domain-specific vocabulary or identifier names that prime the model toward a memorized but incorrect solution; replacing them with vaguer alternatives forces the model to reason from the problem structure instead (Figure 4). A further 13% of cases involve function renaming, in which rewriting under a new signature incidentally corrects parsing or boundary bugs as a side effect. The remaining cases involve softening algorithmic language, which frees the model to select simpler correct strategies over over-fitted templates (10%), rephrasing input descriptions, which triggers regeneration of parsing code and corrects I/O handling bugs (10%), and constraint rewording that disambiguates boundary conditions left unclear in the original phrasing (8%). Under under-specification, constraint removal disrupts two distinct failure modes. Table 8 identifies constraint-triggered wrong input parsing as the dominant mechanism (39%): specific constraint lines prime the model to write LeetCode-style I/O handling incompatible with the actual stdin format, and removing them forces a correct rewrite. The second mechanism, constraint-anchored wrong algorithm (35%), is equally systematic: numerical bounds activate memorized algorithm shortcuts that are misapplied to the problem at hand, and removing the bound allows the model to select a general correct strategy (Figure 1). Together, these two mechanisms account for 74% of US improvement cases. In a further 17% of cases, removing a bound simplifies the problem sufficiently that the model defaults to a fully general implementation, forward DP, full BFS, direct enumeration, that is structurally harder to get wrong than the optimized solution the original constraint had scoped it toward. These findings carry direct implications: for practitioners, reducing specification precision can sometimes improve generation; for benchmark designers, completeness of specification does not guarantee prompt quality; and for model developers, sensitivity to retrieval cues points to a generalisation gap that scale alone does not resolve.
Takeaway. Code generation improvements occur when original prompts contain retrieval cues, domain terms, identifier names, or constraint lines that trigger memorized but incorrect solution strategies. Removing or softening these cues forces models to reason from the problem structure, often arriving at correct solutions via simpler algorithms. Specification completeness does not guarantee prompt quality, and retrieval-cue sensitivity reveals a generalization gap that model scale alone does not resolve.
6 Related work 6.1 LLM-based code generation and evaluation Code generation benchmarks have evolved from static, singlelanguage test suites to more rigorous and diverse evaluation frameworks. Earlier benchmarks such as HumanEval [9] and MBPP [2] established the standard paradigm for evaluating functional correctness using unit-test pass rates (e.g., pass@k). However, subsequent work has shown that these evaluations can overestimate the model’s performance. For example, EvalPlus [22] increases HumanEval with substantially more test cases, demonstrating that increasing test coverage can significantly reduce reported scores and expose weaknesses in generated solutions. More recent benchmarks have sought to address additional limitations of earlier evaluations. LiveCodeBench [17] mitigates benchmark contamination by continuously introducing new competitiveprogramming problems, while BigCodeBench [45] extends evaluation to more complex, library-intensive tasks that require diverse API usage and more realistic programming workflows. Alongside these benchmarks, the ecosystem of code-oriented language models has expanded rapidly [3]. Open-weight models include Code Llama [33], StarCoder2 [23], DeepSeek-Coder [13], and Qwen2.5-Coder [15], while proprietary systems such as GPT-5 [28] and Claude Code [1] represent the state of the art in commercial deployments. Despite the growing diversity of benchmarks and models, evaluation practices remain largely focused on correctness under clean, well-specified prompts, leaving model behavior under realistic prompt imperfections unexplored.
6.2
Robustness of code LLMs to prompt permutations
The most closely related line of work studies how prompt perturbations affect code generation quality. ReCode [37] introduced over 30 semantic-preserving transformations, variable renaming, dead-code insertion, and docstring reformatting on HumanEval [9] and MBPP [2], revealing substantial performance degradation. Mastropaolo et al. [24] show that semantically equivalent Javadoc paraphrases alter Copilot output in nearly half of cases. NLPerturbator [6] extended this to 18 natural-language variation categories derived from real developer data, reporting drops of up to 21.2%, while Rabbi et al. [31] broadened the analysis to multiple programming languages. Sclar et al. [34] demonstrated that formatting changes alone cause accuracy swings of up to 76%. More recently, Larbi et al. [21] examined ambiguous, incomplete, and contradictory task descriptions in a restricted set of models and reported large performance drops for all models and defects.
When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation
Table 7: Root-cause analysis of LV prompt mutations inducing behavioral change in Qwen-7B (incorrect → correct) on LiveCodeBench (𝑛 = 52). Root Cause
n
%
Interpretation
Variable rename breaks overfit
14
27
Renaming key variables (e.g. s→seq, n,x→a,b) prevents retrieval of a memorised incorrect solution, forcing reasoning from the problem description.
Terminology generalisation triggers better patterns
14
27
Replacing domain nouns with generic terms (“graph”→“network”, “boxes”→“containers”) activates canonical algorithmic templates (BFS, knapsack, set-union) over problem-specific shortcuts.
Function rename changes approach
7
13
An explicit function signature (e.g. gather_pairs(n)) anchors code structure, incidentally fixing ancillary bugs (wrong parsing, boundary errors) by forcing a full rewrite.
Vague algorithm frees model
5
10
Softening algorithmic language (“find minimum”→“smallest possible”) lets the model choose a simpler correct strategy (brute-force, enumerate-all) over an overfitted template (broken DP, wrong greedy).
Input-format change fixes parsing
5
10
Rephrasing the input description triggers regeneration of parsing code, correcting token-iteration bugs, index offsets, or sys.stdin.read() misuse.
Constraint rephrasing clarifies
4
8
Explicit constraint wording (“divisors smaller than 𝑥”, “no empty spots”) disambiguates boundaries left unclear in the original, preventing off-by-one errors.
Other (stochastic/unclear)
3
6
Surface-form change shifts output stochastically onto a correct solution; no single linguistic feature is identifiable as responsible.
LV mutation on LiveCodeBench/abc341_b — Maximum Resource Accumulation (orig. “Currency Exchange”) Original prompt (excerpt): There are N countries . Takahashi has A_i units of currency of country i. - Pay S_i units of currency i , gain T_i units of currency (i +1). Print the maximum possible units of currency of country N. Key LV substitutions: " countries " -> " entities ", " currency of country i" -> " resource for entity i",
Original × WA (expected 5, got 3) # Backward sweep : n -1 down to 0 for i in range (n -2 , -1, -1): s , t = exchanges [i] num = a[i] // s a[i] -= num * s a[i +1] += num * t # no cascade print (a[n -1])
" pay / gain " -> " give up / acquire "
Mutated ✓ Passes
# Forward DP : accumulate left to right dp = a [:] for i in range (n -1): s , t = exchanges [i] num = dp [i] // s dp [i] %= s dp [i +1] += num * t # cascades print ( dp [n -1])
Why: “Currency” carries a financial frame in which conversion chains are naturally thought of in reverse (as in FX liquidation). The model applied a backward sweep from 𝑖=𝑁 −2 down to 0, missing cascades: resources accumulated at step 𝑖 cannot propagate forward, yielding 3 instead of 5. The neutral term “resource” carries no directional connotation; the model defaulted to a forward DP, correctly propagating values left-to-right.
Figure 4: Example of an LV mutation improving performance by neutralizing a domain-specific vocabulary cue. While these studies collectively establish that code LLMs are sensitive to prompt perturbations, they share a common limitation: all evaluate robustness exclusively on minimal, single-specification benchmarks and frame perturbations as purely degrading.
6.3
Prompt under-specification
A parallel line of work addresses the consequences of ambiguous or incomplete specifications in code generation. ClarifyGPT [27] detects ambiguous requirements and improves Pass@1 through targeted clarification, while TiCoder [11] proposed test-driven interactive clarification to reduce intent mismatch. SpecFix [18] automated ambiguity repair, finding that 43.58% of benchmark descriptions contain actionable ambiguity. These works establish that specification imperfection is pervasive and consequential, but they focus exclusively on repairing ambiguity, treating all specification degradation as harmful. The assumption that more specification is
always better has been questioned in adjacent work. Yang et al. [40] found that over-specified prompts can overwhelm general-purpose LLMs, identifying over-specification as an anti-pattern. Döderlein et al. [10] observed that removing prompts entirely can sometimes improve code generation, and Break-the Chain [5] showed that narrative reframing of coding problems improves accuracy by up to 35.3% for some models. However, these observations are scattered: none provides a controlled comparison across benchmark types, none quantifies improvement consistency across models, and none identifies the linguistic mechanisms responsible. Our work provides the first systematic account of when and why reducing specification precision improves code generation, grounded in root-cause analysis across 47 consistently improved problems.
Akli et al.
Table 8: Taxonomy of underspecification (US) mutations that induced a Fail→Pass flip on Claude-Sonnet-4 / LiveCodeBench (54 instances). Rows describe the mechanism by which removing a constraint steered the model toward a correct solution. Root Cause
N
%
Constraint Types
Mechanism & Example
I/O format primed by constraint text
21
39
value_range (7), size_bound (5), input_format (4), uniqueness (3), other (2)
Seeing a constraint line (e.g. 1 ≤ nums[i] ≤ 10ˆ5) primed the model to use LeetCode-style parsing (input().split()) rather than bracket-formatted competitive-programming stdin. Removing the line eliminated the priming cue; the algorithm itself was never wrong.
Constraint anchored wrong algorithm
a
19
35
value_range size_bound put_format (1)
(11), (5), in(2), other
A numeric or categorical bound triggered a memorised shortcut that did not fit the problem: e.g. Ci ≤ 10ˆ9 activated a star-graph diameter formula instead of a two-BFS approach; M is prime invoked modular-inverse arithmetic instead of a general DP. Without the cue, the model fell back to the correct general algorithm.
Bound pushed model into a buggy optimisation
9
17
value_range (3), size_bound (2), input_format (2), other (2)
The constraint scoped the model toward an optimised solution with a structural bug (wrong DP dimension order, premature greedy pruning). Freed from the bound, the model chose a simpler, fully general implementation (plain forward DP, full enumeration) that was harder to get wrong.
No identifiable causal link
4
7
size_bound put_format (1)
The deleted text was incidental (a minor size cap or formatting note); no direct linguistic trigger is apparent. The improvement is best attributed to re-sampling variance.
Constraint triggered overfit template
1
2
value_range (1)
7
(2), in(1), other
Threats to Validity
Our mutations are produced automatically by gpt-5-mini, which may shape them in ways that reflect the generator’s own tendencies [36, 42]. Although the judge and manual checks confirm high quality, some variants may still be less natural than what real developers would introduce [20]. HumanEval is known to overlap with model training data [25, 41], so some models may recall solutions rather than reason from the prompt, making our measured drops a lower bound. Both benchmarks are Python-only, and results may not hold for other languages [31] or for tasks that require heavy library use [44], where the specification structure plays a different role. We use Pass@1 with greedy decoding to keep results reproducible, but this hides the variance that comes with sampling [29, 35] and underestimates what models can do with multiple attempts [8]. Small individual flips may therefore reflect output variance rather than true sensitivity to the prompt change. Finally, our exploratory analysis [12, 32] forms a first attempt to identify what impacts the LLM-based code generation correctness. Therefore, the observed improvements could, in principle, be coincidental or reflect confounding factors. We mitigate this by requiring consistency across multiple models, verifying test-suite coverage of flipped solutions, and relying on human annotation, but causal confirmation requires future controlled experiments. To our knowledge, this is the first study to examine these mechanisms at this level of granularity; as an exploratory study, it is, by definition, hypothesis-generating rather than hypothesis-confirming, and our findings should be interpreted accordingly.
Domain vocabulary in the constraint surface-matched a memorised but incorrect solution template. Removing it allowed the model to reason from problem structure rather than retrieve a mismatched pattern.
8
Conclusion
This study investigated how prompt structure, task complexity, and specification richness relates to LLM robustness to prompt mutations. Our results show that robustness is not an intrinsic model property but an artifact of prompt structure: the same mutant types that cause substantial Pass@1 drops on HumanEval (US -11.8%, LV -7.1%) produce near-zero net effects on LiveCodeBench, where multi-layered specifications provide redundant specification layers. Crucially, this near-zero effect masks two opposing forces of comparable magnitude, improvements and degradations, which cancel rather than being absent. On 47 consistently improved LCB problems, root-cause analysis identified retrieval-cue disruption as the dominant mechanism: domain-specific vocabulary, identifier names, and constraint lines prime models toward memorized but incorrect solutions, and their removal forces reasoning from problem structure. These findings suggest that practitioners should be careful with over-specified prompts that may silently mislead models. Benchmark designers may consider that minimal-specification benchmarks probably do not generalize to richer task formats, and model developers should consider that words triggering retrieval cues may well hamper the effectiveness of the model reasoning.
When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation
References [1] Anthropic. 2024. Claude 4 Model Card. Anthropic Documentation. https: //www.anthropic.com [2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732 (2021). [3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS). 1877–1901. [4] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel HerbertVoss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium. 2633–2650. [5] Yiorgos Charalambous et al. 2025. Break-The-Chain: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation. arXiv preprint arXiv:2506.06971. [6] Junkai Chen, Zhenhao Li, Xing Hu, and Xin Xia. 2026. NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations. ACM Trans. Softw. Eng. Methodol. 35, 4, Article 89 (March 2026), 20 pages. doi:10.1145/3745764 [7] Mark Chen, Jerry Tworek, Heewoo Jun, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). [8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). [9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [10] Jean-Baptiste Döderlein, Nguessan Hermann Kouadio, Mathieu Acher, Djamel Eddine Khelladi, and Benoît Combemale. 2025. Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic? Journal of Systems and Software 230 (2025), 112562. doi:10.1016/j.jss.2025.112562 [11] Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K Lahiri. 2024. Llm-based test-driven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering 50, 9 (2024), 2254–2268. [12] Ángel González-Prieto, Jorge Pérez, Jessica Díaz, and Daniel López-Fernández. 2023. Reliability in Software Engineering Qualitative Research through InterCoder Agreement. J. Syst. Softw. 202 (2023), 111707. [13] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wenjie Zhang, Wenhu Chen, Kexin Bi, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming—The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024). [14] Asma Hamidi, Ahmed Khanfir, and Mike Papadakis. 2025. Intent-Based Mutation Testing: From Naturally Written Programming Intents to Mutants. In IEEE International Conference on Software Testing, Verification and Validation, ICST 2025 - Workshops, Naples, Italy, March 31 - April 4, 2025. IEEE, 347–357. doi:10.1109/ICSTW64639.2025.10962508 [15] Binyuan Hui, Jian Yang, Zeyu Cui, et al. 2024. Qwen2.5-Coder Technical Report. arXiv preprint arXiv (2024). [16] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186 [cs.CL] https://arxiv.org/ abs/2409.12186 [17] Naman Jain, Jiayi Han, Alex Gu, William Yang, Yiming Li, Koushik Sen, and Ion Stoica. 2024. LiveCodeBench: Holistic and Contamination-Free Evaluation of Large Language Models for Code. arXiv preprint arXiv:2403.07974 (2024). [18] Haoxiang Jia, Robbie Morris, He Ye, Federica Sarro, and Sergey Mechtaev. 2025. Automated Repair of Ambiguous Problem Descriptions for LLM-Based Code Generation. arXiv preprint arXiv:2505.07270 (2025). [19] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2026. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology 35, 2 (2026), 1–72. [20] René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are Mutants a Valid Substitute for Real Faults in Software Testing?. In FSE. 654–665. [21] Maya Larbi, Amal Akli, Mike Papadakis, Rihab Bouyousfi, Maxime Cordy, Federica Sarro, and Yves Le Traon. 2025. When prompts go wrong: Evaluating code model robustness to ambiguous, contradictory, and incomplete task descriptions. arXiv preprint arXiv:2507.20439 (2025). [22] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems 36 (NeurIPS). https://proceedings.neurips.cc/paper_files/paper/ 2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html
arXiv:2305.01210. [23] Anton Lozhkov, Raymond Li, Vaibhav Chaudhary, et al. 2024. StarCoder2 and The Stack v2: The Next Generation. arXiv preprint arXiv:2402.19173 (2024). [24] Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. 2023. On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 2149–2160. doi:10.1109/ICSE48619.2023.00181 [25] Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, and Matthias Gallé. 2024. On Leakage of Code Generation Evaluation Datasets. In Findings of EMNLP. 13215–13223. [26] Mistral AI. 2024. Codestral: Hello, World! Technical report. https://mistral.ai/ news/codestral [27] Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2332–2354. [28] OpenAI. 2025. GPT-5 Technical Report. https://openai.com. Accessed: 2026. [29] Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. 2025. An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Trans. Softw. Eng. Methodol. 34, 2 (2025), 42:1–42:28. [30] Malintha Perera, Aldeida Aleti, Chunyang Chen, and Chakkrit Tantithamthavorn. 2023. Revisiting the Impact of Natural Language Descriptions on the Correctness of Code Generation. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). [31] Fazle Rabbi, Zishuo Ding, and Jinqiu Yang. 2025. A Multi-Language Perspective on the Robustness of LLM Code Generation. arXiv preprint arXiv:2504.19108 (2025). https://arxiv.org/abs/2504.19108 [32] Paul Ralph, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, Michael Felderer, et al. 2021. Empirical Standards for Software Engineering Research. arXiv preprint arXiv:2010.03525 (2021). [33] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, et al. 2023. Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950 (2023). [34] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design. In Proceedings of the International Conference on Learning Representations (ICLR). [35] Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2025. The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism. In NAACL. 4195–4206. [36] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large Language Models are not Fair Evaluators. In ACL. 9440–9450. [37] Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. 2023. ReCode: Robustness Evaluation of Code Generation Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). 13234–13274. [38] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS). [39] Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv preprint arXiv:2302.11382 (2023). [40] Chenyang Yang, Yike Shi, Qianou Ma, Michael Xieyang Liu, Christian Kästner, and Tongshuang Wu. 2025. What Prompts Don’t Say: Understanding and Managing Underspecification in LLM Prompts. arXiv:2505.13360 [cs.CL] [41] Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. 2023. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. arXiv preprint arXiv:2311.04850 (2023). https://arxiv.org/ abs/2311.04850 [42] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36. 46595–46623. [43] Kaijie Zhu et al. 2023. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. arXiv preprint arXiv:2306.04528 (2023). [44] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2025. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In ICLR. [45] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon
Akli et al.
Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, WenDing Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro
Von Werra. 2025. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In Proceedings of the International Conference on Learning Representations (ICLR). arXiv:2406.15877.