Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis Amal Akli
University of Luxembourg Luxembourg [email protected]
arXiv:2604.24703v1 [cs.SE] 27 Apr 2026
Maxime Cordy
University of Luxembourg Luxembourg [email protected]
Abstract Large language models are widely used for code generation, yet they rely on an implicit assumption that the task descriptions are sufficiently detailed and well-formed. However, in practice, users may provide defective descriptions, which can have a strong effect on code correctness. To address this issue, we develop SpecValidator, a lightweight classifier based on a small model that has been parameter-efficiently finetuned, to automatically detect task description defects. We evaluate SpecValidator on three types of defects, Lexical Vagueness, Under-Specification and Syntax-Formatting on 3 benchmarks with task descriptions of varying structure and complexity. Our results show that SpecValidator achieves defect detection of F1 = 0.804 and MCC = 0.745, significantly outperforming GPT-5-mini (F1 = 0.469 and MCC = 0.281) and Claude Sonnet 4 (F1 = 0.518 and MCC = 0.359). Perhaps more importantly, our analysis indicates that SpecValidator can generalize to unseen issues and detect unknown Under-Specification defects in the original (real) descriptions of the benchmarks used. Our results also show that the robustness of LLMs in task description defects depends primarily on the type of defect and the characteristics of the task description, rather than the capacity of the model, with Under-Specification defects being the most severe. We further found that benchmarks with richer contextual grounding, such as LiveCodeBench, exhibit substantially greater resilience, highlighting the importance of structured task descriptions for reliable LLM-based code generation.
Keywords Code generation, LLMs, prompt engineering, Robustness, Benchmarking
1
Introduction
Large language models (LLMs) such as Codex [5], AlphaCode [21], Code Llama [35], and Qwen2.5 Coder [14], along with related LLM agents [1, 31], increasingly support developers across a broad spectrum of programming tasks, such as code generation [18]. Previous work demonstrates their practical impact: AI-based coding tools boost developer productivity [32]. In parallel, the evaluation methodologies for these systems have evolved substantially. Benchmarks such as HumanEval [5] and MBPP [2] established the unit-test pass rate as a standard evaluation
Mike Papadakis
University of Luxembourg Luxembourg [email protected]
Yves Le Traon
University of Luxembourg Luxembourg [email protected] paradigm. However, subsequent work has revealed important limitations. EvalPlus [22] shows that increasing the test coverage significantly reduces the reported performance, highlighting the fragility of previous assessments. More recently, LiveCodeBench [16] addressed additional concerns by continuously updating problem sets to mitigate data contamination. In addition, it introduces richer problem specifications, incorporating explicit numerical constraints, formal input–output contracts, and illustrative examples alongside natural language descriptions. Interestingly, the above studies evaluate the generation of code on well-formed and sufficiently described task descriptions. However, in practice, natural language descriptions are inherently imprecise and may be subject to several description issues, similar to specification defects that appear in software specifications [19, 28, 41, 42]. Therefore, task descriptions can include vague terminology, omit edge-case constraints, or may contain surface-level noise, such as typographical and formatting errors, all of which can drastically affect the correctness of the generated code [17, 19, 45, 48]. To address this issue, we developed SpecValidator, a simple classifier to automatically detect such task description defects prior to code generation. SpecValidator is a lightweight LoRA-fine-tuned predictor trained on a set of artificially injected defects. In particular, we define and apply three defect types — Lexical Vagueness (LV), Under-Specification (US), and Syntax and Formatting (SF) — reflecting the key issues task descriptions may have, based on which we fine-tune our predictor and check its ability to detect them. We evaluate 10 LLMs spanning from small (6–7B) to large opensource (15–34B) and state-of-the-art reasoning models across 3 benchmarks and found significant degradation in the models’ code correctness. Interestingly, our results show that the structural characteristics of the task descriptions play an important role on the correctness of the generated code. Under-Specification causes the most significant drops, of up to 15.3%, with no model being robust, while benchmarks with richer contextual grounding, such as LiveCodeBench, are substantially more resilient. Lexical Vagueness produces moderate and benchmark-dependent degradation, whereas surface formatting issues have a negligible effect. SpecValidator shows strong defect detection performance with F1 = 0.804 and MCC = 0.745, outperforming significantly fewshot GPT-5-mini (F1 = 0.462 and MCC = 0.276) and Claude Sonnet 4 (F1 = 0.557 and MCC = 0.412). Perhaps more importantly, by applying SpecValidator on the original set of task descriptions, we
Akli et al.
reveal that these surprisingly contain Under-Specification defects (confirmed via thorough manual inspection). This demonstrates the ability of SpecValidator to detect real specification defects. Thus, in a sense we show that SpecValidator generalizes beyond the artificial defects we used for the fine-tuning, with a precision of 72%, correctly identifying 72% of the real unseen descriptions as defective. The main contributions of this paper are: • A large-scale analysis of 10 models and 3 benchmarks, showing that defects’ impact is shaped both by defect type and task description structure. • SpecValidator A lightweight defect detector, a Qwen2.5Coder-1.5B LoRA fine-tune model, achieving F1 = 0.804 and MCC = 0.745, outperforming both GPT-5 mini and Claude Sonnet 4. SpecValidator is capable of detecting real description defects in the benchmarks we use. • A publicly available dataset of 6,573 defective task description instances, supporting replication and future research on task description and benchmark design.
2 Related Work 2.1 LLM-based code generation and evaluation The HumanEval [6], MBPP [2] and EvalPlus [22] benchmarks evaluate functional correctness using test pass rates (e.g. pass@k). LiveCodeBench [16] mitigates benchmark contamination by continuously introducing new competitive-programming problems, while BigCodeBench [49], and Siddiq et al. [38] extend evaluation to more complex, library-intensive tasks that require diverse API usage and more realistic programming workflows. Open-weight models include Code Llama [35], StarCoder2 [24], DeepSeek-Coder [10], and Qwen2.5-Coder [14], while proprietary systems such as GPT-5 [31] and Claude Code [1] represent the state of the art in commercial deployments. Despite the growing diversity of benchmarks and models, evaluation practices remain largely focused on correctness under well-specified task descriptions, leaving model behavior under realistic description defects unexplored.
2.2
2.4
LLM robustness on prompt perturbations
ReCode [43] introduced over 30 semantic-preserving task description transformations (e.g., variable renaming, dead-code insertion, and docstring reformatting), revealing a substantial but uneven performance degradation of the correctness of the generated code. Mastropaolo et al. [25] showed that semantically equivalent paraphrases of Javadoc can alter the GitHub Copilot output in nearly half of the cases, affecting correctness 28% of the cases. Subsequent work broadened the range of possible perturbations by showing that single token perturbations lead to semantically different code versions [11]. NLPerturbator [4] proposed 18 categories of natural-language variations from real developer data, while Rabbi et al. [33] extended the analysis to multiple programming languages. Shirafuji et al. [37] demonstrated that code LLMs are highly sensitive to superficial modifications of problem descriptions. Sclar et al. [36] highlighted the sensitivity of LLMs to prompt formatting, showing that minor changes such as whitespace or casing can impact code correctness, and PromptRobust[48] showed contemporary LLMs suffer up to 39% average performance drops from word-level perturbations alone [46, 48]. The closest work to ours is that of Larbi et al. [19], which found that code correctness is strongly affected by ambiguous, incomplete, and contradictory task descriptions. However, that work did not propose ways to detect such defects. Overall, our work goes a step ahead by making a controlled analysis of three defect categories (lexical vagueness, under-specification, and syntax/formatting errors) across 10 models, it develops a lightweight LoRA-based defect classifier evaluated on both artificial (mutants) and real defects.
Prompt engineering for code tasks
Li et al. [20] proposed Structured Chain-of-Thought (SCoT) prompting, which decomposes generation through program-structureaware intermediate steps [44], yielding consistent improvements over standard prompting on HumanEval and MBPP. Test-driven feedback loops [9] ask users to refine the task description using feedback from tests generated from the original description, using a multi-turn LLM dialog. Self-Debugging [7] teaches models to self-identify and correct errors in their own outputs via executionguided feedback. These studies focus on the generation of correct code from well-specified, sufficiently detailed descriptions and excluding potential defects.
2.3
using today’s LLMs. Jia et al. [17] and Tian et al. [39] train models to automatically repair contradictory descriptions. This approach is somewhat orthogonal to our detection one, as it aims only at repairing issues. Nevertheless, Jia et al. are using contrastive specification inference, which may not perform well with the under-specified defects we study.
Detecting description defects
ClarifyGPT [30] and Larbi et al. [19], and Wu et al. [45] use LLMs to detect defective task descriptions. These approaches have been evaluated on relatively simple cases, such as HumanEval and MBPP, and on a limited number of defects and types. The baselines used in this work represent the performance of these approaches when
3
Dataset Construction
Figure 1 shows the process of dataset construction. Starting from three well-known code-generation benchmarks, we apply three complementary defect-injection strategies to produce our benchmark. Data quality is assessed at two levels: automated evaluation by an independent LLM judge across all 6,573 mutated instances, and manual inspection of a stratified random sample for groundtruth calibration.
3.1
Benchmarks
We select three code-generation benchmarks that cover a range of task types, from structured function completion to natural-language descriptions to full programming problems. This variety helps us evaluate model performance across different levels of difficulty, description styles, and reasoning requirements. Table 1 records the key characteristics of the benchmarks we use. HumanEval [6] uses structured docstrings with typed function signatures; MBPP [2]
Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis
Figure 1: Dataset construction. Defects are generated with GPT-5-mini with defect quality being scored by a Qwen-32B evaluator. Defective descriptions are then manually verified by 3 human reviewers. provides short, informal descriptions in natural-language with minimal scaffolding; and LiveCodeBench [16] provides competitiveprogramming problem statements with explicit numerical bounds and formal input/output contracts. All benchmarks are for Python.
3.2
Generating defective task descriptions
We construct defective task descriptions by systematically mutating original descriptions. Table 3 shows examples of our mutations. We generate three defects per task description (one per defect type). In cases where the original descriptions are too short or have no constraint to delete, no defective variant is produced. Mutations are applied only to task descriptions: reference solutions, test cases, and evaluation harnesses are not modified, so any observed change (e.g. in Pass@1 metric) is solely attributable to the altered description. We generate defective descriptions with gpt-5-mini (via the OpenAI Batch). We prompt the model to mutate original descriptions using a predefined set of transformation rules. To ensure that the generated descriptions comply with our requirements, we use an LLM-as-a-judge to assign compliance scores. In case the scores are not above 85% for all defect types and benchmarks, we manually inspect the low-compliance defective descriptions and refine our prompt to gpt-5-mini until we reach at least this 85% compliance threshold. All our generation prompts are available in the replication package.1 . 3.2.1 Defect Types. Previous work has identified token perturbations and lexical ambiguity as a source of performance variance in code generation models [4, 19, 43]. Missing or under-specified requirements can also cause models to generate code that is syntactically correct but semantically incorrect [9, 11, 17, 30]. Surface-level formatting noise, while seemingly superficial, has been shown to induce performance swings in open-source LLMs [36, 43]. Together, these three defect classes (lexical vagueness, under-specification, and syntactic/formatting corruption) represent orthogonal failure modes that cover the main ways a natural-language specification defect can appear in the user prompts. Table 3 provides representative examples of each defect type applied to task descriptions from all three benchmarks. 1 https://github.com/Amal-AK/detecting_prompt_defects
3.2.2
Mutation Operators.
Lexical Vagueness (LV). LV mutations replace precise, task-relevant vocabulary with semantically broader or less informative alternatives, reducing the specificity of the task without removing any explicit constraint. This mirrors the paraphrasing and synonymsubstitution perturbations studied by Wang et al. [43] and Chen et al. [4]. Concretely, an LV mutation consists of one to five of the following transformations: • Replace domain-specific terms with broader synonyms (e.g., sort → arrange). • Weaken action verbs to less precise ones (e.g., insert → put). • Rename function and parameter identifiers with less informative synonyms (e.g., delimiter → filler). • Generalize or remove type annotations. • Make multiple lexical changes, compounding the ambiguity. Under-Specification (US). US mutations remove one explicit constraint from the task description, producing a description that remains grammatically well-formed but is semantically under-specified. This defect type is motivated by empirical findings that incomplete requirements cause most of the intent mismatch in LLM-based code generation [19]. To ensure under-specified defects, we restrict deletion to the following elements of the description: • Numeric bounds or threshold values. • Ordering or sorting rules. • Input preconditions or domain restrictions. • Output edge-case behavior. • Formula or behavioral pinning constraints. • Error or exception handling requirements. • Output type or format constraints. Syntax and Formatting (SF). SF mutations introduce surface-level noise into the task description without altering its semantic content, targeting the susceptibility of LLMs to low-level textual corruption [36]. Sclar et al. [36] demonstrated that format variations alone can have a significant impact, motivating this defect type. Each SF mutation applies one of the following corruption types: • Token-level corruption (character transposition, typos). • Delimiter, bracket, or colon corruption.
Akli et al.
Table 1: Overview of the three benchmarks used in our experiments
3.3
Benchmark
Size
Task style
Complexity
Language
Evaluation
HumanEval
164
Function completion (docstring)
Easy–Medium
Python
Unit tests
MBPP
974
NL description of short programs
Easy
Python
Unit tests
LiveCodeBench
1,055
Competitive programming(full problem statement)
Easy–Hard
Python
stdin/stdout
• Indentation or whitespace layout corruption. • Example-block formatting corruption.
Table 2: Evaluating mutations by judge (Qwen2.5-Coder-32BInstruct). All values are proportions (0–1).
Defect validation
Dataset
Mutation
𝑁
Compliance
Naturalness
HumanEval
LV SF US
163 164 160
0.99 1.00 0.89
0.99 0.99 0.81
MBPP
LV SF US
974 974 974
1.00 0.98 0.88
0.99 1.00 0.82
LiveCodeBench
LV SF US
1054 1055 1055
1.00 1.00 0.85
0.96 0.99 0.61
We performed an automated assessment of the generated defects using an independent LLM judge followed by manual analysis of a stratified random sample. Judge model: We select Qwen2.5-Coder-32B-Instruct [15] as the judge model. Three properties motivate this choice: (i) it is a strong open-source, code-specialized model that is categorically different from the generation model (gpt-5-mini), reducing the risk of generator-judge collusion common in LLM-as-a-judge setups [47]; (ii) it is deployed locally on our own infrastructure, ensuring full reproducibility without dependence on external APIs; and (iii) its instruction following capability is sufficient for the structured binaryjudgment protocol described below. The model is run at float16 precision with greedy decoding, and its output is constrained to a single JSON object ({"score": 0} or {"score": 1}). Defect evaluation criteria: All defective descriptions are evaluated on the basis of the following criteria: • LV compliance: Did the mutation replace specific terms, variable names, or descriptions with vaguer counterparts without changing the overall task? • SF compliance: Does the defective description contain only typographical or formatting errors compared to the original? • US compliance: Is exactly one requirement, condition, or detail missing in the defective description compared to the original one? • Naturalness: Does the defective description read like a natural description a real user could write? Defect evaluation results: Table 2 reports compliance and naturalness scores for each benchmark–defect pair. The LV and SF mutations achieve near-perfect compliance (0.98–1.00) and high naturalness (0.96–1.00) in all data. US shows lower compliance (0.85–0.89), mainly due to structural constraints in shorter MBPP prompts and stricter specification structures in LiveCodeBench. Naturalness remains high for LV and SF, but drops for US on LiveCodeBench (0.61), where removing constraints can produce unusually terse problem statements. Manual Validation: To assess the reliability of the automated judge, we manually annotated a stratified random sample of 100 mutants of all types (LV, US, SF) and benchmarks (HumanEval, MBPP, LiveCodeBench), sampling roughly 11 samples per benchmark–mutation pair. Three researchers independently evaluated each instance using the same compliance and naturalness criteria as the LLM judge, with disagreements resolved by majority vote. Human annotators rated 86% of instances as natural and 97% as compliant, closely matching the judge’s assessments.
4 Study Design 4.1 Research questions Developers typically write imperfect descriptions: they may use imprecise vocabulary, omit edge cases, or write noisy statements. The robustness of code LLMs to such issues remains underexplored and there is also no systematic infrastructure to detect or mitigate these defects. Our work explores the following questions: RQ1 (Impact of defects): How lexical vagueness, underspecification, and syntactic noise affect generated code correctness across benchmarks of varying complexity? To answer this question, we evaluate 10 code-generation LLMs covering 3 model sizes: 3 small models (6–7B), 5 medium-to-large models (15–34B), and 2 reasoning models (GPT-5-mini and Claude Sonnet 4). We conduct experiments on 3 widely used benchmarks: HumanEval, MBPP, and LiveCodeBench. For each benchmark, we introduce 3 types of defect (US, LV, SF). We measure the performance of each model in generating correct code using Pass@1 metric. We examine to what extent robustness varies with model scale, benchmark complexity, and defect type. RQ2 (Defect detection): Can a small fine-tuned model detect description defects as well as large model baselines? We develop SpecValidator, an approach based on small, locally deployable classifier to detect dejects in code task descriptions. We aim to assess whether SpecValidator can match the performance of frontier models on four-class defect detection (clean, LV, US, SF). For the small model, we fine-tune Qwen2.5-Coder-1.5B under three regimes of increasing parameter budget: linear probing, LoRA, and full fine-tuning. We compare it with GPT-5-mini and Claude Sonnet 4 queried with zero shot and few shots. RQ3 (Defect detection on original prompts): Does SpecValidator detect (unseen) real-world defects? We want to assess whether SpecValidator can generalize to unseen defects in realworld specifications, beyond the synthetic defects we previously
Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis
Table 3: Representative examples of the three mutation strategies. US: removes a constraint; LV: paraphrases vocabulary and renames identifiers; SF: introduces typographical and formatting noise. Mutation
Prompt
HumanEval / 5 Original
def intersperse(numbers: List[int], delimeter: int) -> List[int]: “Insert a number ‘delimeter’ between every two consecutive elements of ‘numbers’.” »> intersperse([1,2,3], 4) ⇒ [1,4,2,4,3]
US
def intersperse(numbers: List[int], delimeter: int) -> List[int]: “Insert a number ‘delimeter’ into the input list ‘numbers’.” »> intersperse([1,2,3], 4) ⇒ [1,4,2,4,3] def place_between(items, filler): “Put a value ‘filler’ between neighboring entries of the sequence.” »> place_between([1,2,3], 4) ⇒ [1,4,2,4,3] def intersperse(numbers: List[int), delimter: int) -> List[int) ”’ Insert a number ‘delimter’ between every two consecutive elements . . .
LV SF MBPP / 12 Original
“Write a function to sort a given matrix in ascending order according to the sum of its rows.”
US
“Write a function to sort a given matrix according to the sum of its rows.”
LV
“Write a function to arrange a given matrix in ascending order according to the total of its rows.”
SF
“Write a function to sort a given matrix in ascdcending order according to the sum of its rows.”
LiveCodeBench / 2811 Original
“An array of distinct positive integers is called a k-avoiding array if no pair sums to 𝑘. Return the minimum possible sum of length 𝑛.” n=5,k=4⇒18 n=2,k=6⇒3
US
“An array of integers is called a k-avoiding array if no pair sums to 𝑘. Return the minimum possible sum of length 𝑛.” n=5,k=4⇒18 n=2,k=6⇒3
LV
“A list of different positive numbers where no two add up to 𝑘. Return the smallest total of length 𝑛.” n=5,k=4⇒18 n=2,k=6⇒3
SF
INPUT;{You are given two integres 𝑛 and 𝑘 . . . **INVALID****OCUMENT**}**
generated. To this end, we apply SpecValidator on the original descriptions of our benchmarks. and examine whether flagged instances reflect genuine defects in the ground truth descriptions. Once a defect is detected, its type is reported to the user in order to understand the issue with the description and update it accordingly.
4.2
Defect detection and SpecValidator
We classify task descriptions in one of four categories: Clean, LV, US, or SF. We evaluate two families of approaches. Proprietary LLM systems. We query GPT-5-mini and Claude Sonnet 4 with a structured prompt that presents the code generation task description and instructs the model to identify the defect type, if any. In addition to the zero-shot setting, we also evaluate a fewshot variant [3] where the model is provided with four labeled examples prior to classification. These serve as strong baselines that do not require training data and leverage the broad knowledge of frontier models.
The prompt used for Zero-Shot classification. (4 Examples are added in case of few-shot) You are a code-benchmark quality auditor. Classify the following coding prompt into exactly one of: LV – The prompt uses vague or imprecise wording. SF – The prompt contains syntax or formatting errors. US – The prompt is missing a constraint or condition. CLEAN – The prompt is complete and well-formed.
SpecValidator (fine-tuned classifier) We fine-tune Qwen2.5Coder-1.5B [14] as a sequence classifier on a labeled dataset constructed from our mutation approach. Each example consists of a task description (original or mutated) paired with its defect label. We train in three parameter budgets to assess the cost–accuracy trade-off: (i) linear probing, where only the classification head is trained and the backbone is frozen; (ii) LoRA adaptation, where lowrank updates are applied to attention projection layers as described in the following implementation section; and (iii) full fine-tuning, where all parameters are updated. The training set is composed of equal numbers of original and mutated description (8,766 samples in total). The dataset is split 80/10/10 into train, validation, and test sets, stratified by label. The finetuning is performed twice with two different seeds.
4.3
Selected LLMs
To study the impact of defects, we choose a diverse set of code generation models to ensure that our findings are not artifacts of a
Akli et al.
single architecture or training regime. Models are grouped into the following three categories/families: • Small open-source models (∼7B parameters): Qwen2.5Coder-7B [14], DeepSeek-Coder-6.7B [10], and CodeLlama7B [35]. These models are representative of deployable, resource-constrained settings and are widely used as baselines by the literature. • Large open-source models (15B–34B parameters): Qwen2.5Coder-32B [14], DeepSeek-Coder-33B [10], CodeLlama-34B [35], Codestral-22B [29], and StarCoder2-15B [24]. This tier allows us to examine whether larger capacity yields greater robustness to prompt mutations, independent of proprietary training data or alignment procedures. • Closed-source LLM-based systems: GPT-5 mini [31], and Claude Sonnet 4 [1]. These represent the current stateof-the-art in code generation and serve as upper-bound references. Their inclusion enables a direct comparison between frontier models and open-source alternatives under identical evaluation conditions. Collectively, this selection spans five model families, two provider categories (open-source and proprietary), and two scale regimes, enabling controlled cross-family and cross-scale comparisons.
4.4
Implementation
4.4.1 Metrics. Pass@1 [5] is used as the primary correctness metric, as it reflects the real-world scenario where a developer accepts the model’s first suggestion. This metric is the de facto standard in code generation evaluation [2, 5] and enables a direct comparison with previous work. For the four-class mutation classification task (LV, SF, US, CLEAN), we report the Matthews Correlation Coefficient (MCC) [26], macroF1, precision, recall, and accuracy. MCC is preferred over accuracy when there is class imbalance as it accounts for all entries in the confusion matrix. 4.4.2 Technical Details. Inference. All models are run with greedy decoding and temperature = 0), ensuring that the outputs are theoretically deterministic given a fixed prompt. Under this setting, each model produces exactly one outcome per problem, and re-running the same experiment yields identical results. The reasoning models (GPT and Claude) are queried via the OpenAI and Anthropic batch APIs, while open-source models are served locally on NVIDIA A100 GPUs in float16 precision. Generated solutions are extracted via markdown fenced code block parsing, with fallback to model-specific delimiters (e.g., CodeLlama’s [PYTHON] tags). Each solution is executed in an isolated subprocess with a 20-second timeout. The code and data are available. 2 LoRA fine-tuning. We apply LoRA [13] to the attention projection layers with rank 𝑟 =16 and 𝛼=32, which prior work has identified as an effective configuration for code model adaptation [8, 12, 13]. Training uses AdamW [23] with learning rate 2 × 10−4 , weight decay 10−4 , cosine decay scheduling, and 100 warmup steps. An effective batch size of 16 is achieved via gradient accumulation. Early stopping (patience = 4) is applied based on 2 Replication package: https://github.com/Amal-AK/detecting_prompt_defects
the validation performance to prevent overfitting. All finetuning experiments use both seeds 42 and 123456; the results are averaged.
5 Experimental Results 5.1 Impact of defects on code generation Table 4 summarizes the evaluation of ten code LLMs on three benchmarks (HumanEval, MBPP, and LiveCodeBench) under three types of mutation: Under-Specification (US), Lexical Vagueness (LV), and Syntactic/Formatting errors (SF). The results show that these mutation types induce substantially different levels of performance degradation in models and benchmarks, suggesting that description defects vary in their impact and should not be treated uniformly. Under-specification is consistently the most damaging mutation. As shown in Table 4, US mutations cause Pass@1 drops of 7.9–15.3% on HumanEval across all model scales. Even state-ofthe-art reasoning models are substantially affected: GPT-5-mini and Claude Sonnet 4, despite baselines of 96.3% and 95.7% respectively, each decline by 9.7%. A consistent pattern holds on MBPP, where US induces drops of 5.1–10.8% across all tiers. Figure 2 confirms that the US column is mostly red in both benchmarks and in all model families, with no model showing immunity. The uniformity of this degradation across architectures and scales indicates that the cause is intrinsic to the mutation itself: removing a constraint leaves the specification compatible with multiple plausible implementations, most of which are incorrect. Lexical vagueness produces benchmark-dependent, moderate drops. On HumanEval, LV mutations produce drops ranging from 1.3% (Qwen2.5-Coder-32B) to 12.2% (DeepSeek-Coder-6.7B), with CodeLlama-7B and StarCoder2-15B losing 7.9% and 11.6%, respectively (Table 4). The nearly negligible drop for Qwen2.5-Coder32B suggests that larger Qwen-family models are better calibrated to lexical synonymy. On MBPP, however, LV-induced drops shrink markedly across all tiers, ranging from only 1.5% to 3.2%, indicating that the shorter and more informal task description style of MBPP limits the impact of vocabulary-level ambiguity. Figure 2 makes this benchmark dependence visually apparent: the LV column shows a mixed pattern on HumanEval but near-uniform light shading on MBPP. Surface-level formatting noise has minimal impact. SF mutations produce drops below 3.7% on HumanEval and below 1.6% on MBPP for virtually all models (Table 4), with several models showing marginal gains (e.g., DeepSeek-Coder-33B: +1.8% on HumanEval; Qwen2.5-Coder-32B: +1.2%). The SF column in Figure 2 is the lightest across all benchmarks, visually confirming that typographical noise is the weakest of the three mutation types. This suggests that code-specialized LLMs are robust to surface-level corruption, extending the observations of Sclar et al. [36], from general-purpose LLMs to code generation ones. LiveCodeBench is nearly immune to all mutation types. In all models, Pass@1 deltas on LCB remain within ±2.0% for all defect types (Table 4), with many models recording marginal gains under US (e.g., StarCoder2-15B: +1.2%; CodeLlama-7B: +0.3%). Figure 2 confirms this near-zero sensitivity: the LCB rows are visually the least affected, in sharp contrast to HumanEval and MBPP. This is explained by the benchmark design: LCB problems include explicit
Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis
Table 4: Pass@1 results on all benchmarks and mutation types. For each mutation (US, LV, SF), Pass@1 deltas wrt to the original are shown in red (↓ drop) and green (↑ gain). US = Under-Specification; LV = Lexical Vagueness; SF = Syntax and Formatting. HumanEval
MBPP
LiveCodeBench
Model
Orig
US
LV
SF
Orig
US
LV
SF
Orig
US
LV
SF
Small models (≤7B) CodeLlama-7B DeepSeek-Coder-6.7B Qwen2.5-Coder-7B
37.2 72.6 82.3
29.3 ↓7.9 57.3 ↓15.3 67.7 ↓14.6
29.3 ↓7.9 60.4 ↓12.2 75.6 ↓6.7
37.2 = 68.9 ↓3.7 81.7 ↓0.6
33.3 44.1 50.1
26.8 ↓6.5 36.2 ↓7.9 41.2 ↓8.9
31.2 ↓2.1 41.6 ↓2.5 48.6 ↓1.5
33.7 ↑0.4 43.5 ↓0.6 49.0 ↓1.1
8.1 16.3 20.5
8.4 ↑0.3 16.0 ↓0.3 20.3 ↓0.2
8.2 ↑0.1 14.8 ↓1.5 22.2 ↑1.7
7.8 ↓0.3 15.2 ↓1.1 20.5 =
Large models (15B–34B) StarCoder2-15B Codestral-22B CodeLlama-34B DeepSeek-Coder-33B Qwen2.5-Coder-32B
65.9 72.6 51.2 72.0 85.4
51.2 ↓14.7 58.5 ↓14.1 42.1 ↓9.1 61.6 ↓10.4 73.2 ↓12.2
54.3 ↓11.6 65.2 ↓7.4 45.1 ↓6.1 68.3 ↓3.7 84.1 ↓1.3
62.8 ↓3.1 70.7 ↓1.9 50.0 ↓1.2 73.8 ↑1.8 86.6 ↑1.2
42.1 47.6 37.9 47.0 51.6
35.3 ↓6.8 38.9 ↓8.7 29.7 ↓8.2 39.8 ↓7.2 40.8 ↓10.8
39.7 ↓2.4 45.0 ↓2.6 34.7 ↓3.2 45.4 ↓1.6 48.6 ↓3.0
42.1 = 47.7 ↑0.1 36.3 ↓1.6 46.4 ↓0.6 51.4 ↓0.2
6.8 23.0 13.6 20.3 32.0
8.0 ↑1.2 22.5 ↓0.5 11.6 ↓2.0 19.4 ↓0.7 30.6 ↓1.4
7.3 ↑0.5 22.7 ↓0.3 13.1 ↓0.5 19.9 ↓0.4 31.7 ↓0.3
7.6 ↑0.8 22.6 ↓0.4 12.4 ↓1.2 18.8 ↓1.5 31.2 ↓0.8
Reasoning models (API) GPT-5-mini Claude Sonnet 4
96.3 95.7
86.6 ↓9.7 86.0 ↓9.7
89.0 ↓7.3 89.0 ↓6.7
93.3 ↓3.0 95.7 =
49.7 53.0
44.6 ↓5.1 44.1 ↓8.9
46.9 ↓2.8 50.3 ↓2.7
49.5 ↓0.2 53.5 ↑0.5
52.5 51.1
48.5 ↓4.0 49.6 ↓1.5
52.5 = 49.9 ↓1.2
53.2 ↑0.7 50.7 ↓0.4
Figure 2: Pass@1 drop (%) with all defect types in all models and benchmarks, computed as (Orig − Mutated)/Orig × 100. Red cells indicate performance degradation; blue cells indicate a slight gain. Column labels: US = Under-Specification; LV = Lexical Vagueness; SF = Syntax and Formatting. sample input/output examples, so models can reconstruct the correct solution from the examples alone. US mutations remove one textual constraint while the sample I/O, which often encodes the same constraint implicitly, remains intact. Consequently, we find that descriptions with richer context leads to more robustness in code generation, highlighting an important boundary condition for the generality of our findings. Model size does not relate to robustness; sensitivity is description-dependent. Comparison of small and large models
shows that model size does not relate to model robustness. GPT5-mini and Claude Sonnet 4 lose nearly 10% on code correctness due to US defects. Sensitivity appears to be description-dependent rather than capacity-dependent: Figure 2 shows different decreases in Pass@1 across all pairs of model-defect types, suggesting that model size alone does not determine a clear trend.
Akli et al.
RQ1 Takeaway. Not all defect types harm code generation equally: Under-Specification degrades performance by up to 15% on HumanEval and MBPP, LV causes moderate losses, and SF is negligible. Benchmarks with richer, more realistic descriptions, such as LiveCodeBench, are inherently more resilient to description defects. Model size and architecture alone do not determine robustness.
5.2
Defect detection
The RQ1 results suggest that the robustness to defects is primarily determined by the characteristics of the input description, rather than the underlying models. Consequently, improving the correctness of code generation requires identifying and addressing defects before code generation. Table 5 reports the classification performance of all five approaches, and Figure 3 visualizes the embedding space learned under each fine-tuning regime. Zero-shot frontier models perform poorly in defect classification. GPT-5-mini achieves an F1 of 0.469 and an MCC of 0.281, barely above the chance for a four-class problem. Claude Sonnet 4 does slightly better (F1=0.518, MCC=0.359) but still falls well short of practical utility. This result is notable because both models achieve the highest Pass@1 scores in RQ1; strong code generation does not transfer to meta-reasoning about description quality in a zero-shot regime. Few-shot prompting provides only limited gains for frontier models. Supplying four in-context examples yields marginal improvements for Claude Sonnet 4 (F1=0.557, MCC=0.412), while GPT-5-mini shows no consistent benefit and slightly degrades overall performance (F1 = 0.462, MCC = 0.276). This suggests that, unlike generation tasks, defect classification is not easily improved through in-context learning alone. Even a frozen linear probe substantially outperforms fewshot frontier models. Training only a linear classification head on frozen Qwen2.5-Coder-1.5B embeddings yields F1=0.748 and MCC=0.665, a jump of 0.23% F1 over Claude Sonnet 4 while updating just 0.0004% of parameters. SpecValidator with LoRA fine-tuning achieves the best classification performance, outperforming full fine-tuning. With only 0.14% of the parameters updated, LoRA achieves F1=0.804 and MCC=0.745, the best results across all classifiers. In contrast, complete fine-tuning (100% parameters) performs slightly worse (F1=0.788, MCC=0.722). The cause is likely overfitting: the dataset, while containing 8766 instances, is still constrained in size, and full fine-tuning may degrade the backbone’s generalization across the input distribution. This result advocates for LoRA as the practical choice: superior accuracy, dramatically lower computations, and local deployability. The ambiguity between clean and under-specified descriptions is the dominant error mode. Unlike LV and SF mutations, which introduce detectable surface signals, US mutations remove a constraint without leaving any positive textual trace: the resulting defective description is grammatically well-formed and stylistically indistinguishable from a clean one. This makes the clean/US boundary structurally harder to learn, as the classifier must detect the
absence of information rather than the presence of an anomaly. Figure 3 makes this concrete: under all three fine-tuning techniques, SF and LV form well-separated clusters, while US and clean exhibit the greatest residual overlap between parameter budgets. The LoRA projection shows much better separation compared to the linear probe, yet the US/clean boundary remains challenging. RQ2 Takeaway. SpecValidator with LoRA fine-tuning of a 1.5B code model achieves F1=0.804 and MCC=0.745, outperforming both zero-shot and few-shot GPT-5-mini and Claude Sonnet 4 by a wide margin, and also outperforming full finetuning. The dominant confusion is between the clean and US classes.
5.3
Generalization to real (unseen) task descriptions
The previous RQ results evaluate the classification on test samples that include the mutated data. A natural concern is whether the classifier generalizes beyond this setting to detect real specification issues in the original descriptions. To investigate this, we analyze running the classification on 456 clean descriptions from the three benchmarks, we use the test data of RQ2 (40 HumanEval, 197 MBPP, 219 LiveCodeBench). For each description, the classifier predicts either clean or one of the three defect classes (LV, US, SF); we treat any non-clean prediction as a flagged defect. We then assess whether flagged defects are truly problematic by measuring Pass@1 on the original descriptions across all ten models and by performing a manual inspection. As shown in Figure4, 83 of the clean descriptions are classified as US (75) and LV(8). Table 6 reports the classification of clean descriptions from the test set. Of the 456 descriptions, SpecValidator flags 83 (18.2%) as defective, all of which are Under-Specified: 0 from HumanEval, 33 from MBPP, and 50 from LiveCodeBench. The absence of false positives in HumanEval is consistent with its structured docstring format, which tends to be more explicit than the short MBPP descriptions or the challenging LCB problem statements. The defects flagged by the classifier yield near-zero Pass@1 across all ten models, suggesting that they are originally defective rather than arbitrarily misclassified. Table 7 reports Pass@1 on the 83 flagged defective descriptions. For MBPP, 90.9% of flagged defects fail across all models (failing on an average of 6.7 out of 10); for LiveCodeBench, 98.0% fail (failing on 7.7 out of 10). These descriptions consistently prevent correct code generation despite being labeled clean in the ground truth benchmark. 72.7% of flagged MBPP descriptions are confirmed as underspecified upon manual inspection. Three researchers independently reviewed the 33 flagged MBPP descriptions, which are short enough for human assessment, and 24 of 33 (72.7%) cases are confirmed as under-specified. Representative examples are shown in Table 8. The 50 LiveCodeBench cases could not be inspected due to the length and complexity of competitive programming statements, but their near-zero Pass@1 is consistent with the same pattern. Taken together, these results indicate that the classifier generalizes beyond the injected defects and indicates latent quality issues in the benchmark ground truth.
Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis
Table 5: Classification results for LV, SF, US, clean; macro-averaged metrics, reported as the mean over two seeds. GPT-5-mini and Claude Sonnet 4 are evaluated zero-shot and few-shot (4 examples) as baselines. Classifier
Trainable %
Accuracy
Precision
Recall
F1
MCC
GPT-5-mini (zero-shot) Claude Sonnet 4 (zero-shot)
0% 0%
0.46 0.51
0.490 0.550
0.460 0.520
0.469 0.518
0.281 0.359
GPT-5-mini (few-shot) Claude Sonnet 4 (few-shot)
0% 0%
0.45 0.55
0.520 0.620
0.450 0.560
0.462 0.557
0.276 0.412
0.0004% 0.14% 100%
0.75 0.81 0.79
0.750 0.810 0.800
0.745 0.805 0.785
0.748 0.804 0.788
0.665 0.745 0.722
Linear Probe LoRA Fine-tune (r=16) Full Fine-tune
Figure 3: t-SNE projections [40] of the embedding space of the classifiers (Qwen2.5-Coder-1.5B backbone). From left to right: Linear Probe (frozen backbone), LoRA fine-tune, and Full fine-tune.
RQ3 Takeaway. SpecValidator flags 83 of 456 clean benchmark descriptions as under-specified. These flagged descriptions yield extremely low Pass@1 across all ten models (90.9% failure rate on MBPP, 98.0% on LiveCodeBench), and 72.7% of inspectable MBPP cases are confirmed as under-specified upon manual review. These results indicate that SpecValidator generalises beyond its training distribution and detects specification defects in benchmark ground truth.
6 Threats to Validity Internal validity A primary validity concern is whether our mutants are genuinely defective rather than arbitrarily corrupted. We mitigate this through a two-tier validation strategy: An independent LLM judge (Qwen2.5Coder-32B) scores all 6,573 mutants for compliance and naturalness, and a stratified manual sample of 100 instances is manually checked. Human annotators closely rate the automated scores, providing confidence on the representativeness of our mutations. Another concern is the use of greedy decoding (temperature = 0) during inference. While this ensures reproducibility, it may underestimate the variance introduced by temperature sampling and overestimate
Figure 4: Confusion matrix of the LoRA classifier. sensitivity for problems where the model’s top-1 beam is near the decision boundary. We treat this as a conservative choice: if greedy decoding shows sensitivity, sampling would only amplify it.
Akli et al.
Table 6: SpecValidator on the 456 real unseen description. TN = correctly predicted CLEAN; FP = incorrectly flagged as defective (all as US). Specificity = TN / Total; FP rate = FP / Total. Benchmark
Total
TN
FP
Specificity
FP Rate
HumanEval MBPP LiveCodeBench
40 197 219
40 164 169
0 33 50
100.0% 83.2% 77.2%
0.0% 16.8% 22.8%
Total
456
373
83
81.8%
18.2%
External validity The finding that LCB is immune to prompt-level perturbations may not generalize well. Conversely, findings on HumanEval and MBPP may not transfer to more complex, library-intensive tasks such as those in BigCodeBench [49], where the relationship between prompt precision and functional correctness is likely to differ. All open-source models are evaluated in their instruction-tuned variants; base models may exhibit a different and potentially higher sensitivity. Our findings on model-family-specific behavior (e.g., the unusually low LV sensitivity of Qwen2.5-Coder-32B) should be interpreted with caution as it may be architecture- or trainingdata specific. Furthermore, although we include HumanEval and MBPP despite their known contamination risks [27, 34], contamination may artificially compress the measured sensitivity: if models have partially memorized solutions, they may be more robust to prompt changes than they would otherwise be. Including LCB as a contamination-free benchmark is intended to mitigate this threat.
Table 7: Pass@1 results on CLEAN test samples predicted as non-CLEAN by the classifier. F = pass@1 False (failing); P = pass@1 True (passing). The Misclassified row aggregates across all samples; the parenthetical reports the average number of models on which a misclassified sample fails. Model
N
F
P
%F
CodeLlama-34B CodeLlama-7B StarCoder2-15B DeepSeek-Coder-33B Qwen2.5-Coder-32B DeepSeek-Coder-6.7B Codestral-22B GPT-5-mini Qwen2.5-Coder-7B Claude Sonnet
33 33 33 33 33 33 33 33 33 33
25 24 23 21 20 20 20 19 15 15
8 9 10 12 13 13 13 14 18 18
75.8 72.7 69.7 63.6 60.6 60.6 60.6 57.6 45.5 45.5
Misclassified (Fail on avg 6.7/10 models)
33
30
3
90.9
StarCoder2-15B CodeLlama-7B CodeLlama-34B DeepSeek-Coder-33B DeepSeek-Coder-6.7B Codestral-22B Qwen2.5-Coder-7B Qwen2.5-Coder-32B GPT-5-mini Claude Sonnet
50 50 50 50 50 50 50 50 50 50
49 45 43 43 41 41 39 36 27 15
1 5 7 7 9 9 11 14 23 35
98.0 90.0 86.0 86.0 82.0 82.0 78.0 72.0 54.0 30.0
Misclassified (Fail on avg 7.7/10 models)
50
49
1
98.0
MBPP
LiveCodeBench
Construct validity We use Pass@1 as a correctness metric, which treats all failures identically. Pass@1 is the standard in the code generation literature and reflects the real-world scenario in which a developer accepts the model’s first suggestion, but it cannot distinguish near misses from total failures. User prompts may contain mixed or compound defects that do not fall cleanly into our categories. The judgment of one constraint removed for US mutations is necessarily subjective; despite our quality-control pipeline, some US mutants in MBPP were rated as under-specified even in their original (CLEAN) form, a limitation of our LoRA classifier’s false-positive analysis explicitly surfaces as a finding rather than an error.
7
Table 8: Example Under-Specified (real) defects detected by SpecValidator. ID
Prompt
Ambiguity
MBPP Filter a dictionary based on 277 values.
By what condition? A threshold?
MBPP Find the peak element in 466 the given array.
Local peak or global max? Index or value?
MBPP Find the number of rota382 tions in a circularly sorted array.
Left or right rotation? Ascending or descending?
Conclusion
This study investigates the impact of task description defects on the LLM-based code generation and ways to automatically detect such defects. We find that defect impact is not uniform: it depends jointly on defect type and benchmark specification structure. Under-Specification is the most damaging class, causing Pass@1 drops of up to 15.3% with no model immune, including state-ofthe-art reasoning models above 95% baseline. Lexical Vagueness produces moderate, benchmark-dependent degradation, whereas surface formatting noise has a negligible effect on code-specialized models. Benchmarks with richer contextual grounding, such as LiveCodeBench, are substantially more resilient to prose-level defects,
establishing the specification structure as a first-order moderator of robustness that has been invisible in prior work. On the detection side, SpecValidator, a lightweight LoRA-finetuned 1.5B classifier, achieves F1 = 0.804 and MCC = 0.745, outperforming few-shot GPT 5-mini and Claude Sonnet 4 by a wide margin despite updating only 0.14% of parameters, confirming that strong code generation ability does not transfer to the prompt metareasoning. More importantly, SpecValidator generalises beyond its training distribution: it flags 83 clean benchmark descriptions as under-specified, 72.7% of which are confirmed as genuinely defective upon manual inspection, with near-zero Pass@1 across all ten
Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis
models supporting the same conclusion for the remaining cases. This demonstrates that SpecValidator can surface latent quality issues in benchmark ground truth, a finding with direct implications for how code generation benchmarks are constructed and evaluated. Taken together, these results argue that description defects should be treated as structurally distinct phenomena requiring targeted fixing. Robustness evaluations that aggregate across perturbation types or rely exclusively on simple single-source benchmarks risk drawing conclusions that do not transfer to realistic settings. We release our dataset of 6,573 defective task descriptions to support reproducibility and future work.
Akli et al.
References [1] Anthropic. 2024. Claude 4 Model Card. Anthropic Documentation. https: //www.anthropic.com [2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732 (2021). [3] Tom B. Brown, Benjamin Mann, Nick Ryder, et al. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33. 1877–1901. [4] Junkai Chen, Zhenhao Li, Xing Hu, and Xin Xia. 2026. NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations. ACM Trans. Softw. Eng. Methodol. 35, 4, Article 89 (March 2026), 20 pages. doi:10.1145/3745764 [5] Mark Chen, Jerry Tworek, Heewoo Jun, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). [6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [7] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching Large Language Models to Self-Debug. In Proceedings of the 12th International Conference on Learning Representations (ICLR). https://openreview.net/forum? id=KuPixIqPiq [8] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36. [9] Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K Lahiri. 2024. Llm-based test-driven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering 50, 9 (2024), 2254–2268. [10] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wenjie Zhang, Wenhu Chen, Kexin Bi, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming—The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024). [11] Asma Hamidi, Ahmed Khanfir, and Mike Papadakis. 2025. Intent-Based Mutation Testing: From Naturally Written Programming Intents to Mutants. In IEEE International Conference on Software Testing, Verification and Validation, ICST 2025 - Workshops, Naples, Italy, March 31 - April 4, 2025. IEEE, 347–357. doi:10.1109/ICSTW64639.2025.10962508 [12] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML). 2790–2799. [13] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR). [14] Binyuan Hui, Jian Yang, Zeyu Cui, et al. 2024. Qwen2.5-Coder Technical Report. arXiv preprint arXiv (2024). [15] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186 [cs.CL] https://arxiv.org/ abs/2409.12186 [16] Naman Jain, Jiayi Han, Alex Gu, William Yang, Yiming Li, Koushik Sen, and Ion Stoica. 2024. LiveCodeBench: Holistic and Contamination-Free Evaluation of Large Language Models for Code. arXiv preprint arXiv:2403.07974 (2024). [17] Haoxiang Jia, Robbie Morris, He Ye, Federica Sarro, and Sergey Mechtaev. 2025. Automated Repair of Ambiguous Problem Descriptions for LLM-Based Code Generation. arXiv preprint arXiv:2505.07270 (2025). [18] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2026. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology 35, 2 (2026), 1–72. [19] Maya Larbi, Amal Akli, Mike Papadakis, Rihab Bouyousfi, Maxime Cordy, Federica Sarro, and Yves Le Traon. 2025. When prompts go wrong: Evaluating code model robustness to ambiguous, contradictory, and incomplete task descriptions. arXiv preprint arXiv:2507.20439 (2025). [20] Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2025. Structured Chain-of-Thought Prompting for Code Generation. ACM Transactions on Software Engineering and Methodology 34, 2 (2025), 1–23. doi:10.1145/3690635 [21] Yujia Li et al. 2022. Competition-level Code Generation with AlphaCode. Science 378, 6624 (2022), 1092–1097. [22] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems 36 (NeurIPS). https://proceedings.neurips.cc/paper_files/paper/ 2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html
arXiv:2305.01210. [23] Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations (ICLR). [24] Anton Lozhkov, Raymond Li, Vaibhav Chaudhary, et al. 2024. StarCoder2 and The Stack v2: The Next Generation. arXiv preprint arXiv:2402.19173 (2024). [25] Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. 2023. On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 2149–2160. doi:10.1109/ICSE48619.2023.00181 [26] Brian W. Matthews. 1975. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta 405, 2 (1975), 442–451. [27] Alexandre Matton et al. 2024. On Leakage of Code Generation Evaluation Datasets. In Findings of the Association for Computational Linguistics: EMNLP. 13215–13223. [28] Bertrand Meyer. 1985. On Formalism in Specifications. IEEE Softw. 2, 1 (1985), 6–26. doi:10.1109/MS.1985.229776 [29] Mistral AI. 2024. Codestral: Hello, World! Technical report. https://mistral.ai/ news/codestral [30] Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2332–2354. [31] OpenAI. 2025. GPT-5 Technical Report. https://openai.com. Accessed: 2026. [32] Shunyu Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv preprint arXiv:2302.06590 (2023). [33] Fazle Rabbi, Zishuo Ding, and Jinqiu Yang. 2025. A Multi-Language Perspective on the Robustness of LLM Code Generation. arXiv preprint arXiv:2504.19108 (2025). https://arxiv.org/abs/2504.19108 [34] Martin Riddell, Ansong Ni, and Arman Cohan. 2024. Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models. In Proceedings of ACL. 14116–14137. [35] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, et al. 2023. Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950 (2023). [36] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design. In Proceedings of the International Conference on Learning Representations (ICLR). [37] Atsushi Shirafuji, Yutaka Watanobe, Takumi Ito, Makoto Morishita, Yuki Nakamura, Yusuke Oda, and Jun Suzuki. 2023. Exploring the Robustness of Large Language Models for Solving Programming Problems. arXiv preprint arXiv:2306.14583 (2023). [38] Mohammed Latif Siddiq, Simantika Dristi, Joanna C.S. Saha, and Joanna Santos. 2024. The Fault in our Stars: Quality Assessment of Code Generation Benchmarks. In Proceedings of the IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 201–212. [39] Zhao Tian, Junjie Chen, and Xiaofei Zhang. 2025. µFiX: Fixing Large Language Models’ Specification Misunderstanding for Better Code Generation. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE). IEEE/ACM, 1514–1526. [40] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9 (2008), 2579–2605. [41] Axel van Lamsweerde. 2009. Requirements Engineering - From System Goals to UML Models to Software Specifications. Wiley. http://eu.wiley.com/WileyCDA/ WileyTitle/productCd-EHEP000863.html [42] Andreas Vogelsang, Alexander Korn, Giovanna Broccia, Alessio Ferrari, Jannik Fischbach, and Chetan Arora. 2025. On the Impact of Requirements Smells in Prompts: The Case of Automated Traceability. In Proceedings of the IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). IEEE/ACM. [43] Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. 2023. ReCode: Robustness Evaluation of Code Generation Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). 13234–13274. [44] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. [45] Jie J.W. Wu and Fatemeh H. Fard. 2025. HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agents. ACM Transactions on Software Engineering and Methodology 34, 7 (2025). doi:10. 1145/3715109 [46] Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang. 2024. EvoEval: Evolving Coding Benchmarks via LLM. In Proceedings of the Conference on Language Modeling (COLM).
Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis
[47] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36. 46595–46623. [48] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, and Xing Xie. 2024. PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. In Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis (LAMPS). ACM, 57–68.
[49] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, WenDing Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. 2025. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In Proceedings of the International Conference on Learning Representations (ICLR). arXiv:2406.15877.