SPARK: Semantic-preserving Prompt Mutation for Code Generation using Consensus-based Verification Framework

SPARK : Semantic-preserving Prompt Mutation for Code Generation using Consensus-based Verification Framework Alireza Daghigh

Chung-Yu Wang

Luca Filippelli

Hamed Taherkhani Hadi Hemmati Hung Viet Pham

arXiv:2611.13401v1 [cs.SE] 10 Nov 2026

York University York University York University York University York University York University Toronto, Canada Toronto, Canada Toronto, Canada Toronto, Canada Toronto, Canada Toronto, Canada alirezadeq@yorku.ca cywang14@yorku.ca lucafili@my.yorku.ca hamedth@yorku.ca hemmati@yorku.ca hvpham@yorku.ca

Abstract—Automated code generation with large language models (LLMs) has improved in recent years. With the lack of test cases in real-world usages, generated code is not automatically verifiable and often requires manual validation. Previous studies have leveraged consensus verification, where code solutions and corresponding test cases are generated and evaluated based on functional consistency. Building on this, SPARK incorporates semantic-preserving mutations to problem descriptions such as paraphrasing, grammatical adjustments, function renaming, and other alterations to enhance the robustness and correctness of LLM-generated code. For each mutated description, SPARK generates a set of code solutions and test cases. These candidate solutions are evaluated using consensus-based verification, where the solutions’ quality is measured by how consistently they pass generated tests. We assess the effectiveness of SPARK on three benchmarks (LiveCodeBench, BigCodeBench, and DLBench) across three LLMs (o4-mini, DeepSeek-R1-Basic, and Gemini2.5-Pro). Our evaluation demonstrates that SPARK outperforms state-of-the-art consensus-based verification approaches up to 4.1%. Our generalization experiment highlights the practical potential of SPARK as an effective strategy for enhancing the reliability and correctness of LLM-based code generation systems in scenarios where traditional validation resources are scarce, while our qualitative analysis shows insight into how SPARK’s mutation helps LLMs generate better code solutions. Index Terms—Code Generation, Test Generation, LLM4SE,

I. Introduction Automated code generation [1], [2] with large language models (LLMs) [3], [4], [5], [6] has advanced rapidly in recent years, offering promising tools for software development. Despite these gains, reliably producing a single correct, executable solution remains a significant challenge [7], [1], [8]. Many models achieve high success rates when allowed to generate multiple candidate solutions, as measured by metrics such as pass@k (the probability that at least one of the top-k generated solutions passes the test suite) [9], [10]. However, their top-ranked outputs often fail to meet correctness requirements in practice [11]. This gap highlights the practical difficulty of selecting a reliable solution from This work was supported by the NSERC Discovery Grant and Vector Institute Scholarship in AI.

Original Prompt Read a list of dictionaries from a JSON file, calculate the mean and median for each key, and write the results to a CSV file. ...

Generated Code 1 def task_func(...): ... with open(output_file, 'w', newline='') as csvfile: fieldnames = ['Field', 'Mean', 'Median'] ...

Generated Code 2 def task_func(...): ... writer.writerow(['field', 'mean', 'median']) ...

Declarative to Interrogative mutation

Mutated Prompt Could you read a list of dictionaries from a JSON file, calculate the mean and median for each key, and write the results to a CSV file? ...

Generated Code 3 def task_func(...): ... writer.writerow(['key', 'mean', 'median']) ...

Fig. 1: An example showing a prompt mutation from declarative to interrogative leads the LLM to generate code with a corrected CSV header (key instead of field).

multiple generated candidates, especially in real-world scenarios where comprehensive, human-written test suites are scarce or unavailable [12], [13], [14]. To address the lack of ground-truth tests in practice, prior work such as the CodeT framework [15] leverages generated corresponding test cases to verify code solutions. Inspired by consensus-based algorithms [16] such as RANSAC [17], this dual agreement approach can identify groups of solutions that consistently pass groups of generated test cases and thus exhibit highly cohesive functionality. CodeT demonstrated improvements in code generation benchmarks without relying on hand-crafted test cases. However, since CodeT relies on a single prompt, the generated solutions are often tied to a single LLM’s understanding of that one specification, leading to some misinterpretation. Figure 1 illustrates such a failure: when tasked with computing the mean and median from a JSON file and writing the results to a CSV, the original prompt repeatedly fails validation because the solution outputs a field header while the validation suite expects a key header. This scenario demonstrates how single-prompt generation can fail due to subtle specification ambiguities.

To mitigate this shortcoming, we propose SPARK , which introduces semantic-preserving prompt mutations to systematically create diverse, semantically equivalent variants of the original problem description. These instances increase diversity in both generated solutions and tests, improving the chances that more candidate matches the intended specification. Concretely, SPARK uses a common, language-focused LLM (GPT-4o) to apply controlled changes in phrasing, grammatical structure, function naming, and other lexical/syntactic features (see Table I). For each mutated prompt, we generate code solutions and test cases; all candidates then participate in a dual-execution agreement selection. Candidate solutions are then grouped by their test-case pass patterns under the hypothesis that correct implementations exhibit consistent functional behavior across differently phrased prompts. We then select the cluster with the strongest agreement as the final output. As shown in Figure 1, reformulating the original declarative instruction as an interrogative prompt leads the model to produce the correct key header, enabling it to pass the validation suite. We evaluate SPARK on three complementary benchmarks, LiveCodeBench [18] (algorithmic/data-structure tasks), BigCodeBench [19] (real-world repository/API tasks), and DLBench [20] (deep learning–oriented code generation) using three state-of-the-art LLMs: O4-Mini [21], DeepSeek R1 Basic [22], and Gemini 2.5 Pro [23]. All prompt mutations are produced with GPT-4o. Our experiments show that incorporating a single prompt mutation in most cases already improves the pass@1 performance across most dataset–model combinations. While the magnitude of improvement varies, we observe clear patterns. For example, verb-based modifications (e.g., converting between active and passive forms or replacing verbs with nearsynonyms) provide the strongest and most reliable gains, particularly on LiveCodeBench and DLBench. Context-based modifications, such as altering or removing function names, also contribute meaningfully, particularly on repository-style tasks in BigCodeBench. Styling-oriented mutations, though more superficial, still produce measurable improvements by prompting models toward more precise code generation. These results suggest that different classes of mutations address complementary weaknesses in LLMs, with verb modifications enhancing semantic alignment, context modifications improving robustness to naming and specification noise, and styling changes introducing additional syntactic diversity. We expand upon the analysis of individual mutations by also studying the compositions of mutations. We first explore combinations of mutations on a calibration dataset to identify the best-performing mutation set. Such combinations are then applied to the tasks from a different dataset. Specifically, on average, SPARK achieves up to 4.1% improvement in pass@1 score over CodeT, the state-of-the-art consensus-based validation technique. This experiment demonstrates SPARK ’s expandability and practical usage in a real-world scenario where additional mutation can be added, and tuning on the calibration dataset is generalizable to deployment time.

Our qualitative analysis shows that mutations improve code generation through several mechanisms. The most common is generating diverse implementations that reduce overfitting to a single phrasing. Mutations also increase resilience to misleading or distracting context, clarify implicit requirements, expose boundary and corner cases, and act as a filter against overfitted solutions by aligning with harder tests. Together, these effects explain why mutations often lead to stronger and more reliable programs. Overall, SPARK not only raises accuracy in diverse benchmarks but also demonstrates that systematic mutation strategies can be tuned to the strengths of specific models and datasets, offering a practical path to more dependable LLMbased code generation. This paper makes the following contributions: • We propose SPARK , a code generation technique that combines semantic-preserving prompt mutations with consensus-based verification to select reliable code solutions. • We conduct an extensive evaluation with three LLMs on three complementary code-generation benchmarks, showing consistent improvements over baselines of up to 4.1% pass@1. • We provide qualitative insights into when and why different mutation strategies help or hurt, and we study a general strategy to select effective mutation combinations that can generalize in real-world scenarios. Data Availability: We release our code and data1 . II. Approach Figure 2 provides SPARK ’s overview. When given a programming problem, SPARK first performs 1 Prompt Mutation. Using the original and mutated problems, SPARK then queries the LLM to execute the 2 Code & Test Generation step. By executing the generated test cases and applying consensus-based validation, SPARK identifies the best code solution in the 3 Solution Selection step. A. Prompt Mutation One of the strengths of the consensus-based validation process proposed by CodeT is that it can estimate the solution quality by verifying the majority agreement of code solutions with generated test cases. However, in cases where the LLM failed to generate correct code solutions from the original prompt, CodeT would select the incorrect solutions. SPARK utilizes a set of semantic-preserving mutations of the problem prompt to help trigger additional correct solutions. As the example in Figure 1 demonstrates, a tone modification from declarative to interrogative can induce more diversity in the generated code, thereby helping the LLM produce additional correct solutions that can improve the overall accuracy of the code generation system. In step 1 , SPARK influences solution diversity by creating problem description variance through semantic-preserving 1 https://github.com/Smart-System-testing-lab/SPARK

1

Prompt Mutation

A programming problem

2

Code & Test Generation

Mutation operators

Code prompts with mutated problems

Mutated problems

Test prompts with mutated problems

3 Solution Selection

Code1 Code2 Code3

...

Consensus-based validation

LLM Test1

Test2

Test3

The best code solution ...

Fig. 2: Overview of SPARK TABLE I: Mutation operation, their class, their prompt template, and corresponding input–output problem descriptions. Class

Operation (Abbr.) Remove Function Name (RFN)

Context Modification

Unrelated Function Name (UFN) Adversarial Function Name (AFN) Rename Variables (RV) Lowercase to Uppercase (L2U)

Styling Modification

Verb Modification

Tone Modification

Mutation Prompt Template

Example Input

Modify the following sentence to exclude references to the function’s name Modify the following sentence to make the function’s name unrelated to its meaning Modify the following sentence to refer to a function with a misleading name that doesn’t represent the actual task Modify the following sentence to rename any explicitly named variable

Generate a Python function multinomial_sampling... Generate a Python function multinomial_sampling...

Change the following sentence to uppercase letters

Uppercase to Lowercase (U2L)

Change the following sentence to lowercase letters

Star Highlight (SH)

Encase important phrases with two asterisks per side

Uppercase Highlight (UH)

Highlight important phrases with uppercase letters

Active to Passive (A2P)

Change the following sentence to the passive voice

Passive to Active (P2A)

Change the following sentence to the active voice

Generate a Python function which takes an attribute named var1...

Generate a Python function which takes an attribute named var2...

Generate a Python function called hinton... Generate a Python function called hinton... Generate a Python function called hinton... Generate a Python function called hinton...

GENERATE A PYTHON FUNCTION CALLED HINTON... generate a python function called hinton... Generate a **Python function** called hinton... Generate a PYTHON FUNCTION called hinton... A Python function cot_laplacian should be written to compute... Write a Python function cot_laplacian that computes... A Python function will be written for the following task:... A Python function was written for the following task:... Construct a Python function for the subsequent task: Illustrate... Is it possible to generate a Python function called top_k_accuracy...

Change the following text to future tense phrasing

Present To Past Tense (2P)

Change the following text to past tense phrasing

Verb to Similar Verb (V2V)

Change the verbs of the following sentence to a similar verb

Declarative to Interrogative (D2I)

Change the following sentence to interrogative voice

Generate a Python function called top_k_accuracy...

Modification focuses on mutating the context of the code generation task. For example, some problem descriptions include function names that can influence the model to generate a specific solution. By mutating this context, the prompt variances can influence different solutions to be generated. • Styling Modification focuses on modifying the styling and highlighting of the problem descriptions. Different highlighting and styling can influence the attention of the models, which can induce additional solution variance. • Verb Modification focuses on modifying the verbs tenses. Modifying these action verbs can directly induce changes in the generated solutions. • Context

Generate a Python function chocolate_cheesecake... Generate a Python function called compute_angle_of_projection...

Present To Future Tense (2F)

Table I shows the mutation prompt template and an example for each mutation operation. Each mutation class aims to introduce changes that can induce additional solution generation:

Generate a Python function...

Generate a Python function called get_perspective_transform...

Write a Python function cot_laplacian that computes... A Python function cot_laplacian should be written to compute... Write a Python function for the following task... Write a Python function for the following task... Write a Python function for the following task: Describe...

mutation operations. These operators aim to vary the prompt styling, tone, syntactic structure, or context. This form of variation can trigger changes in the generated code, which can improve the overall correctness, efficiency, and readability of the candidate solution set (refer to V for further discussion). Specifically, SPARK applies fourteen prompt mutations operators to the original problem description in four major thematic classes: Context Modification, Styling Modification, Verb Modification, and Tone Modification.

Example Output

Modification focuses on the tone of the request. Changing the tone could indirectly instigate additional solution variances.

• Tone

B. Code and Test Generation Once the prompt variances are collected, SPARK queries LLMs to generate both code solutions and test cases to prepare for the consensus-based validation. In step 2 , SPARK utilizes different prompt templates for code and test generation for each problem description to generate candidate code solutions and test cases. Code Solution Generation: To generate code solutions, SPARK directly queries LLMs with the problem description. Since the origin and mutated problem description already contain code generation commands, no additional request is needed. Due to the generated code sometimes containing missing imports, inconsistent function names compared to those expected in the test cases, or mismatched variable names between the function definition and the test inputs, additional post-processing is needed. This includes automatically inserting or correcting import statements, renaming functions to align with the required signatures, and adjusting variable names so that arguments match the test cases. Each generated code solution is then added to the solution candidate pool. Test Case Generation: In addition to generating code solutions, SPARK also generate test cases which will be used

in the consensus-based validation phase to estimate the quality of solution candidates. Specifically, a test case includes a specific input along with the expected output for the function defined in the problem context. For example, in a task that determines whether there are elements in a list that are closer than a given threshold, a test case would provide an input list and threshold value, as well as the correct Boolean result. To generate these test cases, instead of directly querying LLMs using the original problem description, we apply a test generation template (Figure 3). This template replaces the code generation command with the test case generation command. Similar to code generation, SPARK generates test cases with both the original problem description and the mutated ones and adds them to the test cases pool. Specifically, our template tailors the test case generation to support different benchmark requirements. Specifically, we produce three distinct types of test cases to match the expected formats of the major usages corresponding to the major benchmarks: input-output assertions: These are direct assert statements which check that the function produces the correct output for a given input. This style is compatible with the LiveCodeBench benchmark, which requires concise, clear input-output pairs. • Unittest format: These test cases follow the structure of Python’s unittest framework, organizing checks into test classes and methods. This format aligns with the needs of BigCodeBench, which uses standard unit testing conventions. • Pytest format: These tests use the pytest style, often employing function-based tests with decorators or simple assertions that integrate easily with the pytest runner. This format ensures compatibility with DL-Bench, which adopts pytest as its testing standard. • Simple

In terms of testing frameworks, our method considers the two most popular libraries in Python, along with the default assertion style. By capturing these testing formats, our approach generalizes to most development environments. In the event the above test styles are not used, the user can create a new template by making trivial modifications to the existing prompt templates. By generating test cases in these varied formats, we evaluate the generalization of our approach to any development environment with various unique testing requirements. Solutions Selection: In step 3 , SPARK executes the test cases to validate the code solutions in a consensus-based validation process (i.e., dual agreement[15]). Since the generated test cases are not guaranteed to be correct, this step estimates the quality of the solution candidates. While we directly adopt the original validation step, our diverse prompts help tease out more correct solutions, hence overall improve the validation process effectiveness. Tests execution: To compute the quality score for each code solution candidate, SPARK first validates each solution candidate by running it against all test cases in the pool.

To execute the test cases, SPARK needs to perform different execution steps for different development environments: • For simple input–output assertion tests, SPARK wraps the candidate in a lightweight runner that (i) feeds the provided stdin to the program or function, (ii) captures stdout and stderr, (iii) normalizes whitespace and line endings, and (iv) compares the observed output against the expected string(s). Each case is executed in a fresh process with a wall-clock timeout and resource limits; non-zero exit codes, timeouts, or mismatches are recorded as failures. • For pytest tests, SPARK materializes a temporary project (candidate file plus tests), ensures the target symbol is importable (fixing the module path and function name if needed), and runs pytest -q with per-test timeouts. It captures pass/fail/skip/xfail outcomes and assertion messages. • For unittest tests, SPARK discovers and runs the suite via unittest loaders, importing the candidate module, applying any required shim for function renaming, and executing each TestCase with buffered output and timeouts. It collects per-test outcomes (OK/FAIL/ERROR/SKIP) and condensed tracebacks. Consensus-based validation: Once the pass and fail statistics are collected, SPARK applies consensus-based validation by computing a score for each consensus set. Following prior work [15], the score is defined as 𝑓 (𝑆) = |𝑆 𝑥 | · |𝑆 𝑦 |,

(1)

where: • 𝑥 ∈ 𝑋 = {𝑥 1 , 𝑥 2 , . . . , 𝑥 𝑁 } denotes a code solution generated by the model, • 𝑦 ∈ 𝑌 = {𝑦 1 , 𝑦 2 , . . . , 𝑦 𝑀 } denotes a test case generated for the problem, • 𝑆 𝑥 ⊆ 𝑋 is the set of code solutions that pass exactly the same subset of test cases, • 𝑆 𝑦 ⊆ 𝑌 is the set of test cases passed by all code solutions in 𝑆 𝑥 . Intuitively, the score measures the strength of a consensus set by combining the number of agreeing solutions (|𝑆 𝑥 |) and the number of test cases they jointly satisfy (|𝑆 𝑦 |). A higher score indicates stronger evidence that the corresponding functionality is correct. Once the scores are computed. SPARK ranks and selects the highest-scoring solutions. However, the consensus-based nature of the dual execution agreement may result in multiple candidate solutions passing the same set of tests, which results in multiple candidates with the same score. In such cases, we introduce a tiebreaker mechanism based on comparing the average scores of the tied solutions’ code quality metrics. This approach assumes that the solution with higher structural quality is more likely to generalize and align with the groundtruth reference when functional behaviour is indistinguishable. To quantify code quality, we extract established metrics from the previous work [24], [25]: (i) the Maintainability Index (MI), which aggregates readability and complexity on a 0–100

Simple Pair

Pytest

Unittest

Generate 5 test cases for {question}. Return as a Python list of {'input': ..., 'expected_output': ...}.

Generate 5 unittest cases for {question}. Use a TestCase class with methods like test_*, using self.assert*.

Generate 5 pytest test cases for {question}. Use functions starting with test_, each with assert statements.

(a) Input-output assertion

(b) Unittest format

(c) Pytest format

Fig. 3: Prompt templates for test case generation. scale (higher is better); (ii) Lack of Cohesion (LCOM1), where lower values indicate stronger cohesion among functions; (iii) Coupling Between Objects (CBO), which captures the number of distinct external modules used (lower is better); and (iv) Cyclomatic Complexity (CC), measuring the decision complexity of functions (lower is better). These metrics are combined into a single normalized score to represent the overall structural quality of each program. This secondary ranking allows SPARK to prefer programs that are not only correct on the given tests but are also more maintainable and less error-prone in practice. Code solutions that are ranked highest in both rankings are selected as SPARK ’s final solutions. III. Experimental Setup Large Language Models Since semantic-preserving prompt mutation is a natural language task that is LLM’s strength, in general, any recent LLM is sufficient. Hence, SPARK employs GPT-4o, a low-cost general-purpose LLM that is capable of such tasks [26], [27]. On the other hand, code and test generation is a complex task that requires much more sophisticated LLMs. We evaluate SPARK on three state-of-the-art code generation models (o4-mini, DeepSeek R1 Basic, and Gemini 2.5 Pro) to ensure a robust comparison across different architectures and design philosophies. For fair evaluation, all models are queried in zero-shot mode using identical decoding parameters, with a temperature of 0.3, and top-p of 1.0. Code Generation Benchmarks We evaluate SPARK on three complementary benchmarks with varied developmental environmental settings and complexity. LiveCodeBench includes a suite of algorithmic and data-structure problems drawn from community coding challenges, testing general problem-solving ability. BigCodeBench consists of large-scale code synthesis tasks extracted from real-world repositories, emphasizing API usage, project structure, and realistic development practices. Finally, DLBench is a deep-learning–specific benchmark that covers model definitions, training loops, and tensor manipulation tasks, providing a targeted evaluation for machine learning workflows. These diverse benchmarks create a complementary evaluation experiment that assesses SPARK ’s generalization and effectiveness across a broad range of programming tasks. Evaluation Metric Following common practice, we adopt the pass@k metric, which measures the probability that at least one of the top-𝑘 generated solutions correctly solves the given problem, meaning it can pass all the ground-truth test cases.

Original Generate a Python function called rgb_to_rgba that Convert an image from RGB to RGBA.

Mutant Generate a Python function called rgb_to_hsv that Convert an image from RGB to RGBA.

Fig. 4: Adversarial Function Name mutation that failed to apply the intended adversarial renaming. IV. Result In this section, we present the findings to answer our research questions that aim to investigate the generalization and effectiveness of SPARK in code generation on three complementary benchmarks. A. RQ1: How well do LLMs perform prompt mutation while preserving problem description semantics? This research question aims to assess LLM’s ability to generate variances of problem descriptions while preserving the semantics that are needed for the model to generate correct code. To validate the correctness of the mutants, we randomly sampled 500 mutated problem descriptions from 12,586 instances across all three benchmarks. Mutations were grouped into four broader categories—Context Modification, Styling, Verb Modification, and Tone Modification—and were rated along two dimensions: (1) whether the mutated prompt preserved semantic equivalence with the original, and (2) whether the intended transformation was applied correctly. Three co-authors independently performed manual labeling and achieve an inter-annotator agreement of 93.2% for transformation correctness and 88.9% for semantic equivalence, indicating the reliability of our labeling process. Given the prevalence of positive ratings (over 90%), observed agreement provides a more interpretable metric than chance-corrected scores [28], [29]. Table II summarizes the results, showing the percentage of mutations that are semantic equivalent, applied correctly, and both semantic equivalent & applied correctly along with the corresponding margins of error (±). Overall, mutation operations are applied correctly with an accuracy of 95.4 ± 1.8%. However, in some cases, such as when the adversarial function name is requested, the mutation application’s correctness is lower (88.4 ± 8.5) due to the name not being changed to something misleading to the original task. The example shown in Figure 4 shows such cases where the function name “rgb_to_rgba” is changed to “rgb_to_hsv”, which is still related to the color space conversion task. In terms of semantics, the mutation preserved the problem description semantics in most cases, with an overall

TABLE II: Mutant quality across mutation types. Applied Correctly (%)

Semantically Equivalent (%)

Applied Correctly & Semantically Equivalent (%)

Remove Function Name Unrelated Function Name Adversarial Function Name Rename Variables

80.0 ± 14.1 100.0 ± 0.0 88.4 ± 8.5 90.0 ± 10.6

100.0 ± 0.0 100.0 ± 0.0 100.0 ± 0.0 100.0 ± 0.0

80.0 ± 14.1 100.0 ± 0.0 88.4 ± 8.5 90.0 ± 10.6

Styling Modification

Lowercase to Uppercase Uppercase to Lowercase Star Highlight Uppercase Highlight

100.0 ± 0.0 100.0 ± 0.0 93.7 ± 8.3 88.8 ± 10.1

97.6 ± 4.5 95.4 ± 6.2 100.0 ± 0.0 94.4 ± 7.4

97.6 ± 4.5 95.4 ± 6.2 93.7 ± 8.3 83.3 ± 11.9

Verb Modification

Active to Passive Passive to Active Present to Future Tense Present to Past Tense Verb to Similar Verb

96.0 ± 7.6 100.0 ± 0.0 98.3 ± 3.2 100.0 ± 0.0 98.9 ± 2.0

96.0 ± 7.6 100.0 ± 0.0 100.0 ± 0.0 100.0 ± 0.0 98.9 ± 2.0

96.0 ± 7.6 100.0 ± 0.0 98.3 ± 3.2 100.0 ± 0.0 97.8 ± 2.9

Tone Modification

Declarative to Interrogative

100.0 ± 0.0

79.6 ± 9.9

79.6 ± 9.9

95.4 ± 1.8

96.4 ± 1.6

92.0 ± 2.3

Class

Operation

Context Modification

Overall

Original Generate a Python function called softmax that computes the softmax of an input tensor along a given dimension, with parameters input, dim, _stacklevel, and dtype. Include a note recommending log_softmax for NLLLoss compatibility.

Mutant Can you generate a Python function called softmax? Does it take input, dim, _stacklevel, and dtype? Should it include a note about using log_softmax for NLLLoss?

(a) Altered semantic Original Define a Python function that plots a normal distribution with given mean and standard deviation, overlays a histogram of random samples, titles it Normal Distribution, and returns the figure.

Mutant Can you define a Python function that plots a normal distribution with given mean and standard deviation, overlays a histogram of random samples, titles it Normal Distribution, and returns the figure?

(b) Preserved semantic

Fig. 5: Examples of Declarative to Interrogative mutation. correctness rate of 96.4 ± 1.6%. The only exception is for the declarative to interrogative mutation, where the semantic correctness drops to just below 80%. Figure 5a shows an example where the mutation alters a clear instruction into many questions that allow the model to choose which details to implement. However, in most cases, this mutation operation is still able to modify the description correctly (example in Figure 5b), where the interrogative form is a request and the details are not given as questions. If both criteria are considered, the correctness rate is still reasonable at 92.0 ± 2.3%. These result shows that GPT4o can be relied upon for a wide range of prompt mutations, particularly those involving verb transformations, tense shifts, and casing changes. However, naming-based modifications require intensive verification to ensure full compliance, and modality-changing transformations (Declarative-toInterrogative) remain unreliable without additional validation. Finding 1: LLM-based mutations have overall high reliability with an average of 96.4 ± 1.6% semantic equivalence. B. RQ2: Can SPARK improve zero-shot prompting in code generation? This question investigates if SPARK can utilize prompt mutations alongside the original prompt to improve generated code quality and correctness. Table III reports the pass@1

when comparing between the original prompt (Zero-Shot) and SPARK with single mutation or all mutations. On LiveCodeBench, nearly all mutation types boost performance across the three models. For O4-Mini, gains are broad and consistent, with the strongest improvements from syntactic shifts such as Lowercase to Uppercase (+3.1%), Active-toPassive (+4.4%), and Passive-to-Active (+5.2%). DeepSeek R1 shows even larger swings, with substantial gains from emphasizing or rephrasing text (e.g., Star Highlight +9.1%, Unrelated Function Name +7.8). Gemini 2.5, though more conservative, also benefits—particularly from Declarative-toInterrogative (+4.8%). These results indicate that relatively minor mutations can reliably expand the search space of correct solutions for natural-language-heavy coding tasks. On BigCodeBench, improvements are smaller and less uniform. O4-Mini and DeepSeek R1 both benefit modestly from substitutions and voice changes (+0.7% to +2.0%), while Gemini 2.5 remains more fragile, with several mutations producing negligible or even negative shifts (–1.4% with Lowercase to Uppercase, –0.5% with Remove Function Name). Still, targeted changes like Declarative-to-Interrogative (+0.6%) occasionally yield incremental improvements, suggesting some value even in highly structured algorithmic problems. On DL-Bench, the effects are modest but steady. O4-Mini shows consistent gains, with the highest gain of +2.5 %. DeepSeek R1 improves most with Active-to-Passive (+2.0) and maintains positive deltas across nearly all other mutations, albeit smaller. Gemini 2.5 again shows sensitivity: Declarativeto-Interrogative and Passive-to-Active both deliver +2.5%, while careless renaming or excessive formatting can reduce performance (–1.0% with Remove Function Name, –1.1% with Past/Future tense shifts). The overall pattern indicates that even in the complex domain of deep learning code, single controlled mutations help the consensus-based validation uncover additional correct outputs, though the absolute gains are more narrow when compared to general coding tasks. Overall, these findings show that SPARK with a single mutation is a broadly effective yet model and benchmarksensitive strategy. The strongest benefits arise from linguistic transformations that change surface form without altering

TABLE III: Pass@1 (%) comparison between original prompt and SPARK with single mutation or all mutations. The deltas vs. Zero-shot are highlighted. LiveCode Bench

Benchmark

BigCode Bench

DL-Bench

O4-Mini

DeepSeek R1

Gemini 2.5

O4-Mini

DeepSeek R1

Gemini 2.5

O4-Mini

DeepSeek R1

Gemini 2.5

Original Prompt

61.4

51.5

61.0

30.4

29.1

29.1

35.4

34.1

35.4

SPARK w/ Remove Function Name SPARK w/ Unrelated Function Name SPARK w/ Adversarial Function Name SPARK w/ Rename Variables SPARK w/ Lowercase to Uppercase SPARK w/ Uppercase to Lowercase SPARK w/ Star Highlight SPARK w/ Uppercase Highlight SPARK w/ Active to Passive SPARK w/ Passive to Active SPARK w/ Present to Future Tense SPARK w/ Present to Past Tense SPARK w/ Verb to Similar Verb SPARK w/ Declarative to Interrogative

63.6 (+2.2↑) 63.6 (+2.2↑) 65.0 (+3.6↑) 66.6 (+5.2↑) 64.5 (+3.1↑) 65.8 (+4.4↑) 64.1 (+2.7↑) 64.5 (+3.1↑) 65.8 (+4.4↑) 66.6 (+5.2↑) 65.0 (+3.6↑) 60.1 (-1.3↓) 66.2 (+4.8↑) 65.8 (+4.4↑)

57.1 (+5.6↑) 59.3 (+7.8↑) 60.6 (+9.1↑) 54.9 (+3.4↑) 52.8 (+1.3↑) 55.4 (+3.9↑) 60.6 (+9.1↑) 56.2 (+4.7↑) 56.2 (+4.7↑) 54.9 (+3.4↑) 58.4 (+6.9↑) 58.4 (+6.9↑) 54.9 (+3.4↑) 53.4 (+1.9↑)

62.2 (+1.2↑) 65.8 (+4.8↑) 64.0 (+3.0↑) 60.6 (-0.4↓) 63.2 (+2.2↑) 62.3 (+1.3↑) 63.6 (+2.6↑) 62.3 (+1.3↑) 64.0 (+3.0↑) 63.2 (+2.2↑) 64.0 (+3.0↑) 59.7 (-1.3↓) 63.6 (+2.6↑) 65.8 (+4.8↑)

32.4 (+2.0↑) 30.4 (—) 31.1 (+0.7↑) 31.1 (+0.7↑) 30.4 (—) 29.7 (-0.7↓) 31.1 (+0.7↑) 32.4 (+2.0↑) 31.1 (+0.7↑) 31.1 (+0.7↑) 31.1 (+0.7↑) 27.7 (-2.7↓) 32.4 (+2.0↑) 31.1 (+0.7↑)

29.7 (+0.6↑) 29.1 (—) 29.7 (+0.6↑) 29.7 (+0.6↑) 30.4 (+1.3↑) 29.1 (—) 31.1 (+2.0↑) 30.4 (+1.3↑) 31.1 (+2.0↑) 31.1 (+2.0↑) 29.7 (+0.6↑) 29.7 (+0.6↑) 31.1 (+2.0↑) 30.4 (+1.3↑)

28.6 (-0.5↓) 29.7 (+0.6↑) 29.1 (—) 29.1 (—) 27.7 (-1.4↓) 29.1 (—) 29.1 (—) 29.7 (+0.6↑) 29.7 (+0.6↑) 29.7 (+0.6↑) 29.1 (—) 29.1 (—) 29.7 (+0.6↑) 29.7 (+0.6↑)

35.4 (—) 36.9 (+1.5↑) 36.9 (+1.5↑) 34.3 (-1.1↓) 37.9 (+2.5↑) 35.4 (—) 37.9 (+2.5↑) 36.3 (+0.9↑) 37.4 (+2.0↑) 37.4 (+2.0↑) 36.9 (+1.5↑) 34.3 (-1.1↓) 35.0 (-0.4↓) 37.9 (+2.5↑)

34.6 (+0.5↑) 34.3 (+0.2↑) 34.6 (+0.5↑) 35.2 (+1.1↑) 35.4 (+1.3↑) 35.1 (+1.0↑) 34.1 (—) 35.4 (+1.3↑) 35.2 (+1.1↑) 35.4 (+1.3↑) 33.9 (-0.2↓) 35.4 (+1.3↑) 36.1 (+2.0↑) 35.2 (+1.1↑)

36.2 (+0.8↑) 36.4 (+1.0↑) 35.0 (-0.4↓) 34.4 (-1.0↓) 35.6 (+0.2↑) 36.2 (+0.8↑) 35.4 (—) 35.8 (+0.4↑) 37.9 (+2.5↑) 37.9 (+2.5↑) 34.3 (-1.1↓) 35.4 (—) 36.6 (+1.2↑) 36.6 (+1.2↑)

Model

Avgerage

64.8 (+3.4↑)

56.6 (+5.1↑)

63.2 (+2.2↑)

30.9 (+0.5↑)

30.2 (+1.1↑)

29.2 (+0.1↑)

36.4 (+1.0↑)

35.0 (+0.9↑)

36.0 (+0.6↑)

SPARK w/ All Mutations

67.3 (+5.9↑)

57.8 (+6.3↑)

63.2 (+2.2↑)

31.1 (+0.7↑)

32.4 (+3.3↑)

29.1 (—)

36.3 (+0.9↑)

32.2 (-1.8↓)

37.9 (+2.5↑)

semantics (voice changes, interrogatives), whereas arbitrary renaming or over-stylization risks degrading performance. If applying all mutations together, SPARK provides a significant uplift from using the original prompt with consistent improvement across all datasets and models. However, in some cases, the gain is less than applying a certain individual mutation. This indicates that selecting an appropriate mutation set for each situation is important to produce the best result. Finding 2: SPARK with single mutation broadly improves the original prompt with average gain between +0.6% and +5.1% across benchmarks and models. Apply all mutations together overall always improve the original prompt; however, the magnitude and reliability of improvements are not optimal, highlighting the need for a better mutation selection. C. How does SPARK compare to the state-of-the-art baseline? In this research question, we investigate if SPARK can improve the current state-of-the-art zero-shot prompting techniques. Specifically, we compare SPARK with the original prompt, the chain-of-thought prompting, and CodeT. CodeT also utilizes the dual execution agreement on multiple generations from the original prompt. In this experiment, we use eight generation queries for each sample, with CodeT, as more queries do not improve the performance. Table IV presents pass@1 for SPARK and baselines across benchmarks and models. The difference vs. the best baseline is highlighted in brackets. When using all available mutations, SPARK fails to improve the best baseline on four out of nine benchmark and model combinations. Findings from RQ2 indicate that selecting an appropriate set of mutations can optimize SPARK performance. In this RQ, we explore a tuning strategy where techniques can be tuned with a validation set and evaluated on a test set. Specifically, we select the best mutation set for SPARK in one of the three datasets (i.e., tuned with a validation dataset), and evaluated on the other two datasets.

Table IV also presents the tuning evaluation result. For all models across all benchmarks, combining multiple prompt mutations through tuning helps SPARK consistently match or surpass the strongest baselines (on average up to +4.1%). SPARK ’s average gains are statistically significant with the Wilcoxon Signed Rank test p-value of 0.03. Overall, combining different mutation types—such as rephrasing, verb-tense changes, and function renaming—encourages models to explore a wider and more diverse solution space. This strategy reduces overfitting to specific phrasing patterns and increases robustness of code generation, ensuring more reliable Pass@1 performance across heterogeneous benchmarks. Finding 3: Tuned SPARK can consistently match or surpass the strongest baselines (on average up to +4.1%) by leveraging the diversity introduced by multiple prompt mutations. V. Discussion How prompt mutations improve SPARK code generation? To investigate how mutations can help SPARK improve code generation, we manually evaluated a random sample of 100 task instances. For each instance, we inspected SPARK output, the strongest non-selected candidate(s), and the most discriminative tests, then annotated the primary improvement mechanism(s) using a compact taxonomy: • Diverse generated code: where prompt diversity (different voice, verbs, names) leads to generated code diversity (i.e., different algorithms or data structures). • Diverse generated test: where prompt diversity (different voice, verbs, names) leads to generated test diversity (i.e., more complex and comprehensive tests). • Look up prevention: where function renaming mutations prevent models from looking up code directly from the original function name, and instead focus on the task description. • Edge case tests: where mutations produce tests that stress edge cases such as min/max, empty, single-item, special Unicode characters, and large inputs.

TABLE IV: Pass@1 (%) comparison between SPARK and baselines. The deltas vs. the best baseline are highlighted. Benchmark

LiveCode Bench

BigCode Bench

DL-Bench

Model

O4-Mini

DeepSeek R1

Gemini 2.5

O4-Mini

DeepSeek R1

Gemini 2.5

O4-Mini

DeepSeek R1

Gemini 2.5

Original Prompt Chain-of-Thought CodeT

61.4 63.6 66.6

51.5 52.8 55.8

61.0 61.4 64.0

30.4 30.4 32.4

29.1 28.4 31.1

29.1 29.1 25.6

35.4 35.8 37.4

34.1 34.3 35.4

35.4 35.6 37.9

SPARK w/ All Mutations

67.3 (+0.7↑)

57.8 (+2.0↑)

63.2 (-0.8↓)

31.1 (-1.3↓)

32.4 (+1.4↑)

29.1 (—)

36.3 (-1.1↓)

32.2 (-3.2↓)

37.9 (—)

SPARK with Tuning

66.7 (+0.1↑)

59.3 (+3.5↑)

65.8 (+1.8↑)

32.4 (—)

31.1 (—)

29.7 (+4.1↑)

37.9 (+0.5↑)

35.4 (—)

38.0 (+0.1↑)

TABLE V: Improvement mechanisms by mutation type. Each row shows the count of instances where a given mutation type contributed to a specific improvement mechanism. An empty cell indicates a count of 0. Improvement mechanism

Diverse generated code Diverse generated test Look up prevention Edge case tests Spec clarification

Context

Styling

Verb

Tone

AFN

RFN

RV

UFN

UC

LC

UH

SH

A2P

P2A

2F

2P

V2V

D2I

6 1 5

12 3 10 2 1

15 11 4 4

6 9 4 2

5 3

6 1

1

12 10

13 6

2 1

7 3

4 2

4 1

15 8

2

4

1 2

2

6

Total

108 59 23 15 11

TABLE VI: Average token usage per sample across datasets for original prompts and SPARK with single mutation. Dataset

LiveCode Bench BigCode Bench DL Bench

Original Prompt 2,457.3 800.7 2,744.8

Context

Styling

Verb

Tone

w/ RFN

w/UFN

w/AFN

w/RV

w/UC

w/LC

w/SH

w/UH

w/A2P

w/P2A

w/2F

w/2P

w/V2V

w/D2I

10,666.3 3,205.8 7,409.6

10,689.5 3,197.5 7,397.5

11,449.6 3,225.5 8,070.8

10,704.1 3,225.1 7,438.1

11,480.8 3,344.9 8,453.4

9,240.3 3,223.4 8,448.9

10,164.5 2,829.5 7,434.8

9,479.8 4,445.6 8,511.9

8,658.1 3,279.9 8,076.9

10,019.1 3,471.3 7,489.6

9,826.2 3,230.1 8,483.1

12,327.9 3,218.6 8,466.2

10,822.5 3,234.6 8,846.1

11,376.6 3,314.5 8,076.9

• Spec clarification: where changing tone or style clarifies

hidden rules so new tests expose those details and reward code that truly matches the intended specification. An instance could receive up to three tags if multiple mechanisms jointly explained the gain. We used the agreement matrix and hard-test list to ground each label, and we cross-checked annotations by reviewing code diffs and failing/passing test clusters. Table V reports the result. Overall, the two most common improvements were Diverse generated code and Diverse generated test. First, semantic-preserving mutations, such as Context Modification mutations (AFN, UFN, RFN, RV) and several Verb Modifications (A2P, V2V), elicited meaningful implementation deviations from the norm (e.g., switching from ad-hoc parsing to structured APIs) which SPARK favored because they agreed on a broader and more discriminative subset of tests. Second, some mutation families (e.g., styling edits like UC, LC, highlights) can trigger more diverse test case generation that better capture the semantics of each task, which can help the consensus-based validation identify more correct code solutions. Function name mutations, such as AFN, RFN, or UFN, prevent LLMs from generating code with hard-coded identifiers, positional parsing, or case-sensitive comparisons by reducing LLMs’ direct lookup from their training data (Lookup prevention). For example, figure 7 shows that subtle renaming of the function from left_jacobian to right_jacobian forces the model to generate new code instead of looking up existing code with the common function name. The first code snippet, generated from the original prompt, includes a simple small-angle approximation that may lead to errors.

Original Create a non-blocking echo server that appends the current time to received data, sends it back to the client, handles exceptions, and returns a status message with the server's run duration.

def task_func( ... ): server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) ... end_time = start_time + timedelta(seconds=run_duration) while datetime.now() < end_time: readable, writable, exceptional = select.select(inputs, outputs, inputs, 1.0) for s in exceptional: ... server_socket.close()

Mutant Create a **non-blocking echo server** that appends the **current time** to received data, sends it back to the client, and handles **exceptional conditions**. The function should return a **status message** with the server's **run duration**

def task_func( ... ): end_time = start_time + timedelta(seconds=run_duration) while inputs and datetime.now() < end_time: readable, writable, exceptional = select.select(inputs, outputs, inputs, 1.0) for s in exceptional: ... for s in inputs: s.close()

Fig. 6: An example of a mutation that increases diversity in the code and checks the boundary conditions. Original Generate a Python function named left_jacobian to compute the left Jacobian of SO(3) for a batch of rotation vectors (axisangle tensors).

def left_jacobian(vec: torch.Tensor) -> torch.Tensor: B = vec.shape[0] ... A_small = 0.5 - theta2 / 24.0 B_small = 1.0 / 6.0 - theta2 / 120.0 ... mask = theta <= eps A = torch.where(mask, A_small, A) B_coef = torch.where(mask, B_small, B_coef)

Mutant Generate a Python function named right_jacobian to compute the left Jacobian of SO(3) for a batch of 3D vectors.

def right_jacobian(vec): B = vec.shape[0] ... eps = 1e-8 a = torch.where( theta2 < eps, 0.5 - theta2 / 24, (1 - torch.cos(theta)) / theta2) b = torch.where( theta2 < eps, 1.0 / 6 - theta2 / 120, (theta - torch.sin(theta)) / theta3) a = a.view(B, 1, 1) b = b.view(B, 1, 1) ...

Fig. 7: Slight function name variations improve mathematical stability and precision and prevent look up.

The second snippet, prompted by the variation, uses a more robust mathematical formulation, particularly for small-angle handling, using Taylor expansions for more accurate calculations. Edge case tests and Spec clarification are some other benefits of prompt mutation where mutations emphasize edge conditions, and clarification helps improve code and test candidates generation. Overall, across the 100 manually reviewed instances, we observed a consistent pattern: prompt mutants help trigger diverse code and test generation while preventing the model from performing “look up” and instead generating new code patterns that are more generalizable and robust. SPARK ’s cost compared to the original prompt: Table VI presents the average token usage per sample across datasets for original prompts and SPARK with a single mutation, which indicates token usage increases associated with mutated prompts (up to four times compared to the original prompts). However, this cost is compatible with prior approaches such as CodeT that often require more than four times the token usage due to ineffective variance generation without mutation. VI. Related work Code Generation: Code generation tasks are categorized into three paradigms: description-to-code, code-to-description, and code-to-code transformations [30], [31]. Solution ranging from recurrent neural networks (RNNs) [32] to transformers, and convolutional neural networks (CNNs) [33]. Recent transformer-based models, such as Codex and AlphaCode [34], have demonstrated remarkable capabilities, particularly in competitive programming. Li et al. [34] introduce AlphaCode, a transformer-based system capable of generating multiple diverse code candidates, which achieves human-level performance in competitive programming By conditioning generation on bidirectional context, InCoder [35] significantly improves zero-shot performance across tasks such as type inference, comment generation, and variable renaming, effectively bridging pure synthesis and code editing tasks. Additionally, CODET [15] leverages dual execution agreement on both solutions and comprehensive test cases generation to enhance solution selection accuracy. Recently, foundation models such as ChatGPT or Copilot have been shown to have practical utility in code generation, but with some limitations [36]. Specifically, these tools are predominantly utilized in smaller, individually maintained projects involving simpler, low-complexity code, requiring minimal modifications. Test Generation: Traditional techniques for automated test case generation have primarily relied on search-based [37], constraint-driven [38], and random strategies [39], with the primary goal of maximizing code coverage. Recently, LLMs have been utilized for automatic generation of human-like unit tests using pre-trained and fine-tuned models [40], [41], [42], [43], [44], reinforcement learning [45], and hybrid approaches combining LLMs with search-based methods [37]. Despite encouraging progress, LLM-generated test cases often suffer from correctness issues [46], [47], [48] and demonstrate

relatively low coverage on certain benchmarks. These limitations highlight the importance of continued refinement and systematic comparisons with established frameworks such as EvoSuite [49], [50] and Pynguin [51], [50]. Prompt Mutation: Prompt mutations are effectively utilized in LLM-assisted test and code generation [52], [53], [54], [55], [56]. Specifically, skeleton prompts enforce generation uniformity [53] or iterative human-in-the-loop refine requirement [52]. ReCode [54] and ADVPRO [55], respectively, present robustness-enhancement and paraphrase prompt mutation techniques. Differing from ReCode and ADVPRO, SPARK applies a broader set of sentence-level mutations and operates entirely on natural-language prompt mutations. Finally, Chen et al [56] explore the mutations from the perspective of human-induced variance in semantically similar prompts and places a significant emphasis upon minor prompt perturbations. SPARK instead utilizes real-world natural language variance to get semantically similar prompts that can generate diverse code and tests. VII. Threats to Validity Firstly, using the same LLM to produce both solutions and tests reduces the configuration effort but risks circularity: if the model generates flawed or overly simplistic tests, incorrect solutions may pass evaluation. Although our mutation strategies seek to mitigate this by inducing diversity, they cannot guarantee absolute coverage of all edge cases or complex requirements found in real-world software. Secondly, our carefully designed diverse set of LLM-based prompt mutations to simulate real-world variation in problem descriptions might not capture the full range or nuance of naturally occurring variances. Additionally, the manually validated sample set indicates GPT-4 has high accuracy; it may still introduce unintended biases or fail to preserve the original task semantics perfectly. Finally, while the three evaluation benchmarks, LiveCodeBench, BigCodeBench, and DLBench, offer complementary coverage—from algorithmic puzzles to real-world API usage and deep learning code—they may not fully represent all software engineering contexts or programming languages. VIII. Conclusion In this work, we presented SPARK , a framework that leverages semantic-preserving prompt mutations and consensusbased verification to enhance the reliability of LLM-driven code generation. Evaluation showed that SPARK consistently improved pass@1 accuracy when controlled linguistic and structural mutations diversified candidate solutions and strengthened consensus selection. Our analysis revealed complementary benefits across mutation classes while highlighting trade-offs in token cost when applying larger mutation sets. Overall, SPARK provides a practical and extensible approach to improving code generation. References [1] A. Odeh, N. Odeh, and A. S. Mohammed, “A comparative review of ai techniques for automated code generation in software development: advancements, challenges, and future directions,” TEM Journal, 2024.

[2] J. Shin and J. Nam, “A survey of automatic code generation from natural language,” JIPS, 2021. [3] J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” arXiv, 2024. [4] B. Chen and Z. Abedjan, “Duetcs: Code style transfer through generation and retrieval,” in 2023 ICSE, 2023. [5] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” TOSEM, 2024. [6] A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” in 2023 ICSE-FoSE, 2023. [7] B. A. Becker, P. Denny, J. Finnie-Ansley, A. Luxton-Reilly, J. Prather, and E. A. Santos, “Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation,” in SIGCSE, 2023. [8] T. Ridnik, D. Kredo, and I. Friedman, “Code generation with alphacodium: From prompt engineering to flow engineering,” arXiv, 2024. [9] Z. Lyu, X. Li, Z. Xie, and M. Li, “Top pass: improve code generation by pass@ k-maximized code ranking,” Frontiers of CS, 2025. [10] J. Liu, C. S. Xia, Y. Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” NeurIPS, 2023. [11] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv, 2022. [12] J. Shin, R. Aleithan, H. Hemmati, and S. Wang, “Retrieval-augmented test generation: How far are we?” arXiv, 2024. [13] W. Wang, C. Yang, Z. Wang, Y. Huang, Z. Chu, D. Song, L. Zhang, A. R. Chen, and L. Ma, “Testeval: Benchmarking large language models for test case generation,” arXiv, 2024. [14] D. Xu, “A tool for automated test code generation from high-level petri nets,” in Petri Nets, 2011. [15] B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “CodeT: Code generation with generated tests,” in ICLR, 2023. [16] S. Bonandin and M. Herty, “Consensus-based algorithms for stochastic optimization problems,” arXiv, 2024. [17] M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM, 1981. [18] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” arXiv, 2024. [19] T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul et al., “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” arXiv, 2024. [20] A. Daghighfarsoodeh, C.-Y. Wang, H. Taherkhani, M. Sepidband, M. Abdollahi, H. Hemmati, and H. V. Pham, “Dl-bench: Deep learning benchmark dataset for code generation,” arXiv, 2025. [21] OpenAI, “Openai o3 and o4-mini system card,” 2024. [22] Z. Deng, W. Ma, Q.-L. Han, W. Zhou, X. Zhu, S. Wen, and Y. Xiang, “Exploring deepseek: A survey on advances, applications, challenges and future directions,” IEEE/CAA Journal of Automatica Sinica, 2025. [23] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv, 2023. [24] J. Pantiuchina, M. Lanza, and G. Bavota, “Improving code: The (mis) perception of quality metrics,” in ICSME, 2018. [25] T. Heričko and B. Šumak, “Exploring maintainability index variants for software maintainability measurement in object-oriented systems,” Applied Sciences, 2023. [26] H. Lee, G. An, and S. Yoo, “Metamon: Finding inconsistencies between program documentation and behavior using metamorphic llm queries,” in LLM4Code, 2025. [27] Z. Li and D. Shin, “Mutation-based consistency testing for evaluating the code understanding capability of llms,” in CAIN, 2024. [28] A. R. Feinstein and D. V. Cicchetti, “High agreement but low kappa: I. the problems of two paradoxes,” J Clin Epidemiol, 1990. [29] K. L. Gwet, “Computing inter-rater reliability and its variance in the presence of high agreement,” Br. J. Math. Stat. Psychol., 2008.

[30] E. Dehaerne, B. Dey, S. Halder, S. De Gendt, and W. Meert, “Code generation using machine learning: A systematic review,” IEEE Access, 2022. [31] M. Abdollahi, R. Zhang, N. S. Harzevili, J. Shin, S. Wang, and H. Hemmati, “Surveying the benchmarking landscape of large language models in code intelligence,” TOSEM, 2025. [32] A. Sherstinsky, “Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,” Physica D, 2020. [33] L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. AlShamma, J. Santamaría, M. A. Fadhel, M. Al-Amidie, and L. Farhan, “Review of deep learning: concepts, cnn architectures, challenges, applications, future directions,” Journal of big Data, 2021. [34] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago et al., “Competition-level code generation with alphacode,” Science, 2022. [35] D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” arXiv, 2022. [36] X. Yu, L. Liu, X. Hu, J. W. Keung, J. Liu, and X. Xia, “Where are large language models for code generation on github?” arXiv, 2024. [37] M. Harman and P. McMinn, “A theoretical and empirical study of search-based testing: Local, global, and hybrid search,” TSE, 2009. [38] X. Xiao, S. Li, T. Xie, and N. Tillmann, “Characteristic studies of loop problems for structural test generation via symbolic execution,” in ASE, 2013. [39] C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, “Feedback-directed random test generation,” in ICSE, 2007. [40] P. Delgado-Pérez, A. Ramírez, K. J. Valle-Gómez, I. Medina-Bulo, and J. R. Romero, “Interevo-tr: Interactive evolutionary test generation with readability assessment,” TSE, 2022. [41] N. Rao, K. Jain, U. Alon, C. Le Goues, and V. J. Hellendoorn, “Cat-lm training language models on aligned code and tests,” in ASE, 2023. [42] H. Taherkhani and H. Hemmati, “Valtest: Automated validation of language model generated test cases,” arXiv, 2024. [43] Z. Xie, Y. Chen, C. Zhi, S. Deng, and J. Yin, “Chatunitest: a chatgptbased automated unit test generation tool,” arXiv, 2023. [44] A. M. Dakhel, A. Nikanjam, V. Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing,” IST, 2024. [45] B. Steenhoek, M. Tufano, N. Sundaresan, and A. Svyatkovskiy, “Reinforcement learning from automatic feedback for high-quality unit test generation,” in DeepTest, 2025. [46] Z. Yuan, Y. Lou, M. Liu, S. Ding, K. Wang, Y. Chen, and X. Peng, “No more manual tests? evaluating and improving chatgpt for unit test generation,” arXiv, 2023. [47] V. Li and N. Doiron, “Prompting code interpreter to write better unit tests on quixbugs functions,” arXiv, 2023. [48] V. Guilherme and A. Vincenzi, “An initial investigation of chatgpt unit test generation capability,” in SAST, 2023. [49] Y. Tang, Z. Liu, Z. Zhou, and X. Luo, “Chatgpt vs sbst: A comparative assessment of unit test suite generation,” TSE, 2024. [50] S. Bhatia, T. Gandhi, D. Kumar, and P. Jalote, “Unit test generation using generative ai: A comparative performance analysis of autogeneration tools,” in LLM4Code, 2024. [51] C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in 2023 ICSE, 2023. [52] L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, J. Zhou, G. Liang, Q. Wang, and J. Chen, “On the evaluation of large language models in unit test generation,” in ASE, 2024. [53] A. Lops, F. Narducci, A. Ragone, and M. Trizio, “Agonetest: Automated creation and assessment of unit tests leveraging large language models,” in ASE, 2024. [54] S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V. Kumar, S. Tan, B. Ray, P. Bhatia, R. Nallapati, M. K. Ramanathan, D. Roth, and B. Xiang, “ReCode: Robustness evaluation of code generation models,” in ACL. ACL, 2023. [55] X. Li, G. Meng, S. Liu, L. Xiang, K. Sun, K. Chen, X. Luo, and Y. Liu, “Attribution-guided adversarial code prompt generation for code completion models,” ser. ASE ’24. ACM, 2024. [56] J. Chen, L. Zhenhao, H. Xing, and X. Xin, “Nlperturbator: Studying the robustness of code llms to natural language variations.” ACM Trans. Softw. Eng. Methodol., 2025.

Related documents

• MUCOCO: Automated Consistency Testing of Code LLMs#924550
• MUCOCO: Automated Consistency Testing of Code LLMs#920318
• COVE: A Task-Adaptive, Code-Based Ensemble for Proactive Verification of LLM Outputs#1587121
• COVE: A Task-Adaptive, Code-Based Ensemble for Proactive Verification of LLM Outputs#1528042
• AutoINV: Automated Invariant Generation Framework for Formal Verification on High-Level Synthesis Designs#134622