M U C O C O: Automated Consistency Testing of Code LLMs
arXiv:2604.19086v1 [cs.SE] 21 Apr 2026
Chua Jin Chou1 Khant That Lwin2 Ezekiel Soremekun1 1 Singapore University of Technology and Design (SUTD) 2 Royal Holloway University of London (RHUL) {jinchou_chua, ezekiel_soremekun}@sutd.edu.sg [email protected]
Abstract
tent. Particularly, lack of consistency may lead to fatal consequences in critical use cases. Practitioners currently employ manually curated coding benchmarts to assess Code LLMs. Several benchmarks have been proposed to measure the effectiveness of Code LLMs, including HumanEval (Chen et al., 2021), MBPP (Odena et al., 2021), CodeMMLU (Nguyen et al., 2025), CruxEval (Gu et al., 2024) and BigCodeBench (Zhuo et al., 2025a). Likewise, many leaderboards measure/rank LLMs performance on varying coding tasks (Liu et al., 2023; llm stats, 2025; Zhuo et al., 2025b; Zhang et al., 2025; Fang et al., 2025). However, existing benchmarks are manually curated and focus on measuring the correctness of LLMs (Guan et al., 2025; Chen et al., 2021; Zhuo et al., 2025a; Gu et al., 2024; Nguyen et al., 2025; Odena et al., 2021). Manual benchmark curation is challenging and these benchmarks do not apply to consistency property. Besides, benchmarks quickly leak to newer generation of LLMs over time (Guan et al., 2025). Beyond correctness, it is important to assess non-functional properties of Code LLMs like consistency, since LLMs often produce inconsistent results across multiple runs, varying prompts or even similar tasks (Rajan et al., 2024). Inconsistent LLM behaviors may have severe consequences during programming. Researchers have observed that Code LLMs may generate incorrect programs for similar code queries, syntax or logic (Guan et al., 2025). Table 1 highlights an example of inconsistent behaviors in state-of-the-art Code LLMs. To this end, this work poses the following scientific question: How can we automatically assess the consistency property of Code LLMs? We propose an automated technique for assessing the con-
Code LLMs often portray inconsistent program behaviors. Developers typically employ benchmarks to assess Code LLMs, but most benchmarks are hand-crafted, static and do not target consistency property. In this work, we pose the scientific question: how can we automatically discover inconsistent program behaviors in Code LLMs? To address this challenge, we propose an automated consistency testing method, called M U C O C O, which employs semantic-preserving mutation analysis to expose inconsistent behaviors in code LLMs. Given a coding query, M U C O C O automatically transforms its program into semantically equivalent programs (aka mutants) and detects inconsistencies between the mutants and the original program (e.g., different output or test failure). We evaluate M U C O C O using four (4) coding tasks and seven (7) LLMs. Results show that M U C O C O is effective in exposing inconsistency and outperforms the closest baseline (T URBULENCE). About one in seven (15%) inputs generated by M U C O C O exposed inconsistencies. Our work motivates the need to test Code LLMs for consistency property.
1
Introduction
Software practitioners are increasingly employing Code Large Language Models (Code LLMs) for programming and software engineering (SE) tasks. Similarly, researchers are increasingly deploying LLMs to address SE tasks, such as code generation and execution prediction (Fakhoury et al., 2024), with remarkable performance (Bouzenia et al., 2025; Jin et al., 2023; He et al., 2025). Given the increasingly high reliance on LLMs for coding tasks, it is important to assess the performance of LLMs and ensure that they are reliable and consis1
Table 1: S AMPLE CONSISTENCY TEST GENERATED BY M U C O C O AND ITS LLM O UTPUTS . (✓= CONSISTENCY, ✗= INCONSISTENCY, OG = LLM A NSWER FOR O RIGINAL Q UESTION , MU = LLM A NSWER FOR M UTATED Q UESTION ). A PPENDIX (TABLE 9) PROVIDES MORE EXAMPLES FOR EACH MUTATION TYPE . Mutation Type
Original Question 1 2 3 4
Random
5 6 7 8 9 10
def f( text , n ): if n < 0 or len ( text ) <= n: return text result = text [0: n ] i = len ( result ) - 1 while i >= 0: if result [ i] != text [i ]: break i -= 1 return text [0: i + 1]
Mutated Question 1 2 3 4 5 6 7 8 9 10
def dOdtZZSuisMB ( TYesQRBm , mXDbZtJMZFV ) : if mXDbZtJMZFV < 0 or len ( TYesQRBm ) <= mXDbZtJMZFV : return TYesQRBm result = TYesQRBm [0: mXDbZtJMZFV ] i = len ( result ) - 1 while i >= 0: if result [i ] != TYesQRBm [ i ]: break i -= 1 return TYesQRBm [0: i + 1]
sistency of Code LLMs called M U C O C O1 . The main idea of our approach is to employ semanticpreserving mutation analysis to discover inconsistency in Code LLMs. Figure 1 illustrates the workflow of M U C O C O: Given an original dataset (e.g., HumanEval (Chen et al., 2021)), M U C O C O automatically mutates its code queries into semantically equivalent queries. Then, its metamorphic oracle detects consistency error (aka inconsistency) by examining the test correctness of the LLM outputs for the original query versus the mutated query. It discovers inconsistency when the LLM outputs of both queries differ, e.g., when the output of the mutated query fails a test but the output of the original passes, or vice versa. For instance, Table 1 shows that M U C O C O’s random renaming of CruxEval-789 (Gu et al., 2024) triggered consistency errors (✗) in five LLMs (GPT-4o, Qwen, DeepSeek, and CodeStral). In summary, our contributions are as follows:
2
GPT-4o GPT-5 Qwen Llama Gemma DeepSeek Codestral (OG) (OG) (OG) (OG) (OG) (OG) (OG) (MU) (MU) (MU) (MU) (MU) (MU) (MU)
CruxEval_789 (’br’)
✗ (’br’) ([’bR’])
✓
✗ (’br’) ([’bR’])
✓
✗ (’br’) ([’bR’])
✗ (’br’) ([’bR’])
✗ (’br’) ([’bR’])
Overview
Code Consistency Property: Table 2 highlights the types of inconsistency detected by M U C O C O. Given a consistency test (a pair of semanticallyequivalent code queries), we say that an LLM is inconsistent (✗) if one of these conditions is met: • the LLM output is correct (✓ ⃝) for a coding query, but incorrect (⊗) for an equivalent query (aka correctness-based inconsistency). ⃝) or incorrect (⊗) • the LLM output is correct (✓ for a query, but invalid (") for an equivalent query (aka invalidity-based inconsistency). • LLM output for both queries are incorrect (⊗), albeit for different reasons, e.g., failed different tests (aka incorrectness-based inconsistency). Analogously, we say an LLM is consistent (✓) for a pair of semantically-equivalent code queries, ⃝), incorrect (⊗) or inif both outputs are correct (✓ valid (") for the same reasons, i.e., there is no test case that distinguishes both outputs. Consistency property is different from correctness property: Unlike traditional model performance properties (e.g., accuracy), (in)consistency does not require LLM outputs to be correct. In contrast to robustness property (Honarvar et al., 2025), (in)consistency property requires that code queries are semanticallyequivalent but produce different test outcomes. We determine correctness by (a) comparing the LLM output to the expected output, and (b) ensuring the LLM-generated outputs pass the test suite. Concretely, correctness (✓ ⃝) means that the LLM output matches the expected output and passes all test cases. Incorrectness (⊗) means the LLM output does not match the expected output or fails at least one test case. Invalidity (") means the LLM output violates the expected output specification, e.g., non-executable program or an empty output.
1. We propose an automated method (M U C O C O) that employs mutation analysis and metamophic testing to discover inconsistency in Code LLMs. M U C O C O provides 11 semantic-preserving mutations (see section 3 and Table 9). 2. We evaluate M U C O C O (section 4) using seven state-of-the-art LLMs and four benchmarks. We found that 14.82% (22K/148K) of inputs generated by M U C O C O reveals inconsistency in Code LLMs (RQ1, section 5). 3. We compare M U C O C O to a hand-crafted benchmark – T URBULENCE (Honarvar et al., 2025). Results demonstrate that M U C O C O is up to six (6) times more effective than T URBULENCE in consistency testing, and discovers new classes of consistency errors (RQ2, section 5). 1
Dataset (Canon. Ans.)
M U C O C O = Mutation-based Code Consistency Testing.
2
Table 2: C ONSISTENCY E RRORS BASED ON LLM O UTCOMES (OG = O RIGINAL , M UT = M UTATED ).
Figure 1: M U C O C O Workflow
An in-depth explanation of our correctness and inconsistency oracles is provided in Section 3.2. Key Insight: To automatically discover consistency errors, we propose an automated testing technique called M U C O C O. The main idea of our technique is to employ mutational analysis and metamorphic testing to detect consistency errors in Code LLMs. The key insight of M U C O C O is that
OG Task
Mut Task
(In)Consistency
Correct Incorrect
Incorrect Correct
Correctness-based Inconsistency (✗)
Incorrect Incorrect (different test outcomes)
Incorrectness-based Inconsistency (✗)
Correct Incorrect Invalid Invalid
Invalid Invalid Incorrect Correct
Invalidity-based Inconsistency (✗)
Correct Invalid
Correct Invalid
Consistent (✓) Consistent (✓)
Incorrect Incorrect (similar test outcomes)
Consistent (✓)
that targets consistency property in Code LLMs. It does not require manual effort/intervention and applies to arbitrary code benchmarks. Existing benchmarks are manually curated (Zhuo et al., 2025a; Gu et al., 2024; Nguyen et al., 2025) and do not apply to consistency testing. They often focus on the correctness of LLMs. Besides, static LLM benchmarks quickly leaked to newer models (Guan et al., 2025). Existing dynamic benchmarks (Guan et al., 2025) target the correctness property, while M U C O C O focuses on code consistency. T URBULENCE (Honarvar et al., 2025), provides program templates that can be instantiated to discover robustness errors in LLMs. T URBULENCE is the closest related work to our technique, but it is manually crafted and focuses on model robustness instead of consistency testing like M U C O C O. In this work, we compare M U C O C O to T URBULENCE and illustrate that M U C O C O outperforms and complements T URBULENCE (see RQ2 section 5). Similar to M U C O C O, KonTest (Rajan et al., 2024) employs mutation analysis and metamorphic properties to automatically assess consistency, albeit in text-based LLMs, it does not apply to Code LLMs.
a pair of code queries with the same semantics (meaning) should produce LLM outputs that exhibit similar behaviors. In particular, we posit that given a pair of semantically equivalent code queries, the LLM should produce a pair of outputs that exhibit similar behaviors, i.e., same outputs and test correctness. We hypothesize that if a pair of equivalent code queries produce differing LLM outputs or behaviors, then the LLM is inconsistent in handling the query. Leveraging this metamorphic property, we design M U C O C O to generate semantic-preserving code queries. Given an existing LLM input (e.g., prompt or code from HumanEval), M U C O C O mutates it into a semantic-preserving equivalent query, feeds them into the LLM and examines whether the behavior of the LLM is similar for both queries. M U C O C O employs lexical, syntactic and logical mutations that preserve the program semantics, e.g., variable renaming, for-to-while loop transformation and constant folding (Guan et al., 2025) (Orvalho and Kwiatkowska, 2025), respectively. Novelty w.r.t. the State-of-the-art: To the best of our knowledge, M U C O C O is the first automated method for assessing the consistency of Code LLMs. M U C O C O is unique with its combination of semantic-preserving mutations and metamorphic testing, and its focus on programming tasks. Specifically, M U C O C O is the first approach
3
Methodology
Figure 1 highlights M U C O C O’s workflow. 1) Original Benchmark Testing: M U C O C O’s consistency testing starts with retrieving the original benchmark dataset (e.g., HumanEval). Next, for each task in the benchmark, M U C O C O executes 3
Table 3: D ETAILS OF TASKS AND B ENCHMARKS
its canonical solution using the benchmark’s test suite to ensure dataset correctness. The LLM under test is fed the task, and its output is collected and stored. Then, M U C O C O executes the benchmark’s test suite on the LLM output and stores the results. Figure 1 highlights these steps in blue lines.
Original Task Multiple Choice Questions
HumanEval
Code Generation
CruxEval BigCodeBench
Input/Output Prediction Code Generation
3.2
2) Mutated Benchmark Testing: M U C O C O mutates each task in the original benchmark (e.g., HumanEval/1) for each semantic-preserving mutation operator. First, M U C O C O checks that the mutation operator is possible for the task at hand, e.g., it checks that there is a for loop in the program during a for-to-while mutation. It then mutates the task at hand using the desired mutation operator and the mutated task is tested against its corresponding test suite to ensure that correctness is preserved post-mutation. Next, the LLM is fed the mutated task and its results are collected and stored. These steps are outlined in Figure 1 using red lines.
MuCoCo Task Multiple Choice Questions Code Generation, Input/Output Prediction Input/Output Prediction Code Generation
Release Date
# Qns.
9 Apr 2025
164
14 Jul 2021
164
5 Jan 2024 1 Apr 2025
800 1140
M U C O C O Test Oracle
M U C O C O employs two test oracles: (1) Correctness Oracle: The correctness oracle determines the correctness of an LLM output. Correctness of LLM outputs are evaluated individually for each task. (2) Consistency Oracle: the performance of an LLM on a mutated dataset versus an original dataset. The test suite correctness of the mutated and original datasets are first evaluated using the correctness oracle. Next, each task in the original dataset is paired with its corresponding mutated version and their outputs and test suite correctness are compared. M U C O C O’s consistency oracle is applied to each pair of original and mutated tasks, and the resulting outcomes are aggregated to compute the consistency error rate (see Equation 1). Further details about the correctness and consistency oracles are provided in Appendix A. We also report additional experiments investigating the direction of inconsistency and invalidity in Appendix E.
3) Consistency Error Detection: M U C O C O detects a consistency error by comparing the LLM output and test suite correctness of the original task (step 1) versus the mutated task (step 2). A pair of original and mutated tasks is a consistency test case. It then computes the consistency error rate by aggregating the number of consistency test cases generated by M U C O C O and the number of consistency errors triggered by M U C O C O. This step is outlined in step 10 of Figure 1 and it is repeated for all valid code mutations for each task and dataset. 3.1
Dataset CodeMMLU (Code Completion)
4
Experimental Settings
4.1
Research Questions
We pose the following research questions (RQs): RQ1 Effectiveness: How effective is M U C O C O in discovering code inconsistency in LLMs? RQ2 Baseline Comparison: How does M U C O C O compare to T URBULENCE? RQ3 Probing Study: What is the impact of different mutators on M U C O C O’s effectiveness? We also report additional experiments in the Appendix, namely the effectiveness of multiple mutations (Appendix H), direction of inconsistency (Appendix E), distance of inconsistency (Appendix F) and impact of model confidence (Appendix I).
Semantic-preserving Mutations
M U C O C O’s provides 11 mutation operators categorised into three (3) groups, namely lexical, syntactic and logical mutations. Appendix B explains each mutation type/operator with examples. a) Lexical Mutations refer to mutations involving renaming of variable names, function names, or Python strings. Lexical mutations include random, sequential and literal format. b) Syntactic Mutations refers to semanticpreserving mutations that modify the program syntax with equivalent syntax, e.g., for-to-enumerate and for-to-while.
4.2
Coding Tasks and Benchmarks
We evaluated M U C O C O using four benchmarks, namely BigCodeBench (Zhuo et al., 2025a), CodeMMLU (Nguyen et al., 2025), HumanEval (Chen et al., 2021) and CruxEval (Gu et al., 2024) (see Table 3). These benchmakrs provide the following coding tasks: Multiple Choice Questions (MCQ), Input Prediction, Output Prediction, Code
c) Logical Mutations modifies the program logic in a manner that preserves the original program behavior, e.g. decomposing a constant (5) into equivalent operations (4 + 1). This includes DeMorgan mutation, boolean literal, commutative reorder and constant unfolding. 4
Table 4: M U C O C O’s Effectiveness Across LLMs showing inconsistency rate (Inc.) and Model Accuracy (Acc.). Mutation Type Logical Syntactic Lexical All (Avg.)
Qwen2.5 Inc. Acc. 9.74 76.23 6.72 77.91 16.17 67.63 12.73 71.86
Gemma-3 Inc. Acc. 15.52 66.13 12.30 68.78 20.48 60.66 17.69 63.60
DeepSeek-V3.2 Inc. Acc. 17.43 63.00 14.89 65.80 16.50 69.24 16.53 66.80
LLaMA-3.1 Inc. Acc. 18.02 49.88 15.47 55.70 24.71 48.41 21.22 50.00
GPT-5 Inc. Acc. 1.91 97.52 1.51 98.46 3.91 94.10 2.92 95.84
GPT-4o Inc. Acc. 19.48 64.25 18.00 65.75 17.44 68.90 18.15 66.99
Codestral Inc. Acc. 15.24 66.15 12.24 68.71 14.75 70.96 14.51 69.14
All Models Inc. Acc. 13.91 (6259/45008) 69.01 (31055/45000) 11.59 (2691/23220) 71.58 (16612/23209) 16.28 (12974/79707) 68.48 (54425/79471) 14.82 (21924/147935) 69.13 (102092/147680)
Table 5: M U C O C O’s Effectiveness across tasks showing Inconsistency Rate (Inc.) and Accuracy (Acc.). Mutation Type Logical Syntactic Lexical All (Avg.)
MCQ Inc. Acc. 38.50 46.48 16.95 67.80 10.04 78.36 22.35 64.04
Input Pred. Inc. Acc. 7.08 75.90 5.90 80.53 6.62 80.59 6.65 78.97
Output Pred. Inc. Acc. 18.52 64.16 16.88 62.89 20.11 61.84 18.99 62.82
Code Gen. Inc. Acc. – – – – 27.15 57.66 27.15 57.66
Generation. Appendix C provides additional details about our experimental setup and task procedures. 4.3
using the code base provided by the T URBULENCE authors and a pre-defined seed (1234). To ensure the validity of the generated parameters, they are inserted into their corresponding templates and evaluated against the template’s test suite. Parameters that successfuly pass all the test cases are then stored. During testing, these parameters are retrieved to recreate the T URBULENCE instances. There are 60 templates in the T URBU LENCE benchmark and 100 sets of parameters were generated for each template. In total, we generated 5612 valid sets of parameters using T URBULENCE. We note that the distribution of valid parameters across the templates is uneven.
LLMs under Test
We evaluate M U C O C O using seven (7) LLMs (see Table 14 in appendix for more details). We employ a diverse set of recent LLMs (2024-2025) including models with reasoning capabilities (GPT5 (OpenAI, 2025a)), varying model sizes (8-200 Billion) and architectures. We also employ opensource/weight models (Llama (Grattafiori et al., 2024a) and DeepSeek (DeepSeek, 2025)), proprietary models (GPT-4o (OpenAI, 2024)), and models that are trained on code instruction fine-tuning (QwenCoder (Hui et al., 2024b)).
5 4.4
Results
Metrics and Measures RQ1 Effectiveness: We examine the test effectiveness of M U C O C O using all models, datasets, tasks and mutation operators. Table 4, Table 18, and Table 6 highlights M U C O C O’s effectiveness across all tested models, tasks, and datasets, respectively. We found that M U C O C O is effective in exposing inconsistencies in Code LLMs: About one in seven (14.82%) test cases generated by M U C O C O exposed consistency errors across all settings. M U C O C O exposed thousands of consistency errors (≈22K errors out of 148K test cases), despite a relatively high model accuracy (69.13% on average). Table 4 also shows that the consistency error rate exposed by M U C O C O ranges widely across models. In particular, GPT-5 (OpenAI, 2025a) had the least consistency error rate (2.92%), while LLaMA (Grattafiori et al., 2024b) and GPT-4o (OpenAI, 2024) produced the most consistency error rate with 21.22% and 18.15% consistency error rate, respectively. We attribute the performance of GPT-5
M U C O C O computes consistency error rate (aka Inconsistency/Inc./ErrRate) as follows: Inc. =
Ninconsistent × 100 Ntest_cases
(1)
Ninconsistent refers to the number of consistency errors discovered by M U C O C O by comparing the LLM outputs and test suite correctness of M U C O C O’s mutated dataset versus the original dataset. Ntest_cases is the number of test cases (mutated and original programs pairs) generated by M U C O C O. 4.5
All Tasks Inc. Acc. 13.91 (6259/45008) 69.01 (31055/45000) 11.59 (2691/23220) 71.58 (16612/23209) 16.28 (12974/79707) 68.48 (54425/79471) 14.82 (21924/147935) 69.13 (102092/147680)
Baseline Settings
We
compare M U C O C O to the T URBU (Honarvar et al., 2025) benchmark as a baseline for testing code inconsistencies (seesection 5 (RQ2)). Since T URBULENCE uses a parametrised program templates for instantiating programs, 100 of these parameters are first created LENCE
5
Table 6: M U C O C O’s Effectiveness across benchmarks showing Inconsistency Rate (Inc.) and Accuracy (Acc.). Mutation Type Logical Syntactic Lexical All
CodeMMLU Inc. Acc. 38.50 46.48 16.95 67.80 10.04 78.36 22.35 64.04
HumanEval Inc. Acc. 9.95 73.26 8.35 74.94 10.89 74.55 10.09 74.10
CruxEval Inc. Acc. 21.17 60.57 16.08 66.74 17.42 66.41 18.05 65.11
BigCodeBench Inc. Acc. – – – – 27.26 56.95 27.26 56.95
(OpenAI, 2025a) to its reasoning attribute. We also attribute the relatively average consistency error rate for code-instruct LLMs (CodeStral (team, 2025) and Qwen2.5 (Hui et al., 2024a)) to the impact of code domain adaptation. Table 18 and Table 6 show that consistency errors are most prevalent for code generation tasks and BigCodeBench (Zhuo et al., 2025a) dataset due to the higher complexity of the task/dataset. We observed lower consistency error rate (6.65%) for input prediction tasks and HumanEval (Chen et al., 2021) datasets due to the simplicity of the problems. As an example, HumanEval (Chen et al., 2021) programs are significantly simpler than those in BigCodeBench (Zhuo et al., 2025a). Overall, these results demonstrate that M U C O C O effectively exposes coding inconsistency in LLMs and consistency errors are prevalent across LLMs.
All Datasets Inc. Acc. 13.91 (6259/45008) 69.01 (31055/45000) 11.59 (2691/23220) 71.58 (16612/23209) 16.28 (12974/79707) 68.48 (54425/79471) 14.82 (21924/147935) 69.13 (102092/147680)
LENCE . M U C O C O generates (4X) more consistency test cases than T URBULENCE (4,302,713 vs. 826,084). These results demonstrate the test generation effectiveness of M U C O C O versus T URBU LENCE .
Qualitative analysis shows that M U C O C O discovers a fundamentally different type of consistency errors in comparison to T URBULENCE. While T URBULENCE discovers inconsistencies across different parameterized instances of the same program, M U C O C O discovers inconsistencies across semantic-preserving lexical, syntactic or logical variants of a program. Consider the T URBULENCE program Q18 (Table 7). M U C O C O’s DeMorgan mutation on instances of Q18 (column 3) results in the mutated program (column 4). M U C O C O reveals a consistency error (✗) via a logical mutation of the if condition of the return statement (line 3):
M U C O C O is effective in exposing consistency errors in Code LLMs: 14.82% of its generated inputs triggered consistency errors.
if i % $0 == 0 or i % $1 == 0]
→
if not (not i % $0 == 0 and (not i % $1 == 0))]
In this example (Q18), M U C O C O detects a consistency error (✗) since M U C O C O’s logical mutation results in a correct GPT-4o output for the first instance (“-672=-672” in column 6), but an incorrect output for the logically modified instance (“486 ̸= 698” in column 8). Meanwhile, performing only parameter instantiation for this query (Q18), as envisioned in T URBULENCE, does not reveal a consistency error (✓). Specifically, different parameter instantiations ($0- $3) in instance one (column 5) versus instance two (column 7) are consistently incorrect (✓) – since their expected output consistently fails to match the expected output for both instances– “-1114 ̸= -672” (column 6) and “486 ̸= 698” (column 8), respectively. Since this is an output prediction task, the expected output and actual GPT-4o answer are compared. As such, the model outputs are consistent for T URBULENCE instances, according to Table 2 (last row). This example demonstrates that M U C O C O complements T URBULENCE: In comparison to T URBULENCE, M U C O C O discovers different classes of coding in-
RQ2 Baseline Comparison: This experiment compares the effectiveness of M U C O C O to a closely-related robustness benchmark called T UR BULENCE (Honarvar et al., 2025). T URBULENCE provides manually-crafted, parameterised program templates, such that parameters can be instantiated to form multiple instances of the same program. Table 7 shows a consistency error revealed by M U C O C O even though T URBULENCE’s instances are consistent. We compare pairs of parameterized instances of T URBULENCE versus executing M U C O C O on T URBULENCE instances. Table 8 presents our findings. We observed that M U C O C O reveals more consistency errors than T URBULENCE. M U C O C O has a (22.46%) higher consistency error rate than T UR BULENCE (33.8% vs. 41.39%). Specifically, Table 8 shows that M U C O C O reveals five times (5X) more consistency errors (1,781,288 vs. 279,241) than T URBULENCE. In addition, M U C O C O generates significantly larger test suites than T URBU 6
Table 7: T URBULENCE VERSUS M U C O C O I NSTANCES. Table 16 in Appendix provides additional examples. ID
Mutation Type
Turbulence Template
1
Q18 DeMorgan
2 3
def sum_ints_div_by_either_nums ( l: List [ int ]) -> int : l = l[ $2 : $3 + 1] return sum([i for i in l if i % $0 == 0 or i % $1 == 0])
Instance 1 Parameters
Turbulence Template + MuCoCo Mutation
1 2 3
def sum_ints_div_by_either_nums ( l : List [ int ]) -> int : l = l[ $2 : $3 + 1] return sum([i for i in l if not (not i % $0 == 0 and (not i % $1 == 0))])
Template Params: $0: -4 $1: 4 $2: 1 $3: 6
Instance 1 LLM Output
Instance 2 Parameters
Instance 2 LLM Output
Turbulence MuCoCo Inc. Inc.
Template Params: Expected Output: $0: -6 698 $1: 4 $2: 2 Turbulence Output: Turbulence Output: $3: 4 -1114 486 Expected Output: -672
Program Input: MuCoCo Output: [212, -451, -512, 337, -672 486, -442, -160, -422 ]
✓
✗
Program Input: MuCoCo Output: [337, -512, 212, 486 -451, 486, -422]
Table 8: Effectiveness of M U C O C O versus T URBULENCE using the T URBULENCE dataset. Best performance per (sub)category is marked in bold text. (“#” = Number of , "errs” = Consistency Errors, “tests” = Test Cases, “ErrRate” = Error Rate, “Impr. ” = Improvement) Approach T URBULENCE MUCOCO % Impr.
Code Generation #errs #tests ErrRate 20359 272888 7.46% 31458 542627 5.80% 54.52% 98.85% -22.25%
Input Prediction #errs #tests ErrRate 54809 276598 19.82% 363593 1882518 19.31% 563.38% 580.60% -2.57%
Output Prediction #errs #tests ErrRate 204073 276598 73.78% 1386284 1877568 73.83% 579.31% 578.81% 0.07%
#errs 279241 1781288 537.90%
All Tasks #tests ErrRate 826084 33.8% 4302713 41.39% 420.86% 22.46%
Figure 2: C ONSISTENCY ERROR RATE AND ACCURACY BY MUTATION OPERATOR .
consistency, while exposing more inconsistencies. M U C O C O is up to five times (5X) more effective than T URBULENCE in test generation. It discover 22.46% more consistency errors than T URBULENCE.
Percentage (%)
100
RQ3 Probing Study: We inspect the impact of M U C O C O’s mutators on its effectiveness. Figure 2 shows that M U C O C O’s lexical mutations (random and sequential renaming) and logical mutations (constant unfolding and DeMorgan) induce the most consistency errors across all settings. Table 4, Table 18 and Table 6 also show that M U C O C O’s logical and lexical mutation operators induce the most consistency errors, 13.91% and 16.28%, respectively. We attribute the higher consistency error rate for logical mutations to their higher code complexity. Meanwhile, we attribute the higher consistency error rate for lexical mutations to LLM’s inherent natural language dependence, in particular, dependence on meaningful variable names and method names.
Inconsistency
Accuracy
80 60 40 20 0
n l r d d lt n e e t al tioliteraorde nfol d ad mu orga erat whilormandomenti a t l e u - f u d mu an e r nt nfo fol dem num -to al raseq no oole tativ nsta nt u t un -to-e for liter b mu co sta tan for n s m co con co
ordering and constant unfolding). We attribute this to the complexity of logical mutations which makes it more challenging for LLMs. Overall, these results imply that LLMs are less robust to method/variable renaming and changes in program logic. Logical mutations induce a lower model accuracy than the original program or other mutation types (lexical and syntactic mutations).
Logical and lexical mutations induce a higher consistency error rate than syntactic mutations.
6
Inspecting the mutation operators that impact model accuracy, we observed that boolean literal improves model accuracy versus the original program without mutation (“no_mutation” in Figure 2). We also found that syntactic mutations (for-to-enumerate) and logical mutation (DeMorgan) also slightly improve model accuracy. Most mutations improve or preserve model accuracy, except logical mutations (commutative re-
Additional Experiments
In additional experiments, we find that inconsistencies were often induced by M U C O C O’s mutated query rather than the original query (Appendix E). We also find that most inconsistencies exposed are categorised under correctness-based inconsistencies (55.08%) (Appendix D). Moreover, we measure the inconsistency distance across models, we observe that open-source/-weight models have 7
a greater inconsistency distance than proprietary models (Appendix F). We also examine the impact of model confidence on inconsistency using opensource/-weight models. We found that increasing the confidence threshold reduces inconsistencies in LLMs from 8.03% to 1.89% (Appendix I). Finally, we inspect the impact of multiple (two) mutations on inconsistency. Our findings (Appendix H) show that multiple mutations led to a 5.81% increase in inconsistency in comparison to atomic mutations.
uating the correctness of LLMs and are static by nature. Meanwhile, M U C O C O automatically transforms an existing coding dataset to generate test suites for evaluating code consistency. Orvalho et al. (Orvalho and Kwiatkowska, 2025) proposed a code mutation framework for testing LLM robustness. However, their approach exposed code inconsistencies through manual, expert analysis of the LLM’s reasoning (Orvalho and Kwiatkowska, 2025), which differs from M U C O C O’s automatic testing approach.
7
Dynamic Testing Approaches: Guan et al. (Guan et al., 2025) proposed a dynamic benchmarking framework to keep static benchmarks relevant over time. Similar to M U C O C O, their dynamic testing framework utilizes code mutation to transform static benchmarks including VarNormI, VarNormII, For-to-While and ConstUnfold (Guan et al., 2025). However, the dynamic testing framework is targeted at evaluating LLM’s correctness (Guan et al., 2025), which differs from M U C O C O which is targeted at exposing code inconsistencies. Additionaly, M U C O C O supports a total of 11 mutation variations, offering a greater diversity than the dynamic benchmark framework. M U C O C O also broadens the LLM evaluation methodology by including MCQ and code generation tasks, while the dynamic testing framework tests with code execution and code translation (Guan et al., 2025).
Related Works
LLM Robustness and Consistency Testing: Honarvar et al. (Honarvar et al., 2025) presented the T URBULENCE benchmark as a novel templatebased approach for evaluating LLM correctness and robustness. LLM robustness is evaluated by comparing the differences in outputs within the same neighbourhood (Honarvar et al., 2025). M U C O C O differs from T URBULENCE by assessing LLM inconsistency through code mutation on existing benchmarks, and conducting one-to-one comparison of results of mutated and original tasks. M U C O C O also includes tasks like MCQ, input prediction and output prediction tasks. The KONTEST framework proposed by Rajan et al. (Rajan et al., 2024) exposes knowledge gaps in LLMs through an automatic testing framework, utilizing a knowledge graph approach to construct test cases. On the other hand, M U C O C O utilizes rule-based mutation of existing benchmarks for testing. KONTEST also differs in scope with M U C O C O, it is focused on general knowledge testing (Rajan et al., 2024), while M U C O C O is aimed at exposing inconsistencies in Code LLMs. Code LLM Benchmarks and Evaluation Methods: The HumanEval benchmark (Chen et al., 2021) is a well-established dataset used for evaluating LLM’s correctness through code generation tasks. BigCodeBench by Zhuo et al. (Zhuo et al., 2025a) presents a series of diverse Python code generation tasks aimed at challenging LLMs with a range of function calls. CodeMMLU (Nguyen et al., 2025) introduces a comprehensive dataset of MCQ-type questions, testing LLMs on a variety of code-related tasks such as code repair, execution reasoning and code completion. CruxEval (Gu et al., 2024) is a benchmark focused on testing LLM code knowledge through input prediction and output prediction tasks. The aforementioned benchmark datasets are primarly focused on eval-
8
Conclusion
In this work, we propose an automated testing approach for assessing the consistency property of LLMs for coding tasks. Our technique (M U C O C O) employs mutation analysis and metamorphic properties to discover code consistency errors in LLMs. It provides 11 semantic-preserving mutation operators spanning lexical, logical and syntactic mutations. We evaluate M U C O C O using seven LLMs, four benchmarks and four datasets. Results show that M U C O C O is effective in consistency testing and code inconsistency is prevalent in state-of-theart LLMs. We also compare M U C O C O to a stateof-the-art baseline (T URBULENCE) and demonstrate its superiority and complementarity. Our work motivates the need to automatically assess LLMs for other properties, beyond correctness, in particular, consistency. 8
9
Limitations
In particular, CruxEval (Gu et al., 2024) only supports zero-shot prompting, while BigCodeBench (Zhuo et al., 2025a) supports up to one-shot prompting. Although few-shot prompting is widely considered as the superior prompt technique, such configurations are not uniformly available across benchmarks. As a result, performance comparisons across datasets may partially reflect differences in allowable prompt configurations rather than model capability alone.
Construct Validity: The main threat to construct validity of this work is the prompting of LLMs and processing of LLM outputs. For each dataset, we have used the best prompting option out of the three settings (zero-shot, one-shot, few-shot), based on the availability of examples or lack thereof. In addition, to ensure that LLM outputs are processed correctly, we have used system prompts to ensure models provide responses in the expected format. We have also tested this with the canonical solution (before mutation) to ensure correctness. We have also used open-source/-weight models (Llama and DeepSeek) with frozen weights to mitigate and reduce non-determinism. We have set temperature to zero for all models and reasoning to minimal for GPT-5 to mitigate randomness. Finally, we mitigate the effect of model updates by limiting our experiments to a short time and tracking update news. Internal Validity: This refers to the risk that our implementation of M U C O C O performs the intended code consistency testing. To mitigate this, we have conducted manual and automated tests, code review and manual inspection of mutation operators, sampled experiments and results. In addition, we have conducted probing studies (section 5 RQ3 to examine the correctness of M U C O C O. Finally, we have reported additional ablation and sensitivity studies (section 6 and Appendix) to examine the stability/robustness of M U C O C O. External Validity: The main threat to external validity is the generalizability of our approach and findings. To mitigate this, we have employed several state-of-the-art models, tasks and datasets with varying sizes, maturity and complexity. We acknowledge that our approach has been applied and evaluated using only Python programs, thus, it may not directly apply to other programming languages (PLs). However, we note that several mutations in our approach apply to most high-level programming languages, especially our lexical and logical mutations. We acknowledge that syntactic mutations are PL dependent and may require redesign for other languages. Prompt Configuration: Our experiments are conducted using the strongest available prompt configuration for each benchmark, selecting among zeroshot, one-shot, and few-shot prompting (Brown et al., 2020). However, this choice is also constrained by the design of the benchmark datasets.
Model Randomness: LLMs exhibit inherent stochasticity in their generation process. To mitigate this, all models except GPT-5 (OpenAI, 2025a) are evaluated with a temperature setting of zero (0). GPT-5 API does not offer the option to set model temperature hence we set its reasoning mode to minimal (OpenAI, 2025b). Despite these measures, nondeterminism cannot be fully eliminated, and our results may therefore be affected by some stochastic variations in model behavior
10
Ethical Considerations
This section outlines the ethical considerations associated with our study. Datasets. We exclusively use established benchmark datasets obtained from their official releases on HuggingFace. These datasets have been made publicly available, widely used by the research community (Zhuo et al., 2025a; Gu et al., 2024; Nguyen et al., 2025; Chen et al., 2021). Methodology and Model Usage. Pre-trained LLMs by the official companies were used to evaluate M U C O C O. We hereby acknowledge that LLMs may reflect biases present in their training data or modeling assumptions. Additionally, we limit our experiments to inference on pre-trained models, thereby avoiding the significant environmental costs (such as energy and water consumption) incurred from model training.
References Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llmbased agent for program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 694–694. IEEE Computer Society. Chris Bowdon. 2025. How many parameters does gpt-5 have? https://www.r-bloggers.com/2025/08/ how-many-parameters-does-gpt-5-have/. Published on R-Bloggers.
9
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
Shahin Honarvar, Mark van der Wilk, and Alastair F. Donaldson. 2025. Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code . In 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), pages 80–91, Los Alamitos, CA, USA. IEEE Computer Society.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code.
Josh Howarth. 2025. Number of parameters in gpt4 (latest data). https://explodingtopics.com/ blog/gpt-parameters. Last updated June 17, 2025.
DeepSeek. 2025. Deepseek — api docs news. https: //api-docs.deepseek.com/news/news250929.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024a. Hugging face — qwen/qwen2.5-coder-14b. https://huggingface. co/Qwen/Qwen2.5-Coder-14B.
Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K Lahiri. 2024. Llmbased test-driven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering, 50(9):2254–2268.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024b. Qwen2.5-coder technical report. Preprint, arXiv:2409.12186.
Sen Fang, Weiyuan Ding, and Bowen Xu. 2025. Evalooop. https://evalooop.github.io/. Python Software Foundation. 2025. Python 3 library — ast module. https://docs.python.org/ 3/library/ast.html. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024a. The llama 3 herd of models. Preprint, arXiv:2407.21783.
Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. Inferfix: End-to-end program repair with llms. In Proceedings of the 31st ACM joint european software engineering conference and symposium on the foundations of software engineering, pages 1646–1656.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024b. Hugging face — meta-llama/llama-3.1-8b. https://huggingface. co/meta-llama/Llama-3.1-8B. Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024. CRUXEval: A benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 16568–16621. PMLR.
LifeArchitect.ai. 2025. Gpt-5 (2025). https:// lifearchitect.ai/gpt-5/. Online article. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. 2023. Evalplus leaderboard. https://evalplus.github.io/ leaderboard.html. llm stats. 2025. Llm-stats humaneval benchmark. https://llm-stats.com/benchmarks/ humaneval. Dung Manh Nguyen, Thang Chau Phan, Nam Le Hai, Tien-Thong Doan, Nam V. Nguyen, Quang Pham, and Nghi D. Q. Bui. 2025. CodeMMLU: A multitask benchmark for assessing code understanding & reasoning capabilities of codeLLMs. In The Thirteenth International Conference on Learning Representations.
Batu Guan, Xiao Wu, Yuanyuan Yuan, and Shaohua Li. 2025. Is your benchmark (still) useful? dynamic benchmarking for code language models.
Augustus Odena, Charles Sutton, David Martin Dohan, Ellen Jiang, Henryk Michalewski, Jacob Austin, Maarten Paul Bosma, Maxwell Nye, Michael Terry, and Quoc V. Le. 2021. Program synthesis with large language models. In n/a, page n/a, n/a. N/a.
Junda He, Christoph Treude, and David Lo. 2025. Llmbased multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology, 34(5):1–30.
OpenAI. 2024. Openai platform docs — gpt-4o. https: //platform.openai.com/docs/models/gpt-4o.
10
OpenAI. 2025a. Openai platform docs — gpt-5. https: //platform.openai.com/docs/models/gpt-5.
A
OpenAI. 2025b. Openai platform docs — latest model guide. https://platform.openai.com/ docs/guides/latest-model. OpenAI. Google. Google colaboratory (colab). https: //colab.google/. Pedro Orvalho and Marta Kwiatkowska. 2025. Are large language models robust in understanding code against semantics-preserving mutations? Sai Sathiesh Rajan, Ezekiel Soremekun, and Sudipta Chattopadhyay. 2024. Knowledge-based consistency testing of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10185–10196, Miami, Florida, USA. Association for Computational Linguistics. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Hugging face — google/gemma-3-12b-it. https://huggingface. co/google/gemma-3-12b-it. Mistral AI team. 2025. Mistral ai — codestral 25-08. https://mistral.ai/news/codestral-25-08. The HuggingFace Team. 2024. Hugging face docs — transformers. https://huggingface.co/docs/ transformers/en/index. Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. 2025. Swe-bench live. https: //swe-bench-live.github.io/. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 14 others. 2025a. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In International Conference on Representation Learning, volume 2025, pages 66602–66656. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 14 others. 2025b. Bigcode benchmark. https://bigcode-bench. github.io/.
11
M U C O C O Test Oracle
This section contains additional details for M U C O C O’s correctness and consistency oracles. Correctness Oracle: The correctness oracle determines the correctness of an LLM output. Correctness of LLM outputs are evaluated individually for each task. For code generation, the program generated by the LLM is tested against the test suite provided by the respective benchmarks. For input and output prediction tasks, the LLM’s response is evaluated for equivalence against the expected output specified in the benchmark’s test suite. Lastly, inconsistency in MCQ task is determined by checking whether the LLM’s output (i.e., selected option) matches the canonical output in the benchmark. A correctness score is not assigned when M U C O C O’s code mutation is not possible, i.e., the original benchmark task cannot be mutated (e.g., due to the absence of a valid for-loop). Formally, M U C O C O’s correctness oracle is defined as:
Pass, Ocorr (Ai ) = AssertionError, Invalid Answer,
if Ai is correct, if Ai is incorrect, otherwise.
(2) where Ai and Ocorr refers to the LLM answer and the test correctness test oracle, respectively. Invalid Answer occurs when the LLM failed to return valid code (for code generation tasks), or if the LLM failed to return the answer within the given time budget. Consistency Oracle: To determine (in)consistency, M U C O C O’s consistency oracle compares the performance of an LLM on a mutated dataset versus an original dataset. The test suite correctness of the mutated and original datasets are first evaluated using the correctness oracle. Next, each task in the original dataset is paired with its corresponding mutated version and their outputs and test suite correctness are compared. Inconsistencies between LLM outputs are categorised into the following: (1) Correctness-based inconsistency: occurs when with one of the LLM output is correct and the other is incorrect. (2) Incorrectness-based inconsistency: occurs when both LLM outputs are incorrect, and have differing degrees of incorrectness. For code generation tasks, a typical test suite comprises of multiple
generic_function1 and first variable names are replaced with var1 and so on (see Table Table 9 row two). Literal Format replaces string single quotation marks (’) with double quotation (") and vice versa (see Table Table 9 row three).
test cases. The two incorrect LLM outputs are evaluated with the test suites and the sets of test cases passed are compared. If there is a difference in the two sets, an incorrectness-based inconsistency has occurred. Otherwise, the LLM outputs are consistent. For other tasks, the LLM outputs are compared directly, and an inconsistency is recorded if they differ. (3) Invalidity-based inconsistency: occurs when only one of the LLM output is invalid, while the other can either be correct or incorrect. Finally, no inconsistency occurs when both LLM outputs are correct or if both are invalid. Table 2 offers an overview of their inconsistency attribution for each pair of mutation types. M U C O C O’s consistency oracle is applied to each pair of original and mutated tasks, and the resulting outcomes are aggregated to compute the consistency error rate (see Equation 1). Additionally, additional experiments were conducted to investigate the direction of inconsistency and invalidity. These findings are elaborated in Appendix E.
B
B.0.2 Syntactic Mutations It refers to semantic-preserving mutations that modify the program syntax with equivalent syntax. Both Guan et al. (Guan et al., 2025) and Orvalho et al. (Orvalho and Kwiatkowska, 2025) proposed forto-while mutation in their respective works. The mutations under this category include: for-to-enumerate replaces all for loop iterators with enumerate iterators (see Table Table 9 row four). for-to-while Mutation replaces all for loops with while loops. (see Table Table 9 row five). B.0.3 Logical Mutations Logical mutations modifies the program logic in a manner that preserves the original program behavior, e.g., by decomposing a constant (5) into equivalent operations (4 + 1). Constant Unfolding and Condition Augmentation are similar program mutations proposed by Guan et al. (Guan et al., 2025). The mutations under this category include: DeMorgan Mutation: De Morgan’s Law refers to a pair of rules that convert logical operations with conjunction operators into disjunction equivalents. For instance,
Types of Mutations Supported by MUCOCO
This section provides a more detailed description of the types of mutations supported by M U C O C O. B.0.1 Lexical Mutations Lexical mutations refer to mutations involving renaming variable names, function names, or Python strings. These mutations modify the natural language names used in code without modifying the program logic. Code inconsistencies exposed via lexical mutations suggest that the LLM may not fully understand the program logic and relies more on meaningful names for program comprehension. Guan et al. (Guan et al., 2025) proposed similar variable renaming mutations VarNormI and VarNormII in a dynamic benchmarking framework and Orvalho et al. (Orvalho and Kwiatkowska, 2025) also proposed Variable Renaming mutation, which is similar to Random mutation. The mutations under this category includes the following: Random Mutation replaces the function/variable names with random strings (see Table Table 9 row one). Sequential Mutation is similar to random mutation, but it replaces function/variable names with generic names. By default, the function name in the function definition is renamed to
A and B → not A or not B
(3)
A or B → not A and not B
(4)
Table 9 (row six) illustrates how M U C O C O applies the DeMorgan’s law on programs containing logical statements. Boolean Literal transforms True and False, into semantically-equivalent expression using negation (see Table Table 9 row seven). Commutative Reorder: The order of variables in commutative expressions (such as summation and multiplication) are reversed to form a new expression (see Table Table 9 row eight). Constant Unfolding: Integers are decomposed into two factors and expressed using either multiplication or addition expressions (see Table Table 9 row nine). Constant Unfolding (Add): It substitutes all valid constants with equivalent addition expressions (see Table Table 9 row 10). 12
Table 9: M U C O C O M UTATION T YPES AND M ODEL R ESULTS . (✓= CONSISTENCY, ✗= INCONSISTENCY ( AKA CONSISTENCY ERROR ), OG = LLM A NSWER FOR O RIGINAL Q UESTION , MU = LLM A NSWER FOR M UTATED Q UESTION ) Mutation Type
Original Question
1 2 3 4 5
Random
6 7 8 9 10
1
Sequential
2
1 2 3 4 5
Literal Format
6 7 8 9 10
1 2 3
For2Enumerate
4 5 6 7
def f( text , n) : if n < 0 or len ( text ) <= n : return text result = text [0 : n] i = len ( result ) - 1 while i >= 0: if result [ i ] != text [ i ]: break i -= 1 return text [0 : i + 1] def f( text ): return not text . isdecimal () def f( name ): new_name = ' ' name = name [:: -1] for i in range ( len ( name ) ) : n = name [i ] if n != '. ' and new_name . count ( ' . ') <2: new_name = n+ new_name else : break return new_name def f( text ): x = 0 if text . islower () : for c in text : if int (c ) in list ( range (90) ): x +=1 return x
1 2 3 4 5 6 7 8 9 10
1 2
1 2 3 4 5 6 7 8 9 10 11 1 2 3 4 5 6 7 1 2 3
1 2 3
For2While
4 5 6 7
def f( string ): l = list ( string ) for i in reversed ( range ( len ( l )) ) : if l [i ] != ' ': break l . pop ( i ) return ''. join ( l)
4 5 6 7 8 9 10 11 12
1 2 3 4
DeMorgan
5 6 7 8 9 10 1 2 3
Boolean Literal
4 5 6 7 1 2
Commutative Reorder
3 4 5 1 2 3 4 5
Constant Unfolding
6 7 8 9 10 11 12 1
Constant Unfolding (Add)
2 3 4 5
Constant Unfolding (Multi)
1 2 3 4
def f( array , target ) : count , i = 0 , 1 for j in range (1 , len ( array ) ): if (( array [ j] > array [j -1]) and ( array [j ] <= target ) ): count += i elif array [ j] <= array [j -1]: i = 1 else : i += 1 return count def f( lst ) : lst . clear () for i in lst : if i == 3: return False else : return True def f( text , letter ) : if letter in text : start = text . index ( letter ) return text [ start + 1:] + text [: start + 1] return text def f( n) : p = '' if n % 2 == 1: p += 'sn ' else : return n * n for x in range (1 , n +1) : if x % 2 == 0: p += ' to ' else : p += ' ts ' return p def f( nums ): for i in range ( len ( nums ) ) : if nums [i ] % 3 == 0: nums . append ( nums [i ]) return nums def f( lists ) : lists [1]. clear () lists [2] += lists [1] return lists [0]
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 1 2 3 4 5 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 1 2 3 4
Mutated Question
Dataset (Canon. Ans.)
GPT-4o (OG) (MU)
GPT-5 (OG) (MU)
Qwen (OG) (MU)
Llama (OG) (MU)
Gemma (OG) (MU)
DeepSeek (OG) (MU)
Codestral (OG) (MU)
def dOdtZZSuisMB ( TYesQRBm , mXDbZtJMZFV ) : if mXDbZtJMZFV < 0 or len ( TYesQRBm ) <= mXDbZtJMZFV : return TYesQRBm result = TYesQRBm [0: mXDbZtJMZFV ] i = len ( result ) - 1 while i >= 0: if result [ i ] != TYesQRBm [ i ]: break i -= 1 return TYesQRBm [0: i + 1]
CruxEval_789 (’br’)
✗ (’br’) ([’bR’])
✓
✗ (’br’) ([’bR’])
✓
✗ (’br’) ([’bR’])
✗ (’br’) ([’bR’])
✗ (’br’) ([’bR’])
def generic_function1 ( var1 ) : return not var1 . isdecimal ()
CruxEval_518 (True)
✓
✓
✓
✗ (True) (False)
✓
✗ (True) (False)
✓
def f( name ) : new_name = " " # ’ > " name = name [:: -1] for i in range ( len ( name ) ) : n = name [ i ] if n != " . " and new_name . count ( " . " ) < 2: # ’ > " new_name = n + new_name else : break return new_name return a
CruxEval_273 ("NET")
✓
✓
✓
✗ ("NET") (".NET")
✗ ("NET") ("TEN")
✓
✓
def f( text ) : x = 0 if text . islower () : for loop_var0 , c in enumerate ( text ) : if int ( c ) in list ( range (90) ) : x += 1 return x
CruxEval_560 (0)
✓
✓
✗ (0) (9)
✓
✓
✓
✓
def f( string ) : l = list ( string ) new_reversed_var4 = range ( len ( l ) ) [:: -1] length_var3 = len ( new_reversed_var4 ) loop_var0 = 0 while loop_var0 < length_var3 : i = new_reversed_var4 [ loop_var0 ] if l[ i] != ' ': break l. pop (i ) loop_var0 += 1 return ' '. join ( l )
CruxEval_708 (’ jcmfxv’)
✓
✓
✗ (’ jcmfxv’) (’jcmfxv ’)
✓
✓
✗ (’ jcmfxv’) (’jcmfxv’)
✗ (’ jcmfxv’) (’jcmfxv’)
def f( array , target ) : count , i = (0 , 1) for j in range (1 , len ( array ) ) : if not ( not array [ j ] > array [ j - 1] or not array [ j ] <= target ) : count += i elif array [ j ] <= array [ j - 1]: i = 1 else : i += 1 return count
CruxEval_223 (1)
✗ (1) (```\n2\n```)
✓
✓
✗ (1) (2)
✓
✗ (1) (0)
✗ (2) (1)
def f( lst ): lst . clear () for i in lst : if i == 3: return not True else : return not False
CruxEval_97 (True)
✓
✓
✓
✗ (True) (False)
✓
✗ (True) (False)
✗ (True) (False)
def f( text , letter ) : if letter in text : start = text . index ( letter ) return text [1 + start :] + text [:1 + start ] return text
CruxEval_786 ("kefp719")
✗ ("kefp719") ("19kefp7")
✓
✗ ("kefp719") ("9kefp71")
✓
✗ ("9kefp71") ("kefp719")
✗ ("kefp719") ("efp719k")
✓
def f( n ): p = '' if n % (1 + 1) == 1: p += ' sn ' else : return n * n for x in range (1 , n +1) : if x % (2 * 1) == 0: p += ' to ' else : p += ' ts ' return p
CruxEval_506 ("snts")
✗ ("snts") ("sn_ts")
✓
✗ ("snts") ("sn")
✗ ("snts") ("sn")
✓
✗ ("sn") ("snts")
✓
def f( nums ) : for i in range ( len ( nums ) ) : if nums [ i ] % (1 + 2) == 0 + 0: nums . append ( nums [ i ]) return nums
CruxEval_226 ([1,3,3])
✓
✓
✓
✗ ([1,3,3]) ([1,3,1,3])
✗ ([1,3,3]) ([1,3])
✗ ([1,3,3]) ([1,3,1,3])
✗ ([1,3,3]) ([1,3,1,3])
CruxEval_564 ([395,666,7,4])
✓
✓
✓
✗ ([395,666,7,4,666]) ([395,666,7,4])
✓
def f( lists ) : lists [1 * 1]. clear () lists [2 * 1] += lists [1 * 1] return lists [0 * 1]
Constant Unfolding (Multiplication) replaces constants with equivalent multiplication expressions (see Table Table 9 row 11).
C
a description of the program including the incomplete code, along with several completion options. Four completion options are provided to the LLM (A,B,C,D) and the LLM has to choose the option that completes the program.
Task-specific Experimental Setup
This section provides additional details on M U C O C O’s experimental setup and testing procedures for Multiple Choice Questions, Input Prediction, Output Prediction and Code Generation tasks. C.0.1
✗ ✗ ([395,666,7,4]) (395) ([4223,111]) ([395,666,7,4])
C.0.2
Input Prediction
The LLM is tasked with predict whether a provided input will result in given output when the provide program is executed. We provide the LLM with a canonical program, a program description and an input-output pair. The allowed LLM response is either True or False. We employ CruxEval (Gu et al., 2024) and HumanEval (Chen et al., 2021).
Multiple Choice Questions (MCQ)
Using the CodeMMLU benchmark (Nguyen et al., 2025), the LLM is tasked with selecting the correct completion of an incomplete program when given 13
Table 10: Distribution of inconsistency types across mutation categories.
We employ the HumanEval dataset for this task by splitting its test cases into input - output pairs, which are then used for input prediction. Hence, the expected LLM output is always True. C.0.3
Output Prediction
The LLM is tasked with predicting the correct program output when given the input, the canonical program and a program description. CruxEval (Gu et al., 2024) and HumanEval (Chen et al., 2021) benchmarks are used for this task. Test pairs for HumanEval benchmark were generated using the same approach for Input Prediction. The Guan et al. (Guan et al., 2025) and Orvalho et al. utilized output prediction on CruxEval for LLM testing in their respective works. C.0.4
Mutation Type
Count
Percentage
Incorrectness-based Inconsistency
Logical Syntactic Lexical
2401 1213 6143
10.95% 5.53% 28.02%
Correctness-based Inconsistency
Logical Syntactic Lexical
3825 1460 6790
17.45% 6.66% 30.97%
Invalidity-based Inconsistency
Logical Syntactic Lexical
33 18 41
0.15% 0.08% 0.19%
All
All
21924
100.00%
E
Inconsistency Direction in M U C O C O
We examine the direction of inconsistency in order to attribute the cause of inconsistency to M U C O C O’s mutation or the original input. As an example, a correctness-based inconsistency arising from the LLM’s output being correct on the original task and incorrect on the mutated counterpart has an incorrectness direction caused by mutation. Additionally, we also account for the invalidity direction to determine if applying mutation operators led to more invalid LLM outcomes. Table 11 provides the directional attribution for incorrectness and invalidity for all possible outcome pairs. Table 12 highlights our findings. Across all three mutation categories, we find that mutated tasks are consistently the primary contributor to inconsistency. In addition, most incorrectness are caused by mutated tasks. On the other hand, we find that original dataset is the primary invalidity direction for lexical and syntactic mutations, while mutations are the invalidity direction for logical mutations. However, original tasks contributes more to the invalidity direction. Overall, these results show that the consistency errors in Code LLMs, reported in this work, are caused by M U C O C O’s mutations, rather than the original inputs.
Code Generation
The LLM is tasked with generating a program when given a program description and a snippet of the program. The snippet only contains the function definition and input variable names. The LLM must complete the program such that it matches the program description and in accordance to the given function and variable names. BigCodeBench (Zhuo et al., 2025a) and HumanEval (Chen et al., 2021) are used for this task.
D
Inconsistency Type
Inconsistency Errors per Error Type
Table 10 reports the distribution of consistency errors per error type. We found that correctnessbased inconsistency accounts for the majority of inconsistencies found by M U C O C O (55.08%), this is followed by incorrectness-based inconsistency (44.05%). Invalidity-based inconsistencies contributes the least number of inconsistencies, accounting for only 0.42% of consistency errors. Table 10 shows that logical mutations causes the most number of inconsistencies (59.18%). We attiribute this to the complexity of code generation tasks which only have lexical mutations due to the lack of program snippets. Meanwhile, logical and syntactic mutations account for 28.55% and 12.27% of the discovered inconsistencies. It should be noted that syntactic mutations have the least number of mutations (for-to-while and for-to-enumerate). These results suggest that LLMs are more robust to mutations in (Python) programming syntax than lexical or logical changes in programs.
F
Inconsistency Distance in M U C O C O
We propose an additional metric for quantifying the degree of inconsistency between LLM outputs. This involves measuring the correctness differences across test suites. This metric applies only to code generation tasks, which include multiple test cases within test suites for evaluating output correctness. Unlike binary correctness judgments, employing the test suites proffers a more fine-grained assess14
Table 11: Directional inconsistency attribution based on original (OG) and mutated (Mut) task outcomes.
OG Task
Mut Task
Incorr Dir
Inv Dir
Correct Correct Correct
Incorrect Correct Invalid
Mut – –
– – Mut
Incorrect Incorrect Incorrect
Incorrect Correct Invalid
OG + Mut OG OG
– – Mut
Invalid Invalid Invalid
Incorrect Correct Invalid
Mut – –
OG OG OG + Mut
for a test suite is shown in Equation 5.
Inc Dist =
# of inconsistent test cases # of test cases
(5)
We can then aggregate the inconsistency distance (Agg Inc Dist) of each task to determine the model with the highest inconsistency distance using Equation 6. P Agg Inc Dist =
Table 19 highlights the consistency distance results. Our findings determined that proprietary models have a smaller inconsistency distance when compared to open-source/-weight models, with a 0.084 difference. This suggests that proprietary models are more consistent than open-source/-weight models even when we consider more fine-grained metrics. We also find that GPT-5 (OpenAI, 2025a) is the most consistent model for code generation, with an aggregated inconsistency distance of 0.094. On the other hand, Llama-3.1-8b (Grattafiori et al., 2024a) is the most inconsistent model with an aggregated inconsistent distance of 0.235.
Table 12: Directional inconsistency statistics across mutation categories. Category
# Mut
# OG
Inc. Dir
Inv. Dir
Lexical Incorrect Dir Logical Incorrect Dir Syntactic Incorrect Dir
25,022 13,939 6,597
24,401 12,562 6,452
Mut Mut Mut
– – –
Lexical Invalid Dir Logical Invalid Dir Syntactic Invalid Dir
268 26 11
431 19 14
– – –
OG Mut OG
Aggregated Incorrect Dir Aggregated Invalid Dir
45,558 305
43,415 464
Mut –
– OG
Inconsistency Distance (6) Total Questions
G
Implementation Details and Platform
M U C O C O utilizes AST Python package (Foundation, 2025) for code mutation processes. Qwen (Hui et al., 2024a), Llama (Grattafiori et al., 2024b) and Gemma (Team et al., 2025) model weights were downloaded using the HuggingFace Transformers Python package (Team, 2024). The experiments on these models were then conducted on Google Colab (OpenAI, Google), using A100 Tensor Core GPU with 80GB RAM. Experiments on GPT-5 (OpenAI, 2025a), GPT-4o (OpenAI, 2024), Codestral (team, 2025) and Deepseek (DeepSeek, 2025) models were conducted using the official APIs provided by their website.
ment of consistency by determining the specific set of test cases that pass or fail. This metric leverages the test suite sensitivity of each model outcome to quantify the degree of consistency errors. By conducting a pair-wise comparison between the outcome of each test case of the LLM output on the original and mutated task, we compute the inconsistency distance between the outputs. As an example, consider a test suite containing three unique test cases for testing correctness on the generated LLM programs. For simplicity, the test cases will be referred to as first, second, and third. Let us assume that, for the original task, the LLM-generated program passed the first and third test cases, while failing the second one. Meanwhile, for the mutated task, the LLM-generated program passed the second and third test cases and failed the first. The correctness of the two solutions differ for two test cases (first and second) and only aligned for the third function. Since there are a total of three test cases within the test suite, this leads to an inconsistency distance of 2/3. Concretely, the inconsistency distance (Inc Dist)
H
M U C O C O’s Scalability to Multiple (Two) Mutations
This experiment examines the scalability of M U C O C O to two simultaneous mutations. Table 15 highlights our findings. Evaluation results show that second-order mutations reveal inconsistencies at a (30.01%) higher rate than atomic mutations. Table 15 shows that M U C O C O’s second-order mutations reveal inconsistency at a 25.17% rate, while atomic mutations 15
HumanEval Inc Dist
BigCodeBench Inc Dist
Aggregated Inc Dist
Open-weight
Qwen2.5-Coder-14B-Instruct Gemma-3-12b-it DeepSeek-V3.2-Exp LLama-3.1-8B
0.301 0.328 0.043 0.331
0.192 0.188 0.146 0.223
0.204 0.203 0.135 0.235
Proprietary
Table 13: I NCONSISTENCY DISTANCE (I NC D IST ) ACROSS H UMAN E VAL , B IG C ODE B ENCH , AND AGGREGATED BENCHMARKS . Model
GPT-5 GPT-4o codestral-2508
0.015 0.037 0.059
0.103 0.118 0.136
0.094 0.109 0.128
Aggregated Open-Source Aggregated Proprietary
0.251 0.037
0.187 0.119
0.194 0.110
Table 14: Details of Tested LLM Models showing their Architectures, Sizes, and Maturity. (“Est.” = Estimated, “B”= Billion Parameters) LLM Name GPT-4o
Company OpenAI
GPT-5
OpenAI
Qwen2.5-Coder
Alibaba Cloud
14.7 B
Gemma-3
12 B
DeepSeek
671 B
Mistral AI Meta AI
N/A 8B
DeepSeek-V3.2 (Non-Thinking) Codestral-2508 LLaMA-3.1-8B
Model Size 200 B (Howarth, 2025) (est.) 125 B (Bowdon, 2025) – 300 B (LifeArchitect.ai, 2025) (est.)
Architecture Transformers (Decorder-only)
Release Date 6 August 2024
Reasoning No
Open Source No
Code Instruct No
Transformers (Decoder-only)
7 August 2025
Yes
No
No
November 2024
No
Yes
Yes
March 2025
No
Yes
No
29 September 2025
No
Yes
No
July 2025 23 July 2024
No No
No Yes
Yes No
Transformers with RoPE, SwiGLU, RMSNorm Transformers (Decoder-only) with GQA, QK-Norm, and RMSNorm Multi-Head Latent Attention (MLA) and Transformer Hybrid Transformers (Decoder-only) Transformers (Decoder-only)
Second -order
Atomic
Mutatn. Level
Table 15: M U C O C O’s Scalability to multiple (2) mutations vs. atomic mutations. ("Inc." = Inconsistency, "Acc." = Accuracy) Mutation Operation (type) No Mutation Random (lexical) Constant Unfold (logical) For2while (syntactic) Constant Unfold + Random For2While + Random For2While + Constant Unfold Atomic (All) Second-order (All)
Input Prediction HumanEval CruxEval Inc. Acc. Inc. Acc. N/A 75.38 N/A 80.75 9.75 75.28 13.25 77.25 10.20 72.80 7.73 78.45 10.49 73.66 15.24 77.14 12.02 72.53 14.36 76.24 13.17 72.37 17.78 69.65 13.03 77.39 13.68 71.58 10.05 74.26 12.96 77.39 12.70 73.51 16.07 71.99
Output Prediction HumanEval CruxEval Inc. Acc. Inc. Acc. N/A 59.10 N/A 53.82 23.42 58.49 33.12 52.12 23.00 59.20 41.99 35.56 22.84 56.17 32.38 48.57 28.51 69.04 44.20 43.09 32.10 70.58 48.25 52.06 33.72 64.37 53.68 38.30 23.17 58.10 34.18 48.96 31.10 68.65 47.88 47.12
I
have a 19.36% consistency error rate. However, in absolute terms, M U C O C O’s atomic mutations reveal (40.38%) more consistency errors that its second-order mutations – 1269 vs. 904 (see Table 15). Overall, this result demonstrate that M U C O C O scales to multiple mutations and multiple mutations reveal more consistency errors than atomic mutation.
MCQ CodeMMLU Inc. Acc. N/A 85.71 7.58 87.12 40.98 44.26 22.03 69.49 39.34 50.82 30.51 61.02 50.00 46.67 19.05 72.62 38.00 54.00
All Tasks & Datasets Inc. Acc. N/A 67.92 (2528/3722) 19.53 (701/3590) 64.63 (2491/3854) 18.80 (256/1362) 61.90 (918/1483) 19.48 (312/1602) 62.85 (1081/1720) 22.71 (290/1277) 65.69 (919/1399) 26.72 (428/1602) 65.81 (1130/1717) 26.12 (186/712) 65.24 (503/771) 19.36 (1269/6554) 63.62 (4490/7057) 25.17 (904/3591) 65.65 (2552/3887)
Impact of Model Confidence on Model Inconsistency
This section examines the impact of model confidence on consistency errors found by M U C O C O. In particular, we report how the percentage of consistency errors (aka inconsistency) and model accuracy changes as model confidence thresholds varies between 0.50 and 0.99. Table 17 reports our findings. We observed that the percentage of consistency errors reduces by up to 76% as model confidence threshold increases. Table 17 shows that model
M U C O C O scales to multiple mutations and they discover consistency errors at a higher rate than atomic mutations.
16
Table 16: T URBULENCE VS . M U C O C O I NSTANCES E XTENDED E XAMPLES ID
Mutation Type
Turbulence Template
1 2 3 4
Q33 For2While
5 6 7 8 9 10
def return_vowels (s : str ) -> List [ str ]: vowels = [ 'a ' , 'e ' , 'i ' , 'o ' , 'u ' , 'A ' , 'E ' , 'I ','O ' , 'U '] result = [] sliced_s = s[ $0 : $1 ] if not sliced_s : return result for char in sliced_s: if char in vowels and ' $2 ' < char <= ' $3 ': result . append ( char ) return result
2 3 4 5 6 7 8 9 10 11 12 13
1 2 3
Q39
Constant Unfolding
4 5 6 7 8 9
1
Q18 DeMorgan
2 3
Instance 1 Parameters
Instance 1 LLM Output
Template Params: $0: 22 $1: 44 $3: ’z’ $4: ’{}’
Expected Output: []
Program Input: "31 w 014%Xh>00a4"
Turbulence Template + MuCoCo Mutation 1
def return_n_greatest_chars ( s : str ) -> List [ str ]: if len ( s ) < $0 : return [] if " $1 " == " ascending " : sorted_s = sorted ( s) return sorted_s[:$0] else : sorted_s = sorted (s , reverse = True ) return sorted_s[:$0]
def sum_ints_div_by_either_nums ( l : List [ int ]) -> int : l = l [ $2 : $3 + 1] return sum([i for i in l if i % $0 == 0 or i % $1 == 0])
1 2 3 4 5 6 7 8 9
1 2 3
def return_vowels ( s : str ) -> List [ str ]: vowels = [ 'a ' , 'e ' , 'i ' , 'o ' , 'u ' , 'A ' , 'E ' , 'I ' , 'O ' , 'U '] result = [] sliced_s = s [ $0 : $1 ] if not sliced_s : return result loop_var0 = 0 while loop_var0 < len(sliced_s): char = sliced_s[loop_var0] if char in vowels and ' $2 ' < char <= ' $3 ': result . append ( char ) loop_var0 += 1 return result def return_n_greatest_chars ( s : str ) -> List [ str ]: if len ( s ) < $0 : return [] if " $1 " == " ascending " : sorted_s = sorted ( s ) return sorted_s[:$0*1] else : sorted_s = sorted (s , reverse = True ) return sorted_s[:$0*1]
Template Params: $0: 2 $1: "descending" Program Input: "oal.e^"
def sum_ints_div_by_either_nums ( l : List [ int ]) -> int : l = l [ $2 : $3 + 1] return sum([i for i in l if not (not i % $0 == 0 and (not i % $1 == 0))])
Template Params: $0: -4 $1: 4 $2: 1 $3: 6
1
Random
2 3
def all_pos_ints_inclusive( l : List [ int ]) -> List [ int ]: l = l [ $0 : $1 + 1] return [ i for i in l if i > 0]
1 2 3
def eUilVSRGdc( l : List [ int ]) -> List [ int ]: l = l [ $0 : $1 + 1] return [ i for i in l if i > 0]
Instance 2 LLM Output Expected Output: []
Turbulence Output: []
Template Params: $0: 14 $1: 36 $3: ’l’ $4: ’m’
MuCoCo Output: []
Program Input: "4 X4h$<a04 g10"
MuCoCo Output: []
Expected Output: [’o’,’l’] Turbulence Output: [’^’,’o’]
Program Input: [-451, 486, -531, -442, 337, -512, -160, 212, -422]
Turbulence MuCoCo Inc. Inc.
Turbulence Output: []
Turbulence Output: [’|’,’o’,’m’,’l’,’k’] Program Input: "oal.e^|kmTEcdS@" MuCoCo Output: [’^’,’o’,’m’,’k’,’e’]
MuCoCo Output: [’^’,’o’]
Template Params: $0: -6 $1: 4 $2: 2 $3: 4
Expected Output: -672 Turbulence Output: -1114
Turbulence Output: (486, 337, 212) # Tuple type “(..)” MuCoCo Output: (486, 337, 212) # Tuple type “(..)”
✓
✗
✓
✓
✗
✗
✗
Expected Output: 698 Turbulence Output: 486
Program Input: [337, -512, 212, -451, 486, -422]
Expected Output: [486, 337, 212] # List type “[..]”
✓
Expected Output: [’|’,’o’,’m’,’l’,’k’]
Template Params: $0: 5 $1: descending
Program Input: MuCoCo Output: [212, -451, -512, 337, -672 486, -442, -160, -422 ]
Template Params: $0: 0 $1: 7 Q3
Instance 2 Parameters
MuCoCo Output: 486
Expected Output: []
Template Params: $0: 0 $1: 1
Turbulence Output: []
Program Input: [-451, -422, 486]
MuCoCo Output: []
Table 17: Impact of varying model confidence thresholds on inconsistency rate (Inc.) and accuracy (Acc.). Results are reported in the format “Inc. (Acc.)”, with accuracy reported in parenthesis. Confidence
0.50 0.60 0.70 0.80 0.90 0.95 0.99
Gemma
Qwen
Llama
Agg. Model Inc.
HumanEval
CruxEval
CodeMMLU HumanEval
CruxEval
CodeMMLU HumanEval
CruxEval
CodeMMLU
Inc. (Acc.)
5.32 (80.44) 5.32 (80.44) 5.32 (80.44) 2.81 (80.60) 1.62 (81.17) 1.01 (82.03) 0.26 (85.06)
9.42 (73.04) 9.42 (73.04) 9.21 (73.15) 7.58 (73.70) 5.49 (73.75) 3.76 (74.74) 0.82 (72.60)
31.62 (48.51) 31.62 (48.51) 31.56 (48.51) 30.21 (48.66) 28.53 (49.56) 26.91 (49.87) 25.28 (47.48)
6.38 (81.52) 6.38 (81.52) 6.38 (81.52) 0.63 (87.47) 0.15 (91.98) 0.28 (89.37) 0.00 (89.94)
26.61 (61.59) 26.61 (61.59) 25.37 (61.79) 21.07 (64.39) 16.67 (66.30) 13.35 (71.70) 3.73 (78.19)
4.65 (91.29) 33.95 (32.50) 4.65 (91.29) 33.13 (33.00) 4.65 (91.29) 22.16 (35.58) 0.24 (92.97) 4.70 (38.54) 0.00 (90.29) 0.00 (59.80) 0.00 (81.88) 0.00 (72.58) 0.00 (81.63) 0.00 (93.33)
8.03 (72.79) 7.97 (72.83) 7.51 (73.21) 3.46 (77.59) 3.30 (80.86) 3.07 (80.90) 1.89 (80.35)
3.10 (86.89) 3.10 (86.89) 3.10 (86.89) 0.07 (92.97) 0.00 (95.71) 0.00 (96.35) 0.00 (95.30)
confidence above 0.50 results in a 8.03% consistency error rate, but only 1.89% consistency error rate is observed at 0.99 confidence threshold. Likewise, we found that model accuracy increases at higher confidence thresholds: Increasing model confidence threshold increases model accuracy by up to 7.56%, on average. These results suggest that model confidence improves both consistency and accuracy. It also implies that model confidence may be useful in mitigating inconsistency in LLMs, e.g., by only using model outputs with high confidence. However, we note that at higher confidence threshold, the number of abstentions (empty LLM responses) also increases due to the increasing number of LLM outputs that are discarded at high confidence thresholds.
9.38 (43.37) 9.38 (43.37) 9.38 (43.37) 0.12 (41.56) 0.32 (47.19) 0.00 (56.16) 0.00 (53.85)
confidence can serve as a good consistency mitigation method.
Code inconsistency reduces by up to 76% as model confidence increases, implying that
17
Framework
Mutation Testing
Automatic Testing Framework
LLM Benchmarking Automated Manual Generation Generation
Test Oracles Automated
Metamorphic
Code Gen.
Coding Tasks Input Output Pred. Pred.
Testing Goals MCQ
Consistency
Robustness
Accuracy
Natural Language
MuCoCo Turbulence KonTest (Guan et al., 2025) DyCodeEval DynaCode PPM EquiBench CodeCrash CCTest
Lexical
Framework
Syntactic
Sequential Random Literal Format For2While For2Enumerate Boolean Literal DeMorgan
Logical Commutative Constant Constant Unfold Reorder Unfold (Add)
Constant Unfold (Multiply)
MuCoCo Dynamic Benchmarking CodeCrash EquiBench CCTest
Table 18: M U C O C O’s Effectiveness across tasks showing Inconsistency Rate (Inc.) and Accuracy (Acc.). Ensemble Type Input Ensemble Model Ensemble Weighted Model Ensemble Hybrid Ensemble OG M U C O C O
MCQ Inc. Acc. 20.95 65.63 12.54 65.35
Input Pred. Inc. Acc. 6.22 79.68 0.29 86.42
Output Pred. Inc. Acc. 20.67 60.76 10.55 64.8
Code Gen. Inc. Acc. 30.91 51.72 15.48 67.76
Inc. 18.02 (6264/34763) 6.85 (1448/21154)
All Tasks Acc. 65.5 (22714/34678) 74.34 (15726/21154)
# Runtime Err. 85 0
1.69
97.75
1.62
98.86
1.66
95.8
12.63
73.94
2.94 (621/21154)
94.59 (20009/21154)
0
21.47 22.35
81.6 64.04
6.28 6.65
89.32 78.97
15.28 18.99
66.88 62.82
18.13 27.15
73.78 57.66
12.89 (678/5259) 14.82 (21924/147935)
77.16 (4058/5259) 69.13 (102092/147680)
0 255
HumanEval Inc Dist
BigCodeBench Inc Dist
Aggregated Inc Dist
Model Code Gen. Inc
Open-weight
Qwen2.5-Coder-14B-Instruct Gemma-3-12b-it DeepSeek-V3.2-Exp LLama-3.1-8B
0.301 0.328 0.043 0.331
0.192 0.188 0.146 0.223
0.204 0.203 0.135 0.235
0.410 0.387 0.194 0.437
Proprietary
Table 19: I NCONSISTENCY DISTANCE (I NC D IST ) ACROSS H UMAN E VAL , B IG C ODE B ENCH , AND AGGREGATED BENCHMARKS . Model
GPT-5 GPT-4o codestral-2508
0.015 0.037 0.059
0.103 0.118 0.136
0.094 0.109 0.128
0.126 0.162 0.187
Aggregated Open-Source Aggregated Proprietary
0.251 0.037
0.187 0.119
0.194 0.110
0.357 0.158
18