An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations Clarissa Miranda-Pena1* , Andrew Reeson2 , Cécile Paris2 , Josiah Poon1 , Jonathan K. Kummerfeld1 The University of Sydney1 , CSIRO’s Data612 , [email protected]*
arXiv:2604.07755v1 [cs.CL] 9 Apr 2026
Abstract
We analyse the potential for static analysis to detect and mitigate hallucinations in coding output that involves library usage. We consider both errors, i.e., any bug that leads to incorrect code behaviour, and hallucinations, i.e., code that would work if imagined functions or arguments existed.
Despite extensive research, Large Language Models continue to hallucinate when generating code, particularly when using libraries. On NL-to-code benchmarks that require library use, we find that LLMs generate code that uses non-existent library features in 8.1-40% of responses. One intuitive approach for detection and mitigation of hallucinations is static analysis. In this paper, we analyse the potential of static analysis tools, both in terms of what they can solve and what they cannot. We find that static analysis tools can detect 16-70% of all errors, and 14-85% of library hallucinations, with performance varying by LLM and dataset. Through manual analysis, we identify cases a static method could not plausibly catch, which gives an upper bound on their potential from 48.5% to 77%. Overall, we show that static analysis methods are cheap method for addressing some forms of hallucination, and we quantify how far short of solving the problem they will always be.
1
We consider three static analysis methods, applied to the output of four LLMs on three benchmarks. Two methods are off-the-shelf static analysis tools: Mypy and Pyright. One is a grammar that we automatically constructed from docstrings of libraries. All three can be applied after generation, to detect hallucination. The grammar can also be used during generation to constrain decoding. We also compare to prompting o3-mini as a baseline. We evaluate both detection and mitigation using three NL-to-code benchmarks that involve library use: DS-1000, Odex, and BigCodeBench. While static analysis tools provide more coverage of bugs (up to 70%), grammars successfully identify some library-related errors (up to 15%). Our analysis establishes an upper bound on the performance of static analysis methods in a typeinferred language (Python) and identifies which error categories cannot be solved by these approaches. Our results also indicate the importance of code benchmarks that require library use and have clear NL requests, as otherwise this form of hallucination will be missed.
Introduction
LLMs can hallucinate arguments to functions and even entire functions. Spracklen et al. (2025) found that GPT-4 turbo hallucinated packages 4% of the time, while CodeLlama 7B hallucinated them 26% of the time. Tian et al. (2025) found that GPT-4 mapped data types and structures incorrectly 10% of the time, and its output did not match external knowledge sources, such as modules’ imports, in 0.6% of cases. Identifying and fixing these errors creates additional work for programmers (Tanzil et al., 2024). If the errors are missed it can create a security risk, with an attacker who notices a common hallucination creating a malicious package that matches it (Spracklen et al., 2025; Krishna et al., 2025). Prior work has applied static analysis tools to detect syntax errors and logical errors (Ding et al., 2023; Ugare et al., 2024; Poesia et al., 2022), but not hallucinations.
The contributions of this paper are: (1) A comprehensive analysis of various static tools for detecting and mitigating library hallucinations, (2) Manual annotations of three NL-to-code benchmarks on open-source and closed model completions with labels on hallucinations, and (3) A framework for inspecting docstrings and transforming those into grammar form for constrained decoding on open-domain code. 1
2
Related work
However, in the context of bug detection, Chen et al. (2025a) compared static analysis tools with finetuned and pre-trained bug detection models. They found that when the tool and the model were used together, recall increased, and precision also increased on annotated programs. This shows the importance of well-annotated packages for accurate static analysis. We relied on static code analysis to prevent faulty code generation, while instructing the LLM to repair bugs detected by these tools. Inference Based Solutions In package hallucination, Spracklen et al. (2025) demonstrated that hallucinations are not due to sampling, as they still occur with greedy inference. Fu et al. (2024) used constrained decoding to produce more secure code, which is different from our goal, but supports the feasibility of our approach. Roy et al. (2024) perform constrained decoding to check a conversation intent and factuality in self-created APIs, but not in real-world use. Chen et al. (2025b) tested API use, but did not evaluate execution, nor requests from natural language. While they use library aliases as prefixes, when compared to our approach, using a grammar can provide additional validation on syntax and semantics. Grammar-Constrained Decoding Focusing on errors, Olausson et al. (2023) concluded that the effective rate of code repair directly relates to the quality and size of the LLM. In contrast, Geng et al. (2023) found that grammar-constrained decoding, without finetuning and with scarce data, boosts any size of LLM on structured tasks defined through formal grammar. Ugare et al. (2024) used context-free grammars to address syntax errors, reducing them by 96% on Python and Go, which clearly demonstrates the potential of this approach to resolve issues in output. Similar to our work, SYNCHROMESH (Poesia et al., 2022) explored grammar constraints on SQL and JSON output, derived from samples. While flexible, that has the disadvantage that it is not possible to guarantee consistency with an external library. Benchmarks Benchmarks on code hallucinations (Hallucode; Liu et al. 2024a; CodeMirage; Agarwal et al. 2024) treat this problem as a classification task, where the language model needs to detect the type of hallucination from a given snippet of code. They tend to focus on logical errors or inconsistency with the request, whereas we are interested in library use, leading us to evaluate with opendomain code execution benchmarks. CodeRag-Bench (Wang et al., 2025) evaluated
Hallucinations in Code Tian et al. (2025) define code hallucination as code generated by LLMs that might be syntactically and semantically plausible but cannot execute or meet requirements. They consider code errors as a type of code hallucination. Our definition, described in the next section, is slightly different. We do not consider all errors to be hallucinations. Existing work in AI on addressing code hallucinations uses approaches such as retrieval-augmented generation (RAG), iterative grounding, few shot prompting, and fine-tuning (Eghbali and Pradel, 2024; Liu et al., 2024a; Agarwal et al., 2024; Mok et al., 2024; Li et al., 2023). Although these approaches sample better quality tokens, they differ from our work by not addressing detection and mitigation simultaneously (Tanzil et al., 2024), and cannot guarantee that hallucinations have been resolved. There has been related work in Software Engineering on Automatic Program Repair (APR), but this is a more general challenge (Xia et al., 2023; Koutcheme et al., 2023; Prenner et al., 2022; Liu et al., 2024b; Zhang et al., 2023; Wuisang et al., 2023; Ni et al., 2024). Our research aims to mitigate hallucinations by detecting and constraining them before the code is executed and has an error. Hallucinations and External Knowledge Sources in Code LLMs pre-trained on specific tools produce inconsistencies and hallucinations in APIs (Roy et al., 2024), and LLMs optimised for code produced higher rates of package hallucinations (Krishna et al., 2025). Ayala and Bechard (2024) explored finetuning on JSON-structured workflows with annotated suggestions. When they compared a model with and without augmentation, they observed a loss in diversity and an increase in hallucinations. Eghbali and Pradel (2024) proposed a RAG approach for factual open-domain code, which iteratively queries an LLM with API references as context. RAG reduced hallucinations by 3%, and the iterative approach led to a 15% improvement in matching the exact imports, though they did not confirm that the code was executable. Static Analysis in Code Generation Ding et al. (2023) suggest that static program analysis from generated code can reduce errors and resources when evaluating code generation with executionbased benchmarks. Jaoua et al. (2025) showed that RAG with knowledge bases and static code analysis can be a cost-efficient method for code reviews. 2
the use of RAG to improve accuracy in several NL-to-code benchmarks. CodeRag-Bench focused on Python benchmarks, as it is the most widely used language for code generation. This popularity contributes to a higher number of benchmarks compared to other programming languages, which limits the analysis outside of Python.
3
docstrings from the documentation for the library and a list of common aliases (e.g., np for numpy). We constructed our list of common aliases automatically based on how they are used in the benchmarks we consider. For additionally information about the construction of the grammar please see Appendix A. To extend to other languages would involve more manual effort. We only consider Python because of its wide use and the prevalence of Python-based benchmarks.
Defining Code Hallucination
LLMs can make a variety of errors in code generation. We aim to be consistent with the definition of a hallucination from the ‘Speech and Language Processing’ textbook by Dan Jurafsky and James H. Martin: “A hallucination is a response that is not faithful to the facts of the world. That is, when asked questions, large language models sometimes make up answers that sound reasonable” (Jurafsky and Martin, 2024). We apply this definition to code by treating the ’facts of the world’ as existing libraries, APIs, or user-provided context. For example, generating a function name that does not exist in a library is a hallucination, as it violates external factual knowledge. On the contrary, not all errors are hallucinations. For example, a syntax error in code is comparable to a grammatical error in language: a violation of internal language constraints.
4
4.1.1
We use the grammar to constrain the output of the LLM to be consistent with what the grammar permits. During inference, we constrain the probability distribution of the next token to only have positive values for tokens that are valid according to our grammar. Which tokens can be positive is determined by a parser that matches partial sequences with the grammar we provide. This approach has the advantage that it only filters out invalid outputs, with no impact on scoring for valid outputs. We use the parser and decoding integration provided by llama.cpp. This has the advantage that our approach is compatible with any of the opensource LLMs hosted in Ollama. To be efficient, llama.cpp actually samples without filtering the space of output tokens, and if the sample is accepted by the grammar it continues without needing to compute the mask over the output token options. To further improve speed, we implemented caching the states from the non-deterministic pushdown automata used by llama.cpp.
Detection Methods Evaluated
To detect hallucinations, we consider three types of approaches: (1) using a grammar that checks library calls, (2) off-the-shelf static code analysis tools, and (3) a simple LLM-as-a-judge baseline. To mitigate the errors these tools identify, we instruct an LLM to repair the code using the tool’s analysis output as guidance. For the grammar based method, we also consider constrained decoding as a mitigation strategy. 4.1
Grammar-constrained decoding
4.2
Off-the-Shelf Analysers
Our second approach is to use off-the-shelf Static Code Analysis tools. Specifically, we consider mypy (an explicit type annotation tool) and pyright (a type inference tool). These were developed to detect errors in human-written Python code. They are more general than our grammar approach, but are also more language-specific. We configured both tools with automatically generated type stubs to provide information about the functions in the standard library and third-party libraries. Providing these annotations is critical for identifying the library features for our use case.
Automatically Extracted Grammars
In this approach, a GBNF grammar defines whether code is consistent with library definitions. The grammar contains the core language definition and additional symbols to cover the contents of libraries. This can be used either with grammar-constrained decoding, or with post-generation analysis. We developed a method to analyse the documentation for a Python library and automatically construct a grammar that will only accept code that matches the library definition. To create grammars for additional libraries all that is required is the
4.3
LLM-as-a-judge
As a baseline alternative, we ask o3-mini to judge whether code will be executable. To do so, we provided the generated code in conjunction with a 3
set of test cases (Haque et al., 2023)2 . Execution Rate (ER) measures how frequently code executes, without considering correctness. RIF is a measure of how many imaginary features remain after repair. For examples of these errors, and which token causes the issue, see Table 6 in Appendix.
closed-answer question to the model. We present an example of this prompt and LLM’s response in Appendix D.2. This approach could be considered a static analysis method since the LLM does not execute the code. However, it does not have the low overhead or behaviour guarantees of traditional static analysis methods.
5
Generation Models We test our approaches on the output of four LLMs: Claude-3, GPT-4, and GPT-3.5, and IBM-Granite 3B. For the API based models, we used a temperature of one, and for the open source model, we used 0.4. These values are as reported in prior work using these models on these benchmarks. We chose IBM-Granite because of its high scores on DS-1000. The model is finetuned for tasks such as code fixing and explanation (Mishra et al., 2024). We validated samples for all 4 models on all questions in DS-1000 and BigCodeBench, and 550 questions from Odex (not the full dataset because we skipped questions that do not require library imports).
Experiments
Benchmarks Recent benchmarks developed to target hallucinations in code (Collu-Bench; Jiang et al. 2024; CodeHalu; Tian et al. 2025, APIHulBench; Chen et al. 2025b) either lack NL descriptions or evaluate with an exact match, rather than using test cases. In this work, we are concerned with hallucinations in code as a response to instruction prompts related to library usage. So, we required benchmarks with two key characteristics: (1) a natural language prompt that demands library usage and (2) test cases. Our evaluation focuses on Python because it is the most widely used language for coding tasks, and benchmarks with the stated characteristics are not available in other programming languages. Therefore, we evaluate with three open-domain code execution benchmarks: DS1000, which has 1,000 problems over 7 libraries (Lai et al., 2023), Open-Domain EXecution-based natural language (ODEX), which has 945 problems in four natural languages (en, es, ja, ru) over 79 libraries (Wang et al., 2023) and BigCodeBench (instruct), which has 1,140 problems over 139 libraries (Zhuo et al., 2025). In all of them, answers involve multiple libraries, either explicitly specified in the request or implicitly needed. Rather than using grammars that cover all libraries for every question, we use the ones needed: explicitly indicated ones and common related libraries.
5.1
Detection
Table 1 presents precision and recall for our methods and the baseline. We see three distinct patterns of results. First, the grammar has the lowest recall, reflecting the focus on library-related errors, which constitute a small subset of the errors, ranging from 8.1% to 40% of errors. While grammars are commonly used to fix syntax bugs, we noticed that SOTA models occasionally generate these errors with a minimum of one occurrence (0.01%) and a maximum of 15.6%. The highest cases occur due to the interchangeable generation of code and natural language explanations, reinforcing the relevance of inspecting library-related hallucinations. Specifically, in hallucination detection from SOTA models, the static analysis tool Pyright outperforms the use of LLM-as-a-judge. On all types of errors, the baseline, o3-mini, has high precision and low recall. Mypy and Pyright, the off-the-shelf static analysis tools, have very similar results, with higher recall than precision, while Pyright better balances between the two than o3-mini most of the time. However, this depends on the benchmark.
Metrics For detection, we consider (1) all errors, and (2) hallucination specific errors. For (1), we measure precision and recall. For (2), we introduce precision and recall on Imaginary Features (IF), which focuses on library hallucinations by measuring the percentage of errors that are one of four types: attribute, import, type 1 , and module not found. For mitigation, we use three metrics. Pass@k (P@k) evaluates functional correctness and returns the ratio of k samples passing a
Annotations remain a challenge for type inference. One key issue with the grammar approach is the limited information in docstrings. For example, docstrings do not define namespaces, and so int32 in result = tf.random.uniform(...,
1 We think type errors are important to measure since an LLM might generate nonexistent parameters in functions, or map incorrect data types to function calls.
2
We evaluate using Pass@1; so the LLM has a single run to give the correct answer.
4
claude-3 OP OR
All Errors gpt-3.5 gpt-4 OP OR OP OR
Granite OP OR
DS-1000 Mypy Pyright o3-mini Grammar
0.22 0.61 0.63 0.36
0.27 0.70 0.43 0.14
0.26 0.57 0.61 0.45
0.34 0.60 0.45 0.11
0.16 0.50 0.55 0.53
0.23 0.62 0.43 0.15
0.63 0.95 0.79 0.57
0.26 0.55 0.51 0.10
0.22 0.85 0.29 0.16
0.23 0.75 0.29 0.15
0.26 0.84 0.26 0.20
0.21 0.52 0.68 0.17
Odex Mypy Pyright o3-mini Grammar
0.85 0.76 0.61 0.35
0.15 0.22 0.09 0.08
0.39 0.39 0.61 0.43
0.66 0.68 0.22 0.08
0.43 0.43 0.79 0.52
0.81 0.82 0.32 0.11
0.81 0.72 0.74 0.59
0.18 0.34 0.34 0.15
0.17 0.26 0.08 0.11
0.60 0.64 0.11 0.10
0.70 0.71 0.08 0.12
0.17 0.36 0.34 0.17
BigCode Mypy Pyright o3-mini Grammar
0.16 0.32 0.64 0.19
0.05 0.20 0.12 0.04
0.22 0.28 0.56 0.24
0.07 0.16 0.16 0.12
0.24 0.34 0.57 0.23
0.09 0.20 0.11 0.06
0.47 0.62 0.77 0.45
0.09 0.30 0.48 0.09
0.03 0.24 0.15 0.06
0.03 0.14 0.14 0.11
0.08 0.24 0.14 0.06
0.10 0.36 0.50 0.06
claude-3 IF R
Hallucinations gpt-3.5 gpt-4 IF R IF R
Granite IF R
Table 1: On the left: Detection results for all query models, detection methods, and datasets. We defined OP as the overall precision and OR as the overall recall when detecting all types of bugs. On the right: IF R represents recall only on imaginary features. We do not show IF precision as it was 1 in all cases, except for o3-mini in IBM-Granite on DS-1000, which was 0.98.
dtype=tf.int32) is marked as an error. However, this also occurs with more robust methods, such as Pyright and Mypy, where we observe instances where calls to functions are resolved with type stubs containing kwargs as a parameter in their annotations, for example, on date_range = pd.date_range(start=start, end=end, format=’%Y%m%d’), static analysis tools are unable to detect format as an imaginary feature. With inaccurate and unspecified data types and dimensions in data structures, those hallucinations will be missed. Another limitation is that the grammar must match a library name or alias on every call. To see the issue this causes, consider cv = CountVectorizer(stop_words=’english’, punctuation_pattern=r’\\W’). punctuation_pattern is a hallucination, but the grammar does not identify it because this code doesn’t use the library name or alias in the call, instead importing the object using from ... import CountVectorizer. We could modify the grammar to not require explicit use of the library name or alias, but then we would have additional false positives. Tracking import statements would require features beyond those supported by llama.cpp. 5.2
Caught
TP Overlap Capable
DS-1000 Static Grammar
56.7 37.6 11.2 7.9
17.9 85.7
77.0 18.0
Odex Static Grammar
19.0 6.5
4.0 2.5
12.5 20.0
70.5 10.5
BigCode Static Grammar
23.0 10.0 8.5 5.0
5.0 100.0
48.5 7.0
Table 2: Manual analysis of a sample of hallucinations across benchmarks. Where Static is the union of a hallucination been flagged by either Mypy or Pyright. See 5.2 for metric definitions.
200 failure cases for Imaginary Features on each benchmark, with an even sample size across models. We labeled them according to whether we believed a static analysis approach could feasibly catch them, and why or why not this is possible due to their root cause. Annotation procedure 25% of the annotations were labeled by two domain annotators for two purposes: a) to validate reliability and b) to establish guidelines and a systematic rubric for annotation, which can be found as a supplemental material of this work. This resulted in an average Cohen’s kappa of 0.7863 across the eight annotated labels. All open-closed labels exceeded 0.8 agreement, while the categorical labels ranged from 0.69 to
Investigating Potential
To identify the upper bound on performance of static methods, we manually inspected a sample of 5
0.77. These results demonstrated sufficient annotator reliability to proceed with a single annotation for the remaining dataset. More details on inter-annotation agreement and a description of the categorical labels can be found in Appendix D.1. We consider four metrics: Caught is the percentage of cases that were flagged. TP (True Positives) is the percentage of cases where the hallucination was correctly identified for the right reason. Overlap is the percentage of cases identified by one tool that were also identified by the other tool. Capable is the percentage of cases we believe the method could potentially catch (and for the correct reason rather than by chance). In Table 2, for Caught and TP columns, we observe a high rate of False Positives (FP) when manually examining why the hallucination was generated. Due to the FP, we observe variability in the Overlap column between the detected samples. However, in Odex, this is when the grammar is far from static. This is particularly evident in the datetime library, where the way the library is imported collides with the built-in and module definitions, resulting in high precision and recall for the grammar, but opposite to static tools.
Hallucination cause
DS-1000 Odex BigCode
During generation Flow Library Data-Logic Ambiguous input
57.1 14.7 8.5 0.0
42.4 6.6 6.1 6.6
34.2 6.0 7.5 7.5
During test Test case error Ambiguous output Test breadth
2.8 1.1 15.8
4.0 30.3 4.0
6.5 26.6 11.6
Prompt (Char count)
871.8
87.5
663.2
Table 3: Categories that triggered a hallucination and their percentages on the sample of each benchmark.
capable of detecting 70.5% of hallucinations; however, this is because many cases of Odex do not return anything in the function, resulting in returning None being an easy case for the Static tools to identify. However, this is not the case for BigCode, with a more diverse source of hallucinations. What causes hallucinations? For each hallucination in the sample, we also annotated the cause and whether it occurred during the LLM’s code or while processing the test. These results yield insights into what hallucinations are feasible to solve. There were three types of issues we considered infeasible, all of which related to prompt specification matching the test cases. (1) When a calculation requires the input data to design the code, e.g., a column with mixed data types. (2) When the request does not specify the input or output data type, or the prompt is ambiguous about the library it uses. (3) Execution errors that occur in testing code and are not part of the generated code. Table 3 presents categories of the source of each hallucination in the sample of each benchmark in percentage. We consider hallucinations that are feasible to detect as those related to Control Flow and Data Flow (Flow) in static tools, as well as rule definitions that match the Library definitions. These two represent the majority of cases in DS1000. However, for Odex and BigCode, we have a more diverse distribution. For type 1, hallucinations that depend directly on the knowledge of the LLM, such as setting a variable to a negative value without reason and then encountering an exception because it requires positive numbers, are definitely outside the scope of the tools.
Blind spots in static tools Note, the Capable column is our manual analysis of cases where we believe the general method has the potential to detect a hallucination, not necessarily to fix it. Based on our observations in the process of doing the analysis, we believe that the static tools can do better on Data and Control Flow operations. For the first case, depending on the problem and library information, datatypes might mutate during the program. For example, when given a dataframe as input and calculating the average of a single column, it returns a series. However, in the case of two columns, a dataframe is returned. Therefore, static tools should consider these cases as part of their data flow. In the case of control flow, the benchmarks use a function in their test pipeline that will be automatically run in the test suite later. In several cases, the static tool cannot access the code inside the function and therefore misses analysing this code. Another instance of this type of error occurs in Lambda functions, within an apply or map function. The tools skip this nested flow, and if implemented, it will increase detection in the DS-1000 benchmark, as this is a standard for data science libraries using dataframes. In Table 2, we see that on Odex, static tools are
Blind spots in benchmarks’ design The remaining cases are independent of the LLM or the static analyser and depend entirely on the quality of the 6
claude-3 ER P@1
gpt-3.5 ER P@1
gpt-4 ER P@1
Granite ER P@1
claude-3 RIF
gpt-3.5 RIF
gpt-4 RIF
Granite RIF
DS-1000 o3+Self o3+Mypy o3+Pyright No repair
82.3 80.0 80.3 73.1
44.3 43.7 44.3 42.7
82.8 80.3 79.9 73.2
37.9 37.7 37.4 36.6
85.3 82.5 82.1 76.8
49.3 49.1 48.7 47.7
56.3 45.5 44.5 33.6
12.0 10.1 10.1 8.9
2.7 2.9 3.2 9.1
2.6 3.3 3.5 9.7
1.9 1.7 2.4 8.1
5.2 8.1 8.6 11.4
Odex o3+Self o3+Mypy o3+Pyright No repair
70.3 71.1 71.3 66.9
43.6 44.2 44.0 42.2
67.3 67.3 67.7 59.6
40.4 40.2 40.6 35.8
67.5 67.5 68.1 57.0
43.2 43.0 43.2 35.2
50.3 48.5 50.3 46.3
19.6 19.0 19.4 18.0
16.6 16.2 16.2 20.4
15.8 16.8 16.4 15.8
16.0 15.6 16.0 13.7
23.0 24.2 23.4 24.0
BigCode o3+Self o3+Mypy o3+Pyright No repair
82.6 79.0 80.0 79.0
47.3 45.5 45.9 45.5
82.1 77.9 78.3 77.6
41.3 39.4 39.4 39.3
82.6 80.0 80.4 79.6
47.2 46.1 46.1 46.0
77.7 62.2 66.7 61.0
26.7 21.1 23.2 20.6
29.8 30.5 28.9 30.1
34.3 35.7 36.4 35.7
35.4 36.4 34.5 36.6
39.0 39.0 38.2 40.0
Table 4: Code performance with various repair methods. On the left: The percentage of responses that are executable (ER) and the ones that are correct (P@1). On the right: The percentage of imaginary features that remain after repair.
5.3
benchmark. For type 2, an execution error may occur during the generated code if the prompt does not specify the type of input. A common case in BigCode is using the keyword ‘Dataset‘. The LLM designs code for a dataframe, but the function is meant to use a NumPy array as input and output. Another example is test cases containing None in the data. While some prompts specify the requirement to consider data that might contain None values, others do not, and have a test case that evaluates this. These two causes in DS-1000 represented 1.1%, while in Odex 36.9%, and in BigCode (instruct) 34.1%. We associate these with the prompt character count; DS-1000 has almost 800 more characters than Odex and almost 200 more characters than BigCodeBench.
Repair
In this section, we turn from detecting errors to repairing them. We evaluate on the examples that contained an error when executed: 1383 samples in DS-1000, 1171 in BigCode, and 825 in Odex (across all four generation models). To repair the error, we prompt an LLM with the code, the error, and a request to resolve the issue. We try o3-mini combined with static analysis tools, or no tool at all. We average over three runs. Table 4 shows that our approach consistently improved all metrics, particularly execution rate (ER) and how many Imaginary Features remain (note that for RIF, lower is better). Interestingly, o3-mini performs better without information from static analysis on DS-1000 and BigCodeBench, but not on Odex; these results might be related to the quality of the benchmarks.
For type 3, we observed that the three benchmarks contain test cases that fail during the initialisation of input data. We labeled these as Test case error and found that they appear 2.8-6.5% of the time. Finally, with benchmarks that test the breadth and depth of library function calls, we consider Test breadth as whether the assertion cases have different ways to query the library from a valid solution. An example is when asking to set a title in a plot; there are several ways to evaluate this. One could use either ax.get_title() or plt.gca().get_title() depending on how the LLM sets the title; however, these three benchmarks only consider one of those options. In reality, we need to consider different valid options in the library to account for diverse valid solutions.
5.4
Mitigation
Finally, we consider avoiding generation of errors entirely by using constrained decoding. This can only be done with the grammar approach and an open-source model. Figure 1 shows an example of how the grammar can help. Without constrained decoding, the LLM produced an imaginary parameter ‘use_line_collection‘, but with constrained decoding, it generates valid parameters. Table 5 shows that the benchmarks resulted in a lower rate of imaginary library features when the model was constrained in DS-1000, but a moderate improvement 7
# make a stem plot of y over x # and set the orientation to be horizontal plt.stem(x, y, use_line_collection=True) plt.show() # make a stem plot of y over x # and set the orientation to be horizontal plt.stem(x, y, linefmt=’C0-’, markerfmt=’o’, basefmt=’C0-’) plt.xlabel('x') plt.ylabel('y=e^{sin(x)}')
ER
P@1
RIF
DS-1000 Unconstrained Constrained
33.6 36.3
8.9 6.5
11.4 8.0
Odex Unconstrained Constrained
46.3 45.9
18.0 18.6
24.0 23.0
BigCode Unconstrained Constrained
58.1 54.2
16.0 12.7
44.1 46.6
Table 5: Benchmark results with and without constrained generation using our grammar.
Figure 1: Example of how constrained decoding impacts output. On the top: Unconstrained response with imaginary feature. On the bottom: Constrained response with factual parameters.
added approximately three to eight samplings per request, whereas in Odex, one to three samplings were added. This could be explained by the distribution of rules in each grammar, since in DS-1000, we found more uniform and larger grammar rules, as seen in Figure 6 in Appendix.
in Odex, and no improvement in BigCodeBench 3 . The results for ER and P@1 are more variable. All shifts are small, which is consistent with the low detection rate of the grammar-based approach. Why did Pass@1 decrease? This relates to the difficulty of getting accurate information from docstrings. When the correct answer is not defined in the grammar, the constrained output may be wrong. This is consistent with work on extracting hyperparameter constraints from machine learning operators, which found low precision in docstrings and proposed other methods like weakest-precondition (Rak-Amnouykit et al., 2021). Another case is when the LLM has memorised an answer or only knows one approach to the problem; even correct candidates in the grammar might not appear in the LLM’s pretraining data. Recent work on hallucination benchmarks (Ravichander et al., 2025) suggests that coding tasks, such as library hallucinations, appear in pretraining data examples. Code that may be right in a specific document, in isolation, may be incorrect when used later. We observed that an unconstrained response takes an average of 22.3 seconds, while a constrained one takes an average of 34.76 seconds, which is 10 seconds longer; however, this variance depends on the number of resamplings. Therefore, we evaluated Sampling cost (Olausson et al., 2023). This is the total number of tokens sampled from the model. We found that the evaluation on Odex resampled half of the time compared to DS-1000. In DS-1000, the constrained approach
6
Discussion
The results we found indicate that static tools can detect up to 85% of hallucinations. However, manual inspection shows that some of these results are by chance, and the true upper bound is closer to 77%. These tools excel in efficient resource utilization for decoding and evaluating code, compared to compilation and runtime steps. Integrating Static Analysis functionality into the decoder will not alleviate all types of hallucinations. This approach was studied in the past by Melcer et al. (2024), who combined lexer functionality in the left-most parser to account for indentation. They suggested that their approach could be used with static analysis for verifying partial programs. Similiar ideas have been proposed by Ding et al. (2023) in Python and by Mündler et al. (2025) in TypeScript. In these works, the completeness of static analysis tools is assumed, as demonstrated in our detection experiment. Static analysis requires further work on inferring types in dynamic type languages without annotations, such as Python. Previous work on grammar-constrained decoding suggests that API calls could be a possible application (Koo et al., 2024; Poesia et al., 2022). Grammar constraints gained popularity on locally friendly frameworks, such as llama.cpp. These motivated us to design and test a grammar that can identify when the library is being used through an alias or name. However, we find that at most 11% of hallucinations can be detected through grammar
3 In Table 4 we used IBM-Granite’s responses from the BigCodeBench Leaderboard; however, there was no specification of the IBM-Granite version, so we re-run IBM-Granite with our version to do a fair comparison with our constrained model.
8
constraints. From our evaluation, this approach is less precise, as it cannot track of scopes and code’s data flows as fully as static analysis tools. We expect this analysis paper to provide insights into the missing elements of static tools, with the aim of helping NL-to-code users detect bugs introduced by these hallucinations.
7
terminal trees. In Syncode (Ugare et al., 2024), they precompute this set through the union of boolean masks, highlighting the importance of grammarconstrained decoding optimisation for its adoption in structure but non-deterministic tasks. As mentioned by Geng et al. (2023), abstract vs. concrete syntax problem occurs on LLMs trained with different tokenisation methods like Byte-Pair Encoding (BPE) since the token can be tokenised in more than one way and reject plausible candidates; for example, in our use case, the LLM might tokenise pd.DataFrame as pd. ... Data ... Frame. Adapting methods such as the one Koo et al. (2024) proposed by which detokenise characters into tokens and back to text using Finite-state transducers (FST) could decrease sampling cost by accepting partial tokens and decrease time through traversing multiple symbols in the PDA at once; this means reducing the number of next states to parse. Given the similarity of their method to current llama.cpp implementation, this is a viable direction for our work. Although this analysis is Python-specific, our approach can be replicated in other programming languages; however, to do so, we will need more benchmarks with natural language requests and test cases to evaluate a diverse set of libraries. Additionally, Section 5.1 revealed that current codeexecution benchmarks are far from perfect. We discovered that up to 6.5% of the questions contain erroneous code in their test cases, and up to 30.3% of the prompts are ambiguous, hindering accurate evaluation. We suggest that future code execution benchmarks require stricter standards to create natural language prompts that accurately match test cases, as well as broader test suites to account for different valid solutions across various libraries, thereby enabling a fair evaluation of LLM capabilities.
Conclusions
We investigate the potential of static analysis methods to mitigate hallucinations in a type-inferred programming language, such as Python. This work is the first to evaluate grammar-constrained decoding as a method for preventing LLMs from generating code that uses imaginary library features. While the biggest strength of static tools relies on detection, it might not be transferable for repair or mitigation. Our manual analysis reveals the blind spots that emerge when encountering hallucinations with these methods, providing further opportunities for improvement. We also note that methods were affected by poorly designed prompts that influenced the output quality and, consequently, the performance of the detection tool. Finally, we suggest that work on detecting code hallucinations will need to employ different methods; one possible avenue is to consider the internal state of the model.
8
Limitations
One of the main limitations in our approach is mining docstrings, since the quality of the grammar depends on how well these are retrieved. As mentioned in Section 5.1, docstrings do not define namespaces, and ctype libraries have less welldefined docstrings. Another limitation is that we only analysed the first error in execution; we did not track whether our method solved this same error or encountered a new one of the same type in another line of code. Another limitation in the analysis of the Section 5.1, is that we used a sample; the percentages we show are a summary of the sample. We are unsure whether the ratios might not hold for the full dataset; however, we are making the annotations publicly available so that more researchers can either agree with or refute our work. On another subject, our approach explores the memoisation of parsing states as an optimisation for grammar-constrained decoding. Other approaches, such as Domino (Beurer-Kellner et al., 2024), have optimised grammar-constrained decoding by unifying the transverse of precomputing sub-
References Vibhor Agarwal, Yulong Pei, Salwa Alamir, and Xiaomo Liu. 2024. Codemirage: Hallucinations in code generated by large language models. In 2nd IJCAI Workshop on no-code copilots Automates, Korea. Orlando Ayala and Patrice Bechard. 2024. Reducing hallucination in structured outputs via retrievalaugmented generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track),
9
pages 228–238, Mexico City, Mexico. Association for Computational Linguistics.
Daniel Jurafsky and James H. Martin. 2024. Question Answering, Information Retrieval, and RetrievalAugmented Generation, 3rd edition, page 289–306. Pearson.
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2024. Guiding LLMs the right way: Fast, noninvasive constrained generation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 3658–3673. PMLR.
Terry Koo, Frederick Liu, and Luheng He. 2024. Automata-based constraints for language model decoding. In First Conference on Language Modeling. Charles Koutcheme, Sami Sarsa, Juho Leinonen, Arto Hellas, and Paul Denny. 2023. Automated program repair using generative models for code infilling. In Artificial Intelligence in Education, pages 798–803, Cham. Springer Nature Switzerland.
Boqi Chen, José Antonio Hernández López, Gunter Mussbacher, and Dániel Varró. 2025a. The power of types: Exploring the impact of type checking on neural bug detection in dynamically typed languages. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, ICSE ’25.
Arjun Krishna, Erick Galinkin, Leon Derczynski, and Jeffrey Martin. 2025. Importing phantoms: Measuring llm package hallucination vulnerabilities. Preprint, arXiv:2501.19012.
Yujia Chen, Mingyu Chen, Cuiyun Gao, Zhihan Jiang, Zhongqi Li, and Yuchi Ma. 2025b. Towards mitigating api hallucination in code generated by llms with hierarchical dependency aware. Preprint, arXiv:2505.05057.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. PMLR.
Hantian Ding, Varun Kumar, Yuchen Tian, Zijian Wang, Rob Kwiatkowski, Xiaopeng Li, Murali Krishna Ramanathan, Baishakhi Ray, Parminder Bhatia, and Sudipta Sengupta. 2023. A static evaluation of code completion by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 347–360, Toronto, Canada. Association for Computational Linguistics.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3102–3116, Singapore. Association for Computational Linguistics.
Aryaz Eghbali and Michael Pradel. 2024. Dehallucinator: Mitigating llm hallucinations in code generation tasks via iterative grounding. Preprint, arXiv:2401.01701. Yanjun Fu, Ethan Baker, Yu Ding, and Yizheng Chen. 2024. Constrained decoding for secure code generation. Preprint, arXiv:2405.00218.
Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. 2024a. Exploring and evaluating hallucinations in llm-powered code generation. Preprint, arXiv:2404.00971.
Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. 2023. Grammar-constrained decoding for structured NLP tasks without finetuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10932– 10952, Singapore. Association for Computational Linguistics.
Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D. Le, and David Lo. 2024b. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Trans. Softw. Eng. Methodol.
Md Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, and Chris Brown. 2023. Fixeval: Execution-based evaluation of program fixes for programming problems. In 2023 IEEE/ACM International Workshop on Automated Program Repair (APR), pages 11–18. IEEE.
Daniel Melcer, Nathan Fulton, Sanjay Krishna Gouda, and Haifeng Qian. 2024. Constrained decoding for fill-in-the-middle code language models via efficient left and right quotienting of context-sensitive grammars. Preprint, arXiv:2402.17988.
Imen Jaoua, Oussama Ben Sghaier, and Houari Sahraoui. 2025. Combining large language models with static analyzers for code review generation. In 22nd IEEE/ACM International Conference on Mining Software Repositories, MSR 2025, Ottawa, Canada, April 28-29, 2024. ACM.
Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, Manish Sethi, Xuan-Hong Dang, Pengyuan Li, Kun-Lung Wu, Syed Zawad, Andrew Coleman, Matthew White, Mark Lewis, Raju Pavuluri, and 27 others. 2024. Granite code models: A family of open foundation models for code intelligence. ArXiv, abs/2405.04324.
Nan Jiang, Qi Li, Lin Tan, and Tianyi Zhang. 2024. Collu-bench: A benchmark for predicting language model hallucinations in code.
10
Jisoo Mok, Mohammad Kachuee, Shuyang Dai, Shayan Ray, Tara Taghavi, and Sungroh Yoon. 2024. LLMbased frameworks for API argument filling in taskoriented conversational systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages 419–426, Mexico City, Mexico. Association for Computational Linguistics.
Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, and Murtuza Jadliwala. 2025. We have a package for you! a comprehensive analysis of package hallucinations by code generating llms. In SEC’25: Proceedings of the 34th USENIX Conference on Security Symposium. USENIX Association.
Niels Mündler, Jingxuan He, Hao Wang, Koushik Sen, Dawn Song, and Martin Vechev. 2025. Type-aware constraining for code LLMs. In ICLR 2025 Third Workshop on Deep Learning for Code.
Minaoar Hossain Tanzil, Junaed Younus Khan, and Gias Uddin. 2024. Chatgpt incorrectness detection in software reviews. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, New York, NY, USA. Association for Computing Machinery.
Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2024. Next: teaching large language models to reason about code execution. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
Yuchen Tian, Weixiang Yan, Qian Yang, Xuandong Zhao, Qian Chen, Wen Wang, Ziyang Luo, Lei Ma, and Dawn Song. 2025. Codehalu: Investigating code hallucinations in llms via execution-based verification. Proceedings of the AAAI Conference on Artificial Intelligence, 39(1).
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023. Is self-repair a silver bullet for code generation? In The Twelfth International Conference on Learning Representations.
Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. 2024. Syncode: Llm generation with grammar augmentation. Submitted to Transactions on Machine Learning Research. Under review.
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations.
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. 2023. Execution-based evaluation for open-domain code generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1271–1290, Singapore. Association for Computational Linguistics.
Julian Aron Prenner, Hlib Babii, and Romain Robbes. 2022. Can openai’s codex fix bugs? an evaluation on quixbugs. In Proceedings of the Third International Workshop on Automated Program Repair, APR ’22, page 69–75, New York, NY, USA. Association for Computing Machinery.
Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. 2025. CodeRAG-bench: Can retrieval augment code generation? In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3199–3214, Albuquerque, New Mexico. Association for Computational Linguistics.
Ingkarat Rak-Amnouykit, Ana Milanova, Guillaume Baudart, Martin Hirzel, and Julian Dolby. 2021. Extracting Hyperparameter Constraints from Code. In ICLR Workshop on Security and Safety in Machine Learning Systems, Virtual, United States.
Marchel Christhoper Wuisang, Marcel Kurniawan, Komang Andika Wira Santosa, Alexander Agung Santoso Gunawan, and Karen Etania Saputra. 2023. An evaluation of the effectiveness of openai’s chatgpt for automated python program bug fixing using quixbugs. In 2023 International Seminar on Application for Technology of Information and Communication (iSemantic), pages 295–300.
Abhilasha Ravichander, Shrusti Ghela, David Wadden, and Yejin Choi. 2025. HALoGEN: Fantastic LLM hallucinations and where to find them. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1402–1425, Vienna, Austria. Association for Computational Linguistics.
Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1482–1494.
Shamik Roy, Sailik Sengupta, Daniele Bonadiman, Saab Mansour, and Arshit Gupta. 2024. FLAP: Flow-adhering planning with constrained decoding in LLMs. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 517–539, Mexico City, Mexico. Association for Computational Linguistics.
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 769–787, Toronto, Canada. Association for Computational Linguistics.
11
from use in the benchmark with a regular expression. As shown in Appendix 4, each step takes less than a second, as indicated by the distribution across all the libraries tested, demonstrating the viability of this approach for integration at a low cost into engineering pipelines. To expand to more libraries, defining common aliases is the only step that requires validation. One limitation of our approach is that it cannot handle atypical aliases because llama.cpp does not allow the grammar to track state. However, atypical aliases are rare and likely to become even more rare as LLM use rises and so the same alias is consistently proposed. Figure 2 shows a slightly simplified snippet of Numpy’s grammar. We highlight in bold the main symbols that verify a program with the grammar. Starting from the symbol root, we can accept all tokens that describe instructions in natural language, except for the prefix of the library name numpy or alias np. If a library prefix is matched, all the following tokens should also match those from the library symbol, e.g., numpy methods, builtins, or constants. Similarly, matching the token import will add as next states the import-from and import-name symbols, forcing the verification of subsequent tokens as Python’s builtins and environment modules. We conducted our experiments on a local computer with an M2 processor, 8-core CPU, 10-core GPU, and 24GB RAM.
Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 14 others. 2025. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions.
A
Grammar Creation
Our grammar is in GBNF syntax4 since that can be directly used with llama.cpp to constrain decoding. Our grammar has three core symbols: Except-library: accepts any token aside from the library’s alias and name, and the keywords "import" and "from". Library: uses the library name or alias as a prefix for built-ins, initialisers, and constants. Import: uses the keywords "import " and "from " as a prefix to validate packages in the environment of the benchmark and Python built-in modules. These allow us to handle any output not related to libraries, including code and even natural language. This is useful as it allows us to focus on identifying issues with library use. Also, it allows for the style of output where there is a brief explanation, followed by a code snippet, and then further explanation in natural language. We construct the rest of our grammar in two steps. First, we use the inspect module to retrieve the target library’s documentation and represent the key information in JSON. Second, we combine information from the JSON data with Python’s core language specification5 to create a single GBNF grammar that covers library use in the context of other code. Each rule accounts for one of the library functions, with variations to account for variations such as optional and repeated arguments. A twostep approach provides modularity that could make adaptation to other programming languages easier in future. Almost all of this processing is automatic. The Python language formal grammar in EBNF form was manually converted into GBNF in 4 hours of manual work. The library grammars were automatically extracted. Common aliases for libraries (e.g., np for numpy) were automatically extracted
B
Data
We used BigCodeBench under the Apache 2.0 license, and DS-1000 and ODEX under cc-by-sa-4.0. We use those for evaluation, and we did not modify their original content.
C
Risks
We do not see any significant risk introduced by this work.
D
Imaginary features
In Table 6, we present an analysis of the generated code and highlight the tokens that we consider to be imagined by an LLM in the library usage context, and that will result in an error when the code is executed.
4 This is a variant of Extended Backus-Naur Form (EBNF) developed for llama.cpp that adds some features from regular expressions. 5 This is available in EBNF, and so we converted it to GBNF.
D.1
Annotation
We provided annotators with an annotation guideline containing five sections. This guideline is avail12
# Connector rules between the library and the GBNF root ::= ( except-library | library | import )+ except-library ::= ( "numpy" [^.] | "np" [^.] | "from" [^ ] | "import" [^ ] ) library ::= ( "numpy." | "np." ) ( np-numpy-methods | np-numpy-builts | np-numpy-const ) class-name ::= ( "n"[^p] | "nump"[^y] ) class ::= class-name (trailer)* # Import statements import ::= ( import-name | import-from ) "\n" import-from ::= "from " dotted-name " import " name ( " as " name)? import-name ::= "import " dotted-name (" as" name)? dotted-name ::= ( python-modules | environment-modules ) # Built-ins and constant rules np-numpy-builts ::= "char" | "compat" | "compat.py3k" | "compat.tests" | "core" ... np-numpy-const ::= "e" | "euler_gamma" | "inf" | "nan" ... np-numpy-methods ::= np-numpy-copyto | ...
Figure 2: Python’s partial GBNF with a subset of rules defining the numpy library.
D.2
able as supplemental material.
Detect
In Figure 3, we show an example of the prompt we used for the LLM-as-a-judge experiment. While the incorrect token is highlighted in red, we also show the LLM’s response. Note that this example is an incorrect assertion made by an LLM. In Figure 5, we show the distribution of the time taken on each step in the grammar construction. In Figure 4, we compare the time taken for each method to detect a bug. Mypy was the fastest detection tool with a mean of 1s, followed by Pyright and grammar with an average of 2.5s, and o3-mini with a mean of 4s.
• Context: here we define what a hallucination is in code and educate annotators about the relevance of each hallucination in a coding environment, using an example. • Data: we describe each column that they need to annotate and summarize the expected output for each column. • Tools: we provide an overview of each tool’s capabilities and limitations (static analyzer and grammar), and we highlight examples of difficult cases.
D.3
Mitigation
Figure 6 shows the distribution of the number of rules used in each question between the benchmarks, and Figure 7 compares the distribution of resample tokens for each benchmark. Figure 8 shows the distribution of the time taken between the constrained and unconstrained versions of IBMGranite in BigCodeBench.
• Examples: we provided 16 pages of examples, each containing the reasoning behind the annotation. Hard cases were run step by step, and the reasoning was provided in a Table. Table 8 lists the labels in the categorical column for the reasons behind the tool’s detection capability, and it summarizes the annotation guideline in the third section, "Tools". Table 7 reports the Cohen’s Kappa agreement for each labeled column. N (valid pairs), which includes only cases where the tool detected something; otherwise, they have the NA "nan" value. As you can see, caught grammar has fewer pairs, since it is the tool with fewer detected hallucinations. When the "caught static reason" column is higher than the "caught static" column, it indicates that an annotator thought a "test error (that occurred in test)" did not apply to the tool’s detection; this was later clarified in the instructions and guidelines.
D.3.1 Memoisation As seen in Figure 9, we found no gains in time spent per token decoded. Further analysis is needed on the latency generated by Ollama and resampling. As for the remaining work on memory usage on pushdown states in both methods.
13
Imaginary Features
Description
TypeError
A common example is a call to a function that does not match the function’s definition, such as parameters with incorrect datatype or name.
Generated code
result = pd.DataFrame(df, columns=df.columns, suffixes=(’_d’, ’_z’))
AttributeError
This occurs when trying to access an attribute that does not belong to the class, module or submodule.
Generated code
C = tf.tensordot(A, B, axes=[[2], [2]]) sess = tf.Session() print(sess.run(C))
ImportError
This happens when trying to solve a built-in class or function from a library, and it is not a member.
Generated code
from sklearn.externals import joblib joblib.dump(fitted_model, ’sklearn_model.pkl’)
ModuleNotFoundError
This module does not exist in the environment and/or in real life.
Generated code
import numpy_indexed as npi result = npi.group_by(accmap).sum(a)
Table 6: Examples and descriptions of type of bugs considered as imaginary features. The tokens considered as imaginary are highlighted in red.
Label Open-close (Yes / No / NA) Caught Static Capable Static Caught Grammar Capable Grammar Categorical (capabilities / NA) Caught Static Reason Capable Static Reason Caught Grammar Reason Capable Grammar Reason Average
N (valid pairs)
% Agreement
Cohen’s Kappa
Interpretation
98 163 17 163
94.9 89.57 94.12 92.02
0.8849 0.7181 0.8496 0.767
Almost Perfect Substantial Almost Perfect Substantial
104 163 17 163
88.46 77.91 88.24 77.3
0.8422 0.7315 0.8111 0.6857 0.7863
Almost Perfect Substantial Almost Perfect Substantial Substantial
Table 7: Cohen’s kappa analysis of inter-annotator agreement between 25% of annotated labels.
14
Static Analysers Can do the same as in the grammar (Syntax + Library) Can keep data and control flow (Flow) cont = CountVectorizer() is of type CountVectorizer, and should have N attributes. Should keep track of (Lambda) Can know global and local scopes (Scope) def random(char) vs from random import random def shift(something) vs shift = Grammars Find syntax errors (Syntax) Closing parenthesis. Variable names cannot start with digits. A toke must be at the right of = Functions and if statements should not be empty and must be followed by a statement block. Docstring grammar (Library) Defines the function in a library Defines a parameter in the function Defines a constant in the library Infeasible TEST: the error occurs during the test suite, so an error message generated from a test case is not handled by these tools. AMBIGUOUS: the tool needs to know the input and output descriptions of datatypes in the prompt to generate code accordingly. LOGIC-DATA: the tool cannot know if filtering a dataframe will make the series empty, it just knows it is a dataframe. Or if the LLM output is too vague or has no code e.g. END SOLUTION. Table 8: Examples and descriptions of annotation categories for the capability of tools to detect errors and their source. Prompt: import numpy as np import pandas as pd import matplotlib.pyplot as plt x = np.arange(10) y = np.arange(10) # Make two subplots. Make the first subplot three times wider than the second subplot but they should have the same height. # SOLUTION START fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4), gridspec_kw={'width_ratios': [3, 1]}) ax1.plot(x, y) ax1.set_title('First␣Subplot') ax2.plot(x, y**2) ax2.set_title('Second␣Subplot') plt.tight_layout() plt.show() Tell me if this code will execute, answer yes or no, followed by a one-line explanation. LLM’s response: Yes - The code correctly creates two subplots with the first being three times wider than the second.
Figure 3: Example prompt and response from LLM-as-judge to detect code that is executable.
15
Figure 4: Distribution of time taken on each step to build the grammar.
Figure 5: Comparison of time taken on bug detection analysis tools.
16
Figure 6: Distribution of the number of rules in the grammar for each benchmark.
Figure 7: Distribution on resampling in each benchmark.
17
Figure 8: Comparison of time taken between unconstrained and constrained.
Figure 9: Comparison of time taken for approaches on grammar-constrained parsing.
18