arXiv:2605.09304v1 [cs.SE] 10 May 2026
Generating Complex Code Analyzers from Natural Language Questions Amirmohammad Nazari
Sadra Sabouri
Wang Bill Zhu
University of Southern California Los Angeles, CA, USA [email protected]
University of Southern California Los Angeles, CA, USA [email protected]
University of Southern California Los Angeles, CA, USA [email protected]
Robin Jia
Souti Chattopadhyay
Mukund Raghothaman
University of Southern California Los Angeles, CA, USA [email protected]
University of Southern California Los Angeles, CA, USA [email protected]
University of Southern California Los Angeles, CA, USA [email protected]
Abstract Many software development tasks, such as implementing features and fixing bugs, begin with developers posing questions about a codebase. However, answering questions about codebases that span millions of lines of code across thousands of files is non-trivial. Standard tools like grep cannot answer questions requiring semantic or inter-procedural reasoning, and large language models (LLMs) struggle with large codebases due to resource and context constraints. In this paper, we present Merlin, a new system for answering free-form questions that require analytical reasoning about code. Merlin integrates an LLM with CodeQL, a program analysis framework that supports expressive queries over large codebases. We face two principal challenges in the design of such systems: First, program analysis queries are diverse and semantically complex; as a result, even syntactically well-formed queries frequently produce degenerate/empty results. Furthermore, relatively few CodeQL queries are available online, limiting the out-of-the-box effectiveness of LLMs as CodeQL query generators. We address these challenges by developing a RAG-based iterative query-generation approach and a novel self-test technique. Our query debugging technique builds on the idea of assistive queries, which generate concrete witnesses that expose and explain semantic flaws in candidate queries. We evaluate Merlin through both experimental and user studies. Over a set of natural language questions derived from common bug-finding tasks, Merlin discovered not only the majority of software issues reported by other approaches, but also issues that would have otherwise remained undetected. Through a within-subject user study, we found that access to Merlin increased task accuracy by an average of 3.8× and simultaneously reduced the time for programmers to complete all tasks by 31%. ACM Reference Format: Amirmohammad Nazari, Sadra Sabouri, Wang Bill Zhu, Robin Jia, Souti Chattopadhyay, and Mukund Raghothaman. 2026. Generating Complex Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
Code Analyzers from Natural Language Questions. In . ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
Asking questions about code is foundational to the software engineering process [21, 36]. Developers routinely query their codebases throughout the entire software lifecycle, from initial onboarding and active feature development to post-deployment maintenance [20, 22]. However, obtaining accurate answers to these questions is a significant bottleneck, known to consume a substantial fraction of development time and effort, even among experts [22, 29]. Among the many questions, a particularly challenging class consists of deep analytical questions that require reasoning over the global structure and semantics of a codebase. Examples include identifying all locations where a specific API is used, finding methods that are overloaded in particular ways, or locating calls to a method where arguments satisfy specific semantic constraints. Unlike local or syntactic questions, these queries cannot be answered by inspecting a single file or code fragment in isolation. Instead, they require combining information across multiple program elements, such as types, control flow, and call relationships. Common programming tools offer limited support for answering these analytical questions. Developers often rely on ad hoc utilities such as grep or IDE features to incrementally explore a codebase. While these can surface individual code locations, they do not provide a mechanism for expressing complex queries or systematically combining partial results into a coherent answer. Alternatively, programmers may turn to large language models (LLMs), which provide a natural interface for posing free-form questions about code. However, applying LLMs to large, real-world codebases remain challenging. Current LLM-based tools typically operate over plain text representations of code and lack a structured, persistent view of the codebase. As a result, answering questions that require reasoning over large codebases is often unreliable or prohibitively expensive, as relevant context may exceed the model’s context window or be only partially represented [2, 9]. To address these limitations, we introduce Merlin, a natural language question answering system designed to support analytical queries over large codebases. Merlin combines the convenience of a natural language interface with CodeQL [4], a declarative program
Conference’17, July 2017, Washington, DC, USA
Amirmohammad Nazari, Sadra Sabouri, Wang Bill Zhu, Robin Jia, Souti Chattopadhyay, and Mukund Raghothaman
analysis framework that provides a structured representation of a codebase and supports a wide range of static analyses. CodeQL models a codebase as a relational database containing syntactic and semantic information about the program, including types, control-flow and data-flow relationships, and other program properties. Analyses are then expressed using a SQL-like language, enabling precise reasoning about program structure and behavior at scale. Merlin uses an LLM to translate a developer’s natural language question into a corresponding CodeQL query, which is then executed over the codebase. The resulting set of code locations and program elements is returned to the user, grounding answers in the results of program analysis rather than in the LLM’s internal textual representation of the codebase. Developing Merlin required addressing two technical challenges. First, the availability of CodeQL queries are limited, which makes it difficult for pre-trained LLMs to reliably generate wellformed analysis queries or to fine-tune open-weight models. To address this, Merlin combines the LLM with CodeQL documentation and compiler feedback in an iterative, RAG-like loop to ensure syntactic correctness. Second, even syntactically valid CodeQL queries may fail to capture the programmer’s intended semantics, as closely related queries can differ subtly in meaning. We address this challenge using a self-test mechanism in which the LLM generates code examples that should or should not satisfy the query, allowing us to detect degenerate queries before applying them to the codebase. While self-tests are effective at detecting semantic mismatches, they do not explain why a query fails or how it should be corrected. To support diagnosis and refinement, we introduce assistive queries, which allow the LLM to generate auxiliary CodeQL queries that expose intermediate program properties relevant to the original question. These assistive queries help illuminate the causes of semantic errors, functioning analogously to diagnostic print statements during debugging. In order to evaluate the Merlin system, we started by developing a set of benchmarks: We observed that bug-finding problems (e.g., finding calls to suspiciously named methods such as Java’s Array.equals(), calls to non-final methods in object constructors, or direct manipulation of string data possibly leading to code injection vulnerabilities) are a rich source of analytical questions about program behavior and structure. We constructed a set of benchmarks by consulting the analyses performed by SpotBugs [15, 40] and GitHub Security Lab [38], and phrasing the underlying problems as natural language questions that programmers might naturally ask. We found that Merlin is not only effective at reproducing responses produced by these baseline tools, but also flags many code snippets that were not identified by the baselines. To understand why Merlin finds so many additional code snippets, we conducted a survey in which programmers annotate identified code snippets as relevant or irrelevant. This survey confirmed that Merlin finds genuinely relevant code snippets that are overlooked by other tools. In fact, when taking human relevance judgments as ground truth, Merlin achieves both higher precision and higher recall than SpotBugs.
Finally, we conducted an end-to-end user study to determine whether Merlin can truly assist programmers in realistic software engineering tasks. We started by presenting programmers with guidelines from the SEI CERT coding standards [28], and asking them to find and fix all violations in a given codebase. As before, identifying violations involved answering some analytical question about the codebase. One group had access to Merlin, while the control group did not (but could use any other tool, including AIbased tools). Having access to Merlin increased task accuracy by an average of 3.8× and simultaneously reduced time to complete all tasks by 31%. Contributions. To summarize, in this paper, we: (1) Present Merlin, a system that integrates large language models with program analyzers to effectively and reliably answer free-form natural language questions about large codebases. (2) Develop a RAG-based iterative procedure and a self-test technique combined with the idea of assistive queries to reliably generate complex program analysis queries, despite the unavailability of large query corpora. (3) Conduct an empirical evaluation, a survey, and a user study which show that Merlin answers realistic developer questions with high precision and coverage.
2
Motivation and Interaction Mechanism
Consider the following coding guideline included as part of the CERT Oracle Coding Standard for Java, a library of guidelines for writing secure Java code [28, 30]: «Ensure that constructors do not call overridable methods.» The goal is to forbid patterns of code such as that shown in Figure 1b: If the constructor of a class 𝐶 invokes a method 𝑚, and the class 𝐶 is subsequently extended into a subclass 𝐶 ′ with an overridden implementation of 𝑚, then the original class 𝐶 might invoke code operating on the subclass before the subclass has had an opportunity to fully initialize itself. After becoming aware of this guideline, the programmer might wish to search their codebase for instances of its violation. Today, they might approach this task in one of three ways: hand inspection, using tools such as grep and relying on various forms of IDE support, or by consulting an AI coding assistant such as Claude Code [3], Copilot [12], or Cursor [8]. Hand inspection is infeasible for all but the smallest codebases. Tools such as grep are focused on the task of finding textual patterns and are unsuited for this task. Finally, there are several challenges in using contemporary LLMs for this type of analysis: (1) The quality of the output is inconsistent. Given the above question and the codebase from our user study, Gemini 3 Pro reports eight locations, all of which turn out to be false positives. In fact, there are only four locations in our codebase that violate the guideline. In other situations, the LLM frequently flags empty or irrelevant lines of code. (2) LLMs and AI coding assistants need large amounts of context to answer such questions. At the same time, they impose resource-use limits, such as on the number of uploaded files. Specifically, Gemini 3 Pro limits uploads to 1000 files. This necessitates a time-consuming partitioning of the codebase into smaller subsets. In addition, this might also lead to errors,
Generating Complex Code Analyzers from Natural Language Questions
as semantically related files may be accidentally separated across partitions. (3) LLM-based analysis suffers from high latency and occasional non-termination. Queries can require substantial execution time and may return no result, necessitating repeated query attempts. These issues are particularly pronounced on large codebases and were frequently observed in our user study.
Together, these observations illustrate the limitations of purely LLM-based tools in reliably performing precise and exhaustive program reasoning tasks. Answering this question fundamentally involves analytical reasoning about the codebase: one has to find all class constructors, identify the methods invoked, and determine whether any of them are overridable. AI agents such as Claude Code are a natural first candidate for tackling this kind of complex reasoning. Given the question and the codebase, Claude Code performs step-by-step reasoning by sequentially invoking external tools such as grep and then aggregating information from their outputs to produce an answer. However, in our experiments, Claude Code (nondeterministically) reports only two locations, neither of which corresponds to any of the four correct locations. Observe also that this question—“Which constructors in my codebase call overridable methods?”—is just one example of a wide variety of questions that are routinely asked by software developers. Asking and answering such questions is often the first part of many software development processes, where engineers gather information about their codebase, and make plans for how to fix bugs and add features. Our system, Merlin, is an LLM-backed program analysis system which allows the user to answer a variety of such questions about their code. We describe the high-level interaction mechanism between Merlin and the user in Figure 1a, and a screenshot of its interface in Figure 2. The user opens the codebase in their IDE, and provides a textual description of their question and the desired schema for the response. The schema specifies the desired number of columns in the output table and a description of the expected contents of each column. Under the hood, Merlin integrates the LLM with CodeQL, a customizable program analysis tool [4]. CodeQL exports a view of the codebase as a database which contains various forms of information about the program, including its syntactic structure, its type declarations and their subtype / supertype relations, the types of its variables, and information about its data and control flow patterns. By querying this database using a language superficially similar to SQL, the system can be used to answer a variety of ad-hoc queries about the codebase. Guided by the user’s input, Merlin repeatedly queries the LLM until it synthesizes a CodeQL query that is subsequently discharged over the entire codebase. The result of the query is presented to the user in tabular form as the system output. The user may click on the presented program locations to directly navigate to different parts of their codebase. The final CodeQL query is also included as justification for the output presented. We show the query generated in response to the “constructors calling overridable methods” prompt in Figure 1c. In our user study,
Conference’17, July 2017, Washington, DC, USA
participants in the control group (i.e., without access to Merlin) frequently expressed concern about the possibility of missed program locations. On the other hand, even though none of the participants had prior experience with CodeQL, a cursory inspection of the query provided them with confidence regarding the exhaustiveness of the analysis results. Unlike purely LLM-based approaches, this queries therefore enable a repeatable, auditable and exhaustive analysis of the entire codebase. As discussed in the introduction, the primary challenge with realizing this approach is that CodeQL is a highly expressive, low resource language with “delicate” semantics (i.e., even slight variations can result in ill-formed queries or queries with substantially different semantics). In the next section, we show how a combination of retrieval-augmented generation (RAG), compiler feedback, self tests, and assistive queries can be used to overcome this difficulty.
3
Workflow of the Merlin System
We start by presenting the high-level workflow of Merlin in Figure 3. Merlin receives a natural language goal, the schema of the desired output table, and a codebase as input, and produces an output table as the result. The algorithm consists of three main components: a preprocessing step, a retrieval-augmented generation based query debugging phase, and a self-test based query verification mechanism. We now describe each of these components in detail. All prompt templates may be found in the directory named prompts/merlin in the attached artifact.
3.1
Setup and Running Example
As another example, say the programmer wishes to find all locations in their codebase where the Object.equals() method is used to compare two arrays. Although arrays in Java permit comparison using the Object.equals() method, this merely checks for reference equality, rather than an elementwise comparison of their contents. Programmers who wish to compare the contents of two arrays must instead use the static two-argument Arrays.equals() method. This method considers two arrays equal if both arrays contain the same sequence of elements, themselves compared according to Object.equals(). To test for reference equality, it is recommended to use the reference equality operators, == and !=. Because of the potential for confusion, the SEI CERT Coding Standards therefore discourage the use of Object.equals() to compare arrays. The programmer writes the goal and the output table schema. Here, they may specify their goal as: «Identify all method calls of the form array1.equals(array2), where array1 and array2 are two arrays.» They may specify the desired output schema as a simple one-column table: «The location of the method call.»
3.2
Question Preprocessing
Merlin begins with a preprocessing phase that obtains two key pieces of information: (a) a set of test cases that reflects the user’s goal, and (b) a subset of the CodeQL documentation covering potentially relevant library constructs. Both of these automatically obtained pieces of information will be used to guide the generation of the analysis query by subsequent modules in the Merlin workflow.
Conference’17, July 2017, Washington, DC, USA Goal
Schema
Identify calls to overridable methods within constructors.
First column: location of the method call. Second column: …
LLM
Codebase 55. class Document { 56. public Document() { 57. render();
User
Merlin Output Table 1. Document.java:57 2. Rendering.java:61 3. MyProject.java:…
CodeQL
Amirmohammad Nazari, Sadra Sabouri, Wang Bill Zhu, Robin Jia, Souti Chattopadhyay, and Mukund Raghothaman
1 class SuperClass { 2 public SuperClass () { 3 doLogic(); 4 } 5 public void doLogic() { 6 System.out.println("Superclass!"); 7 } 8 } 9 class SubClass extends SuperClass { 10 private String color = "red"; 11 public void doLogic() { 12 System.out.println("Subclass! Color: " + color); 13 } 14 }
(a)
(b)
import java from Constructor c, MethodCall mc, Method m where mc.getEnclosingCallable() = c and m = mc.getMethod() and m.isOverridable() select mc.getLocation(), m.getLocation()
(c)
Figure 1: The Merlin user interaction model (1a). Example Java program in which a constructor calls an overridable method (1b). The concern is that the subclass.doLogic() method might access the subclass.color variable before it was initialized. Automatically generated CodeQL query to find violations of the MET05-J coding guideline (1c). Notice the need to access various kinds of information about the program, such as whether methods are overridable and to navigate between method calls and the enclosing function, and to relate these pieces of information using Boolean connectives.
Figure 2: The Merlin user interface. While working with a codebase, the user specifies a high-level question along with the desired output table schema. In response, Merlin returns the resulting output table together with the corresponding CodeQL query. Automatic generation of “self-tests”. An important failure mode for LLM-generated static analyzers is their tendency to produce empty outputs. This is because, even when the query is syntactically well-formed, inherent ambiguities in natural language and the delicate semantics of CodeQL queries lead to candidate queries with degenerate behavior. We see an example in Figure 4c which— despite being otherwise well-formed—fails to detect any of the violations in Figure 4a. We therefore start by prompting the language model to write test cases for a potential static analyzer with the user-provided goal. The snippet in Figure 4a is an example of an automatically generated test case for the question regarding array equalities. We subsequently use these automatically generated snippets as a simple
litmus test that flags an incorrect understanding of the goal by the ultimately synthesized program analyzer. Note also that we use the same session for the entire interaction process with the LLM. As a result, in addition to testing the analyzer, these automatically generated test cases also provide context to improve the quality of LLM responses. As we will observe in our experimental evaluation, these self-tests have the greatest impact on the effectiveness of Merlin. Documentation retrieval. Next, we hope to use a RAG-based approach [24] to provide the LLM with documentation about the necessary CodeQL constructs during the query generation phase. However, we find that the language model frequently hallucinates
Generating Complex Code Analyzers from Natural Language Questions
Input/Output
No
LLM
Yes
Valid
Web Retrieval
Conference’17, July 2017, Washington, DC, USA
CodeQL Compiler
Docs & Assistive Info
Docs
Syntax Correction Loop
Get Docs
Main Synthesizer Update Constructs
Constructs
Get Docs
Docs
Get Constructs
Constructs
Get Keywords
Keywords
CodeQL Query
Assistive Synthesizer
Yes
Output Table/Errors
Semantic Correction Loop
Errors
Assistive Query
Output Table/Errors
Get Assistive CodeQL Query
Execute Assistive CodeQL Query
No
Get CodeQL Query
Execute CodeQL Query
Empty
Yes
No Final Query Question & Schema
Get Test Cases
Test Cases
Figure 3: Overall architecture of Merlin. Merlin first uses an LLM to retrieve relevant documentation and generate test cases that reflect the user’s goal. It then repeatedly uses the LLM to generate a candidate CodeQL query and addresses syntax errors using RAG-based debugging and resolves semantic errors by issuing assistive queries. The final query is executed the entire codebase in order to produce the final table that is presented to the user.
public class myClass { public static void main(String[] args) { Object[] arr1 = {1, 2, 3}; int[] arr2 = {1, 2, 3}; int[] arr3 = {4, 5, 6}; boolean r1 = arr1.equals(arr2); boolean r2 = arr1.equals(arr3); boolean r3 = arr1.equals(arr1); System.out.println(r1); System.out.println(r2); System.out.println(r3); } }
(a)
import java from Call c, Method m, Expr e1, Expr e2 where c.getCallee() = m and m.hasName("equals") and c.getAnArgument() = e2 and c.getQualifier() = e1 and e1.getType().hasName("int[]") and e2.getType().hasName("int[]") select c.getLocation() import java from MethodCall c, Variable v1, Variable v2 where c.getMethod().getName() = "equals" and c.getQualifier() = v1.getAnAccess() and c.getAnArgument() = v2.getAnAccess() and v1.getType().(ArrayType) and v2.getType().(ArrayType) select c.getLocation()
(b)
(c) import java from MethodCall c where c.getMethod().getName() = "equals" select c.getQualifier().getType(), c.getArgument(0).getType()
(d)
import java from MethodCall c, Expr e1, Expr e2, Type t1, Type t2 where c.getMethod().getName() = "equals" and c.getQualifier() = e1 and c.getArgument(0) = e2 and e1.getType() = t1 and e2.getType() = t2 and t1 instanceof Array and t2 instanceof Array select c.getLocation()
(e)
Figure 4: Our running example in Section 3. (4a): The test case generated by the LLMs with an example use of Object.equals() to compare arrays. The snippet contains three violations of the coding guideline, when each of the variables r1, r2 and r3 are initialized. (4e): The final CodeQL query generated by Merlin which successfully detects all three violations of the guideline. (4b): The first CodeQL query in Merlin’s workflow, which contains syntactic errors. (4c): Initial syntactically correct suggestion. Note that because this query is looking specifically for comparisons between arrays of type int[], it does not detect any of the violations in the test case. (4d): The assistive query generated by Merlin. It prints the types of o1 and o2 for every method invocation of the form o1.equals(o2), and helps the LLM to discover that the objects o1 and o2, despite needing to be arrays, need not necessarily only contain integers. when asked to directly list / request the necessary documentation. This is because focusing solely on concepts mentioned in the query leads to a listing of non-existent / irrelevant constructs within the CodeQL standard library. We instead follow an iterative refinement loop, where we first ask the LLM to identify keywords in the user’s question. For the case of our running example, the LLM identifies keywords such as “identify”, “method”, “calls”, “array1”, “equals”, “array2”, “compare”, and “arrays”. Next, we ask the language model to identify constructs from the CodeQL standard library that correspond to these keywords. This results in the list: MethodCall, MethodCall::getReceiver,
MethodCall::getArgument, ArrayType, and ArrayType::getElementType. We validate each proposed construct by consulting the online CodeQL documentation. In the case of our example, we discover that the class ArrayType and the predicates MethodCall::getReceiver and ArrayType::getElementType do not exist. We therefore provide feedback about their non-existence to the LLM, and ask for an updated list of constructs. We iterate until we obtain a list of complete and valid constructs from the LLM. Another appealing aspect of this iterative documentation retrieval process is that it aligns more closely with the hierarchical structure of the CodeQL language reference. For example, analyzer
Conference’17, July 2017, Washington, DC, USA
Amirmohammad Nazari, Sadra Sabouri, Wang Bill Zhu, Robin Jia, Souti Chattopadhyay, and Mukund Raghothaman
classes such as MethodCall and Array are identified first, and their members can be readily identified using documentation from the parent. For example, the documentation for MethodCall contains references to its predicates such as MethodCall::getArgument, thereby reducing the possibility of hallucination [13].
3.3
RAG-Based Query Debugging
Having identified the potentially useful constructs for synthesizing a program analyzer, we now leverage these constructs to generate a well-formed query. We start by prompting the LLM to generate an initial CodeQL query based on the user-specified goal, the desired output table schema, the synthesized test cases, and the relevant retrieved documentation. This initial query is then compiled using the CodeQL compiler. In most cases, the generated query contains one or more syntax errors and misuses of the standard library. For instance, despite our previous discovery that the predicate ArrayType does not exist, the candidate query in Figure 4b hallucinates its existence and therefore fails to compile. To address such errors, we extract the compiler’s error messages and use them to guide a targeted documentation retrieval from the CodeQL language reference. In this case, we fetch the documentation for Expr::MethodCall and use it to prompt the LLM to revise the query accordingly. If the revised query compiles successfully, we move to the next stage. Otherwise, we iterate using the new error messages to fetch more relevant documentation and prompt further revisions. We show the final refined, successfully compiled query for this example in Figure 4c.
3.4
Fixing Incorrect Analyzers with Assistive Queries
Unfortunately, as previously discussed in Section 3.2, even syntactically well-formed queries are overly selective and frequently result in empty outputs when applied to both the generated self-tests and the user’s codebase at large. Indeed, this is the case with the candidate analysis query in Figure 4c, which incorrectly assumes that both array1 and array2 are arrays of integers, while in reality they may contain objects of other types. In particular, instead of checking whether arg1.getType() and arg2.getType() have the textual name "int[]", the query should instead check whether the types are instances of Array type. Notably, the Array class from the CodeQL standard library was not included in the final list of classes identified by the system at the end of the query preprocessing phase. At this point, we suggest to the LLM that we can discharge a query that will help it diagnose and fix the problem, and request it to propose such an assistive query. In response, the LLM proposes a query similar to that shown in Figure 4d. The role of the assistive query is similar to print statements in exploratory programming [18]: These queries are usually not very selective (i.e., have permissive where clauses), and help the system in discovering entities and possible relations between them. In the case of the Object.equals example, the assistive query of Figure 4d causes the language model to discover the Array class which forms a crucial ingredient in the final analysis query in Figure 4e. After feeding back the results of the assistive query to the LLM, we repeat the compile-test-assist loop as shown in Figure 3 until the
query produces a non-empty output on the litmus test. At this point, we discharge the last synthesized query on the entire codebase and present the results to the user as output. From a particular viewpoint, Merlin’s use of a compile-testassist loop, as illustrated in Figure 3, is similar to an AI agent. Merlin can be viewed as an agent that interacts iteratively with the CodeQL and Java compilers within this loop to synthesize a well typed and semantically correct program analyzer that answers a user’s question about the codebase.
4
Experimental Evaluation
Recall that Merlin produces its output in the form of a table, identifying locations in the codebase that match the user’s question. Our evaluation therefore focuses on the following research questions: RQ1. Does Merlin’s output include known answer locations in the codebase? RQ2. Does Merlin identify previously unknown answer locations? RQ3. How heavily (in terms of prompt length) does Merlin use the LLM? Benchmarks. Bug-finding and vulnerability detection systems are a particularly rich source of analytical questions of the kind answered by Merlin. E.g., those discussed in Sections 2 and 3. Our first benchmark is based on the 142 detectors included as part of SpotBugs, an open source static analyzer which flags coding issues within Java projects [40]. Examples include detecting the use of floating-point variables as loop counters, suspiciously named methods such as Object.equal instead of Object.equals, and instances of double-checked locking. For each of these benchmarks, we reused the textual description available on the website, and added the desired output schema by hand. The output schema is used solely to standardize results and enable consistent comparison across tools, and includes basic information about which source locations to report and how. The second set of benchmarks was published by the GitHub Security Lab [38]. It includes CodeQL queries designed to warn of various categories of issues in code written in a range of languages, including Java, C++, C# and JavaScript. We excluded 33 queries that were no longer supported by CodeQL, and 17 queries that were longer than 1,000 characters. This resulted in a set of 40 benchmarks. We manually wrote natural language descriptions for each of these queries. We also used the provided CodeQL query as the reference implementation and attempted to find issues in the example codebases provided in the Security Lab repository. Overall, our benchmark suite consists of 182 natural language questions and reference analyzers, and 13 codebases consisting of 220–5,785 files each and spanning 148,690–6,414,820 lines of code. The questions and schemas may be found in the files named task.txt in the attached artifact. Baselines. We compare Merlin to two alternative approaches to answering questions using LLMs: (a) Submitting the codebase and question text to an LLM and directly requesting an answer. We
Generating Complex Code Analyzers from Natural Language Questions
Conference’17, July 2017, Washington, DC, USA
Table 1: Macro- and micro- precision and recall statistics relative to the SpotBugs and Security Lab reference analyses and median total query length for each approach. The last column lists the total prompt length supplied to the LLM, aggregated as the median across all questions. While Claude Code does not provide fine-grained accounting, we estimate that each question costs 1.08% and 0.04% of the weekly usage limit for paid users, respectively. Numbers in parentheses indicate how many cases were removed due to division by 0. Algorithm
Macro
Micro
Size
Prec.
Recall
Prec.
Recall
Gemini/Question Gemini/CQL Claude-Code/Question Claude-Code/CQL
0.07 (2) 0.24 (148) 0.32 (3) 0.24 (143)
0.13 (24) 0.05 (24) 0.50 (24) 0.06 (24)
0.05 0.06 0.08 0.06
0.04 0.01 0.21 0.02
7.7 MB 451 bytes -
GPT-4o/CQL GPT-4o+Docs GPT-4o+Docs+Compiler
0.50 (175) 0.44 (142) 0.35 (92)
0.02 (24) 0.16 (24) 0.31 (24)
0.99 0.02 < 0.01
0.08 0.32 0.3
451 bytes 22 KB 25 KB
Merlin
0.40 (32)
0.62 (24)
0.03
0.64
29 KB
choose Gemini 3 Pro and Claude Code Sonnet 4.5 for this purpose.1 (b) Submitting the question text to an LLM and requesting a corresponding CodeQL query. We use gpt-4o, Gemini 3 Pro, and Claude Code Sonnet 4.5. We refer to these baselines as Gemini/Question, Claude-Code/Question, GPT-4o/CQL, Gemini/CQL, and ClaudeCode/CQL, respectively. We also compare the performance of Merlin to three ablated versions: (a) directly requesting gpt-4o for a CodeQL query (GPT-4o/CQL), (b) including the documentation generated as part of the preprocessing step before requesting the query (GPT-4o+Docs), and (c) using the compiler to validate the candidate queries for well-formedness (GPT-4o+Docs+Compiler).
4.1
RQ1: Does Merlin’s Output Include Known Answer Locations in the Codebase?
For each benchmark question, we measured the overlap between locations identified by Merlin to those flagged by the reference analyzers. We visualize our measurements in Figure 5. In each figure, the 𝑥-axis represents the number of issues found by the reference analyzer, while the 𝑦-axis represents the number of issues found by both the reference analyzer and the target algorithm. Points located near the 𝑥 = 𝑦 line indicate that the target algorithm identifies most of the issues found by the reference analyzer. From Figure 5a to Figure 5d, we observe a gradual increase in the number of points near this line, indicating improved performance from each component of our approach. In particular, Figure 5d shows the largest cluster of points around the 𝑥 = 𝑦 line, suggesting that Merlin is effective at reproducing the results of the reference analyzers. Quantifying the performance that we visually see in Figure 5 with concrete recall values is tricky because SpotBugs itself does not flag any warnings across the entire codebase for 24 of our benchmarks. Naively calculating task-wise recall can therefore lead to divideby-zero errors. In addition, notice that the number of reported
locations ranges across multiple orders of magnitude across our 182 benchmark questions. Aggregating the results and calculating a single recall value can lead to over-representation from a subset of the benchmarks. We therefore separately calculate macro- and micro-recalls of the different approaches [32], which we report in Table 1. Notice that both macro- and micro-recall statistics improve significantly across the ablated versions of Merlin from 0.02 to 0.62 and from 0.08 to 0.64 respectively. In addition, notice that the most significant improvement comes from a combination of self-tests and assistive queries. In contrast, Figures 5e and 5f show that Gemini and Claude Code struggle to generate CodeQL queries for answering these questions: they often fail to produce even syntactically correct queries. When asked to directly answer the question, notice that Gemini rarely detects issues identified by the baseline (Figure 5g), a result which is consistent with its low macro- and micro-recall values. Anecdotally, many of its reported locations are meaningless (e.g., closing braces around code blocks) or even point to empty lines in the codebase. Claude Code performs better than other baselines in the direct question answering mode (Figure 5h): This is because Claude Code is an agentic LLM which incorporates the output of external tools such as ls and grep as part of its reasoning process. Still, it is less effective than Merlin at reproducing the results of the reference analyzers (macro-recall: 0.5 vs. 0.62, micro-recall: 0.21 vs. 0.64). A deeper investigation reveals that Claude Code is more effective in identifying locations from the SpotBugs benchmarks (macrorecall: 0.55) than over the Security Lab dataset (macro-recall: 0.33). While it is hard to say with certainty, we suspect that this is because the SpotBugs test suite contains numerous hints about target locations, including conspicuously named files and variables and helpful comments surrounding the code in question. Finally, we investigated the points located far away from the 𝑥 = 𝑦 diagonal in Figure 5d. Many of these points are attributable to ambiguities in the natural language questions rather than to failures of the Merlin query generation system itself. One example is the benchmark description from the SpotBugs dataset:2 «Find code that constructs an HTTP Cookie using an untrusted HTTP parameter. If this cookie is added to an HTTP response, it will allow an HTTP response splitting vulnerability.» Terms such as “untrusted” and “suspicious” are inherently vague and open to multiple interpretations, which can lead to variations in how code patterns are identified.
4.2
RQ2: Does Merlin Identify Previously Unknown Answer Locations?
Next, Figure 6 shows the fraction of reported locations that were not also reported by the reference analyzers. In each figure, the 𝑥-axis represents the number of locations reported by the evaluated algorithm, and the 𝑦-axis shows the number of locations that were reported by the evaluated algorithm but not by SpotBugs or the Security Lab queries. These measurements may be quantified as the precision of the respective algorithms which we report in Table 1. 2 https://spotbugs.readthedocs.io/en/latest/bugDescriptions.html#hrs-http-response-
1 File upload limits preclude the use of ChatGPT.
splitting-vulnerability-hrs-request-parameter-to-http-header
100.0 102.0 |Reference|
|Gemini/CQL Reference|
SpotBugs GitHub Security Lab
102.0
(b)
|Claude-Code/CQL Reference|
(a)
103.0
SpotBugs GitHub Security Lab
102.0
101.0
(e)
SpotBugs GitHub Security Lab
100.0 102.0 |Reference| SpotBugs GitHub Security Lab
100.0
100.0 102.0 |Reference|
(f)
103.0
101.0
100.0
100.0 102.0 |Reference|
(d)
102.0
101.0
100.0
100.0 102.0 |Reference|
100.0 102.0 |Reference|
102.0
101.0
100.0
100.0
(c)
103.0
SpotBugs GitHub Security Lab
101.0
100.0
100.0 102.0 |Reference|
103.0 102.0
101.0
100.0
100.0
SpotBugs GitHub Security Lab
|Merlin Reference|
101.0
101.0
103.0 102.0
102.0
102.0
103.0
SpotBugs GitHub Security Lab
|Claude-Code/Question Reference|
103.0
|Gemini/Question Reference|
SpotBugs GitHub Security Lab
|GPT-4o+Docs Reference|
|GPT-4o/CQL Reference|
103.0
Amirmohammad Nazari, Sadra Sabouri, Wang Bill Zhu, Robin Jia, Souti Chattopadhyay, and Mukund Raghothaman
|GPT-4o+Docs+Compiler Reference|
Conference’17, July 2017, Washington, DC, USA
(g)
100.0 101.0 102.0 103.0 |Reference| (h)
Figure 5: Overlap between the issues found by Merlin (5d), its ablated variants (5a–5c), and the baselines (5e–5h), as compared to the reference solutions provided by SpotBugs and Security Lab, respectively. Each point indicates the number of locations identified by one benchmark/detector across the entire code repository. Figure 6d shows that Merlin reports a number of previously unreported locations. While it is tempting to classify these additional locations as being spurious reports, a deeper investigation reveals that these frequently represent bugs that were mistakenly not reported by the reference analyzer. For example, the following detector from SpotBugs: «Find classes which are declared to be final, but which also declare fields that are protected. Since the class is final, it cannot be derived from, and the use of protected is confusing.» fails to report obvious instances of this pattern, including the following (from the file ConfusingParenting.java in the SpotBugs test suite): public final class ConfusingParenting { protected int a; protected Object b; }
This and numerous other similar observations suggested that there were discrepancies between the documentation and what the actual SpotBugs analyzer was doing, and prompted us to conduct the survey in which we asked users to label the ground truth for a randomly chosen subset of reported locations. The results suggest that many of the locations found by Merlin are in fact relevant to the question, and that the precision measurements in Table 1 are likely to be undercounts with respect to the actual ground truth.
4.3
RQ3: How Heavily Does Merlin Use the LLM While Answering Questions?
Finally, we measured how extensively Merlin and each of the baseline approaches used the LLM. The lowest LLM prompt lengths are observed for Gemini, Claude Code, and gpt-4o when they are asked to generate CodeQL queries. This is expected, since the codebase is not provided as part of the context in this setting, resulting in lower prompt lengths and a lighter processing workload. In contrast, the highest context sizes occur for Gemini and Claude Code when they are asked to directly answer questions. In this setting, both the codebase and the question are provided to the LLM, which significantly increases the prompt lengths and computational burden. This direct-question setting also introduced several practical challenges for Gemini and Claude Code. Gemini enforces a limit of 1,000 files per upload, requiring us to split large codebases into smaller sub-codebases. This process is time-consuming and errorprone, as dependent files may be separated across different subcodebases. Moreover, we had to run Gemini multiple times for each sub-codebase, and in some cases the model terminated without producing a response. Claude Code also posed substantial difficulties. It enforces rate limits both across 5-hour windows and across each week. For our codebases, we were only able to ask only approximately 10 questions per hour. Due to the weekly limits, completing the experiments required nearly two full weekly usage cycles. In addition,
Generating Complex Code Analyzers from Natural Language Questions
102.0 101.0 100.0
100.0 101.0 102.0 103.0 |GPT-4o/CQL|
100.0 102.0 104.0 |GPT-4o+Docs|
(a)
SpotBugs GitHub Security Lab
SpotBugs GitHub Security Lab
102.0
102.0
101.0
101.0
100.0
100.0
100.0 101.0 102.0 103.0 |Gemini/CQL| (e)
100.0 101.0 102.0 103.0 |Claude-Code/CQL| (f)
105.0
103.0 102.5 102.0 101.5 101.0 100.5 100.0 10 0.5
SpotBugs GitHub Security Lab
104.0 103.0 102.0 101.0 100.0
100.0 102.0 104.0 106.0 |GPT-4o+Docs+Compiler|
100.0
(c)
(b)
103.0
|Claude-Code/CQL - Reference|
|Gemini/CQL - Reference|
103.0
SpotBugs GitHub Security Lab
|Merlin - Reference|
103.0
106.0 105.0 104.0 103.0 102.0 101.0 100.0
SpotBugs GitHub Security Lab
102.0 |Merlin|
104.0
(d)
|Claude-Code/Question - Reference|
104.0
SpotBugs GitHub Security Lab
|GPT-4o+Docs+Compiler - Reference|
105.0
|Gemini/Question - Reference|
SpotBugs GitHub Security Lab
|GPT-4o+Docs - Reference|
|GPT-4o/CQL - Reference|
103.0 102.5 102.0 101.5 101.0 100.5 100.0 10 0.5
Conference’17, July 2017, Washington, DC, USA
104.0
SpotBugs GitHub Security Lab
103.0 102.0 101.0 100.0
100.0 101.0 102.0 103.0 |Gemini/Question| (g)
100.0 102.0 104.0 |Claude-Code/Question| (h)
Figure 6: Proportion of new warnings (i.e., unreported by the reference SpotBugs and GitHub Security Lab analyzers) that are reported by Merlin (6d), its ablated variants (6a–6c), and the baselines (6e–6h) respectively. Claude Code frequently requested permission to execute system commands, which we had to manually review and individually approve due to privacy concerns. In contrast, Merlin synthesizes program analyzers independently of the target codebase and therefore does not require the codebase to be included in the context. As a result, Merlin exhibits substantially lower context usage. Another notable advantage of Merlin over the other baselines is that once a CodeQL query is synthesized, it can be reused across arbitrary codebases. In contrast, the LLM-based baselines must be rerun for each new codebase, which is significantly more expensive.
5
User Study on Merlin’s Usefulness
Finally, we designed a user study to validate the usefulness of Merlin in real-world software development and compare it to traditional LLMs and program analysis tools. We asked users to perform three tasks in which we presented them with a coding issue and asked them to: find instances of the issue in the given codebase, and fix the code so as to remove the issue. (Although the scope of Merlin is technically limited to only the “answer questions” / “find locations” phase, we included the “fix code” directive simply because we were curious to understand the impact of our system.) We used this study to investigate the following research questions: RQ4. How does Merlin impact the programmer’s question answering process as compared to conventional program analysis or chat-based LLM tools?
RQ5. How effective is Merlin in helping developers locate and fix code issues compared to other assistive techniques? Participants. Once again, we recruited 18 participants through professional forums and the university mailing list. They ranged in age from 18 to 34 years. 17 were male and one female. Participants included 13 students, 1 researcher, 3 professional software developers, and 1 participant with an occupation outside these categories. 11 participants had 6-10 years of programming experience. All our participants had used IDE tools more than once. Six participants were familiar with debuggers like GDB and LLDB, and some code analyzers like Valgrind. Study tasks. From the SEI CERT Oracle Coding Standard for Java [28], we shortlisted 8 coding guidelines which each had fewer than 10 violations in the codebase, and which we determined were feasible to identify and fix within the time allotted to each task in the user study (40 minutes). We randomly chose the following three coding issues from the shortlist: T1. Ensure that constructors do not call overridable methods (MET05J). T2. Do not use the Object.equals() method to compare two arrays (EXP02-J). T3. Do not return references to private mutable class members (OBJ05-J). Study structure. Participants completed the 3 tasks in a counterbalanced within-subject design, where we randomly assigned each
Conference’17, July 2017, Washington, DC, USA
Amirmohammad Nazari, Sadra Sabouri, Wang Bill Zhu, Robin Jia, Souti Chattopadhyay, and Mukund Raghothaman
task as either a treatment condition (with access to Merlin) or a control condition (without access to Merlin). We ensured that each participant completed at least one task with access to Merlin and at least one task without access to Merlin. We randomized task order to mitigate potential order effects and other confounding factors. We informed participants that they could use any available tool to complete tasks. We recorded screens and audio, and stored modified codebases anonymously on a secure drive for subsequent analysis, as per our university’s IRB guidelines. We set a 40-minute time limit for each task. Upon completion, participants completed a post-study questionnaire regarding their experience and provided feedback on Merlin. Data analysis. We calculated participant accuracy by comparing the number of code locations that they identified to the actual number of coding issues present, which we obtained using a hand-crafted CodeQL query. Next, two researchers performed a negotiated agreement to label each of the participant fixes as either plausible (1) or not (0). We calculated fix accuracy as the average across all three tasks for each participant.
5.1
RQ4: Merlin’s impact on answer seeking process
We identified 3 central themes of how Merlin impacted the participants’ approach: Shifting strategies from pattern matching to high level reasoning. Merlin replaced labor intensive pattern matching with high level reasoning. Without Merlin, most participants relied on literal cues such as keyword searches or regexes during the finding phase: 15 participants (83%) used literal matching, while only 11 (61%) engaged in object identification strategies. In contrast, when using Merlin, object identification became dominant: No participants used literal cues, while 14 participants (77%) adopted object identification approaches. Participants attributed this change to Merlin’s ability to eliminate tedious low level work. As one participant explained, Merlin removed the need to “find all occurrences of something by using regex and judg[e] one by one, [which] is [a] tedious task,” enabling them to focus on more intellectually demanding aspects of debugging. Another participant emphasized the magnitude of this shift, noting that “Merlin definitely made that task easy. Doing Task 1 without [Merlin] was basically impossible.” Lowering reliance on external AI. Merlin significantly changed how participants relied on external AI assistance, affecting both the frequency and purpose of AI use. In the control group, participants frequently turned to AI during the finding phase, using it to generate regex patterns for locating potentially buggy code and, in some cases, to fully delegate bug identification to AI assistants. ChatGPT was typically used for pattern generation, while highercapacity models such as Gemini, Cursor AI, and GitHub Copilot were preferred for broader code analysis due to their larger context windows [7]. When Merlin was available, participants rarely used external AI during the finding phase and instead restricted AI assistance to code-fixing tasks. However, reduced reliance on external AI may also introduce new risks: users may over-trust Merlin’s outputs, potentially leading to confirmation bias [45] in which its suggestions are accepted without sufficient critical scrutiny [19, 34].
T1 T2 T3 10
20 30 Time (minutes)
40
Figure 7: Time needed by participants in the user study.
Combining the strengths of LLMs and program analyzers. When debugging tasks were fully delegated to LLMs or AI code agents, participants frequently experienced high latency and received incomplete or inaccurate results, despite the convenience of natural language interaction. Participants consistently highlighted the Merlin’s ability to correctly interpret natural language intent and to support expressive, criteria-driven searches. This capability was particularly valued by those accustomed to rigid keyword based workflows, who emphasized the benefit of describing bugs directly rather than by searching for specific terms.
5.2
RQ5: The Effectiveness of Merlin in Finding and Fixing Code Issues
Time to locate issues. We observed that participants using Merlin spent 31% less time on average to complete the tasks (Figure 7). Indeed, a one-sided 𝑡-test3 on the completion time comparing cases where participants were using Merlin showed a significant reduction for Task T1 (𝑡 = −1.93, 𝑝 = 0.03, 𝑑 𝑓 = 14) with 35% reduction on average and Task T3 (𝑡 = −2.12, 𝑝 < 0.01, 𝑑 𝑓 = 14) with 40% reduction on average. While the average completion time for Task T2 reduced 18% on average, we couldn’t find a significant difference (𝑡 = −0.06, 𝑝 = 0.47, 𝑑 𝑓 = 14). This is unsurprising, because this task was also the most amenable to a standard grep search. Accuracy of identification. Overall, having access to Merlin increased task accuracy by an average of 3.8×. This increase in accuracy was greatest in the case of Tasks T1 and T3, where we observed a substantial 5.6× and 4.9× improvement in average accuracy respectively, and least in the case of Task T2 where we only observed a 1.6× improvement in the average. This is unsurprising, because a simple command such as grep -rn '.equals('. worked for most users in the control group. On the other hand, control group participants in Task T1 attempted but mostly failed to construct a suitable grep query. Some resorted to manually scanning files— an inefficient and ultimately fruitless strategy. In a unique case, one control group participant even tried to directly use CodeQL but struggled to formulate a valid query. Performing a one-sided 3 Shapiro-Wilk tests were non-significant (𝑝 > .05), confirming the normality assump-
tion was met for all tasks spent time.
Generating Complex Code Analyzers from Natural Language Questions
1.00
Merlin Baseline
Average Score
0.75 0.50
Program and Question
3.2
3.1
2.2
2.1
1.2
0.00
1.1
0.25
Figure 8: Accuracy of responses in the usefulness user study. The measurements for Questions 1.1, 2.1, and 3.1 indicate their accuracy in identifying coding issues, while Questions 1.2, 2.2, and 3.2 measure their effectiveness in fixing them. Wilcoxon signed-rank test4 indicated statistically significant improvements in accuracy arising from the use of Merlin for Tasks T1 (𝑊 = 28, 𝑝 < 0.01, 𝑟 = 0.66) and T3 (𝑊 = 34, 𝑝 = 0.01, 𝑟 = 0.63) but did not show statistical significance for Task T2 (𝑊 = 6, 𝑝 = 0.125, 𝑟 = 0.38). Impact on fixing issues. In Task T1, many users addressed the issue by converting overridable methods that were called within constructors into non-overridable ones by marking them final. Experimental group users were more successful than control group users, largely because Merlin had already helped them identify the problematic method calls. Overall, Merlin led to a 4.9× improvement in fix accuracy. Performing a one-sided Wilcoxon signed-rank test revealed a significant improvement with a large effect size in creating fixes while using Merlin (𝑊 = 32, 𝑝 = 0.02, 𝑟 = 55). In Task T2, users commonly fixed the issue by replacing expressions such as array1.equals(array2) with Arrays.equals(array1, array2). However, some users incorrectly applied this fix to arrays with incompatible types. Despite such errors, experimental group users showed a 1.8× improvement in accuracy. A one-sided Wilcoxon signed-rank test revealed a statistically significant improvement with a large effect size (𝑊 = 15, 𝑝 = 0.03, 𝑟 = 0.54). Finally, in Task T3, the typical fix involved cloning private mutable fields before returning them to prevent accidental access to internal state. Some users attempted to incorrectly clone non-cloneable objects. Overall, experimental group users outperformed the control group in both accuracy and time, achieving a 4× gain in correctness. However one-sided Wilcoxon signed-rank test revealed no significant improvement arising from the use of Merlin (𝑊 = 24, 𝑝 = 0.06, 𝑟 = 0.46).
6
Related Work
Customizable program analyzers. CodeQL is a “semantic code analysis engine” that allows programmers to create program analysis queries using a SQL-like language [4]. It is one of a growing family of such systems, including SemGrep [39] and Amazon CodeGuru / GQL [1]. Despite their expressiveness, these systems require the user to learn a domain-specific language (DSL) in order to effectively craft queries. This has motivated the development of natural language 4 Shapiro-Wilk tests were significant (𝑝 < .05) for all task accuracies; therefore, non-
parametric tests were used due to non-normality.
Conference’17, July 2017, Washington, DC, USA
interfaces that are similar to Merlin: Examples include IRIS [26] and MoCQ [25]. To our knowledge, these systems are either limited in the kinds of analysis that are supported (e.g., IRIS is tailored to the setting of taint analysis) or require extensive additional information from the programmer (e.g. MoCQ requires examples of target code and documentation and samples of the analyzer’s DSL). They also target simpler backend analyzers: e.g., [25] acknowledge that CodeQL is a challenging target language and instead primarily focus on synthesizing queries for Joern [17], and [27] target the simpler setting of structural code search, such as that present in IntelliJ [16], the CodeQue Visual Studio extension [6], or Comby [41]. In contrast, Merlin supports a larger range of queries and only requires the question text and schema of the desired output. A recurring challenge in designing such systems is that the underlying DSL is a low-resource language necessitating the use of retrieval-augmented generation (RAG) in helping the LLM to produce the desired analysis query [24]. A notable distinguishing feature of Merlin is our automatic generation of self-tests (instead of user-provided examples in the case of MoCQ), and in the use of assistive queries to diagnose and fix misbehaving queries. In this context, self-tests are closely related to the idea of generating unit tests to verify LLM-generated code [14, 43]. Using LLMs to answer questions about code. Beyond classical transformer-based LLMs [33, 42], more recent agents combine the underlying statistical model with access to external tools such as grep, ls, and find. Examples of such systems include Copilot [12], Claude Code [3], and Cursor [8]. Having access to these external tools greatly improves the question-answering capabilities of these systems, as we observed in Section 4. However, despite using these commands, their final reasoning process remains expensive, opaque, and non-deterministic [10]. In contrast, the final query executed by Merlin serves as a certificate / explanation of its output, and improves the reliability of the system.
7
Limitations and Threats to Validity
Threats to validity. Although we present Merlin as a system to answer analytical questions about code, our benchmarks are only drawn from bug-finding systems. As previously mentioned, this was because research on these topics serves as a ready source of benchmark questions. In addition, as discussed in Section 4, we had to manually compose the natural language questions that described the GitHub Security Lab analyzers. To mitigate possible bias in this process, two other authors of this paper independently reviewed and edited these translations. Third, one might be concerned about the small scale of the user study, which consisted of three tasks and 18 participants of whom 13 were students. We note that the user study of Section 5 required two entire weeks to schedule and conduct. Performing a largerscale evaluation with more participants and a broader range of tasks is a good direction of future work. Finally, we note that Merlin uses gpt-4o as the backend LLM. Therefore, even though we publicly release our artifact, there is the danger of imperfect reproducibility: This is both because of the
Conference’17, July 2017, Washington, DC, USA
Amirmohammad Nazari, Sadra Sabouri, Wang Bill Zhu, Robin Jia, Souti Chattopadhyay, and Mukund Raghothaman
inherent non-determinism of the LLM and its commercial, closedsource nature. Nevertheless, we expect the broader experimental observations to be stable. Limitations of our system. Finally, Merlin is not designed to support the full range of developer workflows, such as interactive debugging or hypothesis-driven exploration, which are known to vary widely across developers and tasks [23, 37]. Supporting a broader spectrum of workflows is an important direction for future work. Our system is currently limited to analyses that the CodeQL backend can perform. Supporting other program reasoning tools such as symbolic execution engines [5] and dynamic analysis frameworks [11, 31, 35] would greatly extend its capabilities. In addition, the system is currently unable to handle imprecise and vaguely defined predicates (e.g., “untrusted” source) and predicates that do not easily permit an exhaustive listing (e.g., functions which return personally identifiable information). Extending the system to detect and interactively resolve ambiguous questions [44] and supporting non-analytical questions (e.g., «Is this code safe?» or «Is this code easy to read?») is an important direction of future research.
References [1] Amazon. 2025. Amazon CodeGuru Reviewer. https://docs.aws.amazon.com/ codeguru/latest/reviewer-ug/welcome.html [2] Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2024. Make your llm fully utilize the context. Advances in Neural Information Processing Systems 37 (2024), 62160–62188. 2025. Your code’s new collaborator. [3] Anthropic. https://www.anthropic.com/claude-code. https://www.anthropic.com/claudecode [4] Pavel Avgustinov, Oege de Moor, Michael Peyton Jones, and Max Schäfer. 2016. QL: Object-oriented Queries on Relational Data. In Proceedings of the European Conference on Object-Oriented Programming (ECOOP). [5] Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (San Diego, California) (OSDI’08). USENIX Association, USA, 209–224. [6] CodeQue.co. 2024. Multiline & Structural Code Search. https://marketplace. visualstudio.com/items?itemName=CodeQue.codeque [7] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261 (2025). [8] Cursor. 2025. The AI Code Editor. https://cursor.com/en. https://cursor.com/en [9] Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753 (2024). [10] Philipp Eibl, Sadra Sabouri, and Souti Chattopadhyay. 2025. Exploring the Challenges and Opportunities of AI-assisted Codebase Generation. In 2025 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 241–252. [11] Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. 2020. AFL++: combining incremental steps of fuzzing research. In Proceedings of the 14th USENIX Conference on Offensive Technologies (WOOT’20). USENIX Association, USA, Article 10, 1 pages. [12] GitHub. 2021. GitHub Copilot. Retrieved 19 July, 2025 from https://github.com/ features/copilot [13] GitHub. 2025. Expr.MethodCall — CodeQL Standard Libraries. https: //codeql.github.com/codeql-standard-libraries/java/semmle/code/java/Expr.qll/ type.Expr$MethodCall.html [14] Siqi Gu, Quanjun Zhang, Kecheng Li, Chunrong Fang, Fangyuan Tian, Liuchuan Zhu, Jianyi Zhou, and Zhenyu Chen. 2025. TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration. arXiv:2408.03095 [cs.SE] https://arxiv.org/abs/2408.03095 [15] David Hovemeyer and William Pugh. 2004. Finding bugs is easy. SIGPLAN Not. 39, 12 (Dec. 2004), 92–106. doi:10.1145/1052883.1052895
[16] JetBrains. 2024. Structural search and replace. https://www.jetbrains.com/help/ idea/structural-search-and-replace.html [17] joern.io. 2024. Joern: The Bug Hunter’s Workbench. Retrieved 29 January, 2026 from https://github.com/joernio/joern [18] Mary Beth Kery and Brad A Myers. 2017. Exploring exploratory programming. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 25–29. [19] Joshua Klayman. 1995. Varieties of confirmation bias. Psychology of learning and motivation 32 (1995), 385–418. [20] Andrew J. Ko, Rebecca DeLine, and Gina Venolia. 2007. Information needs in collocated software development teams. In Proceedings of the 29th International Conference on Software Engineering (ICSE). IEEE, 344–353. [21] Amy J. Ko, Brad A. Myers, Michael J. Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant Information during Software Maintenance Tasks. IEEE Transactions on Software Engineering 32, 12 (2006), 971–987. doi:10.1109/TSE.2006.116 [22] Thomas D LaToza and Brad A Myers. 2010. Hard-to-answer questions about code. In Evaluation and usability of programming languages and tools. 1–6. [23] Thomas D. LaToza and Brad A. Myers. 2010. Hard-to-answer questions about code. In Evaluation and Usability of Programming Languages and Tools (Reno, Nevada) (PLATEAU ’10). Association for Computing Machinery, New York, NY, USA, Article 8, 6 pages. doi:10.1145/1937117.1937125 [24] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 793, 16 pages. [25] Penghui Li, Songchen Yao, Josef Sarfati Korich, Changhua Luo, Jianjia Yu, Yinzhi Cao, and Junfeng Yang. 2025. Automated Static Vulnerability Detection via a Holistic Neuro-symbolic Approach. arXiv:2504.16057 [cs.CR] https://arxiv.org/ abs/2504.16057 [26] Ziyang Li, Saikat Dutta, and Mayur Naik. 2025. LLM-Assisted Static Analysis for Detecting Security Vulnerabilities. In International Conference on Learning Representations. https://arxiv.org/abs/2405.17238 [27] Ben Limpanukorn, Yanjun Wang, Zach Patterson, Pranav Garg, Murali Krishna Ramanathan, Xiaofei Ma, Anoop Deoras, and Miryung Kim. 2025. Structural Code Search using Natural Language Queries. arXiv:2507.02107 [cs.SE] https: //arxiv.org/abs/2507.02107 [28] Fred Long, Dhruv Mohindra, Robert Seacord, Dean Sutherland, and David Svoboda. 2011. The CERT Oracle Secure Coding Standard for Java. Addison-Wesley. [29] Roberto Minelli, Andrea Mocci, and Michele Lanza. 2015. I know what you did last summer-an investigation of how developers spend their time. In 2015 IEEE 23rd international conference on program comprehension. IEEE, 25–35. [30] Dhruv Mohindra. 2008. MET05-J: Ensure that constructors do not call overridable methods. Retrieved 19 July, 2025 from https://wiki.sei.cmu.edu/confluence/ display/java/MET05-J.+Ensure+that+constructors+do+not+call+overridable+ methods [31] Nicholas Nethercote and Julian Seward. 2007. Valgrind: a framework for heavyweight dynamic binary instrumentation. SIGPLAN Not. 42, 6 (June 2007), 89–100. doi:10.1145/1273442.1250746 [32] Juri Opitz. 2024. A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice. Transactions of the Association for Computational Linguistics 12 (2024), 820–836. doi:10.1162/tacl_a_00675 [33] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019). [34] Sadra Sabouri, Philipp Eibl, Xinyi Zhou, Morteza Ziyadi, Nenad Medvidovic, Lars Lindemann, and Souti Chattopadhyay. 2025. Trust Dynamics in AI-Assisted Development: Definitions, Factors, and Implications . In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 736–736. doi:10.1109/ICSE55347.2025.00199 [35] Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitry Vyukov. 2012. AddressSanitizer: A Fast Address Sanity Checker. In USENIX ATC 2012. https://www.usenix.org/conference/usenixfederatedconferencesweek/ addresssanitizer-fast-address-sanity-checker [36] Jonathan Sillito, Gail C. Murphy, and Kris De Volder. 2008. Asking and answering questions during a programming change task. IEEE Transactions on Software Engineering 34, 4 (2008), 434–451. [37] M.-A. Storey. 2005. Theories, methods and tools in program comprehension: past, present and future. In 13th International Workshop on Program Comprehension (IWPC’05). 181–191. doi:10.1109/WPC.2005.38 [38] GitHub Security Lab Team. 2025. GitHub Security Lab. Retrieved 19 July, 2025 from https://github.com/github/securitylab [39] Semgrep Core Team. 2020. Semgrep. Retrieved 19 July, 2025 from https://semgrep. dev [40] SpotBugs Core Team. 2025. SpotBugs: Find Bugs in Java Programs. Retrieved 19 July, 2025 from https://spotbugs.github.io/
Generating Complex Code Analyzers from Natural Language Questions
[41] Rijnard van Tonder. 2024. Comby - Structural code search and replace for every language. https://comby.dev/ [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762 [43] Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. 2025. KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding. arXiv:2503.02951 [cs.LG] https://arxiv.org/abs/2503.02951
Conference’17, July 2017, Washington, DC, USA
[44] Michael JQ Zhang and Eunsol Choi. 2025. Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs. In Findings of the Association for Computational Linguistics: NAACL 2025, Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 5526–5543. doi:10.18653/v1/2025.findings-naacl.306 [45] Xinyi Zhou, Zeinadsadat Saghi, Sadra Sabouri, Rahul Pandita, Mollie McGuire, and Souti Chattopadhyay. 2026. Cognitive Biases in LLM-Assisted Software Development. arXiv preprint arXiv:2601.08045 (2026).