Less Is More: Measuring How LLM Involvement Affects Chatbot Accuracy in Static Analysis Krishna Narasimhan [email protected] F1Re BV Amsterdam, The Netherlands
arXiv:2604.21746v1 [cs.SE] 23 Apr 2026
Abstract Large language models are increasingly used to make static analysis tools accessible through natural language, yet existing systems differ in how much they delegate to the LLM without treating the degree of delegation as an independent variable. We compare three architectures along a spectrum of LLM involvement for translating natural language to Joern’s query language CPGQL: direct query generation (A1), generation of a schema-constrained JSON intermediate representation (A2), and tool-augmented agentic generation (A3). These are evaluated on a benchmark of 20 code analysis tasks across three complexity tiers, using four open-weight models in a 2×2 design (two model families × two scales), each with three repetitions. The structured intermediate representation (A2) achieves the highest result match rates, outperforming direct generation by 15–25 percentage points on large models and surpassing the agentic approach despite the latter consuming 8× more tokens. The benefit of structured intermediates is most pronounced for large models; for small models, schema compliance becomes the bottleneck. These findings suggest that in formally structured domains, constraining the LLM’s output to a well-typed intermediate representation and delegating query construction to deterministic code yields better results than either unconstrained generation or iterative tool use. CCS Concepts: • Software and its engineering → Formal language definitions; Compilers. Keywords: static analysis, large language models, domainspecific languages, code property graphs, program analysis
1
Introduction
Knowledge about software systems exists in two fundamentally different forms. On one side is the unstructured world: natural language requirements, developer questions posed in issue trackers, security policies described in prose, vulnerability reports written for human readers. On the other side is the structured world: formal query languages, typed schemas, deterministic program representations, machineexecutable specifications. Bridging these two forms is a recurring problem in software engineering, and the difficulty of the crossing depends on how far apart the two sides are. Text-to-SQL translation [10, 27] bridges natural language
questions and relational query languages; natural-languageto-code generation [3] bridges task descriptions and executable programs; requirements formalisation bridges prose specifications and formal models. In each case, the core challenge is the same: an intent expressed informally must be rendered in a language with precise syntax and semantics. Static analysis tools sit firmly on the structured side. Tools like Joern [24] construct Code Property Graphs (CPGs) that unify abstract syntax trees, control flow graphs, and data flow graphs into a single queryable structure, and expose CPGQL, a Scala-flavoured traversal language for querying this representation. CPGQL is powerful—it supports structural pattern matching, inter-procedural data flow tracing, and call graph resolution—but writing queries requires fluency in a niche DSL with method chaining, graph traversal semantics, and knowledge of the CPG schema. Despite welldocumented benefits of static analysis, adoption remains limited by usability rather than capability: developers struggle with warning messages [16], false positives discourage routine use [9], and integrating these tools into workflows requires repeated design iterations [20]. The query interface is a key source of this friction. Large language models have emerged as candidate bridges between the unstructured and structured worlds [4]. An LLM can interpret a natural language request and produce output that approximates a structured artefact—a database query, a code snippet, a configuration file [3]. But LLMs are stochastic approximations, not faithful translators. They pattern-match against distributional regularities in training corpora rather than reason over formal semantics, and they hallucinate plausible-looking outputs that may be syntactically valid but semantically wrong [13]. Their reliability degrades further on formalisms that are underrepresented in training data [8, 25], which includes most domain-specific languages encountered in practice. The question, then, is not whether to use LLMs for this translation, but how much latitude to give them: how tightly to constrain their output space while still leveraging their capacity for language understanding. Several systems explore LLM-based translation to address the static analysis usability gap. IRIS [12] restricts the LLM to labelling APIs as taint sources or sinks, keeping query generation entirely deterministic. QLCoder [22] uses an agentic framework to synthesise full CodeQL queries but reports that giving the agent unrestricted compile-and-run access
Conference’17, July 2017, Washington, DC, USA
degraded performance. MoCQ [11] reduces the DSL surface exposed to the model, finding that the full specification overwhelmed it. An earlier position paper proposed a translationbased architecture for Joern specifically [17]. These systems differ in how much they delegate to the LLM, but none treats the degree of delegation as a variable worth studying. There is empirical reason to be cautious about delegation. Anand et al. [1] analysed attention maps and hidden representations in code-LLMs and found that these models encode relations among syntactic tokens and among identifiers, but fail to encode the relations between them. This cross-category reasoning—connecting a keyword like if to the variable it tests, or linking a function call to the data that flows into it—is precisely what static analysis queries require. The implication is that LLMs lack the internal representations needed to generate reliable analysis queries, even when their output looks syntactically plausible. This paper treats LLM involvement as an independent variable. Three architectures are compared along a spectrum of LLM autonomy:
Krishna Narasimhan
3. Empirical evidence that constraining LLM output to a structured intermediate representation outperforms both direct generation and tool-augmented agentic approaches for DSL generation in a formally structured domain. 4. Evidence of a model-size interaction: the benefit of structured intermediates is strongest for large models, while small models face a different bottleneck in schema compliance.
2
Approach
This section first describes LLM involvement as a design variable and why it matters (Section 2.1), then details the three architectures (Section 2.2). 2.1
LLM Involvement as a Design Variable
Consider a system that takes a natural language request (e.g., “trace how user input reaches the database call in processOrder”) and produces the output of a code property graph A1: Direct Generation. The LLM generates CPGQL queries query—a list of methods, a set of data flow paths, a collection of matching code locations. The system has two parts: an directly, aided by retrieval-augmented context from LLM that interprets the request, and deterministic code that Joern documentation and examples. executes the query. The LLM produces some intermediate A2: Structured Intermediate. The LLM produces a JSON output, and the deterministic code consumes it. object conforming to a fixed schema. A deterministic The design question is: what should that intermediate mapper translates this to the correct CPGQL invocaoutput look like? tion. The LLM never sees query syntax. In A1, the intermediate output is a CPGQL query string. A3: Tool-Augmented Agentic. Analysis capabilities are The LLM must produce syntactically and semantically valid exposed as tools via function calling. The LLM selects Scala-flavoured DSL code, a language with minimal repretools and provides arguments in a multi-step loop, folsentation in training corpora. The output space is effectively lowing the ReAct pattern [26]. unbounded: any string could be a query attempt. These architectures are evaluated on a purpose-built benchIn A2, the intermediate output is a JSON object conforming mark of 20 code analysis tasks across three complexity tiers to a fixed schema with five query types, ten possible output (structural, data flow, composite), using four open-weight columns, and three flow endpoint types. The output space is models in a 2×2 design—two model families (Llama, Qwen) compact and well-typed. A deterministic mapper—verified at two scales (7–8B, 70–72B)—with three repetitions each, by unit tests—translates valid JSON into the correct CPGQL yielding 720 trials. The core finding is that the structured invocation. intermediate representation (A2) achieves the highest reIn A3, the intermediate output is a sequence of tool invosult correctness, outperforming both direct generation and cations. At each step, the LLM selects a tool and provides tool-augmented agentic generation. The benefit is most proarguments. The output space per step is bounded (a finite set nounced for large models (15–25 percentage points (pp) imof tools with typed arguments), but errors compound across provement over A1), while for small models the gain is modsteps. A wrong tool choice at step 𝑖 can derail all subsequent est (3–5 pp) and limited by schema compliance failures. The steps. agentic approach (A3) performs worst despite consuming The pattern is: the smaller and more constrained the inapproximately 8× more tokens per task. termediate output space, the less can go wrong. This holds The contributions of this paper are: as long as the deterministic mapping from constrained out1. A benchmark of 20 natural-language-to-CPGQL transput to domain action is implemented correctly, which is a lation tasks across three complexity tiers, machinereasonable assumption when the domain provides schemas validated against Joern 4.0 and publicly available. and validation. 2. A controlled comparison of three LLM architectures This argument does not apply universally. In exploratory representing different points on the autonomy specdomains where the mapping from intent to action is not trum, evaluated with uniform retry policies for fair known upfront, the LLM’s ability to plan and improvise has comparison. value. The claim here is limited to domains with sufficient
Less Is More: Measuring How LLM Involvement Affects Chatbot Accuracy in Static Analysis
formal structure, which includes static analysis query languages. 2.2
Direct, Structured, and Agentic Generation
Figure 1 shows the three architectures side by side. In each, the user provides a natural language description of a code analysis task, and the system produces results from Joern’s code property graph. All three share a uniform retry policy: up to three attempts on recoverable errors, with error messages fed back to the LLM. This ensures a fair comparison that isolates the effect of the architectural choice from the error-correction budget. Figure 2 summarises the procedures. 2.2.1 Approach 1: Direct Generation. The LLM receives the user request together with a CPGQL syntax reference, Joern-specific conventions (operator naming patterns, annotation matchers, output formatting idioms), and worked examples drawn from Joern documentation. It produces a CPGQL string, which is sent to the Joern server for execution. On execution failure, the error is appended to the conversation history and the LLM is asked to correct its query. This is the most direct approach, and the one with the largest output space. The LLM must handle Scala method chaining, the CPG node type hierarchy, and traversal semantics, none of which it has seen much of during training. To avoid benchmark leakage, all examples in the prompt are drawn from Joern documentation and are distinct from the 20 benchmark tasks. 2.2.2 Approach 2: Structured Intermediate. The LLM produces a JSON object that conforms to a predefined schema. The schema captures the parameters of the analysis task— query type, scope, filters, output columns, flow endpoints— but not the CPGQL syntax. A deterministic mapper covering all benchmark tasks, translates the JSON to the correct query. For example, given “trace how user input reaches the database query in processOrder,” the LLM produces: Listing 1. Example JSON for a data flow query. { " query_type " : " data_flow " , " source " : { " kind " : " parameter " , " method " : " p r o c e s s O r d e r " }, " sink " : { " kind " : " c a l l " , " name " : " e x e c u t e " }, " output_columns " : [ " code " , " lineNumber " ] } The mapper selects the appropriate CPGQL traversal template, fills in the entity references, and sends the query to
Conference’17, July 2017, Washington, DC, USA
Joern. Note that the natural language request mentions “database query” while the JSON specifies execute—the LLM must infer that database queries in Java are typically invoked via JDBC methods such as PreparedStatement.execute(). The LLM’s task thus narrows to selecting among a small set of typed fields and extracting or inferring the relevant entity names. Some domain knowledge is still required, but this inference is confined to filling a well-typed field rather than being entangled with traversal syntax and method chaining. Crucially, the LLM never needs to relate an identifier to the syntactic structure of a CPGQL expression, which is precisely the cross-category reasoning that Anand et al. [1] showed code-LLMs fail to encode reliably. 2.2.3 Approach 3: Tool-Augmented Agentic. Code analysis capabilities are exposed as tools via the HuggingFace function-calling API. From the model’s perspective, this interface is equivalent to MCP [2] and similar tool-use protocols: tool schemas are provided as structured descriptions in the system context, and tool results are injected as conversation turns. The model sees the same sequence of descriptions, invocations, and results regardless of the underlying transport. The implementations differ in how they parse the model’s tool-call output; we discuss this distinction in Section 4.3. Five tools cover common analysis operations: find_methods, find_calls, trace_data_flow, find_reachable_by, and run_custom_query. The LLM operates in a ReAct-style loop [26]: observe request, select tool, observe result, decide next action. Each step requires the LLM to make a correct tool-selection decision with correct arguments. Unlike A1 and A2, which each produce a single CPGQL query whose output is compared to the ground truth, A3 runs multiple CPGQL queries internally through its tool calls and synthesises their results into a final text answer. There is no single generated query to evaluate; the comparison is between this text answer and the Joern output of the ground truth query. Even with high perstep accuracy, errors compound: a task requiring four tool invocations at 90% per-step accuracy yields roughly 66% endto-end success. This is consistent with MCP-Universe [14], where longer task chains show steeper performance drops.
3
Evaluation
The three approaches are evaluated on a benchmark of code analysis tasks executed against open-source Java projects using Joern. All experiments use open-weight models accessed through the HuggingFace Inference API. 3.1
Benchmark
The benchmark consists of 20 tasks organised in three tiers of increasing complexity. Tasks target two real-world Java projects: Apache Commons Lang (a utility library with rich
Conference’17, July 2017, Washington, DC, USA
Krishna Narasimhan
A1: Direct
A2: Structured
A3: Agentic
NL request
NL request
NL request
LLM generates CPGQL string
LLM generates JSON ∈ schema
LLM selects tool + args
Mapper → CPGQL
Tool executes
Joern executes
LLM interprets result
Results
Answer
retry
Joern executes Results
loop
Figure 1. The three architectures. Grey boxes are LLM-mediated; white boxes are deterministic. Algorithm 1 A1: Direct generation. Input: NL task 𝑡 Output: Result or failure msgs ← [sys(ref), usr(𝑡 ) ] for 𝑖 ← 1 to 3 do 𝑞 ← extract(LLM(msgs) ) (ok, 𝑟, 𝑒 ) ← Joern(𝑞) if ok then return 𝑟 msgs += [𝑞, 𝑒 ] return fail
Algorithm 2 A2: Structured intermediate. Input: NL task 𝑡 Output: Result or failure msgs ← [sys(schema), usr(𝑡 ) ] for 𝑖 ← 1 to 3 do 𝑗 ← parse(LLM(msgs) ) if invalid then msgs += [ 𝑗, err ]; continue 𝑞 ← mapper( 𝑗 ) return Joern(𝑞) return fail
Algorithm 3 A3: Agentic loop. Input: NL task 𝑡 Output: Answer or failure msgs ← [sys(tools), usr(𝑡 ) ] for 𝑠 ← 1 to 10 do 𝑟 ← LLM(msgs) if tool call then exec; append else if answer then return answer else return fail return max steps
Figure 2. The three approach procedures, side by side. Grey boxes in Figure 1 correspond to LLM-mediated steps; white boxes to deterministic steps. structural patterns) and OWASP WebGoat (a deliberately vulnerable web application with security-relevant data flows). Nine tasks target Commons Lang; eleven target WebGoat. Before each experiment run, the Joern server imports the target project’s Java source tree and builds a code property graph in memory. All generated queries—whether produced directly by the LLM (A1), by the deterministic mapper (A2), or by the agent’s tool calls (A3)—execute against this live CPG. Result match is determined by running both the generated and the ground truth query against the same graph and comparing their outputs after normalisation (stripping REPL prefixes and collapsing whitespace). The evaluation is end-to-end: from natural language input to analysis output over real code. All 20 ground truth queries were manually authored and verified by execution against Joern 4.0.488; an automated validation script confirms that each query runs without error and returns results on the target project. No benchmark task or its ground truth query appears in any LLM prompt; the examples used in A1 and the schema documentation used in A2 are drawn from Joern’s official documentation. The three tiers are as follows: 1. Structural queries (7 tasks): These map to a single CPGQL traversal over the AST or type hierarchy. Examples include listing public methods in a class, finding assignments to a specific variable, and identifying methods with particular annotations.
2. Data flow queries (7 tasks): These require traversals over the data flow graph using CPGQL’s reachableBy and reachableByFlows operators. Examples include tracing how a method parameter reaches a call site and finding all variables that influence a given condition. 3. Composite queries (6 tasks): These combine structural and data flow reasoning, requiring multiple traversals or joint reasoning about control and data flow. Examples include finding paths from HTTP request parameters to database calls that bypass sanitisation, and identifying methods that transitively call a given sink and accept string parameters. To our knowledge, no existing benchmark addresses naturallanguage-to-CPG query translation. Related benchmarks such as Spider [27] and BIRD [10] target SQL over relational databases; CodeSearchNet [7] targets code retrieval rather than DSL generation; the OWASP Benchmark evaluates static analysis tools rather than query formulation. We make this benchmark publicly available as a contribution.1 3.2
Models
Each approach is evaluated with four open-weight models arranged in a 2×2 design to isolate model-size effects from model-family effects:
1 Repository URL omitted for review.
Less Is More: Measuring How LLM Involvement Affects Chatbot Accuracy in Static Analysis
Large (70–72B) Small (7–8B)
Llama family
Qwen family
Llama 3.3 70B Instruct Llama 3.1 8B Instruct
Qwen 2.5 72B Instruct Qwen 2.5 7B Instruct
All models are accessed through HuggingFace’s Inference API with temperature 0 and fixed random seeds (42, 43, 44) for three repetitions, yielding 60 trials per model×approach combination. For A3, tool schemas are passed via HuggingFace’s tools parameter in chat_completion, which both model families support natively. 3.3
Conference’17, July 2017, Washington, DC, USA
Table 1. Result match rate (%) and execution success rate (%) across approaches and models. 60 trials per cell (20 tasks × 3 reps). Best result match per model in bold. A3 for Llama 70B excluded due to infrastructure failures (see text). A1
Metrics
Token consumption. Total input and output tokens across all LLM calls for each task, including retries and multi-step interactions. LLM invocations per task. For A2 this is typically 1; for A1 it is 1–4 (with retries); for A3 it varies with the agentic loop length (observed range: 2–10). Execution success rate. Whether the approach produces a result without runtime failure. The definition varies by approach. For A1, this means the generated CPGQL query executes on Joern without error. For A2, it additionally requires that the JSON parses, passes schema validation, and maps successfully before Joern execution. For A3, it means the agentic loop terminates with a final answer rather than exceeding the step limit or failing on a tool-call parse error; individual tool calls execute CPGQL internally, but those successes and failures are not surfaced by this metric. Execution success is therefore a higher bar for A2 than for A1, and a lower bar for A3, where it indicates only that the agent produced an answer, not that any particular query ran correctly. 3.4
Family Scale Res. Exec.
Res. Exec. Res. Exec.
Qwen Llama Qwen Llama
58.3 55.0 35.0 35.0
60 Result match (%)
Exact match. Whether the generated CPGQL string is identical to the ground truth query. Applicable to A1 and A2 only; A3 produces results directly rather than CPGQL strings.
A3
72B 70B 7B 8B
43.3 30.0 31.7 30.0
98.3 100 100 100
100 25.0 90.0 100 —a 65.0 15.0 88.3 53.3 15.0 100
a Excluded; see footnote above.
Five quantities are measured for each trial: Result match. Whether the returned results match the ground truth output, tolerating differences in ordering and whitespace formatting. This is the most significant metric because syntactically different CPGQL queries can produce identical results.
A2
A1
58.3
A2
A3
55
43.3
40
31.7
30
35
35 30
25
20
15
0
0
72B Qwen
15
Llam
a 70B
Qwen
7B
a 8B Llam
Figure 3. Result match rate by approach and model. A2 (structured intermediate) leads across all models. A3 for Llama 70B is omitted (infrastructure failures). 3.4.1 Finding 1: Structured intermediates outperform direct generation. Table 1 presents the main results. A2 achieves the highest result match rate for every model tested, with the advantage being most pronounced for the large models: +15.0 pp for Qwen 72B and +25.0 pp for Llama 70B. For the small models, the improvement is modest (+3.3 pp for Qwen 7B, +5.0 pp for Llama 8B). Figure 3 visualises this pattern. The structured intermediate consistently outperforms direct generation, and the gap widens with model scale. A3 underperforms both alternatives on every model. Takeaway: The structured intermediate (A2) outperforms direct generation by 15–25 pp on large models and 3–5 pp on small models. The ranking A2 > A1 > A3 holds across all models.
Results
A total of 660 trials produced usable data: all 480 trials for A1 and A2 across four models, plus 180 A3 trials for three models.2 This section reports findings organised by theme. 2 A3 for Llama 70B is excluded because 36 of 60 trials failed at the infras-
tructure level (HuggingFace API rate limits and tool-call parsing errors), leaving too few completed trials for meaningful comparison. The 24 trials that did complete are broadly consistent with the pattern reported here. See Section 4.3.
3.4.2 Finding 2: Model size interacts with approach. The benefit of the structured intermediate depends on model scale. For large models, A2 provides a substantial improvement because these models are capable enough to fill the JSON schema correctly—both Qwen 72B and Llama 70B achieve 100% execution success on A2, meaning every LLM output was valid JSON conforming to the schema that the mapper could translate.
Conference’17, July 2017, Washington, DC, USA
Krishna Narasimhan 45,000
Execution success Result match
Total tokens per task
Rate (%)
100
50
0
40,000 35,000 30,000 25,000 20,000 15,000 10,000 5,000 0
Q7
1 2-A
Q7
2 2-A
-A1
L70
-A2
L70
-A1
Q7
2
-A Q7
A1 L8-
A2 L8-
Figure 4. Execution success vs. result match for A1 and A2. Small models (Q7, L8) show high execution success on A1 but low result match; on A2 their execution success drops but result match improves slightly.
For small models, the picture is different. Qwen 7B achieves only 65.0% execution success on A2, and Llama 8B only 53.3%. Nearly half of their outputs fail JSON parsing or schema validation before reaching the mapper. This creates a ceiling: even though A2’s architecture is sound, the small models cannot reliably produce the constrained output it requires. By contrast, A1 achieves near-perfect execution success (98.3–100%) across all models, because the threshold for “valid CPGQL” that Joern will attempt to run is lower than the threshold for “valid JSON conforming to a typed schema.” The small models write syntactically plausible CPGQL that executes but returns wrong results; A2 forces them to be precise about intent, and they often fail at that precision. Note that A3’s high execution success rates (88–100%) are not directly comparable to those of A1 and A2: they reflect whether the agent produced a final answer, not whether a generated query ran on Joern. Llama 8B achieves 100% execution success on A3 but only 15.0% result match, confirming that this metric measures a different property for the agentic approach. Figure 4 illustrates this tradeoff. Takeaway: Large models fill the JSON schema reliably (100% execution success) and reap the full architectural benefit. Small models fail schema validation in 35–47% of attempts, creating a ceiling that limits the structured approach’s advantage.
3.4.3 Finding 3: Agentic generation underperforms at higher cost. A3 achieves the lowest result match rates across all three models with clean data: 25.0% for Qwen 72B, 15.0% for Qwen 7B, and 15.0% for Llama 8B. For every model, A3 falls below both A1 and A2. Even a relaxed match criterion that tolerates partial overlap raises the numbers only modestly (35.0%, 15.0%, and 20.0% respectively).
A1
A2
A3
Figure 5. Distribution of total token consumption per task (Qwen 72B). A1 and A2 cluster tightly below 2,000 tokens. A3 spans a 14× range (3,081–42,790) with a median of 6,756— consuming 4× more tokens at the median and 8× more on average, yet achieving the lowest accuracy.
The small models complete the agentic loop more quickly— averaging 3.1 steps (Qwen 7B) and 2.8 steps (Llama 8B) compared to 4.8 for Qwen 72B—but this reflects shallower exploration rather than efficiency. They tend to produce a final answer after fewer tool calls, often without gathering sufficient information. On Qwen 72B, 8 of 60 trials reached the maximum of 10 steps without producing a final answer. The agent explored tool results without converging on a conclusion. The average trial used 4.8 steps and 3.9 tool calls, with an average latency of 32.1 seconds—compared to under 5 seconds for A1 and A2. Takeaway: Agentic generation achieves the lowest accuracy across all models (15–25%) while taking 6× longer per task. Small models exit the loop quickly without gathering sufficient information; large models explore without converging.
3.4.4 Finding 4: Token cost does not predict accuracy. Figure 5 compares the distribution of total token consumption across approaches for Qwen 72B. The box plot makes the variance story vivid: A1 and A2 are barely distinguishable at the bottom of the scale, each clustered in a narrow band below 2,000 tokens, while A3 sprawls across a 14× range from 3,081 to 42,790. At the median, A3 consumes 4× more tokens than A2; at the mean, 8× more. The interquartile range of A3 alone (4,768–21,883) is wider than the entire range of A1 and A2 combined. This variance reflects the unpredictability of the agentic loop: some tasks resolve in two tool calls, others hit the 10-step ceiling without converging. Takeaway: A3 consumes 8× more tokens on average than A2, with a 14× range in per-task cost, yet achieves 33 pp lower accuracy. More compute does not compensate for architectural mismatch.
Less Is More: Measuring How LLM Involvement Affects Chatbot Accuracy in Static Analysis
Conference’17, July 2017, Washington, DC, USA
A3
Tier
#
of
%
#
of
%
#
of
%
Structural Data flow Composite
11 7 8
21 21 18
52.4 33.3 44.4
12 15 8
21 21 18
57.1 71.4 44.4
8 4 3
21 21 18
38.1 19.0 16.7
All
26
60
43.3
35
60
58.3
15
60
25.0
3.4.5 Finding 5: Tier difficulty varies by approach. Table 2 shows the per-tier breakdown for Qwen 72B across all three approaches. Structural queries are the most accessible tier across all approaches. The tier difficulty ordering varies across approaches. For A1, composite queries are more accessible than pure data flow queries, suggesting that the structural component provides grounding. For A2, data flow queries are the most accessible tier (71.4%), likely because the schema explicitly captures flow source and sink endpoints, reducing the task to classification. Composite queries are the hardest tier for A2 (44.4%), which may reflect the difficulty of jointly specifying multiple schema fields. For A3, the pattern is different. The agent handles structural queries reasonably (38.1%), but its performance on data flow and composite queries drops sharply (19.0% and 16.7%). On the smaller models, A3 solves only structural queries: Qwen 7B achieves 42.9% on structural tasks but 0% on both data flow and composite tasks. This suggests that multi-step tool use collapses when the task requires coordinating information across tool calls. Takeaway: A3 solves only structural queries on small models (0% on data flow and composite). The agentic loop fails when tasks require coordinating information across multiple tool calls.
3.4.6 Finding 6: No complementary coverage from agentic generation. A natural hope for tool-augmented generation is that it might solve different tasks than the other approaches—perhaps succeeding where structured intermediates fail, making a combined system worthwhile. The data rule this out. Table 3 shows the per-task breakdown for Qwen 72B across all three approaches. Each cell reports how many of three repetitions produced a correct result. Two patterns are immediately visible. First, the set of tasks solved by A3 (shaded cells in the rightmost column) is a strict subset of those solved by A2—the agentic approach never solves a task that the structured intermediate cannot. Second, six tasks (C02, C05, D02, D07, S03, S05) are never solved by any approach; manual inspection reveals these involve uncommon CPGQL operators or multi-hop reasoning that no model approximates reliably.
Tier
Task
A1
A2
A3
Structural
A2
S01 S02 S03 S04 S05 S06 S07
✓ ✓ — • — ✓ —
✓ ✓ — ✓ — ✓ —
✓ • — — — ✓ —
Data flow
A1
Table 3. Per-task results for Qwen 72B (3 repetitions per cell). Each cell shows the number of repetitions that produced a correct result. Tasks grouped by tier. ✓ = 3/3; • = 1–2/3; — = 0/3.
D01 D02 D03 D04 D05 D06 D07
— — ✓ — • ✓ —
✓ — ✓ ✓ ✓ ✓ —
— — • — — ✓ —
Composite
Table 2. Result match by tier (Qwen 72B, 3 reps). Structural: 21 trials (7 tasks × 3); Data flow: 21; Composite: 18 (6 × 3).
C01 C02 C03 C04 C05 C06
✓ — ✓ — — •
• — ✓ • — ✓
— — ✓ — — —
This strict-subset relationship holds across all three models with clean A3 data. On Qwen 72B, A2 solves 13 of 20 tasks at least once; A3 solves 6 of those 13 and no others. On Qwen 7B, A2 solves 7 tasks; A3 solves 3 of those 7. On Llama 8B, A2 solves 7 tasks; A3 solves 3 of those 7. There is no task for which the agentic approach offers unique coverage. Takeaway: The tasks solved by A3 are a strict subset of those solved by A2, across all models. The agentic approach offers no complementary value that would justify its cost in a combined system.
3.4.7 Finding 7: Exact match understates correctness. The gap between exact match and result match reveals that LLMs frequently write semantically equivalent but syntactically different queries. For Qwen 72B on A1, exact match is 26.7% while result match is 43.3%—a 16.6 pp gap. The LLM traverses the CPG via a different but valid path and arrives at the same result. For A2, this gap is smaller because the deterministic mapper produces exactly one canonical CPGQL string per JSON specification. Any syntactic variation in the output reflects variation in the JSON (e.g., different filter orderings that the mapper normalises), not in the CPGQL itself. This finding has methodological implications: evaluations of DSL generation that rely solely on exact string matching understate actual correctness. Result-based evaluation,
Conference’17, July 2017, Washington, DC, USA
while more expensive (it requires a running execution environment), is more appropriate for measuring practical utility. Takeaway: Exact string matching understates correctness by 16.6 pp on A1. DSL generation should be evaluated by executing queries and comparing results, not by comparing strings.
4
Discussion
4.1
Effectiveness of Structured Intermediates
The structured intermediate (A2) outperforms direct generation (A1) across all four models. The data are unambiguous on this point: 15 pp for Qwen 72B, 25 pp for Llama 70B, and 3–5 pp for the smaller models. What requires interpretation is why. One explanation—and we emphasise that this is our reading of the data, not an experimentally isolated claim—is that A2 transforms the LLM’s task from code generation to something closer to classification. Rather than producing arbitrary CPGQL traversal code requiring fluency in a Scala-flavoured DSL with method chaining, graph traversal semantics, and niche operator names, the LLM selects from a small set of well-typed fields: five query types, ten output columns, three flow endpoint types. The deterministic mapper, verified by 22 unit tests, handles all syntactic and semantic details of CPGQL generation. This framing aligns with the findings of Anand et al. [1], who showed that code-LLMs fail to encode cross-category relations between syntactic tokens and identifiers. A2 sidesteps this limitation: the LLM never needs to relate the identifier processOrder to the syntactic structure of a CPGQL method traversal; it only places processOrder in the correct JSON field. Whether this is the full explanation or merely one contributing factor, we cannot determine from our experimental design alone. What we can establish is that the magnitude of the improvement is comparable to the gains reported by MoCQ [11] from reducing the CodeQL DSL surface, and by IRIS [12] from restricting the LLM to classification rather than generation. All three systems arrive at the same architectural principle from different starting points, which lends credibility to the pattern even if the underlying mechanism remains a hypothesis.
4.2
Error Compounding in Agentic Generation
The tool-augmented agentic approach (A3) performs worst across all models for which we have clean data. On Qwen 72B, A3 achieves 25.0% result match compared to 58.3% for A2— less than half. On the smaller models the gap narrows in absolute terms but not in the ordering: Qwen 7B reaches 15.0% and Llama 8B reaches 15.0%, both below their A1 scores.
Krishna Narasimhan
Three failure patterns are consistently observed. First, the agent selects a tool that returns a large but partially relevant result set, then reasons incorrectly over that set in subsequent steps. Second, the agent enters exploratory loops, calling tools to gather context rather than to answer the question, consuming steps without converging. Third, the agent occasionally produces a correct intermediate result from a tool call but misinterprets or reformats it when composing the final answer. A striking finding is that A3 never solves a task that A2 cannot (Section 3.4.6). Across all three models with clean data, the set of tasks that A3 answers correctly is a strict subset of those that A2 answers correctly. The agentic approach does not even offer complementary coverage—it provides no unique capability that could justify its substantially higher cost. These findings are consistent with MCP-Universe [14], where task success degrades with chain length, and with QLCoder [22], where unrestricted tool access degraded performance. The compounding-error hypothesis—that even a modest per-step error rate accumulates across multi-step interactions—fits the data, though we note that the hypothesis is structural rather than experimentally isolated. With an observed average of 4.8 steps per trial on Qwen 72B, even 90% per-step accuracy would yield roughly 59% end-to-end success, consistent with the observed gap.
4.3
Threats to Validity
The benchmark covers 20 tasks across three complexity tiers. With 60 trials per model×approach cell, the differences between A1 and A2 on large models exceed 15 pp and are consistent across both model families, but we do not claim comprehensive coverage of all possible analysis tasks. The benchmark is author-constructed due to the absence of existing NL-to-CPG benchmarks. To mitigate bias, all ground truths are machine-validated by execution against Joern, and no benchmark content appears in any LLM prompt. Expanding the benchmark is future work; the current version is publicly released for others to extend. The deterministic mapper in A2 is hand-written for the schema defined here. Tasks outside the schema would require schema extension, which is manual work. The mapper’s coverage defines the ceiling of what A2 can express. A3 uses HuggingFace’s function-calling implementation rather than MCP directly. From the model’s input perspective, the two are equivalent: the same tool schemas and result injections appear in the context window. They differ in output parsing: HuggingFace’s parser rejected some of Llama 70B’s tool-call formatting attempts (the model generated <function=...> syntax instead of JSON), contributing to infrastructure failures that led us to exclude Llama 70B from the A3 analysis. An MCP implementation might parse differently, though this would affect only the transport layer,
Less Is More: Measuring How LLM Involvement Affects Chatbot Accuracy in Static Analysis
not the model’s reasoning. Comparing tool-use protocol implementations is orthogonal to the question of how much to involve the LLM and is a direction for future work. Results are specific to the four open-weight models tested. However, the relative ordering (A2 outperforming A1, which outperforms A3) holds across both families at both scales for which we have clean data, which suggests the pattern is not model-specific. All experiments use temperature 0. Higher temperatures may produce different variance characteristics but are unlikely to change the central finding about output-space constraint. The comparison procedure differs slightly for A3. For A1 and A2, both the generated and ground truth queries execute on Joern, and the two outputs are compared directly. For A3, the agent produces a text-rendered final answer (e.g., List("foo", "bar")) rather than a CPGQL string; this text is compared against the Joern output of the ground truth query after the same normalisation. The relaxed match metric (which extracts quoted strings and compares as sets) partially addresses formatting differences, but the comparison is inherently noisier than Joern-to-Joern output comparison. For small models (7–8B parameters), the bottleneck shifts from query correctness to schema compliance. Guided decoding or constrained generation [25] may address this gap by enforcing JSON schema compliance at the token level, effectively removing the schema-compliance burden from the model. This is a direction for future work.
5
Related Work
5.1
LLM-Assisted Static Analysis
Recent systems that combine LLMs with static analysis tools differ primarily in what the LLM produces and how much of the pipeline it controls. At one end of the spectrum, IRIS [12] restricts the LLM to classifying APIs as taint sources or sinks. These labels are fed into CodeQL via templates; the LLM never writes query code. On CWE-Bench-Java, IRIS with GPT-4 detects 55 vulnerabilities compared to CodeQL’s 27, with fewer false positives. The deliberate restriction of the LLM’s role is a design choice that pays off, though the authors do not frame it as such. At the other end, QLCoder [22] asks the LLM to synthesise complete CodeQL queries using an agentic framework with tool access. It achieves 100% compilation and 53.4% success on CWE-Bench-Java. The authors report that giving the agent unrestricted access to compile-and-run degraded performance because the LLM overused it. This is a concrete data point in favour of constraining LLM involvement. QLPro [6] uses a three-role mechanism (Writer, Repair, Execute) to fix syntax errors in generated queries. CQLLM [23] combines RAG with LoRA fine-tuning. Both implicitly acknowledge that LLMs produce unreliable DSL code, then add
Conference’17, July 2017, Washington, DC, USA
corrective layers rather than questioning whether generation is the right task. MoCQ [11] takes a middle position: it extracts a reduced subset of the CodeQL DSL and provides only that subset as context. Providing the full specification overwhelmed the model; the subset was tractable. This is output-space reduction applied at the prompt level. None of these systems compares architectures with different levels of LLM involvement in a controlled setting. Each picks a point on the spectrum and optimises it. While these works target CodeQL specifically, the architectural question generalises. This paper instantiates it for Joern’s CPGQL, a DSL with even less representation in LLM training corpora, which makes the output-space argument sharper. 5.2
What Code-LLMs Fail to Encode
The hypothesis that LLMs are unreliable generators of static analysis queries has empirical grounding beyond anecdotal compilation failures. Anand et al. [1] analysed attention maps and hidden representations of code-LLMs at the token level. They categorised tokens into syntactic tokens (keywords, operators, delimiters) and identifiers (variable names, function names), and found that models encode relations within each category but fail to encode relations across them. Paradoxically, larger models encoded less structural information than smaller ones. Fine-tuned models performed worse than their pre-trained counterparts on these structural probes. These findings matter directly for DSL generation. A typical CPGQL query—e.g., one that retrieves the parameters of a method named foo—requires the model to connect the identifier foo to the syntactic structure of a method traversal. If code-LLMs cannot reliably represent these cross-category links, generating correct DSL queries from natural language is inherently fragile, regardless of model scale. 5.3
Structured Output from LLMs
LLMs struggle with the syntax of domain-specific languages, and this is not specific to any single tool. StructEval [25] evaluates LLMs across 18 structured formats and reports gaps even for frontier models. Work on JSON processing shows that generating code to parse JSON outperforms direct inspection by 3–50%, suggesting LLMs handle structured data better when the task is constrained. The implication is direct: reducing the LLM’s task from “generate CPGQL” to “produce a JSON object conforming to a known schema” moves the problem to a representation where LLMs are more competent. Schema-constrained JSON output is now a standard feature of major LLM APIs and can be enforced through guided decoding on open-weight models, further reducing failure modes. The analogy to Text-to-SQL [19] is useful. That community has studied how schema information and intermediate representations affect query generation. The intermediate
Conference’17, July 2017, Washington, DC, USA
JSON approach presented here is the static-analysis equivalent of schema-linking: generate a constrained intermediate, let deterministic code produce the query. 5.4
Tool-Augmented LLM Generation
The ReAct framework [26] established the pattern that A3 implements: the LLM alternates between reasoning about a task and invoking external tools, observing results at each step. Toolformer [21] and Gorilla [18] explore the same pattern from different angles. In all cases, what matters from the model’s perspective is the tool schema it sees and the result it gets back, not the transport mechanism—whether tools are invoked via MCP, OpenAI function calling, or HuggingFace’s tools parameter, the model’s context window contains the same sequence of tool descriptions, invocations, and results. Recent work on uncertainty in tool-augmented LLM systems provides theoretical context for the hypothesis. Tools in the Loop [15] proposes a framework for jointly modelling the predictive uncertainty of the LLM and the external tools, showing that both contribute to overall reliability. MCPUniverse [14] reports that GPT-5 achieves only 43.72% overall success in realistic MCP environments, with failures concentrated in content generation rather than format compliance. CA-MCP [5] proposes reducing LLM involvement in MCP workflows by letting servers coordinate without continuous LLM orchestration. These findings support the general principle: less LLM mediation, better outcomes in structured domains. 5.5
Code Property Graphs and Joern
Code Property Graphs were introduced by Yamaguchi et al. [24] as a unified representation combining ASTs, CFGs, and DFGs. Joern implements this representation and provides CPGQL, a Scala-based query language for traversing the graph. The CPG representation has since been adopted by other tools and standardised through efforts such as the Open Source Security Foundation’s CPG specification. Joern supports multiple languages (Java, C/C++, Python, JavaScript) and is actively maintained. CPGQL queries use method chaining over graph traversals. A typical query reads the CPG, filters nodes by type or name, traverses edges, and collects results. The language is expressive but niche: it is underrepresented in LLM training data compared to mainstream languages, which makes it a suitable test case for studying LLM limitations in DSL generation.
6
Conclusion
This paper investigates how much of a static analysis pipeline should be delegated to an LLM. Three architectures are instantiated along a spectrum of LLM involvement—direct
Krishna Narasimhan
CPGQL generation, structured JSON intermediates, and toolaugmented agentic generation—and evaluated on 20 code analysis tasks using four open-weight models. The structured intermediate representation achieves the highest result correctness across all models, with improvements of 15–25 pp over direct generation on large models. The agentic approach performs worst despite consuming 8× more tokens: 25.0% result match on Qwen 72B compared to 58.3% for the structured approach, and 15.0% on both smaller models. The set of tasks solved by the agentic approach is a strict subset of those solved by the structured approach, offering no complementary coverage. Model size interacts with approach: the benefit of structured intermediates is strongest for large models that can reliably fill the schema, while small models face a bottleneck in schema compliance that limits the architectural advantage. The principle is not specific to CPGQL. Any domain with a formal schema, typed query language, and deterministic execution can benefit from the same decomposition: let the LLM handle what it handles well—natural language understanding and classification into a schema—and let deterministic code handle the rest. The key precondition is that the space of valid queries can be captured by a schema compact enough for the LLM to fill reliably, which is not guaranteed at all model scales. The benchmark and all experimental infrastructure are publicly available to support replication and extension.3
Acknowledgements The artifact and the data to validate the work’s claims are listed are made available and the authors request to be included for the Artifact evaluation badge. Artifact. The experimental infrastructure, benchmark, raw results, and analysis scripts accompanying this paper are archived at https://zenodo.org/records/18888136. The artifact includes all 660 trial outputs, the deterministic mapper with its test suite, and scripts to reproduce the paper’s tables and figures from the included data. We apply for the Artifacts Evaluated – Functional badge.
References [1] Abhinav Anand, Shweta Verma, Krishna Narasimhan, and Mira Mezini. 2024. A Critical Study of What Code-LLMs (Do Not) Learn. In Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, Bangkok, Thailand, 15869–15889. doi:10.18653/v1/2024.findings-acl.939 [2] Anthropic. 2024. Model Context Protocol Specification. (2024). https: //modelcontextprotocol.io [3] Mark Chen, Jerry Tworek, Heewoo Jun, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). [4] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large 3 Repository URL omitted for review.
Less Is More: Measuring How LLM Involvement Affects Chatbot Accuracy in Static Analysis Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024). doi:10.1145/3695988 [5] Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2025. Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions. arXiv preprint arXiv:2503.23278 (2025). [6] Junze Hu, Xiangyu Jin, Yizhe Zeng, Yuling Liu, Yunpeng Li, Dan Du, Kaiyu Xie, and Hongsong Zhu. 2025. QLPro: Automated Code Vulnerability Discovery via LLM and Static Code Analysis Integration. (2025). arXiv:2506.23644 [cs.SE] https://arxiv.org/abs/2506.23644 [7] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. CoRR abs/1909.09436 (2019). arXiv:1909.09436 http://arxiv.org/abs/1909.09436 [8] Sathvik Joel, Jie Wu, and Fatemeh Fard. 2025. A Survey on LLMbased Code Generation for Low-Resource and Domain-Specific Programming Languages. ACM Trans. Softw. Eng. Methodol. (Oct. 2025). doi:10.1145/3770084 Just Accepted. [9] Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why Don’t Software Developers Use Static Analysis Tools to Find Bugs?. In Proc. ICSE. 672–681. doi:10.1109/ICSE.2013. 6606613 [10] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, {Kevin C.C.} Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. Advances in Neural Information Processing Systems 36. [11] Penghui Li, Songchen Yao, Josef Sarfati Korich, Changhua Luo, Jianjia Yu, Yinzhi Cao, and Junfeng Yang. 2025. Automated Static Vulnerability Detection via a Holistic Neuro-symbolic Approach. CoRR abs/2504.16057 (2025). arXiv:2504.16057 doi:10.48550/ARXIV.2504. 16057 IRIS: LLM[12] Ziyang Li, Saikat Dutta, and Mayur Naik. 2025. Assisted Static Analysis for Detecting Security Vulnerabilities. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025. 35735– 35758. https://proceedings.iclr.cc/paper_files/paper/2025/file/ 582d4e27fa24168f3af1f4582655034b-Paper-Conference.pdf [13] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 21558– 21572. https://proceedings.neurips.cc/paper_files/paper/2023/file/ 43e9d647ccd3e4b7b5baab53f0368686-Paper-Conference.pdf [14] Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. 2025. MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers. (2025). https://openreview.net/forum?id=juQnezS1vw [15] Panagiotis Lymperopoulos and Vasanth Sarathy. 2025. Tools in the Loop: Quantifying Uncertainty of LLM Question Answering Systems That Use Tools. (2025), 2645–2647. [16] Marcus Nachtigall, Michael Schlichtig, and Eric Bodden. 2022. A large-scale study of usability criteria addressed by static analysis tools. (2022), 532–543. doi:10.1145/3533767.3534374 [17] Krishna Narasimhan. 2024. Bridging Natural Language and Static Analysis. In Proc. BENEVOL. doi:publications/narasimhan2025 [18] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Gorilla: Large Language Model Connected with Massive APIs. 37 (2024), 126544–126565. doi:10.52202/079017-4020
Conference’17, July 2017, Washington, DC, USA [19] Mohammadreza Pourreza and Davood Rafiei. 2023. DINSQL: Decomposed In-Context Learning of Text-to-SQL with SelfCorrection. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 36339– 36348. https://proceedings.neurips.cc/paper_files/paper/2023/file/ 72223cc66f63ca1aa59edaec1b3670e6-Paper-Conference.pdf [20] Caitlin Sadowski, Edward Aftandilian, Alex Eagle, Liam Miller-Cushon, and Ciera Jaspan. 2018. Lessons from Building Static Analysis Tools at Google. Commun. ACM 61, 4 (2018). doi:2020/papers/google-analysiscacm.pdf [21] Timo Schick, Jane Dwivedi-Yu, Roberto Dessí, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: language models can teach themselves to use tools. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 2997, 13 pages. [22] Claire Wang, Ziyang Li, Saikat Dutta, and Mayur Naik. 2025. QLCoder: A Query Synthesizer For Static Analysis of Security Vulnerabilities. arXiv:2511.08462 [cs.CR] https://arxiv.org/abs/2511.08462 [23] Le Wang, Chan Chen, Junyi Zhu, Rufeng Zhan, and Weihong Han. 2026. CQLLM: A Framework for Generating CodeQL Security Vulnerability Detection Code Based on Large Language Model. Applied Sciences 16, 1 (2026). doi:10.3390/app16010517 [24] Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. 2014. Modeling and Discovering Vulnerabilities with Code Property Graphs. In Proceedings of the 2014 IEEE Symposium on Security and Privacy (SP ’14). IEEE Computer Society, USA, 590–604. doi:10.1109/SP.2014.44 [25] Jialin Yang, Dongfu Jiang, Lipeng He, Sherman Siu, Yuxuan Zhang, Disen Liao, Zhuofeng Li, Huaye Zeng, Yiming Jia, Haozhe Wang, Benjamin Schneider, Chi Ruan, Wentao Ma, Zhiheng Lyu, Yifei Wang, Yi Lu, Quy Duc Do, Ziyan Jiang, Ping Nie, and Wenhu Chen. 2026. StructEval: Benchmarking LLMs’ Capabilities to Generate Structural Outputs. (2026). arXiv:2505.20139 [cs.SE] https://arxiv.org/abs/2505. 20139 [26] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS. Publisher Copyright: © 2023 11th International Conference on Learning Representations, ICLR 2023. All rights reserved.; 11th International Conference on Learning Representations, ICLR 2023 ; Conference date: 01-05-2023 Through 05-05-2023. [27] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-toSQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, Brussels, Belgium, 3911–3921. doi:10.18653/v1/D18-1425