arXiv:2606.04641v1 [cs.DB] 3 Jun 2026
Bridge the Last-Mile Gap to Semantic Analytics: Compiling Natural-Language Queries into Semantic Operator Pipelines Wenkai Dong
Ruyu Li
University of Hawaii at Manoa [email protected]
University of Hawaii at Manoa [email protected]
Sairam Gurajada∗
Yifan Wang
LinkedIn [email protected]
University of Hawaii at Manoa [email protected]
ABSTRACT Automated AI workflows increasingly rely on natural-language reasoning over heterogeneous data, but they still lack a practical way to execute such reasoning through optimized semantic data systems. Recent semantic operator systems, such as Palimpzest and LOTUS, expose declarative operators for filtering, joining, mapping, and aggregating over tables, text, images, and other data sources using natural-language predicates. However, these systems require users or workflow developers to manually choose operators, order them, write predicates, and adapt the resulting pipeline to backendspecific APIs. This manual process is difficult for non-experts, brittle across backends, and infeasible for automated workflows where queries and data contexts vary at runtime. We present NL2Pipe, a middleware system that compiles naturallanguage questions into executable semantic operator pipelines. NL2Pipe treats this task as a three-phase compilation problem. First, a Query-Data Linker grounds question entities against the actual data and discovers implicit bridge entities needed to connect tables, text, images, and other sources. Second, a Semantic Planner produces a backend-agnostic action plan consisting of semantic operators and natural-language predicates. Third, a Code Generator translates the plan into executable code for a target backend using an auto-generated reference document that captures operator signatures, example pipelines, and backend constraints. This design separates data-aware reasoning from backend-specific code generation, enabling the same planning logic to support multiple semantic operator systems. The evaluation results show that NL2Pipe substantially improves pipeline quality on complex cross-source workloads. Comparing to baselines, NL2Pipe achieves higher pipeline quality (e.g., up to 60% higher F1 score) while maintaining bounded cost and competitive latency for building the pipelines. These results demonstrate that automatic compilation from natural language to semantic operator pipelines is both practical and effective for bringing semantic analytics to non-expert users and automated AI workflows. PVLDB Reference Format: Wenkai Dong, Ruyu Li, Sairam Gurajada, Yifan Wang. Bridge the Last-Mile Gap to Semantic Analytics: Compiling Natural-Language Queries into Semantic Operator Pipelines. PVLDB, XX(X): XXX-XXX, 2026. doi:XX.XX/XXX.XX ∗ Work done outside LinkedIn.
This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by
PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/dongw-netize/NL2Pipe-Compiling-Natural-LanguageQuestions-into-Semantic-Operator-Pipelines.
1
INTRODUCTION
Semantic operator systems [5, 12, 14, 16, 23, 31] have emerged in recent years which integrate semantic processing functionality into data management systems and frameworks. With a series of builtin semantic operators implemented on top of Large Language Model (LLM), like semantic filter (sem_filter), semantic join (sem_join), semantic map (sem_map), semantic aggregation (sem_agg) and so on, these systems fuse the semantic processing capability into the original data systems like Python Pandas Dataframe or relational databases. These operators accept natural language predicates and conduct corresponding operations based on the semantic predicates. Additionally, they are often backed by built-in optimizers to select best models, prompts, and execution plans automatically. Users are able to programmatically call these operators to process natural language (NL) queries without external tools. For example, a semantic filter can be called with predicate "the course requires a lot of math" to filter all courses that need solid math background. Such semantic operator systems provide a new paradigm for semantic analytics: neither text-to-SQL + traditional database, nor relying on tools outside the databases, but an advanced in-database solution to process semantics natively. For complex multi-hop queries, a pipeline consisting of multiple operators has to be built. For instance, as shown in Figure 1, consider answering “How often does the folk festival in the South Moravian city with fewer people than Boskovice but more than others take place?” given a table of South Moravian towns with their populations (the table source) and a collection of Wikipedia passages describing each town (the text source). Answering this requires a precise sequence of semantic operators: first, a semantic filter over the table to retain rows whose population is strictly less than Boskovice’s (11,622); then, a cross-source semantic join that links each surviving row to the passage describing that town and locates the single row (target town) whose population is the largest among these candidates; finally, a semantic aggregation or semantic map that extracts the emailing [email protected]. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. XX, No. X ISSN 2150-8097. doi:XX.XX/XXX.XX
festival’s frequency from the matched passage for the target town. Each operator is parameterized by a natural-language predicate derived from the corresponding task, like sem_agg uses predicate "how often the festival takes place". The challenge is that no existing system can automatically and accurately compile the question into this operator sequence. The user must manually determine which operators to invoke, in what order, with what predicates, and over which data sources. These decisions require understanding not only the question and data semantics, but also the data structure behind the operators and datasets. For instance, the descriptive phrase “fewer people than Boskovice but more than others” must first be resolved against the table through a non-trivial ranking inference, which takes the row with the maximum population among those strictly smaller than Boskovice, to identify the bridge entity (Kyjov, population 11,218) that is an anchor to link cross-source data (which are table and text in this example). Specifically, bridge entity is an entity that never appears in the original question, but has to be found as the join key to link different data sources. In this example, the name "Kyjov" is not mentioned in the question, but it must serve as the join key linking the filtered table row to the text passage that records the festival’s quadrennial cadence. Getting any of these steps wrong leads to missed cross-source connections, redundant processing, or incorrect answers. For complex queries like multi-hop queries over multi-modal data, the pipelines are usually complicated and making those decisions are time-consuming and hard to be optimal if relying on human. A bigger challenge is raised since the required operator pipeline is query-dependent, i.e., the pipelines to solve each query are possibly different. Some queries only need filtering on table, while some needs cross-source semantic join between different modalities and semantic aggregation on multiple records, etc. Even if two queries have the same form like "Which 𝑋 has the highest 𝑌 ", their pipelines could be different due to other conditions like data sources involved. Such a query-and-pipeline dynamics makes manually writing pipelines for each query not scalable. Additionally, even the decisions for a pipeline have all been made, generating an executable pipeline is still non-trivial: with different semantic operator systems (called backend systems in this paper), the APIs and pipeline organizations are different, which requires fundamentally different code, with distinct execution models, column reference syntax, and result extraction patterns. This introduce additional complexity to the pipeline building. Furthermore, if people want to use these semantic operator systems to facilitate any automated workflow where queries arrive at runtime (like multi-agent systems which heavily rely on semantic processing for NL queries), manually building the operator pipeline for each different query in the workflow is impossible. To address the aforementioned challenges, save human effort in building semantic operator pipelines, and better utilize the operator systems in automated workflows, we propose NL2Pipe, a middleware that closes the gaps by automatically and accurately compiling natural-language queries into executable semantic operator pipelines. The key insight is to treat the mapping from query to operators as a compilation problem with three tasks: understanding the data and query intent, deciding what operations to perform, and generating engine-specific code. Accordingly, NL2Pipe operates
in three phases, illustrated in Figure 1. In Phase A (Query-Data Linker), the system extracts key entities from the query, locates them in corresponding data sources , and discovers bridge entities across different data sources, i.e., values absent from the query but needed to connect data sources (e.g., the city name “Kyjov” in Figure 1, which links the population table to the text passages describing each town’s festival). Phase B (Semantic Planner) produces a backend-agnostic plan: an ordered sequence of query execution steps (e.g., filter, join, and extract) with natural-language instructions, informed by the Linker’s entity groundings and verified bridges. Phase C (Code Generator) translates this plan into executable code for a target backend, guided by an auto-generated reference document that includes API signatures, golden examples of operator pipelines, and backend-specific constraints. The generated code is then executed by the target engine and the result is returned to the user. To our knowledge, NL2Pipe is one of the first middleware systems that bridges the emerging semantic operator systems with non-expert users and automated workflows. We evaluate NL2Pipe on several datasets of multi-modal and multihop question-answering and data analytics tasks, and show that the pipelines compiled by NL2Pipe produce significant gains over general-purposed, fixed-structured, and advanced code-generation baselines on cross-source workloads. NL2Pipe reaches up to 63% higher result accuracy than naive LLM-generated pipelines in the evaluation. NL2Pipe is not a Text-to-SQL system. The two problems differ in their input assumptions, target programs, execution engines, and reasoning scope. Text-to-SQL assumes that the data is stored in a relational database and translates a natural-language question into a standard SQL query. Its main challenge is schema and value grounding: mapping phrases in the question to tables, columns, and cells, then expressing the computation using relational operators. In contrast, NL2Pipe assumes a heterogeneous data context that may include tables, texts, images, or other modalities, and compiles the question into a pipeline of semantic operators executed by systems like Palimpzest. The generated program is not SQL but an ordered composition of semantic operators whose predicates are naturallanguage instructions interpreted by LLM. As a result, NL2Pipe can express operations that SQL cannot naturally capture, such as joining a table row to a passage or image through semantic content and extracting answers from unstructured evidence. Thus, Text-toSQL solves relational query translation, whereas NL2Pipe solves semantic pipeline compilation for multi-modal and automated AI workflow settings. Our contributions are as follows: • We identify the gap that non-expert users / automated workflows lack effective automated solutions to construct semantic operator pipelines, and propose one of the first middleware to fix the gap by accurately building the pipelines without domain expertise or human intervention. • We propose a novel three-phase, reference-documentguided workflow, consisting of data-aware query analysis, backend-agnostic planning, and backend-specific code generation, with optimizations like bridge entity identification and reference document quality enhancement. 2
NL2Pipe: Compiling Natural-Language Questions into Semantic Operator Pipelines NL question
"How often does the folk festival in South Moravian city with fewer people than Boskovice but more than others take place?"
2 LLM
FRONT-END
ⒶQuery-Data Linker
what operations, in what order? → backend-agnostic Intermediate Representation
ActionPlan (logic plan) select cities whose population is below Boskovice's 11,622
what does this question mean against this specific data? "Boskovice" ~ table_cell
1 LLM
MIDDLE-END
ⒷPlanner
join:matched town ↔ text passages describing folk festival
(pop=11,622)
extract:how often the festival takes place
"South Moravian" ~ table_subject (implicit)
💡 bridge discovery (key novelty) The city isn't named in the question. Among those smaller than Boskovice (11,622), the largest is Kyjov (11,218).
“text passage #1 about Kyjov”
⛳predicate each step carries a natural-language — no backend API yet
Answer： "Every four years"
Ⓒ Code Generator
1 LLM
emit code for a chosen target engine
target backends: PZ
filtered = table_ds.sem_filter( “text passage #2 about Kyjov” "select cities whose population is below Boskovice's 11,622") “text passage #3 about Kyjov”
joined = filtered.sem_join(text_ds, "town name appears in passage") joined.sem_agg("how often the festival takes place", output="answer")
LOTUS Nirvana
images tables
③
Critic Agent
others..
offline (bootstrap) raw data source
✓ winner
reference.md
N candidates reference.md v1
①
dataset.json Dataset Profiling Agent
texts
Official Doc
②
writes N candidate reference.md drafts
reference.md v2
Reference-Doc Agent
... reference.md vN
Figure 1: End-to-end architecture of NL2Pipe, illustrated on the running example “How often does the folk festival in the South Moravian city with fewer people than Boskovice but more than others take place?”. The front-end Query Linker (Phase A, two LLM calls) grounds question entities against the data (“Boskovice” maps to a table cell, population 11,622) and discovers bridge entities not named in the question— ranking by population just below Boskovice yields “Kyjov”, which is verified against text passages mentioning the folk festival. The grounded entities and matched passages found by the bridges are then handed to the Semantic Planner (Phase B, one LLM call), which emits a backend-agnostic ActionPlan—a sequence of logical operations with natural-language predicates. The Code Generator (Phase C, one LLM call) compiles the plan into executable code for a chosen target engine (Palimpzest, LOTUS, Nirvana, etc). Executing the code returns the final answer (“every four years” ). The offline bootstrap (bottom) is a one-time setup per (backend, dataset) pair: a Dataset Profiling Agent (A) derives the dataset summary from raw files, a Reference-Doc Agent (B) generates 𝑁 candidate reference documents, and a Critic Agent (C) selects the winning document used as Phase C’s system prompt. • We conduct extensive evaluation, showing that our method significantly outperforms general code generation approaches including Codex in terms of pipeline quality and monetary cost while remaining portable across different backends and competitive efficiency.
invocations and LLM calls. These range from fixed, developerauthored pipelines to fully autonomous agents that plan their own steps at runtime. Agentic systems are a prominent instance of this paradigm: representative architectures include ReAct [26], which interleaves reasoning traces with actions; HuggingGPT [18], which routes sub-tasks to specialist models; and MetaGPT [7], which assigns distinct roles to collaborating agents. More broadly, such workflows have been applied to code generation [3], text-to-SQL [15], and scientific research [2]. A shared limitation across these systems is that each reasoning step executes as an independent LLM call with no cross-step optimization. Two strategies dominate: single-pass
2 RELATED WORK 2.1 LLM-Based Automated Workflows LLM-powered automated workflows increasingly handle complex tasks by decomposing them into sub-steps executed through tool 3
code generation, which lacks the ability to adapt operator choices per query, and rigid tool-calling sequences, which cannot handle the diversity of data modalities and reasoning patterns encountered in practice. AgenticData [19] builds a semantic data analytic system from scratch, implementing everything from storage to planner. Particularly, they implement their own semantic operators rather than using any existing systems. Unlike it, NL2Pipe acts as a middleware that can adopt to any existing semantic operator systems, including the semantic operators of AgenticData.
2.2
entities needed for cross-source reasoning, and targets multiple backends through a single planning framework.
3 METHOD 3.1 Problem Formulation Given a natural-language question 𝑞 and a heterogeneous data context which may include a relational table, a collection of text passages, a collection of images, any other data modalities, or any combination of these, the task is building a semantic operator pipeline which will return an answer to 𝑞 after being executed, in other words, compiling the query 𝑞 into such a pipeline. We formulate a semantic operator pipeline as such: an ordered composition of declarative semantic operators whose predicates are natural-language strings and whose semantics are realized by LLM calls at execution time. Recent semantic operator systems [12, 14, 31] expose a variety of such operators, among which the most widely used are a filter 𝜎𝜙 that retains items satisfying natural-language predicate 𝜙, a join ⊲⊳𝜙 that pairs items from two collections under predicate 𝜙, and an aggregation 𝛾𝜙 that reduces a collection of data to a single result under 𝜙. Additionally, per-row semantic mapping, ranking, and other specialized operators are also commonly available. Based on such operators, given the question 𝑞, data context D, and a target backend 𝐵, an operator pipeline 𝑃𝐵 is a finite ordered sequence of the operators that is specifically executable on 𝐵. Executing the pipeline on backend 𝐵 within the data context D yields the answer to question 𝑞. Particularly, the question and data context are backend-agnostic while the produced pipeline is backendspecific. To make this mapping (𝑞, D, 𝐵 → 𝑃𝐵 ) more accurately, we split it into two stages: (1) Intermediate plan generation (𝑞, D → 𝐼 ) that generates a backend-agnostic step-by-step plan 𝐼 about how to answer 𝑞 based on D, as shown in Figure 1 Query-Data Linker and Planner phases, and (2) Plan-to-executable-pipeline mapping (𝐼, 𝐵 → 𝑃𝐵 ), where the backend specification (like operator APIs) is introduced to translate the intermediate plan into executable backend-specific pipeline program, shown as the Code Generator phase in Figure 1.
Multi-Modal Question Answering
A large body of work answers multi-hop, multi-modal questions directly. For multi-hop text QA, the dominant method is retrievethen-read: iteratively retrieve supporting passages and feed them to a reader that chains the evidence into an answer [8, 21, 24]. For questions over tables and text, methods either linearize the table into the reader’s input or train a hybrid reasoner that operates over cells and passages jointly, sometimes with a numerical-reasoning module for aggregation and arithmetic [4, 9, 29, 30]. For questions that further span images, a modality-selection or decomposition step routes each sub-question to a text, table, or vision module and recombines the partial answers [6, 20, 27]. These methods are effective on semantic query processing in their target tasks, but they are end-to-end solutions tuned to one benchmark’s structure, and they usually require significant engineering effort to be seamlessly integrated into existing data systems for semantic data operation, which limits their utilization in end-to-end data analytics. This gap motivates expressing such queries as native semantic operators in a data system, which we review next: doing so turns an ad-hoc QA model into optimizable, reusable in-database operations.
2.3
Semantic Operator Systems
Unlike standalone QA solutions, semantic operator systems provide declarative operators implemented on top of LLM, like LLMpowered semantic filter, map, join, and aggregate, which can understand natural language semantics and operate/analyze data based on natural language predicates. The operators may additionally pair with optimizers that search over models, prompts, and execution orders. Palimpzest [12] pioneered this paradigm, exploring cost– quality tradeoffs across execution plans. Subsequent systems have extended it along different axes: Abacus [16] adds cost-based optimization using validation examples; LOTUS [14] provides formal accuracy guarantees through model cascades; Nirvana [31] supports multi-modal analytics with per-operator backend selection. Related paradigms include DocETL [17] for document processing, CAESURA [22] for multi-modal SQL, SUQL [13] and TAG [1] for hybrid text-SQL queries, and SemBench [10] for benchmarking. Despite their optimization capabilities, all these systems require users to manually compose operator pipelines, i.e., selecting operators, ordering them, and writing natural-language predicates for each. Unify [23] takes a step toward automation by generating query plans directly from natural language, but it targets a single backend and does not handle multi-modal data or cross-source entity linking. NL2Pipe addresses the remaining gap: it automates the full compilation from question to executable pipeline, discovers bridge
3.2
Three-Phase Compilation
NL2Pipe factors the compilation process into three phases, each includes a small number of LLM calls and pivoted by an intermediate representation. Phase A discovers bridge entities that link crosssource data for multi-modal scenarios; Based on the linked data items and their modalities, Phase B emits a backend-agnostic action plan with abstract data operations and corresponding predicates; Phase C translates the plan into executable code for a target backend. Phases A and B are fully backend-agnostic, where the working logic is not impacted by the execution backend, and only Phase C is backend-specific. Comparing to general code generation that produces final code in only one step, this multi-phase approach decouples query processing logic and backend-specific specification, guaranteeing high compatibility (easy switch) between different backends. Figure 1 illustrates the pipeline on our running example: “How often does the folk festival in the South Moravian city with fewer people than Boskovice but more than others take place?”, paired with 4
a table of South Moravian cities and their populations and a collection of Wikipedia passages describing each town. Answering this question requires cross-source reasoning because any single source (table or passages) does not include full information required by the complete reasoning path: the table includes city name and population but not the festival information while the text includes festival information but no population. Therefore, finding a pivot entity to link different sources is critical, for which the central difficulty is that such entities are often implicit (hidden) in the question. Like in our running example, the target city “Kyjov” is never named in the question, instead, it must be resolved against the data. And only when we identify it, we can join the information between different data sources (corresponding passages and the table in the example) to enable a complete cross-source reasoning path. Such values that must be resolved against the data to allow proper data source joining is called bridge entities: they are often absent in the query but necessary to bind operators (like sem_join) across data sources. Bridge entities guarantee accurate collection of complete key information.
3.3
identified for being joined with the table later. By such bridge entities, different sources are linked to provide complete information. Bridge entities usually start from the tables to other sources like text or image collection. When no row is directly pinned down, the grounder will identify a column that could serve as the natural join key with another source, yielding every entry of that column as a bridge entity from the table to the other source. After identifying all bridge entities, an additional LLM call could be conducted to verify whether they truly appear in the other source, to reduce the risk of filtering based on non-existing targets. We preprocess all data items offline to generate short previews for each of them, and the verification will let LLM check the existence of the entities against those previews. Non-existing entities will be discarded before sending to Phase B.
3.4
Phase B: Semantic Planning
Phase B compiles the question 𝑞, dataset schema and the Phase A evidence into a backend-agnostic plan. The evidence consists of three forms of information: matched phrases with their locations in corresponding data sources, topical phrases which describe sources rather than appear in them, and verified bridges with their cardinality and supporting document identifiers. Phase B Planner will consume them to construct a backend-agnostic step-by-step plan to answer the query, which we call the action plan. An action plan is a finite sequence of steps, where each step is a triple (𝑜𝑖 , src𝑖 , 𝜙𝑖 ): 𝑜𝑖 ∈ {𝜎, ⊲⊳, 𝛾 } is the operator type, src𝑖 identifies the source(s) the operator applies to, and 𝜙𝑖 is a natural-language predicate attached to the operator. Our current implementation supports sem_filter (𝜎), sem_join (⊲⊳) and sem_agg (𝛾) which are three most commonly used semantic operators that can handle most queries. An action plan does not reference any backend API, instead, representation of each step is a loosely structured natural language description for the operation and predicate, as shown in Figure 1 Phase B. Generating this action plan (i.e., the planning) uses a single LLM call over four inputs: the question, dataset schema, the Phase A output, and a short list of backend-agnostic planning hints that encode failure modes empirically observed across all backends (for example, text filters should use broad entity-based predicates joined by disjunction rather than narrow conjunctive conditions, which downstream operators handle poorly). Phase A’s three forms of evidence are wired into the plan in distinct ways: matched phrases become precise column-referencing filter predicates, topical phrases are excluded from all filter conditions to avoid eliminating all rows, and verified bridges become either filter anchors or join keys depending on their cardinality. For the running example, Phase B emits a three-step plan: filter the table to retain cities whose population is below Boskovice’s 11,622, join the matched town with the text passages describing the folk festival, and aggregate over the joined records to extract how often the festival takes place. The separation of planning from code generation is deliberate: under a single combined prompt, the model frequently drops join steps or produces type errors when forced to juggle abstract plan logic and concrete syntax simultaneously. Treating the action
Phase A: Query-Data Linker
Phase A prepares the evidence that Phase B needs to select proper operators and write precise operator predicates. It performs up to two LLM calls: one used by entity grounder that jointly grounds question entities and identifies bridge entities, and another optional call used by bridge verifier to double check the bridge entities truly exist in the linked data sources. Entity grounding and bridge discovery. The entity grounder first identifies question entities in the question that explicitly exist in the question and may be part of the operator predicates, e.g., noun phrases and numeric constraints. Each candidate entity/phrase will be assigned one of three labels. (1) A phrase is matched if it appears as a value in some data source (e.g., a cell in the table, or a span in some text passages or image captions). Matched phrases will form column-aware filter predicates in Phase B. (2) A phrase is topical if it describes the subject of a source rather than appearing as content within it. Topical phrases are prohibited in filter predicates, since filtering on them would eliminate all data items as they never exist in the data content. In the running example, “South Moravian” is topical: it matches the title of the table instead of any cell in the table. Therefore, although such phrases are certainly related to the data sources, they should not be used as keywords/constants in predicates. (3) A phrase is absent if it is neither matched nor topical and must therefore be resolved through a different source. Particularly, if the data context includes tables, the matched/topical/absent will be labeled only based on the tables without checking text, image or other sources, to avoid LLM overthinking. In the running example, “folk festival” is absent because there is a table and the table contains no festival information. This phrase has to be resolved against the text passages other than table. The grounder simultaneously identifies bridge entities by inspecting the data sources for values that could plausibly connect two sources. Figure 2 zooms in this mechanism for the running example: the ranking constraint “fewer people than Boskovice but more than others” grounds row 4, then its cell City=Kyjov become a bridge entity, by which the text passages related to “Kyjov” are 5
"How often does the folk festival in the South Moravian city with fewer people than Boskovice but more than others take place?" List of South Moravian cities by population
Text Passages
Rank
City
District
Population
1
Brno
Brno-City
380,681
2
Hodonín
Hodonín
24,065
3
Boskovice
Blansko
11,622
4
Kyjov
Hodonín
11,218
/wiki/Boskovice : "Boskovice is a town in the Blansko District of the South Moravian Region of the Czech Republic..." /wiki/Kyjov : "Kyjov is a town in the Hodonín District, famous for its folk festival 'Slovácký rok' held every four years..."
/wiki/Hodonín_District : "The Hodonín District is one of the seven districts in the South Moravian Region..."
"Boskovice" in question
City = "Boskovice" (row 3)
rank below row 3 → row 4 (City="Kyjov", District="Hodonín")
Doc /wiki/Kyjov matched
entity from question bridge entity (not in question, discovered by Query Linker)
Figure 2: Bridge entity discovery on a HybridQA example. Top: the table and text passages seen by the system. The yellow cell marks the entity grounded from the question ("Boskovice" maps to the City of row 3); amber cells mark bridge entities discovered by the Query Linker (City = Kyjov, District = Hodonín) — absent from the question but needed to retrieve the relevant passage. Bottom: the step flow shows how "Boskovice" anchors row 3, the rank-below row’s cells become bridge entities, and bridge "Kyjov" identifies the candidate text passages. The grounded entities and matched passages are then handed to the downstream Planner (Phase B), which generates the operator pipeline that ultimately extracts the answer. plan as an explicit intermediate representation also makes the system portable across backends, since only the final translation step changes.
3.5
requires cross-source reasoning between text and table. Each example shows a complete end-to-end pipeline for that type, so the model can pattern-match its plan to the closest precedent and adapt the predicates rather than synthesizing the API calls from scratch. Layer 3 appends runtime constraints, i.e., format requirements, common pitfalls, and engine-specific invariants, which the generated code must satisfy. For our running example, Phase C receives the three-step action plan from Phase B (filter → join → aggregate) and consults the Compose(TextQ, TableQ) entry in Layer 2 of the reference document. Then the model adapts the Phase B generated template by substituting the plan’s natural-language predicates into the corresponding operator API calls and emits the final backend-specific program. The generated program, when executed, invokes the backend’s implementation of the semantic operators to operate data and generate result. The execution time and LLM cost are then managed by the backends themselves and lie outside our scope.
Phase C: Code Generation
Phase C translates the action plan into executable code for a target backend. The translation is a single LLM call with two inputs: the action plan, and an auto-generated reference document placed in the system prompt. The reference document is pre-generated offline and organizes everything Phase C needs into three layers, illustrated in Figure 3 which uses that of Palimpzest as an example. Layer 1 lists the backend’s operator signatures, so the model knows what functions are available and the parameters for each function. Layer 2 provides one runnable golden pipeline per question type, where a question type is defined by the data source modalities required to answer it, e.g., TableQ can be answered from a table alone, TextQ from text passages, ImageQ from images, and Compose(TextQ, TableQ) 6
4 Reference Doc
EVALUATION
Palimpzest, MMQA
We evaluate NL2Pipe along four dimensions: (i) end-to-end answer quality after running the pipelines it produces, (ii) mapping cost which is the LLM cost spent in building the pipelines, (iii) efficiency, including mapping efficiency for building the pipelines and execution efficiency for running the pipelines, and (iv) component contribution via ablation study.
Layer 1: API Signatures dataset.sem_filter(filter, depends_on) dataset.sem_join(other, condition, depends_on) dataset.sem_agg(col, agg, depends_on)
Layer 2: Golden Pipeline (TableQ)
4.0.1 Datasets and Backends. We use five evaluation datasets: MMQA [20], HotpotQA [25], HybridQA [4], ManyModalQA [6], and TAT-QA [30]. They span three modalities (table, text, image) and several reasoning patterns. MMQA and ManyModalQA both involve all three modalities, but MMQA requires multi-hop reasoning whereas ManyModalQA is mostly single-hop. HybridQA pairs every question with a Wikipedia table and linked passages, so all its questions are inherently cross-source. HotpotQA is text-only and tests multi-hop reasoning without table structure. TAT-QA adds numerical reasoning over financial reports with hybrid table–text contexts. Together, these datasets cover diverse scenarios and complexity, ensuring the conclusion is strong enough. Because semantic-operator execution is slow (many LLM calls per question), we evaluate on a stratified sample of 300 questions per dataset, drawn proportionally to question types (e.g., TableQ/TextQ/ImageQ/Compose for MMQA; bridge/comparison for HotpotQA). This preserves coverage of fine-grained types (e.g., MMQA has totally 15 types of queries) while keeping the grid tractable: the full 5 major_baselines × 3 backends × 5 datasets evaluation already requires 22,500 mapping LLM calls plus far more backend-internal calls, in which case running all of them on the full datasets would take roughly 104 days. We choose three state-of-the-art semantic operator systems as evaluation backends: Palimpzest [12], LOTUS [14] and Nirvana [31]. Each question and pipeline will be tested using all these systems.
table_ds = pz.MemoryDataset(id="t", vals=table_rows) result = table_ds.sem_agg(col={...}, agg=f"{question} {INSTR}")
Layer 3: Runtime Constraints • Call _inject_schema after every MemoryDataset. • ...
Figure 3: Overview of an example auto-generated Reference Doc for Palimpzest, MMQA. Three layers: operator signatures, one runnable pipeline per question type, and runtime constraints. Full document is ∼250 lines.
Auto-generated reference documents. The reference document (Figure 3) is produced offline, once per (backend, dataset) pair, by a bootstrap pipeline (bottom of Figure 1). So it does not introduce runtime overhead and cost. Specifically, a Dataset Profiling agent inspects the raw dataset to produce a structured profile of its data sources. A Reference-Doc agent then generates multiple candidate documents based on the backend’s official API documentation and the dataset profile. By this point, each candidate document is composed of operator signatures (Layer 1) and golden pipeline examples (Layer 2). Finally, a Critic agent scores all candidates against the backend’s own example code (officially provided in their documentation or repository) on criteria such as syntactic correctness, pipeline completeness, and coverage of all question types, and selects the highest-scoring candidate. Finally, runtime constraints (Layer 3) like format requirements, common pitfalls, and engine-specific invariants are appended to the selected document as-is.
3.6
4.0.2 Baselines. We compare NL2Pipe against five baselines that span the design space of LLM-driven pipeline construction. (1) Naive issues a single LLM call that receives the backend’s official documentation, a dataset description, and the question, and directly emits executable code. (2) Hardcoded is a baseline simulating that a human engineer hand-writes one pipeline template per question type (e.g., filter → aggregate for TableQ; filter → join → aggregate for Compose) and phrases the natural-language predicate into each operator within the template. In our implementation an LLM performs the predicate composition at query time to simulate human doing it. And because this is just simulation which is actually completed by human in real world, its token cost does not make sense and is not comparable to other methods. We therefore report Hardcoded’s answer quality and latency, while omit its mapping cost. And actually the mapping latency is also not comparable to other methods due to the same reason, while its pipeline execution latency is still meaningful. Hardcoded estimates the pipeline quality attainable by hand-crafted templates. This baseline proves that pre-defined templates are far from enough to construct effective pipelines for complex workload. (3) CodeTree [11] is an agent-guided code-search baseline. For each question it proposes up to five candidate strategies, generating and executing code for each. If error occurs, additional reflection
Cost Summary
The per-query cost of NL2Pipe is bounded: three LLM calls for single-source query (the Linker’s entity-grounding step, plus one for the Planner and one for Code Generator) and four for composed questions where the Linker may additionally runs bridge verification. This overhead is tiny comparing to the internal LLM calls inside the backend during execution, which typically number in dozens and dominate end-to-end cost. The offline bootstrap (dataset profiling and reference-document generation) adds a one-time cost per (backend, dataset) pair that is amortized across all subsequent questions. Our evaluation reports this online mapping cost (from query until the pipeline built) separately from backend execution cost, and shows that the additional compilation overhead buys substantial quality gains on complex workloads. 7
will inspect and propose a repair, which is then re-executed. We allow the search to expand to at most three branching levels and to take at most ten total propose-execute-repair iterations before returning the best candidate seen. Unlike NL2Pipe that does not need execution and loops during the pipeline construction, CodeTree relies on execution feedback and reflection. We use it to evaluate whether such feedback loops truly have advantage over NL2Pipe’s single-pass compilation. (4) Codex (model) uses the same prompt as Naive but calls specific codex model (gpt-5.1-codex-mini) instead of generalpurpose reasoning model used by the other baselines. It serves as a stronger baseline than Naive as the codex model is optimized for code generation. We use this baseline to evaluate how well a standalone code-specialized model can handle our task. (5) Codex (agent) is the standard Codex agent that relies on the same code-optimized model (gpt-5.1-codex-mini) with a best-ofthree harness: for each question the agent generates three candidate pipelines independently with full tool access, executes each candidate using the backend, and picks the best of the three answers based on the agent’s understanding of the query and data. We set it to generate 3 candidates based on grid search from 1 to 3, where at 3 the agent has already needed longer running time than NL2Pipe but still underperforms NL2Pipe. So we stop at 3. Because each query triggers three full generations and executions, it is markedly slow and costly, so we evaluate it on 100 stratified sample queries per (backend, dataset) combination (reported in Table 4) rather than using the full 300 queries. In our implementation, for pipeline construction, the Codex (model) and Codex (Agent) baselines use gpt-5.1-codex-mini, while all other baselines and NL2Pipe use gpt-5-mini. The backend engines are always using gpt-5-mini for pipeline execution.
For efficiency we measure three average latencies per question (in seconds): mapping latency 𝑡𝑚 (the mapping process), execution latency 𝑡𝑒 (running the generated code using specific backend engine until the result is returned), and end-to-end latency 𝑡 = 𝑡𝑚 + 𝑡𝑒 . To better present the cost–quality trade-off, we additionally report quality-per-unit-cost, i.e., EM/$, F1/$, and LLM/$ (quality score per milli-USD, where the unit of cost is 0.1 cents), higher is better.
4.1
Main Results
Tables 1, 2, and 3 report end-to-end query processing quality and mapping cost on Palimpzest, LOTUS, and Nirvana respectively. Due to the long running time of Codex(agent), we only run it on 100 stratified samples of queries over 6 (backend, dataset) combinations, instead of all 300 queries. So we have a separate table (Table 4) for Codex(model), Codex(agent) and NL2Pipe on those 100 queries. NL2Pipe achieves the best quality among all methods. In the full 300-query end-to-end evaluation (Table 1, 2 and 3), the pipeline NL2Pipe achieves the best quality in most cases with a significantly large advantage over the baselines, especially in complex multi-hop or multi-modal cases where long and cross-source reasoning is needed. On Palimpzest, NL2Pipe perform the highest EM, F1 and LLM-as-judge scores in almost all datasets, where the largest gaps between NL2Pipe and baselines appear in HybridQA (20%, 21.1% and 22.3% higher EM, F1 and LLM-as-judge scores than the second-best baseline), MMQA (17.4%, 14.8% and 10.6% higher quality than second-best baseline) and TAT-QA (21.7%, 14.7%, 3% higher), which are multi-modal and multi-hop, or require crosssource numerical reasoning. In contrast, on text-only HotpotQA and single-hop ManyModalQA, the gaps between NL2Pipe and baselines are smaller. These prove the effectiveness of our method to build high-quality pipelines, especially for complex workload. Similar pattern appears across backends. On LOTUS NL2Pipe achieves the best quality on HybridQA and TAT-QA, as well as ManyModalQA. On Nirvana our best performance occurs on all datasets except HybridQA. Furthermore, as reported in Table 4, NL2Pipe outperforms the Codex coding agent in most cases with significantly higher quality, lower mapping latency, and 1/30 1/20 cost comparing to Codex agent. This result strongly supports the effectiveness of our method. NL2Pipe is cost efficient. Naive is the cheapest method but often loses substantial quality, while CodeTree can recover quality only by spending much more on search, and Codex agent takes biggest time and cost due to overcomplex agent runtime and multiple generations for each pipeline. NL2Pipe is the second-cheapest method in many cases with the highest quality: it achieves a strong overall quality profile at roughly $0.013 per question (averaging on all datasets and backends), ∼ 1.8× cheaper than CodeTree ($0.024), comparable to single-call Codex ($0.014) and 20 ∼ 30 times cheaper than Codex agent. This is because mapping cost of NL2Pipe is bounded: the Linker issues at most two LLM calls, followed by one Planner call and one Code Generator call, regardless of question complexity. CodeTree’s cost grows with reflection rounds and exceeds $0.03 per query where many candidate strategies fail the verification.
4.0.3 Metrics. To measure the pipeline construction quality, each pipeline will be executed using the corresponding backend and get the returned answer. Based on the generated answer, we report three quality metrics, all shown as percentage without explicit "%": (1) Exact Match (EM) is the fraction of generated answers exactly matching the gold answers on string level, after standard normalization (lowercasing, punctuation and article removal, and whitespace collapse); (2) token-level F1 is the harmonic mean of unigram precision and recall against the gold answers; and (3) LLMas-judge (shown as “LLM” in tables) is the fraction of generated answers judged semantically equivalent to the gold answers by an LLM using a fixed six-rule prompt [28]. Each quality metric is averaged over all 300 questions. We also report average mapping cost in USD per question: the money cost of LLM calls to build each pipeline (which is essentially the process of mapping initial query to the final pipeline). In NL2Pipe, mapping process is Phases A–C; in baseline methods, it is the whole process from beginning until the pipeline code is generated, since they do not have offline processing. The cost of backend execution is excluded as it is managed by each backend engine and out of scope of our work. Cost of the Hardcoded baseline is also excluded, for the reason given in Section 4.0.2. 8
Table 1: End-to-end results on the Palimpzest backend. Quality (average EM, F1, LLM) is percentage number without explicit %; $ means average mapping cost per query (in USD). Hardcoded’s mapping cost is excluded because the LLM calls in it is purely for simulating human actions, which is not necessary in real cases that manually build the pipelines. Bold marks the best quality score among all pipelines and the lowest mapping cost among pipelines, while underline marks the second lowest mapping cost. MMQA
HybridQA $
EM
F1
LLM
HotpotQA $
EM
F1
LLM
ManyModalQA $
EM
F1
LLM
TAT-QA
Pipeline
EM
F1
LLM
Naive Hardcoded CodeTree Codex (model)
12.7 34.3 19.7 25.3
13.6 42.3 23.7 29.7
16.0 0.0052 6.7 9.6 10.3 0.0054 46.3 62.1 77.3 0.0045 22.7 27.0 32.7 0.0049 18.0 23.3 33.0 0.0053 52.7 — 39.3 47.5 51.0 — 51.7 68.9 84.3 — 45.3 57.0 71.0 — 24.0 40.5 64.7 — 28.7 0.0332 10.7 13.9 16.7 0.0241 33.0 43.1 57.3 0.0283 31.7 38.7 51.0 0.0191 21.0 28.4 49.7 0.0212 39.0 0.0118 22.3 27.2 30.3 0.0164 35.3 46.9 60.3 0.0088 36.7 45.6 61.7 0.0105 17.3 23.5 40.0 0.0155
NL2Pipe (ours) 51.7 57.1 63.3 0.0134 59.3 68.6 73.3 0.0156 64.7 79.3 93.3 0.0109 48.7 59.0 68.7
$
EM
F1
LLM
$
0.0090 45.7 55.2 67.7 0.0125
Table 2: End-to-end results on the LOTUS backend. Columns and conventions as in Table 1. MMQA
HybridQA $
EM
F1
LLM
HotpotQA $
EM
F1
LLM
ManyModalQA
EM
F1
LLM
Naive Hardcoded CodeTree Codex
24.8 7.7 27.4 10.7
29.2 10.9 36.3 15.3
34.8 0.0059 32.7 42.6 51.7 0.0057 21.0 29.6 39.0 0.0055 31.0 39.7 52.3 0.0055 19.7 27.7 45.0 0.0055 16.4 — 20.3 30.7 47.0 — 22.7 43.7 80.0 — 23.0 28.5 40.0 — 14.0 20.0 42.7 — 47.8 0.0316 36.0 46.3 58.7 0.0249 28.7 42.4 61.7 0.0245 40.7 49.7 68.0 0.0202 29.7 40.7 61.7 0.0245 31.4 0.0191 11.3 20.9 45.3 0.0195 11.3 23.5 54.0 0.0138 11.7 21.2 55.3 0.0153 4.3 13.1 52.0 0.0127
NL2Pipe (ours) 24.4 34.7 51.2 0.0133 32.3 46.3 66.3 0.0177 28.3 45.1 74.7
$
EM
F1
LLM
TAT-QA
Pipeline
$
EM
F1
LLM
$
0.0112 43.0 52.6 68.3 0.0100 36.3 47.6 70.7 0.0126
Table 3: End-to-end results on the Nirvana backend. Columns and conventions as in Table 1. MMQA LLM
HybridQA
Pipeline
EM
F1
$
EM
F1
Naive Hardcoded CodeTree Codex
1.3 0.3 6.3 7.7
2.2 2.7 0.0061 8.7 10.8 0.3 0.3 — 9.7 12.1 8.7 10.3 0.0305 31.0 38.8 13.2 25.7 0.0153 6.7 10.5
LLM
HotpotQA $
EM
F1
LLM
ManyModalQA $
EM
F1
LLM
TAT-QA $
EM
F1
LLM
$
11.3 0.0059 16.0 23.7 33.3 0.0059 36.3 44.6 57.7 0.0055 20.3 27.2 40.0 0.0061 13.7 — 30.0 46.3 70.3 — 24.0 27.8 32.3 — 7.0 7.5 8.7 — 45.7 0.0264 30.3 40.7 54.3 0.0214 40.3 48.1 62.7 0.0127 24.3 33.3 50.7 0.0196 15.7 0.0173 15.7 25.7 49.0 0.0102 21.3 33.5 57.3 0.0082 10.0 19.3 43.3 0.0132
NL2Pipe (ours) 27.7 31.1 36.0 0.0149 29.3 35.4 40.0
0.0179 57.7 69.0 81.0 0.0121 42.7 50.2 56.7
Figure 4 visualizes the quality-cost trade-off with F1 and mapping cost, over varying datasets: on all three backends, NL2Pipe lies near the upper-left portion, outperforming all baselines except Naive, and Naive is often unavailable due to low quality in most time. This highlights that NL2Pipe can reach same quality with less cost, or use same cost to get higher quality. The offline bootstrap (data profiling and reference doc generation) for NL2Pipe (Section 3.5) adds a one-time cost of $0.40–$0.70 per (backend, dataset) pair, which adds only $0.001 ∼ 0.002 to each of the 300 queries and does not impact the conclusions. When more queries come, impact of this cost becomes even less.
0.0102 29.3 37.4 46.7
0.0152
that match no passages. Our Phase B Planner removes this burden by splitting the two decisions into planning and generation steps, such that in each single step LLM only focuses on one decision, improving accuracy. Hardcoded(Manual Programming): Fixed templates are limited and cannot handle many scenarios, like they cannot dynamically look for bridge entities for cross-source operators. Although we use LLM in this baseline to simulate the human programming progress, such programming in real world scenarios is manual and takes much more time than in simulation. So automated solution is necessary. CodeTree: Execution feedback repairs syntactic errors but is hard to handle semantic or logic errors which do not break the pipeline execution but just lead to wrong results. CodeTree’s reflection loop therefore burns extra LLM calls (2x more cost per question than NL2Pipe) on candidates that were doomed from the first generation. In contrast, NL2Pipe pays more attention to guaranteeing LLM has as complete information as possible at the beginning, such
4.1.1 Failure Mode in Baselines. We analyze the failure modes for the baselines and discuss how our three-phase architecture solves them. Naive: A single LLM call must simultaneously decide what operators to invoke and how to write them according to the target API. Under this dual burden the generated programs often make more mistakes, like dropping joins or emitting conjunctive text filters 9
Table 4: Codex (model) vs. Codex (agent) vs. NL2Pipe on 𝑁 =100 stratified queries per dataset. EM/F1/LLM-asjudge scores are percentage without %; $ is average mapping cost per query (USD); 𝑡𝑚 is average mapping latency per query (s), excluding failed execution runs; EM/$, F1/$, LLM/$ are quality scores per unit mapping cost (milli-USD), higher is better. Datasets abbreviated HyQ=HybridQA, HoQ=HotpotQA, MQ=MMQA, TQ=TAT-QA, MMQ=ManyModalQA. Bold marks the best value for quality and quality-per-cost. Back.
DS
Method
EM
F1
LLM
$
mapping and execution). In addition, the two tables include qualityper-unit-cost metrics. Furthermore, Table 5 separates two regimes. Naive and Hardcoded are intentionally simple reference baselines in higher block, while the lower block (CodeTree, Codex, and NL2Pipe) contains the specialized multi-agent or coding-optimized methods which are more comparable with NL2Pipe. Within this latter group, NL2Pipe achieves the best quality-per-unit-cost on every dataset, at a mapping latency (63–107 s) that is up to 90% faster than CodeTree. Comparing to Codex agent (Table 4), NL2Pipe’s pipeline building is up to 2x faster (311.2 VS 111.3s on Palimpzest over HybridQA). These results show that NL2Pipe has an acceptable efficiency in building pipelines, making it practical as an agentic approach, while maintaining the highest quality and second lowest cost among all evaluated methods. The only baseline that has both better efficiency and lower cost than NL2Pipe is Naive, which is often unusable due to extreme low quality (like on Palimpzest and Nirvana). Another observation is that NL2Pipe’s pipelines have the fastest execution among all methods in 3 out of 5 datasets in Table 5. This means the generated pipelines by NL2Pipe are not only high quality, but also well optimized to be highly efficient. This further shows that our method is practical and effective. In summary, NL2Pipe takes a reasonable mapping time and tiny mapping cost to gain significantly high quality and executionoptimized pipelines. Such an approach is practical to bridge semantic operator systems with automated application workflows.
𝑡𝑚 EM/$ F1/$ LLM/$
Palimpzest HyQ
Codex (model) 24.0 27.9 28.0 0.0077 17.3 Codex (agent) 43.0 52.9 60.0 0.2521 311.2 NL2Pipe 57.0 67.3 74.0 0.0156 111.3
3.1 0.2 3.7
3.6 0.2 4.3
3.6 0.2 4.7
Palimpzest MQ
Codex (model) 30.0 33.7 43.0 0.0096 16.1 Codex (agent) 38.0 43.3 50.0 0.2775 143.0 NL2Pipe 47.0 53.0 59.0 0.0137 76.8
3.1 0.1 3.4
3.5 0.2 3.9
4.5 0.2 4.3
LOTUS
TQ
Codex (model) 7.0 14.7 51.0 0.0132 14.8 Codex (agent) 7.0 17.9 66.0 0.2758 189.9 NL2Pipe 41.0 52.3 74.0 0.0107 87.1
0.5 0.0 3.8
1.1 0.1 4.9
3.9 0.2 6.9
LOTUS
Codex (model) 14.0 22.1 56.0 0.0145 16.5 MMQ Codex (agent) 15.0 27.7 73.0 0.2620 169.4 NL2Pipe 45.0 54.3 72.0 0.0080 73.2
1.0 0.1 5.6
1.5 0.1 6.8
3.9 0.3 9.0
Nirvana
HoQ
Codex (model) 20.0 29.8 53.0 0.0099 11.8 Codex (agent) 26.0 36.9 62.0 0.2484 126.6 NL2Pipe 66.0 75.8 86.0 0.0117 79.3
2.0 0.1 5.6
3.0 0.1 6.5
5.4 0.2 7.4
Nirvana
MQ
Codex (model) 5.0 10.8 20.0 0.0158 17.1 Codex (agent) 16.0 22.8 37.0 0.2759 165.4 NL2Pipe 27.0 31.2 35.0 0.0139 101.5
0.3 0.1 1.9
0.7 0.1 2.2
1.3 0.1 2.5
4.2.1 Efficiency of Each Phase. We further measure the fraction of each phase’s latency over the whole mapping latency in NL2Pipe, finding that Phase A is the bottleneck inside NL2Pipe. Table 6 reports the detailed latency fractions. Phase A is the bottleneck on every dataset, taking 53%–61% of the mapping time. Within Phase A, the entity grounder dominates because its input is the largest: it sees the question plus the full data context (table rows and previews of every text and image passage) and must emit a structured labeling of every key phrase. Phase B and Phase C create less latency despite producing the most operationally important output, because their inputs are short structured objects (the Linker output and the action plan) rather than data context.
that LLM can see a broader picture about the datasets, backends and semantics hidden in question and data, by preprocessing like identifying query and bridge entities before planning. Our evaluation shows that such beforehand information completeness is more important than reflection and retry for pipeline built with fixed-scope operators. Codex: As a single-LLM call too, Codex (model) achieves better quality than Naive given its code-generation-optimized model, and unavoidably inherits Naive’s failure mode, limiting its effectiveness. So it is often outperformed by CodeTree and NL2Pipe. Codex (agent) achieves higher quality than Codex(model) in cost of significantly more latency and cost, and it is still underperforming NL2Pipe. This is because as a general-purpose coding agent, Codex agent has extra components and advanced architecture that are overcomplex to the pipeline construction task. In such a fixed-scope problem (e.g., limited scope of operators and limited combinations of them), our evaluation shows that Codex agent tends to build unnecessarily complex and long pipelines to solve even the simplest queries. This proves specialized problem needs specialized solutions like NL2Pipe.
4.2
Potential Optimization on Efficiency. The profile in Table 6 suggests several non-exclusive ways to reduce mapping latency without changing the per-phase semantics: (i) Cheaper entity grounder. The grounder’s task is predominantly label assignment over question spans against data columns, closer to a structured extraction task than to open-ended reasoning; a smaller, faster model (e.g., gpt-5-nano) likely retains most of its accuracy at a fraction of the current latency. (ii) Lexical pre-filter for bridge verification. A cheap inverted-index or BM25 pre-filter can eliminate candidates that lexically cannot match any data items, leaving the LLM to confirm only the survivors that match at least one data item. (iii) Prompt caching for Phase B and Phase C. Both phases share a large invariant prefix per (backend, dataset) pair (the planning hints and the reference document, respectively). So prompt-caching APIs would amortize the prefix’s Time-to-firsttoken (TTFT) across all questions in a dataset. In this paper, we report the un-optimized mapping latency without the solutions above so that the metrics in Table 5 and 4 reflect the algorithm rather than the engineering advantages.
Efficiency
The efficiency results are reported in Table 4 and 5. Table 4 compares our method and two Codex methods over the same 100 sampled queries due to long running time of Codex agent. Table 5 reports the average latency over all three backends per dataset as supplementary to Table 1,2,3, with the 300 queries for each dataset. 𝑡𝑚 , 𝑡𝑒 and 𝑡 are mapping, pipeline execution and total time (sum of 10
F1 (%)
Palimpzest
LOTUS
80 70 60 50 40 30 20
Nirvana
80
50
60
40 40
30
20
20 10
10 1
10 2
Mapping cost (USD per question, log scale) Naive
10 1
10 2
Mapping cost (USD per question, log scale)
CodeTree
Codex (model)
Codex (agent)
0
10 1
10 2
Mapping cost (USD per question, log scale) NL2Pipe (ours)
Figure 4: Quality–cost trade-off on the three backends over varying datasets. Each point is a (pipeline, dataset) combination from Tables 1–3; same-color points trace one pipeline across the five datasets. NL2Pipe (red) occupies the upper-left portion of the figure in most cases, indicating a better quality-cost trade-off than other methods. Naive is often unavailable due to low quality. Table 5: Per-pipeline efficiency on each dataset, averaged across the three backends (Palimpzest, LOTUS, Nirvana). 𝑡𝑚 , 𝑡𝑒 , 𝑡 are mapping, execution, and end-to-end latency per question (seconds). EM/$, F1/$, LLM/$ are quality scores per milli-USD of mapping cost (higher is better). Bold marks the best value per (dataset, column) within the lower group only (CodeTree, Codex (model), NL2Pipe); Naive and Hardcoded are reference baselines and left unbolded. Dataset
TAT-QA
HotpotQA
HybridQA
MMQA
ManyModalQA
Pipeline
𝑡𝑚
𝑡𝑒
𝑡
Naive Hardcoded
29.9 28.2 58.1 0.0043 22.5 20.2 42.7 —
4.5 —
6.1 —
9.2 —
CodeTree 134.7 25.0 159.6 0.0217 Codex (model) 15.6 20.3 36.0 0.0118 NL2Pipe 92.7 16.2 109.0 0.0116
1.1 0.9 3.2
1.6 1.6 4.0
2.5 3.8 5.3
Naive Hardcoded
27.2 37.3 64.4 0.0035 19.9 16.7 36.6 —
7.9 —
11.2 —
15.0 —
CodeTree 138.4 37.5 175.8 0.0243 Codex (model) 12.6 16.2 28.7 0.0106 NL2Pipe 78.0 16.1 94.0 0.0108
1.4 2.0 4.8
1.8 3.0 6.1
2.3 5.1 8.0
Naive Hardcoded
28.3 59.0 87.3 0.0036 26.9 57.6 84.5 —
4.8 —
6.4 —
7.8 —
CodeTree 159.7 53.3 213.1 0.0255 Codex (model) 18.1 54.9 73.0 0.0166 NL2Pipe 117.0 66.6 183.5 0.0156
1.0 1.1 2.6
1.3 1.5 3.2
1.6 2.1 3.8
Naive Hardcoded
27.6 28.6 56.1 0.0034 25.9 31.1 57.0 —
3.8 —
4.4 —
5.2 —
CodeTree 166.7 28.4 195.1 0.0290 Codex (model) 15.2 19.0 34.3 0.0148 NL2Pipe 87.8 15.9 103.7 0.0116
0.6 1.0 2.9
0.7 1.3 3.5
0.9 2.2 4.3
Naive Hardcoded
24.7 15.3 40.0 0.0041 16.6 14.8 31.4 —
7.9 —
9.5 —
12.5 —
CodeTree 107.3 20.1 127.4 0.0191 Codex (model) 12.2 12.1 24.3 0.0109 NL2Pipe 67.2 12.2 79.4 0.0092
2.1 2.1 5.2
2.5 3.1 6.2
3.3 5.3 7.4
$
Table 6: Per-phase latency fraction of NL2Pipe on the Palimpzest backend over selected datasets (seconds per question). Phase A is the Query-Data Linker (entity grounder + optional bridge verifier), Phase B is the Semantic Planner, and Phase C is the Code Generator. Bold marks the largest latency fraction per dataset. Dataset ManyModalQA MMQA HybridQA
EM/$ F1/$ LLM/$
4.3
Phase A (%)
Phase B (%)
Phase C (%)
60.6 53.4 55.3
15.0 21.3 19.1
24.4 25.3 25.6
Ablation Studies
We ablate the three critical building blocks of NL2Pipe–the QueryData Linker (Phase A), the action plan (Phase B), and the autogenerated reference document (Phase C input), by disabling each component independently while holding the rest of the workflow fixed. –Linker presents the workflow that skips Phase A entirely, where the Planner sees only the question and the data schema. – Plan skips Phase B, where the Code Generator instead receives the Linker’s output (entity groundings and verified bridges) directly as part of the prompt. –L&P ablates both phases A and B, and the Code Generator sees only the question and data schema. –Ref replaces the auto-generated reference document input to Phase C with the original official backend documentation, with no golden pipeline examples, no dataset profile, and no backend-specific constraints, while still keeping the Phase B action plan as part of input to Phase C. We report ablations on MMQA and TAT-QA across all three backends (Table 7). The two datasets exercise the most complex reasoning patterns (cross-source Compose for MMQA, numerical aggregation with hybrid context for TAT-QA), making them the most informative stress test. We organize the discussion around three claims: each phase has distinct impact (Section 4.3.1), the contribution of the reference document grows with how unusual 11
Table 7: Ablation of NL2Pipe’s three components (PZ: Palimpzest) on MMQA and TAT-QA across all backends. Quality metrics (EM, F1, LLM) in percentage without %. Each row disables the listed component(s), and the unmodified NL2Pipe’s numbers are reported in Tables 1–3. Bold marks the lowest score in each (backend, dataset, metric) column.
PZ MMQA
PZ TAT-QA
LOTUS MMQA LOTUS TAT-QA Nirvana MMQA Nirvana TAT-QA
Configuration EM
F1
LLM EM
F1
LLM EM
–Linker –Plan –L&P –Ref
54.6 45.1 35.8 22.3
58.7 54.7 40.7 25.7
51.0 44.3 54.2 32.1
65.0 21.4 28.8 58.3 4.3 9.5 72.0 7.0 12.2 46.0 32.1 38.6
49.7 36.3 31.0 19.7
40.3 32.7 41.3 26.7
F1
the backend’s API is (Section 4.3.2), and the Linker and Planner are coupled rather than additive (Section 4.3.3).
LLM EM
F1
LLM EM
F1
LLM
EM
F1
LLM
44.8 20.3 23.7 49.5
37.4 30.0 36.6 46.2
65.3 34.7 38.3 42.3 8.3 9.3 47.0 10.0 10.7 60.7 22.7 27.6
43.7 10.3 11.3 36.0
25.3 4.0 3.7 27.3
31.6 5.4 3.9 34.4
38.7 7.3 4.3 46.7
26.7 24.0 29.0 37.7
pretraining already covers the API conventions. The reference document mechanism also contributes directly to backend portability: the same Phase A and Phase B logic does not vary on backends, and only the Phase C reference document change based on the backend. Most configuration of NL2Pipe keeps consistent across backends.
4.3.1 Each phase has distinct impact. Disabling a phase usually degrades quality, but the ablation table also contains a few nonmonotonic cases. The broad pattern is nevertheless diagnostic: the drops overall align with the role each phase plays in the NL2Pipe. The Planner is overall the most consequential component. Removing it (–Plan) makes the largest single-phase drop on both LOTUS and Nirvana, e.g., on LOTUS MMQA the LLM-as-judge score collapses from 51.2 to 20.3 (−30.9), and on Nirvana MMQA from 36.0 to 10.3 (−25.7). Palimpzest shows a smaller drop (−8.6 on MMQA). This asymmetry reflects how each backend handles under-specification: when no explicit plan is supplied, the Code Generator must simultaneously decide what to compute and how to write it. So the more the backend API requires, the higher chance that mistakes occur in the generated pipelines. Palimpzest’s operators allow free-text predicates while LOTUS and Nirvana requires more, like one predicate has to include the column names it applies to. So missing plan has more impact on LOTUS and Nirvana than Palimpzest. This also indicates that the planning phase contributes information the Code Generator cannot reliably infer on its own. The Linker helps modestly on its own, but its impact grows when the downstream phases weaken. Dropping only the Linker (–Linker) costs little on Palimpzest (−4.6 on MMQA, −2.7 on TAT-QA): when a question already names the entities it asks about, the downstream phases can still locate them without the Linker. The Linker mainly helps on harder questions whose key entity is never explicitly stated in the question and must instead be discovered in the data. The full value of the Linker becomes more visible when the downstream phases are also weakened (Section 4.3.3).
4.3.3 Linker and Planner are coupled, not additive. The combined ablation –L&P behaves non-monotonically and suggests an interaction between Linker and Planner. On Palimpzest TAT-QA –L&P has 72.0 LLM-as-judge score, higher than –Plan alone (58.3) or – Linker alone (65.0). The reason is when only the Plan is removed, the Linker output is input to Phase C and occasionally biases the model toward over-specific filters (e.g., narrow text predicates that match no passages), while removing the Linker as well leaves the model more loosely constrained, sometimes producing simpler and robust pipelines. This suggests Phase A and Phase B are coupled: the Plan validates and structures the Linker’s evidence. Specifically, Linker output without Planner to discipline it can hurt, and a Plan without Linker output to ground it loses much of the informational content. Taken together, the four ablation conditions show that the three phases contribute distinct benefits. The Linker supplies entity-level evidence that cannot be easily captured by following phases. The action plan in Phase B structures the evidence into a backendagnostic operator sequence. The reference document closes the gap between the abstract plan and a specific backend’s API, and absorbs some compilation burden depending on how unusual the API is.
5
CONCLUSION
We presented NL2Pipe, one of the first middlewares enabling automated AI workflows to fully utilize semantic operator systems for query processing, by automatically compiling natural-language questions into executable semantic operator pipelines. We design a three-phase workflow with related optimizations. Such design guarantees high effectiveness, low cost, and good cross-backend compatibility. Evaluation shows that NL2Pipe achieves highest quality over baselines, with tiny cost and reasonable efficiency to build semantic operator pipelines for different queries, especially complex multi-hop or multi-modal workload. We see NL2Pipe as a big step towards seamlessly integrating semantic operator systems into automatic AI workflow, such that the AI applications can benefit from the latest advance in semantic data systems.
4.3.2 The reference document matters more for less familiar backends. The –Ref ablation shows that the contribution of the autogenerated reference document is significant when a backend’s API differ from “standard” Python operations. On Palimpzest, whose API style (lazy datasets, custom schema-bound operators) is the least conventional, –Ref shows the largest drop on both datasets (−37.6 on MMQA, −21.7 on TAT-QA). On LOTUS and Nirvana, whose DataFrame-style and functional-style APIs are closer to popular programming patterns that LLM has seen during pretraining, the effect is smaller and just occasionally non-monotonic. Therefore, the reference document bridges the gap when the LLM is not familiar with the backend syntax, and contributes less where 12
REFERENCES
[23] Jiayi Wang and Jianhua Feng. 2025. Unify: An Unstructured Data Analytics System. In Proc. ICDE. [24] Wenhan Xiong, Xiang Lorraine Li, Srinivasan Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oğuz. 2021. Answering Complex Open-Domain Questions with MultiHop Dense Retrieval. In International Conference on Learning Representations (ICLR). [25] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-Hop Question Answering. In EMNLP. [26] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR. [27] Le Zhang, Yihong Wu, Fengran Mo, Jian-Yun Nie, and Aishwarya Agrawal. 2023. MoqaGPT: Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model. arXiv preprint arXiv:2310.13265 (2023). [28] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MTBench and Chatbot Arena. In NeurIPS Datasets and Benchmarks Track. [29] Yongwei Zhou, Junwei Bao, Chaoqun Duan, Youzheng Wu, Xiaodong He, and Tiejun Zhao. 2022. UniRPG: Unified Discrete Reasoning over Table and Text as Program Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 7494–7507. https://aclanthology.org/2022. emnlp-main.508 [30] Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance. In ACL. [31] Junhao Zhu, Lu Chen, Xiangyu Ke, Ziquan Fang, Tianyi Li, Yunjun Gao, and Christian S. Jensen. 2025. Beyond Relational: Semantic-Aware Multi-Modal Analytics with LLM-Native Query Optimization. arXiv preprint arXiv:2511.19830 (2025).
[1] Asim Biswal, Liana Patel, Siddarth Jha, Amog Kamsetty, Shu Liu, Joseph E. Gonzalez, Carlos Guestrin, and Matei Zaharia. 2024. Text2SQL is Not Enough: Unifying AI and Databases with TAG. arXiv preprint arXiv:2408.14717 (2024). [2] Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. 2023. Autonomous Chemical Research with Large Language Models. Nature 624 (2023), 570–578. [3] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). [4] Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data. In Findings of EMNLP. [5] Hanjun Dai, Bethany Yixin Wang, Xingchen Wan, Bo Dai, Sherry Yang, Azade Nova, Pengcheng Yin, Phitchaya Mangpo Phothilimthana, Charles Sutton, and Dale Schuurmans. 2024. UQE: A Query Engine for Unstructured Databases. arXiv:2407.09522 [cs.DB] https://arxiv.org/abs/2407.09522 [6] Darryl Hannan, Akshay Jain, and Mohit Bansal. 2020. ManyModalQA: Modality Disambiguation and QA over Diverse Inputs. In AAAI. [7] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. 2024. MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework. In ICLR. [8] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for OpenDomain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 6769–6781. [9] Vishwajeet Kumar, Yash Gupta, Saneem Chemmengath, Jaydeep Sen, Soumen Chakrabarti, Samarth Bharadwaj, and Feifei Pan. 2023. Multi-Row, Multi-Span Distant Supervision For Table+Text Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). 8080– 8094. [10] Jiale Lao, Andreas Zimmerer, Olga Ovcharenko, Tianji Cong, Matthew Russo, Gerardo Vitagliano, Michael Cochez, Fatma Özcan, Gautam Gupta, Thibaud Hottelier, H. V. Jagadish, Kris Kissel, Sebastian Schelter, Andreas Kipf, and Immanuel Trummer. 2025. SemBench: A Benchmark for Semantic Query Processing Engines. arXiv preprint arXiv:2511.01716 (2025). [11] Jierui Li, Hung Le, Yingbo Zhou, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2025. CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models. In NAACL. [12] Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baile Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, Rana Shahout, and Gerardo Vitagliano. 2025. Palimpzest: Optimizing AI-Powered Analytics with Declarative Query Processing. In Proc. CIDR. [13] Shicheng Liu, Jialiang Xu, Wesley Tjangnaka, Sina Semnani, Chen Yu, and Monica Lam. 2024. SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models. In Findings of NAACL. [14] Liana Patel, Siddharth Jha, Melissa Pan, Harshit Gupta, Parth Asawa, Carlos Guestrin, and Matei Zaharia. 2025. Semantic Operators and Their Optimization: Enabling LLM-Based Data Processing with Accuracy Guarantees in LOTUS. Proc. VLDB Endow. 18, 11 (2025), 4171–4184. [15] Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. In NeurIPS. [16] Matthew Russo, Sivaprasad Sudhir, Gerardo Vitagliano, Chunwei Liu, Tim Kraska, Samuel Madden, and Michael Cafarella. 2026. Abacus: A Cost-Based Optimizer for Semantic Operator Systems. Proc. VLDB Endow. 19, 5 (2026), 1060–1073. [17] Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. 2025. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing. Proc. VLDB Endow. 18, 9. [18] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2024. HuggingGPT: Solving AI Tasks with ChatGPT and Its Friends in Hugging Face. In NeurIPS. [19] Ji Sun, Guoliang Li, Peiyao Zhou, Yihui Ma, Jingzhe Xu, and Yuan Li. 2025. AgenticData: An Agentic Data Analytics System for Heterogeneous Data. arXiv preprint arXiv:2508.05002 (2025). [20] Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. MultiModalQA: Complex Question Answering over Text, Tables, and Images. In ICLR. [21] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving Retrieval with Chain-of-Thought Reasoning for KnowledgeIntensive Multi-Step Questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). 10014–10037. [22] Matthias Urban and Carsten Binnig. 2024. Demonstrating CAESURA: Language Models as Multi-Modal Query Planners. In Companion of the 2024 International Conference on Management of Data (SIGMOD).
13