arXiv CSopen access

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language

· arxiv_cs

arXiv CS · Papers · License: Open Access

databases, sql, data management, storage

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language [Experiment, Analysis & Benchmark] Hang Zhang1 , Chaokun Wang1,∗ , Yuzhi Pan1 , Ziyao Zhong1 , Shuo Cao1 , Yue Xue1 , Zeyu Huang1 , Xingwei Zhou1 , Fang Niu1 , Bofan Xie1 , Guanchen Ge1 , Leqi Zheng1 , Ziyang Liu1 , Xiannian Cao2 , Pengcheng Ge2 1 Tsinghua University, Beijing, China

2 Lenovo Group Limited, Beijing, China

arXiv:2609.06527v1 [cs.CL] 6 Sep 2026

{zhanghang24, panyz24, zhongzy25, caos24, y-xue24, huangzy24, zhouxw24, nf21, xbf25, ggc22, zhenglq24, liu-zy21}@mails.tsinghua.edu.cn, [email protected], {caoxn3, gepc1}@lenovo.com

ABSTRACT Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasing attention from the database community. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and optimization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multiscenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an executionbased benchmark covering both Direct and Interactive modes. ProcArena comprises 3,998 executable tasks over 157 databases, spanning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic Enhancement and scenario-specific adapters, and derive paired Interactive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver–User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particularly in interactive settings. We release the benchmark on the public website https://zeyuhuangzh.github.io/procarena-site/. PVLDB Reference Format: Hang Zhang, Chaokun Wang, Yuzhi Pan, Ziyao Zhong, Shuo Cao, Yue Xue, Zeyu Huang, Xingwei Zhou, Fang Niu, Bofan Xie, Guanchen Ge, Leqi Zheng, Ziyang Liu, Xiannian Cao, Pengcheng Ge. ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language [Experiment, Analysis & Benchmark]. PVLDB, 20(1): XXX-XXX, 2027. doi:XX.XX/XXX.XX

This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected]. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 20, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX

PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/ZhanGHanG9991/ProcArena.

INTRODUCTION

By integrating procedural constructs with SQL, PL/SQL1 enables developers to implement data access and business logic directly within database systems, reducing communication overhead between applications and databases [16]. It has therefore become an important component of modern database systems [3, 6, 8, 9]. For example, Microsoft Azure SQL Database alone hosts over two billion PL/SQL codes that serve billions of daily invocations [14, 35]. However, developing PL/SQL codes is difficult in practice, as it requires knowledge of database schemas, business logic, and PL/SQL syntax [16, 38]. Recently, large language models (LLMs) have been increasingly explored for automating the translation from natural language (NL) to PL/SQL (NL-to-PL/SQL) [50]. Despite this progress, systematically assessing LLMs for practical NL-to-PL/SQL development remains challenging, as no benchmark captures the diverse development scenarios and interactions involved in real-world PL/SQL programming. Prior work provides only partial foundations for such an assessment. PLForge [50] introduces NL-to-PL/SQL datasets, models, and execution-based evaluation, but primarily studies direct PL/SQL code generation from complete requirements. NL-to-SQL benchmarks [21, 27, 45, 49] examine how models translate NL requests into SQL by grounding them in database schemas and contents, while interactive Text-toSQL benchmarks such as BIRD-Interact [18] further incorporate interactions with users and database environments. However, both lines of work target declarative SQL rather than PL/SQL codes. General code-generation benchmarks [5, 19, 31, 40] cover generation and modification tasks in languages such as Python, Java, and C++, but do not capture key characteristics of PL/SQL development, including database environment and procedural logic. To the best of our knowledge, ProcArena is the first benchmark to evaluate LLMs on NL-to-PL/SQL across multiple development scenarios in both Direct and Interactive modes across PostgreSQL and Oracle. Figure 1 illustrates the Direct and Interactive task under from scratch development scnario. To construct the benchmark, we outline two main challenges. C1: How can we construct realistic 1 In this paper, we use PL/SQL to refer to procedural extensions of SQL, including

Oracle PL/SQL [1], PostgreSQL PL/pgSQL [2], and related dialects.

ExampleGuided

Functional Modification

Code Composition

Performance Optimization

Code Repair

Synthesis (5)

Editing (4)

From Scratch DocumentGrounded

Template Instantiation

we have enough sto ck, an d then generate a ne w order. We als o need to figure out the total amount correct ly and make sure t he cart i s empty afterwards. The current proce ss is a bit messy an d I want to make s ure the data stays con si stent.

Multi-Turn Interaction

ask_ se le ctor

Which reading of the input scope is correct?

get_schema() -- what tables am I working with?

1. N o n e o f th e fo llo wi ng are co rrect. 2. The procedure takes only user_id; theactive cart is looked up internally. 3. Th e p ro ced u re takes b oth us er_i d and an exp li cit cart_i d . 4. A list of product_ids and quantities, without reading the cart.

table carts, columns=[cart_id(int|PK), user_id(int)]| table carts_item, columns=[cart_id, product_id, cart_item_quantity] | ……

Complete Requirement

Create an Ora cle PL/SQL stored proced ure that terminates employment for transportati on st aff who se years of service exceed twice the target service duratio n, but only proceed with terminati ons if the department maintain s adeq uate staff ing levels above the minimum requ irement.

LLM Solver

CREATE OR REPLACE PROCEDURE sp(para_Years_Working NUMBER, para_Min_Staff NUMBER) IS staff_count NUMBER; BEGIN SELECT COUNT(*) INTO staff_count FROM "transportation_staff" WHERE "Employment_Status" = 'Active'; IF staff_count > para_Min_Staff THEN DELETE FROM "transportation_staff" WHERE "Years_Working" > para_Years_Working * 2; END IF; END;

Interaction 3

Interaction 1 get_ schem a

Direct (Single-Turn)From Scratch

Initial Request DECLARE I need a stored proced ure to h andle t he che ckout fl ow. Bas ica lly, when a user with a specific user ID wa nts to user_cart_id INT; buy something , the system needs to grab everything from their sho pping cart , check the inventory to make sure (Underspecified)

Code Completion

CREATE OR REPLACE PROCEDURE create_order(uid_in INT) LANGUAGE plpgsql AS $procedure$

Interactive (Multi-Turn) From Scratch

Interaction 2 ask_ use r

2.Only user_id; the active cart is inferred.

I need to join the cart data with product details. Could you provide the specific foreign key relationship between the carts_item table and the products table? They are connected by the productID. The products table contains the price information, and the shopping cart table contains the quantity foreach item.

Interaction 4

execute_ scrat ch

UPDATE products SET stk_num = stk_num - 2 WHERE product_id = 101;SELECT stk_numFROM products WHERE product_id = 101; Ok, 1 row affected: stk_num | 8

LLM Solver submit

Complete requirement → PL/SQL code

new_order_id INT; order_amount DECIMAL := 0; BEGIN SELECT cart_id INTO user_cart_id FROM carts WHERE user_id = uid_in; INSERT INTO orders (user_id, total_amount) VALUES (uid_in, order_amount)RETURNING order_id INTO new_order_id; FOR rec IN SELECT * FROM carts_item JOIN products ON carts_item.product_id = products.product_id WHERE carts_item.cart_id = user_cart_id LOOP UPDATE products SET stk_num = stk_num - rec.cart_item_quantity WHERE product_id = rec.product_id; order_amount := order_amount + (rec.price * rec.cart_item_quantity); INSERT INTO orders_item (order_id, product_id, order_item_quantity) VALUES (new_order_id, rec.product_id, rec.cart_item_quantity); END LOOP; UPDATE orders SET total_amount = order_amount WHERE order_id = new_order_id; DELETE FROM carts_item WHERE cart_id = user_cart_id; END; $procedure$

Underspecified requirement → Interaction → PL/SQL code

Figure 1: ProcArena evaluates the same PL/SQL development target under two modes. Left: the nine subscenarios, five under Synthesis and four under Editing, and a Direct episode that maps a complete requirement to PL/SQL code in a single turn. Right: an Interactive episode that clarifies an underspecified request via ask_selector and ask_user turns with the User , execute turns against the Database Environment before the code is submitted. and difficult NL-to-PL/SQL tasks across multiple PL/SQL development scenarios? Existing publicly available NL-to-PL/SQL datasets rarely cover these diverse development scenarios. They primarily focus on from-scratch generation with complete NL requirements, resulting in relatively limited scenario diversity and task difficulty. C2: How can we construct an executable interactive environment for realistic multi-turn NL-to-PL/SQL development? Interactive NL-to-PL/SQL development requires the Solver to resolve incomplete requirements against both user intent and the database environment. A fixed transcript cannot assess whether the Solver appropriately seeks clarification or inspects database information, while unrestricted interaction may alter database state or leak evaluation signals. The environment must therefore support controlled interactions between the Solver and both the User and the Database Environment under an executable protocol. For C1, we use iterative logic enhancement to add procedural logic, jointly update the PL/SQL code and NL requirement, and validate their executability and semantic consistency. We then apply scenario-specific adapters to construct executable Direct tasks for nine subscenarios. For each subscenario, we rewrite the user requirement and prepare any required supporting artifact, such as a document, behavioral examples, a template, or existing code, while preserving the gold PL/SQL code. For C2, we apply knowledge injection and requirement perturbation to derive paired Interactive tasks from their Direct counterparts by introducing information omissions and resolvable contradictions while preserving the gold PL/SQL code. We maintain an elicitation ledger of the facts, definitions, and resolutions available to the User Simulator during interaction. We further design the Solver–User Simulator protocol to coordinate user interactions with database environment. We propose ProcArena, which comprises 3,998 NL-to-PL/SQL tasks across nine PL/SQL development scenarios in PostgreSQL and Oracle. We evaluate 7 models across Direct and Interactive tasks. On

the Direct and Interactive modes, the best average scores achieved by the models are 62.2% and 57.8%, respectively. The evaluation further compares generation across subscenarios and analyzes the construction components and User Simulator. In summary, we make the following contributions: (1) We propose ProcArena, the first benchmark to evaluate LLMs on NL-to-PL/SQL across multiple development scenarios in both Direct and Interactive modes across PostgreSQL and Oracle. (2) We construct PL/SQL-specific interactive environments through knowledge integration and requirement perturbation, and propose the Solver–User Simulator protocol to coordinate Solver–User interactions and Database Environment tools. (3) We conduct comprehensive experiments to evaluate seven models on 3,998 tasks across nine development scenarios in both Direct and Interactive modes, and further analyze the benchmark construction and interaction behavior.

PROBLEM FORMULATION

An NL-to-PL/SQL task asks a Solver to produce PL/SQL code from an NL requirement, a database description, and an optional task attachment. We represent the task input as 𝑋 = ⟨𝑞, D, A⟩,

(1)

where 𝑞 states the NL requirement. The database description D = ⟨D𝑠 , D𝑚 ⟩ contains the database schema D𝑠 and its metadata D𝑚 . A denotes the optional task attachment. Let P denote the space of PL/SQL codes. Given a task input 𝑋 , the Solver ultimately produces a candidate PL/SQL code 𝑃ˆ ∈ P. We model the Solver as S = ⟨M, A⟩, (2) where M is the LLM and A is its action space. 2

2.1

Direct NL-to-PL/SQL

Seed Sources

The direct NL-to-PL/SQL task provides a task 𝑋𝑑 = ⟨𝑞𝑑 , D𝑑 , A𝑑 ⟩ to a Solver S𝑑 = ⟨M𝑑 , A𝑑 ⟩, where A𝑑 = {submit}. The LLM directly generates a candidate PL/SQL code 𝑃ˆ𝑑 ∈ P from the task input: 𝑃ˆ𝑑 = M𝑑 (𝑋𝑑 ). (3)

Seed Task Logic enhancement Scenario Adaptation

2.2

frozen once, then read-only: gold code, call suite, initial state

Generation follows 𝑞𝑑 under D𝑑 and may use the task attachment A𝑑 . The Solver then submits 𝑃ˆ𝑑 for evaluation.

Interactive NL-to-PL/SQL

An interactive NL-to-PL/SQL task is a multi-turn collaboration among a Solver S𝑖 , a User Simulator U𝑖 , and a database environment E𝑖 . The S𝑖 receives the initial task input from U𝑖 : 𝑋𝑖 = ⟨𝑞𝑖 , D𝑖 , A𝑖 ⟩,

(4)

where 𝑞𝑖 may be underspecified. For a paired ProcArena task, D𝑖 = D𝑑 and A𝑖 = A𝑑 ; interactive construction changes the requirement view while preserving the database description and task attachment. The S𝑖 can clarify information through the U𝑖 and inspect the database through E𝑖 . The interaction action space A𝑖 of S𝑖 is A𝑖 = A U ∪ A E ∪ {submit},

𝑎𝑡 ∈ A𝑖 .

Contradiction injection

Task Interactive task

task construction, we apply knowledge injection and requirement perturbation to transform the direct NL-to-PL/SQL tasks into interactive NL-to-PL/SQL tasks that necessitate interaction between the Solver and the user to achieve correct task completion.

3.1

Seed Task Construction

We construct the seed NL-to-PL/SQL tasks from two sources. The first consists of the Hard and Webridge datasets from PLForge, whose underlying databases are derived from Spider. These datasets provide difficult NL-to-PL/SQL pairs in PL/pgSQL and Oracle PL/SQL, respectively. The second source is LogicCat [33], which contains databases and NL-to-SQL tasks that require more sophisticated logical reasoning and domain knowledge. Using the LogicCat databases together with their original complex NL-to-SQL examples as prompts, we employ LLMs to generate corresponding seed NL-to-PL/SQL pairs.

(6)

(7)

(8)

where ⊕ denotes appending the current action–response pair to the interaction history. After zero or more nonterminal interaction turns, the LLM generates a candidate PL/SQL code 𝑃ˆ𝑖 ∈ P based on the initial task input and the interaction history: 𝑃ˆ𝑖 = M𝑖 (𝑋𝑖 , ℎ𝑇 ), (9)

3.1.1 Iterative Logic Enhancement. Due to the strong code generation capabilities of state-of-the-art LLMs, we need to ensure that the NL-to-PL/SQL pairs in ProcArena are sufficiently challenging to effectively evaluate the capabilities of LLMs. In practice, users can extend a requirement by adding business rules. We leverage this characteristic and propose an iterative logic enhancement method, which let a held-out LLM to add one reasonable business rule in each round. Through iterative enhancement, we increase the procedural complexity of the PL/SQL code while ensuring that it remains grounded in realistic business scenarios. Concretely, the iterative procedure starts from an NL-to-PL/SQL pair in the source set and proceeds over multiple rounds. In each round, given the current NL-to-PL/SQL pair, the held-out LLM proposes a business rule, generates a PL/SQL code snippet that implements the rule, integrates the snippet into the current PL/SQL code, and updates the NL requirement to describe the resulting PL/SQL code.After 𝑁 rounds, we execute the final PL/SQL code on the database and use a held-out validation LLM to check whether

where 𝑇 is the number of interaction turns. The Solver S𝑖 then submits 𝑃ˆ𝑖 for evaluation, which terminates the task.

Requirement Perturbation Information omission

(5)

and the protocol updates the interaction history as ℎ𝑡 = ℎ𝑡 −1 ⊕ ⟨𝑎𝑡 , 𝑟𝑡 ⟩,

Reasoning Commonsense

Figure 2: The ProcArena construction pipeline.

For a nonterminal action, the corresponding interaction target returns a response 𝑟𝑡 = R (𝑎𝑡 , ℎ𝑡 −1 ; U𝑖 , E𝑖 ),

Editing B1 –B4

Knowledge Integration Domain

Direct task

where A U and A E denote the action spaces for interacting with U𝑖 and E𝑖 , respectively. The Solver S𝑖 initiates each interaction turn. Starting from an empty interaction history ℎ 0 = ∅, at an interaction turn 𝑡, the LLM M𝑖 selects an action based on the initial task input 𝑋𝑖 and the interaction history ℎ𝑡 −1 : 𝑎𝑡 = M𝑖 (𝑋𝑖 , ℎ𝑡 −1 ),

Synthesis A1 –A5

BENCHMARK CONSTRUCTION

In this section, we describe the construction of ProcArena tasks. Figure 2 illustrates the overall construction pipeline. We first construct the seed PL/SQL codes and enhance their procedural logic through an iterative logic enhancement method. We then rewrite the corresponding NL descriptions to preserve semantic consistency with the enhanced PL/SQL codes. For direct NL-to-PL/SQL task construction, we employ scenario adapters to transform the enhanced NL-to-PL/SQL pairs into direct NL-to-PL/SQL tasks spanning nine development scenarios. For interactive NL-to-PL/SQL 3

• A2: Document-Grounded Synthesis. The user also provides a design or requirement document that the code must follow. • A3: Example-Guided Synthesis. The user provides examples that illustrate the expected behavior of the target code. • A4: Template Instantiation. The user provides a generic code template and requests a target implementation that fills its task-specific bindings. • A5: Code Composition. The user provides complete, callable PL/SQL components and requests new target code that coordinates their calls, parameters, and procedural logic.

Algorithm 1: Iterative Logic Enhancement Input : Source pair ⟨𝑞 0, 𝑃0 ⟩; database description D; initial state 𝜎0 ; held-out construction LLM M𝑐 ; enhancement rounds 𝑁 ; repair budget 𝐵 Output : Validated seed task 𝜏 + , or Fail 1 (𝑞, 𝑃) ← (𝑞 0 , 𝑃 0 ) /* Add one rule per round */ 2 for 𝑛 ← 1 to 𝑁 do ˆ ← Enhance(M𝑐 , 𝑞, 𝑃, D) ˆ 𝑃) 3 (𝑞, ˆ ˆ 𝑃) 4 (𝑞, 𝑃) ← (𝑞,

For Editing, we define four subscenarios based on the type of requested code change:

5 𝜂 ← Execute(𝑃, D, 𝜎0 )

if Failed(𝜂) then /* Repair from execution feedback */ 7 for 𝑏 ← 1 to 𝐵 do 8 𝑃 ← Repair(M𝑐 , 𝑞, 𝑃, D, 𝜂) 9 𝜂 ← Execute(𝑃, D, 𝜎0 ) 10 if Succeeded(𝜂) then 11 break 12 if Failed(𝜂) or CheckAlignment(M𝑐 , 𝑞, 𝑃, D) = false then 13 return Fail 14 𝐶ˆ ← GenerateCalls(M𝑐 , 𝑞, 𝑃, D) ˆ 𝑃, D, 𝜎0 ) 15 𝐶 ← KeepStateChanging(𝐶, 16 if 𝐶 = ∅ then 17 return Fail + 18 return 𝜏 = ⟨𝑞, D, ∅, 𝜎0 , 𝑃, 𝐶⟩ 6

• B1: Functional Modification. The user requests changes that add, remove, or modify business behavior in existing code. • B2: Performance Optimization. The user requests better execution performance while preserving the behavior of existing PL/SQL code. • B3: Code Repair. The user reports a defect in PL/SQL code and requests a repair that restores the intended behavior. • B4: Code Completion. The user provides incomplete PL/SQL code and requests completion of a coherent missing block. 3.2.2 Scenario-Specific Adapters. We use scenario-specific adapters driven by held-out LLM to transform a validated seed task into a Direct task for each subscenario. The validated seed task defines the target behavior. Each scenario adapter creates the starting point for one development subscenario by rewriting the user requirement and adding a task attachment when needed. The attachment may contain a document, examples, a template, reusable components, or existing PL/SQL code. Let 𝜏 + denote the validated seed task. The scenario contract 𝐺 𝑗 defines the preconditions under which adapter 𝐹 𝑗 applies and the acceptance criteria that its output must satisfy. For subscenario 𝑗, the adapter constructs the complete Direct task

the NL requirement and code remain semantically aligned. We then generate a call suite and retain only calls that change the database state. The validated pair and its execution assets are collectively referred to as the seed task for subsequent Direct and Interactive task construction. We illustrate the iterative logic enhancement method in Algorithm 1.

3.2

𝐹 𝑗 (𝜏 + ; 𝐺 𝑗 ) = 𝜏𝑑,𝑗 = ⟨𝑞𝑑,𝑗 , D, A 𝑗 , 𝜎0, 𝑃, 𝐶⟩.

(10)

Here, 𝑞𝑑,𝑗 denotes the scenario-specific requirement, and A 𝑗 denotes the task attachment. The remaining elements correspond to the database description, initial database state, reference PL/SQL code, and call suite, all of which remain unchanged. Each scenario adapter constructs a task by rewriting the original requirement into a scenario-specific requirement and, when needed, generating a task attachment, while leaving all other task elements unchanged. The resulting NL requirement and attachment must contain all information necessary to complete the task and remain consistent with the database description. If candidate content contains conflicting statements, the adapter resolves the conflict and retains a single consistent statement. To construct the scenario-specific requirement and attachment, the adapter combines LLM-guided rewriting with scenario-specific artifact preparation. The LLM handles content that requires semantic rewriting. For Synthesis subscenarios that require an attachment, the adapter places the appropriate supporting artifact in the attachment: a document, behavioral examples, the provided generic template, or callable components. For Editing subscenarios, deterministic transformations derive code attachments from the reference PL/SQL code without modifying the reference code itself.

Direct Task Construction

In this section, we first define the scenarios and then explain how to construct Direct tasks based on the seed tasks. 3.2.1 Development Scenarios. We collect 1,535 unique posts from Stack Overflow and Database Administrators Stack Exchange by searching for keywords such as PL/SQL and PL/pgSQL. We then analyze the types of user requests in these posts and find that most could be broadly categorized into two categories. In 48.7% of the posts, users have no existing PL/SQL code and need to generate code from scratch. In another 38.4%, users have PL/SQL code but need to modify, optimize, fix, or complete it. We refer to these two categories as Synthesis and Editing, respectively. Based on the analysis, we further divide Synthesis and Editing into five and four subscenarios, respectively. For Synthesis, we define five subscenarios according to the information or code artifact provided with the task: • A1: From-Scratch Synthesis. The user provides an NL requirement and the database description and requests complete target code from scratch. 4

These transformations revert a business rule, introduce a behaviorpreserving slowdown, inject a defect, or remove a coherent code block. After each attempt, we validate the candidate against its scenario contract. If validation fails, the LLM receives the failed checks as feedback for the next attempt. Every candidate must pass three common checks to ensure that the constructed Direct task is valid, non-trivial, and semantically aligned with the reference seed task. ① Artifact validity verifies that the attachment has the type and structure required by the target subscenario. ② Task non-triviality ensures that the candidate neither exposes the reference code nor leaves no meaningful work for the Solver. ③ Reference alignment verifies semantic consistency with the reference code through execution. In addition to these common checks, each candidate must pass the scenario-specific checks defined in Table 1. After an adapter passes all checks, we materialize the resulting Direct task. The Solver receives 𝑋𝑑,𝑗 = ⟨𝑞𝑑,𝑗 , D, A 𝑗 ⟩.

Original sentence and controlled change

Logic Enhancement

Base: Add a wine and set every vintage to the requested year. Logic: Add the wine (skip duplicates), then assign vintages by priority: premium, target, or Napa Valley → requested year; otherwise → previous year.

Knowledge Injection

Original: Otherwise, set the vintage to one year before the requested year. Injected: Otherwise, apply the prior-vintage rule. [Glossary: prior-vintage rule = requested year − 1.]

Information Omission

Original: For wines marked “now,” increase Cases by exactly 1; treat NULL as 0. Omitted: For wines marked “now,” increase Cases. [The increment and NULL rule are hidden.]

Contradiction Injection

Intended: The premium-score threshold is 90. Injected: The main rule says score ≥ 90, but a closing note says score ≥ 80.

Figure 3: Controlled requirement edits on one example task. Each row pairs the original sentence with the rewritten one produced by a single operator, with the changed span highlighted: Iterative Logic Enhancement, Knowledge Injection, Information Omission, and Contradiction Injection.

(11)

This input contains only the scenario-specific requirement, database description, and task attachment. We keep the gold PL/SQL code, call suite, and initial database state hidden for offline evaluation. In Direct, the Solver is allowed to submit only once and cannot call ask_user, ask_selector, or any environment tool before submission. We construct Direct tasks for all nine subscenarios.

3.3

Type

Algorithm 2: Knowledge Injection Input : Task 𝜏𝑑,𝑗 ; contract 𝐺 𝑗 ; knowledge budget 𝜅; attempt budget 𝐴 𝐾 and ledger Output : Interactive requirement candidate 𝑞𝑖,𝑗 𝐾 Λ 𝑗 , or Fail 1 Π ← EditableProse(𝑞𝑑,𝑗 , 𝐺 𝑗 ); fb ← ∅ 2 if Π = ∅ then 3 return Fail 4 for 𝑎 ← 1 to 𝐴 do 5 ( Π̂, Λ̂𝐾 ) ← LLMInject(Π, 𝑃, D, 𝐺 𝑗 , 𝜅, fb) 6 fb ← KnowledgeGates( Π̂, Λ̂𝐾 ; Π, 𝑃, D, 𝐺 𝑗 ) 7 if fb = ∅ then 𝐾 ← ApplyRewrite(𝑞 , Π, Π̂) 8 𝑞𝑖,𝑗 𝑑,𝑗 𝐾 , Λ̂𝐾 9 return 𝑞𝑖,𝑗 10 return Fail

Interactive Task Construction

3.3.1 Interactive Requirement Construction. We construct Interactive tasks from the Direct tasks in Section 3.2 through two stages. In real-world development, user requirements are often not fully explicit: users may express their intent using domain-specific knowledge, logical reasoning, or commonsense knowledge, and may also omit necessary information or provide conflicting constraints. We model these realistic forms of underspecification through Knowledge Injection and Requirement Perturbation. Figure 3 illustrates both stages on one airline-operations task, together with the Iterative Logic Enhancement step that produces the underlying Direct requirement. Both stages modify only the requirement and preserve the task attachment, reference PL/SQL code, call suite, and initial database state. We apply this construction to eight subscenarios; B2 remains Direct-only because it uses runtime-based evaluation. Knowledge Injection transforms a complete Direct requirement by replacing selected explicit statements with formulations that rely on implicit knowledge. It uses three operators:

commonsense knowledge and is available to the Solver during interaction. Algorithm 2 summarizes the Knowledge Injection process. For each rewritten requirement fragment produced by Knowledge Injection, the algorithm records its knowledge type, the corresponding source fact, the rewritten text, and any queryable knowledge entry. It then validates the rewritten fragment by checking whether it preserves the scenario characteristics, remains consistent with the gold PL/SQL code, avoids schema-identifier leakage, provides sufficient ledger coverage, and can be traced back to the source fact. If a rewritten fragment fails any validation check, a new rewrite is generated within the fixed attempt budget. 𝐾 after Requirement Perturbation applies two operators to 𝑞𝑖,𝑗 Knowledge Injection.

• Domain Knowledge replaces an expanded definition or formula with a domain term. • Logical Reasoning replaces procedural steps with an invariant or constraint. • Commonsense Knowledge removes information that users infer from common conventions in the given scenario. All three operators preserve the scenario contract and the reference PL/SQL code. For all three operators, we extract the knowledge needed to interpret each rewrite and store it in the online knowledge ledger Λ𝐾𝑗 . Each entry contains a domain definition, reasoning logic, or

• Information Omission removes necessary but recoverable information from the visible requirement. 5

Table 1: Scenario-specific adapters. A1–A5 cover Synthesis and B1–B4 cover Editing. All adapters preserve the database description, reference code, call suite, and initial database state. Each constructed user input is written as requirement; attachment. The last column lists the checks applied after the three common checks. Subscenario

Applicable seed task

Constructed user input

Additional acceptance criteria

A1: From-Scratch

Any validated seed task.

Complete behavioral requirement; no attachment.

❶ The task has no attachment. ❷ The requirement fully specifies the target behavior.

A2: Document-Grounded

The target behavior depends on facts that can form a separate document.

Task request without the document facts; document stating them.

❶ Some implementation that ignores a document fact differs from the reference under the call suite. ❷ The document exposes no code.

A3: Example-Guided

The call suite contains inputs that illustrate the target behavior.

Task request; calls selected from the call suite with their observed behavior.

❶ The reference code reproduces every example. ❷ A leakage check rejects PL/SQL implementation text.

A4: Template Instantiation

The target admits a reusable skeleton and task-specific bindings.

Task-specific bindings; reusable skeleton with unfilled placeholders.

❶ A placeholder-coverage check matches every binding to the template. ❷ The check rejects a target-specific body.

A5: Code Composition

Existing callable components can implement the target when coordinated.

Composition request; callable PL/SQL components without the coordination code.

❶ The components compile and are callable. ❷ A structural check confirms the target coordination code is absent.

B1: Functional Modification

Reference code has a business rule that functional reversion can invert.

Original rule as the requested change; reference code with that rule reverted.

❶ At least one call distinguishes the unchanged code from the reference behavior.

B2: Performance Optimization

A behavior-preserving slowdown operator applies to the target.

Optimization request; behavior-preserving slowdown of the reference code.

❶ The slower code matches the reference outputs and state. ❷ Its warm median runtime exceeds the reference by the contract margin.

B3: Code Repair

A defect operator applies to the working code.

Repair request; reference code with an injected defect.

❶ At least one call exposes the defect: an exception, or a mismatched output or persistent state.

B4: Code Completion

The reference code contains a coherent removable block.

Completion request; reference code with a coherent block removed and the site marked.

❶ A structural check validates the removed block. ❷ Compilation or execution confirms the incomplete code fails the target behavior.

• Contradiction Injection introduces conflicts within the requirement or between the requirement and the database environment.

requirement-environment contradictions, we additionally require that the mismatch can be detected from the database environment. 3.3.2 Task Materialization and Interaction Protocol. Task Materialization. After Knowledge Injection and Requirement Perturbation, we package the resulting requirement and its supporting assets into an Interactive task:

Information Omission removes an explicit but recoverable requirement detail from the visible requirement. We regard the removed detail as necessary if its omission permits an implementation that satisfies the remaining requirement but behaves differently from the reference PL/SQL code when both are executed on the same call suite and initial database state. For each omitted detail, the requirement ledger records the original statement, where it appears in the source requirement, who can provide the missing information, and how the detail can be restored during interaction. Contradiction Injection introduces conflicting information that cannot be resolved from the visible requirement alone. We consider two forms of contradiction. An Intra-requirement contradiction introduces two incompatible statements into the same requirement, such as different thresholds. Because both statements are visible, the Solver can detect the conflict directly from the requirement, but must interact with the user to determine which statement the user actually intends. A Requirement-environment contradiction introduces a statement that conflicts with the actual database schema or metadata. For example, the requirement may refer to a nonexistent table or column, or specify an incorrect data type or constraint. The Solver can use environment tools to detect such mismatches. Environment tools can detect the mismatch, but cannot determine the user’s intended correction. The conflict therefore requires interaction with the user for resolution. Both perturbation operators modify only the visible requirement while preserving the database schema, metadata, gold PL/SQL code, call suite, and initial database state. Each injected conflict has a unique resolution that is consistent with the gold PL/SQL code. For

𝜏𝑖,𝑗 = ⟨𝑞𝑖,𝑗 , D, A 𝑗 , 𝜎0, 𝑃, 𝐶, Λ𝑖,𝑗 ⟩.

(12)

The Solver receives only the requirement, database description, and task attachment. The router assigns each ledger entry to its designated information holder. The pipeline retains only tasks whose ledger entries identify the corresponding construction operators. Interaction Protocol. The User Simulator represents the user in each interactive episode and responds to the Solver’s questions and clarification requests. Since the interaction proceeds over multiple turns in free-form natural language, we implement the simulator with a held-out LLM. The Solver–User Simulator protocol governs how the Solver interacts with the simulator. It defines the actions available to the Solver, the task information that the simulator may access for each action, and the responses that the simulator may provide. By restricting the simulator to user-facing information, the protocol prevents hidden evaluation assets from leaking into the interaction. Within the simulator, the requirement holder and knowledge holder handle requests that match entries in their corresponding ledgers. During task construction, we create an NL description of the gold PL/SQL behavior for the fallback channel, which answers questions without a matching ledger entry. Separately, environment tools expose the database environment defined in Section 2.2, allowing the Solver to inspect task-relevant database information through the prescribed tool interface. 6

Table 2: Nine actions available to the Solver in an Interactive episode. Every issued action consumes budget units, even if malformed or failed. Action

Env.

ask_user ask_selector

User question User options

Arguments

Natural-language answer Index of the intended option

Returned observation

2 2

get_schema describe_table sample_rows compile_plsql execute_scratch reset_scratch

DB DB DB DB DB DB

– table table, limit code sql –

Schema of the whole database Metadata of one table Sample rows of one table Compiler diagnostics Execution result Scratch context restored

1 0.5 0.5 0.5 1 0.5

submit

Eval

code

Table 3: ProcArena and prior NL-to-PL/SQL datasets. Tasks reports all evaluation instances, and Interactive reports the Interactive subset. Stmt and Cyclo report the mean statement count and cyclomatic complexity per reference procedure.

Cost

Scale & coverage

Action Space. The router sends each action to the component selected by the Solver, and the offline evaluator remains outside the interaction. The Solver is the only action initiator in the protocol. It may inspect the environment, request information through ask_user, ask the user to select among candidate interpretations through ask_selector, or end the episode through submit. Table 2 summarizes the nine concrete actions, their target components, inputs, returned observations, and interaction costs. The two userfacing actions return a natural-language answer and the index of the intended option, respectively. The analysis groups malformed or unrecognized names under malformed/other; this category falls outside the action space. The task attachment A remains visible throughout the episode. Every environment action and every ask_user or ask_selector action consumes the interaction budget; submit is cost-free but terminates the episode immediately and returns no execution-accuracy feedback, preventing the Solver from iteratively refining its solution against the hidden oracle. The protocol provides two user-facing actions. ask_user routes the question to the holder responsible for the matched ledger entry, falling back to a default channel when no entry matches. ask_selector lets the Solver choose a holder and supply candidate interpretations; the holder returns the index consistent with its ledger. When candidates mix requirement facts with injected knowledge, the Solver may query the two holders separately to resolve each source independently. Resolving contradictions between the requirement and the database requires two complementary information sources. The Solver first calls a schema or metadata action to identify the objective mismatch, such as a referenced column that does not exist, then uses ask_user or ask_selector to obtain the user’s intended resolution. Environment actions expose database facts only and never infer user intent, ensuring a clean separation between objective state and subjective preference.

Per-task difficulty

Dataset

Tasks

DBs

Dialects

Scen.

Interactive

Stmt

Cyclo

PLForge Simple PLForge Hard ProcBench WeBridge

300 300 300 209

130 113 123 40

1 1 1 1

0 0 0 0

2.8 10.7 3.3 29.5

3.0 6.8 3.5 9.4

ProcArena

3,998

157

1,879

32.1

18.6

summarizes its scale, coverage, and difficulty; Table 4 breaks it down by subscenario; and Table 5 quantifies the transformations that construct Interactive requirements.

4.1

Comparison with Prior Datasets

Table 3 compares ProcArena with four prior NL-to-PL/SQL datasets: PLForge Simple, PLForge Hard, ProcBench, and WeBridge [16, 50]. ProcArena expands both scale and coverage. The prior test splits contain 209–300 Direct tasks, one dialect, one scenario, and no Interactive episodes. ProcArena contains 3,998 tasks across two dialects and nine subscenarios, including 1,879 Interactive episodes. ProcArena procedures also contain more statements and have higher cyclomatic complexity on average. They average 32.1 statements and 18.6 cyclomatic complexity, compared with 10.7 statements and 6.8 cyclomatic complexity for PLForge Hard and 29.5 and 9.4 for WeBridge. ProcArena therefore covers more settings and has higher average procedure complexity.

4.2

Corpus Composition

Table 4 breaks down the corpus by subscenario. Direct task counts range from 111 to 120 per dialect. ProcArena contains 1,060 PostgreSQL and 1,059 Oracle Direct tasks. Each Direct task has a paired Interactive episode except B2, which has no Interactive task because it uses a runtime score after semantic-equivalence checking. This pairing yields 940 PostgreSQL and 939 Oracle Interactive episodes. The corpus spans 157 databases; B3 uses 110, while A1 and B2 each use 76. The Input rows report the mean size of the requirement and task attachment. NL tokens include the requirement and any textual attachment, such as the A2 design document and A3 behavioral examples. Code tokens count PL/SQL code in the attachment. A1– A3 have no code attachment, so their Code-token cells are shown as dashes. The Reference procedure rows summarize each target procedure and its call suite: statements, lines, parameters, calls, and tables written. Across the corpus, reference procedures average 39.0 lines, 4.0 parameters, 6.3 calls, and 4.9 tables written.

PROCARENA STATISTICS

ProcArena spans nine Direct subscenarios and eight paired Interactive subscenarios across PostgreSQL and Oracle. This breadth follows from the task itself: NL-to-PL/SQL development spans synthesis and editing, complete and underspecified requirements, and two dialects. Varying the scenario, the mode, and the dialect therefore keeps evaluation grounded in the full task rather than in one narrow setting. This section reports the resulting corpus: Table 3

4.3

Interactive Requirement Transformations

Table 5 reports six transformation types for Interactive requirement construction. Section 3.3.1 groups them into Knowledge, Contradiction, and Omission. Knowledge replaces explicit facts with domain 7

Table 4: ProcArena corpus by subscenario. All gives the per-subscenario mean for task counts, the corpus-wide distinct count for databases, and the per-applicabletask mean for other metrics. A dash denotes a missing Interactive task or a subscenario without a code attachment. A. Synthesis

through 5.5 report what the models achieve and how they fail. Section 5.6 then checks that the task construction and the interactive environment earn their place.

5.1

Evaluation Modes. ProcArena presents all nine subscenarios in the Direct mode and eight of them in the Interactive mode. B2 Performance Optimization is Direct-only, because its score is execution time rather than state equality. In the Direct mode the Solver receives the complete requirement and submits one procedure. In the Interactive mode it receives an underspecified requirement. It may ask the User Simulator for a fact with ask_user, confirm a candidate reading with ask_selector, and inspect the database with six environment tools before it submits. Each paired task shares its database description, task attachment, gold procedure, call suite, and initial state.

B. Editing

Metric

All

Scale & coverage Direct, PG Direct, Oracle Interactive, PG Interactive, Oracle Databases

114 115 114 115 76

119 116 119 116 85

118 120 118 120 85

119 116 119 116 87

111 118 111 118 88

119 118 119 118 90

120 120 – – 76

120 120 120 120 110

120 116 120 116 107

117.8 117.7 117.5 117.4 157

Input (per task, avg) NL tokens 609.5 Code tokens –

898.5 –

764.3 –

1125.6 519.4

496.4 334.3

385.5 446.5

149.3 429.1

732.2 410.2

330.9 244.7

609.6 402.3

Reference procedure (per task, avg) Statements 36.9 31.2 Lines 39.8 45.4 Parameters 4.0 4.0 Calls 𝑘 6.4 6.1 Tables written 5.1 4.8

43.6 54.4 4.1 6.4 5.8

40.7 39.2 3.6 6.4 5.9

22.7 45.1 4.9 6.7 4.2

41.9 54.5 3.9 6.4 5.8

15.0 13.3 3.9 5.2 2.3

26.7 32.7 3.9 6.3 4.6

30.5 26.7 4.1 6.5 5.2

32.1 39.0 4.0 6.3 4.9

Evaluation Metrics. Execution accuracy (EX) is the metric for the eight paired subscenarios in both modes. We adapt PLForge’s execution-match principle [50] to the stateful behavior of procedural code. Exec(𝑃, 𝐶, 𝜎0 ) installs 𝑃 from the initial state 𝜎0 , runs the ordered call suite 𝐶, and returns its outputs together with the ˆ the task-level execution final persistent state. For a candidate 𝑃, indicator is ˆ 𝑃, 𝐶, 𝜎0 ) = 1 Exec(𝑃, ˆ 𝐶, 𝜎0 ) = Exec(𝑃, 𝐶, 𝜎0 ) . EX(𝑃; (13)

Table 5: Interactive requirement transformations per task and subscenario. B2 has no Interactive task. Avg reports the mean across Interactive tasks. A. Synthesis

B. Editing

Fact kind

Avg

Knowledge Jargon (terminology) Reasoning Commonsense

1.89 0.84 0.84

3.01 0.78 0.79

2.05 0.86 0.82

1.00 0.43 0.37

1.56 0.76 0.74

1.81 0.84 0.79

0.00 0.00 0.00

1.07 0.85 0.83

1.54 0.67 0.65

Contradiction Requirement Environment

0.68 0.32

0.69 0.31

0.68 0.32

0.67 0.33

0.68 0.32

0.69 0.31

0.68 0.33

0.68 0.32

Omission Omitted facts

2.00

2.60

2.00

1.99

1.58

1.94

2.00

1.00

1.89

Transformed facts

6.56

8.19

6.73

4.78

5.64

6.38

3.00

4.75

5.75

Execution accuracy averages this indicator over tasks. B2 is scored differently, because its input is a degraded version of the gold procedure and already satisfies Equation 13. Semantic equivalence is therefore a gate rather than a score. An admitted candidate is measured against the gold procedure by two ratios, acceleration 𝑇𝑃 /𝑇𝑃ˆ and work 𝑊𝑃 /𝑊𝑃ˆ , where 𝑇 is a median wall-clock time and 𝑊 a deterministic work counter. Both are oriented so that 1.0 means the candidate matched the gold procedure and larger is better. Section 5.3 gives the measurement protocol. For the Interactive mode we additionally report the number of interaction turns and the total tokens per episode. Retention is defined in Section 5.4, where it is first used.

terminology, reasoning constraints, or commonsense conventions. Domain definitions enter the elicitation ledger, while reasoning and commonsense rewrites remain in the offline trace. Contradiction introduces incompatible readings within the requirement or between the requirement and the environment. Information Omission removes a registered fact from the visible requirement and stores it in the elicitation ledger. ProcArena applies 5.75 transformations per Interactive task on average. A4 has fewer reasoning and commonsense rewrites than the other non-B3 subscenarios, while B3 has zero Knowledge transformations. The construction pipeline changes only the requirement, so a subscenario can retain transformations even when its attachment contains much of the target code.

Experimental Setup

Baseline Models. We evaluate seven language models: GPT-5.5, GPT-5.4 Mini, and GPT-5.2 [36], Gemini-3.1 Pro [12], GLM-5.3 and GLM-5.2 [11], and DeepSeek V4 Pro [7]. Implementation Details. Every task runs on a freshly restored database, so no episode observes the writes of another. ProcArena ships tasks in two dialects, PL/pgSQL on PostgreSQL 12.22 and Oracle PL/SQL on Oracle Database 21c Express Edition. A single fixed model, DeepSeek V4 Flash, drives the User Simulator across every Solver, so an EX difference between two Solvers never reflects a difference in the simulator. We set reasoning effort to high for every model that exposes it, and we run each task once. The Solver may take at most 60 recorded steps. Its tool budget is 6 + 2|Λ| + 3 units, where Λ is the task’s elicitation ledger. ask_user and ask_selector cost two units each, environment tools cost between 0.5 and one unit, and submit is free. Both the budget and the price list are shown to the Solver, and every tool result reports the units spent so far.

EXPERIMENTS

We evaluate seven language models on ProcArena, across nine Direct subscenarios, eight paired Interactive subscenarios, and two SQL dialects. The strongest model reaches 62.2% execution accuracy in the Direct mode and 57.8% in the Interactive mode. Sections 5.2 8

Table 6: Interaction lowers accuracy on Synthesis and raises it on Editing. Execution accuracy (%) per subscenario and dialect, with Direct above the shaded band and Interactive below it. Subscenarios run along each category’s ordering axis of Section 3.2. Within each mode block, the best value per column is in bold and the second best is underlined. PostgreSQL A. Synthesis Model Direct DeepSeek V4 Pro GLM-5.2 GLM-5.3 GPT-5.4 Mini GPT-5.2 GPT-5.5 Gemini-3.1 Pro Interactive DeepSeek V4 Pro GLM-5.2 GLM-5.3 GPT-5.4 Mini GPT-5.2 GPT-5.5 Gemini-3.1 Pro

5.2

Oracle B. Editing

Overall

A. Synthesis

B. Editing

Overall

39.5 44.7 62.3 57.9 64.9 68.4 67.5

36.1 45.4 52.1 52.9 52.9 58.0 58.8

27.1 33.9 50.0 50.0 44.1 55.1 55.1

44.5 51.3 59.7 46.2 60.5 70.6 74.8

33.3 37.8 57.7 51.4 58.6 64.0 59.5

40.3 49.6 61.3 43.7 47.1 58.8 64.7

19.2 31.7 29.2 41.7 38.3 63.3 54.2

41.7 53.3 65.8 60.0 65.0 69.2 69.2

35.2 43.5 54.7 50.4 53.8 63.4 63.0

46.1 47.0 52.2 63.5 62.6 64.3 64.3

34.5 40.5 47.4 55.2 55.2 55.2 55.2

41.7 48.3 52.5 65.8 63.3 66.7 64.2

44.8 50.9 49.1 56.0 60.3 60.3 62.1

30.5 28.0 28.0 33.9 34.7 36.4 36.4

44.1 45.8 52.5 44.9 52.5 59.3 62.7

15.0 32.5 46.7 45.0 49.2 68.3 80.8

50.9 51.7 54.3 59.5 62.9 67.2 65.5

38.3 43.0 47.8 52.9 55.1 59.7 61.4

5.3 25.4 53.5 30.7 47.4 37.7 39.5

10.9 51.3 73.9 50.4 68.1 52.1 61.3

5.1 21.2 49.2 46.6 55.1 38.1 31.4

10.1 14.3 26.1 21.0 31.1 49.6 43.7

12.6 25.2 31.5 18.0 30.6 44.1 44.1

28.6 47.9 60.5 42.9 58.0 71.4 79.8

29.2 52.5 55.0 54.2 60.8 71.7 74.2

33.3 42.5 53.3 52.5 60.8 78.3 75.0

17.0 35.2 50.5 39.8 51.7 55.6 56.4

29.6 33.0 51.3 42.6 41.7 57.4 51.3

52.6 51.7 72.4 54.3 69.0 79.3 75.0

28.3 27.5 49.2 46.7 46.7 53.3 54.2

35.3 37.1 45.7 40.5 49.1 57.8 52.6

19.5 16.9 23.7 22.0 22.0 29.7 28.0

29.7 33.1 53.4 37.3 42.4 54.2 63.6

7.5 33.3 52.5 45.0 44.2 70.8 76.7

42.2 41.4 53.4 52.6 62.1 64.7 71.6

30.5 34.2 50.2 42.6 47.1 58.4 59.1

Overall Results

reproduce a reported defect. Both gain from the database tools the Interactive mode provides, and here that gain outweighs the missing requirement facts. A Synthesis task offers no such anchor, so it absorbs the full cost of an underspecified requirement.

Table 6 reports EX for every model on every subscenario, in both dialects. The Direct block sits above the shaded band and the Interactive block below it. The two blocks use the same paired tasks and the same row order, so a reader compares them cell by cell. B2 is not in this table; Section 5.3 reports it. Gemini-3.1 Pro is the strongest model in both modes, at 62.2% and 57.8% pooled over the two dialects. GPT-5.5 follows at 61.6% and 57.0%. DeepSeek V4 Pro is last in both, at 36.8% and 23.7%. Even in the Direct mode, where the requirement is complete and no interaction is needed, the best model is wrong on more than a third of the tasks. The two dialects behave alike, and their differences are small and not consistently signed. In the Direct mode Gemini-3.1 Pro scores 63.0% on PostgreSQL and 61.4% on Oracle, while GPT5.4 Mini scores 50.4% and 52.9%. A5 Orchestration is the hardest subscenario for every model. No model exceeds 49.8% on it in the Direct mode or 36.7% in the Interactive mode. A5 asks the Solver to compose existing procedures into one transactional workflow while respecting their signatures. The two categories respond to interaction differently. In the Direct mode Synthesis and Editing are close: 59.8% against 66.2% for Gemini-3.1 Pro, and 59.9% against 64.4% for GPT-5.5. In the Interactive mode they separate, to 48.1% against 73.5% for Gemini3.1 Pro. Two subscenarios even score higher in the Interactive mode. A2 rises from 57.0% to 68.1% for Gemini-3.1 Pro and from 49.8% to 73.2% for GLM-5.3, and B3 rises from 67.5% to 75.5% for Gemini-3.1 Pro. Two properties of the tasks account for this. An Editing task hands the Solver the procedure it must change, and that attachment already fixes most of what requirement perturbation removed, so less has to be recovered by asking. A2 and B3 are the two subscenarios whose difficulty is grounding rather than specification: A2 must map a design document onto the real schema, and B3 must

5.3

Performance Optimization

Figure 4 reports B2, the one subscenario that execution-state equivalence cannot rank; the figure labels the gold procedure as the reference. B2 hands the Solver a degraded version of the gold procedure and asks for a faster one that behaves the same way. The degradation applies real inefficiency patterns: a set-based UPDATE becomes a cursor loop with one statement per row, a join becomes an N+1 lookup, and an aggregate is recomputed inside a loop. The Solver is told that the procedure is correct but slow, and is not told where the inefficiency is. Seven models answer 120 tasks in each dialect, which gives 1,680 measurements. Acceleration and work are defined in Section 5.1; what remains is how they are measured. Every table is scaled to the row count recorded with the task, inside a transaction that is rolled back afterwards. Each procedure is then timed five times, with the gold procedure, the degraded input and the candidate taking turns in rotating order, so that cache warmth favors none of them. The median of the five runs is reported. We take the ratio between a procedure’s own slowest and fastest run as the measurement noise, whose median is 1.66–2.05 on PostgreSQL and 1.42–1.48 on Oracle. Results are aggregated with the harmonic mean, which penalizes a regression instead of letting one large win hide it. Panel (a) bins the tasks by speedup. Every model matches or beats the gold procedure on most tasks in both dialects. GPT-5.4 Mini places 36% of its PostgreSQL tasks in the 2–10× band. GLM-5.2 has the largest slower block, 14% on PostgreSQL and 17% on Oracle. Panel (b) reports the two harmonic means. Five models sit just above the gold procedure on both. GPT-5.4 Mini has the highest 9

(a) Distribution of speedup over the reference

(b) Acceleration and work, harmonic means

Figure 4: Most candidates match or beat the gold procedure, but two models buy their speed with far more database work. Each model contributes one bar or marker pair per dialect, PostgreSQL then Oracle. Panel (a) stacks the faster speedup bins above the axis and the slower ones below. Panel (b) uses a log axis on which 1.0 matches the gold procedure, and its connector spans the acceleration and work means. In panel (c), a zero marks a model with no rejected task.

5.4

80 60

-V4 -5.2 .4M -5.2 -5.3 -5.5 Pro DS GLM PT-5 GPTGLM GPTni 3.1 G mi Ge

(a) EX and retention

Mean tokens

10 0 -V4 -5.2 .4M -5.2 -5.3 -5.5 Pro DS GLM PT-5 GPT GLM GPTni 3.1 G mi Ge

Mean tokens (k)

EX (%)

60 40 20 0

Interactive EX Retention Mean turns 100 20

Retention (%) Mean turns

Direct EX

acceleration, 1.51, but a work mean of 0.18. GLM-5.2 reaches 0.02 on work, and is the only model whose acceleration falls below the gold procedure, at 0.72. Panel (c) counts the answers the semantic gate rejects, split by compile failure, wrong final state, and runtime error. GPT-5.5, Gemini-3.1 Pro and GPT-5.2 have none in either dialect. GLM-5.2’s rejections are mostly a wrong final state, and DeepSeek V4 Pro’s are mostly compile errors, seven of its eight on Oracle. Only three of the 1,680 answers raised a runtime error. Panel (a) counts tasks and panel (b) weighs them, which is why they disagree. GLM-5.2’s two worst PostgreSQL answers perform 4,445× and 1,058× the gold procedure’s operations. GPT-5.4 Mini’s worst performs 516×, and its measured speedup is 0.97, close enough to the gold procedure to disappear into the matched band of panel (a). An answer can therefore look correct on the clock while doing hundreds of times the database work. On a scaled test table those extra operations cost little wall-clock time; on production volumes they would not. The same two models also account for most of the rejected answers, and GLM-5.2’s are changes of behavior rather than compile failures. We read this as code that is locally plausible without a model of what the database will actually do, which is why B2 reports work beside time.

100 50

(b) Interaction cost

Figure 5: Interaction cost does not predict retention. Panel (a): Direct and Interactive EX as bars, their ratio as a line. Panel (b): mean turns as bars, mean tokens per episode as a line.

Retention is Interactive EX divided by Direct EX. It measures how much of a model’s own generation ability survives requirement resolution. Raw Interactive EX cannot measure this, because a model with weak generation and strong elicitation scores like a model with the opposite profile. GLM-5.3 retains the most, 98.2%, and DeepSeek V4 Pro the least, 64.5%. The other five fall between 79.4% and 92.6%. Cost does not follow. GPT-5.5 spends the fewest tokens, 38.5k per episode, and the second fewest turns, 7.1, yet retains 92.6%. GLM-5.2 spends the most of both, 123.8k tokens over 19.8 turns, and retains 80.2%. DeepSeek V4 Pro spends the fewest turns, 6.9, and retains the least. Figure 6 shows what the turns are spent on, and the models split into two strategies. GLM-5.3 spends 54.5% of its actions on execute_scratch, GLM-5.2 40.0%, and Gemini-3.1 Pro 39.1%: these models iterate against a private copy of the database. GPT5.5 spends 7.2% there, and 18.0% on sample_rows and 18.1% on compile_plsql instead: it reads the data and compiles the code

Interaction Analysis

This subsection asks how much accuracy survives requirement resolution, what that costs, and what the cost is spent on. Figure 5 plots Direct and Interactive EX as bars with their ratio as a line, and mean turns as bars with mean total tokens as a line, where tokens are the input plus output usage reported for an episode. Figure 6 gives each model’s share of recorded actions over the nine tools, plus one bucket for any action name outside the protocol. Figure 7 gives action counts by recorded step for four models spanning the Interactive EX range. It plots counts rather than shares, so the outline of each stack is that model’s episode attrition. All three pool the paired PostgreSQL and Oracle tasks. 10

get_schema describe_table

1500 1000 500 0

Action count

Action share (%)

Gemini 3.1 Pro 9.30 2.13 2.97 9.82 39.05 1.11 14.40 12.02 9.20 0.00 GPT-5.5 13.54 5.32 17.97 18.14 7.22 0.08 15.08 8.36 14.30 0.00 GLM-5.3 6.60 1.32 9.67 5.58 54.45 3.26 7.95 4.49 6.63 0.07 GPT-5.2 6.26 9.25 17.42 14.35 33.31 4.10 6.80 2.43 6.08 0.00 GPT-5.4M 9.68 5.44 22.14 20.56 13.76 0.56 12.40 4.83 10.63 0.00 GLM-5.2 4.99 11.45 14.51 6.97 40.04 3.08 6.46 7.13 5.08 0.29 DS-V4-Pro 9.99 12.06 3.59 15.54 18.23 2.48 19.19 2.51 16.01 0.39 a le s ql ch ch er or it er hem e_tab le_row le_pls scrat scrat sk_us select subm ed/oth c s b i _ _ a sk_ _ rm get descri samp comp ecute reset a alfo ex m Trace tool

20 10 0

execute_scratch reset_scratch

(a) Gemini 3.1 Pro

Recorded tool step

ask_user ask_selector

submit malformed/other

(b) GLM-5.3

(d) DS-V4-Pro

Figure 7: Models follow visibly different interaction trajectories. Action counts by recorded step, two models per row with shared axes. The bands use the tool names of Figure 6.

Figure 6: Models split between iterating against the database and inspecting it. Share of all recorded actions per model. Tool names are verbatim from the traces; names outside the protocol are grouped as malformed/other.

loop and return structure between the candidate and the gold procedure separates Control flow from State effect. This comparison is used only in post-hoc analysis and is never shown to the Solver. Control flow is the largest category for six of the seven models, between 17.6% and 31.2%. State effect is second, between 7.9% and 15.4%. Exception is negligible everywhere, at most 2.9%. GLM-5.2 has the highest rate in both dominant categories, 31.2% and 15.4%. Gemini-3.1 Pro and GPT-5.5 have the lowest rates in almost every category. DeepSeek V4 Pro has a different profile: its Schema rate is 20.1% and its Contract rate is 27.9%, several times any other model’s, while its control-flow rate is the lowest at 17.6%. Models rarely crash. They compile, they run, and they compute the wrong thing. Two of the five categories have no analogue in declarative SQL, because a query has no branch or loop to get wrong and no persistent write to get wrong. Together they account for between 25.5% and 46.6% of the tasks each model ran. DeepSeek V4 Pro is the one model that fails before the logic rather than in it. Its wrong tables and wrong signatures are consistent with its low retention in Section 5.4, and with the fact that it asks many questions but confirms few readings.

rather than running it. The two channels to the User Simulator split the models the same way. DeepSeek V4 Pro asks the most, 19.2% of its actions on ask_user, and confirms the least, 2.5% on ask_selector. Gemini-3.1 Pro inverts this, at 12.0% on ask_selector, the highest in the figure. Action names outside the protocol never exceed 0.4%, so tool-call formatting affects none of these numbers. Figure 7 places the same behavior in time. Gemini-3.1 Pro concentrates ask_user in steps two to five and then stops asking. GLM-5.3’s execute_scratch band runs to step 25, and its episodes are the longest in the figure. DeepSeek V4 Pro’s episodes are nearly over by step ten. Both strategies work. GLM-5.3 iterates against the database and GPT-5.5 barely does, yet they are the two models that retain the most, at 98.2% and 92.6%, and GPT-5.5 reaches its figure on less than half the tokens. Neither the volume of interaction nor the volume of questions predicts retention. What fails is the absence of a strategy rather than the choice between them: DeepSeek V4 Pro takes the fewest turns, spends the largest share of them asking, confirms the fewest readings, and retains the least. Retention is also unstable when the denominator is small, so GLM-5.3’s 98.2% should be read against its Direct EX of 51.3%.

5.5

sample_rows compile_plsql

5.6

Benchmark Validation

To verify the reliability of the User Simulator, we conduct a human audit of its ask_user replies (Figure 9). We sample 180 turns stratified over solver models and sub-scenarios, one third from the LOC pool; auditors blind to model identity judged each reply against the task’s oracle evidence for acceptability, failure reason, and answer leakage. Overall, 85.6% of replies are acceptable and the residual failures scatter over six categories with no dominant mode (a); LOC-pool replies pass at a similar rate, with only 3.3% leaking answer content (b). The simulator thus provides reliable feedback with negligible answer leakage.

Error Analysis

Figure 8 reports Interactive failures by type, as a percentage of the tasks each model ran, so a cell reads directly against the Interactive EX columns of Table 6. Schema is a wrong table or column, Control flow a wrong branch or loop, State effect a wrong persistent write, Exception a remaining runtime call failure, and Contract an installation or interface mismatch. We assign these types after evaluation with deterministic rules. Schema and Contract come from installation, interface, argument, signature, component-use, and database diagnostics, and a remaining runtime call failure becomes Exception. For a state mismatch, a lexical comparison of branch,

RELATED WORK

NL-to-PL/SQL. To the best of our knowledge, PLForge [50] represents the first study specifically targeting NL-to-PL/SQL, achieving 11

5.7 7.1 0.3 1.4 4.4 3.5 20.1

18.9 18.3 22.4 25.1 26.7 31.2 17.6

10.1 10.1 13.7 11.6 14.6 15.4 7.9

1.1 0.7 1.6 0.7 1.2 2.9 2.8

6.4 6.9 11.7 11.8 11.9 12.3 27.9

ma fect low tion ontract Sche ontrol f State ef Excep C C

25 20

Failures / tasks (%)

Gemini 3.1 Pro GPT-5.5 GLM-5.3 GPT-5.2 GPT-5.4M GLM-5.2 DS-V4-Pro

Wrong fact

Unsupported assertion

Irrelevant answer

Incomplete answer

Wrong boundary

Other

Pass 154 (85.6%)

10 5

2.8% 1.7% 2.8% 3.3% 2.8% 1.1%

Outcome (n = 61)

Fail 13.1%

Pass 86.9%

Failure reasons (n = 8) 12.5% 12.5%

20%

Wrong boundary 50.0%

40%

60%

12.5% 12.5%

80%

100%

(a) Outcome and failure breakdown (b) LOC audit summary

Classified failure type

Figure 9: Human audit (a) Pass rate and failure reasons over all audited ask_user replies. (b) The same audit restricted to the sampled LOC pool.

Figure 8: Control flow and persistent state dominate, and neither exists in declarative SQL. Failures by error type in the Interactive mode, as a percentage of the tasks each model ran. Table 7: Every construction operation lowers EX on its own, and the full pipeline lowers it most. Base is EX (%); indented rows are paired percentage-point changes against the same tasks. Condition

GPT-5.5

Gemini-3.1 Pro

GPT-5.2

GLM-5.2

Direct task + Knowledge injection + Contradiction injection + Requirement omission Direct task + all three methods

86.67 -16.27 -10.92 -10.82 -23.35

82.92 -9.09 -13.87 -3.90 -17.18

72.92 -13.40 -10.92 -7.79 -22.91

73.33 -25.36 -15.13 -25.54 -39.21

controllability, leveraging domain-oriented training [23, 25, 37, 46] and structural constraints [42, 51]. Agentic workflows treat synthesis as an iterative process involving planning and feedback, exemplified by MCTS-based Alpha-SQL [22], multi-agent collaboration [47], and user-in-the-loop clarification [34, 52]. Although BIRD-INTERACT [18] introduces dynamic and stateful multi-turn evaluation for Text-to-SQL, it remains focused on SQL statements and does not evaluate dialogue-based alignment and revision of procedural PL/SQL specifications. Code Language Models. Existing code language models [17, 20, 30, 43] rely on massive polyglot training data to achieve broad coverage. However, this general-purpose objective often leads to suboptimal performance on specialized, low-resource languages such as PL/SQL. The scarcity of high-quality PL/SQL data in public repositories remains a major obstacle to PL/SQL-specific model adaptation. Rather than proposing a new model-specialization method, ProcArena complements existing model-centric research with a modelagnostic, executable benchmark for evaluating procedural database programming across multiple development and interaction settings.

SOTA performance through continued pre-training on a PL/SQLcentric corpus. However, PLForge relies on a template-based method for dataset generation, which may restrict syntactic diversity and limit its ability to reflect the complex distribution of real-world workloads. Although real-world procedural workloads also contain UDFs and triggers [14], both PLForge and the current version of ProcArena focus on SPs. ProcArena instead broadens evaluation along a different dimension by covering multiple SP development scenarios in both direct and interactive settings. Text-to-SQL. Recent progress on Text-to-SQL is increasingly driven by language models, involving two main directions: supervised PLM-based encoder-decoder parsers and LLM-based approaches utilizing prompting and workflows. PLM-based encoderdecoder parsers typically map serialized inputs to SQL. Key enhancements include injecting graph structures for schema-aware learning [4, 26], utilizing intermediate representations to decouple schema linking [10, 13, 24], and applying constrained decoding to ensure syntactic validity [44]. These methods establish reliable baselines for encoding and constraining schema information. LLM-based Textto-SQL methods have shifted Text-to-SQL toward prompt-centric and agentic paradigms [15, 29, 32], including in-context learning (ICL), fine-tuning (FT), and agentic workflows. ICL exploits prompting via decomposition and execution-guided refinement [15, 29, 39], improving schema grounding through retrieval and dynamic fewshot construction [28, 41, 48]. FT approaches adapt models for

CONCLUSION

We proposed ProcArena, to our knowledge the first benchmark to evaluate LLMs on NL-to-PL/SQL across multiple development scenarios in both Direct and Interactive modes: an executable suite of 3,998 tasks spanning nine Direct subscenarios and eight paired Interactive subscenarios in two dialects. Iterative Logic Enhancement, live validation, and scenario-specific adapters constructed verifiable Direct tasks, from which Knowledge Integration and Requirement Perturbation derived paired Interactive tasks whose executable target an isolated Offline Evaluator preserved. Across seven models, the strongest reached 62.2% execution accuracy in the Direct mode and 57.8% in the Interactive mode, interaction cost accuracy on Synthesis and recovered it on Editing, and failures concentrated in control flow and persistent state, the two error types that declarative SQL does not have. We released the benchmark, the interaction traces, and the construction pipeline for reproducible evaluation. 12

REFERENCES

[25] Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024. CodeS: Towards Building Open-source Language Models for Text-to-SQL. Proceedings of the ACM on Management of Data 2 (2024), 1 – 28. [26] Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. 2023. Graphix-T5: Mixing PreTrained Transformers with Graph-Aware Layers for Text-to-SQL Parsing. ArXiv abs/2301.07507 (2023). [27] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM already serve as a database interface? a big bench for large-scale database grounded text-to-SQLs. In Proceedings of the 37th International Conference on Neural Information Processing Systems. Red Hook, NY, USA, Article 1835, 28 pages. [28] Xiuwen Li, Qifeng Cai, Yang Shu, Chenjuan Guo, and Bin Yang. 2025. AID-SQL: Adaptive In-Context Learning of Text-to-SQL with Difficulty-Aware Instruction and Retrieval-Augmented Generation. 2025 IEEE 41st International Conference on Data Engineering (ICDE) (2025), 3945–3957. [29] Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yu xin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2024. A Survey of Text-to-SQL in the Era of LLMs: Where Are We, and Where Are We Going? IEEE Transactions on Knowledge and Data Engineering 37 (2024), 5735–5754. [30] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel LamyPoirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173 (2024). [31] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021). [32] Yuyu Luo, Guoliang Li, Ju Fan, Chengliang Chai, and Nan Tang. 2025. Natural Language to SQL: State of the Art and Open Problems. Proc. VLDB Endow. 18 (2025), 5466–5471. [33] Xutao Mao, Dixuan Zhang, Yifan Li, Jiaming Hou, Rui Li, YunLong Li, Aoze Zheng, Zhiqiang Zhang, Luo Zhewei, Hongying Zan, et al. 2026. LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 29958–29966. [34] Aritra Mazumder, Parth Desai, Fuheng Zhao, and Anna Fariha. 2026. AtomSQL: Interactive Disambiguation of NL-to-SQL via User-Guided Atom-Level Alignment. Proceedings of the VLDB Endowment 19 (2026). [35] Microsoft. n.d.. Azure SQL Database. https://azure.microsoft.com/en-us/services/ sql-database/. Accessed: 2026-01-16. [36] OpenAI. 2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/. Accessed: 2026-04-18. [37] Minjun Park, Yongju Seong, Myoseop Sim, Kyungkoo Min, and Stanley Jungkyu Choi. 2026. ReSQL: Self-Improving Framework for Reasoning-Aware Text-to-SQL Dataset Generation. In Findings of the Association for Computational Linguistics: ACL 2026. 33582–33602. [38] Andrew Pavlo. 2017. What Are We Doing With Our Lives? Nobody Cares About Our Concurrency Control Research. In Proceedings of the 2017 ACM International Conference on Management of Data. 3. https://doi.org/10.1145/3035918.3056096 [39] Mohammadreza Pourreza and Davood Rafiei. 2024. Din-sql: Decomposed incontext learning of text-to-sql with self-correction. Advances in Neural Information Processing Systems 36 (2024). [40] Nishat Raihan, Antonios Anastasopoulos, and Marcos Zampieri. 2025. mHumanEval-a multilingual benchmark to evaluate large language models for code generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 11432–11461. [41] Tonghui Ren, Yuankai Fan, Zhenying He, Ren Huang, Jiaqi Dai, Can Huang, Yinan Jing, Kai Zhang, Yifan Yang, and Xiaoyang Sean Wang. 2024. PURPLE: Making a Large Language Model a Better SQL Writer. 2024 IEEE 40th ICDE (2024), 15–28. [42] Tonghui Ren, Chen Ke, Yuankai Fan, Yinan Jing, Zhenying He, Kai Zhang, and Xiaoyang Sean Wang. 2025. The Power of Constraints in Natural Language to SQL Translation. Proc. VLDB Endow. 18 (2025), 2097–2111. [43] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). [44] Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 9895–9901. [45] Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2025. A Survey on Employing Large Language Models for Text-to-SQL Tasks. ACM Comput. Surv. 58, 2, Article 54 (Sept. 2025), 37 pages. https://doi.org/10.1145/ 3737873

[1] 2025. Oracle Help Center. https://docs.oracle.com/en/ [2] 2025. PostgreSQL: Documentation. https://www.postgresql.org/docs/ [3] Itzik Ben-Gan, Dejan Sarka, Roger Wolter, Greg Low, Ed Katibah, and Isaac Kunen. 2009. Inside Microsoft SQL Server 2008 T-SQL Programming. Microsoft Press. [4] Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. 2021. LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and NonLocal Relations. ArXiv abs/2106.01093 (2021). [5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [6] Vinicius Ferreira de Sousa, Cláudio de Souza Baptista, André Luiz Firmino Alves, and Hugo Feitosa de Figueirêdo. 2025. Effectiveness of small and large language models for PL/SQL bad smell detection. In Simpósio Brasileiro de Banco de Dados (SBBD). SBC, 399–412. [7] DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348 (2026). [8] Korry Douglas and Susan Douglas. 2003. PostgreSQL: a comprehensive guide to building, programming, and administering PostgresSQL databases. SAMS publishing. [9] Steven Feuerstein and Bill Pribyl. 2005. Oracle pl/sql Programming. " O’Reilly Media, Inc.". [10] Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John Robert Woodward, John Drake, and Qiaofu Zhang. 2021. Natural SQL: Making SQL Easier to Infer from Natural Language Specifications. In Conference on Empirical Methods in Natural Language Processing. [11] GLM-5 Team. 2026. GLM-5: from Vibe Coding to Agentic Engineering. arXiv preprint arXiv:2602.15763 (2026). [12] Google DeepMind. 2025. Gemini 3. https://deepmind.google/models/gemini/. Accessed: 2026-04-18. [13] Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and D. Zhang. 2019. Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation. ArXiv abs/1905.08205 (2019). [14] Surabhi Gupta and Karthik Ramachandra. 2021. Procedural Extensions of SQL: Understanding their usage in the wild. Proceedings of the VLDB Endowment 14, 8 (2021), 1378–1391. [15] Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2024. Next-Generation Database Interfaces: A Survey of LLM-Based Text-to-SQL. IEEE Transactions on Knowledge and Data Engineering 37 (2024), 7328–7345. [16] Gansen Hu, Zhaoguo Wang, Chuzhe Tang, Jiahuan Shen, Zhiyuan Dong, Sheng Yao, and Haibo Chen. 2024. WeBridge: Synthesizing Stored Procedures for LargeScale Real-World Web Applications. Proceedings of the ACM on Management of Data 2, 1 (2024), 1–29. [17] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024). [18] Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, et al. 2026. BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation via Lens of Dynamic Interactions. In International Conference on Learning Representations, Vol. 2026. 44285–44333. [19] Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, Vol. 2025. 58791–58831. [20] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515 (2024). [21] Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin SU, ZHAOQING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. In The Thirteenth International Conference on Learning Representations. [22] Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. 2025. Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search. In Forty-second International Conference on Machine Learning. [23] Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, and Cuiping Li. 2025. OmniSQL: Synthesizing High-Quality Text-to-SQL Data at Scale. Proc. VLDB Endow. 18, 11 (July 2025), 4695–4709. https://doi.org/10.14778/3749646. 3749723 [24] Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. In AAAI Conference on Artificial Intelligence.

Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 3911–3921. [50] Hang Zhang, Chaokun Wang, Hongwei Li, Cheng Wu, Songyao Wang, Yabin Liu, Gengyuan Shi, and Ziyang Liu. 2025. PLForge: Enhancing Language Models for Natural Language to Procedural Extensions of SQL. Proc. ACM Manag. Data 3, 6, Article 348 (Dec. 2025), 28 pages. https://doi.org/10.1145/3769813 [51] Qinggang Zhang, Hao Chen, Junnan Dong, Shengyuan Chen, Feiran Huang, and Xiao Huang. 2025. Structure-Guided Large Language Models for Text-to-SQL Generation. In Forty-second International Conference on Machine Learning. [52] Fuheng Zhao, Shaleen Deep, Fotis Psallidas, Avrilia Floratou, Divyakant Agrawal, and A. El Abbadi. 2024. Sphinteract: Resolving Ambiguities in NL2SQL Through User Interaction. Proc. VLDB Endow. 18 (2024), 1145–1158.

[46] Marko Sterbentz, Kevin Cushing, Cameron Barrie, and Kristian J Hammond. 2026. RingSQL: Generating Synthetic Data with Schema-Independent Templates for Text-to-SQL Reasoning Models. arXiv preprint arXiv:2601.05451 (2026). [47] Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Qian-Wen Zhang, Zhao Yan, and Zhoujun Li. 2023. MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL. In International Conference on Computational Linguistics. [48] Xiangjin Xie, Guangwei Xu, Lingyan Zhao, and Rui Guo. 2025. OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment. Proceedings of the ACM on Management of Data 3 (2025), 1 – 24. [49] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language

Related documents