Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks

Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks Shunfan Zheng1 , Dongsheng Shi1 , Yue Li1 , Xin Yi1 , Linlin Wang1 * , Gerard de Melo2 1 East China Normal University 2 Hasso Plattner Institute/University of Potsdam {sfzheng, dongsheng, yue_li, xinyi}@stu.ecnu.edu.cn, llwang@cs.ecnu.edu.cn, gdm@demelo.org

arXiv:2608.03794v1 [cs.DB] 4 Aug 2026

Abstract Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from “catastrophic forgetting” in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true fullstack database intelligence.

1

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in processing structured data, positioning them as potential successors to traditional database interfaces (Ruan et al., 2023; Kong et al., 2023). Beyond simple query generation, LLMs have increasingly exhibited the potential to serve as autonomous agents capable of planning and executing complex tasks (Shi et al., 2026; Chen et al., 2026; Zhang et al., 2026b), including managing the entire database pipeline (Zhang et al., * Corresponding Author.

I want to build an SQLite database for a library management system. The database should be able to record information such as book loans …

Database: Library

Table 1: books

Table 2: users

id title author …

id name address …

Spider/BIRD Missing 1

Design …

Missing 2

Implement …

…

Table n: loans

id book_id user_id …

DBLifeBench Design a library database management system. Implement the DB based on the designed structure.

Text-to-SQL which members have not returned …

…

…

Missing N

… maintain …

Single Task Evaluation

Text-to-SQL which members have not returned …?

…

You're a CPU agent; call the API to maintain the DB. Comprehensive Evaluation

Figure 1: Comparison between traditional Text-toSQL-centric benchmarks and our holistic DBLifeBench. While existing benchmarks focus solely on query translation, DBLifeBench covers the full spectrum of a DBA’s workflow.

2026a; Zhou et al., 2024b; Li et al., 2024). However, as LLMs are increasingly deployed in complex, high-stakes environments, the need for a rigorous, multifaceted evaluation framework has become paramount. High-quality benchmarks serve not only as evaluation tools but as a strategic “moat” that drives the evolution of model capabilities, providing the reference baseline essential for building robust database products (Schmidt et al., 2025). Unfortunately, current benchmarks fail to fulfill this role. Benchmarks such as Spider (Yu et al., 2018) and BIRD (Li et al., 2023) predominantly focus on the Text-to-SQL task (Hong et al., 2024; Yaghmazadeh et al., 2017; Xu et al., 2017; Zelle and Mooney, 1996). This narrow focus renders them insufficient as reference baselines for enhancing real-world database product capabilities. While valuable, relying solely on them creates a “tunnel vision” effect. Consider a real-world scenario: a database administrator (DBA) for a library system must first design a normalized schema to avoid redundancy, implement it with correct constraints, and continuously maintain it by analyzing error logs. A model that excels at generating SQL SELECT statements (Text-to-SQL) but fails to design

a viable schema or debug a deadlock is insufficient for real-world deployment. Optimizing models purely for existing benchmarks yields diminishing returns for actual product utility, as metrics like Execution Accuracy (EX) fail to assess structural and operational reasoning capabilities. To address these limitations, we introduce DBLifeBench, a comprehensive benchmark designed to evaluate the “Full-stack Database Intelligence” of LLMs. DBLifeBench extends beyond query translation to cover five key phases of the database lifecycle: design, implementation, operation, debugging, and maintenance. Each phase features tailored tasks and metrics, ensuring a multidimensional assessment of whether a model can truly function as a database expert. Moreover, within the Operation phase, we identify a fundamental cognitive gap in traditional Textto-SQL: complex SQL logic is often too intricate to be mapped directly from a single natural language sentence. To mitigate this, we propose ProgressiveText2SQL. This task utilizes a Dynamic Reasoning Graph to decompose complex queries into intermediate logical steps, effectively simulating the iterative reasoning process of human experts. This structure not only improves model performance but also aligns evaluations with genuine cognitive workflows. In summary, our contributions are the follows: • We propose DBLifeBench, the first benchmark to evaluate LLMs across five critical phases of the database lifecycle, moving beyond the single-task paradigm. • Specifically, we introduce ProgressiveText2SQL, a task that leverages reasoning graphs to bridge the gap between natural language ambiguity and SQL complexity, offering a more robust evaluation of reasoning capabilities. • Our comprehensive analysis reveals the phenomenon of the “curse of specialization”, whereby models fine-tuned specifically for SQL generation exhibit significant performance degradation in broader database management tasks.

2

The Proposed Benchmark

2.1

The Composition of DBLifeBench

To comprehensively assess Database Intelligence, DBLifeBench maps the database lifecycle to dis-

tinct cognitive capabilities of LLMs. We devise assessments across five phases: Design (P1: Abstraction & Modeling) This phase evaluates the model’s ability to translate abstract requirements into structured schemas. Given natural language requirements Q, the model must construct a schema S = (E, D, K), comprising entities E, data types D, and foreign key dependencies K. This tests the model’s high-level logical modeling capabilities. Implementation (P2: Syntax Precision) Models must translate the conceptual schema S into executable SQL Data Definition Language (DDL) statements Y. Unlike semantic generation, this phase demands strict syntactic precision to define tables, columns, and constraints that are valid within a specific database engine (e.g., SQLite). Operation (P3: Logic Translation & Reasoning) This phase focuses on converting operational needs into Data Manipulation Language (DML). It includes standard Text2SQL and our novel Progressive-Text2SQL. For a query or graph node Qi , the model generates satisfying SQL Yi . This tests the model’s ability to map natural language intent to database logic. Debugging (P4: Diagnosis & Correction) Simulating a developer’s workflow, the model receives a flawed SQL statement Yflawed and an error message. It must identify the root cause-whether syntactic or semantic and generate a corrected statement Ycorrected . Maintenance (P5: System Management) This phase simulates the role of a Site Reliability Engineer (SRE) or DBA. It consists of: (i) Triage (Assigner): Analyzing error logs to assign specific experts (e.g., IOExpert); (ii) Resolution (Expert): Utilizing simulated API tools (e.g., check_metric) to diagnose and fix system anomalies. This tests the model’s domain knowledge and tool-use capabilities. Figure 7 provides specific examples. 2.2

Benchmark Construction

2.2.1 Overview DBLifeBench integrates diverse data sources, including: S#1, existing open-source datasets; S#2, manually constructed data; S#3, authoritative textbooks, and S#4, automatically generated data. The entire corpus includes 13 distinct databases, with an average of 7.8 tables per database, covering diverse

User

Text2SQL

(1) Design

Database

What is the most popular school in …

SELECT School ID from school where …

Question (Text)

Answer (SQL)

Column name

Data format

Key type

Description

school id

INTEGER

Primary Key

…

…

Table 2: xxx

Table n: xxx

LLM Output: CREATE TABLE `school` (…)

exec

schools

Construct Graph

SQL: SELECT School ID from school where … Error Info: syntax error …

Input

-

(2) Implement

schools

UPDATE xxx (SELECT xxx) xxx Union xxx From xxx

Complex sql statements

(4) Debugging

Table 1: school

……

Progressive-Text2SQL (P-Text2SQL)

(3) Operations

Requirements ### Requirements Analysis for the “schools” Table ……

School ID “School ID”

1 SELECT

(5) Maintenance Alerts

(SELECT, From,)

3 (SELECT, Update,)

2

database

Update

Assigner

school id

TEXT

school id

INT

Data Type

author

TEXT

address

TEXT

Column

status type TEXT

status type TEXT Key Type

prediction

ground truth

Contextual Understanding

6

Data Modeling

(SELECT, Update, Union, From)

Instruction Following

5

Error Detection

(SELECT, Union,)

Table Name

IO Expert

Logical Reasoning

Graph

4

False True

SelfCorrection

CPU Expert Memory Expert

CPU APIs IO APIs: ➢ io_match_diagnose_knowledge ➢ ...

Memory APIs

Database Verification

---

Node

Graph Verify

Edge

Manual Verification

Evaluation Capabilities

Figure 2: An Overview of DBLifeBench. The left part illustrates the lifecycle-based evaluation, while the right part details the data construction process for the Progressive-Text2SQL task.

domains such as finance, education, and sports. We utilize this corpus to construct five datasets (shown in Table 1) for the five core database scenarios introduced in Section 2.1. Specific examples are provided in Appendix A.1. Phase

Dataset

P1 P2 P3 P4 P5

# Examples

Sources

Requirements

265

S#1; S#2

Text2SQL P-Text2SQL SQL-Debugging Maintenance

1,534 1,149 523 518

S#1; S#3 S#1; S#3; S#4 S#1; S#2; S#4 S#1

Table 1: Dataset Statistics

Requirements To ensure high-quality design prompts, we employed a human-AI collaborative loop. The process starts with AI reviewing the schema and generating a description. Three students then draft the initial requirements analysis based on the schema and AI description. AI analyzes this version, suggests improvements, and the students revise it. This iterative process is repeated for three rounds. Text-to-SQL & Progressive-Text2SQL We incorporate BIRD (Li et al., 2023) for standard evaluation and construct a new Progressive-Text2SQL dataset to address the “reasoning hop” challenges found in complex queries. Using the Dynamic Progressive Reasoning Graph method, we dynamically construct progressive reasoning graphs, detailed in Section 2.2.2. To ensure data diversity, we also manually include other statements like UPDATE

and DELETE, while prior work focuses on SELECT (Yu et al., 2019; Lei et al., 2024). SQL-Debugging Consulting BigTable (Zhang et al., 2024), we collect erroneous SQL statements and their corresponding error messages during the execution of Text-to-SQL tasks. Based on this, we construct the SQL Debugging dataset. These erroneous SQL statements cannot be executed properly in SQLite and require correction. Maintenance Based on the DB-GPT framework (Zhou et al., 2024a), we divide tasks into assignment and expert execution. Detailed examples are provided in Figure 7. Automated techniques like regular expression matching and database entity recognition were used for filtering, followed by execution and validation in a database environment to ensure effectiveness. 2.2.2 Progressive-Text2SQL Standard Text-to-SQL tasks often suffer from a misalignment between the brevity of natural language and the complexity of the target SQL. To address this, Progressive-Text2SQL transforms the task into a structured reasoning process (Figure 2, right). SQL Reasoning Graph We define a reasoning graph G = (V, E) to serve as a cognitive scaffold for the model. Nodes V = {v1 , . . . , vn } represent sub-tasks (e.g., “filter by date” or “join tables”), where each node contains a localized natural language description and its corresponding SQL fragment. Edges E represent logical dependencies,

guiding the model to build the final query vn incrementally. This structure mimics the “Chain-ofThought (CoT)” process of human experts. Graph Construction The data construction process is divided into three parts: (i) We collect examples from Text2SQL that are more difficult and where the misalignment between the natural language description and SQL has been manually identified. (ii) To mitigate potential bias from relying on a single model, we construct the reasoning graph using a diverse set of large language models, including GPT-4o, Claude, Gemini, and DeepSeekR1. Each model independently generates candidate graph structures based on complex SQL queries. We also provide a few manually written examples in the prompt to assist the LLM in understanding and generating the graph. The specific process is detailed in Appendix A.2. (iii) Based on the constructed SQL reasoning graph, we then use the LLM to generate the corresponding natural language description for each SQL node in the graph. The natural language description and SQL from the original example are provided in the prompt. Data Validation We employ a two-stage validation strategy. First, all SQL nodes are executed in an SQLite environment, and those failing to execute are discarded. Following this, three graduate students with backgrounds in database systems independently reconstruct graph edges using only the SQL and natural language descriptions for each node, without seeing the original model-generated edges. We then compare their annotations with the model-generated structure using Jaccard similarity. Only examples achieving more than 80% consensus are retained. To quantify inter-rater reliability (Artstein, 2017), we calculate Fleiss’ kappa, obtaining a score of 0.8173, which indicates strong agreement. Annotators are trained with representative edge construction examples and participate in weekly calibration discussions. Additionally, all annotation disagreements are logged and analyzed to identify common error patterns such as alias confusion and aggregation mismatches, which in turn inform future refinements to the data construction pipeline.

3

Evaluation Metrics

As shown in Section 2.1, the evaluations in DBLifeBench vary across different phases, necessitating a tailored set of assessment metrics based

on the unique characteristics of each task. (1) For the design phase: we assess whether the model’s designed database structure is reasonable in this phase. We establish three evaluation metrics: entity name accuracy (ACCi -Entity), entity data type accuracy (ACCi -Data), and foreign key dependency accuracy (ACCi -Key). i denotes the number of output tables, defaulting to 2. ACCi -Entity can computed by: PT t=1 Et ∩ Êt (1) ACCi -Entity = , PT |E | t t=1 where T is the total number of tables, Êt represents the ground-truth set of entity names for table t, and Et represents the set of entity names predicted by the model for table t. Analogously, the same approach is followed for the other metrics. (2) For the implementation phase: in this phase, the model outputs the creation statements for database tables. We execute these statements and evaluate the T resulting tables using table-level accuracy (T-Level) and field-level accuracy (FLevel):  T  1 X   T-Level = I(Nt , N̂t )   T t=1   PT   t=1 Ft ∩ F̂t   F-Level =  PT   F̂t

(2)

t=1

Here, Nt represents the predicted table name for table t and Ft represents the set of fields for predicted table t. N̂t and F̂t represent the corresponding ground truth. Finally, I(·) is an indicator function that yields 1 whenever Y and Ŷ are semantically consistent (not necessarily identical in name), and 0 when Y and Ŷ are not semantically consistent. (3) For the operation phase: in the case of the Text2SQL task, we adopt the widely used Execution Accuracy (EX) metric, which calculates accuracy by comparing the execution results Vˆn of the n-th predicted SQL statement with the execution results Vn of the ground truth SQL: N

1 X EX = I(Vn , V̂n ) N

(3)

n=1

For the Progressive-Text2SQL task, we apply EX for each node, and define the Graph Execution Accuracy (G-EX) as the average of the node accuracies, i.e. G

G-EX =

1 X EXg , G g=1

(4)

Design ACC2 -Entity

ACCi -Data

ACCi -Key

AVG

T-level

F-level

AVG

General

Implementation

GPT-4o GPT-4o-mini Llama3 Mistral DeepSeek Qwen2.5 ChatGLM-4

76.94 69.30 64.90 41.99 33.46 66.31 67.08

65.97 55.84 50.20 31.72 25.96 53.80 52.48

80.99 79.95 73.14 39.16 37.59 67.71 72.62

74.63 68.36 62.74 37.62 32.34 62.61 64.06

85.71 79.03 70.41 56.08 42.88 80.39 78.71

77.35 71.30 68.59 39.23 32.55 65.81 69.24

81.53 75.17 71.00 47.66 37.72 73.10 73.98

Specialized

Model

DeepSeek-Coder SQLCoder CodeQwen Llama3-sqlcoder

66.78 52.79 69.24 45.95

56.62 40.60 59.45 35.45

74.84 59.19 75.33 46.40

66.08 50.86 68.00 42.60

78.28 10.00 73.86 38.45

64.18 7.35 61.83 29.17

71.23 8.68 67.85 33.81

Operation

Maintenance

Text2SQL

P-Text2SQL

Single-round

Multi-round

Assigner

Expert

AVG

General

Debugging

GPT-4o GPT-4o-mini Llama3 Mistral DeepSeek Qwen2.5 ChatGLM-4

56.19 47.72 30.51 19.82 17.54 38.01 35.98

53.79 54.31 45.63 28.55 26.81 39.08 46.13

68.39 60.44 38.04 23.62 23.09 35.38 30.78

72.17 61.83 43.42 36.82 27.81 42.11 40.98

79.38 72.88 59.32 46.47 53.39 67.80 46.89

33.50 33.58 32.77 3.58 31.25 32.25 33.17

56.44 53.23 61.55 25.03 42.32 50.02 40.03

Specialized

Model

DeepSeek-Coder SQLCoder CodeQwen Llama3-sqlcoder

39.31 24.64 37.35 33.25

47.00 29.85 41.22 39.43

41.53 25.74 21.99 48.18

50.69 31.89 23.32 62.52

45.48 0.00 35.31 46.05

33.50 0.00 32.25 32.92

39.49 0.00 33.78 39.48

Table 2: Main Results on DBLifeBench

where G is the number of reasoning graphs, and EXg represents the Execution Accuracy of the g-th graph. (4) For the debugging phase: we execute the corrected SQL generated by the model on the database and evaluate the model’s error correction ability based on the execution results. (5) For the maintenance phase: we evaluate the correctness of expert assignments (by the Assigner) and API calls (by the Expert).

4

Experiments

4.1

Baseline Models

We evaluate the performance of two classes of advanced baseline models in DBLifeBench. The first class consists of general models, including OpenAI’s GPT series (GPT-4o, GPT-4o-mini), Llama3, Mistral, DeepSeek, Qwen, and ChatGLM-4. The second class comprises specialized models that have been fine-tuned on SQL, code, and related data, including DeepSeek-Coder, SQLCoder, CodeQwen, and Llama3-Coder.

4.2

Experimental Setup

We run our extensive experiments using the SQLite database, consistent with previous work such as BIRD (Li et al., 2023). During inference, we consistently set the temperature to 0.3 and the top-p to 0.2 to ensure a balance between diversity and coherence in the generated outputs. 4.3

Main Results

We conducted a comprehensive evaluation of 11 LLMs using our DBLifeBench, examining their performance across five different phases of the database lifecycle. The experimental results are presented in Table 2. GPT-4o and GPT-4o-mini generally outperform other models across almost all aspects, demonstrating their comprehensive database capabilities. However, certain models show varying performance in different phases of evaluation, underscoring the need for a more comprehensive evaluation of different phases. For instance, ChatGLM-4 excels in the Design and Implementation phases, but performs poorly in the Operation and Maintenance phases. Some models, such as SQLCoder, struggle significantly in the Implementation and Maintenance phases, likely

5

Analysis

5.1

Impact of Data Model Variations

To assess the impact of schema variation on LLMs’ database performance, we conducted experiments using three schema variants: v1 (lowest normalization), v2, and v3 (highest normalization), all sourced from (Fürst et al., 2024) and migrated to SQLite. These are evaluated on Text2SQL and P-Text2SQL tasks. Table 5 shows LLM performance across schemas. v2 underperformed due to alias conflicts and redundant UNION operations from bridge tables, which make queries more complex. In con-

Model

Text2SQL

P-Text2SQL

5.30 9.30 13.42 13.42 7.91 11.51

4.30 7.57 5.80 5.80 6.83 5.95

GPT-4o Llama3 Qwen2.5 CodeQwen DeepSeek DeepSeek-Coder

Table 3: Coefficient of Variation 4.2

Iter3

4.0

#5

Iter2

Iter1

#3

#6

3.6

20

3.4 3.2 3.0

#4

#2

3.8

Score

due to their training data being heavily focused on Text2SQL tasks, which adversely affects their performance in other areas. Specialized models struggle with “Full-stack” tasks. DeepSeek, after fine-tuning with code data, shows significant improvements across the lifecycle evaluation, while Llama3, fine-tuned on SQL data, experiences a decline in the Design and Implementation phases. Task-specific fine-tuning can sometimes cause models to over-focus on particular patterns, impairing their generalizability to other types of data. In contrast, general-purpose models appear less affected by this overfitting and can better cope with variation. Additionally, the task itself may have scale dependencies: As shown in Section 5.5, when the number of input tables is low, the task is simpler, and general models might perform better since specialized models’ fine-tuned features may not offer significant advantages. Some models lack the ability to perform multiround debugging. Models such as Llama3 and Mistral show significant improvements in multiround debugging compared to single-round debugging, indicating that they are better at recognizing and correcting their previous mistakes. In contrast, DeepSeek lacks this capability, as its performance in the multi-round setting is consistent with that of single-round debugging, showing no improvement. Model performance is different in the two maintenance tasks. As Assigners, models like GPT-4o and Qwen are adept at discerning error types from logs, but others are not. This reflects differences in the models’ ability to perceive database contexts and the richness of their database knowledge. As Experts, all models exhibit similar performance and obtain low scores because they are not good at using database tools.

Iter1

Iter2

Iteration

Relevance Correctness Expression Iter3

(a) Human Assessment

#7

40

#1 60

#11 #8 #9

#10

(b) Model Performance

Figure 3: (a) Human quality assessment of different versions of requirements analysis. (b) Performance of 11 models with different versions of requirements analysis, ordered as in Table 2.

trast, v3 performed better by merging tables, eliminating aliases, and aligning schema semantics with natural language queries. This suggests that overnormalization should be avoided, with a focus on semantic alignment and minimizing multi-table joins. Additionally, as shown in Table 3, the coefficient of variation (Reed et al., 2002) is smaller in the P-Text2SQL task, indicating that P-Text2SQL helps the model better adapt to database data model variations. 5.2

Iteration of Requirement Analysis

The requirements analysis dataset underwent three rounds of iterative revisions. To assess the improvements, we analyzed the dataset after each iteration from both human evaluation and model performance perspectives. Figure 3(a) shows that three master’s students assessed the dataset on relevance, correctness, and expression. The results indicate significant improvement after each iteration, confirming enhanced clarity and accuracy. Fleiss’ kappa coefficient (Fleiss et al., 1981) yielded a value of 0.8171, indicating strong consistency among annotations. Figure 3(b) shows that 11 baseline models performed better in the database design phase as the quality of the requirements analysis improved, suggesting that higher-quality analysis leads to better model performance and fewer evaluation errors.

Text2SQL Progressive-Text2SQL

50

140 Growth Rate (%)

Execution Accuracy (EX)

60

40 30

120 100 80 60

0 .5 ek 4o en a3 GPT- LLam DeepSe Qwen2 codeqw

Simple Moderate 40 Challenging 4o ama3 Qwen GLM4 pSeek T P t G LL Cha Dee

(a) EX Comparison

(b) Growth Rate Comparison

20 10

Figure 4: Comparison of the model’s EX results on Progressive-Text2SQL and Text2SQL tasks. The Progressive-Text2SQL settings slightly differ from the main experiment. We only compute the EX for the final node of each graph to ensure consistency with the Text2SQL data.

5.3

In-depth Analysis with P-Text2sql

To explore whether the dynamic progressive reasoning graph in Progressive-Text2SQL can better assist models in generating complex SQL queries, we compare the performance of models in Text2SQL and Progressive-Text2SQL tasks, as shown in Figure 4(a). Graphs significantly improve the Execution Accuracy (EX) of all baseline models, indicating that the graph effectively simulates incremental user input, helping the model better understand and complete SQL generation tasks. Additionally, we compare the growth rate of EX for SQL generation tasks of varying difficulty when a graph is applied. As shown in Figure 4(b), as the task difficulty increases, the improvement in SQL generation due to the use of graphs becomes more pronounced, with the most significant increase observed for tasks at the “Challenging” difficulty level. This is because more difficult SQL queries tend to be longer and harder to describe clearly using simple natural language. With graphs, however, the model is guided through a step-bystep reasoning process, helping it generate more accurate SQL queries. 5.4

Robustness Test of P-Text2SQL

In the Progressive-Text2SQL dataset, we constructed high-quality reasoning graphs, but realworld reasoning may contain errors. To better simulate realistic disruptions and evaluate model robustness, we define level-k perturbations as the targeted selection of nodes within the reasoning graph, where each selected node is subjected to one or more disruptive operations. These operations include syntactic corruption of SQL statements, se-

60 50 40 30 20 10 0

Design Metrics AVG (%)

70

1

2

3

Table Num

Llama3 Qwen2.5 DeepSeek Llama3-sqlcoder Qwen2.5-Coder DeepSeek-Coder 4 5

Figure 5: Impact of the number of tables on LLMs in database design tasks.

mantic misalignment with the schema, injection of contradictory logic, or complete removal of the node. This approach introduces realistic flaws that can affect downstream reasoning. As shown in Table 4, model performance declines as the perturbation level increases, yet in most cases, still outperforms baseline Text2SQL models. This highlights the importance of step-by-step structured reasoning and the critical role of graph integrity in SQL generation tasks. 5.5

Influence of Number of Tables for Database Design

In the main experiment, the design phase evaluation requires the model to design two tables within the database. To further investigate how the number of tables to be designed affects the model, we modify the input to include the description of the entire database, with the model being tasked to output the design for a specified number of tables, ranging from 1 to 5. We then evaluate the design capabilities of models under these different requirements. As shown in Figure 5, as the number of tables grows, different models exhibit varying performance. Among them, the general-purpose models show relatively stable and consistent performance, with little impact from the number of tables. In contrast, models fine-tuned on code data exhibit more significant fluctuations in performance, which may be due to the impact of code fine-tuning on the general instruction-following capabilities of the model. Some specialized models, in most cases, perform worse with fewer input tables compared to when they have more tables. This suggests that when the number of input tables is small, specialized models may struggle to fully utilize their fine-tuned features in such simpler tasks.

P-Text2SQL Level-1 Level-2 Level-3 Level-4 Level-5 Text2SQL

Llama3

Mistral

DeepSeek

Qwen2.5

ChatGLM-4

DeepSeek-Coder

sqlcoder

CodeQwen

Llama3-sqlcoder

46.76 43.71 41.48 36.29 33.28 32.14 28.85

30.1 30.02 24.65 22.13 20.40 19.65 15.17

28.11 26.24 22.89 16.36 16.67 16.23 13.68

47.01 46.48 38.81 33.33 29.85 29.85 28.86

46.52 45.27 37.65 33.33 31.59 30.85 28.86

46.27 43.81 39.98 36.07 33.68 32.68 30.85

36.32 34.27 29.58 22.45 20.71 19.65 16.67

41.29 41.18 32.34 25.87 21.60 19.90 16.42

44.03 42.39 36.90 30.35 27.36 26.59 24.88

Table 4: Robustness Test of Progressive-Text2SQL

5.6

New Perspective of Model Rankings

Existing benchmarks are based on SQL generation tasks, such as Text-to-SQL, and do not address other types of database tasks. For further analysis, in Figure 6, we test the model’s performance on non-SQL generation tasks (e.g., design, maintenance). Among them, SQLCoder performs poorly on other tasks compared to its performance on SQL generation tasks, which is due to the fine-tuning on SQL data affecting its capabilities on other tasks. Model Comparison of SQL Tasks GPT-4o 56.19

GPT-4o-mini 47.72

Qwen2.5 38.01

Llama3-SQL 33.25

SQLCoder 24.64

DeepSeek 17.54

Model Comparison of Other Tasks

Figure 6: Ranking of models on SQL generation tasks and other (non-SQL generation) tasks.

6

Related Work

LLM Applications in Database Tasks Large language models (LLMs) have enabled significant advances in a variety of fields, including database management (Zhou et al., 2024b; Li et al., 2024). For example, LLMs have been employed for Text-to-SQL tasks, where they generate SQL queries based on natural language descriptions. A range of LLM-based Text-to-SQL methods have been proposed (Hong et al., 2024; Yaghmazadeh et al., 2017; Xu et al., 2017), showing substantial progress on this task. Additionally, LLMs have been used in database maintenance (Zhou et al., 2024a), a domain where recent work has contributed to advancements. As LLMs become more widely used in the database field, the challenge of

evaluating their database capabilities has grown, necessitating a comprehensive benchmark. Existing Database Benchmarks Several benchmarks have been proposed to evaluate LLMs in the context of database-related tasks. SQLStorm (Schmidt et al., 2025), BIRD (Li et al., 2023) and Spider (Yu et al., 2018; Lei et al., 2024) are Textto-SQL benchmarks designed to assess a model’s ability to generate SQL queries from natural language questions. They are widely used to evaluate the performance of general models in query generation, primarily focusing on execution accuracy (Execution Accuracy, EX). Additionally, Database QA benchmarks (Zheng et al., 2025) have been introduced to assess the richness of models’ database domain knowledge. However, both benchmarks are limited to a single task and fail to comprehensively assess other database-related tasks such as schema design, database debugging, or maintenance. We fill this gap by proposing a more comprehensive evaluation benchmark that covers the entire database lifecycle (Gupta et al., 2011), including design, implementation, debugging, and maintenance.

7

Conclusion

We presented DBLifeBench, a pioneering benchmark that shifts the evaluation paradigm from isolated SQL generation to holistic database lifecycle management. By assessing capabilities across Design, Implementation, Operation, Debugging, and Maintenance, we exposed the limitations of current specialized models, which often sacrifice general reasoning for syntax proficiency. Additionally, our Progressive-Text2SQL task demonstrates that structured, step-by-step reasoning is key to solving complex database problems. DBLifeBench provides the community with a rigorous standard to guide the development of the next generation of LLMs-true Autonomous Database Agents.

Limitations DBLifeBench, while comprehensive, has some limitations. It is primarily designed for LLMs, which may not fully capture the performance of non-LLM approaches. Additionally, some tasks, particularly in design and maintenance, may be challenging to define and evaluate accurately. Lastly, the current benchmark focuses exclusively on textual modalities and does not address multimodal scenarios, such as interpreting visual performance charts. Future work will aim to incorporate multimodal inputs to broaden the scope of evaluation.

References Ron Artstein. 2017. Inter-annotator agreement. Handbook of linguistic annotation, pages 297–313. Yanyu Chen, Yue Li, Yongyi Cui, Dongsheng Shi, and Lichang Dai. 2026. Reinforcement learning for large language model selective evidence adoption from contaminated retrieval results. arXiv preprint arXiv:2607.20090. Joseph L Fleiss, Bruce Levin, Myunghee Cho Paik, et al. 1981. The measurement of interrater agreement. Statistical methods for rates and proportions, 2(212236):22–23. Jonathan Fürst, Catherine Kosten, Farhad Nooralahzadeh, Yi Zhang, and Kurt Stockinger. 2024. Evaluating the data model robustness of text-to-sql systems based on real user queries. arXiv preprint arXiv:2402.08349. Pranshu Gupta, Ramon Mata-Toledo, and Morgan Monger. 2011. Database development life cycle. Romanian Economic Business Review, 5:8–17. Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2024. Next-generation database interfaces: A survey of llmbased text-to-sql. arXiv preprint arXiv:2406.08426. Yilun Kong, Jingqing Ruan, Yihong Chen, Bin Zhang, Tianpeng Bao, Shiwei Shi, Guoqing Du, Xiaoru Hu, Hangyu Mao, Ziyue Li, Xingyu Zeng, and Rui Zhao. 2023. Tptu-v2: Boosting task planning and tool usage of large language model-based agents in realworld systems. Preprint, arXiv:2311.11315. Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2024. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. Preprint, arXiv:2411.07763. Guoliang Li, Xuanhe Zhou, and Xinyang Zhao. 2024. Llm for data management. Proceedings of the VLDB Endowment, 17(12):4213–4216.

Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Preprint, arXiv:2305.03111. George F Reed, Freyja Lynn, and Bruce D Meade. 2002. Use of coefficient of variation in assessing variability of quantitative assays. Clinical and Vaccine Immunology, 9(6):1235–1239. Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, et al. 2023. Tptu: Task planning and tool usage of large language model-based ai agents. In NeurIPS 2023 Foundation Models for Decision Making Workshop. Tobias Schmidt, Viktor Leis, Peter Boncz, and Thomas Neumann. 2025. Sqlstorm: Taking database benchmarking into the llm era. Proceedings of the VLDB Endowment, 18(11):4144–4157. Dongsheng Shi, Yue Li, Xin Yi, Yongyi Cui, Huawei Feng, and Linlin Wang. 2026. Surgent: A surgical multi-agent assistance system across the perioperative workflow. arXiv preprint arXiv:2605.29368. Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sqlnet: Generating structured queries from natural language without reinforcement learning. arXiv preprint arXiv:1711.04436. Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. Sqlizer: query synthesis from natural language. Proc. ACM Program. Lang., 1(OOPSLA). Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. 2019. Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases. arXiv preprint arXiv:1909.05378. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics. John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pages 1050–1055. Bin Zhang, Yuxiao Ye, Guoqing Du, Xiaoru Hu, Zhishuai Li, Sun Yang, Chi Harold Liu, Rui Zhao,

Ziyue Li, and Hangyu Mao. 2024. Benchmarking the text-to-sql capability of large language models: A comprehensive evaluation. arXiv preprint arXiv:2403.02951. Kaituo Zhang, Mingzhi Hu, Hoang Anh Duy Le, Fariha Kabir Torsha, Zhimeng Jiang, Minh Khai Bui, Chia-Yuan Chang, Yu-Neng Chuang, Zhen Xiong, Ying Lin, Guanchu Wang, and Na Zou. 2026a. A survey on evaluating quality and trustworthiness in llm-generated data. Preprint, arXiv:2601.17717. Kaituo Zhang, Zhen Xiong, Mingyu Zhong, Zhimeng Jiang, Zhouyuan Yuan, Zhecheng Li, and Ying Lin. 2026b. Are tools all we need? unveiling the tool-use tax in llm agents. arXiv preprint arXiv:2605.00136. Yihang Zheng, Bo Li, Zhenghao Lin, Yi Luo, Xuanhe Zhou, Chen Lin, Guoliang Li, and Jinsong Su. 2025. Revolutionizing database q&a with large language models: Comprehensive benchmark and evaluation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 5960–5971. Xuanhe Zhou, Zhaoyan Sun, and Guoliang Li. 2024a. Db-gpt: Large language model meets database. Data Science and Engineering, 9(1):102–111. Xuanhe Zhou, Xinyang Zhao, and Guoliang Li. 2024b. Llm-enhanced data management. arXiv preprint arXiv:2402.02643.

A

Appendix

} Edges : 15 { 16 " source ": 1, 17 " target ": 3 18 }, 19 { 20 " source ": 2, 21 " target ": 3 22 } 23 " question ": 24 Update the instructors of all courses held in room 101 of the ' Smith ' building to ' David Brown '. 25 # Output 26 UPDATE teaches SET ID = ( SELECT ID FROM instructor WHERE name = ' David Brown ') WHERE ( course_id , sec_id , semester , year ) IN ( SELECT course_id , sec_id , semester , year FROM section WHERE building = ' Smith ' AND room_number = '101 ') 13 14

A.1

Examples

# Input Requirements Analysis : 3 1. Purpose 4 2. Key Requirements 5 3. Non - Functional Requirements 6 ... 7 # Output 8 Table 1: frpm 9 CDSCode , TEXT , Primary Key 10 ... 11 Table 2: satscores 12 cds , TEXT , Primary Key 13 ... 1 2

Listing 1: Example of Design. # Input { results of design phase } 3 # Output 4 Table 1: CREATE TABLE frpm ( 5 CDSCode TEXT PRIMARY KEY , 6 AcademicYear TEXT , 7 ... 8 ); 9 Table 2: CREATE TABLE satscores ( 10 cds TEXT PRIMARY KEY , 11 ... 12 ) ; 1 2

Listing 2: Example of Implementation. # Input ## Simple DDL : 3 albums ( AlbumId , Title , ArtistId ) 4 artists ( ArtistId , Name ) 5 ... 6 ## Question : 7 Retrieve the album titles and the corresponding artist names . 8 # Output 9 SELECT Title , Name FROM albums INNER JOIN artists USING ( ArtistId ); 1 2

Listing 3: Example of Text2SQL. # Input Simple DDL : ... 3 Nodes : 4 { 5 " id " : 1, 6 " sql statement ": " SELECT ID FROM instructor WHERE name = ' David Brown '", 7 " nl " : " Retrieve the ID of the instructor whose name is ' David Brown '. " 8 }, 9 { 10 " id " : 2, 11 " sql statement ": " SELECT course_id , sec_id , semester , year FROM section WHERE building = ' Smith ' AND room_number = '101 ' " , 12 " nl " : " Retrieve the course ID , section ID , semester , and year for all sections held in room 101 of the ' Smith ' building ." 1 2

Listing 4: Example of P-Text2SQL. # Input ## Simple DDL : ... 3 ## Wrong SQL : 4 SELECT T1 . Phone FROM satscores AS T1 INNER JOIN schools AS T2 ON T1 . cds = T2 . CDSCode WHERE T1 . AvgScrRead > 1500 ORDER BY T1 . NumTstTakr DESC LIMIT 1 5 ## Error Info : 6 OperationalError (' no such column : T1 . Phone ') 1 2

7 8 9

# Output SELECT T2 . Phone FROM satscores AS T1 INNER JOIN schools AS T2 ON T1 . cds = T2 . CDSCode ORDER BY T1 . NumGE1500 DESC LIMIT 1

Listing 5: Example of SQL-Debugging. # Input ## System : 3 You are a role assigner . Your task is to select experts with diverse identity to correctly analyze the root causes of the given alerts . 4 ## User : 5 - Role Description 6 You are the leader of a group of diagnosis experts . Currently , the database in your company may meet problems . The anomaly alert is : 7 During the inspection , it was found that from 2023 -10 -15 11:39:49 to 2023 -10 -15 11:40:49 , the database 's CPU usage was above the threshold of ... 1 2

8

Now you need to select experts with diverse identity to correctly analyze the root causes of the given alert . The names of available experts are : 10 [' CpuExpert ', ' MemoryExpert ', ' IoExpert ', ' WorkloadExpert ', ' QueryExpert ', 9

Requirements Analysis 1. Purpose The `frpm` table is designed to store data related to the Free and Reduced-Price Meal (FRPM) program for schools. … 2. Key Requirements … 3. Non-Functional Requirements... 4. …

Database Design # Table 1: frpm frpmCDSCode, TEXT, Primary Key (other fields …) # Table 2: satscores satscorescds, TEXT, Primary Key ...

Maintenance-Assign

Maintenance-Expert

## System: You are a role assigner. Your task is to select experts with diverse identity to correctly analyze the root causes of the given alerts. ## User: {Role Description} ## Optional Expert: {Alert Information} ['CpuExpert', {Output Rules} 'MemoryExpert', 'IoExpert', 'WorkloadExpert', 'QueryExpert', Output Output 'WriteExpert', 1. CpuExpert 'IndexExpert', 2. MemoryExpert 'ConfigurationExpert'] 3. IoExpert

## System: You are a CPU agent that can use the tool apis to check CPU usage, analyze the root causes of … ## User: {Alert Information} Based on the above history, what will you, CpuExpert, do next? ## Optional APIs: - cpu_match_diagnose_knowledge (args …) - whether_is_abnormal_metric (args …), - …

Expert Output

Thought: {Chain of Thought} Action: whether_is_abnormal_metric Action Input: {args …}

Figure 7: Illustration of Design and Maintenance

Text2SQL

Model GPT-4o Llama3 Mistral DeepSeek Qwen2.5 ChatGLM-4 DeepSeek-Coder llama3-sqlcoder sqlcoder-7b-2 Qwen2.5-Coder-7B-Instruct

P-Text2SQL

v1

v2

v3

v1

v2

v3

19.25 13.75 9.50 11.75 14.25 16.75 15.00 12.00 8.75 9.00

18.00 11.75 9.25 10.00 14.00 15.00 13.00 11.75 8.00 8.25

20.50 14.75 8.75 12.00 17.00 15.00 17.25 15.00 8.75 11.25

54.33 42.67 30.00 36.33 49.33 37.67 46.00 49.00 38.00 36.33

49.19 40.45 22.98 30.74 41.42 37.86 44.01 41.42 32.04 33.01

50.33 48.33 30.00 33.33 49.67 41.00 50.67 53.00 33.67 38.00

Table 5: Comparison of the model’s performance across different schemas. ' WriteExpert ', ' IndexExpert ', ' ConfigurationExpert '] 11

# Output 1. CpuExpert 14 2. MemoryExpert 15 3. IoExpert

- During diagnosis , you have access to the following tools : 8 { Optional APIs ...} 7

12

9

13

10

Listing 6: Example of Maintenance-Assign. # Input ## System : 3 You are a CPU agent that can use the tool apis to check CPU usage ( whether_is_abnormal_metric ) , analyze the root causes of high CPU usage using the metrics , queries and knowledge gained from ( cpu_match_diagnose_knowledge ) , and give optimization solutions (e.g., optimize_index_selection , enable_or_disable_nestloop_operator ) . 4 ## User : 5 You are in a company whose databases meet an anomaly . The anomaly 's start_time is ${ start_time } and end_time is ${ end_time }. The anomaly alert is : { Alert ...} 1 2

6

{ Database history }

11

Based on the above history , what will you , CpuExpert , do next ? 13 # Output 14 Thought : Now that I have obtained the start and end time of the anomaly , I will check whether the CPU usage is abnormal during that time period using the ` whether_is_abnormal_metric ` tool . 12

15 16 17

Action : whether_is_abnormal_metric Action Input : {" start_time ": " 2023 -10 -15 21:01:49 " , " end_time ": " 2023 -10 -15 21:02:49 " , " metric_name ": " cpu_usage "}

18 19

Please execute the above action and provide the result .

Listing 7: Example of Maintenance-Expert.

A.2

Graph Construction of Progressive-Text2SQL

The following is the prompt for constructing graph data in P-Text2SQL, with the Examples provided in Listing 4. 1 2

3

4

5

6

prompt = ( " The goal is to construct a directed graph representation from a given sqlite SQL statement to represent the gradual advancement of functional implementation and eventually complete all requirements ." " Each node in the graph represents a SQL statement , which is a subsequence of a given SQL statement , and the edge represents the execution order . " " Subsequent nodes are the progression of the previous node . ( please attention the last node should be the original given SQL statement . )" " This graph should faithfully reflect the topological execution order of SQL statements to achieve requirements . " " The following includes three cases , including Sql , and Graph .\ n\n "

7 8 9 10 11 12 13 14

15 16 17

" ## Example 1\ n" f " { example_template_1 }\ n\n" " ## Example 2\ n" f " { example_template_2 }\ n\n" " ## Example 3\ n" f " { example_template_3 }\ n\n" " Please build the graph according to the following Sql . Please note that just output the final graph . Do not include any other superfluous descriptions .\ n\n" " # Sql :\ n" f " { raw_sql }\ n" " # Graph :\ n")

Listing 8: Prompt for Graph Construction.

Related documents

• ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language#673625
• ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language#974096
• NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions#120627
• A Model-Driven Approach to Database Migration with a Unified Data Model#1105019
• SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation#986569