SiriusDeliver: Automating Data Warehouse Delivery at Tencent

SiriusDeliver: Automating Data Warehouse Delivery at Tencent Haining Xie† , Xiaokai Zhou‡ , Jiaming Yang† , Siqi Shen† , Ziwei Wang† , Yifeng Zheng† , Tengyue Xu† , Yipeng Shi† , Zefang Zong† , Yang Li† , Peng Chen† , Jie Jiang† , Debiao He‡ , Xiao Yan‡ , Jiawei Jiang‡ † TEG, Tencent Inc.

‡ Wuhan University

† {hainingxie,besmingyang,siqishen,willziwang,yifengzheng,leooxu,

portershi,willzong,thomasyngli,pengchen,zeus}@tencent.com ‡ {xiaokaizhou,hedebiao,yanxiaosunny,jiawei.jiang}@whu.edu.cn

Enterprise data warehouses (DWs) support business-critical analytics, but warehouse task delivery remains a complicated production process involving context retrieval, workflow configuration, code generation, platform submission, and failure diagnosis. Although large language models (LLMs) and coding agents have improved software development, they are insufficient for production DW delivery, which requires dependency-aware orchestration, lifecycleaware artifact control, and continuous adaptation to evolving platform practices. We present SiriusDeliver, an end-to-end delivery automation agent for production warehouse task submission. SiriusDeliver integrates three components: a hierarchical delivery agent that orchestrates warehouse skills, an artifact lifecycle control module that verifies and revises artifacts before and after platform execution, and a trace-driven skill evolution mechanism that maintains reusable skills from delivery trajectories. We evaluate SiriusDeliver through offline datasets and large-scale production deployment on Tencent Cloud WeData. Offline experiments on realworld warehouse delivery cases show that SiriusDeliver improves delivery success and automation efficiency over representative baselines. During a two-month deployment across 6 business teams and 4 warehouse task types, SiriusDeliver served 3,600 monthly active users and supported 18,240 delivery sessions, achieving an 87.2% end-to-end success rate and a 73.5% autonomous submission rate. A one-month A/B test shows that SiriusDeliver reduces median delivery time from 228 to 23 minutes and engineer effort from 95 to 11 minutes, while maintaining comparable final delivery success.

Keywords Data Warehouse Task Delivery, Data Agent, Skill Evolution

1

Introduction

Enterprise data warehouses (DWs) underpin business-critical analytics by transforming raw business events into analytical tables. At Tencent, the production warehouse platform serves over 10,000 monthly active data engineers, manages millions of warehouse tasks, and maintains hundreds of thousands of production tables. Yet warehouse task delivery remains largely manual. As shown in Figure 1, engineers must retrieve context from metadata services, workflow platforms, historical tasks, logs, and troubleshooting documents; generate code and configurations; validate dependencies; submit tasks; and repair failures. This fragmented process Conference’17, Washington, DC, USA 2026. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn

Generate daily user feature tables from transaction logs.

Online Impact

Intent Context Planning

Delivery Agent

Metadata

History

Skill Orchestration

…

arXiv:2608.09185v1 [cs.DB] 10 Aug 2026

Abstract

Planning

Workflow

Engineer

Artifact Generation Codes

Config

Manual Repair & Resubmit

Dependency

Deliver Fail

Failure Diagnosis

Human-driven Delivery

Execution

Artifact Lifecycle Control Verify Deliver

WeData Warehouse

Diagnose Fail Resubmit

6Teams 3600+ MAUs 18,000+ Sessions

Delivery Time 228 → 23 min

Revise Improve

Skill Evolution

Agent-driven Automation

Engineer Effort 95 → 11 min

Auto Submission 73.5%

Figure 1: Traditional human-driven warehouse delivery (left) vs. our agent-driven automation (right). SiriusDeliver automates the end-to-end delivery process, substantially reducing manual intervention in production warehouse systems. is labor-intensive and error-prone: missing dependencies, inconsistent schedules, invalid permissions, or inefficient queries can delay delivery or introduce silent data-quality risks. Evidence from our internal questionnaire survey, supplemented by in-depth followup interviews with experienced practitioners, suggests that SQL development accounts for less than 25% of end-to-end delivery time, while information retrieval and cross-platform coordination collectively constitute the primary sources of effort. Recent advances in large language models (LLMs) [1, 7, 9, 14] and coding agents [2, 10, 15, 25] have demonstrated strong capabilities in automated software development. However, production warehouse task delivery fundamentally differs from repository-level coding: it requires end-to-end coordination over data semantics, workflow dependencies, platform policies, execution feedback, and long-term operational maintenance. Without such DataOps integration, general-purpose coding agents often produce plausible artifacts that remain incomplete, unsafe, or difficult to maintain in production warehouse environments. Our production experience shows that directly applying generalpurpose code agents to this setting leaves three key challenges. (1) Multi-stage task composition. Warehouse task delivery spans multiple interdependent stages—from metadata retrieval and dependency resolution to code generation, configuration, and platform submission. For instance, importing operational data requires resolving source/target schemas, incremental keys, and schedule dependencies before task execution. In practice, one-shot LLM planning frequently omits or misorders such prerequisites, producing invalid configurations or failed submissions. Existing tool-use

Conference’17, July 2017, Washington, DC, USA

agents [16, 19, 20] rely on curated supervision or static tool definitions, which fail to handle enterprise warehouses with complex, implicit, and team-specific dependency constraints. (2) Trustworthy artifact delivery. A deliverable warehouse bundle encompasses executable code, workflow topologies, schedules, permissions, and platform metadata. Syntactically valid artifacts may still contain invalid dependencies, schedule conflicts, or logic bugs that trigger silent data errors or execution failures. Existing code agents [28, 30] perform reactive post-execution repair based solely on logs. In contrast, production delivery demands proactive pre-submission verification alongside post-execution diagnosis to guarantee end-to-end artifact safety. (3) Continuous platform adaptation. As platform APIs, data schemas, scheduling policies, and operational conventions continually evolve, static delivery rules can rapidly become obsolete. Existing lifelong learning methods [22, 26, 31] optimize transient task-level behaviors rather than auditable, reusable skill updates. Furthermore, they overlook signals from successful trajectories, which can reveal potentially redundant context acquisition and reasoning steps that could streamline future execution. To address these challenges, we present SiriusDeliver, an endto-end agent for production warehouse task delivery. For Challenge (1), its delivery automation agent iteratively plans, routes, executes, and updates memory over hierarchical scenario, context, artifact, and platform skills, composing their outputs into workflow specifications, executable artifacts, and task configurations. For Challenge (2), artifact lifecycle control validates and revises artifacts before and after execution by combining deterministic platform evidence, LLM reasoning, execution feedback, and retrieved diagnostic knowledge to detect risks and explain failures. For Challenge (3), trace-driven skill evolution groups trajectories by skill and outcome into structured evidence and applies bounded LLM updates, using failures for repair and successes for compression. We evaluate SiriusDeliver through offline real-world cases and online production deployment. Offline, on 200 real-world delivery cases across 4 warehouse scenarios, SiriusDeliver improves the average end-to-end delivery success rate by 14.5 points (from 71.5% to 86.0%) over the strongest skill-augmented baseline and reduces token consumption by 30%, with ablations validating the contribution of each component. Online, a two-month deployment on Tencent Cloud WeData across 6 business teams and 4 warehouse scenarios served 3,600 monthly active users and supported 18,240 delivery sessions, achieving an 87.2% end-to-end success rate and a 73.5% autonomous submission rate. In a one-month A/B test, SiriusDeliver reduced delivery time from 228 to 23 minutes, engineer effort from 95 to 11 minutes, time to first artifact from 44 to 2.6 minutes, and manual intervention from 100% to 21%. To summarize, we make the following contributions: • We propose SiriusDeliver, a delivery automation agent that organizes warehouse skills hierarchically and generates platformsubmittable artifact bundles from business requirements. • We design artifact lifecycle control with pre- and post-execution diagnosis to detect risks, explain failures, and revise artifacts. • We introduce trace-driven skill evolution, which turns delivery trajectories into bounded skill updates for failure repair and successful-run compression.

Xie et al.

Offline

Business intent Intent Delivery Agent

Trace-driven Skill Evolution Update

Skill Library Manager Executor Memory

Approve Validate Revise

Artifacts

Deliver

Validation and deployment

Trajectory Grouping Bounded Evolution

Artifact Lifecycle Control Pre-execution diagnosis.

Memory

Post-execution diagnosis.

Tencent Cloud WeData Warehouse

Fail and Feedback

Figure 2: Overview of key techniques in SiriusDeliver.

• We evaluate SiriusDeliver on 200 real-world cases and in largeCloud WeData scale production deployment, achieving an Tencent 87.2% end-to-end success rate while reducing median delivery time and engineer effort by 89.9% and 88.4%, respectively.

2 Intent Related Work

Data warehouse platform

Data warehouse platforms. Modern cloud-native data platforms Delivery Agent (e.g., Snowflake [5], BigQuery [13], and Delta Lake [3]) provide highManager performance Executor storage and analytical query engines. However, these systems primarily expose low-level engine capabilities rather than Memory automating end-to-end task delivery. Engineers must still manually Skill Library interpret requirements, resolve dependencies, configure schedules, and diagnose execution failures. In contrast, SiriusDeliver operates as an automation layer above warehouse platforms to close the requirement-to-submission loop using execution feedback. LLM-based NL-to-SQL. NL-to-SQL maps natural language to executable queries via schema linking, in-context learning, and execution feedback [8, 17, 24, 29]. While effective for query synthesis, NL-to-SQL solves only a subset of production warehouse delivery. A deployable warehouse task requires not only valid SQL, but also workflow topology, scheduling parameters, permission configs, and cross-stage failure handling. Thus, NL-to-SQL alone cannot satisfy full-lifecycle production submission requirements. LLM agents for code and data systems. LLM agents extend LLMs to repository editing and data analytics (e.g., SWE-agent [28], AutoCodeRover [30], TaskWeaver [18], and Data Interpreter [12]), while platform assistants (e.g., Databricks Assistant [6], Snowflake Cortex [23], and Gemini in BigQuery [11]) assist in query generation. However, these systems mainly focus on code completion or interactive analysis. They do not address production task delivery, which demands dependency-aware orchestration, lifecycle artifact verification before/after platform submission, and trace-driven skill maintenance as platform practices evolve.

3

SiriusDeliver

Figure 2 shows the workflow of SiriusDeliver. Given a warehouse requirement, SiriusDeliver first invokes the delivery automation agent to select warehouse skills, retrieve context, and generate an artifact bundle with workflow specifications, executable artifacts, and task configurations. The bundle is then passed to artifact

SiriusDeliver: Automating Data Warehouse Delivery at Tencent

lifecycle control for pre-execution diagnosis, where evidence, dependencies, permissions, and configurations are verified before submission. Only approved artifacts are submitted to the warehouse platform by the agent. After execution, post-execution diagnosis analyzes platform feedback and logs, explains failures, and revises the artifacts. The revised artifacts are returned to the agent for resubmission, closing the delivery loop. Throughout this process, SiriusDeliver records skill calls, intermediate artifacts, diagnostic results, execution feedback, and final outcomes as delivery trajectories. Offline, trace-driven skill evolution transforms these trajectories into structured evidence, and proposes bounded skill updates for failure repair and successful-run compression.

3.1

Delivery Automation Agent

Warehouse delivery starts from a business requirement, but executable artifacts depend on progressively acquired context, including data semantics, workflow states, historical configurations, and platform constraints. A one-pass LLM generation process cannot reliably determine which evidence is missing, which skills should be invoked, or when the delivery state is complete. General tool-use agents can invoke tools, but they usually do not explicitly track warehouse-specific prerequisites, dependencies, and submission readiness. To address this, we propose a delivery automation agent that maintains a structured delivery state and orchestrates warehouse skills to incrementally construct a candidate artifact bundle. Delivery objective. The agent aims to produce a candidate artifact bundle for downstream lifecycle validation. Definition 1 (Artifact bundle). Given a user requirement 𝑞, historical workflow configurations 𝐻 , and platform context 𝑀, the delivery automation agent outputs 𝐴 = ⟨𝐺, 𝐶, 𝑃⟩, where 𝐺 is a workflow specification, 𝐶 is a set of node-level executable artifacts, and 𝑃 is a set of task configurations. 𝐺 describes the workflow structure, which can be a single node for single-task delivery or a DAG for workflow-level delivery. 𝐶 contains executable artifacts such as SQL, PySpark, Flink SQL, or Python programs. 𝑃 specifies schedules, dependencies, retry policies, runtime parameters, resource hints, permissions, and other platform options. This allows code, workflow structure, and configurations to be jointly validated before platform submission. Design insights. Our design follows four observations from production warehouse delivery. First, evidence is scenario-specific: synchronization tasks require connections, schemas, mappings, and incremental keys, whereas computation tasks require schemas, partition semantics, dependencies, and execution resources. Second, submission-critical fields must be clarified rather than inferred. Third, existing warehouse assets should be reused whenever possible. Fourth, historical configurations provide grounded references for team-specific schedules, resources, retries, naming, and dependencies. Accordingly, the agent identifies the delivery scenario, resolves critical fields, reuses existing workflows or nodes, and retrieves relevant configurations before artifact generation. Agent and warehouse skills. The delivery automation agent is a closed-loop framework that maintains the delivery state and orchestrates warehouse skills until all required artifacts are complete.

Conference’17, July 2017, Washington, DC, USA

Skill Library

Intent Yes Missing? No

Scenario

Artifact

Context

Platform

Retrieve

Planner Retrieve

Router

Invoke

Memory Long-term Knowledge

Artifacts

Short-term Context

Update

Executor

Yes

Terminator No

Memory

Figure 3: Orchestration loop of the delivery automation agent. The manager plans and routes warehouse skills, the executor invokes them and updates memory, and the loop iterates until a platform-submittable artifact bundle is produced.

A warehouse skill is a reusable executable capability with defined inputs, outputs, prerequisites, and side effects, implemented through metadata retrieval, deterministic scripts, LLM calls, historicaltask search, or platform APIs. Besides task outputs, a skill may emit structured hints, such as the inferred scenario, unresolved prerequisites, and candidate next steps, to guide subsequent orchestration. To reduce planning complexity, SiriusDeliver organizes warehouse skills into four conceptual layers. Scenario skills identify the delivery scenario, such as computation, synchronization, or workflow update. Context skills retrieve warehouse evidence, such as table metadata, workflow structure, and historical configurations. Artifact skills generate executable code, workflow specifications, and task configurations. Platform skills interact with the warehouse platform, including approved submission, status query, and feedback collection. This hierarchy narrows the candidate skill space at each step and makes implicit dependencies easier to resolve. Orchestration. Figure 3 illustrates the iterative delivery orchestration process. Given a user requirement, the agent checks whether the current delivery state contains sufficient information for the next step. It requests targeted clarification when critical inputs are missing; otherwise, it selects and executes warehouse skills based on the current state and historical context. Skill outputs are normalized and written to memory, enabling subsequent steps to use newly acquired metadata, workflow evidence, generated artifacts, and platform feedback. The loop terminates when 𝐺, 𝐶, and 𝑃 are complete, and the resulting candidate bundle is passed to artifact lifecycle control. After lifecycle approval, the agent invokes the corresponding platform skill to submit the bundle and collect execution feedback. The framework has three internal roles. • Manager tracks the delivery state and decides whether to request clarification, continue execution, or terminate with a complete candidate bundle. Its planner decomposes tasks, router selects skills, and terminator checks bundle completeness. • Executor resolves inputs from the requirement and memory, schedules prerequisite-dependent skills, invokes them, and records normalized outputs and execution traces. • Memory stores short-term session context, including metadata, workflow evidence, and skill outputs, as well as long-term knowledge from compressed trajectories, reusable configurations, and

Conference’17, July 2017, Washington, DC, USA

Artifacts

Pre-execution Diagnosis Evidence Grounding

States & Traces

Memory

Xie et al.

Hybrid Verification

Candidate Revision

Approve

Deliver

Revise

Post-execution Diagnosis Failure Extraction

Knowledge Retrieval

Fail Root-causebased Revision

WeData Warehouse

Figure 4: Artifact lifecycle control. Artifacts undergo preexecution verification before submission, and post-execution diagnosis transforms platform feedback into repair suggestions and revised artifacts for subsequent execution. execution patterns. It also retains platform feedback and diagnostic traces for subsequent reasoning, lifecycle control, and skill evolution.

3.2

Artifact Lifecycle Control

The delivery agent produces candidate rather than productionready artifact bundles. Even syntactically valid bundles may contain silent semantic errors, invalid dependencies, permission or scheduling conflicts, and inefficient execution patterns. Direct submission incurs costly trial and error, while pure LLM-based repair may overlook platform evidence, execution feedback, and historical diagnostic knowledge. To address these risks, artifact lifecycle control validates and revises artifacts in two complementary stages, as shown in Figure 4: pre-execution diagnosis verifies artifacts before submission using platform evidence, deterministic checks, and metadata-aware reasoning; post-execution diagnosis analyzes platform feedback, retrieves diagnostic knowledge, and revises failed artifacts for subsequent execution. Diagnosis coverage. Based on practical observations from warehouse development, we define diagnosis coverage for both presubmission risks and post-execution failures, as summarized in Table 3. Pre-execution diagnosis targets defects that can be detected before platform submission from generated artifacts, task configurations, metadata, and pre-compilation evidence. For example, it detects brute-force scans caused by missing partition predicates and business-intent mismatches that may produce silent data errors. Post-execution diagnosis targets failures exposed by platform feedback after compilation, dry-run, or execution. For example, it diagnoses permission exceptions from access-denied logs and table-structure exceptions from partition conflicts. A diagnosis item is the basic unit of this coverage. Each item specifies: (1) the issue description and its risk; (2) judgment conditions for reporting the issue; (3) required evidence, such as code locations, compiler outputs, metadata, logs, or retrieved cases; and (4) the execution workflow for collecting evidence and producing the diagnosis. This structure prevents unsupported or speculative reports and makes diagnosis results comparable across tasks. Pre-execution diagnosis. To detect both execution failures and silent warehouse defects before costly production platform runs, we adopt a hybrid strategy that combines deterministic platform evidence with LLM-based reasoning. Deterministic evidence grounds

each diagnosis in concrete, directly verifiable platform facts, such as current metadata, task configurations, declared dependencies, and pre-compilation results. LLM-based reasoning complements these facts by analyzing user intent, SQL semantics, and complex metadata-dependent risks that are difficult to reliably capture with rules alone. The resulting diagnostic process contains three steps. • Evidence grounding. We construct a canonical diagnosis input from codes, configurations, user requests, and conversation history. Missing parameters are inferred only when supported by existing evidence, following the priority of code and configurations, conversation history, and code comments. If required fields remain unresolved, we ask targeted follow-up questions. • Hybrid verification. We combine statement-level pre-compilation with metadata-aware reasoning. The process decomposes SQL into atomic statements, collects pre-compilation evidence, extracts table and field references, retrieves related metadata, and Feedback checks predefined diagnosis items. • Candidate revision. We report only defects that may cause execution failures, data errors, or severe performance issues. Each issue includes its type, severity, code evidence, and repair suggestion. When sufficient evidence is available, we prompt an LLM with the original artifact, verified evidence, and repair suggestion to generate a revised artifact. Otherwise, we preserve the original artifact and report unresolved issues. Post-execution diagnosis. To convert platform failures into structured repair evidence, we build on log-grounded diagnosis [4, 21, 27] by extracting failure signals from platform feedback and retrieving relevant diagnostic knowledge to guide artifact revision. This avoids direct repair from raw logs, reduces unsupported root causes, and keeps repair suggestions traceable. It contains three steps. • Failure extraction. We collect detailed execution logs, runtime errors, task states, and related platform metadata. We then extract concise, structured failure signals from platform feedback, such as the exception type, failed operator, error message, task stage, or platform component. These signals must be directly derived from platform feedback rather than inferred from the SQL alone. • Knowledge retrieval. We retrieve relevant cases, rules, and troubleshooting documents from a diagnostic knowledge base using the extracted failure signals. The retrieved evidence provides historical fixes and known platform behaviors that are difficult to infer from the current failure alone. • Root-cause-based revision. We diagnose the failure using the original artifact, extracted failure signals, and retrieved knowledge. Raw logs are used as auxiliary context, while localization focuses on the key failure signals. The output includes an exception category, a concise root-cause explanation, and an actionable repair suggestion. When the repair is sufficiently grounded, we prompt an LLM with the original artifact, localized evidence, and repair suggestion to generate a revised artifact for the next execution.

3.3

Trace-driven Skill Evolution

Warehouse skills cannot remain static after deployment, because platform conventions, task patterns, and failure cases continuously change in production. Online delivery further produces reusable

SiriusDeliver: Automating Data Warehouse Delivery at Tencent

User:

Agent:

I will first check metadata...

Session ID: c7f3e36d… Intent: Daily sync user_behavior Grouping Skills: <selected skills> Tools: <involved tools > Compress Artifacts: <generated artifacts >

Task Type: ETL Sync Execution Shape: # Turns: 7 # Tool Calls: 4 # Skills Used: 5

Performance:

# Tokens: 1.1M

…

…

…

[Tool call]:get_table_metadata [Log]: 32 columns retrieved Agent: we can reuse template...

Metadata

Strucured Trajectory

Trajectory

Sync user_behavior table daily

Conference’17, July 2017, Washington, DC, USA

Retrieve

Skill Library Scenario

Artifact

Context

Platform

Experience

Validation Guide Pass Update

LLM Fail Revise

Self External

Experience-guided Refinement

Figure 5: Trace-driven skill evolution. Raw trajectories are transformed into structured experience and metadata, which guide LLM-based skill refinement until validated updates are incorporated into the skill library. experience, including successful execution paths, failed attempts, diagnostic results, and platform feedback. However, these signals are scattered across long trajectories and cannot directly improve reusable warehouse skills. Manual maintenance is costly, while unrestricted LLM rewriting may introduce unstable changes to operational skills. To address this, we propose trace-driven skill evolution, an offline mechanism that converts delivery trajectories into bounded and reviewable skill refinements. As shown in Figure 5, this module proceeds in three steps. Evidence preparation groups and compresses raw trajectories into structured evidence. Experience-guided refinement proposes bounded updates from trajectory evidence and historical update records. Validation and deployment verify candidate skill updates before human-reviewed release and rollback-enabled production deployment. Design insights. Our design follows three observations from production deployment. First, failures often recur around specific skills, scenarios, or platform constraints and should therefore be aggregated at the skill level. Second, successful trajectories also reveal reusable configurations, stable execution patterns, and redundant context acquisition. Third, unrestricted LLM rewriting may propagate regressions across downstream submissions. Therefore, SiriusDeliver derives structured evidence from both outcomes, applies explicit, bounded refinements, and releases updates only after offline replay validation and mandatory human review. Evidence preparation. We convert long, multi-skill delivery sessions into structured evidence for skill-level maintenance. Each trajectory records the request, invoked skills and tools, intermediate artifacts, diagnoses, platform feedback, and final outcome, and may contribute to multiple skill groups. We group trajectories by skill and outcome: failures reveal missing preconditions, unsafe assumptions, and weak recovery or validation, whereas successes reveal reusable context, shorter execution paths, and redundant calls. For each selected session, we retain these key interactions in a compact trajectory and pair it with metadata on turns, skills, tool calls, latency, and errors, while removing verbose reasoning and truncating long outputs. This representation preserves the evidence needed for evolution with lower noise and processing cost. Experience-guided refinement. This stage converts prepared evidence into bounded skill refinements. Directly rewriting a skill

from raw traces is unstable, as the LLM may overfit to one failure or change unrelated behavior. We therefore retrieve two types of evolution experience before prompting the LLM. Self-guided experience retrieves previous updates of the same skill and their modification reasons, helping avoid repeated ineffective changes and maintain revision consistency. External-guided experience selectively retrieves relevant update records from related skills, providing transferable cross-skill repair or compression patterns. Given the structured trajectories and retrieved experience, the LLM proposes a candidate refinement with an update reason linked to supporting evidence. Repair-oriented refinements address recurring failures, while compression-oriented refinements remove redundant context, steps, or tool calls from successful executions. An update budget strictly limits the number of modified fields and rewritten text in each iteration, keeping the evolution process reviewable and preventing free-form skill rewrites. Validation and deployment. This stage prevents skill evolution from introducing regressions into production delivery. Candidate updates are not deployed immediately. We first run offline validation on historical trajectories. The validation set includes targeted failed trajectories, successful trajectories that should remain unaffected, and held-out trajectories from related skills. A refinement is accepted only if it resolves the targeted failure or efficiency issue without introducing regressions on previously successful cases. After offline validation, the candidate refinement and its update reason are sent for human review. The reviewer may accept, reject, or edit the refinement before deployment. For each accepted update, we record the previous skill version, the new version, the update reason, and the validation result. If later monitoring detects degraded performance, the deployed skill can be rolled back to the previous version. This protocol makes production skill evolution trace-driven, offline replay-validated, and safely reversible.

4

Experimental Evaluation

We evaluate SiriusDeliver through offline benchmarks and online production deployment by examining three questions: ① whether it improves end-to-end warehouse delivery over representative baselines; ② how artifact lifecycle control and trace-driven skill evolution affect reliability and efficiency; and ③ whether these gains translate into practical production benefits.

4.1

Experimental Settings

Dataset. We evaluate SiriusDeliver in both offline and online settings. For offline evaluation, we construct a frozen benchmark of 200 production delivery cases from Tencent Cloud WeData, stratified equally across four task types (50 each): real-time synchronization, offline synchronization, real-time computation, and offline computation. Each case pairs a user requirement with the platform context needed for execution. The benchmark snapshots are frozen before evaluation, and all skill-evolution trajectories are strictly disjoint from evaluation cases to prevent data leakage (details in Appendix A.2). Due to proprietary schemas and platform metadata, these cases cannot be released; we instead provide DataClawEval, a public companion benchmark covering general data-engineering https://github.com/Dicemy/DataClawEval

Conference’17, July 2017, Washington, DC, USA

Xie et al.

Table 1: Offline end-to-end delivery performance and ablation on 200 WeData cases (50 per task type). The upper block compares three coding products under two configurations (+ skills vs. + SiriusDeliver); ∆ is the average-success gain of + SiriusDeliver. The lower block ablates the three SiriusDeliver components on the best product (Claude Code). Success is mean ± 95% CI (𝑛=200); turns, time, and tokens are means. Bold: best; underline: second-best. Product

End-to-end success rate (%) ↑

Configuration RT Sync

Off. Sync

RT Comp.

Off. Comp.

Avg. success ↑

Turns ↓

Time (min) ↓

Tokens (K) ↓

Codex

+ skills + SiriusDeliver ∆

72.0 88.0 +16.0

68.0 86.0 +18.0

62.0 80.0 +18.0

56.0 74.0 +18.0

64.5 ± 6.6 82.0 ± 5.3 +17.5

9.8 6.1 −3.7

19.4 11.8 −7.6

151.6 98.4 −53.2

CodeBuddy

+ skills + SiriusDeliver ∆

76.0 90.0 +14.0

72.0 88.0 +16.0

66.0 82.0 +16.0

60.0 76.0 +16.0

68.5 ± 6.4 84.0 ± 5.1 +15.5

9.1 5.7 −3.4

17.6 10.8 −6.8

137.2 91.5 −45.7

Claude Code

+ skills + SiriusDeliver ∆

80.0 92.0 +12.0

74.0 90.0 +16.0

68.0 84.0 +16.0

64.0 78.0 +14.0

71.5 ± 6.2 86.0 ± 4.8 +14.5

8.4 5.4 −3.0

16.0 10.2 −5.8

124.0 86.9 −37.1

Ablation of SiriusDeliver (on Claude Code) w/o hierarchical skill orch. 72.0 w/o artifact lifecycle control 86.0 w/o pre-execution diag. 88.0 w/o post-execution diag. 88.0 w/o trace-driven skill evolution 90.0

72.0 84.0 86.0 86.0 88.0

80.0 74.0 76.0 78.0 82.0

74.0 68.0 70.0 72.0 76.0

74.5 ± 6.0 78.0 ± 5.7 80.0 ± 5.5 81.0 ± 5.4 84.0 ± 5.1

7.7 7.1 6.7 5.9 6.2

14.8 13.7 12.9 11.4 11.7

118.7 111.8 101.4 107.8 112.3

capabilities. The online evaluation uses long-running real-world production deployment logs, as detailed in Section 4.3. Baselines and product configurations. We compare three representative agentic coding products: CodeBuddy [25], Claude Code [2], and Codex [15]. Unless otherwise specified, they use their native default backbones: GLM-5.2, Claude Opus 4.8, and GPT-5.5, respectively. For each product, + skills adds the same base warehouse skill library, including identical skill interfaces, input schemas, executable implementations, and platform tools; + SiriusDeliver additionally enables hierarchical skill orchestration, artifact lifecycle control, and trace-driven skill evolution. Within each product pair, the backbone, product version, skill library, platform tools, execution environment, and refinement budget are fixed, so the paired difference measures the incremental benefit of SiriusDeliver. Because products retain different native backbones and internal agent implementations, their absolute results represent productconditioned systems rather than isolated comparisons of agent frameworks or backbone capabilities. Metrics. We use five metrics. End-to-end delivery success rate is the fraction of cases producing platform-accepted and executable warehouse tasks whose artifacts, dependencies, and configurations pass manual verification. First-pass success rate is the fraction completed by the initial submission without refinement. Turns counts agent reasoning and action rounds; Time measures wall-clock duration from receiving the requirement to the final submission result; and Tokens counts total LLM input and output tokens. For each product or backbone pair, we report the absolute percentage-point success improvement after enabling SiriusDeliver. Because products may use different tokenizers and internal inference procedures, turns, time, and tokens are interpreted primarily within matched pairs rather than by their absolute values across products. Implementation and controls. Each case permits one initial submission and at most two refinement attempts based on platform

feedback. For fair paired comparisons, configurations share task inputs, platform context, permissions, skill access, and refinement budgets. Agent states and temporary execution contexts are reset between cases to prevent cross-case information carryover. To prevent leakage, evolution trajectories are disjoint from evaluation cases and the resulting skill snapshot is frozen before evaluation. The backbone study uses the same 200 cases and CodeBuddy runtime, varying only the underlying model. Additional isolation and backbone controls are provided in Appendix A.1.

4.2

Offline Evaluation

End-to-end performance. Table 1 reports the end-to-end delivery performance on 200 offline cases. SiriusDeliver consistently outperforms all code-agent baselines across the four task types in both delivery effectiveness and efficiency. Compared with the best baseline, SiriusDeliver improves the average end-to-end delivery success rate by 14.5 points (from 71.5% to 86.0%), with the largest per-type improvement of 16.0 points on offline synchronization and real-time computation tasks. Meanwhile, SiriusDeliver reduces the average number of turns from 8.4 to 5.4, shortens the end-toend delivery time from 16.0 to 10.2 minutes, and decreases token consumption by 30% (from 124.0K to 86.9K). These results show that SiriusDeliver improves reliability without relying on larger reasoning budgets. The gain comes from structured skill orchestration, which reduces missing-context and wrong-skill-selection errors, and artifact lifecycle control, which prevents invalid artifacts from being repeatedly submitted and repaired through costly trial and error. Among the four task types, offline computation obtains the lowest success rate because it often involves complex SQL or PySpark semantics, multi-table dependencies, historical workflow reuse, partition management, and scheduling configurations. Nevertheless, SiriusDeliver still achieves a success rate of 78.0% on challenging offline computation and maintains consistently lower

SiriusDeliver: Automating Data Warehouse Delivery at Tencent

Conference’17, July 2017, Washington, DC, USA

Table 2: Paired evaluation of SiriusDeliver across representative LLM backbones on the same 200 cases (50 per task type), all run under the same CodeBuddy runtime. ∆ Success is the absolute gain of + SiriusDeliver over + Skills. Success (%) ↑

∆ Success (pp) ↑

First-pass (%) ↑

Turns ↓

Time (min) ↓

Tokens (K) ↓

Agent + Skills Agent + SiriusDeliver

58.0 74.0

– +16.0

42.0 56.0

11.2 7.6

21.8 14.6

162.0 116.0

GPT-5.5

Agent + Skills Agent + SiriusDeliver

68.0 80.0

– +12.0

54.0 64.0

9.2 6.4

17.6 12.2

130.0 100.0

Claude Opus 4.8

Agent + Skills Agent + SiriusDeliver

70.0 84.0

– +14.0

55.0 68.0

9.0 6.0

17.4 11.4

132.0 95.0

GLM-5.2

Agent + Skills Agent + SiriusDeliver

68.5 84.0

– +15.5

53.0 68.0

9.1 5.7

17.6 10.8

137.2 91.5

Evolved

Tie

Backbone

Configuration

DeepSeek V4 Flash

Original

4 3

Win

163

3.87 ↓25.5% 2.88

↓17.5% 134

100

0

20.8%

150

2 1

Lose

200

Turns

Tokens (M)

5

8.3% 70.9%

50

0

Figure 6: Effect of trace-driven skill evolution. Left: token usage and reasoning turns before and after skill evolution. Right: pairwise artifact quality comparison (Win/Tie/Lose) between the original and evolved skill libraries. overall turns, time, and token cost than the baselines, showing its effectiveness in the most difficult offline delivery setting. Effect of artifact lifecycle control. We evaluate artifact lifecycle control at both the pre-execution and post-execution stages. Pre-execution diagnosis detects 75.9% of real artifact issues with a false-positive rate of 9.5%, while introducing an average latency of only 6.1 s. Post-execution diagnosis directly repairs 73.0% of failed executions within one diagnosis-guided revision, with an average latency of 24.2 s. Together with the component ablations in Table 1, these results show that lifecycle control Artifact lifecycle control improves delivery reliability by blocking risky artifacts before submission and converting a large fraction of execution failures into successful repairs, with modest runtime overhead. Effect of skill evolution. We evaluate trace-driven skill evolution, split on offline production delivery cases into 50% training cases for deriving skill refinements and 50% held-out test cases for evaluation. For each held-out case, we replay the same request with the original and evolved skill libraries, and measure token usage, reasoning turns, and artifact quality. Artifact quality is assessed by pairwise LLM-as-judge comparison between artifacts generated before and after skill evolution, using position swapping and multimodel voting to reduce order bias and model-specific variance. The detailed judging protocol provided in Appendix A.3. As shown in Figure 6, evolved skills reduce token usage by 25.5% and reasoning turns by 17.5% on held-out cases For artifact quality, evolved artifacts are preferred or judged comparable in 79.2% of cases. These results suggest that skill evolution substantially reduces execution overhead without measurable degradation in final artifact quality. Effect of different LLM backbones. To examine whether the effectiveness of SiriusDeliver generalizes across different LLM

backbones, we conduct a paired evaluation using a controlled agent runtime. We adopt CodeBuddy as the agent runtime for this study, since it supports pluggable custom model backbones, whereas Codex and Claude Code do not expose such an interface to the best of our knowledge. For each backbone, we compare two configurations: an agent equipped with the shared warehouse skill library and the same agent augmented with the three components of SiriusDeliver. The backbone is fixed within each pair, while the runtime, tools, prompts, context budget, refinement budget, and execution environment are kept identical. We evaluate representative enterprise-approved models spanning different capability levels and model families. As shown in Table 2, enabling SiriusDeliver consistently improves end-to-end delivery success across all evaluated backbones. The paired success gains range from 12.0 to 16.0 percentage points. Notably, weaker backbones benefit more from SiriusDeliver (e.g., a 16.0-point gain on DeepSeek V4 Flash versus 12.0 points on the stronger GPT-5.5), as the deterministic platform evidence and structured orchestration compensate for limited reasoning capability. Since all backbones are evaluated under the same CodeBuddy runtime, the reported numbers are not directly comparable to the product-conditioned results in Table 1, which use each product’s native runtime and backbone. SiriusDeliver also reduces the number of reasoning turns, wall-clock time, and token consumption for all backbones. These results indicate that the effectiveness of SiriusDeliver does not depend on a particular LLM backbone and that its structured delivery workflow provides complementary benefits across different model capability levels. Ablation studies. On the best-performing product, Claude Code, we remove one SiriusDeliver component at a time while holding the others fixed; Table 1 shows that all three components contribute, with the full model corresponding to the Claude Code + SiriusDeliver row. Removing hierarchical skill orchestration causes the largest degradation, an 11.5-point success drop (86.0%→74.5%), concentrated on synchronization tasks where missing metadata, workflow context, or dependency evidence leads to invalid plans. Removing artifact lifecycle control reduces success by 8.0 points (86.0%→78.0%), mainly on computation tasks with latent correctness and performance risks; its pre- and post-execution sub-ablations both contribute to this drop. Removing trace-driven skill evolution causes a relatively smaller 2.0-point success drop but increases total token consumption from 86.9K to 112.3K, showing that evolution primarily improves efficiency by compressing redundant context acquisition and intermediate reasoning.

Conference’17, July 2017, Washington, DC, USA

Xie et al.

18,240

2

Delivery sessions

Months

Offline comp.

Real-time comp.

Offline sync.

Real-time sync.

75%

87.2%

250

73.5% 61.0%

50%

25% 41%

0%

25%

27%

50%

19%

75%

13%

100%

0%

Figure 7: Online production deployment of SiriusDeliver. Left: deployment scale and workload composition. Right: production delivery metrics.

4.3

Online Deployment

We deployed SiriusDeliver on Tencent Cloud WeData and evaluated it in a real production environment. SiriusDeliver provides a web interface where engineers submit warehouse requirements, inspect generated artifacts, and track delivery progress; an example is shown in Figure 13 in Appendix C.2. The deployment covers 6 business teams and 4 warehouse task types: offline computation, offline synchronization, real-time computation, and real-time synchronization. These tasks support core business scenarios such as WeChat Pay, advertising, and security risk control. Online results. As shown in Figure 7, during a two-month deployment period, SiriusDeliver served 3,600 monthly active users and supported 18,240 delivery sessions. The four task types account for 41%, 27%, 19%, and 13% of online requests, respectively. Across all sessions, SiriusDeliver achieved an end-to-end delivery success rate of 87.2% and an autonomous submission rate of 73.5%, where autonomous submission means that artifact generation, validation, and submission are completed without manual modification of delivery artifacts. Artifact lifecycle control was triggered in 61% of sessions: pre-execution diagnosis detected 4,120 risky artifacts before submission, and post-execution diagnosis repaired 2,890 platform execution failures. Trace-driven skill evolution produced 128 human-approved skill updates, including 89 repair-oriented updates and 39 compression-oriented updates. Based on 3,180 user ratings, SiriusDeliver received an average score of 4.4 out of 5.0. A/B test. We conducted a one-month A/B test on real online delivery requests. We recruited 70 experienced data warehouse engineers who regularly perform warehouse task delivery and randomized them at the user level into a control group and a treatment group, with 35 users in each group. The randomization was stratified by task type and business team. The control group followed the existing manual workflow, while the treatment group used SiriusDeliver. During the experiment, the treatment and control groups completed 718 and 682 delivery tasks, respectively. Since experienced engineers already achieve high final success rates, the A/B test focuses on delivery cost. As shown in Figure 8, compared with the control group, SiriusDeliver reduced the median end-to-end delivery time from 228 minutes to 23 minutes, corresponding to an 89.9% reduction with a 95% confidence interval of [86.4%, 92.1%]. It also reduced net engineer effort from 95 to 11 minutes, time to first executable artifact from 44 to 2.6 minutes, and manual intervention rate from 100% to 21%. The treatment group required 3.2 interaction rounds on average, indicating that most sessions were completed without repeated clarification. Meanwhile,

100%

Ours

200

95↓88.4%

100

0

100%

98.1% 97.6%

75%

150

50

Delivery Autonomous Lifecycle success submission trigger

Human 228↓89.9%

Rate

3,600 Monthly active users

Minutes

100%

6 Business teams

44↓94.1% 23

Delivery time

11

2.6

Engineer First-artifact effort latency

50% 25% 0%

21%

Manual intervention

Final delivery success

Figure 8: Online A/B test comparing the manual workflow and SiriusDeliver. Left: delivery efficiency metrics. Right: human intervention and final delivery quality. final delivery success rates remained comparable between the treatment and control groups, 97.6 % vs. 98.1%, suggesting that efficiency gains did not compromise production submission quality. Production insights. The deployment yields three insights. First, gains in first-artifact latency, engineer effort, and manual intervention show that delivery automation should coordinate the end-toend delivery process rather than focus on code generation alone. Second, lifecycle control is essential for production readiness: its activation in 61% of production sessions confirms the practical need to verify artifacts before submission and ground repairs in execution feedback. Third, the 128 approved updates show that warehouse skills are continually evolving operational assets; repair-oriented updates improve robustness, while compression-oriented updates reliably remove redundant context and interactions. Case studies. Appendix B presents four cases that complement the aggregate results with production traces. In the pre-execution case, SiriusDeliver combines SQL semantics with table metadata to block an excessive partition scan after discovering approximately 1,179 TB in the first matching hourly partition. In the post-execution case, it grounds the repair of a PySpark runtime failure in execution logs, artifact context, and retrieved knowledge, then generates a revised artifact for safe retry. The skill-evolution case abstracts recurring empty-result and file-conflict failures into reusable preventive rules. The end-to-end case follows an offline computation task from intent routing through artifact generation, validation, execution, and feedback-driven resubmission. These cases demonstrate how SiriusDeliver integrates generation, lifecycle control, and skill evolution into a closed-loop production workflow.

5

Conclusion

We presented SiriusDeliver, an end-to-end production warehouse delivery system that goes beyond general coding agents through dependency-aware skill orchestration, lifecycle-aware artifact validation and repair, and trace-driven skill evolution. Offline evaluations demonstrate improved delivery performance, effective diagnosis and repair, and reduced reasoning overhead. A two-month deployment on Tencent Cloud WeData served 3,600 monthly active users and supported 18,240 delivery sessions, achieving an 87.2% end-to-end success rate and a 73.5% autonomous submission rate. A production A/B test substantially reduced delivery time and engineer effort while maintaining comparable final delivery success, demonstrating that production warehouse automation requires coordinated planning, explicit artifact control, and experience-driven skill maintenance beyond code generation.

SiriusDeliver: Automating Data Warehouse Delivery at Tencent

References [1] Anthropic. 2024. Introducing Claude 3.5 Sonnet. https://www.anthropic.com/ index/claude-3-5-sonnet. Accessed: 2026-06-30. [2] Anthropic. 2026. Claude Code overview. https://docs.anthropic.com/en/docs/ claude-code/overview. Accessed: 2026-06-30. [3] Michael Armbrust, Tathagata Das, Sameer Paranjpye, Reynold Xin, Shixiong Zhu, Ali Ghodsi, Burak Yavuz, Mukul Murthy, Joseph Torres, Liwen Sun, Peter Boncz, Mostafa Mokhtar, Herman Van Hovell, Adrian Ionescu, Alicja Luszczak, Michal Switakowski, Takuya Ueshin, Xiao Li, Michal Szafranski, Pieter Senster, and Matei Zaharia. 2020. Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores. Proc. VLDB Endow. 13, 12 (2020), 3411–3424. [4] Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, et al. 2024. Automatic root cause analysis via large language models for cloud incidents. In Proceedings of the Nineteenth European Conference on Computer Systems. 674–688. [5] Benoît Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, Allison W. Lee, Ashish Motivala, Abdul Q. Munir, Steven Pelley, Peter Povinec, Greg Rahn, Spyridon Triantafyllis, and Philipp Unterbrunner. 2016. The Snowflake Elastic Data Warehouse. In Proceedings of the 2016 International Conference on Management of Data. ACM, 215–226. [6] Databricks. 2026. Databricks assistant: your context-aware ai assistant. https: //www.databricks.com/product/databricks-assistant. Accessed: 2026-06-30. [7] DeepSeek-AI. 2026. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948 [cs.CL] doi:10.1038/s41586-025-09422-z [8] Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-sql empowered by large language models: a benchmark evaluation. Proceedings of the VLDB Endowment (2023). [9] Gemini Team. 2024. Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530 [cs.CL] https://arxiv.org/abs/ 2403.05530 [10] GitHub. 2026. GitHub Copilot: your AI pair programmer. https://github.com/ features/copilot. Accessed: 2026-06-30. [11] Google Cloud. 2026. Write queries with gemini Aassistance in bigquery. https: //cloud.google.com/bigquery/docs/write-sql-gemini. Accessed: 2026-06-30. [12] Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, et al. 2025. Data interpreter: an llm agent for data science. In Findings of the Association for Computational Linguistics: ACL 2025. 19796–19821. [13] Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, and Theo Vassilakis. 2010. Dremel: Interactive Analysis of Web-Scale Datasets. In Proceedings of the VLDB Endowment, Vol. 3. 330–339. [14] OpenAI. 2024. GPT-4 technical report. arXiv:2303.08774 [cs.CL] https://arxiv. org/abs/2303.08774 [15] OpenAI. 2025. OpenAI codex. https://openai.com/codex/. Accessed: 2026-06-30. [16] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Gorilla: Large Language Model Connected with Massive APIs. International Conference on Neural Information Processing Systems (2024). [17] Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: decomposed incontext learning of text-to-sql with self-correction. International Conference on Neural Information Processing Systems (2023). [18] Bo Qiao, Liqun Li, Xu Zhang, Shilin He, Yu Kang, Chaoyun Zhang, Fangkai Yang, Hang Dong, Jue Zhang, Lu Wang, Minghua Ma, Pu Zhao, Si Qin, Xiaoting Qin, Chao Du, Yong Xu, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2023. Taskweaver: a code-first agent framework. arXiv preprint arXiv:2311.17541 (2023). [19] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: facilitating large language models to master 16000+ real-world APIs. International Conference on Learning Representations (2024). [20] Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: language models can teach themselves to use tools. International Conference on Neural Information Processing Systems (2023). [21] Yu Shen, Shiyang Liu, Qihang He, Yihang Cheng, Haining Xie, Zhiming He, Huahua Fan, Xianzhi Tan, Teng Ma, Shaoquan Zhang, et al. 2026. SiriusHelper: an LLM agent-based operations assistant for big data platforms. arXiv preprint arXiv:2605.00043 (2026). [22] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. International Conference on Neural Information Processing Systems (2023). [23] Snowflake. 2026. Cortex analyst. https://docs.snowflake.com/en/user-guide/ snowflake-cortex/cortex-analyst. Accessed: 2026-06-30. [24] Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. Chess: contextual harnessing for efficient sql synthesis.

Conference’17, July 2017, Washington, DC, USA

arXiv preprint arXiv:2405.16755 (2024). [25] Tencent Cloud. 2026. Tencent cloud code assistant CodeBuddy: product overview. https://www.codebuddy.cn. Accessed: 2026-06-30. [26] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An Open-Ended Embodied Agent with Large Language Models. Transactions on Machine Learning Research (2023). [27] Weiyuan Xu, Juntao Luo, Tao Huang, Kaixin Sui, Jie Geng, Qijun Ma, Isami Akasaka, Xiaoxue Shi, Jing Tang, and Peng Cai. 2025. LogSage: an LLM-based framework for CI/CD failure detection and remediation with industrial validation. IEEE/ACM International Conference on Automated Software Engineering (2025). [28] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: agent-computer interfaces enable automated software engineering. In International Conference on Neural Information Processing Systems. [29] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 3911–3921. [30] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. [31] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. ExpeL: LLM agents are experiential learners. Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence (2024).

Conference’17, July 2017, Washington, DC, USA

A Experimental Details A.1 Detailed Evaluation Controls Within each product pair, both configurations are initialized from the same base warehouse skill library. Agent states and temporary execution contexts are reset between cases, and no test-case outcome or platform feedback is used to update the skills or execution policy of subsequent cases. Historical trajectories used for skill evolution are disjoint from evaluation cases, and the resulting skill-library snapshot is frozen before evaluation. For the backbone study, all LLMs are accessed through direct APIs and integrated into the same CodeBuddy runtime on the same 200 cases. The prompt template, skill-library snapshot, tool access, context budget, refinement budget, and execution environment are fixed within each pair; only the underlying model differs. Backbone selection criteria are described in Section 4.2.

A.2

Private Benchmark Construction and Public Companion Benchmark

Case collection and stratification. The private benchmark was constructed from production warehouse delivery requests collected from Tencent Cloud WeData during the first half of 2026. The unit of evaluation is one delivery request together with the platform context required to complete it, including the relevant table schemas, workflow metadata, task configurations, historical references, and execution permissions. We retained requests for which the required platform context and the final delivery outcome were available, and excluded incomplete, cancelled, and duplicated requests. Eligible requests were grouped into four task types: real-time synchronization, offline synchronization, real-time computation, and offline computation. For each task type, we applied stratified selection and manually curated 50 representative cases, yielding 200 cases in total. To reduce selection bias, the curated cases were chosen to span different difficulty levels, artifact types (e.g., SQL, PySpark, and Flink SQL), and failure modes, rather than favoring easily solved requests. The same fixed cases and platform-context snapshots were used by all compared methods throughout all evaluations. Deduplication and leakage control. Before evaluation, we removed exact-duplicate delivery requests so that no request appears more than once in the benchmark. The historical delivery trajectories used for trace-driven skill evolution are disjoint from the 200 evaluation cases. The skill-library snapshot, task inputs, metadata, and platform context were frozen before evaluation, and no evaluation outcome or platform feedback was used to update the skills or execution policy of subsequent cases. Success adjudication. A case is counted as successful only when the generated artifact bundle is accepted by the platform, is executable in the designated environment, and satisfies the required artifact, dependency, and configuration constraints. The final outputs were independently verified by three experienced warehouse engineers using a predefined checklist, and disagreements were resolved through adjudication. For each task type and configuration, we report integer success counts together with the corresponding success rates for cross-method performance comparison. Public companion benchmark. The private benchmark cannot be publicly released because the cases contain proprietary schemas,

Xie et al.

workflow configurations, platform metadata, operational traces, and business-sensitive information. To support public research on related data-engineering agent capabilities, we separately provide DataClawEval, a public companion benchmark of 100 executable tasks across PySpark, MySQL, HiveSQL, PrestoSQL, and Flink SQL. Each task provides a sanitized request, an initialized data environment, a ground-truth solution, an automated grader, and a Dockerbased execution harness. DataClawEval complements rather than reproduces the private benchmark: it covers offline and real-time computation tasks, but does not model WeData-specific synchronization, scheduling, permission, platform-submission, or lifecyclediagnosis workflows. The DataClawEval tasks do not contain any of the 200 private evaluation cases and are not used to produce the results reported in this paper under any setting.

A.3

LLM-as-Judge Protocol for Skill Evolution

We use a pairwise LLM-as-judge protocol to assess whether skill evolution affects artifact quality. For each case, we compare two artifacts generated from the same request: one using the original skill library and the other using the evolved skill library. The judge is asked to select the better artifact according to functional correctness, dependency consistency, configuration validity, and production readiness. To reduce position bias, each case is evaluated twice by the same judge with swapped artifact order. If the two judgments are consistent, we record the result as a win for the corresponding artifact. For example, if the judge selects artifact 𝐴 in both the original order and the swapped order, the case is recorded as an 𝐴 win for this judge. If the two judgments are inconsistent, the case is recorded as inconsistent for subsequent analysis. We use three LLM judges: Gemini 3.8 Flash, GPT-5.5, and Claude Opus 4.8. Each judge independently produces one of three outcomes for each case: 𝐴 win, 𝐵 win, or inconsistent. We then aggregate the three judge outcomes by majority voting. If two or more judges select the same artifact, the result is recorded as a win for that artifact. For example, outcomes (𝐴, 𝐴, 𝐵) are recorded as an 𝐴 win. If no artifact receives at least two votes, or if the votes are dominated by inconsistent outcomes, the result is recorded as inconsistent. For example, outcomes (𝐴, 𝐵, inconsistent) are recorded as inconsistent.

B Case Studies B.1 Case Study of Pre-execution Diagnosis Figure 9 presents a case in which a syntactically valid SQL query remains unsafe for production execution. The query contains two overlapping lower-bound predicates on partition_time, whose conjunction is equivalent to partition_time >= 2026062423. Predicate simplification alone does not reduce the effective scan range. Pre-execution diagnosis further retrieves table metadata and finds that the first matching hourly partition alone contains approximately 1,179 TB of data. Combined with the subsequent GROUP BY aggregation, this introduces a high risk of excessive shuffle, memory pressure, and OOM failure. Since no semanticspreserving rewrite can sufficiently reduce the scanned data, lifecycle control removes the redundant predicate but does not approve the artifact for direct submission. Instead, it recommends narrowing the time range, batching the query by time or business dimensions, or revising the table partition strategy.

SiriusDeliver: Automating Data Warehouse Delivery at Tencent

Conference’17, July 2017, Washington, DC, USA

Table 3: Diagnosis coverage in SiriusDeliver. Pre-execution diagnosis detects risks before task submission, while postexecution diagnosis structures feedback from platform logs. Stage

Issue type

Main signal

Risk or feedback

Pre-execution

Syntax error Brute-force scan Cartesian product Implicit conversion Empty-table read Basic logic defect Where-clause logic defect Business-intent mismatch

SQL parser or compiler evidence Missing or ineffective partition pruning Missing or incomplete join conditions Type mismatch in filters, joins, or expressions Empty source table or missing partition Aggregation, join, null, or dedup logic Boundary, null, or predicate logic SQL semantics vs. user intent

Invalid SQL or dialect-specific misuse Excessive scan cost and task delay Data explosion and resource waste Semantic drift or pruning failure Silent empty outputs or zero metrics Silent data distortion Incorrect filtering results Correct execution but wrong business meaning

Post-execution

Syntax exception UDF exception Permission exception System environment exception Data-source read exception Table-structure exception

Parse, alias, function, join, or subquery errors UDF runtime, dependency, or registration errors Access-denied or unauthorized logs Timeout, OOM, network, or service failures Missing file, corrupted data, or source failure Missing table, column, partition, or schema mismatch

Root cause for failed compilation/execution UDF repair or dependency fix Permission application or policy check Runtime or resource-level repair Source availability or format repair Metadata or schema repair

Case: Pre-execution Diagnosis for an Excessive Partition Scan

B.2

Input: Candidate Artifact

Figure 10 presents a case in which a PySpark task fails during runtime with an IndexOutOfBoundsException. The failure log reports an invalid access to index 106 from a container with size 0 during ORC data reading. Post-execution diagnosis first extracts localized failure signals from execution logs and retrieves relevant troubleshooting knowledge from historical cases. Based on the artifact context, failure signals, and retrieved knowledge, the diagnosis examines potential causes including corrupted ORC files, invalid reading logic, and empty input partitions.The evidence indicates that abnormal ORC data is the most likely root cause. The system then generates repair guidance to replace corrupted files, validate partition inputs, and add safeguards against empty data access. Using the localized evidence and repair guidance, an LLM revises the original artifact for the next execution attempt. This case illustrates that execution logs alone are insufficient for reliable failure repair. Effective post-execution diagnosis requires combining platform feedback, artifact context, and historical knowledge to localize root causes and generate grounded revisions rather than directly modifying artifacts based on raw errors.

select context.sdk_version,count(0) as cnt from ams_log_iceberg::etl_pageview_mixer where partition_time>=2026062422 and partition_time>=2026062423 and site_set = 15 and action_info.request_info.ad_pull_mode=1 and site_info.site_id=80503886999934 group by context.sdk_version Status: Syntax-valid but not yet approved for submission.

Evidence Grounding Predicate simplification: partition_time ≥ 2026062422 ∧ partition_time ≥ 2026062423 ≡ partition_time ≥ 2026062423 Platform metadata: The table is hourly partitioned, and the first matching partition alone contains approximately 1,179~TB of data. Execution operators: Large partition scan followed by a GROUP BY aggregation.

Hybrid Verification • • • •

The redundant predicate does not restrict the effective scan range. The first matching partition alone contains = 1,179 TB GROUP BY increases shuffle and memory pressure. No semantics-preserving rewrite can sufficiently reduce the scanned data.

Case Study of Post-execution Diagnosis

Excessive Partition Scan (High Risk of OOM Failure)

Candidate Revision The redundant predicate is removed while preserving the original semantics. Because the resource risk remains unresolved, direct submission is blocked. The user is advised to narrow the time window, batch the query by time or business dimensions, or revise the table partition strategy.

Figure 9: Case study of pre-execution diagnosis for an excessive partition scan. By combining SQL predicate analysis with table metadata, SiriusDeliver identifies a high risk of resource exhaustion, blocks direct submission, and returns a semantics-preserving revision with mitigation guidance.

This case illustrates that syntax validation alone is insufficient for production warehouse delivery. Reliable diagnosis requires jointly reasoning over query semantics, table metadata, and execution characteristics, while avoiding unsupported rewrites when the underlying risk cannot be safely resolved.

B.3

Case Study of Trace-driven Skill Evolution

Figure 11 presents a case in which execution traces are converted into reusable skill updates. The trajectories contain two recurring failures: an empty DataFrame before insertInto() that later triggers a NoSuchElementException, and an upload_file(’etl_task.py’) operation that fails because the target file already exists. Rather than treating them as isolated fixes, trace-driven skill evolution abstracts them into auditable, reusable delivery rules across similar tasks, including empty-result checks before writes, unique file naming, and platform-side existence checks before upload. When the evolved skill is reused, these failures are prevented before platform submission. The agent detects empty intermediate results, revises the transformation before writing, and selects non-conflicting filenames after checking existing platform files. This case illustrates how SiriusDeliver turns failed trajectories

Conference’17, July 2017, Washington, DC, USA

Xie et al.

Case: Trajectory-Driven Skill Refinement and Reuse

Case: Post-execution Diagnosis for Failed PySpark Task Input: Platform Feedback

Input: A Group of Trajectories Case 1 : DWS aggregation task failed. Error on insertInto() With empty data Tool Call: ‘spark.sql(…)’(0 rows returned) Log: NoSuchElementException: None.get

ERROR Executor: Exception in task 1453.0 in stage 32.0 (TID 1453) Java.lang.IndexOutOfBoundsException: Index: 106. impl.RecordReaderInpl.next(RecordReaderInpl.java: 106) mapred.OrcRecordReader.next(OrcRecordReader. java: 215) FileScanRODSSanon$1.hasNextFileScanROD.scala:116)

Case 2 : DWD cleaning task submission failed. File name conflict on platform Tool Call:‘upload_file(‘etl_task.py’)’ Log: Error 8001 - File ‘etl_task.py’ exists.

Failure Extraction • • • •

The task fails with an IndexOutOfBoundsException. Index 106 is accessed from a container of size 0. The failure occurs during PySpark ORC data reading. Raw logs are retained as auxiliary context.

Knowledge Retrieval • • • •

Historical cases associate this failure with corrupted ORC files. Similar errors may also arise from out-of-bound index access. Empty or invalid partitions can produce zero-sized inputs. Troubleshooting documents provide corresponding repair procedures.

Experience-guided Skill Refinement LLM Refinement Output(Key Changes): 1.Stronger Defensive Code Standard: ->Add mandatory empty DataFrame checks before write operations. 2.Unified Naming Strategy Rules: ->Enforce unique naming with timestamp or task-id suffix. 3.Pre-check Guidance: ->Add file existence check before upload.

Next Execution With Updated skill (One-Shot Success) Case 1 : DWS aggregation task success. Tool Call: spark.sql(...) (0 rows returned) Log: Pre-check, Empty DataFrame detected. Tool Call: revise_transformation(...) Tool Call: spark.sql(...) (128734 rows returned) Log: Write completed successfully.

Root-cause-based Revision • • • •

Corrupted ORC files should be repaired or replaced. Empty inputs and out-of-bound accesses should be explicitly guarded. Partition fields should be validated to avoid invalid or empty partitions. The original artifact is revised using the evidence and repair guidance.

Case 2: DWD cleaning task submission success. Tool Call: platform.list_files()(result: ["etl_task.py", "daily_job.py", ...]) Tool Call: upload_file("etl_task_20260723.py") Log: Submission completed successfully.

Output • •

A revised artifact is generated for the next execution attempt. The output includes the exception category, root-cause explanation, and actionable repair guidance.

Figure 10: Case study of post-execution diagnosis for a PySpark runtime failure. By combining execution log analysis with diagnostic knowledge, SiriusDeliver identifies the root cause, provides actionable repair guidance, and generates a revised artifact for subsequent execution. into preventive execution knowledge, reducing repeated delivery failures through reusable skill refinement.

B.4

Case Study of End-to-end Delivery

Figure 12 presents an end-to-end delivery case for an offline computation task. The user requests a daily detail table for cross-day Ray notebook executions, requiring day-boundary splitting, runtime calculation, minute-level GPU metrics, and insertion into the target partition dt=’20260507’. The router identifies the request as a new offline SQL task, selects Spark-compatible SuperSQL, and orchestrates schema retrieval, preview-data acquisition, SQL generation, validation, resource recommendation, and platform operations. Based on the resolved context, SiriusDeliver generates a workflow specification, SuperSQL code, and task configuration. Preexecution diagnosis checks schema consistency, partition semantics, referenced fields, target mappings, and submission settings before the artifact is uploaded to WeData for trial execution. When the trial exposes a column-count mismatch and invalid arithmetic over STRING values, post-execution diagnosis removes the partition column from the SELECT projection, adds explicit casts, and resubmits the revised artifact. This case illustrates how SiriusDeliver connects intent routing, context planning, artifact generation, lifecycle diagnosis, platform execution, and feedback-driven repair into a single production delivery loop.

Figure 11: Case study of trajectory-driven skill refinement and reuse.

C Details C.1 Agent and skills The delivery automation agent operates over a reusable library of warehouse skills. A skill is the basic executable unit for warehouse delivery, consisting of (1) a description of its functionality and applicable scenarios; (2) a specification of input arguments extracted from the user request or context; (3) execution code, which may involve LLM calls, platform APIs, retrieval modules, or deterministic scripts; and (4) optional dependencies that determine execution order. A skill may emit structured planning hints (e.g., inferred scenarios, remaining prerequisites, or downstream candidates) to support subsequent planning.

C.2

Web Interface

Figure 13 shows the production interface of SiriusDeliver, where engineers submit warehouse requirements, inspect generated artifacts, and track delivery progress. The interface provides a unified workspace for initiating new delivery conversations, revisiting historical sessions, and selecting common warehouse task scenarios. Within each session, engineers can review the generated SQL, workflow configurations, and diagnostic feedback, provide clarifications when required information is missing, and monitor the status of validation, submission, and subsequent execution. This interface operationalizes the human-in-the-loop delivery workflow by keeping requirement interaction, artifact inspection, and platform feedback within a single production-facing entry point.

SiriusDeliver: Automating Data Warehouse Delivery at Tencent

Conference’17, July 2017, Washington, DC, USA

Case: End-to-end Delivery of an Offline Computation Task 1. Business Intent and Task Routing

4. Pre-execution Diagnosis

Business Requirement: • Generate a daily detail table for cross-day Ray notebook executions. Split each execution at natural-day boundaries, calculate instance runtime, code runtime, and resource waiting time, attach minute-level GPU metrics, and write the results into partition dt='20260507’

Evidence Grounding: The generated code is checked against table schemas, preview data, partition semantics, and the task configuration. Hybrid Verification: Syntax, referenced fields, target mapping, and submission settings are jointly verified. Decision: Pass：Submit for a platform trial. Fail：Revise and repeat pre-execution diagnosis.

Task Router: New offline SQL task using Spark-compatible SuperSQL.

2. Context Planning and Skill Orchestration Context Acquisition: Schemas and previews are retrieved for five source tables and the target table. Field definitions and example values are combined into Light Schemas. Skill Orchestration: The delivery agent invokes metadata retrieval, SQL generation, resource recommendation, validation, and platform-operation skills according to their prerequisites. Resolved Context: Source fields, partition semantics, table mappings, execution identity, and target write mode.

5. Platform Trial and State Polling Platform Submission: The code is uploaded to WeData, a new offline task is created from the validated configuration, and a trial execution is submitted. States & Traces: SiriusDeliver continuously polls the platform execution state after task submission. COMPLETED → Request publication confirmation. FAILED → Trigger post-execution diagnosis. RUNNING → Continue polling.

3. Artifact Generationand Configuration Executable Artifact: SuperSQL is generated to split cross-day intervals, aggregate execution durations, resolve service-to-pod mappings, and join minute-level GPU metrics. INSERT OVERWRITE TABLE target PARTITION (dt='20260507') WITH daily_split_spans AS (...), trace_time_metrics AS (...) SELECT ... Task Configuration: A new single-node offline task is configured with the Spark runtime, daily schedule, retry policy, execution resource, and static target partition. Candidate Bundle: Workflow specification + SuperSQL code + task configuration.

6. Post-execution Diagnosis Failure Extraction: Platform feedback reports a column-count mismatch and invalid arithmetic on STRING values. Knowledge Retrieval: Relevant historical cases, diagnostic rules, and troubleshooting knowledge are retrieved. Root-cause-based Revision: Remove dt from the SELECT projection, explicitly cast replicas and num_gpu, and resubmit the artifact. Subsequent Execution: The revised artifact is revalidated, resubmitted, and monitored again.

Figure 12: Case study of end-to-end delivery of an offline computation task.

Figure 13: Production web interface of SiriusDeliver.

Related documents

• REAL-TIME DATA STREAM PROCESSING WITH KAFKA- DRIVEN AI MODELS#90020
• Unfair by design: eBPF-based scheduling of mixed database workloads#155413
• DBRepro: Automated Database Synthesis via a Hybrid Constraint-Solving Approach for Reproducing Slow Queries#781110
• DBRepro: Automated Database Synthesis via a Hybrid Constraint-Solving Approach for Reproducing Slow Queries#779735
• Efficient Benchmarking in Production: A Study of an Evolving LLM Agent#1035675