SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking

SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking Lyndon Shi* , Nic Crouch* , Yi Hu* , Zihao Ye* , Choden Konigsmark, Eric Vandenberg, Dionysios Logothetis, Kuba Niemiec, Krzysztof Gadomski, Yuxiong He

arXiv:2611.00449v1 [cs.DB] 30 Oct 2026

Snowflake Inc.

Abstract—SpBench is an agent-native framework for crossengine lakehouse benchmarking, in production use for benchmarking Snowpark, Snowpark Connect for Apache Spark, and other execution environments across standardized suites — such as TPCx-BB and TPC-DI — and custom benchmarks distilled from production workload patterns. It runs continuously across a broad portfolio of workloads, guiding the performance roadmap and detecting regressions for production-scale data engines. Benchmarking has historically been an expensive investment requiring continuous cross-functional collaboration and domain expertise, making it difficult to obtain actionable performance insights. SpBench provides (i) a typed, code-generated benchmark substrate that compiles platform-agnostic benchmark definitions into fair, comparable platform-specific implementations; (ii) a structured-telemetry contract recording every run as queryable fact and dimension data, letting LLM agents query results deterministically; and (iii) an agent-native harness of three composable skills to expedite building benchmarks and analyzing results. This agent-native architecture enabled the rapid development of more than 400 benchmark queries that run on a regular cadence to detect performance regressions; catching these has directly led to improvements in production. Index Terms—lakehouse benchmarking; LLM agents; foundation models for big data; benchmarking frameworks

I. I NTRODUCTION Modern cloud data platforms expose different execution engines for the same logical workload. On Snowflake this includes the Snowpark API for native execution [1] and Snowpark Connect for Apache Spark [2]; SpBench additionally benchmarks Apache Spark [3] across other lakehouse platforms. Comparing these engines fairly is a recurring industrial need that is expensive and time-consuming. Continuous cross-functional collaboration is required as different roles look for different signals from benchmarking, and benchmark implementations require domain experts. Recent advances in foundation models [4] and LLM agents [5] suggest a different operating model: an LLM agent can, in principle, translate a natural-language workload description into engine-equivalent implementations, drive their execution, and produce a structured comparison. Naive “agent-in-a-shell” approaches — an LLM issuing raw shell commands without typed schemas or structured intermediaries — often fail to produce comparable implementations across engines and struggle to fit multi-engine understanding in a single context. Robust primitives and a unified architecture are therefore required to guide AI toward rigorous implementations and reliable results. * These authors contributed equally to this work.

This paper presents SpBench, an agent-native framework for cross-engine lakehouse benchmarking in production use. Three concepts enable LLM agents to act as first-class consumers of the benchmarking workflow: 1) Strictly typed benchmark definitions. A benchmark is expressed step-by-step in a platform-agnostic way to achieve equivalent implementations across engines by construction. 2) Structured-telemetry contract. Every run is recorded as queryable fact and dimension data in a standardized format, allowing deterministic queries for results. 3) Agent-native harness. Composable skills form a closed loop for independent, self-reviewing agent iteration. II. BACKGROUND Cross-engine benchmarking is fundamentally a tripleequivalence problem: a fair comparison between two engines must hold the data, the query semantics, and the compute tier all equal. Workload specifications such as TPC-DS [6] and TPCx-BB [7]1 standardize what to run, but say nothing about how to compose engine-specific implementations of the same logical query, how to match compute tiers of comparable cost, or how to verify that two implementations have produced equivalent results. Those gaps are filled today by ad-hoc engineering effort. Compounding these gaps is the need for cross-platform expertise; engineers are often only well-versed in their own platform. An agent-native framework must remove this bottleneck — not by replacing equivalence policing with prompts, but by promoting equivalences from “by convention” to “by construction”. Benchmark management systems such as OLTP-Bench [9] address portability across relational databases but target single-engine deployments with no agent-facing interface. Lakehouse-specific benchmarks such as LHBench [10] compare storage-format performance across engines but do not address agent-driven authoring or cross-engine query equivalence. SpBench is designed from the ground up for agents by providing strict, composable building blocks to target arbitrary execution engines from a common foundation to ensure comparable cross-engine results. 1 TPCx-BB is based on the BigBench benchmark [8].

BenchmarkDefinition (typed schema objects ← YAML)

Operational loop Agent surface (Section IV): typed schemas at author-time, SQL over telemetry at analysis-time

Templated code-gen Snowpark

Snowpark Connect

Lakehouse platform A

Lakehouse platform B

Regression signal (telemetry)

Classify: bench bug/config/ product bug

Development loop External request

Pluggable BenchmarkRunners (cluster lifecycle, submit, poll)

Reproduce and localize (Build + Analyze)

Distill → spec

Analyze → results summary

Build → code + run

retry

retry

Star-schema telemetry: fact_runs, fact_queries, dim_* → object storage/external tables

Staged acceptance-contract gate

Fig. 1. The SpBench substrate. A typed BenchmarkDefinition compiles via templated code generation into per-engine artifacts executed by pluggable runners, emitting star-schema telemetry (fact_runs, fact_queries, dim_*) that the agent surface (Section IV) reads at author- and analysistime. BenchmarkDefinition name: str created_resources: list[CreatedResource] query_groups: list[QueryGroup]

1-to-many

QueryGroup name: str, serial: bool queries: list[Query] 1-to-many

QueryStep file: Path, function: str arguments: dict[str, Any] sql_file: optional Path

1-to-many

Query name: str, group: str steps: list[QueryStep]

Fig. 2. The BenchmarkDefinition schema hierarchy: each level holds a typed, one-to-many list of the level below, from the top-level benchmark down to individual QuerySteps.

III. A RCHITECTURE SpBench’s substrate is a typed compile-and-execute pipeline: one benchmark definition compiles into enginespecific implementations, runs against pluggable engine runners, and emits structured telemetry to a shared analytics layer. Every layer is deterministic — agents author against typed schemas, not free-form code, and outputs are queryable rather than merely parseable. Fig. 1 illustrates the pipeline. A benchmark decomposes into a hierarchy of typed constructs: BenchmarkDefinition, QueryGroup, Query, and QueryStep, each with a typed schema. Fig. 2 shows the decomposition: a BenchmarkDefinition holds oneto-many QueryGroups, each holding one-to-many Query instances, each holding one-to-many QuerySteps — the same steps whose per-step timing is recorded at run time, as described below. From a single benchmark definition, the substrate generates engine-specific notebooks or scripts via templated code generation, enabling execution against each target engine. The same BenchmarkDefinition produces all artifacts, so the query equivalence of Section II is enforced by construction rather than by convention. Generated artifacts run inside corresponding native environments, each via a pluggable BenchmarkRunner (Fig. 1): Snowpark workloads in the Snowpark Execution Environment [11], a secure sandboxed runtime co-located with Snowflake’s elastic compute architecture [12]; PySpark workloads through Snowpark Connect; and Apache Spark workloads on other lakehouse platforms. Every run emits a star-schema telemetry record to object storage: a fact_runs row identifying the run, one fact_queries row per query with timing and structured error fields, and content-addressed (hash-keyed) dim_* records for the deployment, compute tier, and environment, exposed

Fig. 3. SpBench’s two governed harness loops. The operational loop reproduces, localizes, and classifies a telemetry-detected regression; the development loop turns an external request into a spec, code, and a results summary. Both loops are gated by staged acceptance contracts before completion.

as external tables so deterministic SQL queries power both dashboards and the agent skills of Section IV. Beyond the per-query fact_queries record, a per-step timing table captures sub-query timing, letting the agent localize whether latency is in data loading, compilation, or execution. Users can also add extensions to collect platform-specific telemetry beyond this default contract after each benchmark query run. IV. AGENTIC B ENCHMARKING The substrate of Section III is the deterministic surface for an agent layer comprising three composable benchmark skills and two governed harness loops that drive the skills against operational and development signals (Fig. 3). Benchmark Skills. Each top-level skill encodes a named, composable workflow that an LLM agent can invoke from an IDE or CLI through typed tool interfaces implemented over the Model Context Protocol [13], in the spirit of tool-augmented language models [14]: a dedicated MCP server exposes run monitoring, SQL query access, and artifact retrieval as typed tools, grounding agent interactions in structured calls rather than shell commands. Each skill routes to engine-specific sub-skills, with Analyze further decomposing into run-data retrieval, comparison reporting, and failure diagnosis. The three skills each serve a distinct phase of the workflow: • Distill ingests engine performance telemetry — aggregate workload patterns, execution statistics, job-level metrics — and produces a sanitized benchmark spec (tables, engines, success criteria) containing no customer query content, table values, or user-identifying information. • Build transforms that spec into runnable artifacts — perengine query implementations, configurations, orchestration wiring — using SpBench constructs. • Analyze queries the benchmark telemetry tables after a run completes and emits a plain-language summary covering pass/fail rates, latency deltas, and outliers. The three skills compose by data flow: Distill’s output is Build’s input, Build’s output is the run, and Analyze reads its telemetry. Harness Loops. The skills are driven by two governed loops against complementary signals (Fig. 3): an operational loop that reproduces, localizes, and classifies a telemetry-detected regression via Build and Analyze, and a development loop that turns an external request into a spec, generated code, and

a results summary via Distill, Build, and Analyze — both gated by staged acceptance contracts, automated checks that a stage’s output must satisfy before the loop advances: for Build, engine-specific code checks and matching configurations across engines before a run is launched. A rejected stage is sent back to Build — in the operational loop, via Reproduce and localize — for another iteration. Both signal-generating skills emit structured, directly actionable recommendations rather than free-text findings, enabling an autonomous loop. Distill’s eligibility framework screens each candidate query pattern against a defined eligibility rule set — rejecting trivial or already-covered patterns and prioritizing signals such as window functions, multijoin plans, and novel UDF combinations — then writes a filled benchmark-request file that Build consumes directly, with no manual transcription step. Analyze’s regression scan carries engine-specific diagnostic context, enabling immediate discovery of benchmark implementation issues. Benchmarkside defects route back into Build’s iteration path to patch the affected query and re-run. Decompose, Don’t Monolith. A natural design choice when applying LLM agents to a multi-step workflow is one agent, equipped with all required tools, that plans, writes, debugs, and analyzes in one continuous reasoning chain. This monolithic approach breaks down on the Build skill, which must simultaneously reason about benchmark semantics, generate per-engine query implementations, audit code against a multi-rule constraint set, and coordinate validation across engines. The monolithic approach suffers heavily from context overload and quickly derails as iterations accumulate. We developed the Build skill using TeraSort [15] as a golden end-to-end test case, targeting reliable execution across every supported engine, execution mode, and compute configuration. Starting from a structured user-request template (tables, engines, success criteria), we aimed for a single skill invocation that completed the full authoring cycle. Iteration proceeded through acceptance gates at each delegation boundary, perengine implementation guidance, and graduated sub-agent delegation, until a single invocation reliably reached the validated state without manual intervention. The resulting architecture (Fig. 4) decomposes authoring at four stages: (i) static code generation, parallel per-engine query writers; (ii) code review, one sub-agent per query directory auditing all engine implementations in parallel; (iii) scale-factor-1 (SF1) validation, engine-specific runner subskills operating independently; and (iv) full-scale execution by the same sub-skills. Decomposition localizes context and recovery; errors can be addressed in a targeted manner, leaving other agents’ state intact. This points to a general pattern for multi-step LLM workflows: draw sub-agent boundaries at natural failure points, so a failure can be isolated and retried without restarting the whole workflow.

(i) Static codegen

per-engine sub-agent, parallel

eng A eng B

(ii) Code review

per-query subagent, parallel

dir A

dir B

(iii) SF1 validation

(iv) Full-scale run

per-engine sub-skill

per-engine sub-skill

eng A eng B

eng A eng B

Fig. 4. Decomposed Build skill architecture. Authoring is delegated at four points — static codegen, code review, SF1 validation, full-scale run — each handled by narrow-scope sub-agents (eng: per-engine; dir: per-querydirectory).

V. O UTCOMES SpBench currently comprises more than 400 benchmark queries, drawn from standardized suites such as TPC-DI [16] and TPCx-BB, and from custom benchmarks distilled from observed production patterns. These benchmarks run continuously to guide the Snowpark and Snowpark Connect performance roadmap and uncover regressions. The two engines together serve hundreds of millions of jobs per day in production [1], making rigorous performance tracking a direct driver of quality at scale. Concrete areas of investment driven by benchmark insights include file-to-table data transformations and UDF optimizations. With the benchmark authoring cycle reduced from weeks or months to just one AI development loop, we have been able to rapidly enhance our production benchmark portfolio with custom benchmarks derived from observed real-world usage, filling in blind spots left by industry-standard benchmarks [17]. VI. C ONCLUSION SpBench is an agent-native framework for cross-engine lakehouse benchmarking, pairing a deterministic substrate with three composable skills. Reliable one-shot benchmark authoring has been made possible by decomposed sub-agent harnesses with staged acceptance contracts. This system is used in production today by engineers across Snowpark to build and analyze benchmarks, and the resulting insights have spawned numerous development efforts for real product improvements. These outcomes underscore a broader principle: an LLM agent’s productivity depends heavily on the surrounding structure of typed schemas, unified telemetry, governed loops, and decomposed skills. These findings reflect a single-organization deployment across a fixed engine set; broadening the evaluation across organizations and engines is ongoing work, moving benchmarking toward a model in which a customer-reported signal can be resolved by anyone, guided by AI. ACKNOWLEDGMENT The authors thank colleagues at Snowflake for design discussions, code reviews, and engineering support throughout SpBench’s development — in particular the engineering and product teams whose products SpBench composes, the Snowflake Engineering Systems teams building foundational internal infrastructure powering benchmarking, and the early users whose feedback shaped the agentic interfaces and telemetry contracts presented here.

R EFERENCES [1] B. Baker, E. Brossard, C. Xie, Z. Ye, D. Liu, Y. Xie, A. Zwiegincew, N. K. Sharma, G. Jain, E. Retunsky, M. Halcrow, D. Denny-Brown, I. Cseri, T. Akidau, and Y. He, “Snowpark: Performant, Secure, UserFriendly Data Engineering and AI/ML Next To Your Data,” in Proc. 45th IEEE Int. Conf. on Distributed Computing Systems (ICDCS), Glasgow, UK, 2025. [2] Snowflake Inc., “Snowpark Connect for Apache Spark,” Snowflake Documentation, 2026. [Online]. Available: https://docs.snowflake.com/ en/developer-guide/snowpark-connect/snowpark-connect-apache-spark [3] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica, “Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing,” in Proc. 9th USENIX Symp. on Networked Systems Design and Implementation (NSDI), 2012, pp. 15–28. [4] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., “On the Opportunities and Risks of Foundation Models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07258 [5] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing Reasoning and Acting in Language Models,” in Proc. Int. Conf. on Learning Representations (ICLR), 2023. [6] R. O. Nambiar and M. Poess, “The Making of TPC-DS,” in Proc. 32nd Int. Conf. on Very Large Data Bases (VLDB), 2006, pp. 1049–1058. [7] Transaction Processing Performance Council, “TPC Express Benchmark BB (TPCx-BB) Standard Specification,” Version 1.6.1, Aug. 2023. [Online]. Available: https://www.tpc.org/TPC_Documents_Current_ Versions/pdf/TPCX-BB_v1.6.1.pdf [8] A. Ghazal, T. Rabl, M. Hu, F. Raab, M. Poess, A. Crolotte, and H.-A. Jacobsen, “BigBench: Towards an Industry Standard Benchmark for Big Data Analytics,” in Proc. ACM SIGMOD Int. Conf. on Management of Data, 2013, pp. 1197–1208. [9] D. E. Difallah, A. Pavlo, C. Curino, and P. Cudré-Mauroux, “OLTPBench: An Extensible Testbed for Benchmarking Relational Databases,” Proc. VLDB Endowment, vol. 7, no. 4, pp. 277–288, 2013. [10] P. Jain, P. Kraft, C. Power, T. Das, I. Stoica, and M. Zaharia, “Analyzing and Comparing Lakehouse Storage Systems,” in Proc. 13th Conf. on Innovative Data Systems Research (CIDR), 2023. [11] G. Jain, B. Baker, J. Yin, C. Xie, Z. Ye, S. Kulkarni, S. Abdelrahman, N. Qi, U. Shrestha, M. Halcrow, D. Bailey, and Y. He, “SEE++: Evolving Snowpark Execution Environment for Modern Workloads,” in Proc. IEEE Int. Conf. on Big Data (IEEE BigData), 2025, poster. [12] B. Dageville, T. Cruanes, M. Zukowski, V. Antonov, A. Avanes, J. Bock, J. Claybaugh, D. Engovatov, M. Hentschel, J. Huang, A. W. Lee, A. Motivala, A. Q. Munir, S. Pelley, P. Povinec, G. Rahn, S. Triantafyllis, and P. Unterbrunner, “The Snowflake Elastic Data Warehouse,” in Proc. ACM SIGMOD Int. Conf. on Management of Data, 2016, pp. 215–226. [13] Anthropic, “Model Context Protocol Specification,” 2024. [Online]. Available: https://modelcontextprotocol.io/specification [14] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language Models Can Teach Themselves to Use Tools,” in Proc. 37th Conf. on Neural Information Processing Systems (NeurIPS), 2023. [15] O. O’Malley, “TeraByte Sort on Apache Hadoop,” Yahoo!, Tech. Rep., May 2008. [Online]. Available: https://sortbenchmark.org/ YahooHadoop.pdf [16] M. Poess, T. Rabl, H.-A. Jacobsen, and B. Caufield, “TPC-DI: The First Industry Benchmark for Data Integration,” in Proc. 40th Int. Conf. on Very Large Data Bases (VLDB), 2014, pp. 1367–1378. [17] J. V. Szlang, S. Breß, S. Cattes, J. Dees, F. Funke, M. Heimel, M. Oleynik, I. Oukid, and T. Maltenberger, “Workload Insights From the Snowflake Data Cloud: What Do Production Analytic Queries Really Look Like?,” Proc. VLDB Endowment, vol. 18, no. 12, pp. 5126–5138, 2025.

Related documents

• SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking#1536279
• From Code to Playable Systems: CEGame-Bench for Long-Horizon Game Creation Agents#1401277
• OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration#1345191
• OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration#1336059
• SWE-Residual: Evaluating Starting-State Robustness of Coding Agents#1401090