arXiv:2605.24183v1 [cs.DB] 22 May 2026
AvalancheBench: Evaluating Enterprise Data Agents Through Latent World Recovery Darek Kłeczek
Fuheng Zhao
Alexander W. Lee
Snowflake [email protected]
Snowflake [email protected]
Brown University and Snowflake [email protected]
Julien Tissier
Paweł Liskowski
Uğur Çetintemel
Snowflake [email protected]
Snowflake [email protected]
Brown University and Snowflake [email protected]
Anupam Datta Snowflake [email protected]
Abstract We introduce AvalancheBench, a benchmark for evaluating enterprise data agents through latent world recovery. AvalancheBench improves on existing benchmarks in three ways. First, it evaluates analytical understanding rather than pipeline completion: systems are scored on whether they recover the segments, drivers, temporal events, and relationships that explain the data, not merely on whether they execute a workflow or produce a plausible report. Second, it provides ground truth for goal-driven analytics by generating observations from a known latent world, enabling partial credit for incomplete but valid recoveries. Third, it exposes how early analytical mistakes propagate into later conclusions: missed segments, merged events, or wrong attributions can lead to systematically wrong recommendations. In this sense, AvalancheBench complements real-data benchmarks by providing a controlled setting for diagnosing whether agents recover the analytical structure behind enterprise data. On a first e-commerce use case, the strongest configuration of a leading coding agent recovers only 26% of the rubric, with failures concentrated in generic customer segmentations and merged temporal events. VLDB Workshop Reference Format: Darek Kłeczek, Fuheng Zhao, Alexander W. Lee, Julien Tissier, Paweł Liskowski, Uğur Çetintemel, and Anupam Datta. AvalancheBench: Evaluating Enterprise Data Agents Through Latent World Recovery. VLDB 2026 Workshop: DASHSys.
1
Introduction
Analytical tasks in real enterprise settings are often goal-driven, e.g., diagnose declining customer satisfaction or identify growth opportunities; many valid pipelines can pursue such goals, but correctness remains grounded in the underlying data. To evaluate AI agents on such tasks, we frame analytics as a latent world recovery This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected]. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment. ISSN 2150-8097.
Structural Enforcement
Latent World Specification Customer Personas
Golfer
Patriotic
Temporal Events Satisfaction Impact Timeline
Date
Persona
Product
Event
Rating
1/12
Golfer
Tank
Defect
1
1/14
Golfer
Leggins
None
5
2/23
Patriotic
Leggins
Shipping
3
2/25
Patriotic
Tank
Shipping
2
Surface Realization I thought this would be a perfect tank for my golfing practice, but sadly small holes started appearing after a few washes… These legging are great for golf, 5 stars!
Rubric
Cotton Tank Quality Defect
Global Shipping Incident
Q: Identify all significant inflection points in the month-by-month average rating trend across the full review period. What events or patterns explain each inflection? A: Cotton Tank quality defect in month 1, global shipping incident in month 2.
I paid 20 percent more for these legging because I want to support US companies, but I hate it when my orders don’t arrive on time!
Figure 1: Latent 𝑍 (personas, events) drives both the structured rows and rubric answers; an LLM then renders reviews where persona and defect signals surface only implicitly. The agent sees the right panel and recovers the left.
problem: each task is a tuple (𝑍, 𝑋, 𝑅)—a latent analytical state 𝑍 we control at generation time, observations 𝑋 produced from 𝑍 by a known process, and a rubric 𝑅 whose answers derive from 𝑍 . Evaluation asks how much of 𝑍 a system reconstructs from 𝑋 , scoring analytical correctness rather than pipeline execution. This setting reflects real world workflows, where discovery of latent factors such as customer segments, satisfaction drivers, and temporal events is followed by quantitative measurement of their interdependencies and decision-oriented synthesis. See Table 1 for examples of rubric queries and Figure 1 for a concrete example of a task that an agent has to answer. Existing benchmarks reflect other features of enterprise analytical workloads. Such workloads need to execute pipelines over complex, multi-source, multi-modal data. KramaBench [4] focuses on evaluating data-to-insight pipelines over data lakes. Data Agent Benchmark (DAB) [8] measures end-to-end data agent capabilities on realistic enterprise schemas while explicitly avoiding openended questions. UniDataBench [14], InsightBench [11] and InsightEval [17] target multi-source analytics. The unstructured data aspect handled by semantic operator engines such as AISQL [6], SWAN [15], LOTUS [10], Palimpzest [7], DocETL [12] is measured
Query type
Example (e-commerce)
Latent target in 𝑍
Segmentation Temporal Synthesis
Identify the customer personas and their priorities. Identify the defect onset timeline. Top 5 products for quality investment.
Persona clusters and population weights Onset and peak phases for defects Integration of insights
Table 1: Examples of rubric queries and latent targets in e-commerce use case.
rise to the data. These tasks are compositional: a system extracts signals, organizes them into latent structure, and synthesizes conclusions. Recovery of the latent structure is orthogonal to pipeline correctness: a system can run the wrong pipeline and still recover 𝑍 , or run the right pipeline and misrecover it (Section 4). Standard metrics are poorly suited to this setting. Classification accuracy, rank correlation, or clustering similarity all assume predicted and ground-truth representations are aligned. Analytical outputs rarely are: a system may merge multiple latent factors into one category, name segments differently, or report rankings at a coarser granularity, leaving set- or pair-based comparisons undefined. Outputs admit multiple valid formulations, so exact-match scoring is too strict. We therefore score recovery one query at a time, with each 𝑟𝑖 probing a distinct slice of 𝑍 . Our approach exposes compounding errors: the avalanche effect where mistakes in discovery or segmentation propagate into the final synthesis.
by SemBench [5] at the operator level and DeepScholar-Bench [9] on long-horizon synthesis. A gap remains: these benchmarks measure pipeline execution or answer plausibility, not whether a system recovers the correct analytical understanding of the data. Our preliminary experiments show that AvalancheBench exposes failure modes that existing benchmarks would find hard to detect: agents can produce plausible analyses while recovering the wrong analytical structure. In our e-commerce use case, agents asked to diagnose a satisfaction decline sometimes merged a product-specific material defect with a separate brand-wide shippingdelay incident, despite being asked to identify issue type, timing, scope, affected products, and evidence. Similarly, agents sometimes substituted generic e-commerce segments for the customer personas present in the data. These are not merely failures of wording or presentation; they are failures to recover the structure that generated the observations. The rest of the paper is organized as follows: Section 2 formalizes recovery-based evaluation, Section 3 describes the controlled generation pipeline that yields rubric ground truth from a known latent world, and Section 4 reports preliminary experimental findings highlighting typical agent failure modes—cascading errors, generic segmentations, and merged temporal events—on the e-commerce use case.
2
3 Benchmark Design and Validity 3.1 Controlled Latent World Generation The benchmark is operationalized in four stages: specification, rubric derivation, structural enforcement, and surface realization, visualized with a small example in Figure 1. Specification. The latent state Z is carefully designed by subject matter experts (SMEs) and defined declaratively in a configuration file that specifies every analytical signal of interest: persona clusters with exact population weights, defect timelines, seasonal review distributions, product-level base ratings and persona preferences. This configuration is the single source of truth from which both the dataset and the evaluation rubric are derived. Rubric Derivation. Because 𝑍 is fully specified before generation, the experts manually specify the scoring criteria from the latent world. Quantitive answers, such as product rankings, follow mechanically from 𝑍 . Qualitative answers, such as strategic recommendations, are grounded in 𝑍 but admit multiple valid formulations. For these, scoring criteria specify partial credit conditions that an LLM judge evaluates. Structural Enforcement. A generator maps 𝑍 into structured data. Customers are assigned to persona clusters via weighted sampling matching the configured distribution. Review ratings are computed compositionally from base rating, defect impact, persona bias and noise factor, where each term is controlled by 𝑍 . This layer guarantees that the statistical structure of the generated data faithfully reflects 𝑍 . Surface Realization. Unstructured content is constructed by a generative model conditioned on the structural parameters. For example, each product review is generated from a rich prompt that encodes the assigned persona’s personality traits, motivations, pain points, product-specific satisfaction aspects, and, where applicable,
Latent World Recovery as an Evaluation Framework
We define an analytical task as a tuple (𝑍, 𝑋, 𝑅): • Latent world 𝑍 : the analytical state we instantiate—segments, factors, temporal regimes, events, product/issue hierarchies, and their joint distributions. • Observations 𝑋 : structured tables paired with unstructured text (e.g. customer reviews) produced from 𝑍 by a generative process. • Rubric 𝑅 = {𝑟 1, . . . , 𝑟𝑚 }: each 𝑟𝑖 = (𝑞𝑖 , 𝑎𝑖 , 𝑐𝑖 ) is a query 𝑞𝑖 with a ground-truth answer 𝑎𝑖 derived from 𝑍 and scoring criteria 𝑐𝑖 . A system observes 𝑋 and emits a report 𝐴. We define analytical fidelity as the alignment between 𝐴 and the structure of 𝑍 , operationalized through the rubric: ∑︁ L (𝑍, 𝐴, 𝑅) = 𝑠 (𝑞𝑖 , 𝑎𝑖 , 𝑐𝑖 , 𝐴), 𝑖
where 𝑠 measures how well 𝐴 matches 𝑎𝑖 under criteria 𝑐𝑖 . Because 𝑍 is fixed before 𝑋 is generated, every 𝑎𝑖 is derived from 𝑍 rather than inferred from text. Correctness is defined by consistency with the data-generating structure, not by individual answers in isolation. This framing mirrors our interpretation of goal-driven enterprise analytics tasks: recovering the hidden structure that gives 2
phase-specific defect descriptions. We additionally condition the generative model on human-written reviews to maintain natural style. This produces natural, varied text that embeds the target analytical signals without making them artificially obvious.
3.2
persona-by-aspect priority matrix), temporal and synthesis (e.g. strategic recommendations) queries (examples in Table 1). For our experimental setting, we utilize Snowflake’s coding agent Cortex Code (CoCo) v1.0.80 [13] equipped with the Claude Opus 4.7 [2]. We experiment with two different paradigms in solving these goal-driven analytical questions:
Generation Quality
• Agent as Compiler: An agent uses relevant tools (Python, SQL, etc.) to understand the data and produces a static pipeline with a single semantic query (e.g. AISQL [6] and LOTUS [10]) per question. • Agent as Orchestrator: An agent uses semantic query engine (e.g. AISQL) along with other relevant tools (Python, SQL, etc.) to solve all queries interactively and can refine its reasoning with new observations.
We apply two quality-control checks to the generation pipeline, one per generation stage, to ensure the observations faithfully realize the specification. Structural Conformance. At the structural-enforcement stage we have full metadata for all hidden factors, so we run queries to verify that the rubric ground-truth claims hold over the generated data—e.g., persona distributions, average ratings, defect timelines. Realization Audit. The structured generator guarantees that the intended latent signals are present, but the LLM that renders observations may introduce artifacts: leaking the latent state, diluting signals through vague language, adding plausible but unintended factors not in 𝑍 , or creating stylistic shortcuts that agents exploit instead of performing genuine recovery. We qualitatively spot-check a small random sample of generated observations and adjust the generation process to mitigate these failure modes.
3.3
For the Agent as Compiler paradigm, we experiment with two semantic query systems: 1) the AISQL [6] queries are executed directly on Snowflake; and 2) the LOTUS semantic query (version 1.1.4) using the same model backbone via the Snowflake REST API. Note that the LOTUS semantic rank operator is disabled in this setting, as it requires access to log probabilities that are not currently provided by the Snowflake REST API. We evaluate accuracy using an LLM-as-a-Judge [16] framework with Claude Opus 4.7 as the judge backbone. Each of the 18 rubric items is scored on a 0-5 scale; the reported percentage is the total points awarded divided by the maximum attainable points. For example, a temporal item awards credit per distinct event recovered with correct timing, but penalizes merging separate events into one. To account for variance, we run three trials per approach and report the mean and standard deviation.
Ecological Realism and Anti-Shortcut Design
To ensure the synthetic benchmark reflects enterprise analytics, we involve subject matter experts in two ways. First, SMEs design the latent world: analytics goals, rubric questions, and latent factors (personas, satisfaction aspects, defects) are inspired by real workloads. Second, we perform error analysis and confirm the observed failure modes are relevant to real enterprise analytics. For example, hallucinating plausible customer segments rather than verifying them in the data would be very hard to discover on a real dataset with unknown distribution. This leads us to an important insight—the requirement to avoid LLM-distribution priors in 𝑍 . Instead of letting an LLM propose “plausible” segments, we hand-design personas around concrete domain types (e.g., a golfer who cares about glove grip and walking comfort, or a patriotic buyer who values U.S.-made apparel). This ensures that errors caused by agents falling back on generic LLMprior labels are identified by the rubric-based evaluation.
Approach Score (%) CoCo Compiled LOTUS 12.2 ± 3.0 CoCo Compiled AISQL 17.0 ± 3.3 CoCo as Orchestrator 25.9 ± 5.5 Table 2: Average score and standard deviation of the evaluated paradigms on AvalancheBench across three trials.
4.2
4
Early Experiments: Failure Modes in Analytical Recovery 4.1 Setup
Findings
First, we observe that Agent as Orchestrator scores the highest average accuracy of 26%. A pipeline-completion benchmark would record these runs as successful—reports are produced, SQL executes, AI SQL pipelines return rows—while the latent-world rubric shows the analytical content is largely wrong. Generic Segmentations. Recovering the correct customer segments proves to be a major challenge. The latent representation 𝑍 defines ground-truth segments including golfer, patriotic, and sports_fan, but all three approaches only partially recover them: compiled AISQL misses sports_fan, and compiled LOTUS misses patriotic. CoCo as Compiler, rather than discovering segments from the full data, hard-codes plausible-sounding segments inferred from a small sample of reviews and domain knowledge, effectively providing reasonable-looking but generic segmentation for the domain instead of grounding it in the underlying population.
Our first use case is an e-commerce task which pairs a product catalog (20 products), sales records (23k transactions), and freeform customer reviews (10k) of a fictional performance-apparel brand. The latent world 𝑍 specifies customer personas with explicit population weights, product-tier base ratings, satisfaction aspects with persona-conditioned orderings, and temporal events combining product-level material defects with one brand-wide service incident. Reviews in 𝑋 carry no foreign key to the catalog, so informal mentions must be resolved against product names. The rubric covers six query types: discovery (e.g. satisfaction factors), segmentation (e.g. customer segments), ranking, conditional (e.g. 3
References
Avalanche Effect. Our e-commerce benchmark is structured so that many queries necessitate accurate customer segmentation. When models fail to recover the ground-truth latent representation 𝑍 and instead rely on generic segmentations, they trigger an avalanche effect of error propagations. This misalignment invalidates downstream analytics, such as customer spending patterns and group satisfaction, as these metrics are calculated for groups that do not exist in the actual data. Merged Temporal Events. 𝑍 contains two distinct events: productscoped material defects and a brand-wide shipping incident in a different window. The compiled AISQL and LOTUS systems identify the defect but misattribute its timeframe to the shipping-delay window, effectively collapsing the two events into one. Coarse metrics such as topic recall or rating-trend matching do not register this as a failure: the system did discuss quality issues and did flag the overall rating decline. Temporal rubric items, which probe onset, scope, and affected products, expose both the merge and the omitted incident directly.
5
[1] Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2026. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=RQm2KQTM5r [2] Anthropic. 2026. System Card: Claude Opus 4.7. https://cdn.sanity.io/files/ 4zrzovbb/website/037f06850df7fbe871e206dad004c3db5fd50340.pdf. Accessed: 2026-05-14. [3] Martin Jurkovic, Valter Hudovernik, and Erik Štrumbelj. 2025. SyntheRela: A Benchmark For Synthetic Relational Database Generation. In Will Synthetic Data Finally Solve the Data Access Problem? https://openreview.net/forum?id= ZfQofWYn6n [4] Eugenie Lai, Gerardo Vitagliano, Ziyu Zhang, Om Chabra, SIVAPRASAD SUDHIR, Anna Zeng, Anton A. Zabreyko, Chenning Li, Ferdi Kossmann, Jialin Ding, Jun Chen, Markos Markakis, Matthew Russo, Weiyang Wang, Ziniu Wu, Mike Cafarella, Lei Cao, Samuel Madden, and Tim Kraska. 2026. KRAMABENCH: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=fZfUdeCC5X [5] Jiale Lao, Andreas Zimmerer, Olga Ovcharenko, Tianji Cong, Matthew Russo, Gerardo Vitagliano, Michael Cochez, Fatma Özcan, Gautam Gupta, Thibaud Hottelier, H. V. Jagadish, Kris Kissel, Sebastian Schelter, Andreas Kipf, and Immanuel Trummer. 2026. SemBench: A Benchmark for Semantic Query Processing Engines. arXiv:2511.01716 [cs.DB] https://arxiv.org/abs/2511.01716 [6] Pawel Liskowski, Bowei Chen, Anupam Datta, Benjamin Han, Boxin Jiang, Nitish Jindal, Zihang Li, Aaron Lin, Paritosh Aggarwal, Jay Tayade, Dimitris Tsirogiannis, Nathan Wiegand, and Weichen Zhao. 2025. Cortex AISQL: A Production SQL Engine for Unstructured Data. ArXiv abs/2511.07663 (2025). https://api.semanticscholar.org/CorpusID:282922358 [7] Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baile Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, Rana Shahout, and Gerardo Vitagliano. 2025. Palimpzest: Optimizing AI-Powered Analytics with Declarative Query Processing. In Proceedings of the Conference on Innovative Database Research (CIDR) (2025). [8] Ruiying Ma, Shreya Shankar, Ruiqi Chen, Yiming Lin, Sepanta Zeighami, Rajoshi Ghosh, Abhinav Gupta, Anushrut Gupta, Tanmai Gopal, and Aditya G. Parameswaran. 2026. Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents. arXiv:2603.20576 [cs.DB] https://arxiv.org/abs/2603.20576 [9] Liana Patel, Negar Arabzadeh, Harshit Gupta, Ankita Sundar, Ion Stoica, Matei Zaharia, and Carlos Guestrin. 2025. DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis. In NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling. https://openreview.net/forum?id=M4iVZtEDX4 [10] Liana Patel, Siddharth Jha, Melissa Pan, Harshit Gupta, Parth Asawa, Carlos Guestrin, and Matei Zaharia. 2025. Semantic Operators and Their Optimization: Enabling LLM-Based Data Processing with Accuracy Guarantees in LOTUS. Proc. VLDB Endow. 18, 11 (July 2025), 4171–4184. doi:10.14778/3749646.3749685 [11] Gaurav Sahu, Abhay Puri, Juan A. Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vázquez, Nicolas Chapados, Christopher Pal, Sai Rajeswar, and Issam H. Laradji. 2025. InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation. In ICLR. https://openreview.net/forum? id=ZGqd0cbBvm [12] Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. 2025. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing. Proc. VLDB Endow. 18, 9 (May 2025), 3035–3048. doi:10. 14778/3746405.3746426 [13] Snowflake. 2026. Snowflake Cortex Code. https://www.snowflake.com/en/ product/features/cortex-code/. Accessed: 2026-05-14. [14] Han Weng, Zhou Liu, Yuanfeng Song, Xiaoming Yin, Xing Chen, and Wentao Zhang. 2025. UniDataBench: Evaluating Data Analytics Agents Across Structured and Unstructured Data. arXiv:2511.01625 [cs.DB] https://arxiv.org/abs/2511. 01625 [15] Fuheng Zhao, Divyakant Agrawal, and Amr El Abbadi. 2024. Hybrid Querying Over Relational Databases and Large Language Models. arXiv:2408.00884 [cs.DB] https://arxiv.org/abs/2408.00884 [16] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MTBench and Chatbot Arena. In Advances in Neural Information Processing Systems, Vol. 36. [17] Zhenghao Zhu, Yuanfeng Song, Xin Chen, Chengzhong Liu, Yakun Cui, Caleb Chen Cao, Sirui Han, and Yike Guo. 2025. InsightEval: An ExpertCurated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents. arXiv:2511.22884 [cs.AI] https://arxiv.org/abs/2511.22884
Limitations and Open Challenges
LLM-based Evaluation. Even with 𝑍 -anchored criteria, the LLM judge has its own biases—length, position, lexical overlap with the rubric—well documented in the LLM-as-a-judge literature [16]. We plan to address this by evaluating multi-judge inter-agreement on a calibration subset against human experts. We do not yet claim judge robustness across the full rubric; this is active work. Synthetic-Real Gap. Our generator buys soundness with synthesis: every signal in 𝑋 traces to 𝑍 . This trade-off is shared with synthetic-data efforts such as SyntheRela [3]. Real enterprises have signals nothing in 𝑍 would predict—legacy quirks, exogenous events, organically evolved taxonomies. AvalancheBench is not a substitute for real-data; it is a controlled lab for failure-mode discovery, complementary to real-data benchmarks like KramaBench [4] and DAB [8]: they probe whether a system survives the real distribution, we probe whether it recovers a known structure. Ambiguity in Analytical Ground Truth. Even though we control for spurious factors, some different interpretations for specific rubric items are possible (e.g. granularity of discovered clusters). We currently audit this manually through error analysis and adjust the rubric for more clarity. In the future, we plan to use automated evolutionary approaches to address this [1].
6
Conclusion and Future Work
AvalancheBench proposes making the latent analytical structure of a domain the unit of evaluation: instantiate it explicitly and score agents on how much they recover from observations they did not generate. Our early experiments show that systems looking adequate at pipeline completion miss segments, merge events, and adopt LLM-prior taxonomies, and recovery-aware rubrics can see this. We view AvalancheBench as complementary to recent realdata and insight benchmarks [4, 8, 9, 11, 14]: it evaluates analytical understanding, not just task execution, provides ground truth for goal-driven analytics, and exposes cascading failures in the analytical process. Future work extends the approach to additional modalities (documents, images, audio) and enterprise domains (customer support, marketing analytics). 4