Conceptio › Archive › arXiv CS

arXiv CSopen access

PrismaDV: Automated Task-Aware Data Unit Test Generation

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

PrismaDV: Automated Task-Aware Data Unit Test Generation Hao Chen

Arnab Phani

Sebastian Schelter

BIFOLD & TU Berlin [email protected]

arXiv:2604.21765v1 [cs.LG] 23 Apr 2026

Abstract

allows its users to annotate data pipelines with “pipeline expectations” [10] for data unit testing, Google Dataplex [20] enables customers to choose from auto-suggested data quality rules for catching data anomalies in their data pipelines, and GXCloud recently announced an AI-driven constraint suggestion feature [27].

Data is a central resource for modern enterprises, and data validation is essential for ensuring the reliability of downstream applications. However, existing automated data unit testing frameworks are largely task-agnostic: they validate datasets without considering the semantics and requirements of the code that consumes the data. We present PrismaDV, a compound AI system that analyzes downstream task code together with dataset profiles to identify data access patterns, infer implicit data assumptions, and generate taskaware executable data unit tests. To further adapt the data unit tests over time to specific datasets and downstream tasks, we propose “Selective Informative Feedback for Task Adaptation” (SIFTA), a prompt-optimization framework that leverages the scarce outcomes from the execution of data unit tests and downstream tasks. We evaluate PrismaDV on two new benchmarks spanning 60 tasks across five datasets, where it consistently outperforms both task-agnostic and task-aware baselines in generating unit tests that reflect the end-to-end impact of data errors. Furthermore, we show that with SIFTA, we can automatically learn prompts for PrismaDV’s modules that outperform prompts written by hand or generated from a generic prompt optimizer. We publicly release our benchmarks and prototype implementation.

Shortcomings of current approaches. Despite their popularity, existing frameworks suffer from several shortcomings: (𝑖) authoring and maintaining data unit tests remains tedious and errorprone, since data engineers often write checks one column at a time, which does not scale to wide production tables with hundreds of columns [70]; as a result, validation coverage is typically partial and focuses on a subset of columns only. Frameworks such as Deequ and TFDV alleviate this burden by automatically suggesting constraints from sample data via data profiling, but this automation introduces additional challenges: (𝑖𝑖) heuristically suggested constraints are often either too strict or too general; overly strict tests produce false alarms, leading to alert fatigue and costly on-call triage, while overly general tests tend to miss domain-specific data errors and can lead to production incidents that require substantial manual intervention. Furthermore, (𝑖𝑖𝑖) designing effective data unit tests still requires manual post-editing by a data engineer with domain knowledge, and (𝑖𝑣) over time, data unit tests typically evolve in a reactive way only: data problems become apparent in production systems, are manually fixed and tests are extended with additional constraints to prevent recurrence. This reactive cycle imposes a recurring tax on data teams: scarce engineering time is diverted to debugging, test maintenance, and incident response, slowing down feature and model iteration. Researchers proposed several extensions to address these shortcomings in recent years, which either leverage statistics from historical executions [12, 31, 58, 67, 70, 73] or require a human in the loop [29, 32, 46] to label data examples, neither of which directly leverages the downstream task code that actually consumes the data.

Introduction

Data is a central resource for modern enterprises and institutions, and data issues, such as missing or incorrect information [1, 80, 86], can seriously impact their operations. Data errors propagating through data systems lead to serious impact in production, such as outages of mobile apps [79], bank customers losing access to their accounts [78], outages of flights in the US [9], and the loss of medical records [74]. Furthermore, data errors are one of the major reasons for the silent performance degradation of deployed ML models [51, 57, 60]. A reason for this is that many organizations have adopted a “collect first, analyze later” workflow [35], relying on the schema-onread interpretation of data in downstream applications. Therefore, corrupted data often propagates unnoticed until it causes failures in production.

Task-aware data unit test generation. We argue that a major limitation of current approaches is that they rely on observed data only and ignore the characteristics of the downstream tasks that consume the data to validate. This leads to several missed opportunities to improve data unit tests and address some of the outlined shortcomings. First, certain downstream tasks might only access parts of the data, especially for large denormalized datasets common in enterprise data lakes, which means that data unit tests for these tasks should focus on the subset of accessed columns only. Second, the code of downstream tasks is often written by experienced data engineers, with implicit domain knowledge about the data “baked in”, which may be helpful to extract into a data unit test. Some downstream tasks like ML training tasks might even be naturally robust against certain types of noise in data, which a data unit test could account for. We illustrate these limitations with a running example in Section 3.

Current landscape of data unit testing frameworks. As a consequence, data unit testing frameworks such as TensorFlow Data Validation [57], Amazon’s Deequ [51, 61, 64], and Great Expectations [13] have become widely used in industry in recent years. These frameworks generate data unit tests: declarative data constraints, often expressed in an easy-to-use DSL, such as null-value checks, completeness and uniqueness constraints, value-range checks, and distributional sanity checks. These constraints are inferred by profiling a data sample and subsequently applying heuristics, against which to validate unseen data. Major cloud providers offer data unit testing as part of their data infrastructure: Amazon’s AWS Glue Data quality service [64] defines a domain-specific language to enable non-coders to define data unit tests with Deequ, Databricks 1

Hao Chen, Arnab Phani, & Sebastian Schelter

• We design two novel benchmarks for evaluating task-aware data unit test generation: ICDBench for individual constraint discovery from data–code pairs (63 cases with ground-truth constraints) and EIDBench for end-to-end error impact detection with five datasets, 60 tasks, and 25 error cases per dataset (Section 6). • We conduct an extensive experimental evaluation showing that PrismaDV outperforms strong baselines by more than 20 points in F1 score on ICDBench, more than 26 points in F1 score on EIDBench, and that SIFTA outperforms a general prompt optimizer (Section 8). • We make our code and benchmark available under an open license at https://github.com/deem-data/PrismaDV.

A way forward is to improve the automated generation of data unit tests by specializing them to the downstream tasks for which they are deployed. However, this specialization is inherently difficult as it requires an “understanding” of downstream task code. In practice, production data pipelines support a diverse set of downstream tasks, ranging from recurring BI/ETL processing to web applications to display and edit data, to feature engineering and ML training or inference, and these tasks often encode different semantics and assumptions about the same data. Even seemingly simple problems like identifying which columns a piece of code accesses are challenging and typically handled via static code analysis with hand-curated knowledge bases [49]. Approaches like fuzzing-based testing [57] are also difficult to apply in practice, as they assume that one can generate synthetic input data and repeatedly execute the downstream tasks in a “sandbox mode”. In many industry settings, repeated execution is impractical because tasks have external side effects, for example materializing intermediate results, triggering actions in other systems, or customer interactions such as sending notification emails.

Background

We briefly introduce the required background on data unit tests. Data unit tests are typically deployed as part of data pipelines which move data between different systems and applications [10, 51, 57, 61, 64]. The goal of a data unit test is to flag potentially erroneous data early to allow engineers to intervene before the data already caused issues in downstream applications. Data unit tests are crucial for the data operations in large organizations, where data updates are regularly produced and consumed by hundreds of downstream applications. Formally, a data unit test 𝐶 = {𝑐 1, . . . , 𝑐𝑛 } consists of a set of constraints {𝑐 1, . . . , 𝑐𝑛 }. Each constraint 𝑐𝑖 is a variant of a primitive aggregation constraint [59]. In data unit testing libraries such as pydeequ [65], constraints are declared as follows hasCompleteness("colA", lambda x: x >= 0.99).where("colB > 10"). This constraint states: The column “colA” must have at least 99% non-null values in rows where the corresponding value of “colB” is larger than ten. Data unit tests are explicitly designed to rely on efficiently computable aggregates, since these tests must be runnable on datasets with billions of tuples [61]. For that reason, they often use approximations for expensive statistics, e.g., hyperloglog sketches [28] for cardinality estimates or KLL sketches [37] for approximating percentiles. Evaluating the data unit test 𝐶 on a dataset 𝐷 requires the evaluation of each constraint 𝑐𝑖 ∈ 𝐶. The data unit test rejects 𝐷 if there exists a constraint which is not satisfied on 𝐷. Designing data unit tests is challenging since it requires intricate knowledge about invariants of the data and the domain in which it is used. Furthermore, there is a tension between overly strict constraints, which may produce many false alarms and too general constraints, which may not be helpful in identifying issues in the data. Popular libraries like Deequ [3] and Tensorflow Data Validation [72] offer automated ways to suggest constraints based on data profiling, which must typically be post-edited by data engineers.

Overview and contributions. We propose to take downstream tasks into account for automated data unit test generation. To this end, we introduce PrismaDV, a task-aware data validation system that proactively [84] generates specialized data unit tests for individual downstream tasks by jointly analyzing dataset profiles and task code. We motivate this direction with a running example (Section 3). We then describe the design of PrismaDV, a compound AI system [36], which decomposes task-aware data unit test generation into multiple steps (data profiling, column access detection, assumption inference, and constraint code generation). We discuss these steps and how we leverage the code understanding [21] and code synthesis capabilities of LLMs [32, 39] in Section 4. To improve validation quality over time for a particular dataset, we propose “Selective Informative Feedback for Task Adaptation” (SIFTA), a lightweight prompt-optimization approach that leverages the scarce outcomes from the execution of data unit tests and downstream tasks as supervision signal (Section 5). SIFTA identifies informative constraint failures via “failure precision” (the fraction of constraint failures that coincide with task failures), and backtraces these failures to the underlying data assumptions in code as input to an optimizer. Finally, we introduce two complementary benchmarks in Section 6: ICDBench, a hand-crafted benchmark for constraint discovery from data–code pairs, and EIDBench, an end-to-end benchmark with 60 downstream tasks across five public datasets. In summary, we provide the following contributions.

• We introduce the problem of task-aware data unit test generation (Section 3). • We present PrismaDV, a compound AI system that analyzes downstream task code together with dataset profiles to identify data access patterns, infer implicit data assumptions, and generate task-aware executable data unit tests (Section 4). • We propose SIFTA, a prompt optimization procedure for PrismaDV that leverages the scarce outcomes from the execution of data unit tests and downstream tasks, together with structured backtraces from constraints to assumptions and code (Section 5).

Problem Statement

We discuss the shortcomings of task-agnostic data unit tests and introduce the problem in the focus of this paper with a running example in a fictitious scenario that mirrors the ETL and downstreamconsumption patterns common in production data platforms. Note that we provide an executable version of this example in a Jupyter notebook at https://github.com/deem-data/PrismaDV/blob/main/ toy-example.ipynb. 2

PrismaDV: Automated Task-Aware Data Unit Test Generation Batch processing name

location

guest_cat

revenue

status

Aisha

aisha@…

100

COMPLETED

Mateo

mateo@…

100

COMPLETED

Keiko

keiko@…

2000

IN_PROGRESS

NULL

2500

CANCELED

Ingrid

ingrid@…

100

IN_PROGRESS

name

location

guest_cat

revenue

Leila Matthias

leila@…

US EU

1 1

2000 100

300

Sakura

NULL sakura@…

Crashes batch processing task

location

guest_cat

revenue

status

Linda

linda@… dieter@…

3 1

1000 1000

IN_PROGRESS

Dieter

GER US

Chen NULL

NULL li@…

1 0

1000

CANCELED CANCELED

1000

ETL Pipeline with Task-agnostic 1 Data Unit Test

Analytics

status

Check() .areComplete("guest_cat", "location", "status", "revenue") .hasCompleteness("name", lambda x: x>.44) .hasCompleteness("email", lambda x: x>.44) .areNonNegative("guest_cat", "revenue") .isContained("location", ["EU", "US"]) .isContained("guest_cat", [0, 1])

COMPLETED COMPLETED IN_PROGRESS

name

GER

COMPLETED

Overly strict constraints produce false alarms for D2

Crashes ML training task

report = duckdb.sql(""" SELECT guest_cat, COUNT(*) FROM df WHERE status = 'IN_PROGRESS' GROUP BY guest_cat""").df() save_to_s3(report, datetime.today())

ML training

Task assumes that each tuple with “COMPLETED” as status has an email Task assumes that standard deviation of revenue is nonzero

cost_normalized = (df["revenue"] - df["revenue"].mean()) / df["revenue"].std() df.loc[df["location"]=="GER", "location"] = "EU" locations = OneHotEncoder().fit_transform(df[['location']]) X = np.column_stack((locations, cost_normalized)) Code shows that y = df["booking_status"]=="COMPLETED" “GER” is valid for model = LogisticRegression().fit(X, y) deploy_model(model) location

Downstream Tasks with implicit domain knowledge and data assumptions

Data Batches to import from Data Lake

Code shows that guest_cat has import polars as pl range of 0 to 3 discount_per_cat = [5, 10, 25, 500] df = df.with_columns(pl.col("guest_cat") \ .map_elements(lambda x: discount_per_cat[x]).alias("discount")) completed = df.filter(pl.col("status") == "COMPLETED") for booking in completed.iter_rows(named=True): send_confirmation_mail(booking["email"], booking["discount"])

Sample data Dsample to create data unit test via data pro ling and heuristics

Figure 1: Toy example to exemplify the need for task-aware data unit tests: 1 An ETL pipeline employs a task-agnostic data unit test (generated by AWS Deequ) to validate new batches of data before forwarding them to three downstream tasks. 2 A hidden dependency among different columns in the code of batch processing task causes a crash, and was missed by the Deequ test that only looked at data; 3 The overly strict data unit test flags data conditions to which downstream tasks are robust, and thereby causes false alarms; 4 A hidden assumption about the aggregate statistics of a column in the code of the ML task causes another crash. The example shows that a single data unit test derived from sample data alone is insufficient, since it fails to account for implicit data assumptions and domain knowledge in the code of the downstream tasks. Instead, a task-aware solution is required with a data unit test per downstream task, specialized to the task’s access pattern and data assumptions. Running example. Imagine that a large travel corporation acquired a small startup which produced a successful booking app. As part of the integration, the central devops team from the corporation now needs to connect several downstream services of the startup with a large shared data lake from the corporation via ETL pipelines. These ETL pipelines regularly push new data into the downstream services (e.g., on a nightly basis). We visualise one such example pipeline in Figure 1. This pipeline handles records, which detail ongoing and completed bookings as well as their financial impact, with the following six columns name, email, location, guest_cat, revenue, status. The ETL pipeline regularly feeds new batches of booking data into the following three downstream services developed by the startup:

the form of a set of constraints on the completeness and value range of various columns. The devops team then deploys the generated test in their ETL pipeline. Reactive handling of data issues. At night, the data batch 𝐷 1 arrives in the ETL pipeline, which evaluates the data unit test on it. Since the test passes, the pipeline forwards to the data batch to the downstream tasks. 2 However, the batch processing task crashes with an error, resulting in the devops team getting alerted. Their investigation uncovers that the code of the batch processing task contains the hidden assumption that each record with a “COMPLETED” value in the status must also have a valid value in the email column, which was not the case for the second tuple in 𝐷 1 . This subtle condition has been missed by Deequ’s constraint suggestion. The devops engineers now have to manually make sure that all customers receive their correct discount emails. Afterwards, they manually extend the data unit test to also account for the subtle data condition. During the next night, the data batch 𝐷 2 arrives in the pipeline. 3 The data unit test rejects this batch, which leads to the quarantining of the data and again to alerts for the devops team. The engineers investigate the test results and find that the test flagged the unexpected value "GER" in the location column, as well as the value 3 for guest_cat. After contacting the startup engineers, the devops team learns that this was a false alarm, the value "GER" is sometimes produced by legacy booking systems, and 3 is a rare but valid value for guest_cat, which indicates a special guest category. The startup engineers confirm that both cases can be handled by their services, leading to the insight that the data unit test from Deequ was overly strict. 4 The engineers now make the ETL pipeline forward 𝐷 2 , which unfortunately leads to an unexpected crash in the ML task. Investigating the code of the ML task uncovers that the data preparation code produces NaN values in the training data, which the ML model cannot handle. The devops

• Batch processing – a batch processing task which computes discounts for customers and sends them notification emails. • Analytics – a task which runs a SQL query to generate a daily report on active bookings and stores it in a distributed file system. • Machine learning model training – a task which uses the booking data to train and deploy a model to predict the probability of a booking completion. In the past, the devops team of the corporation has repeatedly had to handle data quality incidents where downstream tasks failed due to issues in the data, and the engineers had to spend their weekends fixing the data and rerunning the affected downstream tasks. 1 To avoid such problems in this scenario, they decide to implement a data unit test for their ETL pipeline, which is evaluated on each new data batch to ingest, and is supposed to tell them whether it is safe to forward the newly arriving data. For that, they leverage the automated generation of data unit tests from Deequ (via “constraint suggestion” [3]). The engineers take a sample 𝐷 sample of the existing booking data, and provide it to Deequ. Deequ profiles the data sample and applies several heuristics to generate a data unit test in 3

Hao Chen, Arnab Phani, & Sebastian Schelter

Module

API methods

Output

Description

Profiling & Discovery

ProfileData DiscoverColumnAccess DiscoverJointColumnAccess

✗ ✓ ✓

Data profile List of columns List of sets of columns

Compute basic statistics about the input data Determine columns accessed by the downstream code Determine columns jointly accessed by the downstream code

Assumption Inference

ColumnDataflowAnalysis MultiColumnDataflowAnalysis SummarizeAndLinkAssumptions

✓ ✓ ✓

Code locations Code locations Data-code assumption graph

Find code lines operating on a column Find code lines operating on a set of input columns Summarize implicit data-code assumptions

Constraint Code Generation

GenerateColumnConstraints GenerateMultiColumnConstraints

✓ ✓

Executable constraint code Executable constraint code

Generate constraints for a column Generate constraints for a set of columns

Post-Processing

PreCheckConstraint

✗

Flag indicating validity

discard buggy / invalid constraints

LLM?

Table 1: Modules and API methods of PrismaDV, implemented via external tools, custom code, and LLM invocations.

engineers realize that this is due to the fact that revenue values are normalized by dividing through their standard deviation which is zero in this data batch. They again realize that this subtle data assumption was not covered in their data unit test.

depends on whether these assumptions continue to hold. We define the boolean task validity of a data batch 𝐷𝑖 with respect to 𝑇 as Valid𝑇 (𝐷𝑖 ) ∈ {0, 1}, where Valid𝑇 (𝐷𝑖 ) = 1 if executing 𝑇 on 𝐷𝑖 completes successfully and exhibits the intended behavior, and Valid𝑇 (𝐷𝑖 ) = 0 if 𝑇 crashes, raises an exception, or silently produces an incorrect result. This outcome captures the ground-truth suitability of the data batch for the task. The objective is to generate, for each downstream task 𝑇 , a specialized data unit test in the form of a constraint set 𝐶𝑇 whose acceptance behavior aligns with the true task validity signal:

The need for the automated generation of task-aware data unit tests. The examples show that a single central data unit test, derived from the data, is insufficient to adequately address potential data issues that can occur. Instead, an intricate understanding of the code and data assumptions of the downstream tasks are required. Ideally, a custom test for each downstream task, tailored to its specific data assumptions and access pattern is deployed. This would be to avoid both false alarms (which cause unnecessary work for devops engineers and on-call sessions on the weekend) and missed data issues (which may crash downstream services). However, creating specialized data unit tests is very tedious since popular datasets in large data lakes maybe consumed by hundreds of downstream services, often with hard-to-understand codebases (e.g., legacy code). Furthermore, both data and downstream services continuously change and evolve in large organizations, requiring a regular adjustment of the data unit tests.

𝐶𝑇 (𝐷𝑖 ) ⇔ Valid𝑇 (𝐷𝑖 ), for both observed and, critically, unobserved dataset batches. Given the downstream task code 𝑇 and an observed data sample 𝐷 sample of 𝐷 on which 𝑇 runs successfully, the goal is to infer the implicit data assumptions that 𝑇 relies on to operate correctly and synthesize from them a constraint set 𝐶𝑇 that approximates the validity function Valid𝑇 (·) on new data batches.

Research question. This leads us to the research question in the focus of this paper: Can we automate the generation of data unit tests, such that they are tailored to downstream code? Our goal is to change the development of data unit tests from its reactive nature (adjusting tests after production incidents) to a proactive nature [84], where comprehensive tests are generated upfront. At the same time, we aim to alleviate the need for domain experts to write custom data unit tests. An automated system should leverage task code in addition to sample data to design, specialize and improve tests, by uncovering and including the hidden domain knowledge and data assumptions in the code.

PrismaDV

In PrismaDV, we leverage Large Language Models (LLMs) for task code summarization, assumption inference, and constraint code generation, since these models have recently shown strong capabilities in code generation [4, 16, 66], data preprocessing [5, 14, 15, 18, 40, 69], and program understanding [47, 55, 68]. Note that our modular API isolates LLM interactions within specific methods.

4.1

System Modules

We decompose task-aware data unit test generation into three modules, each exposing a well-defined API contract: (i) Profiling and Discovery, (ii) Assumption Inference, (iii) Constraint Code Generation, and (iv) Post-processing. Each module exposes a set of API methods for the data unit test generation workflow detailed in Table 1. Formally, PrismaDV takes as input a data sample 𝐷 sample with columns 𝐴 = [𝐴1, . . . , 𝐴𝑛 ] and the source code of the downstream task 𝑇 .

Formal definition of task-aware data validation. We formalize the task-aware data validation problem introduced above. Consider a downstream task 𝑇 —implemented as code artifact—that consumes a tabular dataset 𝐷 over time via regularly incoming data batches {𝐷 1, . . . , 𝐷𝑚 }. We assume that 𝑇 executes correctly on a sample 𝐷 sample of 𝐷; however, other batches may violate implicit assumptions embedded in the task logic, as illustrated in our running example. Whether 𝑇 succeeds or fails on a new data batch

Profiling and discovery. The profiling and discovery module collects descriptive statistics from sample data and analyzes the code of downstream tasks to identify the columns, and combinations of columns, accessed by downstream tasks. This establishes the foundation for connecting data characteristics with task semantics. 4

PrismaDV: Automated Task-Aware Data Unit Test Generation Accessed Data Columns

…

… guest_cat …

status

leila@… …

1 …

COMPLETED …

…

Task Code with Per-Column Data ow Annotations discount_per_cat = [5, 10, 25, 500] df = df.with_columns( pl.col("guest_cat") .map_elements(discount_per_cat) .alias(“discount")) completed = df.filter("status" == "COMPLETED")

Identi cation and annotation of column access patterns in task code

for booking in completed.iter_rows(): send_confirmation_mail( booking["email"], booking["discount"]) Extraction of Implicit Data Assumptions

Implicit Data Assumptions of Code Synthesized in Natural Language The 'guest_cat' column must be complete (no nulls) for all rows, since it is used directly as a positional index into a list without any null handling. The 'guest_cat' column must contain integer values only, as Python list indexing requires ints. The 'guest_cat' values must be within the allowed index range of the discount list: 0 to 3 inclusive (no negatives and no values >= 4). For all rows where booking_status == 'COMPLETED, the email column must be present, because the value is passed directly to without null-handling. For all rows where booking_status == 'COMPLETED, the email column should match a basic email format.

Task-Aware Data Unit Test Synthesized from Implicit Data Assumptions

Check .satisfies("guest_cat >= 0 AND guest_cat <= 3", "guest_cat within [0, 3]") .satisfies("booking_status != 'COMPLETED' OR email IS NOT NULL", "Email not null for completed bookings") .containsEmail("email").where("booking_status = 'COMPLETED'", “Valid email for completed bookings") Synthesis of code for each constraint from one or more implicit data assumptions

Figure 2: During data unit test generation for a downstream task, PrismaDV builds a bipartite “data-code assumption” graph, which connects accessed input columns to the implicit data assumptions in the task code about them (synthesized in natural language). For that, our system annotates the code lines that operate on an input column (or data derived from it). The code generation module, which synthesizes the task-aware data unit test, leverages the assumption graph as input.

fi fl

Given 𝐷 sample and 𝑇 , the method ProfileData(𝐷 sample ) → 𝑆 computes descriptive statistics 𝑆, which include types, completeness, approximate number of distinct values, histograms for lowcardinality columns, and the mean for numeric columns. The methods DiscoverColumnAccess(𝑇 , A, 𝑆) → 𝐴accessed and DiscoverJointColumnAccess(𝑇 , 𝐴accessed, 𝑆) → 𝐴accessed_jointly detect the subset of columns accessed by 𝑇 and column groups jointly referenced in the code, respectively. The resulting metadata provides the basis for the subsequent assumption inference stage.

columns 𝐴relevant expressed in the syntax of a target data validation framework. Multiple constraints may arise from a single assumption, and conversely, one constraint may aggregate several related assumptions. Post-processing. The last module ensures syntactic validity of the generated constraints. Each candidate constraint 𝑐 is first validated via PreCheckConstraint(𝑐, 𝐷 sample ) → {0, 1}, which evaluates parseability and consistency. Secondly, constraints that do not hold on 𝐷 sample are discarded.

Assumption inference and data-code graph construction. The assumption inference module forms the conceptual core of PrismaDV. It bridges the gap between data and code by analyzing the task code to derive explicit representations of the implicit data assumptions encoded within. This module transforms the task code into a structured, interpretable intermediate representation, producing natural language descriptions of these hidden assumptions. The assumption inference module constructs a bipartite data–code assumption graph 𝐺 = ((𝐴relevant, 𝐻 ), 𝐸), where 𝐴relevant = 𝐴accessed ∪ 𝐴accessed_jointly denotes accessed columns, 𝐻 the set of inferred data assumptions, and 𝐸 the labeled edges linking them, annotated with code locations. For each column 𝐴𝑖 ∈ Aaccessed , ColumnDataflowAnalysis(𝑇 , 𝐴𝑖 ) → 𝐿𝑖 locates the statements in 𝑇 operating on 𝐴𝑖 or its derivatives. The code locations 𝐿𝑖 are then used to create an annotated code variant 𝑇 ′ of the code 𝑇 . This annotated code is then summarized through summarizeAndLinkAssumptions(𝑇 ′, 𝐴𝑖 , 𝑆), yielding the set of inferred natural language assumptions 𝐻𝑖 connected to code spans in 𝐿𝑖 . For multi-column cases, MultiColumnDataflowAnalysis(𝑇 , 𝐴 𝑗 ) identifies joint access of column sets 𝐴 𝑗 ∈ Aaccessed_jointly in the code. The resulting graph 𝐺 serves as the intermediate representation passed to the constraint synthesis stage. We refer to Figure 2 for a visualization of this process on a downstream task from our running example.

Implementation. We implement the proposed modules of PrismaDV in DSPy [38] with support for response caching and asynchronous execution to run dataflow analysis, assumption generation, and pydeequ code generation in parallel across accessed columns.

Optimization via Selective Informative Feedback for Task Adaptation (SIFTA)

In real-world deployments, data validation runs as part of data pipelines that continuously ingest new data batches. Over time, teams accumulate execution outcomes that indicate whether a task run succeeded or failed on specific batches. These observations are scarce, but they provide feedback for improving validation quality on future data. Moreover, multiple tasks often consume the same input dataset. They can share latent data assumptions or business logic, such as preprocessing steps, joins, or feature engineering. This creates transfer opportunities: outcomes from existing tasks and batches can help improve validation for new batches and tasks. Taken together, these properties motivate an optimization approach for task-aware data unit tests generation. Since PrismaDV is a compound AI system with LLM-based modules, there are several optimization choices such as updating LLM parameters, e.g., via reinforcement learning or fine-tuning or adjusting prompts of the LLM-based modules. In production settings, however, updating LLM parameters for each new task is often infeasible, as it incurs significant training cost, fine-tuning parameter storage, and deployment and versioning overhead [85]. We therefore focus on prompt optimization for PrismaDV’s LLM-based modules, as it requires no model training, fits within existing inference pipelines, and can be driven by the scarce execution outcomes collected in production.

Constraint code generation. The constraint code generation module synthesizes executable validation logic from the data–code assumption graph 𝐺, translating the inferred data assumptions linked to code and columns directly into the syntax of a target data validation framework (e.g., Deequ or Great Expectations). The methods GenerateColumnConstraints(𝐴𝑖 , 𝐺) and GenerateMultiColumnConstraints(𝐴 𝑗 , 𝐺) translate the data-code assumption graph 𝐺 into executable constraints for the accessed 5

Hao Chen, Arnab Phani, & Sebastian Schelter

5.1

5.2

Optimization Setting

We extend the formal problem statement introduced in Section 3. Over time, downstream tasks are executed on new data batches, yielding task–batch pairs with an observed binary execution outcome indicating whether the task completed correctly. Although these observations are scarce and expensive to obtain, they provide valuable feedback for adapting the system to future data batches and downstream tasks. Our goal is to tune the prompts Π of the LLM-based modules used by PrismaDV for a fixed dataset, using observed execution outcomes together with the corresponding data batches and task code. We consider three within-dataset generalization settings: (i) new batches for known tasks, (ii) new tasks on known batches, and (iii) new tasks on new batches. 𝑟 Formally, for dataset 𝐷 with 𝑚 data batches {𝐷𝑖 }𝑚 𝑖=1 , let {𝑇ℓ } ℓ=1 be the 𝑟 downstream tasks that consume the data over time. Let 𝑇obs denote known observed tasks, and 𝑇new denote new tasks for the same dataset. Analogously, let 𝐷 obs denote observed data batches and 𝐷 new unseen new data batches. The observation set is O ⊆ 𝑇obs × 𝐷 obs . PrismaDV’s LLM-based modules depend on a backbone LLM 𝐿 and the set of prompts Π. Concretely, we optimize Π while keeping 𝐿 fixed; we omit 𝐿 from the notation for readability. The optimization target Q can be defined in different ways, covering (i) new data batches 𝐷 new for observed tasks 𝑇obs , (ii) new tasks 𝑇new on observed batches 𝐷 obs , or even (iii) new tasks 𝑇new on new data batches 𝐷 new . For a given set of prompts Π, PrismaDV generates for each task 𝑇ℓ a task-specific data unit test 𝐶 ℓ(Π) , whose evaluation (Π) on data batch 𝐷𝑖 yields a binary prediction b 𝑣 ℓ,𝑖 = 𝐶 ℓ(Π) (𝐷𝑖 ). We denote the actual binary execution outcome for a task–batch pair as 𝑣 ℓ,𝑖 = Valid𝑇ℓ (𝐷𝑖 ); note that these ground-truth outcomes are only observed for deployed data unit tests.

“Failure Precision” as Informative Signal

Informative learning signals. Task-aware data validation generates multiple signals from the execution of constraints and downstream tasks. However, the available supervision is limited to binary execution outcomes, and not all observed signals are equally informative for assessing the quality of a constraint. We analyze under which conditions the outcome of a constraint evaluation provides reliable information about whether the constraint captures task-relevant data errors. We consider the behavior of an individual constraint. Let 𝑐 ℓ,𝑘 denote a constraint defined on column 𝐴 𝑗 , evaluated on a data batch 𝐷𝑖 . For each evaluation, we observe the binary constraint outcome b 𝑣 ℓ,𝑖,𝑘 ∈ 0, 1 and the corresponding task execution outcome 𝑣 ℓ,𝑖 ∈ 0, 1. Whether 𝐴 𝑗 contains a task-relevant data error in batch 𝐷𝑖 is not directly observable and is treated as a latent variable. Based on the observable outcomes, four cases can occur: (1)b 𝑣 ℓ,𝑖,𝑘 = 1 and 𝑣 ℓ,𝑖 = 1: the constraint passes and the task succeeds. This outcome is inconclusive, as the absence of observed failures does not imply that the constraint would detect relevant errors in other batches; (2) b 𝑣 ℓ,𝑖,𝑘 = 1 and 𝑣 ℓ,𝑖 = 0: the constraint passes while the task fails. This case is also ambiguous, since the task failure may be caused by errors in other columns or by interactions that the constraint does not capture; (3) b 𝑣 ℓ,𝑖,𝑘 = 0 and 𝑣 ℓ,𝑖 = 1: the constraint fails while the task succeeds. This outcome corresponds to a clear false alarm and therefore provides a reliably informative negative signal; and (4) b 𝑣 ℓ,𝑖,𝑘 = 0 and 𝑣 ℓ,𝑖 = 0: both the constraint and the task fail. This outcome is potentially informative, but remains ambiguous because the task failure may or may not be attributable to an error in column 𝐴 𝑗 . The key asymmetry arises when a task fails (𝑣 ℓ,𝑖 = 0): there are two latent possibilities, namely that the failure is caused by a taskrelevant error in column 𝐴 𝑗 , or that it originates from other columns or interactions. Since this distinction is unobserved, multiple latent error configurations collapse into the same observable outcome. Consequently, constraint passes (b 𝑣 ℓ,𝑖,𝑘 = 1) are inherently uninformative under this supervision regime, as they are compatible with both the absence of errors and undetected errors. In contrast, constraint failures (b 𝑣 ℓ,𝑖,𝑘 = 0) are the only outcomes that can yield informative learning signals: failures with task success indicate definitive false alarms, while failures with task failure correspond to plausible detections. This observation motivates focusing optimization exclusively on constraint failures and quantifying how often a constraint failure coincides with a task failure.

Optimization objective. Our goal is to choose a set of prompts Π that maximize validation quality on the target scenarios Q. Formally, the objective is: h i (Π) Π∗ = arg max E 𝜇 {(b 𝑣 ℓ,𝑖 , 𝑣 ℓ,𝑖 ) : (𝑇ℓ , 𝐷𝑖 ) ∈ Q} , Π

where 𝜇 (·) is a validation-quality metric such as F1 score. Challenges. The optimization problem is challenging due the following reasons: Scarce and delayed supervision. The primary feedback available in practice is a binary execution outcome for a task run on a data batch. Obtaining more specific supervision signals (e.g., which column caused a failure or which assumption was violated) is difficult and expensive, and may require substantial debugging. This issue is exacerbated in ML pipelines, where the manifestation of data issues can be delayed (e.g., gradual performance degradation), making it hard to collect fine-grained error feedback at scale.

Column-level failure precision. Our optimization operates on task–column units. For a column 𝐴 𝑗 , let 𝐶 ℓ,𝐴 𝑗 be the set of constraints on 𝐴 𝑗 and define the column-level prediction 𝑤 bℓ,𝑖,𝑗 on a Ó batch 𝐷𝑖 as 𝑤 bℓ,𝑖,𝑗 = 𝐶ℓ,𝑘 ∈𝐶ℓ,𝐴 b 𝑣 ℓ,𝑖,𝑘 . The column-level failure pre𝑗

Multiple intermediate trajectories with localized errors. Task-aware validation generates rich intermediate artifacts (e.g., column access patterns, assumption summaries, constraint candidates, and code), which can be long and heterogeneous. At the same time, data issues in a batch are typically localized to a small subset of columns or interactions. This mismatch makes it hard to directly apply existing prompt-optimization methods (e.g., MIPROv2 [53], GEPA [2]), which are commonly evaluated in settings with smaller trajectories and more dense feedback.

cision is an empirical estimate of Pr[𝑣 ℓ,𝑖 = 0 | 𝑤 bℓ,𝑖,𝑗 = 0] over observed batches 𝐷 obs : Í CFPr(𝐶 ℓ , 𝐴 𝑗 , 𝐷 obs ) =

𝑤 ℓ,𝑖,𝑗 = 0 ∧ 𝑣 ℓ,𝑖 = 0] 𝐷𝑖 ∈𝐷 obs 1[b Í

𝑤 ℓ,𝑖,𝑗 = 0] 𝐷𝑖 ∈𝐷 obs 1[b

and we only consider columns with non-zero denominator as informative training units. 6

PrismaDV: Automated Task-Aware Data Unit Test Generation

Algorithm 1 Prompt optimization for PrismaDV with columnlevel failure-precision backtracking via SIFTA.

Constraint-level failure precision. For diagnosis and backtracing within an informative column, we also compute constraint-level failure precision for individual failing constraints: Í 𝑣 ℓ,𝑖,𝑘 = 0 ∧ 𝑣 ℓ,𝑖 = 0 𝐷𝑖 ∈𝐷 obs 1 b FPr(𝐶 ℓ,𝑘 , 𝐷 obs ) = . Í 𝑣 ℓ,𝑖,𝑘 = 0 𝐷𝑖 ∈𝐷 obs 1 b

Require: training observations Otrain , eval observations Oeval , initial constraint-generation prompts Π0 , rounds 𝑛 round , eval budget 𝑏 eval , train sample size 𝑛 train , feedback constraints per column 𝑛 fb , eval sample size 𝑛 eval , assumption graph 𝐺 1 Π ← Π0 Constraints that never fail (zero denominator) are assigned FPr = 0, 2 𝑏 remain ← 𝑏 eval since they provide no actionable information. 3 for 𝑡 ← 1 . . . 𝑛 round : 4 trainCond ← Condense(Otrain, Π) 5.3 Optimization Procedure 5 evalSample ← SampleColumns(Oeval, 𝑛 eval ) Based on the analysis above, we introduce Selective Informative Feed6 evalScore ← MeanCFPr(Π, evalSample) back for Task Adaptation (SIFTA), a lightweight prompt-optimization 7 𝑏𝑡 ← ⌊𝑏 remain /(𝑛 round − 𝑡 + 1)⌋ procedure that leverages informative failure signals captured by 8 candidates ← {(Π, evalScore)} failure precision. SIFTA concentrates optimization on task–column 9 while 𝑏𝑡 > 0 : units with constraint failures and backtraces failing constraints to 10 trainSample ← SampleColumns(trainCond, 𝑛 train ) the assumptions and code locations that produced them, providing 11 trainScore ← MeanCFPr(Π, trainSample) targeted feedback for prompt updates. 12 colCFPr ← ComputeCFPr(Π, trainSample) Overview. SIFTA iteratively updates PrismaDV’s constraint-generation 13 constraintFPr ← ComputeFPr(Π, trainSample) related prompts 𝜋, i.e., the prompts used by the LLM-based API 14 lowFPr ← SelectBottomKPerColumn(constraintFPr, 𝑛 fb ) methods ColumnDataflowAnalysis, SummariseAndLinkAssump- 15 traces ← Backtrace(lowFPr, 𝐺) tions, and GenerateColumnConstraints (Table 1), using scarce 16 Π ′ ← Propose(Π, colCFPr, lowFPr, traces) task outcomes across multiple tasks on the same dataset. At the 17 if MeanCFPr(Π ′, trainSample) ≥ trainScore beginning of each round, SIFTA (i) condenses the training obser18 evalScore′ ← MeanCFPr(Π ′, evalSample) vations using the current 𝜋, (ii) samples 𝑛 eval task–column units 19 candidates ← candidates ∪ {(Π ′, evalScore′ )} for evaluation, and scores the current Π on this fixed evaluation 20 𝑏𝑡 ← 𝑏𝑡 − 1; 𝑏 remain ← 𝑏 remain − 1 sample, and (iii) allocates the remaining evaluation budget 𝑏 eval 21 (Π, evalScore) ← BestByEval(candidates) across rounds. Within a round, it repeatedly resamples 𝑛 train train22 return Π ing units to generate constraints, compute failure-precision scores, and construct backtraces; a candidate Π ′ is only scored on the round’s evaluation sample if its mean training CFPr on the sampled training units does not decrease. Algorithm 1 shows the procedure. Training set condensation. To reduce the search space, we first only generate candidate constraints for the actual accessed columns of a task 𝑇ℓ . At the beginning of each optimization round, we condense the training set Otrain using the current prompts Π by selecting task–column units (𝑇ℓ , 𝐴 𝑗 ) for which at least one constraint fails on an observed training batch. This concentrates the optimization budget on columns that surface failures. We do not condense Oeval ; instead, we uniformly sample task–column units from all task–column combinations in Oeval and keep this eval sample fixed within the round.

Implementation. We implement SIFTA using DSPy’s promptoptimization API. In our pipeline, the prompts for dataflow analysis, assumption inference, and constraint code generation are tightly coupled through shared intermediate artifacts. As a result, updating only one module prompt can create contextual mismatches across modules. SIFTA therefore uses a global prompt proposer that can jointly update one or multiple module prompts within a single proposal. Each proposal is conditioned on a shared instruction prompt that provides task-aware data validation context and motivates failure precision as the proxy optimization target.

Selection of training targets via failure precision. We use mean column-level failure precision CFPr as the primary optimization objective. For each sampled training unit, we compute constraintlevel failure precision FPr for failing constraints and rank them within each column. For feedback, we select the 𝑛 fb constraints with the lowest FPr per column, emphasizing negative signals that are directly actionable for prompt updates.

Benchmarking Task-Aware Data Validation

We introduce ICDBench and EIDBench, two carefully designed benchmarks to evaluate task-aware data unit tests generation. These benchmarks provide a standardized way to compare future frameworks as well as various baselines such as LLMs, outlier detection methods and agentic systems [56, 71] in terms of their ability to automatically generate effective, task-aware data unit tests. Our benchmarks integrate publicly available datasets with LLM-generated tasks covering diverse domains and applications. We make both benchmarks available under an open license at https://github.com/ deem-data/PrismaDV/blob/main/benchmarks and plan to maintain and update them with new use cases and baselines.

Backtracing feedback context. For each selected low-FPr constraint, we backtrace to the linked assumptions and code locations via the data–code assumption graph (Figure 2), and provide these traces (together with CFPr and FPr scores) as feedback context to the prompt proposer. The proposer returns a candidate prompt set Π ′ , which we score on the evaluation sample only if its mean training CFPr on the sampled units does not decrease. 7

Hao Chen, Arnab Phani, & Sebastian Schelter

6.1

ICDBench – Individual Constraint Discovery from Data-Code Pairs

(1) Table summarization: we profile 𝐷 sample and produce a compact summary of its schema and value distributions (types, missingness, ranges, frequent categories, and example rows). (2) Task proposal: conditioned on the summary, the LLM proposes a concrete task description that mimics applied business logic over the table. (3) Assumption generation: the LLM enumerates implicit data assumptions that a developer would rely on for the task to behave correctly. Each assumption is phrased as a predicate whose violation would lead to a crash or an abnormal (possibly silent) behavior. (4) Code generation: the LLM implements the task as a single Python script. We prompt it to (i) rely on the assumptions in the task logic and (ii) embed each assumption as an executable assertion block (see below). We generate 30 candidate tasks per dataset and retain the executable tasks after the verification procedure described below.

The building block of a task-aware data unit test generation system is the ability to discover constraints about the data which are implicitly defined in code. To evaluate this ability (independent of the other end-to-end building blocks of the system), we design ICDBench, a novel benchmark for constraint discovery from data-code pairs. This benchmark includes 63 cases, each of which consists of a tabular data sample, a piece of code written to process the data, a natural language description of the hidden assumption in the code, and the corresponding ground truth constraint in PyDeequ syntax. Furthermore, each case features two held-out pieces of data: a positive example of data to pass (on which the constraint holds), and a negative example of data to reject, where the constraint is not satisfied. ICDBench includes hand-designed cases as well as a large number of data-code pairs obtained from public GitHub repositories. We explicitly design it to include a diverse range of hidden assumptions in code, ranging from simple cases like explicit asserts, to tough cases like column dependencies expressed in control flow or knowledge about semantics in ML libraries (e.g., scikit-learn operations). Moreover, the benchmark covers a wide range of domains, including payment processing, cricket sports rules, and in-game auctions in video games.

6.2

Assertion blocks and leakage control. Each assertion block is delimited by sentinel comments (e.g., # ASSERTION_START / # ASSERTION_END) to enable programmatic removal and reinsertion. We use the blocks in two ways: (i) for inference, we remove all assertion blocks and provide the resulting code to the system under test, preventing trivial extraction of assumptions from explicit asserts; (ii) for labeling, we execute the script with assertion blocks enabled on evaluation batches. A data batch is labeled erroneous for a task if its execution crashes or any assertion fails; otherwise it is labeled safe. To prevent leakage, we ensure that removing assertion blocks leaves the task executable and that no program state used by the core task is defined or modified inside assertion blocks. In addition, we manually review each task’s code to remove bugs and eliminate assumption leakage via comments or task logic.

EIDBench – End-to-End Error Impact Detection

Following the task-aware data validation setting introduced in Section 3 (a downstream task 𝑇 consuming a dataset over data batches {𝐷 1, . . . , 𝐷𝑚 }), we introduce EIDBench, an end-to-end benchmark for evaluating full pipelines. Again, a batch is “safe” for 𝑇 only if executing 𝑇 completes and exhibits the intended behavior; otherwise it is labeled “erroneous”.

Task selection and verification. Each generated task goes through two verification stages: (i) an automated repair-and-test loop that runs the task on 𝐷 sample in three modes: (a) with all assertion blocks enabled, (b) with all assertion blocks removed, and (c) with exactly one assertion block enabled at a time (removing all others), to ensure that each assumption check is executable and independent. This guards against implementation artifacts where the core task code accidentally relies on variables, imports, or intermediate results created or redefined inside assertion blocks, which would break when assertions are removed. Upon failure, we prompt the same LLM to edit the code up to five rounds, discarding tasks that remain non-executable; (ii) a manual audit to remove remaining bugs, confirm expected behavior on 𝐷 sample , and check for any assumption leakage. Following this procedure, we retain 60 final tasks (roughly 12 per dataset). We release the initially generated task versions, edit histories, and final tasks with the benchmark.

Benchmark design. Each EIDBench dataset provides: (i) an initial data sample 𝐷 sample used to author and validate task code, (ii) a set of twenty-five evaluation batches, each obtained by injecting synthetic errors of a certain type into 𝐷 sample , and (iii) a suite of downstream tasks in Python written to consume the dataset. The code of each task embeds data assumptions as executable assertion blocks, which we use as ground truth to label whether an evaluation batch is safe or erroneous for that specific task. Datasets. We include five datasets from diverse domains and sources, with varying fractions of numerical, categorical, and textual attributes. Downstream tasks. A core challenge in building an end-to-end benchmark is obtaining diverse, executable downstream tasks for the same tabular dataset: such code is common in industry but rarely shared with the academic community, while public notebooks, like the ones from Kaggle, often follow repetitive EDA/model-training templates. We therefore synthesize downstream tasks and their ground-truth assumptions using an LLM-assisted, human-in-theloop pipeline.

Error injection. To mimic real-world data issues, we extend the tabular error injection framework Jenga [62] with 19 operator types covering structural, integrity, numerical, textual, and format corruptions. For each dataset, we instantiate these operators into 25 error configurations, each producing one corrupted data batch from the clean sample 𝐷 sample . Each configuration corrupts only a small subset of columns and rows, reflecting how production issues are typically localized. We design task-targeted corruptions by inspecting scripts’ assumption blocks so that a given corruption may break

Task generation pipeline. For each dataset, we treat the sample data 𝐷 sample as the development data and generate an initial pool of candidate tasks via four stages: 8

PrismaDV: Automated Task-Aware Data Unit Test Generation

only a subset of scripts while leaving others unaffected. This setting challenges task-aware data validation methods, which must catch harmful corruptions for affected tasks while avoiding false alarms for robust tasks. All error configurations are released with the benchmark in our repository. Evaluation. Each pair of (task, evaluation batch) must be classified as pass (safe) or reject (erroneous). We compare these predictions to the ground-truth labels produced by executing tasks with assertion blocks enabled, and report precision, recall, and F1 score for detecting erroneous batches.

8.1 7

Experimental Evaluation

We experimentally evaluate PrismaDV in the following. We start by assessing its ability to discover individual constraints from datacode pairs (Section 8.1) and the ability of its tests to accommodate for the end-to-end impact of errors (Section 8.2), based on our benchmarks from Section 6. Next, we evaluate the ability of our proposed SIFTA approach to optimize the prompts of our system in Section 8.3. Finally, we conduct an ablation study in Section 8.4 to quantify the impact of our individual system modules.

Constraint Discovery from Data-Code Pairs

We evaluate the ability of our system to discover individual constraints from data-code pairs, which is the foundation for generating high-quality data unit tests.

Related Work

Data validation. Existing data validation systems vary in how rules are specified and inferred. Great Expectations offers a flexible assertion grammar but relies on manually defined expectation suites, limiting automation. Deequ [61] and TFDV [67] infer statistical constraints via data profiling, while Auto-Test [7] and AutoValidate [70] learn semantic constraints from large table corpora, with Auto-Validate focusing on string columns. These approaches are largely task-agnostic and depend primarily on observed data. DataPrism [19] incorporates downstream systems, using causal reasoning to identify data-profile violations that trigger failures. In contrast, PrismaDV generates task-aware validation rules across heterogeneous columns by jointly reasoning over data and code, capturing both explicit violations and latent data issues that induce abnormal program behavior.

Experimental setup and baselines. We leverage ICDBench, our individual constraint discovery benchmark, introduced in Section 6.1. Our evaluation protocol is as follows. For each of the 63 cases in the benchmark, we first expose the data-code pair in the form of a passing data sample and the example code to the method to evaluate. Next, we expose the data-to-pass and the data-to-reject from each case to the method and ask it to decide whether the data is valid. This leads to 126 binary decisions per method to evaluate, for which we compute the F1 score as quality metric. We evaluate PrismaDV with different LLMs from OpenAI and Google as well as a large range of additional baselines: • Outlier detection – We evaluate classic ML methods for outlier detection such as an isolation forest [43] (refered to as isolationforest) and a one-class SVM [63] (one-class-svm). In addition, we evaluate an adaption of the “partition summarisation” approach proposed in [58] (to which we refer as stats-novelty), where we compute the proposed descriptive statistics on the data, and decide upon rejection via a Maximum Mean Discrepancybased test [22]. Note that these methods are not task-aware and make their decisions based on the data alone. For each case to evaluate, we fit the outlier detection model (with default parameters) on the data sample and subsequently ask it to detect outliers in the data-to-pass and the data-to-reject. We mark either of them as invalid if the model says that outliers are present. We cannot include autotest [8] in our evaluation, as we did not manage to get its code to run despite several hours of trying. • Task-agnostic data unit test generation – We evaluate task-agnostic methods for data unit test generation, which generate their data unit test solely based on the data sample for each case to evaluate. In particular, we evaluate Deequ [61] (deequ) via its “constraint suggestion” feature [3], and Tensorflow Data Validation [57] (tensorflow-dv) via its “schema inference” feature. We apply the generated data unit on the data-to-sample and data-to-pass to make decisions for the benchmark. Note that we cannot include Great Expectations as a baseline, as it currently does not provide an automated way to generate data unit tests. • In-context learning with LLMs – We design two task-aware baselines, which use a single LLM call with a custom prompt to generate a data unit test in pydeequ syntax. The first baseline, referred to as zero-shot uses zero-shot prompt to an LLM which contains the data sample, the downstream code, the target column and asks for the list of constraints as result. The second baseline

Code understanding with LLMs. PrismaDV’s performance in extracting data assumptions depends on the code understanding capabilities of LLMs [34, 44, 47, 48, 50]. Recent studies have demonstrated that LLMs can reason about code execution behavior and program semantics [6, 41, 42]. LLMDFA [75] further shows that LLMs can serve as effective tools for performing data-flow analysis over source code, and RepoAudit [24] leverages such reasoning for repository-level auditing. Multiple benchmarks evaluate LLM code understanding and reasoning abilities [11, 23, 33, 45]. However, none of these benchmarks explicitly target implicit data assumptions embedded in code, which our ICDBench and EIDBench capture (Section 6). Prompt optimization for compound AI systems. LLM performance can vary substantially with prompt quality. Methods such as Chain-of-Thought [77] and Plan-and-Solve prompting [76] have been shown to improve reasoning performance. Beyond manual prompt design, prompt optimization methods [17, 26, 81, 87] aim to automatically improve prompts using feedback from previous invocations. Inspired by PyTorch’s abstraction philosophy [54], DSPy [38] offers a declarative framework for defining and optimizing prompt modules using textual feedback. Building on DSPy, MIPROv2 [53] selects high-performing instructions and demonstrations via Bayesian optimization. GEPA [2] evolves prompts with Pareto-based selection and can outperform reinforcement learning methods such as GRPO. TextGrad [83] treats prompts and intermediate outputs as optimizable variables and updates them via backpropagated natural-language feedback. EvoPrompt [25] applies evolutionary search to prompt optimization, while AlphaEvolve [52] extends this idea to code through execution-based evaluation. 9

Hao Chen, Arnab Phani, & Sebastian Schelter

Data to Pass Passed↑ False alarm↓

Data to Reject Rejected↑ Missed↓

Method

Taskaware?

Generates test?

one-class-svm isolation-forest stats-novelty

0 14 0

63 49 63

61 51 63

deequ tensorflow-dv

✓ ✓

1.9 -

44 32

19 31

zero-shot [gemini-2.5-flash] zero-shot [gemini-2.5-pro] zero-shot [gpt-4.1] zero-shot [gpt-5-mini] zero-shot [gpt-5]

✓ ✓ ✓ ✓ ✓

2.4 2.8 3.1 3.5 3.1

22 19 19 30 30

few-shot [gemini-2.5-flash] few-shot [gemini-2.5-pro] few-shot [gpt-4.1] few-shot [gpt-5-mini] few-shot [gpt-5]

✓ ✓ ✓ ✓ ✓

2.0 1.9 2.1 2.4 2.5

pocketflow-agent swe-agent

✓ ✓

prismaDV [gemini-2.5-flash] prismaDV [gemini-2.5-pro] prismaDV [gpt-4.1] prismaDV [gpt-5-mini] prismaDV [gpt-5]

✓ ✓ ✓ ✓ ✓

Avg. num. const.

False alarm or missed↓

F1 Score↑

2 12 0

65 61 63

0.0% 31.5% 0.0%

27 32

36 31

55 62

61.5% 50.8%

41 44 44 33 33

59 60 56 57 57

4 3 7 6 6

45 47 51 39 39

49.4% 44.7% 42.7% 60.6% 60.6%

27 34 29 33 36

36 29 34 30 27

51 54 52 59 54

12 9 11 4 9

48 38 45 37 36

52.9% 64.2% 56.3% 66.0% 66.7%

2.2 2.2

38 35

25 28

32 54

31 9

56 37

57.6% 65.4%

2.1 1.7 2.1 2.5 2.6

59 59 56 60 61

4 4 6 3 2

40 42 38 46 48

23 21 25 17 15

27 25 31 20 17

81.4% 82.5% 77.8% 85.7% 87.8%

Table 2: Results for constraint discovery from data-code pairs in ICDBench, with outlier detection methods, task-agnostic data unit test generation, LLM-based prompting and software engineering agents as baselines. The best values are marked in bold, the second best values are underlined. PrismaDV outperforms all baselines by a large margin of more than 20 points in F1 score.

few-shot extends this prompt with two manually crafted fewshot examples. We evaluate both baselines with various LLMs from OpenAI and Google. • Agentic systems – We treat data unit generation as a software engineering task and ask two agentic systems for software engineering to generate a data unit test in pydeequ syntax, based on the data sample and code from the benchmark, as well as a short task description. In particular, we evaluate the pocketflow agentic code generator [30] (pocketflow-agent), which is based on repeated series of code generation, testing and revision. We provide it with GPT-4.1 as base model, give it a budget of three full iterations, and manually restart it up to three times when it crashes. In addition, we evaluate a variant of the popular software engineering agent SWE Agent [82] (referred to as swe-agent), which is designed to automatically fix GitHub issues. We use the “swe-agent-mini” implementation with GPT-5 as base model, give it a budget of $0.50 per case, as well as a task description with two few-shot examples.

A first observation is that the outlier detection based approaches exhibit an extremely low performance. This is a validation of our benchmark design, since it shows that inspecting data alone is not sufficient to make correct decisions, but that an “understanding” of the code is required. The task-agnostic data unit tests, LLM prompting approaches and software engineering agents provide a mixed performance with the largest F1 scores in the high sixties (deequ with a score of 61.5%, few-shot [gpt-5] with a score of 66.7%, and swe-agent with a score of 65.4%). We observe that prompting approaches and agentic systems often generate the correct ground truth constraint, but suffer from the fact that they generate additional constraints which do not hold on the data. Our system PrismaDV manages to outperform them all by a large margin of more than 20 points with an F1-score 87.8% for prismaDV [gpt5]. Furthermore, it provides the lowest number of mispredictions, which is an important metric for operational deployments, where a misprediction might lead to a data engineer having to inspect the data manually to no avail. In summary, these findings confirm that constraint discovery from code is a difficult task, which benefits from a dedicated system and cannot be sufficiently solved with prompting or general engineering agents alone. Beyond the F1 margin, PrismaDV’s step-wise decomposition makes each stage easier to inspect.

Results and discussion. We list the results for this experiment in Table 2. For methods that generate data unit tests in the form of pydeequ constraints, we detail the average number of constraints generated. Furthermore, we count how often a method correctly identified the data to pass (Data to Pass > Passed) and how often it produced a false alarm by flagging this data as invalid (Data to Pass > False alarm). Analogously, we count how often each method correctly flagged data to reject as invalid (Data to Reject > Rejected) and how often it missed the rejection (Data to Reject > Missed). We compute the F1-score from these counts as the final quality metric.

8.2

End-To-End Error Impact

Next, we evaluate the ability of PrismaDV to detect the impact of data errors on downstream task behavior end-to-end, a setting that directly reflects the practical value of task-aware data unit tests. 10

PrismaDV: Automated Task-Aware Data Unit Test Generation

Data to Pass Passed↑ False alarm↓

Data to Reject Rejected↑ Missed↓

Precision↑

Metrics Recall↑

F1 Score↑

113 217

65.8% 65.1%

14.4% 42.7%

24.2% 50.3%

668 595 594 584 661

5 78 79 89 12

81.9% 65.0% 51.0% 65.7% 80.3%

2.4% 22.8% 12.7% 20.0% 8.7%

11.0% 31.0% 18.7% 30.4% 31.9%

696 597 675 612 636

615 566 613 570 614

58 107 60 103 59

65.8% 69.9% 66.8% 67.1% 75.8%

15.2% 26.8% 18.5% 25.2% 21.4%

35.5% 43.0% 40.4% 51.3% 47.2%

179

648

622

78.6%

20.3%

47.2%

551 597 484 560 715

276 230 343 267 112

533 478 551 484 368

140 195 122 189 305

81.7% 76.1% 81.6% 76.0% 70.5%

67.0% 73.0% 57.8% 68.9% 86.6%

72.9% 73.9% 67.0% 71.7% 77.4%

Method

Exec.

Non-exec.

deequ tensorflow-dv

67.0 -

0.0 -

134 352

693 475

560 456

zero-shot [gemini-2.5-flash] zero-shot [gemini-2.5-pro] zero-shot [gpt-4.1] zero-shot [gpt-5-mini] zero-shot [gpt-5]

48.3 12.4 26.1 17.1 34.1

20.3 2.8 4.6 4.7 3.5

20 203 100 164 75

807 624 727 663 752

few-shot [gemini-2.5-flash] few-shot [gemini-2.5-pro] few-shot [gpt-4.1] few-shot [gpt-5-mini] few-shot [gpt-5]

33.8 9.6 23.2 18.8 25.8

9.8 1.1 3.6 3.0 2.0

131 230 152 215 191

swe-agent [gpt-5]

24.1

1.7

prismaDV [gemini-2.5-flash] prismaDV [gemini-2.5-pro] prismaDV [gpt-4.1] prismaDV [gpt-5-mini] prismaDV [gpt-5]

36.4 28.5 34.1 42.9 44.5

0.0 0.0 0.0 0.0 0.0

Table 3: Detection performance with respect to the impact of data errors on downstream tasks in EIDBench. The best scores are highlighted in bold, the second-best underlined. PrismaDV outperforms all baselines by a large margin. Exec. reports the average number of executable constraints, and Non-exec. reports the average number of non-executable constraints.

Experimental setup. We leverage EIDBench, our end-to-end benchmark introduced in Section 6.2. Our evaluation protocol is as follows. We evaluate on the five datasets in the benchmark, which contain 60 downstream tasks in total. For each task, we provide 𝐷 sample and the “assertion-stripped” task code to the method to generate a data unit test. We then evaluate this test on the 25 corresponding data batches and compare pass/reject decisions to the ground-truth labels computed by EIDBench (Section 6.2), yielding 1,500 (task, batch) decisions. Note that while each batch 𝐷𝑖 is corrupted, it is not necessarily erroneous for every task; thus, the evaluation set contains both safe and erroneous cases (827 safe, 673 erroneous). We evaluate PrismaDV with a selection of the baselines used in Section 8.1. We omit anomaly detection approaches, the pocketflow agent and smaller LLM variants due to consistently poor performance in ICDBench from the the previous Section 8.1.

strict constraints. With our post-processing, all constraints generated by PrismaDV in the testing stage are executable. Across different backend LLMs, PrismaDV achieves consistent performance, with prismaDV [gpt-5] reaching the highest F1 score of 77.4%. At the same time, different backbone LLMs exhibit different trade-offs between missed rejections and false alarms. For example, prismaDV [gpt-5] raises relatively few false alarms (112), but it misses many erroneous cases (305). In contrast, prismaDV [gemini-2.5pro] misses very few erroneous cases (195), but it raises more false alarms (230). This highlights that PrismaDV’s preference depends on the backbone LLM, and practitioners can choose a model based on their tolerance for false alarms versus missed rejections.

Results and discussion. We list the results for this experiment in Table 3. For methods that generate data unit tests in the form of pydeequ constraints, we report the average number of executable constraints. We also count the non-executable constraints, which we exclude from the pass/reject decision (i.e., non-executable constraints do not cause rejection). We then report how often a method correctly marks a safe data partition as pass (Data to Pass > Passed) and how often it raises a false alarm on safe data (Data to Pass > False alarm). Similarly, we report how often erroneous data is correctly rejected (Data to Reject > Rejected) and how often it is missed (Data to Reject > Missed). We compute the F1 score from these counts as the final metric. Zero-shot and few-shot baselines perform substantially worse than PrismaDV. They show a strong tendency to generate more false alarms than misses, which is reflected in the gap between precision and recall. This suggests that these models generate overly

In the following, we validate that our proposed prompt optimization approach SIFTA (Section 5) improves PrismaDV’s performance.

8.3

Optimizing PrismaDV with SIFTA

Experimental setup. We evaluate prompt optimization for PrismaDV on EIDBench under the three generalization targets defined in Section 5.1: New Data, New Tasks, and New Data + New Tasks. We follow the end-to-end pass/reject protocol from Section 8.2 and report F1 scores. For each dataset 𝐷, we split its downstream tasks into 𝑇train , 𝑇eval , and 𝑇test with a 3:3:4 ratio. We also split the corresponding 25 data batches into an observed set 𝐷 obs and a held-out set 𝐷 new with a 1:1 ratio. We optimize prompts on 𝑇train × 𝐷 obs and use 𝑇eval ×𝐷 obs for evaluation; we then test on three target scenarios: New Data: (𝑇train ∪ 𝑇eval ) × 𝐷 new , New Tasks: 𝑇test × 𝐷 obs , and New Data + New Tasks: 𝑇test × 𝐷 new . We compare against the following baselines: Manual, our handtuned set of prompts for PrismaDV used in Section 8.2; and GEPA [2], a general-purpose prompt optimizer that learns from the Pareto frontier of optimization attempts. We include two GEPA variants. First, for GEPA[S], we configure GEPA following principles aligned 11

Dataset

F1 Score per Prompt Type GEPA[P] GEPA[S] Manual SIFTA

students hr_analytics sleep_health IPL imdb

70.50% 71.54% 54.85% 58.17% 62.28%

63.44% 68.44% 69.63% 62.50% 68.08%

70.10% 69.00% 58.42% 68.55% 64.06%

86.88% 70.69% 71.30% 64.27% 68.60%

Mean

65.34%

66.42%

67.71%

72.84%

students hr_analytics sleep_health IPL imdb

76.07% 84.17% 52.45% 43.50% 48.01%

76.96% 82.84% 51.54% 58.22% 49.58%

75.66% 83.19% 58.70% 55.53% 56.43%

85.30% 82.37% 56.98% 57.03% 54.75%

Mean

64.81%

63.83%

67.29%

69.76%

students hr_analytics sleep_health IPL imdb

73.33% 76.76% 61.31% 46.88% 51.58%

57.34% 78.37% 60.40% 58.97% 55.45%

65.85% 75.62% 68.33% 56.44% 64.81%

75.68% 78.92% 65.55% 60.76% 55.41%

Mean

64.45%

62.11%

66.11%

67.75%

New Data + New Tasks

New Tasks

Target Scenario

New Data

Hao Chen, Arnab Phani, & Sebastian Schelter

Results and discussion. We list the optimization performance in Table 4; for each method, we run the optimization twice and report the averaged results. SIFTA outperforms GEPA[P], GEPA[S] and the Manual baseline on average across all three target scenarios. In the New Data scenario, SIFTA achieves the largest average gain, exceeding the GEPA variants by 5.13 points. In the more challenging scenarios that require generalizing to new tasks, SIFTA’s optimization is slightly better than the Manual baseline. By contrast, the GEPA variants do not consistently match the performance of the Manual baseline across all three scenarios. This aligns with our observation that GEPA[P] struggles to propose effective updates for PrismaDV’s coupled, multi-module prompts. This indicate that GEPA cannot optimize PrsimaDV when the feedback text didn’t contains enough information like SIFTA. GEPA[S] can improve the single initial prompt but the optimization result can not match SIFTA.

8.4

Next, we conduct an ablation study to systematically validate that each system component introduced in Section 4 contributes meaningfully to the overall performance of PrismaDV. Specifically, we manually disable individual system modules and API methods in prismadv [gpt-5] and evaluate each resulting system variant on EIDBench, reporting the mean F1 score averaged over all five datasets included in the benchmark. The results shown in Table 5 confirm that every component plays a beneficial role in the system, as disabling any single component consistently leads to a measurable decrease in F1 score.

Table 4: Impact of optimizing the prompts of PrismaDV for different scenarios. SIFTA outperforms all baselines on average across all three test scenarios, with the largest average gain in the New Data scenario.

F1 score on EIDBench

Delta

prismaDV [gpt-5]

77.35%

w/o multicolumn constraints w/o dataflow analysis w/o assumption inference

76.43% 76.87% 77.18%

-0.92% -0.48% -0.17%

Variant

Ablation Study

Conclusion

This paper introduced PrismaDV, a task-aware data validation approach that synthesizes specialized data unit tests based on both the data and downstream tasks. To improve PrismaDV’s performance on specific datasets, we further proposed SIFTA, an optimizer that efficiently adapts the prompts used in PrismaDV’s modules based on failure precision on training examples. To evaluate task-aware data validation systems from diverse perspectives, we introduced two novel benchmarks, ICDBench and EIDBench.

Table 5: Ablation results on EIDBench for different variants of PrismaDV with GPT-5 as backing model. Each system module contributes to the overall performance.

Limitations and Future Work. Our prototype focuses on singlefile tasks over a single table. Supporting multi-file code and multitable inputs will require additional engineering, such as tracking dataflow across scripts and reasoning about joins and derived tables, but remains conceptually compatible with task-aware assumption inference. A promising next step is to extend our benchmarks with real industry workloads, enabling evaluation under more realistic codebases, data distributions, and operational constraints. By treating textual data assumptions as an intermediate representation, PrismaDV has the potential to support multiple validation DSLs, such as Great Expectations and Python assert statements. SIFTA currently relies on training batches with observed errors, which can be difficult to obtain in practice. Automating error injection could synthetically generate training batches, enabling optimization without relying on naturally occurring failures and making PrismaDV more suitable for cold-start settings. In SIFTA, we adopted a simple greedy search strategy, we leave evaluating more advanced strategies (e.g., Monte Carlo tree search) for future work.

with the original paper. For each training or evaluation example, we provide GEPA with a single simple prompt that takes the task and column profile information as input; after obtaining the output, we compute the F1 score between the prediction and the ground-truth safeness of 𝐷 new on the task. Second, for GEPA[P], we use GEPA to optimize PrismaDV modules, where each prompt describes a module’s function in one sentence. The same set of basic prompts is optimized by SIFTA. For all optimization approaches, we apply early stopping and terminate if the proposer fails to outperform the current best prompts for 20 proposing iterations. GEPA[P], GEPA[S], and SIFTA are all assigned a budget of 15 full evaluation executions, which is more than enough for the optimization process to converge. For SIFTA, we recalculate the training set condensation stage every 5 full evaluation executions. We repeat the experiment with two different random seeds, and set the training sample batch size to three for both methods. We use gpt-5 as the prompt proposer and gpt-4.1-mini as the backbone LLM of the data validation system for both methods. 12

PrismaDV: Automated Task-Aware Data Unit Test Generation

References

[24] Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. 2025. RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing. arXiv preprint arXiv:2501.18160 (2025). [25] Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. [n. d.]. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers. In The Twelfth International Conference on Learning Representations. [26] Sairam Gurajada, Eser Kandogan, and Sajjadur Rahman. 2025. Effectiveness of Prompt Optimization in NL2SQL Systems. In Novel Optimizations for Visionary AI Systems Workshop at SIGMOD 2025. https://openreview.net/forum?id= BnLbe5eQaP [27] GXCloud. 2025. ExpectAI. https://greatexpectations.io/blog/gx-expectAI%20/. [Online; accessed Aug-2025]. [28] Hazar Harmouch and Felix Naumann. 2017. Cardinality estimation: An experimental survey. Proceedings of the VLDB Endowment 11, 4 (2017), 499–512. [29] Alireza Heidari, Joshua McGrath, Ihab F Ilyas, and Theodoros Rekatsinas. 2019. Holodetect: Few-shot learning for error detection. In Proceedings of the 2019 International Conference on Management of Data. 829–846. [30] Zachary Huang. 2025. PocketFlow. https://github.com/The-Pocket/PocketFlow. [Online; accessed Aug-2025]. [31] Zhipeng Huang and Yeye He. 2018. Auto-detect: Data-driven error detection in tables. In Proceedings of the 2018 International Conference on Management of Data. 1377–1392. [32] Zezhou Huang and Eugene Wu. 2024. Cocoon: Semantic table profiling using large language models. In Proceedings of the 2024 Workshop on Human-In-the-Loop Data Analytics. 1–7. [33] Hamed Jelodar, Mohammad Meymani, and Roozbeh Razavi-Far. 2025. Large language models (llms) for source code analysis: applications, models and datasets. arXiv preprint arXiv:2503.17502 (2025). [34] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 (2023). [35] Joe Hellerstein. 2024. The Data School with Professor Joe Hellerstein – Big Shifts in Data and Analytics. https://www.youtube.com/watch?v=-J0dy3jtLDk. Online; accessed 12 Jan. 2026. [36] Eser Kandogan, Nikita Bhutani, Dan Zhang, Rafael Li Chen, Sairam Gurajada, and Estevam Hruschka. 2025. Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI. In 2025 IEEE 41st International Conference on Data Engineering Workshops (ICDEW). 18–27. doi:10.1109/ICDEW67478. 2025.00007 [37] Zohar Karnin, Kevin Lang, and Edo Liberty. 2016. Optimal quantile approximation in streams. In 2016 ieee 57th annual symposium on foundations of computer science (focs). IEEE, 71–78. [38] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. [n. d.]. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models. [39] Hoa Thi Le, Angela Bonifati, and Andrea Mauri. 2025. Graph Consistency Rule Mining with LLMs: an Exploratory Study. (2025). [40] Shuocheng Li, Yihao Liu, Silin Du, Wenxuan Zeng, Zhe Xu, Mengyu Zhou, Yeye He, Haoyu Dong, Shi Han, and Dongmei Zhang. 2025. Jupiter: Enhancing LLM Data Analysis Capabilities via Notebook and Inference-Time Value-Guided Search. arXiv preprint arXiv:2509.09245 (2025). [41] Changshu Liu and Reyhaneh Jabbarvand. 2025. A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models. arXiv preprint arXiv:2501.18482 (2025). [42] Changshu Liu, Shizhuo Dylan Zhang, Ali Reza Ibrahimzada, and Reyhaneh Jabbarvand. 2024. Codemind: A framework to challenge large language models for code reasoning. arXiv preprint arXiv:2402.09664 (2024). [43] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008. Isolation forest. In 2008 eighth ieee international conference on data mining. IEEE, 413–422. [44] Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and LINGMING ZHANG. 2024. RepoQA: Evaluating Long Context Code Understanding. In First Workshop on Long-Context Foundation Models @ ICML 2024. https://openreview.net/forum?id=hK9YSrFuGf [45] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. [n. d.]. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). [46] Mohammad Mahdavi, Ziawasch Abedjan, Raul Castro Fernandez, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, and Nan Tang. 2019. Raha: A Configuration-Free Error Detection System. SIGMOD (2019). [47] Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13.

[1] Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, and Nan Tang. 2016. Detecting data errors: Where are we and what needs to be done? Proceedings of the VLDB Endowment 9, 12 (2016), 993–1004. [2] Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. 2025. Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457 (2025). [3] Amazon. 2025. Automatic Suggestion of Constraints. https://github.com/ awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/examples/ constraint_suggestion_example.md. [Online; accessed March-2025]. [4] Nastaran Bassamzadeh and Chhaya Methani. 2024. A comparative study of DSL code generation: Fine-tuning vs. optimized retrieval augmentation. arXiv preprint arXiv:2407.02742 (2024). [5] Jeffery Cao, Lampros Flokas, Yujian Xu, Eugene Wu, Xu Chu, and Cong Yu. 2025. Prompt Editor: A Taxonomy-driven System for Guided LLM Prompt Development in Enterprise Settings. In Companion of the 2025 International Conference on Management of Data (Berlin, Germany) (SIGMOD/PODS ’25). Association for Computing Machinery, New York, NY, USA, 59–62. doi:10.1145/3722212.3725124 [6] Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2024. Reasoning runtime behavior of a program with llm: How far are we? arXiv preprint arXiv:2403.16437 (2024). [7] Qixu Chen, Yeye He, Raymond Chi-Wing Wong, Weiwei Cui, Song Ge, Haidong Zhang, Dongmei Zhang, and Surajit Chaudhuri. 2025. Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables. Proc. ACM Manag. Data 3, 3, Article 133 (June 2025), 27 pages. doi:10.1145/3725396 [8] Qixu Chen, Yeye He, Raymond Chi-Wing Wong, Weiwei Cui, Song Ge, Haidong Zhang, Dongmei Zhang, and Surajit Chaudhuri. 2025. Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables. Proceedings of the ACM on Management of Data 3, 3 (2025), 1–27. [9] CNN. 2023. A corrupt file led to the FAA ground stoppage. It was also found in the backup system. https://edition.cnn.com/travel/article/faa-ground-stopcauses/index.html. [Online; accessed Aug-2025]. [10] Databricks. 2025. Manage data quality with pipeline expectations. https://docs. databricks.com/aws/en/dlt/expectations. [Online; accessed Aug-2025]. [11] Saman Dehghan. 2024. Assessing Code Reasoning in Large Language Models: A Literature Review of Benchmarks and Future Directions. (2024). [12] Sijie Dong, Soror Sahri, Themis Palpanas, and Qitong Wang. 2025. Automated Data Quality Validation in an End-to-End GNN Framework. (2025). [13] Great Expectations. 2024. Great Expectations. https://greatexpectations.io/. [Online; accessed January-2025]. [14] Meihao Fan, Ju Fan, Nan Tang, Lei Cao, Guoliang Li, and Xiaoyong Du. 2025. AutoPrep: Natural Language Question-Aware Data Preparation with a MultiAgent Framework. PVLDB 18, 10 (2025), 3504–3517. https://www.vldb.org/ pvldb/vol18/p3504-fan.pdf [15] Anna Fariha, Ashish Tiwari, Alexandra Meliou, Arjun Radhakrishna, and Sumit Gulwani. 2021. CoCo: Interactive Exploration of Conformance Constraints for Data Understanding and Data Cleaning (SIGMOD ’21). Association for Computing Machinery, New York, NY, USA, 2706–2710. doi:10.1145/3448016.3452750 [16] Saeed Fathollahzadeh, Essam Manfsour, and Matthias Boehm. 2025. Demonstrating CatDB: LLM-based Generation of Data-centric ML Pipelines. In Companion of the 2025 International Conference on Management of Data. 87–90. [17] Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797 (2023). [18] Lampros Flokas, Jeffery Cao, Yujian Xu, Eugene Wu, Xu Chu, and Cong Yu. 2025. Towards a Framework for Hierarchical Text Segmentation using Large Language Models. In Proceedings of the Workshop on Data Management for End-to-End Machine Learning. 1–9. [19] Sainyam Galhotra, Anna Fariha, Raoni Lourenço, Juliana Freire, Alexandra Meliou, and Divesh Srivastava. 2022. Dataprism: Exposing disconnect between data and systems. In Proceedings of the 2022 International Conference on Management of Data. 217–231. [20] Google. 2023. Deliver trusted insights with Dataplex data profiling and automatic data quality. https://cloud.google.com/blog/products/data-analytics/dataplexdata-profiling-and-automatic-data-quality-are-ga?hl=en. [Online; accessed Aug-2025]. [21] Stefan Grafberger, Hao Chen, Olga Ovcharenko, and Sebastian Schelter. 2025. Towards Regaining Control over Messy Machine Learning Pipelines. In Workshop on Data-AI Systems (DAIS) at ICDE. [22] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A kernel two-sample test. Journal of Machine Learning Research 13, Mar (2012), 723–773. [23] Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065 (2024). 13

Hao Chen, Arnab Phani, & Sebastian Schelter

[48] Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an LLM to Help With Code Understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 97, 13 pages. doi:10.1145/3597503.3639187 [49] Mohammad Hossein Namaki, Avrilia Floratou, Fotis Psallidas, Subru Krishnan, Ashvin Agrawal, Yinghui Wu, Yiwen Zhu, and Markus Weimer. 2020. Vamsa: Automated provenance tracking in data science scripts. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 1542–1551. [50] Dung Manh Nguyen, Thang Chau Phan, Nam Le Hai, Tien-Thong Doan, Nam V Nguyen, Quang Pham, and Nghi DQ Bui. [n. d.]. CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs. In The Thirteenth International Conference on Learning Representations. [51] David Nigenda, Zohar Karnin, Muhammad Bilal Zafar, Raghu Ramesha, Alan Tan, Michele Donini, and Krishnaram Kenthapadi. 2022. Amazon sagemaker model monitor: A system for real-time insights into deployed machine learning models. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3671–3681. [52] Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. 2025. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131 (2025). [53] Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. 2024. Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 9340–9366. doi:10.18653/v1/2024.emnlpmain.525 [54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019). [55] Alberto Sánchez Pérez, Alaa Boukhary, Paolo Papotti, Luis Castejón Lozano, and Adam Elwood. 2025. An LLM-Based Approach for Insight Generation in Data Analysis. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 562–582. doi:10.18653/v1/2025.naacl-long.24 [56] Pouya Pezeshkpour, Eser Kandogan, Nikita Bhutani, Sajjadur Rahman, Tom M. Mitchell, and Estevam Hruschka. 2024. Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutions. CoRR abs/2402.01108 (2024). https://doi.org/10.48550/arXiv.2402.01108 [57] Neoklis Polyzotis, Martin Zinkevich, Sudip Roy, Eric Breck, and Steven Whang. 2019. Data validation for machine learning. MLSys 1 (2019), 334–347. [58] Sergey Redyuk, Zoi Kaoudi, Volker Markl, and Sebastian Schelter. 2021. Automating Data Quality Validation for Dynamic Data Ingestion.. In EDBT. 61–72. [59] Kenneth A Ross, Divesh Srivastava, Peter J Stuckey, and S Sudarshan. 1998. Foundations of aggregation constraints. Theoretical Computer Science 193, 1-2 (1998), 149–179. [60] Sebastian Schelter, Felix Biessmann, Tim Januschowski, David Salinas, Stephan Seufert, and Gyuri Szarvas. 2015. On challenges in machine learning model management. IEEE Data Engineering Bullettin (2015). [61] Sebastian Schelter, Dustin Lange, Philipp Schmidt, Meltem Celikel, Felix Biessmann, and Andreas Grafberger. 2018. Automating large-scale data quality verification. Proceedings of the VLDB Endowment 11, 12 (2018), 1781–1794. [62] Sebastian Schelter, Tammo Rukat, and Felix Biessmann. 2021. JENGA: A framework to study the impact of data errors on the predictions of machine learning models. EDBT (2021). [63] Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. 2001. Estimating the support of a high-dimensional distribution. Neural computation 13, 7 (2001), 1443–1471. [64] Amazon Web Services. 2025. AWS Glue Data Quality. https://docs.aws.amazon. com/glue/latest/dg/glue-data-quality.html. [Online; accessed Aug-2025]. [65] Amazon Web Services. 2025. pyDeequ. https://github.com/awslabs/pythondeequ. [Online; accessed Aug-2025]. [66] Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G Parameswaran, and Eugene Wu. 2024. Docetl: Agentic query rewriting and evaluation for complex document processing. arXiv preprint arXiv:2410.12189 (2024). [67] Shreya Shankar, Labib Fawaz, Karl Gyllstrom, and Aditya Parameswaran. 2023. Automatic and precise data validation for machine learning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2198–2207. [68] Shreya Shankar, Haotian Li, Parth Asawa, Madelon Hulsebos, Yiming Lin, J. D. Zamfirescu-Pereira, Harrison Chase, Will Fu-Hinthorn, Aditya G. Parameswaran,

and Eugene Wu. 2024. spade: Synthesizing Data Quality Assertions for Large Language Model Pipelines. Proc. VLDB Endow. 17, 12 (Aug. 2024), 4173–4186. doi:10.14778/3685800.3685835 [69] Chen Shen, Jin Wang, Sajjadur Rahman, and Eser Kandogan. 2024. Demonstration of a Multi-agent Framework for Text to SQL Applications with Large Language Models. In CIKM. 5280–5283. doi:10.1145/3627673.3679216 [70] Jie Song and Yeye He. 2021. Auto-validate: Unsupervised data validation using data-domain patterns inferred from data lakes. In Proceedings of the 2021 International Conference on Management of Data. 1678–1691. [71] Charlie Summers, Haneen Mohammed, and Eugene Wu. 2025. Please Don’t Kill My Vibe: Empowering Agents with Data Flow Control. arXiv preprint arXiv:2512.05374 (2025). [72] Tensorflow. 2025. TensorFlow Data Validation - An Example of a Key Component of TensorFlow Extended. https://colab.research.google.com/github/tensorflow/ tfx/blob/master/docs/tutorials/data_validation/tfdv_basic.ipynb. [Online; accessed March-2025]. [73] Dezhan Tu, Yeye He, Weiwei Cui, Song Ge, Haidong Zhang, Shi Han, Dongmei Zhang, and Surajit Chaudhuri. 2023. Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4991–5003. [74] The Verge. 2020. Excel spreadsheet error blamed for UK’s 16,000 missing coronavirus cases. https://www.theverge.com/2020/10/5/21502141/uk-missingcoronavirus-cases-excel-spreadsheet-error. [Online; accessed Aug-2025]. [75] Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, Xiaoheng Xie, and Xiangyu Zhang. 2024. LLMDFA: analyzing dataflow in code with large language models. Advances in Neural Information Processing Systems 37 (2024), 131545– 131574. [76] Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chainof-thought reasoning by large language models. arXiv preprint arXiv:2305.04091 (2023). [77] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837. [78] Wired. 2018. Timeline of trouble: how the TSB IT meltdown unfolded. https://www.theguardian.com/business/2018/jun/06/timeline-of-troublehow-the-tsb-it-meltdown-unfolded. [Online; accessed Aug-2025]. [79] Wired. 2020. How a Facebook Bug Took Down Your Favorite iOS Apps. https: //www.wired.com/story/facebook-sdk-ios-apps-spotify-tiktok-crash/. [Online; accessed Aug-2025]. [80] Jing Nathan Yan, Oliver Schulte, MoHan Zhang, Jiannan Wang, and Reynold Cheng. 2020. Scoded: Statistical constraint oriented data error detection. In SIGMOD. 845–860. [81] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers. In The Twelfth International Conference on Learning Representations. [82] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37 (2024), 50528–50652. [83] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. 2025. Optimizing generative AI by backpropagating language model feedback. Nature 639, 8055 (2025), 609–616. [84] Sepanta Zeighami, Yiming Lin, Shreya Shankar, and Aditya Parameswaran. 2025. LLM-Powered Proactive Data Systems. arXiv preprint arXiv:2502.13016 (2025). [85] Sepanta Zeighami, Shreya Shankar, and Aditya Parameswaran. 2025. Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees. Proc. ACM Manag. Data 3, 6, Article 311 (Dec. 2025), 26 pages. doi:10.1145/3769776 [86] Shuo Zhang, Zezhou Huang, and Eugene Wu. 2025. Data Cleaning Using Large Language Models. In 2025 IEEE 41st International Conference on Data Engineering Workshops (ICDEW). 28–32. doi:10.1109/ICDEW67478.2025.00008 [87] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. In The eleventh international conference on learning representations.

Record · ID 126511 · SHA-256 95dd87185fff868a
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.