From Data Querying to Data Investigations: Rethinking Natural Language Interfaces for Databases

From Data Querying to Data Investigations: Rethinking Natural Language Interfaces for Databases Fabian Wenz

Zixuan Chen

Carsten Binnig

TU Darmstadt and MIT Darmstadt, Germany

TU Darmstadt Darmstadt, Germany

TU Darmstadt, hessian.AI and DFKI Darmstadt, Germany

Natural language (NL) interfaces to databases have been optimized for the wrong problem. The dominant Text-to-SQL paradigm assumes that users ask questions that can be answered by single SQL queries. In practice, however, users seek assistance with solving data problems. This requires searching a database by sequences of SQL queries while reasoning over intermediate results instead of just running one SQL query. This paper therefore introduces a new paradigm for NL interfaces to data, which we call data investigations. We present 𝐷 2 , a first prototype of a data investigation system that embodies this vision by autonomously searching, reasoning over, and collecting data to solve data problems. Using a newly constructed benchmark based on the Murder Mystery dataset, we demonstrate the potential of 𝐷 2 for tasks that require data investigations with evidence-backed decisions, extending beyond the capabilities of traditional single-query question answering.

1

Introduction

Text-to-SQL is the wrong abstraction. Natural language (NL) interfaces to databases have advanced rapidly in recent years. Their dominant formulation is Text-to-SQL: a user asks a question in NL, and the system translates it into a SQL query whose result answers that question. Benchmarks such as Spider [10] and BIRD [6] have driven impressive progress under this paradigm. Yet they are built on a fundamental assumption. We argue that it is way too limiting to assume that users formulate requests that correspond to a single SQL query. In practice, they do not [3]. Users neither think in terms of SQL queries nor care whether their request requires one SQL statement or multiple ones. They simply expect a system to help them solve a problem using their data. From querying data to investigating data. Consider a physician interacting with a hospital database. Asking for the average age of patients with COVID-19 is a valid database query, but it is rarely the question that matters. A physician is much more likely to ask: Which treatment is more effective for older COVID-19 patients? Answering this question cannot be reduced to generating a single SQL query. Instead, the system must identify alternative treatment strategies, compare medications, account for differences in patient populations, analyze recovery outcomes, and reason about all factors contributing to successful treatments. Also, the outcome is not a single query answer but a collection of competing hypotheses (i.e., different treatments), each supported by different pieces of evidence. Recent studies on natural language interfaces suggest This paper is published under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution, provided that you attribute the original work to the authors and CIDR 2027. 17th Annual Conference on Innovative Data Systems Research (CIDR ’27). January 24-27, Amsterdam, The Netherlands.

DATA QUERYING

DATA INVESTIGATIONS

NL: What is the death

NL:

Simple 1-Shot Querying

Multiple Queries and Reasoning over Data

rate for COVID-19 patients?

SQL SELECT death_rate FROM

patients

WHERE diagnosis LIKE “%COVID-19%”;

DB OUTPUT death_rate

0.1%

Which treatment is most effective for older COVID-19 patients?

Is … a good treatment?

2

TEXT-TO-SQL

VS .

reason

arXiv:2609.03898v1 [cs.DB] 3 Sep 2026

Abstract

explor

e

What

1 treatments exists for COVID-19?

query

What effects did diseases have?

3

rea

son

DB

query query

rationalize

OUTPUT (evidence based)

Drug A is effective Is this or that 1) SUPPORTED if people have causing ASTHMA, SQL:… the effect? 2) CONTRADICTED if people have diabetes, SQL:… treatment X is … treatment Y is …

4

Figure 1: Comparison of traditional Text-to-SQL and Data Investigations. Text-to-SQL maps an NL question to a single SQL query, producing a direct result. In contrast, Data Investigations decompose a high-level NL question into multiple exploratory queries and reasoning steps, iteratively collecting and synthesizing evidence to verify or falsify potential solutions as output for the user question. that this investigative interaction is what users actually expect: they seek assistance in solving data-driven problems, which we call data investigations, rather than asking simple database queries [5]. Figure 1 contrasts this data investigative paradigm with the traditional Text-to-SQL workflow. By exploring alternative treatments, gathering supporting and contradicting evidence, and making the reasoning process transparent, data investigations enable users to understand, verify, and ultimately trust the system’s conclusions. This is a new data management workload. And the principle of data investigations extends far beyond healthcare. A business analyst may ask "why customer retention suddenly declined", a manufacturer may ask "where to build the next production facility", or a city planner may ask "why a traffic light behaved unexpectedly". None of these questions has a unique SQL query that provides the answer. Instead, the system must iteratively explore the available data, formulate possible hypotheses based on the data, collect supporting and contradicting evidence, revise its understanding as new observations emerge, and ultimately communicate well-supported conclusions to the user. We argue that this investigative process constitutes a fundamentally new data management workload that is distinct from translating natural language into SQL. Current systems are not designed for this workload. Today’s Text-to-SQL approaches primarily optimize the translation from NL into SQL [2, 6, 10]. While this is an important capability, it addresses only one component of the broader problem of a data investigation. Investigative workloads require managing an evolving search over the data, involving sequences of queries whose execution depends on reasoning over intermediate results rather than generating a single query upfront. One might argue that recent LLM agents equipped with database tools already provide this

DATA INVESTIGATIONS

CIDR’27, January 24-27, 2027, Amsterdam, The Netherlands

Wenz et al.

Data Detective + Investigation Board

NL:

Which treatment is most effective for older COVID-19 patients?

DATA DETECTIVE (multi-agent team)

DATA INVESTIGATION BOARD (the detectives’ shared memory)

store results

HYPOTHESES

H1: drug A helps older patients? EVIDENCE

old patients

SQL: … WHERE

organize hypothesis & agents

incon? ✓supported clusive drug per per deaths cured E3 E2 E3

age > 65 …

& comorbidity old patients

dosage & drug old patients

SQL: … JOIN

SQL: …

comorbidities…

dose_mg>=200 …

=22

asthma|drug_B|n=

…n=12|deaths=1

NULL|n=224|death=24 ↳no drug has less deaths

214|deaths=20

…n=202|deaths=24

drug_A|n=84|death

↳drug B & asthma has low death rate

NEW POTENTIAL HYPOTHESIS

asthma subgroup query

dosage effect

1 HYPOTHESIZER

que

ry

& s explores data and creates earch hypothesis

H2: effect depends on comorbidity?

𐄂contradict

drug E1 deaths per ed E1

🤖

↳low cure rate doesn’t say much

str evi uctu den re ce

eva

lua

te

) ate

re

c (&

rationalize

🤖 🤖

the data investigation question

2 ORCHESTRATOR

r r fo fo ce ok lo den esis and coordinates i ev oth p 3 EVIDENCE COLLECTOR hy

DB

organizes explanations

find data based facts to support or falsify hypothesis

🤖

4 JUDGE

🤖

5 REPORTER

weighs support vs. counter-evidence

summarizes & rationalizes

OUTPUT (evidence based)

Drug A is effective 1) SUPPORTED if people have ASTHMA, SQL:… 2) CONTRADICTED if people have diabetes, SQL:… treatment X is … treatment Y is …

Figure 2: Architecture of a Data Investigation System with two main system abstractions: (1) The Data Detective, a multiagent team that iteratively inquires the database, formulates and evaluates hypotheses. (2) The team stores supporting and contradicting evidence on a shared Data Investigation Board before producing an evidence-backed result. The result is the verdict on each hypothesis (supported or contradicted) and the summary of its evidence. 2 Our Vision: Data Investigation Systems capability [4, 8, 9]. Such systems represent an important step toward data investigations. However, as we show later in this paper, Figure 2 illustrates our vision for future data investigation systems today’s pure agentic approaches remain insufficient. First, agents using a running example where a clinician asks Which treatment typically tend to terminate early once they identify one plausible is most effective for older COVID-19 patients? A data investigation answer to a question, whereas many real-world data problems have system acts as a detective for data: it explores the database, develops multiple equally relevant solutions—for example, several effective hypotheses such as "Drug A helps older patients" by looking at the treatments for COVID-19. Second, their reasoning process is often data, and gathers evidence from data to support or falsify hypothedifficult for users to inspect as it is "buried" in a long text trace. ses. At the core, we argue that data investigation systems consist Towards Data Investigation Systems. The central argument of of two components: the Data Detective, a multi-agent system with this paper is that NL interfaces should no longer be viewed primarspecialized roles for hypothesis generation, evidence collection, ily as Text-to-SQL. The ultimate goal should instead be to help users and hypothesis evaluation (i.e., the hypothesis is true or false based solve problems based on the available data. We argue that this reon the evidence) and the Data Investigation Board, an externalized quires a new systems layer between users and relational databases, shared memory that organizes all state for data investigations, as which we coin data investigation systems. As a first step toward shown on the left of the figure. In the following, we first discuss this vision, we present 𝐷 2 , a prototype that introduces two new how data investigation workloads are different from existing worksystem abstractions for hypothesis-driven data analysis: the Data loads and then describe the design of the two main components to Detective and the Data Investigation Board. The Data Detective is a support such workloads. team of specialized agents that collaboratively investigates a data What are Data Investigations? A data investigation differs fundaproblem by generating hypotheses, and collecting supporting or mentally from Text-to-SQL. Rather than translating a NL question contradicting evidence for each hypothesis — i.e., how efficient difinto a single SQL query, the system receives a high-level investigaferent treatments are in our example. The Data Investigation Board tive objective, such as Which treatment is most effective for older serves as the detective’s workspace, externalizing the reasoning COVID-19 patients? in Figure 2. At the start, neither the relevant process as shared investigation state between agents. By making queries nor the relevant tables may be known to answer such a the investigation state a first-class systems object, 𝐷 2 enables sysuser question. A data investigation system must therefore explore tematic exploration of all alternative hypotheses and enables an the data first and see what treatments are available in the data, easier verifiability of the outcome of an analysis for humans. which involves exploring the schema and executing several queries Our contributions. To summarize, as the main contribution, this already. Based on this, a system can then formulate candidate hypaper introduces data investigations, a new paradigm for NL interpotheses (e.g., H1 in Figure 2). However, finding "Drug A" as a faces to data that shifts the focus from directly answering queries treatment for older patients does not mean it is an effective treatto solving more open-ended data problems. We present 𝐷 2 , a very ment. As such, the system next needs to find evidence in the data first prototype realizing this paradigm. We further introduce a for (or against) this hypothesis by further querying the data. In benchmark of data investigation workloads and use it to study the the example, the evidence E1, which is the result of a SQL query, capabilities and limitations of current systems. Finally, we outline contradicts H1 because treated and untreated patients over 65 show an agenda for building future data investigation systems. similar death rates. Also H1 was only one hypothesis; as such, the system continues collecting evidence for the other hypothesis. An

From Data Querying to Data Investigations: Rethinking Natural Language Interfaces for Databases

interesting characteristic of a data investigation workload is that it progressively discovers relevant hypotheses and evidence through exploration, querying the data as well as reasoning to decide which hypotheses hold and which do not. The Data Detective. The core component of our architecture of a data investigation system to conduct such a data investigation process is the Data Detective. Given the high-level user query, it coordinates hypothesis generation, evidence collection, and evidencebased reasoning toward a final conclusion, such as that "Treatment A helps patients with asthma but not those with diabetes". A simple implementation of a Data Detective would be to equip an agent with a SQL tool and let it iteratively explore and query the database. However, this not only keeps the investigation process implicit inside the LLM context, making it hard for the agent to keep track of an investigation state, but it also causes the agent to prematurely converge on a single hypothesis rather than systematically explore all alternatives available in the data, as we show in our initial evaluation in Section 4. To overcome these limitations, we thus design our Data Detective as a multi-agent system, which divides responsibilities of data investigation to avoid these shortcomings. As shown in Figure 2, each agent has a specialized role: a Hypothesizer independently generates candidate hypotheses (H1, H2) from data and thus guarantees a breadth of evaluation, while an Orchestrator coordinates the investigation and oversees that all hypotheses will be checked based on the available data. Furthermore, a team of Evidence Collectors gathers supporting or contradicting data as evidence for or against a hypothesis, while a Judge evaluates the evidence and comes to a verdict per hypothesis. Finally, once all hypotheses are checked, a Reporter summarizes the collected evidence for users to understand the reasoning process. The Data Investigation Board. The second key component is the Data Investigation Board, the externalized memory of the agents shown on the left of Figure 2. Instead of hiding the investigation state in the context of an LLM, the board explicitly represents the state of evolving investigation, linking hypotheses to evidence that was found so far to support or contradict hypotheses. While recent work on agentic context management advocates externalizing memory to efficiently support long-horizon reasoning [7], the Data Investigation Board goes beyond this and structures the memory in particular for data investigations. Important is that this structure is problem-independent and can be used for data investigations in any domain. Moreover, the investigation board provides a persistent representation that can be shared across all agents to coordinate investigations and also resume in case agents fail (which happens). In addition, it also supports parallel investigative branches (e.g., different collectors working on different hypotheses), but it also preserves all outcomes of hypotheses (rejected and supported ones), and allows users to understand not only the final conclusion but also the evidence and reasoning that led to it.

3

Our Prototype: The 𝐷 2 System

Based on the architecture outlined before, we have been building a first data investigation system we call 𝐷 2 . Figure 3 shows a representative run of 𝐷 2 over a COVID-19 patient database. We want to highlight that the workflow of 𝐷 2 is completely domain-independent. Replacing the COVID-19 database with a police, manufacturing, or any other database leaves the investigation process completely

CIDR’27, January 24-27, 2027, Amsterdam, The Netherlands

unchanged. In the following, we discuss in detail how 𝐷 2 works as well as the technical challenges we faced when building it.

3.1

The Data Detective

Unlike a ReAct-style agent that incrementally generates SQL until it reaches a first plausible answer, the task of the Data Detective is to systematically analyze and collect data-based evidence for all hypotheses that can be derived from the data. As shown in Figure 3, the Data Detective decomposes the investigation into five specialized agent roles. It is important that all agents work fully independently with the data, and the steps outlined below are not a strict execution sequence. Instead, agents collaborate and share state through the shared Data Investigation Board with clear interfaces for how they can contribute to or read from the board. Hypothesizer (step 1). Every investigation begins with understanding what potential answers to the user query might be. The role of the hypothesizer, however, is not to produce a single answer, but to explore the search space of possible hypotheses. For this, the hypothesizer agent has an SQL tool to explore the schema and execute queries. Given an investigative objective, it uses these tools and, based on the data, it proposes multiple candidate hypotheses that deserve further investigation by adding them to the investigation board. In our example, the hypothesizer first issues a broad query Q1 that computes statistics for all treatments for patients aged 65 and older, and from the result proposes three competing candidates: Drug A improves outcomes, the effect depends on comorbidities, and the effect depends on dosage. Orchestrator (step 2). The orchestrator provides the global control loop of the investigation. It picks hypotheses and decides which hypotheses should be investigated next, which should be abandoned (as it is not really answering the user query), and when sufficient evidence has accumulated to terminate the investigation. In the example, it selects two of the hypothesizer’s candidates – H1 (“Drug A helps older patients”) and H2 (“effect depends on comorbidities”) – and dispatches them to independent evidence collectors in parallel. Importantly, the orchestrator never plans and executes any SQL queries on the data. Instead, it reasons entirely at the level of all hypotheses which are available in the investigation board. Evidence collectors (steps 3a+b). Each hypothesis is assigned its own evidence collector. Collectors independently explore the database by issuing SQL queries to collect data-based evidence that helps to support or reject a hypothesis and adds the evidence to the Data Investigation Board. In the example, the collector assigned to H1 issues Q2 and Q3 to compare death rates by treatment and by severity, and concludes that H1 is contradicted; concurrently, the collector assigned to H2 issues Q4 and Q5 to compare outcomes across comorbidity subgroups. Since collectors share persistent investigation state via the board rather than holding it in their private transient reasoning context, multiple hypotheses can be investigated concurrently without interfering with one another. Judge (step 4). Once a collector decides sufficient evidence has been collected, it calls the judge. Continuing the example, the judge in our case weighs the two verdicts based on the collected evidence. This does not involve any database queries, but the main task of the judge is to reason over the evidence on the board and decide if a hypothesis is confirmed or contradicted based on the evidence. However, a judge can also decide that additional evidence needs

CIDR’27, January 24-27, 2027, Amsterdam, The Netherlands

Wenz et al.

DATA DETECTIVE (D²) DATA INVESTIGATION (question)

Which treatment is most effective for older COVID-19 patients?

1. HYPOTHESIZER queries DB broadly to surface candidates

🤖

Q1: overview by treatment for patients age ≥ 65 SELECT treatment, COUNT(*) AS n_patients, SUM(death) AS n_deaths, AVG(age) AS avg_age FROM patients WHERE age >= 65 GROUP BY treatment;

🤖

2. ORCHESTRATOR creates and queues hypotheses

Create hypotheses • H1: Drug A helps older patients • H2: Effect depends on comorbidities

Result / candidate hypotheses

(in parallel)

H2

H1

>SQL query ▦ query results

>SQL query ▦ query results

reason + query

Dispatch to evidence collectors

DB

• Drug A improves outcomes? • Effect depends on comorbidities? • Effect depends on dosage?

Hypothesizer searches for new leads (e.g., asthma, diabetes, dosage). Orchestrator queues H3, H4. hypothesis test them …

SQL Q2: outcomes by treatment (age ≥ 65)

continue

SELECT treatment, COUNT(*) AS n, SUM(death) AS deaths, ROUND(100.0*SUM(death) /COUNT(*),1) AS death_rate FROM patients WHERE age >= 65 GROUP BY treatment;

reason + query

>SQL query ▦ query results

SQL QM: adjusted by severity SELECT treatment, severity, COUNT(*) AS n, ROUND(100.0*SUM(death) /COUNT(*),1) AS death_rate FROM patients WHERE age >= 65 GROUP BY treatment, severity;

🤖

3b. EVIDENCE COLLECTOR (H2) 🤖🤖 🤖 tests H2: Effect depends on comorbidities SELECT comorbidity, treatment, COUNT(*) AS n, ROUND(100.0*SUM(death) /COUNT(*),1) AS death_rate FROM patients WHERE age >= 65 GROUP BY comorbidity, treatment;

4. JUDGE evaluates accumulated evidence

Label SQL evidence

structure, label, deduplicate, and classify each SQL with the global knowledge

Render audit graph and final report E2

contradicts

H1

Assessment

E4

supports

H2

E6

leads-to

H3

New leads to pursue

• Which comorbidities drive the difference (e.g., asthma, diabetes)? • Does dosage have an effect within key subgroups?

SELECT comorbidity, p_value_between_treatments FROM stats_comorbidity_effects WHERE age_group = '>=65';

5. REPORTER finalize and present

H1 (Drug A helps older patients): CONTRADICTED H2 (Effect depends on comorbidities): SUPPORTED Evidence indicates comorbidities explain treatment effect in older patients.

SQL QN: significant differences between treatments?

reason + query

🤖

Current verdicts

Verdict: CONTRADICTED

SQL Q4: outcomes by comorbidity (subgroups, age ≥ 65)

(COVID-19 data)

conclude

NEXT ROUND expand leads

3a. EVIDENCE COLLECTOR (H1) 🤖🤖 🤖 tests H1: Drug A helps older patients

…

Re-verify verdicts

Re-evaluate, check consistency.

Decision Propose new hypotheses • H3: Drug A helps older patients with asthma • H4: Drug A helps older patients with diabetes

Verdict: SUPPORTED

read for reporting (no changes)

DATA INVESTIGATION BOARD / HYPOTHESIS STORE (SQLite)

HYPOTHESES

EVIDENCE (facts + SQL provenance)

id

id

description

status

confidence

updated_at

H1 H2

Drug A helps older patients Effect depends on comorbidities Drug A helps older patients with asthma

contradicted supported

0.1 0.9

10:18 10:20

E1

open

0.5

10:21

E2

H3

…

description (fact) sql_text (truncated) # patients & deaths by treatment SELECT treatment, COUNT(*), (age ≥ 65) SUM(death) ... Similar death rate for Drug A vs SELECT treatment, COUNT(*), others (age ≥ 65) SUM(death) ...

…

LINKS (relationships / interpretations)

raw_result

recorded_at

by

▦ ▦

10:05

Investigator H1

10:07

Investigator H1

id L1 L2

src_type evidence evidence

src_id E2 E3

relation contradicts contradicts

…

dst_type hypothesis hypothesis

dst_id H1 H1

created_at 10:08 10:11

Figure 3: Our 𝐷 2 prototype in action. An investigation over a COVID-19 patient database answering “Which treatment is most effective for older patients?”. The Data Detective (top, steps 1–5) is a multi-agent system: the hypothesizer (1) explores the database and proposes hypotheses (e.g., Drug A benefits older patients; effects depend on comorbidities); the orchestrator (2) assigns them to evidence collectors (3a, 3b), which query the database in parallel for evidence and counter-evidence; the judge (4) evaluates evidence and comes to a verdict, (→1/2) it also refines hypotheses if needed, or triggers (5) the reporter to produce a provenance-backed report. The Data Investigation Board (bottom) provides the shared state, enabling consistent collaboration across agents and investigation rounds independently for all hypotheses. to be collected by starting the evidence collection again with hints maintains the externalized state of the investigation and also acts of what is missing, just like in a real investigation. Moreover, a as the interface between agents that read from and write to the judge can also identify new potential hypothesis that stems from board. In our prototype, the board is implemented as a SQLite the verdict and the collected evidence. For example, in our case, as database with three tables centered around the main concepts of a efficiency seems to depend on comorbidities, the judge proposes data investigation. two refined hypotheses, H3 (“Drug A helps older patients with The three investigation tables. The Hypotheses Table stores candiasthma”) and H4 (“Drug A helps older patients with diabetes”) date hypotheses together with their status and confidence, allowing which are processed by the agents as described before. 𝐷 2 to explore competing hypotheses without prematurely convergNext round (→ step 1/2). When the judge creates new potential ing to one. The Evidence Table records each observation along with hypotheses instead of concluding, the orchestrator queues them – its SQL query, a textual description of the query, the query rehere H3 and H4 – for a further round of evidence collection, closing sult, and provenance (timestamp and investigator), making findings the loop back to step 1 and 2. This is what allows the investigation traceable. The Links Table connects hypotheses and evidence, and a in Figure 3 to refine “effect depends on comorbidities” into concrete, state capturing whether evidence supports or contradicts a hypothtestable subgroup hypotheses. esis. Together, these tables store all relevant state of investigations Reporter (step 5). Finally, once the investigation for all hypotheses and thus separate state from execution in the detective. Moreover, terminates, the reporter summarizes all investigations from the the board also preserves the complete investigation history. Data Investigation Board with the goal of making it easy for the How does the board help? By externalizing hypotheses, evidence, user to verify the reasoning process based on the hypothesis and and their relationships, it provides a shared representation that evidence. More precisely, the reporter therefore cleans redundant enables agents to collaborate across investigation rounds while evidence, renders an audit graph connecting hypotheses to their each agent can focus on its particular role. We show in our initial cleaned evidence (e.g., evidence E2 contradicts H1, while E4 supports evaluation that state-of-the-art agents that keep everything in their H2). The goal of this audit graph is to be easily comprehensible by context lead to much lower coverage of which hypotheses are humans. As such, it is designed to be concise and structured. This being explored, and also the reasoning process is hard to follow by is in stark contrast to a reasoning trace that agents use, which is humans. Using the board, users can audit the investigation directly purely text-based and really hard for humans to understand easily. from the board without inspecting the underlying LLM reasoning traces, which are hard to comprehend cognitively by humans, as 3.2 The Data Investigation Board all reasoning is hidden in a long unstructured text trace. Moreover, the board makes long-running investigations resumable if agents The second core component of a data investigation is the Data crash or facts change. Investigation Board which we have also implemented as part of 𝐷 2 , shown at the bottom of Figure 3. The board, as discussed before,

From Data Querying to Data Investigations: Rethinking Natural Language Interfaces for Databases

4

CIDR’27, January 24-27, 2027, Amsterdam, The Netherlands

Initial Evaluation

Benchmark. As no benchmark exists for data investigations, we introduce a new benchmark inspired by the SQL Murder Mystery1 where the goal is to find a murderer given the facts in a database. However, unlike the original benchmark, which contains a single murderer case, our benchmark comprises 50 fictional murder cases with varying complexity. The task is to identify the murderer(s) per case and produce an evidence-backed hypothesis. The benchmark captures the core characteristics of data investigations: agents must first explore the database to discover relevant information, develop and refine competing hypotheses (i.e., who are potential murderers), and collect evidence (or counter-evidence) which is distributed across multiple tables in the entire database. Thus, success requires more than generating one correct SQL query; it requires a multi-step investigation that combines data exploration, reasoning, and evidence validation. Furthermore, we added cases of different difficulty which is determined by the combined number of difficulty dimensions (e.g., number of hints per suspect, normalized number of suspects per case, number of missing clues, and number of unreliable witnesses), yielding overall four categories: easy, medium, hard, and very hard. Harder cases require investigating more suspects, following longer chains of evidence, and, in some cases, identifying multiple suspects as murderers in one case. For checking quality, each case includes the ground-truth murderer set and the evidence required to solve the case (which we manually annotated), enabling evaluation of both the final verdict and the quality, completeness, and provenance of the investigation. Baselines. We compare 𝐷 2 against Claude Code and OpenClaw, two general-purpose agentic systems supporting iterative SQL tool use to explore and query the database. All three systems (𝐷 2 and the baselines) use Claude Sonnet 5 as their underlying model and receive identical case descriptions, schemas, database access, and instructions, and may iteratively issue SQL queries before producing a final answer. All collected evidence is kept in the system context. Metrics. We evaluate investigations along three dimensions: Correctness measures whether the system found the right answer (i.e., who was the murderer?). It is not about whether all potential suspects (i.e., hypotheses) were analyzed, which is the goal of completeness. Correctness is measured as F1 over precision (did we find the correct murderers?), which penalizes false accusations, and recall, which penalizes missed perpetrators (did we miss murderers?). Completeness measures whether the system explores the full set of all hypotheses (i.e., all suspects). A good data investigation system should explore them all, as sometimes multiple possible solutions exist, which is a big difference to querying data. We report the F1 score of hypotheses analyzed over the set of true hypotheses. Verifiability Ease measures how easily humans can understand and verify the reasoning of a data investigation system. Systems that explicitly structure their reasoning, for example by linking hypotheses to evidence, reduce the human effort required for verification. In contrast, approaches such as Claude Code or OpenClaw represent reasoning as unstructured text. Formally, the metric measures graph similarity between the generated reasoning structure and an ideal hypothesis-evidence-graph by solving a bipartite matching problem. Due to space constraints, we omit the formal definition 1 https://mystery.knightlab.com/

Figure 4: Overall investigation quality. Correctness, completeness, and verifiability ease. Higher-is-better. 𝐷 2 excels in all dimensions over baselines. and provide it in the technical report, which we will publish on arXiv.

4.1

Exp. 1 & 2: Overall Quality & Breakdown

Our first experiment evaluates the end-to-end investigation quality of 𝐷 2 , Claude Code, and OpenClaw across all 50 cases in our benchmark. For every system, we report the average correctness, completeness, and verifiability ease. Figure 4 shows the results higher is better along all dimensions. All three systems identify the correct murderer: 𝐷 2 reaches 95.26% correctness, compared with 94.64% for OpenClaw and 93.37% for Claude Code. If correctness was the only metric, one might conclude that all systems are equivalent. They are not. 𝐷 2 achieves 76.02% completeness, while both baselines remain near 32%. In other words, the baselines often find a plausible culprit and stop; 𝐷 2 instead is the only system that investigates the larger hypothesis space. The difference is even larger for verifiability ease: 68.88% for 𝐷 2 , compared with 2.22% for OpenClaw and 2.91% for Claude Code. The baselines produce walls of text. 𝐷 2 produces a concise and structured audit graph. Our second experiment examines how investigation quality changes as the difficulty of the investigation increases. Table 1 shows the expected trend: correctness declines as cases become harder. All systems solve the easy cases, while performance drops to roughly 79–84% on very hard cases. 𝐷 2 is therefore not immune to difficult investigations. The more interesting result is what happens beyond correctness. Across every difficulty level, 𝐷 2 remains substantially more complete and more verifiable than the baselines. Even on the very hard cases, it reaches high completeness while OpenClaw and Claude Code remain low or even degrade. 𝐷 2 ’s verifiability ease also remains above 65%, while both baselines stay below 3%. Overall, 𝐷 2 is comparable in correctness but clearly outperforms the baselines in completeness and verification ease.

4.2

Exp. 3: Ablation Study

Our final experiment studies which parts of our architecture contribute to the performance of 𝐷 2 . For this, we collapse specialized agent roles and compare the resulting variants with the full 𝐷 2 system. This preserves the same tools, data, and general execution capabilities while changing how investigative responsibilities are

CIDR’27, January 24-27, 2027, Amsterdam, The Netherlands

Wenz et al.

Table 1: Correctness, Completeness & Verification Ease by case difficulty. System

Corr.

Easy Comp.

Ver. Ease

Corr.

Medium Comp. Ver. Ease

Corr.

Hard Comp.

Ver. Ease

Corr.

Very Hard Comp. Ver. Ease

OpenClaw Claude Code 𝐷2

100.00% 100.00% 100.00%

32.64% 42.44% 86.06%

2.30% 2.80% 74.01%

98.31% 98.31% 100.00%

30.66% 33.21% 75.50%

93.55% 93.55% 95.24%

30.90% 25.31% 69.42%

2.15% 2.88% 65.85%

84.34% 79.47% 84.15%

35.25% 25.65% 71.07%

Table 2: Ablation study of the 𝐷 2 system. Configuration

Corr.

Compl.

Ver. Ease

Full 𝐷 2 Without Reporter Without Judge Merged Hyp. & Orch. Single Agent

95.26% 89.58% 77.25% 89.67% 86.63%

76.02% 73.54% 61.19% 29.81% 22.59%

68.88% 50.15% 62.98% 39.21% 21.34%

distributed. Moreover, all variants have access to the same investigation board. We evaluate the following configurations: (1) Full 𝐷 2 : the complete architecture with all roles. (2) Without Reporter: the final answer is generated directly from the accumulated evidence, without a reporter that cleans up evidence. We expect lower verification ease. (3) Without Judge: The verdict per hypothesis is taken by the evidence collector instead of a separate judge, which critically looks at the evidence. Also no new refined hypotheses are created. We expect lower completeness and correctness. (4) Merged Hypothesizer and Orchestrator: one agent both generates candidate hypotheses and manages the global investigation. We expect lower completeness. (5) Single Agent (as Detective): one agent manages all tasks, but we keep the Data Investigation Board. This should affect verification ease positively against baselines (Claude and OpenClaw) of Exp. 1. Table 2 shows the results. Without the judge, irrelevant hypotheses survive, and new relevant ones are not created, reducing correctness and completeness. Verifiability is not that much affected due to the separate reporter. Without the reporter, the investigation remains largely intact, but its structure is lost, causing the strongest drop in verifiability ease. Merging the hypothesizer and orchestrator produces too few hypotheses, primarily reducing completeness, while verifiability and correctness remain comparatively stable. Finally, the single-agent variant approaches the baselines: completeness drops substantially. However, as we keep the board, verification ease is higher as for the baselines in Exp. 1.

5

Research Roadmap

This paper argues that data investigations constitute a new data management workload and presents 𝐷 2 as a first step toward supporting it. Building general-purpose data investigation systems raises fundamental questions about how investigations should be represented, planned, optimized, executed, and verified. We outline several additional promising directions below. Optimization of execution. Classical query optimization assumes a fixed declarative query and optimizes its execution plan [1]. Data investigations introduce a broader optimization problem: the system must decide both what to investigate and how to execute it. Objectives extend beyond runtime to accuracy, completeness, verifiability, monetary cost, and resource consumption. An important

2.18% 2.84% 69.13%

2.27% 3.12% 66.21%

research direction is therefore multi-objective investigation optimization, including cost models, adaptive investigation plans, and principled trade-offs between competing objectives. Provenance & auditing. Investigation results require provenance connecting conclusions to the evidence and queries that produced them. The Data Investigation Board provides a natural substrate for such provenance, extending classical “why” and “why-not” analysis to investigations. More broadly, reverse investigation could use the investigation graph to identify missing, contradictory, or weak evidence on how an investigation reached an incorrect conclusion. Benchmarking. A key challenge is scaling from our prototype to realistic workloads and establishing more benchmarks for data investigation systems. This requires larger schemas, larger tables, more complex relationships, longer evidence chains, and more diverse questions, including open-ended “why” and “how” questions. Benchmarks with progressively more complex investigations are essential for evaluating systems beyond simple fact-finding tasks. As the number of agents grows, coordination also becomes more and more a systems problem. Heterogeneous data & specialized operators. Our prototype currently relies primarily on SQL and tables as its investigation substrate. A general investigation system should integrate heterogeneous data and specialized operators such as images and text as well as optimization solvers and satisfiability engines as operators. This raises database-style questions around cost-based tool selection, operator composition, intermediate-result reuse, and provenance across heterogeneous execution engines.

References [1] S. Chaudhuri. An overview of query optimization in relational systems. In Proceedings of the Seventeenth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, pages 34–43. ACM Press, 1998. [2] P. B. Chen, F. Wenz, Y. Zhang, M. Kayali, N. Tatbul, M. J. Cafarella, Ç. Demiralp, and M. Stonebraker. BEAVER: An Enterprise Benchmark for Text-to-SQL. CoRR, abs/2409.02038, 2024. [3] C. Dörschner, F. Jäkel, and C. Binnig. Towards More Realistic Natural Language Queries in Text-to-SQL Benchmarks. In Proceedings of the Workshop on HumanIn-the-Loop Data Analytics, HILDA 2026, pages 1–8. ACM, 2026. [4] T. Eckmann and C. Binnig. A vision for autonomous data agent collaboration: From query-by-integration to query-by-collaboration. In 16th Conference on Innovative Data Systems Research, CIDR 2026, 2026. [5] Y. Guo, D. Shi, M. Guo, Y. Wu, Q. Chen, and N. Cao. Talk2data: A natural language interface for exploratory visual analysis via question decomposition, 2023. URL https://arxiv.org/abs/2107.14420. [6] J. Li, B. Hui, G. Qu, et al. Can LLM Already Serve as A Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2023. [7] X. Li, R. Ming, M. Chu, S. Shao, R. Jin, and C. Xiong. Acm: Agentic context management for long horizon tasks, 2026. URL https://arxiv.org/abs/2607.23809. [8] C. Liu, M. Russo, M. J. Cafarella, L. Cao, P. B. Chen, Z. Chen, M. J. Franklin, T. Kraska, S. Madden, R. Shahout, and G. Vitagliano. Palimpzest: Optimizing ai-powered analytics with declarative query processing. In 15th Conference on Innovative Data Systems Research, CIDR 2025, 2025. [9] F. Wenz, F. Treutwein, K. Arenja, Çagatay Demiralp, and M. Stonebraker. Rubicon: Agentic ai for messy enterprise data, 2026. URL https://arxiv.org/abs/2604.21413. [10] T. Yu, R. Zhang, K. Yang, et al. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In EMNLP, pages 3911–3921, 2018.

Related documents

• Query-Guided Analysis and Mitigation of Data Verification Errors (Extended Version)#677516
• Query-Guided Analysis and Mitigation of Data Verification Errors (Extended Version)#674318
• Property Testing for Recursive Query Languages#926776
• Ten simple rules for teaching data science#273306
• Time to Move on: Querying without Nulls and Bags#444718