Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities So Hasegawa1 , Shailaja Keyur Sampat1 , Lei Liu1 , and Wei-Peng Chen1
arXiv:2607.06482v1 [cs.CL] 7 Jul 2026
Fujitsu Research of America, Santa Clara CA 95054, USA [email protected]
Abstract. Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.
1
Introduction
The ability to reason over structured data is a cornerstone of modern data science and a long-standing challenge in artificial intelligence. With the advent of Large Language Models (LLMs), we have witnessed a paradigm shift in how humans interact with complex information [17,25,32,8]. These models have led to the development of sophisticated agents designed to democratize data analysis, promising a future where any user can pose natural language questions to a dataset and receive accurate answers [9,24]. The ultimate vision is an autonomous system that not only retrieves information but also uncovers the knowledge hidden within raw data. However, a significant gap persists between this vision and reality, as the benchmarks lack real-world complexity. While datasets like WikiTableQuestions [20] and Spider [33] propelled research in semantic parsing and text-to-SQL, their controlled environments use small-scale tables. They largely neglect practical challenges such as massive table scales, the need to merge multiple tables,
2
So Hasegawa et al. Datasets
Tables > 5K Rows
Metadata
External Knowledge
Col_A
Question ・・・
Q1: Can you show the progression of the count …? Q2: In 2014, what were the average percentages …?
Answers LLM
A1:
X
Title: Boston Buildings Inventory Description: This dataset pulls… Publisher: City of Boston … Col_A: Column Label: … Column Description: … Invalid values: X = Less than 10 …
A2: [13.6, 6.44, 17.0]
(a) Table QA (b) Table Insight
DataGovBench
or
Insights
LLM Agents
Insight1: Widespread Retrofit Potential … Insight2: … In summary, the dataset provides…
Fig. 1: DataGovBench evaluates LLMs and agents on two table reasoning tasks using large, multi-table datasets supplemented with metadata and external knowledge. (a) The Table QA task requires models to answer simple or decomposable questions with textual or visual answers. (b) In contrast, the Table Insight task challenges models to perform open-ended exploratory analysis, proactively generating a list of insights and a summary without a specific user query. and the essential role of metadata and external knowledge. Beyond these data limitations, existing benchmarks have focused on direct fact retrieval [31,9,34]. Tasks like QA and text-to-SQL are about retrieving information to a query, while missing the capability of proactive insight discovery that data analysts exhibit. Consequently, this discovery-oriented skill remains largely unevaluated, as few benchmarks have formalized insight generation as a primary task [22,23]. To bridge these notable gaps in both data realism and task scope, we introduce DataGovBench, a comprehensive benchmark sourced from public repositories like Data.gov [28]. Our benchmark features two complementary tasks: Table Question Answering (Table QA) and Table Insight Generation (Table Insight), as illustrated in Figure 1. The Table QA task assesses factual reasoning over decomposable questions that explicitly include multiple sub-questions, requiring models to produce answers either in text or in visualizations. In contrast, the Table Insight task challenges models to perform expert-level insight derivation, requiring in-depth analysis and the discovery of trends. For the Table QA task, we use LLMs, aided by a novel table serialization, to generate a diverse corpus of QA pairs that are then meticulously verified by human annotators. For the Table Insight task, we address the challenge of subjectivity by using the expert-authored reports accompanying datasets as a ground truth. This process yields a novel benchmark that surpasses existing alternatives by holistically combining the complex Table QA and Table Insight with the diverse data complexities, such as large-scale, multi-tabular datasets that require metadata and external knowledge, as detailed in Table 1. Comprehensive evaluation unearths that latest top-performing LLMs achieve low accuracy for both tasks even with the agentic support. These results demonstrate that DataGovBench captures real-world challenges. Furthermore, we provide detailed qualitative analyses for both tasks, identifying common failure modes and uncovering two missing capabilities of current LLM agents: narrativelevel reasoning over tabular data and accurate fact retrieval from complex tables.
Data Analysis in the Wild Benchmark
Insight
WTQ [20] OTT-QA [3] FeTaQA [15] DataBench [19] TableBench [31] MMQA [30]
✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗
InsightBench [22] MT-RAIG [23] T2R-Bench [34]
✓ ✓ ✓
✗ ✗ ✗
✗ ✗ ✗
✗ ✗ ✓
DataGovBench
✓
✓
✓
✓
3
QA Dataset Characteristics Decomposable QA Visualization Large-table Multiple Tables Metadata External Knowledge Existing Benchmarks for Table Question and Answering ✗ ✗ ✗ ✗ ✓ ✗
✗ ✗ ✗ ✓ ✗ ✗
✗ ✓ ✗ ✗ ✗ ✓
✗ ✓ ✓ ✗ ✗ ✗
✗ ✓ ✗ ✗ ✗ ✗
✓ ✓ ✓
✗ ✓ ✗
✗ ✗ ✗
✓
✓
✓
Existing Benchmarks for Table Insight Generation
Table 1: Comparison of existing Table QA and Table Insight benchmarks with respect to task coverage and dataset characteristics.
In summary, our paper makes the following three contributions: – We introduce DataGovBench, featuring tasks for both multifaceted Question Answering and Insight Generation with ground-truths annotated by domain experts. These tasks comprehensively handle large, multi-tabular, and heterogeneous datasets representing complexity in real-world scenarios. – Extensive evaluation of state-of-the-art models reveals a crucial performance gap, as even top models achieve low accuracy despite agentic support. This underscores the benchmark’s difficulty and its alignment with actual challenges. – We provide a detailed qualitative analysis and ablation studies that identify key failure modes, offering a clear guide for the community to focus on primary areas for LLM-based data analytics.
2
Overview of DataGovBench
DataGovBench is a new benchmark designed to evaluate tabular reasoning in realistic scenarios. The data is derived from governmental open data portals (e.g., Data.gov [28], Data.gov.uk [27]), which host publicly available datasets from official institutions. Datasets on these portals reflect real-world complexity, consisting of a single or multiple tables containing a large number of records, and rich contextual information. This context is provided through metadata, such as textual descriptions of the dataset, and often supplemented with external knowledge like data dictionaries. The benchmark is designed to evaluate performance on two core data science tasks: Table QA and Table Insight. Table QA: This task requires models and agents to answer simple or decomposable questions with multiple sub-questions. The answers are provided in text or visualizations. Table Insight: This task challenges models and agents to perform exploratory analysis, generating substantive insights directly from the tabular data without an explicit user query.
4
2.1
So Hasegawa et al.
Benchmark Construction
The construction of DataGovBench involved a three-stage process as shown in Figure 2: 1) curating datasets from open data portals through a systematic filtering process, 2) annotating QA pairs via a human-in-the-loop, and 3) compiling ground-truth insights by leveraging professional reports. Data Curation Given the vast and decentralized nature of open data available online, a systematic collection and filtering process was imperative. Our process began by identifying 53 open data platforms with English as the primary language (a complete list is provided in the Table 6). We downloaded all available datasets from these platforms and then applied a series of filtering criteria including at least one of the tabular files covering over 5,000 records and metadata with description to understand the context of the dataset. A more detailed filtering procedure is outlined in the Appendix A.1. Annotation of Question and Answer Pairs To construct a high-quality, complex, and challenging set of QA pairs at scale while mitigating the need for resource-intensive manual annotation, we designed a four-stage generation pipeline that leverages LLMs with human-in-the-loop verification. Inspired by prior work [31], this approach ensures both diversity and correctness. The procedure is detailed below. 1. Question Generation: The initial stage focused on generating a diverse pool of candidate questions. To guide the output of the LLM, we first defined eight question types as presented in Appendix A.2. These types encompassed not only simple queries (e.g. Ranking, Aggregation) shared with existing benchmarks [31,30] but also complex decomposable questions. The prompt contains the table contents, the title and description of the dataset from the metadata, external knowledge when available, and the designated question type. Generating diverse and meaningful questions with LLMs requires providing them with a representative view of the table contents and value distributions. While prior works mainly focus on smaller datasets such that the entire table or the first several rows could be embedded into the prompt [31], we cannot apply similar approaches to tables with millions of records in our benchmark as it exceeds the limits of LLM context window. To address this challenge, we propose a technique called Feature type-specific table serialization, which creates a compact yet informative summary of a table by representing each column according to its data type. For instance, instead of listing all values in a categorical column, we provide only the set of unique categories. This serialization is a core component of our workflow, and a detailed description of the logic for various feature types is presented in Appendix A.2. To mitigate modelspecific biases in the generated questions, we employed an ensemble of four high-performance LLMs: GPT-4o, GPT-4o-mini [18], Gemini 2.0 Flash [5], and Gemini 1.5 Pro [26].
Data Analysis in the Wild
5
2. Question Scoring: Following generation, the candidate questions underwent an automated scoring and selection phase. Each question was evaluated against four qualitative criteria: relevance to the dataset, its potential to yield insightful or actionable information, sufficient analytical complexity with novelty, and clarity expressed in natural language. In an approach to quality filtering, we tasked each of the four aforementioned LLMs with acting as a judge, selecting its top-5 preferred questions from the generated pool based on the above criteria for each dataset. A score from 5 (highest) to 1 (lowest) was assigned to these selections. The scores from all four models were then aggregated for each question, resulting in a maximum possible score of 20. Based on this aggregated score, we selected the top-10 questions to proceed to the next stage. 3. Answer Generation: For each of the top-10 questions per dataset, an LLM was prompted to generate Python code that produces the correct answer. After executing the code from all four models, we measured the answer consensus to filter out questions that yielded unanimous agreement across all four LLMs. Such instances were deemed to indicate a low level of analytical complexity (e.g. single column filtering or aggregation), making them unsuitable for a benchmark in a real-world setting. An analysis of the generated questions shows that only 6% of the questions from the previous stage fall into this category, indicating that their removal does not drastically distort the original distribution as shown in Appendix A.3. 4. Human Verification: The remaining candidate QA pairs were subjected to a human verification and refinement process. Using a custom-developed annotation GUI as shown in Figure 7, human annotators with expertise in data analysis reviewed each component. Their tasks included: (1) revising the natural language question for clarity and precision; (2) validating, debugging, and refining the Python code for correctness and efficiency; and (3) verifying the final answer derived from the code. During this stage, annotators also filtered out questions for qualitative reasons, such as leading to uninformative answers, being too ambiguous to permit a definitive answer, or requiring external knowledge that was unavailable. This human-in-the-loop process yielded a curated set of 211 high-quality QA pairs with 178 datasets. Furthermore, all the questions were rephrased by separating the output format (e.g. bar chart, list of tuples) from the question, enhancing the naturalness, and paraphrasing the column names mentioned in the questions. As a final quality control measure, a second group of annotators with much experience in data science, who were not involved in the initial revision phase, performed a concluding review by using the different annotation GUI as presented in Figure 8. This step was designed to validate the quality and logical soundness of the final QA pairs, with a particular focus on ensuring the Python code was robust and accurately addressed the corresponding question. After the question scoring stage, we generated a total of 1,840 candidate questions, from which we curated 211 high-quality QA pairs, requiring 9,246
6
So Hasegawa et al.
Question Generation
Data.gov Data.gov.uk Open.canada
Col_A
Tables
・・・
Feature type-specific Table Serialization
Question type and description
LLMs
Prompt Engine for Question Generation
Question Scoring Metadata
Feature type-specific Table Serialization
Title: Description: …
Score Aggregation
Prompt Engine for Question Scoring
Human Verification
Answer Generation
External Col_A: Knowledge …
Feature type-specific Table Serialization
Code Execution
Prompt Engine for Answer Generation
(b) Annotation of Table QA (c) Annotation of Table Insight (a) Data Curation
Col_A
Tables
・・・
Report
Extract List of Insights
DataGovBench List of Insights
LLM
Summarized Insight
Fig. 2: Overview of the three-stage construction process of DataGovBench LLM calls in total. A detailed breakdown of the reasons for discarding candidate questions is provided in Appendix A.3. Annotation of Insight Establishing a ground truth for insight generation is inherently more complex than for question answering. The subjective nature of what constitutes a meaningful finding makes achieving consensus difficult, posing a significant challenge for both automated generation and evaluation. To address this limitation, we adopted official reports that accompany the open datasets. These human-authored documents contain the key findings and conclusions originally derived by specialists. Our process involved curating six datasets (Table 9) that included such reports. We then systematically extracted the principal findings from each document depending on the representation of the key findings in the report. If the report presents insights as bullet points, we directly treated each bullet point as one insight. If the report expresses findings in free text, we uploaded the report to NotebookLM [6] and prompted it to suggest ten insights from the results sections, followed by the manual verification of the quality of the extracted insights. We then synthesized the extracted insights into a standardized set of declarative sentences, and converted the set of sentences into a summary via Gemini 2.5 Flash [4]. The set of insights and the summary together form the ground-truth corpus for our insight generation task. 2.2
Benchmark Statistics
Dataset Statistics: A statistical overview of the datasets in DataGovBench is presented in Table 2. The benchmark comprises 178 unique datasets, featuring tables with an average of approximately 210K rows and 18 columns. The scale of the data is substantial, with the largest table containing up to 11.9M rows and 213 columns that exceed those found in most existing table QA benchmarks. The distribution of original open data websites is presented in Table 8. Figure 3 (a) illustrates the distribution of tables per dataset; notably, over 36% of the
Data Analysis in the Wild (a) Number of Tables per Dataset 1 63.5%
(b) Formats of External Knowledge
no external knowledge 42.7% 3.4% 3
13.5% 2 16.3% 3.4% 4 5
15.2%
json
3.4% docx 9.6% csv 8.4% 3.4% pdf 14.6% 2.8% html xml xlsx
(c) Number of Sub-questions 3 31.3%
2 19.0%
0.5% 5 4.3% 4 45.0% 1
7
(d) Question Types Multi-turn Insight Generation31.0%
Multi-turn 23.8%Follow-up
1.0% Multi-hop Lookup 10.0% Aggregation 5.7% Counting 5.2% 6.2% Multi-hop Complex Data 17.1% Numerical Reasoning Transformation Ranking
Fig. 3: Distributions of (a) number of tables per dataset, (b) format of external knowledge, (c) number of sub-questions, and (d) question types. datasets are multi-tabular, with five tables as the most frequent configuration. Furthermore, as shown in Figure 3 (b), over 57% of the datasets are accompanied by external knowledge to aid data interpretation, provided in various formats (e.g. PDF, XLSX). These characteristics—including large, multi-table schemas and the integration of external knowledge—underscore the alignment with realistic data analysis scenarios. Task Statistics: Table 2 provides a statistical summary of tasks in the benchmark. The Table QA task includes 211 question Table 2: Statistics of datasets & sets, which are categorized into 95 simple tasks questions and 116 decomposable questions. When these decomposable questions are Properties Value broken down into their constituent parts, #Datasets 178 the benchmark contains a total of 414 indi- #Average Rows 210K vidual questions. The distribution of sub- #Max Rows 11.9M questions per question set is detailed in #Average Columns 18.4 Figure 3 (c), which shows that over 55% #Max Columns 213 of all question sets include multiple sub- #Datasets for Table QA 173 questions. Figure 3 (d) also presents the #Datasets for Table Insight 6 distribution of question types. For the Ta- #Simple Questions 95 ble Insight task, a curated subset of six #Decomposable Questions 116 414 datasets, each accompanied by a report #Individual Questions from domain experts, is designated for insight generation evaluation.
3
Experimental Setup
Our benchmark evaluates performance on two distinct tasks: Table QA and Table Insight. For Table QA, models receive a user question that specifies the desired output format (text or visualization). In decomposable questions, the preceding conversational history is also provided. The goal is to produce a precise textual or visual answer. The Table Insight task challenges models to generate a list of findings and a summary directly from the given files. Implementation details, including model configurations, hyperparameters, and settings for fair comparison are available in Appendix B.1.
8
So Hasegawa et al. Dataset
(a) Answer Agent
・・・
Feature type-specific Table Serialization
Coding Module
Visualization Reflection Module
Self-correction Module
Text-answer Reflection Module loop
(b) Insight Agent Feature type-specific Table Serialization
Insert if not blank
History
Answer Agent
Question Generator
Insight Generator
add
add ・・・
Insert if not blank
add
・・・
Extract insights after the final loop
Fig. 4: Architecture of (a) Answer Agent and (b) Insight Agent
3.1
Evaluation Comparisons for Table QA
We evaluate a range of baselines, from general-purpose LLMs to specialized agents. LLM: We evaluate a diverse set of LLMs to investigate the capability of table reasoning. We selected open-source models from several categories: generalpurpose (Llama 3.1 [7], DeepSeek-R1 [8], and Qwen3 [32]), code generationspecific (Devstral [14] and Qwen3-Coder [21]), and table-specific (TableGPT2 [24]). We also include high-performance closed-source models, namely GPT-4o [18], GPT-5.1 [16], Claude Sonnet 4.6 [1], and Gemini-2.5 Flash [4]. Answer Agent: We leverage the Answer Agent, designed to robustly generate Python code for answering questions. The agent is composed of a practical assembly of existing following mechanisms shown in Figure 4 (a). – Feature type-specific table serialization: This module first processes the raw tables using the table serialization based on feature types as detailed in Appendix A.2. The resulting structured, textual representation of the data is then utilized as input for the subsequent modules. – Coding: With the serialized table, metadata, external knowledge, and the input question, this module generates Python code to produce an answer. It incorporates a self-correction mechanism: if code execution fails, a subsequent LLM is called to revise the code based on the error message with up to three revision attempts allowed. – Reflection: This module addresses cases where successfully executed code produces semantically incorrect outputs (e.g., a visualization with no data points or calculation resulting in NaN). A Vision-Language Model (VLM) or Multimodal Large Language Model (MLLM) is employed to analyze visual outputs, while an LLM analyzes textual results. If an issue is detected, the module triggers a revision loop to refine the code, and this can be repeated to three times. We initially performed preliminary experiments with the specialized table agents as baselines, including InfiAgent-DABench [9] and tablegpt-agent [24]; however, their performance on our benchmark was near-zero due to the complexity of our benchmark, so they were excluded from the final comparison.
Data Analysis in the Wild
3.2
9
Evaluation Comparisons for Table Insight
For the Table Insight task, we evaluate the following baseline agents. We employ only closed-source LLMs in these agents, given the relatively low performance of open-source models on Table QA. AgentPoirot [22]: This agent is designed for goal-oriented insight generation. It operates by first extracting the data schema and then generating a set of high-level questions. For each question, it generates an answer, interprets it, and recursively poses follow-up questions to dive deeper, and finally summarizes the obtained insights. This process follows a tree-like exploration structure (Figure 9 (a)), where each branch represents a dive into a analytical path. Insight Agent: We employ the Insight Agent shown in Figure 4 (b) that extends the core ideas of AgentPoirot in a applicable way. Similar to AgentPoirot, it iteratively generates questions, produces answers, and derives insights. The agent begins by generating the fixed number of high-level questions, which are then processed by our Answer Agent to obtain correct answers. Insights are subsequently synthesized from multiple QA pairs. These initial insights then seed the generation of new follow-up questions by combining multiple insights, continuing the cycle, ending by the summarization. Unlike AgentPoirot’s treestructured approach, the Insight Agent employs a directed acyclic graph (DAG)based approach as explained in Figure 9 (b), as the generation of new questions and insights selects and aggregates the context from all previously generated information instead of single insight or QA pair in the previous depth. 3.3
Evaluation Metrics
To assess the performance of agents and LLMs on our benchmark, we compare their outputs against the ground-truth references. Distinct evaluation protocols are employed for Table QA and Table Insight tasks. Table QA: The QA task is evaluated on accuracy under two settings: Whole, where all sub-questions in a decomposable question must be correct, and Individual, which measures sub-question-level accuracy. Correctness is determined by the modality of the answer. Text-based answers are judged by Exact Match (EM). Visualizations are evaluated using an MLLM-as-a-judge protocol, where four MLLMs (GPT-4o, GPT-4o-mini, Gemini 2.5 Flash, and Gemini 2.5 Pro [4]) assess the semantic equivalence between the predicted and ground-truth outputs. The judges are provided both the rendered images and their source code, and a prediction is deemed correct upon a majority consensus, requiring positive assessments from at least three of the four models. Table Insight: To evaluate the quality of generated insights, we adopt the methodology from InsightBench [22] computing LLaMA-3-Eval scores. We employ GPT-4o as the evaluator by replacing LLaMA3-70b [7]. We verified that the evaluator aligns with human perception through a meta-evaluation, as shown in Appendix B.3. The evaluation is conducted at two levels of granularity: – Summary-level Score: This metric assesses the holistic quality of the entire set of generated insights by comparing it against the ground-truth summary.
10
So Hasegawa et al. LLM
Table QA Table Insight w/o Answer Agent w/ Answer Agent AgentPoirot Insight Agent Whole Individual Whole Individual Insight Summary Insight Summary Closed-source LLMs
Gemini 2.5 Flash 0.310 Claude Sonnet 4.6 0.337 GPT-5.1 0.289 GPT-4o 0.242
0.401 0.478 0.411 0.333
Qwen3-30B 0.134 Qwen3-Coder-30B 0.123 Devstral-Small 0.152 DeepSeek-R1-14B 0.038 Llama3.1-8B 0.019 TableGPT2-7B 0.057
0.216 0.191 0.221 0.061 0.017 0.088
0.393 0.389 0.348 0.270
0.502 0.534 0.489 0.391
0.283 0.338 0.295 0.292
0.359 0.397 0.355 0.345
0.315 0.323 0.311 0.319
0.405 0.453 0.401 0.399
Open-source LLMs 0.199 0.186 0.186 0.095 0.019 0.066
0.309 0.280 0.316 0.140 0.034 0.109
Table 3: Main results of Table QA and Table Insight with LLMs and specific agents. w/o Answer Agent is without the agentic support.
– Insight-level Score: This metric provides a more fine-grained analysis. It measures the semantic alignment between each individual ground-truth insight and the most relevant insight from the predicted set, with the final score being the average of these individual comparisons. 3.4
Implementation Details
Answer Agent requires a VLM or a MLLM within its Reflection Module to validate visual outputs. For experiments involving closed-source models, we utilized their native multimodal capabilities across all modules. For the open-source agent configurations, we paired various LLMs with a specialized VLM, ChartR1-7B [2]. For the Table Insight task, each experiment was executed five times per model, and the scores were averaged across these runs to ensure the stability of our result. All results were obtained with the model temperature set to 0.0 to promote deterministic output. The other details about LLM configurations and hyperparameters are available in Appendix B.1.
4
Evaluation Results
4.1
Quantitative Evaluation Results
The main results for the Table QA and Table Insight tasks are presented in Table 3, where w/o Answer Agent means that the LLM is executed to generate the Python codes once based on the first 10 rows of the tables instead of the specific serialization. For the Table QA task, LLMs with the agentic support by Answer Agent consistently outperforms the base LLMs. With Gemini 2.5 Flash, for instance, it achieves relative improvements of approximately 27% in the Whole setting and 25% in the Individual setting. This suggests that a structured agentic framework is crucial, as standalone LLM reasoning is insufficient for such complex tasks. Despite these gains, the top absolute score in
Data Analysis in the Wild
11
the Whole setting remains below 0.4, highlighting the difficulty of the benchmark. Among the open-source models, Qwen3-30B achieves the highest score in the Whole setting, while Devstral-small performs best in the Individual setting. However, their performance still lags behind that of the closed-source models. Notably, TableGPT2-7B, a model specialized for tabular data, scores below 0.1 even when paired with our Answer Agent. In the Table Insight task, Claude Sonnet 4.6 has the best contributions to the agentic approach, achieving 0.338 for the Insight-level score by AgentPoirot and 0.453 for the Summary-level score by Insight Agent. However, the absolute scores remain low even with the top-performing model (below 0.35 for the Insightlevel score and below 0.5 for the Summary-level score), indicating substantial challenges remain in automated insight generation. 4.2
Qualitative Analysis
Error Analysis of Table QA We conducted an error analysis on the incorrect answers from Gemini 2.5 Flash with Answer Agent, and the results are categorized in Figure 5. The most prevalent issue, Condition Filter Error (32.4%), occurs when the model fails to apply implicit conditions not explicitly stated in the question. A common example involves datasets with aggregated and disaggregated data (e.g., population counts for ‘male’, ‘female’, and ‘total’); models often fail to apply appropriate filters to avoid double-counting, leading to incorrect calculations. The second most frequent category is Transformation Error (23.2%), which involves failures in data wrangling. Common mistakes include parsing-related failures various datetime formats (e.g. day-first format) by using pandas.to_datetime method or neglecting to convert numerical strings (e.g., "1,234,567") into integer or float types. To mitigate these errors, the more comprehensive yet efficient view of tables (e.g. exploratory data analysis results) is required in addition to the feature type-based representations for future work. Errors also arise from the inherent complexity of the tasks. ConCondition filter error text Handling Errors (8.7%) oc32.4% cur in decomposable questions where Transformation error 23.2% the model incorrectly uses the output 3.9% Misunderstanding external knowledge 4.3% from a flawed previous turn though 6.8% Wrong choice of tables 13.0% 8.7%7.7% Visualization error the generated logic is correct in most Output format error Syntax error Context handling error cases. Visualization Errors (6.8%) typically involve incorrect axis ranges, Fig. 5: Error distribution with Answer such as a timeline that does not match Agent the period specified in the question. Finally, the complexity of the benchmark’s data structure leads to specific errors. These include Wrong Choice of Tables (4.3%) in multi-table scenarios. This error occurs when the generated code fails to select the correct table from a multi-table dataset based on information provided in the metadata. For
So Hasegawa et al. Topic Relevance
Narrative Alignment
Insight Agent AgentPoirot
250
100
150 100
100
50
50
0
0
0
1
2
3
4
Topic Relevance Score
5
1
2
3
4
Narrative Alignment Score
5
200
150
50
Insight Agent AgentPoirot
250
200
Frequency
150
Quantitative Details Match
Insight Agent AgentPoirot
250
200
Frequency
Frequency
200
Qualitative Details Match
Insight Agent AgentPoirot
250
Frequency
12
150 100 50
1
2
3
4
5
0
Qualitative Details Match Score
1
2
3
4
5
Quantitative Details Match Score
Fig. 6: Insight perspective distributions by Insight Agent and AgentPoirot. example, a dataset may contain separate tables for annual statistics, with the year covered by each table specified only in the metadata. An error arises if a question pertains to a specific year, but the model fails to refer to the metadata and consequently queries the wrong table. Misunderstanding External Knowledge (3.9%), where the model fails to correctly apply information from provided data dictionaries to interpret the data. Fine-grained Analysis of Table Insight While the metrics provides a single value to measure the alignment between predicted and target insights, we conduct a more fine-grained analysis to understand model capabilities. We decompose the evaluation into four distinct perspectives: Topic Relevance (Does the predicted insight address the same topic as the target? ), Narrative Alignment (Does the prediction make the same core argument or conclusion as the target? ), Qualitative Details Match (Does the prediction mention the same specific names or entities as the target? ), and Quantitative Details Match (Does the prediction mention the same specific quantitative values as the target? ). For each of the 280 pairs that are randomly sampled 1 , we prompted GPT-4o to score the prediction on each perspective using a 1–5 scale. The score distributions for both Insight Agent and AgentPoirot are presented in Figure 6. The results show that both agents perform relatively well on Topic Relevance, the highest-level criterion. However, their performance drops on Narrative Alignment and Qualitative Details Match, where high scores (3–5) are less frequent. This suggests that although the agents identify the general topic of the target insight, they frequently fail to reproduce the core argument or the qualitative entities that support it. As a result, many generated insights lack the fundamental elements needed to convey the intended conclusion, indicating that the predictions remain misaligned with the target insights. This interpretation is supported by Table 11 where the generated insights lacking the correct core argument and qualitative entities receive lower scores than those that include them. Finally, both agents fail on Quantitative Details Match, with none of the predicted insights scoring 3 or higher. This highlights that both agents lack the ability to accurately calculate and present specific numerical values in their generated insights. 1
5 executions × 56 GT insights (10 per 5 datasets, 6 for 1 dataset)
Data Analysis in the Wild
13
These findings reveal two missing capabilities of LLM agents when handling real-world tables: narrative-level reasoning and accurate fact retrieval. For the former, both agents struggle to identify the central analytical argument underlying the target insight, suggesting that stronger reasoning over complicated tabular data is required. The latter capability is closely related to Table QA; therefore, the same observations discussed in the previous subsection apply here. In particular, the results suggest that a more comprehensive yet compact representation of tables is necessary to reduce various types of errors. 4.3
Ablation Study
Components of Answer Agent To assess the contribution Table 4: Ablation Study of Answer Agent of each Answer Agent Gemini 2.5 Flash GPT-4o module quantitatively, Settings Baseline 0.310 0.242 we conducted an ab- + Schema 0.308 0.244 + Schema + More number of rows 0.275 0.232 lation study against + Serialization 0.360 0.257 a naive baseline us- + Serialization + Reflection 0.379 0.267 0.393 0.270 ing the first 10 table + Serialization + Reflection + Self-correction (Answer Agent) rows (Table 4) for the Python code generation. Adding the table schema consisting of column names, data types, and statistics on missing and unique values does not affect the performance. We further increased the number of provided rows whose number was increased up to a maximum of 200, adjusted as needed to fit within the LLM context window. The performance degraded noticeably, suggesting that LLMs struggle to identify the key information needed for solution construction when faced with a larger pool of numerical or textual entries, compared to the first 10 rows. In contrast, the feature type-specific serialization led to gains over the baseline by providing a compact and reliable representation of the table that avoids the baseline’s tendency to rely on guessed values. This finding further supports the idea that a more comprehensive and efficient representation of tables leads to higher performance, as discussed in Section 4.2. Adding the Reflection module further improved performance across models by enabling the system to capture implicit constraints and infer missing reasoning steps. SQL vs. Python for Table Question Answering Text-to-SQL has been a primary approach for tackling table question answering, where natural language questions are translated into SQL queries. SQL is widely used for efficient data manipulation and retrieval, making it meaningful to investigate whether SQL is a more suitable output format than Python code for LLM-based table
Table 5: SQL vs. Python LLM
SQL Python
Gemini 2.5 Flash 0.241 0.545 GPT-4o 0.145 0.303
14
So Hasegawa et al.
reasoning. In this study, we exclude visualizationbased QA pairs from DataGovBench, as SQL does not provide native functions for generating visualizations. As a result, the evaluation is conducted on 145 textbased QA pairs. The LLM prompt includes feature type-specific serialization, as information about feature types and data types helps the model construct proper SQL queries. As shown in Table 5, the results indicate that SQL performs significantly worse than Python code. Error analysis of the generated SQL queries reveals that syntax errors and incorrect column selections occur more frequently than in generated Python code.
5
Related Works
Benchmarks for Table Question and Answering. Research in table-based question answering has been largely driven by a series of influential benchmarks. Early work such as WikiTableQuestions [20] established the task of answering natural language questions over Wikipedia tables. While foundational, this benchmark is limited to single tables and relatively simple questions. Although benchmarks like FeTaQA [15] and OTT-QA [3] introduced tasks requiring more complicated reasoning, they still focus on small-scale Wikipedia tables. The subsequent development of text-to-SQL benchmarks marked a significant leap in complexity. Spider [33] and BIRD [10] became the standards requiring models to generate complex SQL queries. More recently, developments in LLMs have enabled models to generate coherent Python code, leading to the construction of benchmarks that assess data analysis capabilities [31,9,19]. While these benchmarks were instrumental in advancing table reasoning capabilities, they do not comprehensively capture the challenges of real-world data analysis. The datasets are typically well-structured in scale. They often lack practical characteristics, such as rich metadata and supplementary external knowledge. Furthermore, many do not address the large multi-tabular datasets. DataGovBench comprehensively targets these limitations by incorporating all of these features: large tables, multi-table schemas, metadata, and extenral knowledge. Benchmarks for Table Insight Generation. Automated insight generation is a nascent and challenging area to benchmark, as the subjective nature of an "insight" complicates objective evaluation [35,12]. Recent work like InsightBench [22] and T2R-Bench [34] have advanced this area by introducing benchmarks for table insight generation. However, they do not fully capture the complexities of real-world data analysis. InsightBench focuses primarily on smallscale tables, while T2R-Bench lacks supporting metadata and external knowledge that are often required to interpret tabular data. In contrast, DataGovBench provides a more comprehensive benchmark by incorporating large-scale collections of tables together with relevant external knowledge sources.
Data Analysis in the Wild
6
15
Conclusion
To address the critical lack of realism in existing benchmarks, we introduced DataGovBench, a new benchmark built from open data. It features large, multitabular datasets, and incorporates external knowledge, and formalizes two key tasks: complex Question Answering (with decomposable questions and visualizations) and a novel Insight Generation task grounded in reports by domain specialists. Our extensive evaluation reveals that even state-of-the-art models with and without the agentic support struggle with low QA accuracy, highlighting the difficulity of both tasks. A detailed qualitative analysis provides a clear roadmap for future research. Future efforts should focus on both addressing these issues and efficiently scaling the benchmark’s size to ensure more robust evaluations. We believe DataGovBench will serve as a catalyst steering research toward building more robust agents capable of handling real-world data complexities. Use of Large Language Models We used LLM for the grammar correction and the words refinement to enhance the quality of the paper.
References 1. Anthropic: ntroducing claude sonnet 4.6. Blog Post (February 2026), https:// www.anthropic.com/news/claude-sonnet-4-6, accessed: March 5, 2026 2. Chen, L., Zhao, X., Zeng, Z., Huang, J., Zhong, Y., Ma, L.: Chart-r1: Chainof-thought supervision and reinforcement for advanced chart reasoner (2025), arXiv:2507.15509 3. Chen, W., Chang, M.W., Schlinger, E., Wang, W.Y., Cohen, W.W.: Open question answering over tables and text. In: International Conference on Learning Representations (2021) 4. Comanici, G., et al: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities (2025), arXiv:2507.06261 5. DeepMind, G.: Introducing gemini 2.0: our new ai model for the agentic era. Blog Post (December 2024), https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/, accessed: May 30, 2025 6. Google: Notebooklm (accessed July 10, 2025), https://notebooklm.google.com/ 7. Grattafiori, A., et al: The llama 3 herd of models (2024), arXiv:2407.21783 8. Guo, D., et al: Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645(8081), 633–638 (Sep 2025) 9. Hu, X., et al: Infiagent-dabench: evaluating agents on data analysis tasks. In: Proceedings of the 41st International Conference on Machine Learning (2024) 10. Li, J., et al: Can LLM already serve as a database interface? a BIg bench for largescale database grounded text-to-SQLs. In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023) 11. Liu, L., Hasegawa, S., Sampat, S.K., Xenochristou, M., Chen, W.P., Kato, T., Kakibuchi, T., Asai, T.: Autodw: Automatic data wrangling leveraging large language models. In: Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. p. 2041–2052 (2024)
16
So Hasegawa et al.
12. Majumder, B.P., Surana, H., Agarwal, D., Mishra, B.D., Meena, A., Prakhar, A., Vora, T., Khot, T., Sabharwal, A., Clark, P.: Discoverybench: Towards data-driven discovery with large language models (2024), arXiv:2407.01725 13. Wes McKinney: Data Structures for Statistical Computing in Python. In: Proceedings of the 9th Python in Science Conference. pp. 56 – 61 (2010) 14. MistralAI: Upgrading agentic coding capabilities with the new devstral models (accessed July 10, 2025), https://mistral.ai/news/devstral-2507 15. Nan, L., et al: FeTaQA: Free-form table question answering. Transactions of the Association for Computational Linguistics 10, 35–49 (2022) 16. OpenAI: Gpt-5.1: A smarter, more conversational chatgpt. Blog Post (November 2025), https://openai.com/index/gpt-5-1/, accessed: March 5, 2026 17. OpenAI, et al: Gpt-4 technical report (2024), arXiv:2303.08774 18. OpenAI, et al: Gpt-4o system card (2024), arXiv:2410.21276 19. Osés Grijalba, J., Ureña-López, L.A., Martínez Cámara, E., Camacho-Collados, J.: Question answering over tabular data with DataBench: A large-scale empirical evaluation of LLMs. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). pp. 13471–13488 (May 2024) 20. Pasupat, P., Liang, P.: Compositional semantic parsing on semi-structured tables. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 1470–1480 (2015) 21. Qwen: Qwen3-coder: Agentic coding in the world (accessed March 6, 2026), https: //qwen.ai/blog?id=qwen3-coder 22. Sahu, G., Puri, A., Rodriguez, J.A., Abaskohi, A., Chegini, M., Drouin, A., Taslakian, P., Zantedeschi, V., Lacoste, A., Vázquez, D., Chapados, N., Pal, C., Rajeswar, S., Laradji, I.H.: Insightbench: Evaluating business analytics agents through multi-step insight generation. In: ICLR (2025) 23. Seo, K., Kwon, D., Lee, D.: MT-RAIG: Novel benchmark and evaluation framework for retrieval-augmented insight generation over multiple tables. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 23142–23172 (2025) 24. Su, A., et al: Tablegpt2: A large multimodal model with tabular data integration (2024), arXiv:2411.02059 25. Team, G., et al: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023) 26. Team, G., et al: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (2024), arXiv:2403.05530 27. UK Government: Data.gov.uk, https://www.data.gov.uk/, accessed: 2026-03-11 28. U.S. General Services Administration: Data.gov, https://www.data.gov/, accessed: 2026-03-11 29. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Huggingface’s transformers: State-of-the-art natural language processing (2020) 30. Wu, J., Yang, L., Li, D., Ji, Y., Okumura, M., Zhang, Y.: MMQA: Evaluating LLMs with multi-table multi-hop complex questions. In: The Thirteenth International Conference on Learning Representations (2025)
Data Analysis in the Wild
17
31. Wu, X., et al: Tablebench: A comprehensive and complex benchmark for table question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 25497–25506 (2025) 32. Yang, A., et al: Qwen3 technical report (2025), arXiv:2505.09388 33. Yu, T., et al: Spider: A large-scale human-labeled dataset for complex and crossdomain semantic parsing and text-to-SQL task. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 3911–3921 (2018) 34. Zhang, J., et al: T2R-BENCH: A benchmark for real world table-to-report task. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 22427–22451 (Nov 2025) 35. Zhang, R., Elhamod, M.: Data-to-dashboard: Multi-agent llm framework for insightful visualization in enterprise analytics (2025), arXiv:2505.23695
A
Benchmark Construction
A.1
Data Curation
Data Collection Table 6 lists 53 websites, from which we downloaded all datasets along with their metadata via APIs (e.g., CKAN API 2 ). Data Filtering All downloaded datasets were subjected to a rigorous filtering and curation process to select those most suitable for real-world data analytics tasks. This process involved three main stages: dataset filtering, external knowledge identification, and metadata standardization. First, data filtering is conducted based on the following conditions: – The dataset had to contain at least one CSV file that was correctly formatted and readable by the pandas.read_csv method. Files that were HTML or XML in content despite having a .csv suffix were excluded. – At least one CSV file within the dataset was required to have more than 5,000 rows. Additionally, tables with five or more blank columns were discarded. – Each dataset needed to be accompanied by a textual description. The license was also required to permit redistribution; for datasets from Data.gov where the license was often unspecified in the metadata, we performed manual verification on the source webpage. – ArcGIS-based datasets, which are primarily geospatial, were excluded from our analysis. Following the filtering stage, we systematically searched for external knowledge (e.g., data dictionaries) within each dataset using a set of heuristic rules: – First, an automated search was performed for files with names containing "data dictionary" or "datadictionary". 2
https://github.com/ckan/ckanapi
18
So Hasegawa et al.
– Next, a platform-specific rule was applied for Data.gov datasets. We observed that when a JSON file is provided alongside CSV, XML, and RDF files, it often contains column-level descriptions. In such cases, the JSON file was designated as external knowledge. – If these automated heuristics failed, we performed a manual inspection of the dataset’s contents to locate any other supplementary documentation that could serve as a data dictionary. Finally, the original metadata for each curated dataset was processed and standardized. This step created a concise metadata format specific to our benchmark by removing redundant or irrelevant information from the source. The following is an example of the specific metadata from Indiana Arrest Data of Indiana Data Hub. Converted Metadata "identifier": "d39f6598-efbb-40a7-a694-6a9b8d2dc2dc" "dataset_title": "INDIANA ARREST DATA" "dataset_description": "This dataset is the underlying data of the Indiana Arrests Dashboard which displays counts of individuals arrested, arrests, charges by offense category, dispositions, country and time period in Indiana beginning in 2008 through the present year. \r\n\r\nArrest data comes from the Criminal History Repository System (CHRIS). Data feeding into the CHRIS system comes from three main sources. Arrest data comes from the LiveScan system, which is used for fingerprinting and capturing other pertinent information at the time of the arrest. Criminal disposition data are maintained by prosecutors in ProsLink system, and by the courts in the Odyssey system. \r\n\r\nData Notes:\r\n\r\n1. Arrest data are sent to ISP soon after the arrest occurs, but disposition data have a lag of approximately seven months as the case makes its way through the legal system. \r\n\r\n2. Text descriptions of the original offenses are provided by the arresting officer when the offender is arrested. Later, the prosecutor's office or court provides a text description of the filed offense, along with the Indiana Code title, article, chapter, and section (e.g.35-48-4-6). The filed offense may be amended later. \r\n\r\n3. Arrest County is determined by the location of the booking agency. If the booking agency is missing, then the arresting agency is used. \r\n\r\n4. The count of individuals/arrests /charges by offense category can add up to more than the grand total because one individual/arrest/charge can fall into multiple categories (e.g. DUI is counted in the \"Drug\" and \"Traffic\" categories. \r\n\r\n5. Arrest categories and subcategories are determined based on keywords found in a free text description of the offense. About 7% of offenses have a description that has not yet been categorized." "publisher": "Indiana State Police" "landingPage": "Indiana State Police" "license": "Creative Commons Attribution" "distribution": [{"file_name": "data9.csv",
Data Analysis in the Wild
19
"file_title": "ARREST DATA 2022 Q3", "file_description": null, "downloadURL": "https://hub.mph.in.gov/dataset/d39f6598efbb-40a7-a694-6a9b8d2dc2dc/resource/00cd698d-e26b -458a-861b-4c355b77ab20/download/ isp_arrest_data_2022_q3.csv", "accessURL": "https://hub.mph.in.gov/dataset/d39f6598efbb-40a7-a694-6a9b8d2dc2dc/resource/00cd698d-e26b -458a-861b-4c355b77ab20/download/ isp_arrest_data_2022_q3.csv"}, {"file_name": "data37.csv", "file_title": "ARREST DATA 2015 Q3", "file_description": null, "downloadURL": "https://hub.mph.in.gov/dataset/d39f6598efbb-40a7-a694-6a9b8d2dc2dc/resource/8b2b54fe-363a-46 f7-9c3b-197cce01616f/download/isp_arrest_data_2015_q3 .csv", "accessURL": "https://hub.mph.in.gov/dataset/d39f6598efbb-40a7-a694-6a9b8d2dc2dc/resource/8b2b54fe-363a-46 f7-9c3b-197cce01616f/download/isp_arrest_data_2015_q3 .csv"}, {"file_name": "data20.csv", "file_title": "ARREST DATA 2019 Q4", "file_description": null, "downloadURL": "https://hub.mph.in.gov/dataset/d39f6598efbb-40a7-a694-6a9b8d2dc2dc/resource/bd011a33-0652-4 ad7-8d90-6c1019d6385c/download/ isp_arrest_data_2019_q4.csv", "accessURL": "https://hub.mph.in.gov/dataset/d39f6598efbb-40a7-a694-6a9b8d2dc2dc/resource/bd011a33-0652-4 ad7-8d90-6c1019d6385c/download/ isp_arrest_data_2019_q4.csv"}, {"file_name": "data15.csv", "file_title": "ARREST DATA 2021 Q1", "file_description": null, "downloadURL": "https://hub.mph.in.gov/dataset/d39f6598efbb-40a7-a694-6a9b8d2dc2dc/resource/9c7960c6-417b-45 e6-9ace-b75958dd91de/download/isp_arrest_data_2021_q1 .csv", "accessURL": "https://hub.mph.in.gov/dataset/d39f6598efbb-40a7-a694-6a9b8d2dc2dc/resource/9c7960c6-417b-45 e6-9ace-b75958dd91de/download/isp_arrest_data_2021_q1 .csv"}, {"file_name": "data14.csv", "file_title": "ARREST DATA 2021 Q2", "file_description": null, "downloadURL": "https://hub.mph.in.gov/dataset/d39f6598efbb-40a7-a694-6a9b8d2dc2dc/resource/1ff2cf5f-69ef -4139-bcb4-036f66787172/download/ isp_arrest_data_2021_q2.csv",
20
So Hasegawa et al.
"accessURL": "https://hub.mph.in.gov/dataset/d39f6598efbb-40a7-a694-6a9b8d2dc2dc/resource/1ff2cf5f-69ef -4139-bcb4-036f66787172/download/ isp_arrest_data_2021_q2.csv"}], "external_knowledge": ["data68.xlsx"]}
A.2
Data Annotation
Question Types During question generation, specific question types are provided to the LLM to guide the formulation of questions. We use the following eight question types, with each type’s name and description supplied to the LLM. Multi-turn Follow-up and Multi-turn Insight Generation correspond to decomposable questions. – Aggregation: Questions involving aggregated answers based on the statistical operations, such as mean, sum or mode – Ranking: Questions involving answers based on the ranking – Counting: Questions involving answers based on counting something – Multi-hop Lookup: Question involving extracting single cell value from the table based on multiple reasoning steps – Multi-hop Numerical Reasoning: Questions involving numerical answers based on multiple reasoning steps – Complex Data Transformation: Question involving complex data transformation, such as aggregation or filtering across multiple dimensions, creating new columns, filtering with context-dependent logic, resolving entity references across rows, or merging multiple tables – Multi-turn Follow-up: Question involving multi-turn follow-up questions that build on previous answers or context from table data, requiring the model to maintain state and context across multiple interactions – Multi-turn Insight Generation: Question involving multi-turn insight generation that requires the model to generate insights or summaries based on previous answers or context from table data, requiring the model to maintain state and context across multiple interactions. Questions in the intermediate turn ask to provide not only text-based answer but also text-based complicated statistical information (e.g. correlation) and visualization Feature type-Specific Table Serialization Our serialization process generates a compact textual representation of a table by summarizing its global properties and providing detailed, feature type-aware information for each column. The serialized text begins with the dimensions of the tables (number of rows and columns), followed by a per-column breakdown. For each column, the serialization includes: the inferred feature type, the Pandas data type [13], the percentage of NaN values, and a feature-specific textual summary. The primary feature type is determined by a Feature Type Inference (FTI). Our FTI module follows a design similar to that in [11]. This model classifies each column into
Data Analysis in the Wild
21
Table 6: List of Open Data Websites Websites
URL
Data.gov https://data.gov/ California Open Data Portal https://data.ca.gov/ Hawaii Open Data https://opendata.hawaii.gov/ Analyze Boston https://data.boston.gov/ City of Houston Open Data https://data.houstontx.gov/ The Indiana Data Hub https://hub.mph.in.gov/ Milwaukee Open Data https://data.milwaukee.gov/ Open Data SA https://data.sanantonio.gov/ Pompano Beach Open Data https://data.pompanobeachfl.gov/ America’s Education data https://data.ed.gov/ Energy Data eXchange https://edx.netl.doe.gov/ California Health and Human Services Open Data Portal https://data.chhs.ca.gov/ California Natural Resources Agency Open Data https://data.cnra.ca.gov/ U.S. Small Business Administration Open Data https://data.sba.gov/ Ireland’s Open Data Portal https://data.gov.ie/ Dublinked: Open Data for the Dublin Region https://data.smartdublin.ie/ Tusla Data Catalogue https://datacatalog.tusla.ie/ DAFM Data Portal https://opendata.agriculture.gov.ie/ Central Bank of Ireland’s Open Data Portal https://opendata.centralbank.ie/ Data.gov.au https://data.gov.au/ The Central Resource for SEED in NSW https://www.seed.nsw.gov.au/ Data.NSW https://data.nsw.gov.au/ NTG Open Data Portal https://data.nt.gov.au/ Data.SA https://data.sa.gov.au/ Ballarat Open Data https://ballaratopendata.org.au/ DATA VIC https://www.data.vic.gov.au/ Data WA https://www.data.wa.gov.au/ Queensland Government Publications Portal https://www.publications.qld.gov.au/ Transport Open Data https://opendata.transport.nsw.gov.au/ openAFRICA https://open.africa/ Data.gov.hk https://data.gov.hk/en/ Data.gov.uk https://www.data.gov.uk/ UK Data Service https://statistics.ukdataservice.ac.uk/ London Datastore https://data.london.gov.uk/ Open Data NI https://admin.opendatani.gov.uk/ ENTSO-E https://docs.entsoe.eu/ Journal Data Archive https://journaldata.zbw.eu/ Data.openstate.eu https://data.openstate.eu/ OPERANDUM https://data-catalogue. operandum-project.eu/ Dataportal.ponderful.eu https://dataportal.ponderful.eu/ OpenCity https://opencity.in/ New Zealand’s Biological Heritage Data Repository https://data.bioheritage.nz/ Datastore.landcareresearch.co.nz https://datastore.landcareresearch.co. nz/ Open.canada https://search.open.canada.ca/opendata/ Open Govermental Portal in Alberta https://www.alberta.ca/ open-government-program Data.gov.bc.ca https://catalogue.data.gov.bc.ca/ Niagara’s Open Data Catalogue https://niagaraopendata.ca/ Ontario Data Catalogue https://data.ontario.ca/ Données Québec https://www.donneesquebec.ca/ Surrey’s Open Data https://data.surrey.ca/ City of Toronto’s Open Data Portal https://open.toronto.ca/ Data.sustain.ubc.ca https://data.sustain.ubc.ca/ Columbia Basin Water Hub https://data.cbwaterhub.ca
22
So Hasegawa et al.
one of 11 types: Numerical, Categorical, Datetime, Sentence, URL, Embedded Number, List, Ignorable ID, Numbers with Unit, Numbers with Sign, Range of Numbers, or Formatted ID. The Pandas data type is inferred using the built-in pandas.api.types.infer_dtype function. While its output would overlap with the feature types, we include it because its ability to identify mixed types (e.g., columns containing both strings and integers) serves as a key signal for potential data quality issues that would require wrangling. The feature type-specific summary is constructed according to the inferred feature type, as follows: – Numerical: The minimum and maximum values in the column are included. – Categorical: If the column contains 20 or fewer unique categories, all are listed. Otherwise, a random sample of 20 unique categories is provided. – Datetime: The earliest and latest date or time values are included. – URL: No sample values are included. This is a deliberate choice to conserve context length, as full URLs are token-intensive and typically have low semantic value for general data analysis tasks. – All Other Types: For all other feature types, a random sample of 10 unique values is included to provide a representative snapshot of the column’s contents. The example of the serialized text is provided in the following from E-bike Field Study of Data.gov. Text Example by Feature type-specific Table Serialization 1st table Dataset title: Comma Separated Values File Dataset description: None Headers and values: Number of columns: 21 Number of rows: 408363 Feature type, pandas type, ratio of missing values, and feature typespecific information is given for each column as below. date (feature type: Datetime) (pandas type: string) (ratio of missing values: 0%): Start date is 2022-04-27 23:42:29.834000+00:00, and end date is 2022-09-23 18:31:05.502000+00:00. lat (feature type: Numerical) (pandas type: floating) (ratio of missing values: 0%): Value range is [42.447303, 42.461437200000006]. lon (feature type: Numerical) (pandas type: floating) (ratio of missing values: 0%): Value range is [-71.3243906, -71.2562746]. spd (feature type: Numerical) (pandas type: floating) (ratio of missing values: 0%): Value range is [0.0, 23.825000000000003]. blind_turn (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. constrained_tunnel (feature type: Categorical) (pandas type: integer) ( ratio of missing values: 0%): All categories are [0, 1].
Data Analysis in the Wild
23
narrow (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. slow_sign (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. trail_hazards (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. trail_junction (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. vehicle_conflict_point (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. walk_bike_sign (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. eb (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. uphill (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. downhill (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. passing (feature type: Categorical) (pandas type: integer) (ratio of missing values: 0%): All categories are [0, 1]. participantid (feature type: Numerical) (pandas type: integer) (ratio of missing values: 0%): Value range is [1, 37]. age (feature type: Numerical) (pandas type: integer) (ratio of missing values: 0%): Value range is [27, 65]. sex (feature type: Categorical) (pandas type: string) (ratio of missing values: 0%): All categories are [female, male]. bike_type (feature type: Categorical) (pandas type: string) (ratio of missing values: 0%): All categories are [conventional, electric]. ebike_class (feature type: Categorical) (pandas type: floating) (ratio of missing values: 56%): All categories are [1.0, 2.0, 3.0].
Human Verification Figure 7 shows the annotation GUI, built with Streamlit 3 , for revising questions and the Python code used to generate answers. Annotators can refer to the LLM-generated answers and code, as well as the underlying tables, metadata, and external knowledge. Figure 8 presents the GUI for reviewing revised QA pairs, where annotators select one of three options—– Good, Ambiguous, or Wrong Answer —–and may leave comments for the latter two. In total, we obtained 211 datasets, with their original website distribution shown in Table 8. Six of these datasets (Table 9) are used for the Table Insight task. A.3
Distribution of Discarded Candidate Questions
We generated 1,840 candidate questions after the question scoring stage and curated them into 211 high-quality questions, discarding the remaining 1,629 based on the criteria below. 3
https://github.com/streamlit/streamlit
24
So Hasegawa et al.
Fig. 7: Annotation GUI for revising questions and answers
Fig. 8: Annotation GUI for checking the revised QA pairs – Unanimous Agreement: During the answer generation stage, we employed four LLMs to produce answers and measured consensus across models. Questions for which all LLMs produced identical answers were removed because they typically require only shallow reasoning and do not align with the intended complexity of our benchmark. Answer consensus was determined using exact string match for text-based answers and manual verification for visualization-based answers using the GUI tool shown in Figure 7. – Insufficient External Knowledge: Some datasets require external knowledge (e.g. data dictionary) to interpret column meanings or specific values. When such information was missing or insufficient, it became impractical to map column names and values to the generated questions, making accurate answer generation infeasible. Questions of this type were removed. – Ambiguous Question: Questions allowing multiple plausible answers were removed to ensure benchmark clarity. For example, when a table contains
Data Analysis in the Wild
25
both calendar-year and fiscal-year columns, a question asking for “the year” satisfying certain conditions becomes ill-posed unless the question explicitly specifies which type of year should be used. – Uninsightful Question: We excluded questions that failed to yield analytically meaningful or practically useful insights despite being answerable. A common example occurs in geospatial datasets, where questions such as “What is the average latitude and longitude under certain conditions?” often result in a coordinate that lacks geographic or analytical relevance (e.g., a point in the ocean). Such questions were deemed non-insightful and removed. The distribution of the reasons is shown in Table 7.
Table 7: Distribution of Discarding Reasons Reason
Count and Ratio
Unanimous Agreement 114 (0.062) Insufficient External Knowledge 550 (0.299) Ambiguous Question 354 (0.192) Uninsightful Question 611 (0.332)
B
Experimental Setup
B.1
Implementation Details
All open-source models are sourced from the HuggingFace’s transformers library [29], and experiments were conducted using 2 × 48 GB NVIDIA L40S GPUs. Table 10 lists the API names of closed-source models and the HuggingFace model names of open-source models. Insight Agent was configured to generate three initial high-level questions and perform four iterations of its question-answer-insight cycle, resulting in 12 insights. Also, the AgentPoirot baseline was configured with parameters that also yielded 12 insights. Furthermore, the summarizing LLM in Insight Agent is instructed to generate the same number of tokens as the summarized sentences from AgentPoirot for a fair comparison. B.2
Insight Agent
The Insight Agent operates through an iterative cycle: it generates questions, answers them using Answer Agent, and then synthesizes insights from the resulting QA pairs. The insights generated in one step are then used to inform the question generation in the next, creating a continuous exploratory process. This process is governed by a Directed Acyclic Graph (DAG) structure, as illustrated
26
So Hasegawa et al.
Table 8: Distribution of Open Data Websites in DataGovBench Websites
Count
Open.canada 63 Data.gov 35 California Open Data Portal 27 Open Govermental Portal in Alberta 7 Data.gov.uk 6 Analyze Boston 5 Ontario Data Catalogue 4 The Indiana Data Hub 4 Data.SA 4 Surrey’s Open Data 4 Open Data NI 4 The Central Resource for SEED in NSW 2 Pompano Beach Open Data 2 Data.NSW 2 U.S. Small Business Administration Open Data 1 Hawaii Open Data 1 City of Houston Open Data 1 openAFRICA 1 City of Toronto’s Open Data Portal 1 Milwaukee Open Data 1 OpenCity 1 DATA VIC 1 Columbia Basin Water Hub 1
Table 9: Datasets for Table Insight Dataset
Website
Domain
Boston Buildings Inventory Analyze Boston Real Estate Number of Weight Loss Surgeries Performed in Data.gov Healthcare California Hospital Cross-Canada Survey of Radon Concentrations Open.canada Environment in Homes Fixed gear sentinel fisheries program - northern Open.canada Marine Biology Gulf of St. Lawrence Canadian Health Measures Survey (CHMS) Hu- Open.canada Environment man Biomonitoring Data for Environmental Chemicals Results from the 2023 Staffing and Non- Open.canada Demographics Partisanship Survey
Data Analysis in the Wild
27
Table 10: List of LLM model names in the experiments Model Name
API name or Huggingface model name
GPT-4o gpt-4o-2024-08-06 GPT-4o-mini gpt-4o-mini-2024-07-18 GPT-5.1 gpt-5.1-2025-11-13 Claude Sonnet 4.6 claude-sonnet-4-6 Gemini 2.5 Flash gemini-2.5-flash Gemini 2.5 Pro gemini-2.5-pro Devstral-Small mistralai/Devstral-Small-2507 Qwen3-30B Qwen/Qwen3-30B-A3B-Instruct-2507 Qwen3-Coder-30B Qwen/Qwen3-Coder-30B-A3B-Instruct DeepSeek-R1-14B deepseek-ai/DeepSeek-R1-Distill-Qwen-14B Llama3.1-8B meta-llama/Llama-3.1-8B-Instruct TableGPT2-7B tablegpt/TableGPT2-7B
in Figure 9 (b). The graph consists of alternating layers of Question-Answer (QA) nodes and Insight nodes. A new Insight node is generated by synthesizing information from one or more preceding QA nodes, and conversely, a new QA node is generated by drawing upon one or more preceding Insight nodes. Crucially, a new node can be connected to parent nodes from any previous iteration, not just the immediately preceding one. This DAG structure facilitates the aggregation of multiple lines of inquiry, enabling the generation of more diversified and comprehensive insights compared to a simpler tree-based exploration as in Figure 9 (a), where insights from different depths and branches are not connected. The decision of which nodes to aggregate is determined by the reasoning capabilities of LLM; to guide this process, our prompt explicitly instructs the model to consider synthesizing information from multiple parent nodes when possible. B.3
Meta-evaluation of Table Insight
We conducted a meta evaluation to assess the validity of the evaluation metrics for Table Insight. We sampled 50 pairs of generated insights (Insight A and Insight B) for each ground-truth insight and asked three independent annotators to assess which of the two was closer to the ground truth. Annotators selected one of five relative options: A+ (A is definitely better ), A (A is slightly better ), N (A and B are comparable), B (B is slightly better ), or B+ (B is definitely better ). Each option was mapped to a normalized score in [-2, -1, 0, 1, 2]. The final human-judgment score for each pair was obtained by averaging the three annotators’ scores. In parallel, we computed a metric-derived score based on the difference between the insight-level scores of B and A. We then quantified the agreement between human judgments and the metric-derived scores using both Pearson and Spearman correlations, which yielded coefficients of 0.669 and 0.663, respectively. These results indicate a solid alignment between the proposed
28
So Hasegawa et al. QA Node
Insight Node
1st Iter
2nd Iter
3rd Iter loop
(a) AgentPoirot (Tree-based)
(b) Insight Agent (DAG-based)
Fig. 9: Insight generaion process by (a) tree-based process used in AgentPoirot, where each question is selected by LLM from the question candidates and (b) directed acyclic graph-based process used in Insight Agent metric and human judgments, providing empirical support for the reliability of our evaluation methodology for the Table Insight task.
Data Analysis in the Wild GT
Insight Agent
Provinces/Territories
with
Highest Radon
ritories
that
exhibited
the
AgentPoirot
concentrations
Prevalence: The provinces and ter- significantly high- provinces
est percentages of participant homes New
across and
29
vary The average radon concentration in Canadian New Brunswick (179.9 Bq/m3) is
territories,
Brunswick,
with more than double that of British
Yukon,
and Columbia
testing above the radon guideline Manitoba consistently showing the the
(70.98
Northwest
were Manitoba, New Brunswick, highest average concentrations and Bq/m3),
Bq/m3)
Territories
highlighting
and (70.96
significant
Saskatchewan, and the Yukon. For a wider spread of values. Notably, regional disparities in radon levels example, New Brunswick had a New Brunswick has the highest across Canada. (score: 0.37) raw percentage of 24.8% and a proportion of homes exceeding population-weighted
percentage
of the
200
Bq/m³
mitigation
20.6% of homes above 200 Bq/m³, guideline, with over 25% of its while Manitoba had 23.7% raw and homes
above
this
19.4% population-weighted.
these
high-concentration
Within
threshold.
regions, specific health regions and Forward
Sortation
Areas
(FSAs)
exhibit even higher localized averages, of
highlighting
granular
the
regional
importance analysis
for
targeted mitigation efforts. (score: 0.78) Localized
Risk
in
Provinces
with While New Brunswick and Yukon The significant variability in radon
Lower Averages: Even in provinces have the highest average radon con- concentrations between Health Rewhere
the
weighted
overall results
population- centrations, Ontario and Manitoba gions and their provincial averages, indicated
a also show significant radon concerns, as highlighted by the large stan-
lower incidence of homes with particularly regarding high outliers. dard deviation of 40.21 Bq/m3, sugelevated radon levels, there were Ontario has a substantial number of gests that localized geological facstill specific Health Regions with measurements exceeding 500 Bq/m³ tors or housing characteristics within high
radon
levels. For example, (49 instances), and Manitoba has specific Health Regions may play a
in Ontario, where the population- a high proportion of measurements more dominant role in radon levweighted
estimate
was
4.6%
of above 200 Bq/m³ (23.67%), second els than broader provincial trends.
homes exceeding the guideline, 13 of only to New Brunswick. This indi- (score: 0.49) 36 Health Regions (over one-third) cates that even provinces with had more than 10% of homes test lower overall average radon conabove the guideline.
centrations can have localized areas with very high radon levels, necessitating targeted mitigation efforts. (score: 0.64)
Age-Related Increases in Mirex and Polychlorinated Marker PCBs: Mirex concentrations (PCBs), increased
with
age
in
Cycle
biphenyls For Polychlorinated biphenyls, the
particularly
1 polychlorinated
’Marker AM-MA values for the ’Total’ gen-
biphenyls
(sum der group consistently increase with
(2007–2009), with the highest mean of PCB 138, 153, 180)’, show a age, with the 60-79 age group showserum concentration found in the clear
age-related
accumulation, ing AM-MA values as high as 9.62,
60–79 years age group (0.019 ng/g with
significantly
higher
serum) compared to younger groups measured
amounts
in
average significantly higher than the 12-19
older
age age group which has values as low as
(e.g., 0.0014 ng/g serum for 6–11 groups (40-79 years) compared to 0.89. (score: 0.31) years).
Similarly,
Marker
PCBs
180)
the
(PCB
generally
sum 138,
showed
of younger
ones
(3-39
years).
This
153, suggests a persistent presence and
an
in- bioaccumulation of these substances
crease in concentrations with in- over a person’s lifetime. (score: 0.71) creasing
age
with
60–79
the
across
all
years
cycles,
age
group
consistently exhibiting the highest arithmetic
means
(e.g.,
140
ng/g
lipid in Cycle 1).
Table 11: Apple-to-apple comparison among GT insight, insight from Table Insight, and one from AgentPoirot.