arXiv CSopen access

TopBench: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

An-Yang Ji 1 2 Jun-Peng Jiang 1 2 De-Chuan Zhan 1 2 Han-Jia Ye 1 2 Traditional TQA: Clear Instructions and Facts-Retrieval

Abstract

arXiv:2604.28076v1 [cs.CL] 30 Apr 2026

Large Language Models (LLMs) have advanced Table Question Answering, where most queries can be answered by extracting information or simple aggregation. However, a common class of real-world queries is implicitly predictive, requiring the inference of unobserved answers from historical patterns rather than mere retrieval. These queries introduce two challenges: recognizing latent intent and reliable predictive reasoning over massive tables. To assess LLMs in such Tabular questiOn answering with implicit Prediction tasks, we introduce T OP B ENCH, a benchmark consisting of 779 samples across four sub-tasks, ranging from single-point prediction to decision making, treatment effect analysis, and complex filtering, requiring models to generate outputs spanning reasoning text and structured tables. We evaluate diverse models under both text-based and agentic workflows. Experiments reveal that current models often struggle with intent recognition, defaulting to just lookups. Deeper analysis identifies that accurate intent disambiguation serves as the prerequisite for leading these predictive behaviors. Furthermore, elevating the upper bound of prediction precision requires the integration of more sophisticated modeling or reasoning capabilities. Our benchmark is available at https://github.com/ LAMDA-Tabular/TopBench

Q: Search the charge for the 37year-old male smoker, whose BMI is 34.1, without kids in the sortheast.

Implicit Predictive TQA: Intent Recognition and Predictive Reasoning Q: My son just turned 18, living in the southeast for college. He's a male, doesn't smoke, has a BMI of 30.14. Since he's our only child and has no dependents of his own, I'm trying to budget for him. What can we expect his health insurance bill to be?

Historical Data Table ages

sex

BMI kids smoker

42 famale 29

male 34.1

region

charges

southwest 7050.64

yes

northwest 40182.24

...... 29 famale 25.9

southeast 3353.28

southeast

male 30.14

A: The actual insurance bill is estimated to be approximately $1131.51.

Figure 1. Comparison between Traditional TQA and Implicit Predictive TQA. Rather than retrieving or aggregating explicit facts based on clear instructions, implicit predictive TableQA requires the model to infer unobserved values.

Chen et al., 2022) and medical interpretation (Lee et al., 2022; Shi et al., 2024) to daily record management (Dong et al., 2025). Driven by the growing demand to automatically analyze these structures and extract insights, Large Language Models (LLMs) are increasingly deployed to interpret tabular content, lowering the technical barrier for users to interact with complex structured data (Dibia, 2023; Wang et al., 2024; Jiang et al., 2023). Table Question Answering (TQA) serves as a representative task in this field (Chen et al., 2021a; Yin et al., 2020; Jin et al., 2022). Distinct from unstructured text QA (Rajpurkar et al., 2016; Fisch et al., 2019), TQA requires models to derive answers from the structured schema of tables (Pasupat & Liang, 2015). For instance, for a medical insurance table, a query to search the charge for a specific profile involves information retrieval. More complex questions, such as get the average cost for non-smokers in the southwest, require aggregation to count or summarize the results. However, both tasks remain confined to analyzing explicit facts strictly present within the table.

1. Introduction Tabular data represents one of the most prevalent formats in real-world applications, serving as a carrier of rich information (Cafarella et al., 2008) across critical domains, ranging from financial analysis (Chen et al., 2021b; Zhu et al., 2021;

Importantly, real-world queries with tables extend beyond the simple retrieval of records to the forecasting of unknown outcomes based on historical data. As illustrated in Figure 1, when a user asks about an expected bill for a new profile, the answer is not present in the database. To address this, the model must first recognize the latent predictive intent by

School of Artificial Intelligence, Nanjing University, China 2 National Key Laboratory for Novel Software Technology, Nanjing University, China. Correspondence to: Han-Jia Ye <[email protected]>. Preprint. May 1, 2026.

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Figure 2. Overview of T OP B ENCH. T OP B ENCH requires inferring unobserved outcomes from historical data across four tasks: Single-Point Prediction, Decision Making, Treatment Effect Analysis, and Ranking and Filtering.

abstracting the textual description into a structured feature set. Subsequently, it needs to perform predictive reasoning over massive historical rows to infer the missing value. This process imposes a dual challenge: the model must abstract the user query into a tabular problem while correctly identifying its predictive nature, and execute precise predictive modeling based on the provided data context.

files; (2) Decision Making compares trade-offs to select optimal solutions; (3) Treatment Effect Analysis assesses how interventions alter future results; (4) Ranking and Filtering screens candidates for high-priority targets based on historical data. Models are required to generate both natural language reasoning and structured outputs. We evaluate performance under two paradigms: direct text-based reasoning and agentic workflows utilizing iterative ReAct loops (Yao et al., 2023). To ensure rigorous assessment, we employ a hybrid pipeline combining statistical metrics with an LLMas-a-Judge approach (Zhang et al., 2025a), incorporating evaluation designs specifically adapted to each task and strict verification steps to eliminate extraction hallucinations.

Crucially, existing benchmarks fail to address this complexity. They mainly focus on retrieving facts from small tables (Pasupat & Liang, 2015; Chen et al., 2020) or following clear execution instructions (Zhong et al., 2017; Yu et al., 2018). On one hand, there is a shortage of datasets that simulate these real-world implicit predictive scenarios. On the other hand, current works fail to analyze the intermediate reasoning process. They are unable to determine whether model failures result from misinterpreting the task intent or from deficiencies in the predictive modeling process itself.

Our experiments across diverse model types reveal that current LLMs face a strict barrier in bridging the gap between intent recognition and predictive modeling. Text-based approaches are frequently constrained by context limits, often degrading from predictive reasoning into hallucinated retrieval. Similarly, agentic workflows often fail to distinguish implicit predictive intents from historical lookups, resulting in low recall for filtering tasks and prone to execution failures. Through further analysis, we identify two pivotal requirements for improvement. First, we validate that effective intent disambiguation serves as the foundational step to activate predictive modes and prevent models from de-

To bridge this gap, we introduce T OP B ENCH, a benchmark designed to comprehensively evaluate LLMs on Tabular questiOn answering with implicit Prediction goals. T OP B ENCH comprises 779 high-quality samples spanning three critical domains: healthcare, finance, and daily consulting. We organize the benchmark into four tasks representing common real-world predictive scenarios: (1) Single-Point Prediction estimates unknown outcomes for specific pro-

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

faulting to retrieval. Second, elevating performance beyond this baseline requires more than simple tool invocation. It needs superior feature engineering and advanced predictive reasoning mechanisms to accurately model complex data dependencies. While the ultimate goal of TableQA involves handling unstructured requests where data must be retrieved or structured from scratch, T OP B ENCH targets a fundamental intermediate challenge. By pairing intent-rich queries with specific historical tables, we isolate the model’s capacity to identify latent predictive goals and execute rigorous modeling, laying the necessary groundwork for future autonomous data intelligence.

refines model capabilities in structure recognition and fact verification (Zhuang et al., 2024; Zhang et al., 2025b). 2.2. Evolution of TQA Benchmarks Existing benchmarks primarily evaluate the ability to retrieve or aggregate explicit information. Foundational datasets like WTQ (Pasupat & Liang, 2015) and SQA (Iyyer et al., 2017) established standards for discrete operations on HTML tables, while TabFact (Chen et al., 2020) introduced verification tasks to assess consistency between text and data. To address the rigidity of extractive answers, generative tasks such as ToTTo (Parikh et al., 2020) and FeTaQA (Nan et al., 2022) required models to synthesize free-form natural language summaries. With the rise of coding capabilities, the field shifted toward evaluating text-toSQL and code generation performance through benchmarks like WikiSQL (Zhong et al., 2017), Spider (Yu et al., 2018; 2019), and BIRD (Li et al., 2023). Recent agent-centric frameworks such as TableBench (Wu et al., 2025) further simulate complex data analysis workflows. Multimodal tabular benchmarks study complementary problems, including compositional condition QA over table images, tabular-tovisual knowledge transfer, and visual tabular reasoning with privileged structured information (Jiang et al., 2025; 2024; Lu et al., 2025; Jiang et al., 2026b). However, a limitation persists across these datasets as they assess the capability to query or summarize existing historical records with clear instructions, but do not evaluate the ability to abstract implicit unstructured intents into tabular tasks or execute rigorous predictive modeling to infer unobserved outcomes, which are key requirements in real-world scenarios.

In summary, our main contributions are as follows: • We formally define and quantify the task of implicit predictive TQA. This formulation aligns with the complexities of real-world industrial applications, addressing a critical capability gap in existing benchmarks. • We introduce T OP B ENCH, establishing a new standard for tabular intelligence research. It contains a four-tier task supported by a rigorous automated evaluation pipeline. • We conduct a comprehensive evaluation on mainstream LLMs. Through systematic failure analysis, we identify key limitations in implicit modeling and execution robustness, offering empirical guidance for future research.

2. Related Work Tabular intelligence spans representation learning, supervised prediction, and question answering. Recent surveys organize tabular representation learning into specialized, transferable, and general-purpose models, clarifying how deep tabular and tabular foundation models differ in their assumptions and deployment scope (Jiang et al., 2026a).

2.3. Tabular Prediction Benchmarks Tabular prediction has a long independent line of work, from tree-based methods such as XGBoost (Chen & Guestrin, 2016) and CatBoost (Prokhorenkova et al., 2018) to modern deep baselines and tabular foundation models. Recent toolboxes and benchmark analyses emphasize that no single model family dominates across all real datasets, with tree ensembles, nearest-neighbor-inspired deep methods, and foundation models showing data-dependent strengths (Liu et al., 2025a; Ye et al., 2025a;b). TabPFN (Hollmann et al., 2025) and its scalable variants further demonstrate the promise and limits of in-context tabular prediction, especially under highdimensional, many-category, or large-scale settings (Liu & Ye, 2025; Ye et al., 2026). Benchmark suites such as OpenML (Vanschoren et al., 2014), OpenML-CTR23 (Fischer et al., 2023), and TabArena (Erickson et al., 2025) provide standardized comparisons for supervised tabular learning, where the target column, feature columns, training data, and evaluation split are specified before modeling begins. In contrast, T OP B ENCH evaluates an intent-grounded predictive TQA setting where the system must first infer

2.1. Methodologies for Table Question Answering Methods for interpreting tabular data have evolved significantly. Traditional approaches primarily focused on modifying model architectures with specialized embeddings and attention mechanisms to capture structural dependencies (Herzig et al., 2020; Yin et al., 2020). The advent of LLMs shifted the paradigm toward prompt engineering and in-context learning, where well-organized representations improve interpretation capabilities (Singha et al., 2023; Zhao et al., 2026). To enhance reasoning depth, recent studies leverage chain-of-thought techniques to elicit stepby-step deduction processes (Wei et al., 2022; Tai et al., 2024). Furthermore, integrating external tools such as SQL executors and Python interpreters has become a standard approach to handle complex logical operations and robust data analysis (Cheng et al., 2023; Wang et al., 2025; Chai et al., 2024). Complementing these inference-time strategies, instruction tuning on diverse tabular datasets further 3

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering Table 1. Statistics of the T OP B ENCH Dataset. The dataset consists of 779 queries derived from 35 unique source tables across three domains. We report the breakdown of queries (#Q.) per task, balanced across predictive objectives (Regression vs. Classification) and interaction perspectives (User vs. Data Holder).

the predictive task from a natural-language question over a paired historical table.

3. T OP B ENCH Dataset In this section, we present T OP B ENCH, a benchmark designed to evaluate implicit predictive reasoning over tabular data. We first introduce the hierarchical task taxonomy that maps real-world queries into formalized predictive problems. Subsequently, we detail the data sourcing and automated synthesis pipeline used to construct high-quality, intent-rich samples. Finally, we provide a statistical analysis of the dataset across domains and task types. 3.1. Hierarchical Task Taxonomy

Task

#Tab.

#Q.

Single-Point Decision Making Treatment Effect Ranking

35 31 12 27

274 186 105 214

Total

779

Objective

Perspective

Reg.

Class.

User

Holder

116 84 80 104

158 102 25 110

138 93 59 -

136 93 46 214

384

395

290

489

ified state and comparing it against the baseline f (x′old ) to quantify the treatment effect.

We structure T OP B ENCH into four sub-tasks as depicted in Figure 2, representing the most common forms of implicit prediction in real-world scenarios. We formalize implicit predictive TQA as a two-stage inference problem. Given a historical table Thist = {(xi , yi )}N i=1 containing featuretarget pairs and a user query Q, the model must first perform intent abstraction to extract a target feature profile x′ that does not exist in Thist . Subsequently, the model must execute predictive inference by learning a mapping function f : x → y from Thist to estimate the unknown target y ′ = f (x′ ). We instantiate this paradigm into the following four diverse tasks derived from real-world user logs.

Ranking and Filtering. This task addresses the industrial demand for screening massive candidate lists. The model must identify a subset of items S ⊂ {x′1 , . . . , x′M } that satisfy explicit constraints while maximizing an implicit predictive criterion. For example, the query requires identifying the “top 3 females” who are projected to generate the “largest payouts”. This challenges the model to simultaneously apply feature filtering based on gender and perform batch regression to rank candidates by their predicted costs. 3.2. Data Sourcing and Construction

Single-Point Prediction. This serves as the fundamental unit of predictive TQA where the objective is to estimate a single unknown value y ′ given a specific query profile x′ . In Figure 2, a user describes a “51-year-old male, southwest, BMI 39.7” and asks for his expected insurance bill. Here, the model must extract these features to form x′ and utilize the historical data to predict the specific numerical charge, requiring a direct regression or classification inference.

To ensure high-quality predictive reasoning, we curated tables containing genuine correlations from Kaggle across healthcare, finance, and daily consulting domains. These tables vary in scale, ranging from small datasets to industrialscale logs, designed to challenge model adaptability. Building on this foundation, we implemented a multi-stage synthesis pipeline to generate natural, intent-rich queries. As shown in Figure 6 of the Appendix, rather than relying on random selection, we employed a logic-driven sampling strategy to select challenging examples with similar feature values or high noise, testing discriminative precision. Furthermore, we adopted a dual-perspective prompting approach that simulates both non-technical “User” descriptions and historical-data-aware “Data Holder” narratives to prevent mechanical outputs. Finally, all samples underwent a hybrid validation process combining independent LLM auditors with expert human review to guarantee the consistency and solvability of the implicit predictive intent.

Decision Making. This task extends prediction to comparative reasoning, simulating trade-off analysis. The model receives multiple profiles x′1 , x′2 , . . . and must determine the optimal choice based on their predicted outcomes. The query in Figure 2 asks the model to compare a “63-year-old male” against a “52-year-old female” to identify who will likely incur the highest medical cost. The model must implicitly predict the charges y1′ and y2′ for both individuals and perform a comparison operation argmax(y ′ ) to derive the final decision. Treatment Effect Analysis. Moving beyond static prediction, this task evaluates causal reasoning by estimating how a change in state variables ∆x impacts the outcome ∆y. As illustrated in the provided case, a user asks if “moving to the NW” and changing their BMI to “33.3” will lower their charges. The model must effectively perform counterfactual reasoning by predicting the outcome f (x′new ) for the mod-

3.3. Dataset Statistics T OP B ENCH comprises 779 high-quality samples derived from 35 unique historical data tables. Each sample pairs a natural language query with a raw CSV file and a structured ground truth JSON. As illustrated in Figure 3(a), the dataset spans three distinct domains: Daily Consulting, Finance, 4

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Med

11.4%

Short

42.5%

22.9%

50.0%

37.1% 14.3%

Daily Medical

28.6% Med

2.9%

Constrained

Short

Finance Short

8.6%

34.3% 5.7%

Long

Structured Prediction Extraction. We employ a specialized judge LLM to parse free-form responses and extract both final conclusions and intermediate predictive values, such as estimates for competing candidates. This step verifies that the model performed the requisite quantitative analysis rather than relying on semantic plausibility. To accommodate the inherent uncertainty in LLM reasoning, we enforce the parallel extraction of predictive intervals alongside single-point estimates. This ensures that valid conservative bounds are evaluated rather than ignored. Importantly, as shown in Figure 7 of the Appendix, we address the issue of judge hallucination by implementing a rigorous verification protocol. By cross-referencing extracted proofs with the original response via fuzzy matching and Natural Language Inference, we ensure that all extracted values are strictly grounded in the model’s actual output.

Long

8.6% 2.9%

Short

8.4%

7.5%

Short

Unconstrained 50.0%

22.9%

Long

Med

(a) Domains & Overall Tables Length

41.6%

Long

(b) Feature Constraints & Candidates Length

Figure 3. Dataset Distributions. (a) Domain distribution (Inner Ring) and the corresponding historical table lengths (Outer Ring) defined as Short (< 1k), Medium (1k-10k), and Long (> 10k). (b) Distribution of Ranking tasks categorized by filtering constraints (Inner Ring) and the length of candidate lists to be processed (Outer Ring) as Short (< 100) and Long (> 100).

Reasoning Quality Assessment. Distinct from numerical accuracy, the judge evaluates logical coherence on a fivepoint scale. It audits the reasoning chain for fatal flaws such as self-contradiction, circular justification, or the hallucination of constraints not present in the table. This metric acts as a quality gate, ensuring high scores are reserved for clear, evidence-based derivation rather than random guessing.

and Healthcare. To test model robustness against data scale, the historical source tables vary significantly in size, ranging from compact datasets with fewer than 1,000 rows to massive industrial logs exceeding 6 million entries. Beyond domain and scale diversity, T OP B ENCH has a structural balance. As shown in Table 1, the predictive objectives are nearly evenly split between regression (384 queries) and classification (395 queries). Furthermore, the benchmark incorporates dual perspectives, featuring 290 user-centric inquiries and 489 data-holder scenarios, thereby testing model adaptability across different role-playing contexts. Finally, for the Ranking and Filtering task shown in Figure 3(b), we ensure a balanced evaluation by equally splitting queries between those requiring explicit additional feature filtering and those relying solely on implicit prediction.

Composite Metric Calculation. We quantify predictive precision using distinct metrics for continuous and discrete outputs. For regression tasks, measuring simple distance to the ground truth is insufficient given that models often generate valid confidence intervals. We define the regression accuracy score (Accreg ) as a weighted combination of pointwise precision and interval coverage, modulated by a penalty for excessive uncertainty: Accreg = (λ1 · Spoint + λ2 · Siou ) × Pwidth

4. T OP B ENCH Evaluation Methodology

(1)

Here, λ1 and λ2 represent the weights for point estimation and interval overlap, set to 0.6 and 0.4 respectively. The point score Spoint is calculated as max(0, 1 − NMAE) to ensure comparability across datasets. To assess confidence bounds, we calculate the Intersection over Union (Siou ) between the predicted and ground truth intervals. We prevent the gaming of this metric via a Width Penalty Pwidth that decays exponentially if the predicted range exceeds the natural data variance: ( w −α( wpred −τ ) gt e if wpred > τ · wgt Pwidth = (2) 1 otherwise

We design a dual-stream evaluation framework to rigorously assess implicit predictive capabilities based on the output modality. The Predictive Reasoning scenarios, comprising Single-Point Prediction, Decision Making, and Treatment Effect Analysis, require natural language explanations with embedded inferential conclusions. Conversely, the Ranking and Filtering task demands the generation of structured CSV files to demonstrate large-scale data manipulation. Our pipeline therefore differentiates between a logic-aware text evaluator and a deterministic structured file evaluator. 4.1. Natural Language Reasoning Evaluation

where w represents interval width, τ is the tolerance threshold set to 2.0, and α controls the decay rate. For Decision Making and Treatment Analysis tasks, we calculate a distinct Decision Score, defined as a binary exact match indicating whether the model successfully identified the optimal candidate or correctly predicted the directional trend (e.g., “increase” vs. “decrease”) compared to the ground truth.

Implicit prediction requires verifying specific outcome values embedded within complex reasoning chains, unlike traditional Table QA tasks that rely on n-gram overlap metrics such as ROUGE (Lin, 2004). We address this by implementing a comprehensive LLM-as-a-Judge pipeline that jointly assesses predictive accuracy and reasoning quality. 5

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

4.2. Structured Output Evaluation

Inference Paradigms. To comprehensively assess both intent recognition and predictive modeling capabilities, we employ two inference paradigms. In Text-Based Reasoning, the model receives the query alongside a serialized table string without external tools. Since large datasets are truncated to fit context limits (Liu et al., 2024), this setup requires the model to perform arithmetic and logical inference using only internal parameters. We keep the table header and randomly retain rows when the serialized table exceeds the context budget; Appendix B.3 compares this policy with head and stratified retention. Conversely, the Agentic Framework employs a ReAct loop (Yao et al., 2023) where the model iteratively executes Python code within a Docker sandbox. The model analyzes execution outputs and error logs to dynamically refine its strategy. We apply this framework to Ranking and Filtering tasks to ensure the accurate processing of full-scale datasets.

The Ranking and Filtering scenario simulates industrial batch processing, where the objective is to screen massive datasets and identify candidates meeting complex implicit criteria. This scenario requires the model to act as a data processor, generating a structured file (CSV) containing the filtered or ranked records. Therefore, our evaluation methodology shifts from natural language parsing to a deterministic file-based assessment, focusing on the precision of the retrieval set and the accuracy of the batch predictions. Metrics for List Filtering. For tasks where the goal is to select a subset of candidates (e.g., “Identify all patients with malignant tumors.”), we calculate the F1 Score by comparing the set of rows returned by the model against the ground truth subset. This measures the model’s ability to strictly adhere to the implicit filtering logic without hallucinating non-existent records or omitting valid candidates.

5.2. Main Results

Metrics for Complex Ranking. For tasks involving regression or prioritized sorting (e.g., “Rank the top 10 most profitable companies.”), we assess performance across three dimensions. We use Set Recall to verify if the model successfully retrieved the correct top-k items from the full dataset. To evaluate the quality of the sorting, we compute the Normalized Discounted Cumulative Gain (NDCG) (Järvelin & Kekäläinen, 2002), which rewards models for placing highvalue items at the very top of the list. Finally, to assess the precision of the predicted numerical values attached to each record, we compute the batch NMAE over the aligned pairs, providing a stable measure of how accurately the model predicted the specific attributes of the filtered candidates.

Table 2 summarizes the performance of various Large Language Models. To ensure alignment, Logic Scores are normalized to the range [0, 1]. Are LLMs Effective Predictive Reasoners? Current LLMs demonstrate significant fragility in implicit predictive tasks, with most scores falling below 0.60—a sharp contrast to their proficiency in fact retrieval. Even the leading model, Gemini 3 Flash, achieves only 0.65 accuracy in the fundamental Single Point scenario. This deficiency is amplified in Decision Making and Treatment Effect Analysis, where performance often approximates random guessing. The corresponding intermediate predictions are even worse. Models frequently hallucinate a final conclusion without generating the supporting quantitative estimates, resulting in zero accuracy scores and reflecting a lack of derivation.

5. Experiments 5.1. Evaluation Details

Does the Agentic Framework Help? Comparing the frameworks reveals distinct behavioral differences. GPT-5.2 demonstrates the benefit of code generation in Treatment Effect Analysis, raising its Trend Score from 0.51 to 0.65 by offloading complex arithmetic. Conversely, Qwen3-Instruct suffers a significant drop in Single-Point Prediction (0.57 to 0.43) within the agentic mode. This degradation stems from the model attempting to retrieve data via code rather than performing the necessary predictive modeling. This highlights a critical alignment failure where the model conflates implicit predictive intent with simple information retrieval, a phenomenon we examine further in Section 6.

Selected LLMs. We assess 9 representative models organized into three categories: General LLMs, ReasoningEnhanced Models, and Tabular Specialists. The General LLMs encompasses both latest proprietary frontier models (GPT-5.2 (OpenAI, 2025), Claude Sonnet 4.5 (Anthropic, 2025), and Gemini 3 Flash (Google DeepMind, 2026)) and standard open-weights instruction-tuned models (DeepSeek-V3.2-Instruct (DeepSeek-AI et al., 2025) and Qwen3-Instruct (Yang et al., 2025)). To analyze the impact of test-time compute (Snell et al., 2025), we evaluate Reasoning-Enhanced Models, including DeepSeek-V3.2Thinking and Qwen3-Thinking. Note that for the Qwen3 family, we utilize the specific Qwen3-235BA22B-2507. Finally, we include Tabular Specialists such as TableLLM8B (Llama3.1 based) and TableLLM-13B (CodeLlama based) (Zhang et al., 2025b) to benchmark the performance gap between generalist capabilities and domain-specific adaptations.

Reasoning Models vs. General Models. Reasoningenhanced models, notably Qwen3-Thinking, exhibit severe instability in text-based settings, often underperforming standard baselines. In over 50% of cases, the model enters repetitive loops that exhaust the context window. This suggests that despite test-time compute scaling, it struggles to main6

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering Table 2. Main Evaluation Results on T OP B ENCH. We report performance across four distinct tasks: Single Point Prediction, Decision Making, Treatment Effect Analysis, and Ranking and Filtering. Metrics include Prediction Accuracy (Acc.), Logic Score (Logic), Decision/Trend Accuracy (Dec./Trend), Recall (Rec.), NDCG, NMAE, and F1 Score. The first part displays results for text-based reasoning, where the Ranking and Filtering task is omitted as it requires structured file generation. The second part presents results for the agentic code-execution framework. The best result in each column is highlighted in bold.

Ranking and Filtering Model

Single Point

Decision Making

Treatment Effect

Acc.

Logic

Acc.

Logic

Dec.

Acc.

Logic

Trend

0.55 0.65 0.59 0.57

0.72 0.75 0.75 0.76

0.22 0.42 0.32 0.36

0.76 0.78 0.76 0.75

0.57 0.62 0.57 0.53

0.18 0.43 0.35 0.28

0.76 0.78 0.77 0.74

0.57 0.44

0.75 0.52

0.27 0.23

0.75 0.53

0.61 0.39

0.35 0.33

0.38 0.27

0.52 0.28

0.17 0.14

0.62 0.23

0.46 0.39

0.71 0.73 0.72 0.76 0.50

0.40 0.50 0.46 0.42 0.25

0.72 0.75 0.74 0.74 0.56

0.61 0.57

0.75 0.67

0.40 0.27

0.10 0.12

0.04 0.03

0.02 0.01

Regression

Class.

Rec.

NDCG

NMAE↓

0.51 0.62 0.59 0.55

0.76 0.52

0.59 0.46

0.17 0.10

0.46 0.19

0.52 0.35

0.55 0.56 0.65 0.59 0.45

0.46 0.52 0.46 0.38 0.30

0.79 0.76 0.77 0.77 0.67

0.65 0.63 0.65 0.57 0.53

0.46 0.52 0.53 0.54 0.31

0.41 0.50 0.50 0.50 0.28

0.41 0.31 0.30 0.26 0.61

0.38 0.55 0.58 0.48 0.28

0.78 0.70

0.58 0.57

0.43 0.42

0.78 0.75

0.65 0.53

0.50 0.43

0.46 0.41

0.35 0.46

0.38 0.42

0.02 0.02

0.12 0.17

0.06 0.05

0.07 0.17

0.00 0.00

1.00 1.00

0.00 0.00

Text-Based Reasoning General LLMs GPT-5.2 Gemini 3 Flash DeepSeek-V3.2-Instruct Qwen3-Instruct

Reasoning-Enhanced Models DeepSeek-V3.2-Thinking Qwen3-Thinking Tabular Specialists TableLLMLlama3.1-8B TableLLMCodeLlama-13B

Agentic Code-Execution Framework General LLMs GPT-5.2 Claude Sonnet 4.5 Gemini 3 Flash DeepSeek-V3.2-Instruct Qwen3-Instruct

0.60 0.64 0.66 0.58 0.43

Reasoning-Enhanced Models DeepSeek-V3.2-Thinking Qwen3-Thinking Tabular Specialists TableLLMLlama3.1-8B TableLLMCodeLlama-13B

tain logical coherence over long tabular inputs, frequently degenerating into circular logic.

for simple calculation, lacking the broader world knowledge necessary to interpret implicit predictive intents. In filtering regimes, these models exhibit catastrophic failure with nearzero recall due to invalid code execution. This confirms that traditional “Table QA” training objectives are insufficient and that robust code generation is a strict prerequisite for industrial-grade tabular workflows.

Challenges in High-Throughput Filtering. For the Ranking and Filtering task, which needs large-scale structured file generation, precise output remains a substantial challenge. While Gemini 3 Flash leads in classification with a F1 score of 0.58, regression precision is generally poor, with average NMAE scores ranging from 0.30 to 0.40. DeepSeek-V3.2Instruct achieves the lowest error (0.26), demonstrating relative high performance during batch processing.

6. Further Analysis In this section, we investigate the underlying causes of performance limitations. Specifically, we examine whether current models can effectively translate implicit unstructured queries into structured tabular prediction tasks and how different modeling strategies impact final precision.

Limitations of Tabular Specialists. Domain-specific models significantly underperform generalist LLMs across all tasks. This shortfall likely stems from their specialized training on explicit information extraction and code generation

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

12% 46%

10% 16%

40%

35%

78%

80%

31%

Table 3. Semantic Information Ablation. Orig. vs. +Info. Single, Decision, and Treat. use accuracy-style metrics; Rank(Reg.) uses NMAE↓, and Rank(Cls.) uses F1.

5% 15%

14%

Single Model

DeepSeekV3.2 Instruct

DeepSeekV3.2 Thinking

7% 49%

6% 15%

25% 45%

Qwen3 Thinking

Qwen3 Instruct

Claude Sonnet-4.5

Gemini3 Flash

GPT-5.2 Gemini 3 Flash DeepSeek-V3.2 Qwen3-Instruct

Random Forest Logistic Regression Linear Regression Others Without Modeling

24%

GPT-5.2 Gemini 3 Flash DeepSeek-V3.2 Qwen3-Instruct

Figure 4. Distribution of Predictive Tool Usage. The chart illustrates the frequency with which different LLMs invoke machine learning libraries versus simple data manipulation methods. It also highlights the most frequently selected algorithm for each model.

Average Performance Score

0.70

10% 6% 0.58

0.56

0.55 0.50

0.67

0.65

0.57

16%

0.55

0.49

0.45 0.40

DeepSeek DeepSeek Qwen3 Qwen3 Claude Gemini3 Instruct Thinking Instruct Thinking Sonnet4.5 Flash

With Modeling

0.59 0.66 0.59 0.59

0.57 0.62 0.57 0.53

0.54 0.62 0.57 0.55

0.51 0.62 0.59 0.55

0.52 0.69 0.57 0.62

0.60 0.66 0.58 0.43

0.64 0.69 0.60 0.56

0.55 0.65 0.59 0.45

0.53 0.61 0.59 0.46

0.65 0.65 0.57 0.53

0.68 0.64 0.68 0.50

0.41 0.30 0.26 0.61

0.35 0.31 0.24 0.55

0.38 0.58 0.48 0.28

0.48 0.57 0.55 0.33

Impact of Code Execution on Predictive Precision. Figure 5 contrasts average performance metrics between instances where models explicitly trained a predictor versus those relying solely on internal reasoning or simple calculation. The data confirms a positive correlation between code-based modeling and task success across all categories. Notably, DeepSeek, which has a higher modeling frequency, significantly outperforms Qwen3. While Gemini 3 Flash achieves a relatively high baseline score of 0.65 through strong intrinsic reasoning, explicit code invocation still yields further performance gains. This indicates that even for advanced models capable of approximate context processing, rigorous predictive modeling remains essential for precision-critical tasks.

0.64

0.53 0.50

0.55 0.65 0.59 0.57

els to favor algorithms that are robust and require minimal hyperparameter tuning.

12%

0.61 0.57

0.66 0.64

0.65 0.60

15%

Rank(Reg.) Rank(Cls.)

Agentic Code-Execution Framework

GPT-5.2

0.75

Treat.

Text-Based Reasoning

Prediction Models

51% 74%

Decision

Orig. +Info Orig. +Info Orig. +Info Orig. +Info Orig. +Info

GPT-5.2

Without Modeling

Figure 5. Performance Impact of Predictive Modeling. We compare average scores across Single Point (Accuracy), Decision (Decision Score), and Treatment Effect (Trend Score) tasks. “With Modeling” means LLMs use the predictive model. “Without Modeling” means they default to data retrieval or simple aggregation.

6.2. Ablation on Intent Recognition and Semantic Understanding To test whether failures come from missing the predictive intent, we inject semantic hints into the instruction: target column, task type, and feature descriptions. Table 3 compares the original models with these info-enhanced variants.

More specific results can be found in Appendix D. 6.1. Analysis of Agentic Predictive Modeling Behaviors

Correction of Intent Misalignment. The clearest gains in Table 3 occur when the original model likely chooses the wrong task framing. In the agentic setting, Qwen3Instruct improves on Single Point prediction from 0.43 to 0.56, DeepSeek-V3.2 improves on Treatment Effect Analysis from 0.57 to 0.68, and GPT-5.2 improves on Single Point prediction from 0.60 to 0.64. Text-based gains are smaller but still visible. These results indicate that a major failure mode is not coding ability, but failure to recognize prediction as the required operation.

To understand the mechanisms driving performance in the agentic framework, we analyze code generation patterns to determine if models utilize the sandbox for rigorous predictive modeling or merely default to data retrieval. Tendency for Predictive Modeling. Figure 4 illustrates a distinct divergence in tool-use strategy. DeepSeek actively imports libraries such as scikit-learn to train predictive models in over 60% of scenarios. In contrast, Qwen3 defaults to pandas-based filtering or heuristic arithmetic, often failing to recognize the implicit predictive intent. Regarding algorithm selection, Random Forest (Breiman, 2001) and Logistic Regression (Hosmer et al., 2013) are dominant. This preference likely stems from the distribution of data science code in the pre-training corpus, leading mod-

Benefits for Complex Batch Processing. For Ranking and Filtering, evaluated only in the agentic setting, semantic information helps feature selection in noisy candidate pools. GPT-5.2 improves on both metrics 8

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering Table 4. Predict-Only Baseline. The ensemble receives the gold structured target/profile and is compared with the strongest Gemini agentic E2E setting.

(NMAE 0.41→0.35, F1 0.38→0.48), as do DeepSeek-V3.2 (0.26→0.24, 0.48→0.55) and Qwen3-Instruct (0.61→0.55, 0.28→0.33). Gemini 3 Flash changes little, suggesting that stronger models may already infer much of the schema semantics from the raw table.

Method Gemini 3 Flash (agentic E2E) Predict-only ensemble

Precision Barrier in Fine-Grained Decisions. The Decision Making columns show a different pattern. Semantic information brings little benefit and sometimes hurts performance. Gemini 3 Flash drops from 0.65 to 0.61 in the agentic setting, and GPT-5.2 drops from 0.55 to 0.53. Candidate pairs are intentionally similar, so recognizing the intent is necessary but not sufficient; the remaining bottleneck is prediction precision, feature engineering, and the tabular model. This aligns with recent tabular prediction studies showing that stronger specialized predictors and TabPFNstyle models can improve prediction quality, but remain sensitive to data scale, dimensionality, and preprocessing choices (Liu et al., 2025a; Liu & Ye, 2025; Ye et al., 2026).

Single Acc. Decision Dec. Treat. Trend 0.66 0.76

0.65 0.72

0.65 0.69

toward such adaptive comparison, but our setting additionally requires recovering the structured task from an implicit user request (Liu et al., 2025b). Thus future agents need stronger predictive capability across the full workflow: targeted data processing, appropriate model selection, contextual comparison, and self-correction.

7. Conclusion In this work, we introduced T OP B ENCH, a benchmark that shifts the paradigm of TQA from explicit fact retrieval to implicit predictive reasoning. Our evaluation exposes a critical capability gap: current models often struggle to distinguish between retrieval and prediction intents, frequently defaulting to simple lookups rather than constructing rigorous models. However, our study operates under a specific constraint where the relevant historical table is explicitly paired with each query. This represents an intermediate step toward realistic data intelligence. In open-ended applications, user requests are often unstructured, requiring systems to not only reason over provided data but also autonomously retrieve relevant historical records or structure undefined information from scratch. We hope T OP B ENCH serves as a foundation for future research, encouraging the development of models that can navigate the full pipeline from data discovery to predictive inference.

6.3. Ablation on Predictive Modeling Capacity Table 4 examines the prediction upper bound after the target and feature profile are provided. The predict-only ensemble selects among HistGradientBoosting (Pedregosa et al., 2011), XGBoost (Chen & Guestrin, 2016), LightGBM (Ke et al., 2017), CatBoost (Prokhorenkova et al., 2018), ExtraTrees (Geurts et al., 2006), and TabPFN (Hollmann et al., 2025), following the observation that robust tabular performance often requires adaptive model selection over heterogeneous datasets (Liu et al., 2025a; Ye et al., 2025a). It is not an end-to-end T OP B ENCH system, but a diagnostic reference for the prediction module that current agentic workflows try to construct through code. Targeted Data Processing and Model Selection. Compared with the strongest Gemini agentic end-to-end setting, predict-only improves Single Point from 0.66 to 0.76, Decision Making from 0.65 to 0.72, and Treatment Effect from 0.65 to 0.69. These gains suggest that stronger prediction requires a task-specific tabular pipeline, not merely invoking a generic estimator. The agent must select useful features, encode categorical variables, handle missing values and scale, and choose model families suited to the target type and table size. Small errors in these steps can directly change a numerical estimate or candidate preference.

References Anthropic. System card: Claude Sonnet 4.5. System card PDF, 2025. Breiman, L. Random forests. Machine Learning, 45:5–32, 2001. Cafarella, M. J., Halevy, A., Wang, D. Z., Wu, E., and Zhang, Y. WebTables: exploring the power of tables on the web. Proceedings of the VLDB Endowment, 1: 538–549, 2008.

Contextual Comparison and Self-Correction. The predict-only baseline is still far from perfect, showing that T OP B ENCH also requires reasoning over difficult prediction contexts. Noisy distributions, long-tail targets, class imbalance, and similar candidates require models to compare scenario-level predictions, check whether effect directions or candidate orderings are plausible, and revise preprocessing or modeling choices when outputs conflict with table evidence. LLM-assisted tabular ensembling is an early step

Chai, L., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., Wang, Z., Wang, B., Wu, X., Wang, B., Li, T., Yang, L., Duan, S., and Li, Z. McEval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436, 2024. Chen, T. and Guestrin, C. XGBoost: A scalable tree boosting system. In KDD, pp. 785–794, 2016. 9

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., and Wang, W. Y. TabFact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164, 2020.

Liu, Y., Zhou, Y., Wu, Z. F., Ren, Z. Z., Zhao, Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, Z., Gou, Z., Ma, Z., Yan, Z., Shao, Z., Huang, Z., Wu, Z., Li, Z., Zhang, Z., Xu, Z., Wang, Z., Gu, Z., Zhu, Z., Li, Z., Zhang, Z., Xie, Z., Gao, Z., Pan, Z., Yao, Z., Feng, B., Li, H., Cai, J. L., Ni, J., Xu, L., Li, M., Tian, N., Chen, R. J., Jin, R. L., Li, S. S., Zhou, S., Sun, T., Li, X. Q., Jin, X., Shen, X., Chen, X., Song, X., Zhou, X., Zhu, Y. X., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Huang, Z., Xu, Z., Zhang, Z., Ji, D., Liang, J., Guo, J., Chen, J., Xia, L., Wang, M., Li, M., Zhang, P., Chen, R., Sun, S., Wu, S., Ye, S., Wang, T., Xiao, W. L., An, W., Wang, X., Sun, X., Wang, X., Tang, Y., Zha, Y., Zhang, Z., Ju, Z., Zhang, Z., and Qu, Z. DeepSeek-V3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025.

Chen, W., Chang, M.-W., Schlinger, E., Wang, W., and Cohen, W. W. Open question answering over tables and text. arXiv preprint arXiv:2010.10439, 2021a. Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Langdon, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B., and Wang, W. Y. FinQA: A dataset of numerical reasoning over financial data. In EMNLP, pp. 3697–3711, 2021b. Chen, Z., Li, S., Smiley, C., Ma, Z., Shah, S., and Wang, W. Y. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering. arXiv preprint arXiv:2210.03849, 2022.

Dibia, V. LIDA: A tool for automatic generation of grammaragnostic visualizations and infographics using large language models. arXiv preprint arXiv:2303.02927, 2023.

Cheng, Z., Xie, T., Shi, P., Li, C., Nadkarni, R., Hu, Y., Xiong, C., Radev, D., Ostendorf, M., Zettlemoyer, L., Smith, N. A., and Yu, T. Binding language models in symbolic languages. arXiv preprint arXiv:2210.02875, 2023.

Dong, H., Zhao, J., Tian, Y., Xiong, J., Xia, S., Zhou, M., Lin, Y., Cambronero, J., He, Y., Han, S., and Zhang, D. SpreadsheetLLM: Encoding spreadsheets for large language models. arXiv preprint arXiv:2407.09025, 2025.

DeepSeek-AI, Liu, A., Mei, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., Lu, C., Zhao, C., Deng, C., Xu, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Li, E., Zhou, F., Lin, F., Dai, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Li, H., Liang, H., Wei, H., Zhang, H., Luo, H., Ji, H., Ding, H., Tang, H., Cao, H., Gao, H., Qu, H., Zeng, H., Huang, J., Li, J., Xu, J., Hu, J., Chen, J., Xiang, J., Yuan, J., Cheng, J., Zhu, J., Ran, J., Jiang, J., Qiu, J., Li, J., Song, J., Dong, K., Gao, K., Guan, K., Huang, K., Zhou, K., Huang, K., Yu, K., Wang, L., Zhang, L., Wang, L., Zhao, L., Yin, L., Guo, L., Luo, L., Ma, L., Wang, L., Zhang, L., Di, M. S., Xu, M. Y., Zhang, M., Zhang, M., Tang, M., Zhou, M., Huang, P., Cong, P., Wang, P., Wang, Q., Zhu, Q., Li, Q., Chen, Q., Du, Q., Xu, R., Ge, R., Zhang, R., Pan, R., Wang, R., Yin, R., Xu, R., Shen, R., Zhang, R., Liu, S. H., Lu, S., Zhou, S., Chen, S., Cai, S., Chen, S., Hu, S., Liu, S., Hu, S., Ma, S., Wang, S., Yu, S., Zhou, S., Pan, S., Zhou, S., Ni, T., Yun, T., Pei, T., Ye, T., Yue, T., Zeng, W., Liu, W., Liang, W., Pang, W., Luo, W., Gao, W., Zhang, W., Gao, X., Wang, X., Bi, X., Liu, X., Wang, X., Chen, X., Zhang, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yu, X., Li, X., Yang, X., Li, X., Chen, X., Su, X., Pan, X., Lin, X., Fu, X., Wang, Y. Q., Zhang, Y., Xu, Y., Ma, Y., Li, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Qian, Y., Yu, Y., Zhang, Y., Ding, Y., Shi, Y., Xiong, Y., He, Y., Zhou, Y., Zhong, Y., Piao, Y., Wang, Y., Chen, Y., Tan, Y., Wei, Y., Ma, Y., Liu, Y., Yang, Y., Guo, Y., Wu, Y., Wu, Y., Cheng, Y., Ou, Y., Xu, Y., Wang, Y., Gong, Y., Wu, Y., Zou, Y., Li, Y., Xiong, Y., Luo, Y., You, Y.,

Erickson, N., Purucker, L., Tschalzev, A., Holzmüller, D., Desai, P. M., Salinas, D., and Hutter, F. TabArena: A living benchmark for machine learning on tabular data. arXiv preprint arXiv:2506.16791, 2025. Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., and Chen, D. MRQA 2019 shared task: Evaluating generalization in reading comprehension. arXiv preprint arXiv:1910.09753, 2019. Fischer, S. F., Feurer, M., and Bischl, B. OpenML-CTR23 – a curated tabular regression benchmarking suite. In AutoML Workshop, 2023. Geurts, P., Ernst, D., and Wehenkel, L. Extremely randomized trees. Machine Learning, 63:3–42, 2006. Google DeepMind. Gemini 3 Flash Model Card. https://storage.googleapis. com/deepmind-media/Model-Cards/ Gemini-3-Flash-Model-Card.pdf, 2026. Herzig, J., Nowak, P. K., Müller, T., Piccinno, F., and Eisenschlos, J. TaPas: Weakly supervised table parsing via pre-training. In ACL, pp. 4320–4333, 2020. Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with a tabular foundation model. Nature, 637:319–326, 2025. 10

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Hosmer, D. W., Lemeshow, S., and Sturdivant, R. X. Applied Logistic Regression. Wiley, 2013.

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024.

Iyyer, M., Yih, W.-t., and Chang, M.-W. Search-based neural structured learning for sequential question answering. In ACL, pp. 1821–1831, 2017.

Liu, S. and Ye, H.-J. TabPFN unleashed: A scalable and effective solution to tabular classification problems. In ICML, pp. 40043–40068, 2025.

Järvelin, K. and Kekäläinen, J. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems, 20:422–446, 2002.

Liu, S.-Y., Cai, H.-R., Zhou, Q.-L., Yin, H.-H., Zhou, T., Jiang, J.-P., and Ye, H.-J. Talent: A tabular analytics and learning toolbox. Journal of Machine Learning Research, 26:1–16, 2025a.

Jiang, J., Zhou, K., Dong, Z., Ye, K., Zhao, W. X., and Wen, J.-R. StructGPT: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645, 2023.

Liu, S.-Y., Zhou, Q., and Ye, H.-J. Make still further progress: Chain of thoughts for tabular data leaderboard. In 1st ICML Workshop on Foundation Models for Structured Data, 2025b.

Jiang, J.-P., Ye, H.-J., Wang, L., Yang, Y., Jiang, Y., and Zhan, D.-C. Tabular insights, visual impacts: Transferring expertise from tables to images. In ICML, pp. 21988–22009, 2024.

Lu, S., Li, Y., Xia, Y., Hu, Y., Zhao, S., Ma, Y., Wei, Z., Li, Y., Duan, L., Zhao, J., Han, Y., Li, H., Chen, W., Tang, J., Hou, C., Du, Z., Zhou, T., Zhang, W., Ding, H., Li, J., Li, W., Hu, G., Gu, Y., Yang, S., Wang, J., Sun, H., Wang, Y., Sun, H., Huang, J., He, Y., Shi, S., Zhang, W., Zheng, G., Jiang, J., Gao, S., Wu, Y.-F., Chen, S., Chen, Y., Chen, Q.-G., Xu, Z., Luo, W., and Zhang, K. Ovis2.5 technical report. arXiv:2508.11737, 2025.

Jiang, J.-P., Zhou, T., Zhan, D.-C., and Ye, H.-J. Compositional condition question answering in tabular understanding. In ICML, pp. 27831–27850, 2025. Jiang, J.-P., Liu, S.-Y., Cai, H.-R., Zhou, Q.-L., and Ye, H.-J. Representation learning for tabular data: A comprehensive survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–20, 2026a.

Nan, L., Hsieh, C., Mao, Z., Lin, X. V., Verma, N., Zhang, R., Kryściński, W., Schoelkopf, H., Kong, R., Tang, X., Mutuma, M., Rosand, B., Trindade, I., Bandaru, R., Cunningham, J., Xiong, C., Radev, D., and Radev, D. Fetaqa: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35–49, 2022.

Jiang, J.-P., Xia, Y., Sun, H.-L., Lu, S., Chen, Q.-G., Luo, W., Zhang, K., Zhan, D.-C., and Ye, H.-J. Multimodal tabular reasoning with privileged structured information. In NeurIPS, 2026b. Jin, N., Siebert, J., Li, D., and Chen, Q. A Survey on Table Question Answering: Recent Advances, pp. 174– 186. Springer Nature Singapore, 2022. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. In NeurIPS, 2017.

OpenAI. Update to GPT-5 system card: GPT5.2. https://cdn.openai.com/pdf/ 3a4153c8-c748-4b71-8e31-aecbde944f8d/ oai_5_2_system-card.pdf, 2025.

Lee, G., Hwang, H., Bae, S., Kwon, Y., Shin, W., Yang, S., Seo, M., Kim, J.-Y., and Choi, E. EHRSQL: A practical text-to-SQL benchmark for electronic health records. In NeurIPS, pp. 15589–15601, 2022.

Parikh, A., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., and Das, D. ToTTo: A controlled tableto-text generation dataset. In EMNLP, pp. 1173–1186, 2020.

Li, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Geng, R., Huo, N., Zhou, X., Chenhao, M., Li, G., Chang, K., Huang, F., Cheng, R., and Li, Y. Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs. In NeurIPS, pp. 42330–42357, 2023.

Pasupat, P. and Liang, P. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305, 2015. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Édouard Duchesnay. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12:2825–2830, 2011.

Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In ACL Workshop: Text Summarization Branches Out, pp. 74–81, 2004. 11

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. CatBoost: unbiased boosting with categorical features. In NeurIPS, 2018.

Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.

Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. Shi, W., Xu, R., Zhuang, Y., Yu, Y., Zhang, J., Wu, H., Zhu, Y., Ho, J. C., Yang, C., and Wang, M. D. EHRAgent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records. In EMNLP, pp. 22315–22339, 2024.

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023.

Singha, A., Cambronero, J., Gulwani, S., Le, V., and Parnin, C. Tabular representation, noisy operators, and impacts on table structure understanding tasks in LLMs. arXiv preprint arXiv:2310.10358, 2023.

Ye, H.-J., Liu, S.-Y., Cai, H.-R., Zhou, Q.-L., and Zhan, D.-C. A closer look at deep learning methods on tabular datasets. arXiv preprint arXiv:2407.00956, 2025a.

Snell, C. V., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In ICLR, 2025.

Ye, H.-J., Yin, H.-H., Zhan, D.-C., and Chao, W.-L. Revisiting nearest neighbor for tabular data: A deep tabular baseline two decades later. In ICLR, 2025b.

Tai, R. H., Bentley, L. R., Xia, X., Sitt, J. M., Fankhauser, S. C., Chicas-Mosier, A. M., and Monteith, B. G. An examination of the use of large language models to aid analysis of textual data. International Journal of Qualitative Methods, 23, 2024.

Ye, H.-J., Liu, S.-Y., and Chao, W.-L. A closer look at tabPFN v2: Understanding its strengths and extending its capabilities. In NeurIPS, 2026. Yin, P., Neubig, G., Yih, W.-t., and Riedel, S. TaBERT: Pretraining for joint understanding of textual and tabular data. In ACL, pp. 8413–8426, 2020.

Vanschoren, J., van Rijn, J. N., Bischl, B., and Torgo, L. OpenML: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15:49–60, 2014.

Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., and Radev, D. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and textto-SQL task. In EMNLP, pp. 3911–3921, 2018.

Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.-W., Yin, D., Sun, X., and Li, Z. MAC-SQL: A multi-agent collaborative framework for text-to-SQL. In COLING, pp. 540–557, 2025. Wang, Z., Zhang, H., Li, C.-L., Eisenschlos, J. M., Perot, V., Wang, Z., Miculicich, L., Fujii, Y., Shang, J., Lee, C.-Y., and Pfister, T. Chain-of-table: Evolving tables in the reasoning chain for table understanding. arXiv preprint arXiv:2401.04398, 2024.

Yu, T., Zhang, R., Er, H., Li, S., Xue, E., Pang, B., Lin, X. V., Tan, Y. C., Shi, T., Li, Z., Jiang, Y., Yasunaga, M., Shim, S., Chen, T., Fabbri, A., Li, Z., Chen, L., Zhang, Y., Dixit, S., Zhang, V., Xiong, C., Socher, R., Lasecki, W., and Radev, D. CoSQL: A conversational text-toSQL challenge towards cross-domain natural language interfaces to databases. In EMNLP-IJCNLP, pp. 1962– 1979, 2019.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, pp. 24824–24837, 2022.

Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A., and Agarwal, R. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 2025a.

Wu, X., Yang, J., Chai, L., Zhang, G., Liu, J., Du, X., Liang, D., Shu, D., Cheng, X., Sun, T., Li, T., Li, Z., and Niu, G. TableBench: A comprehensive and complex benchmark for table question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 39:25497–25506, 2025.

Zhang, X., Luo, S., Zhang, B., Ma, Z., Zhang, J., Li, Y., Li, G., Yao, Z., Xu, K., Zhou, J., Zhang-Li, D., Yu, J., Zhao, S., Li, J., and Tang, J. TableLLM: Enabling tabular data manipulation by LLMs in real office usage scenarios. In Findings of ACL, pp. 10315–10344, 2025b.

Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., 12

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. A survey of large language models. arXiv preprint arXiv:2303.18223, 2026. Zhong, V., Xiong, C., and Socher, R. Seq2SQL: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017. Zhu, F., Lei, W., Huang, Y., Wang, C., Zhang, S., Lv, J., Feng, F., and Chua, T.-S. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv preprint arXiv:2105.07624, 2021. Zhuang, A., Zhang, G., Zheng, T., Du, X., Wang, J., Ren, W., Huang, S. W., Fu, J., Yue, X., and Chen, W. StructLM: Towards building generalist models for structured knowledge grounding. arXiv preprint arXiv:2402.16671, 2024.

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

A. Overview of the Appendix This Appendix provides supporting details for T OP B ENCH. It is organized as follows: • Appendix B gives dataset details, including source-table statistics, table-shape analysis, truncation analysis, the data synthesis pipeline, and the human authenticity audit. • Appendix C gives evaluation details, including hallucination verification, judge configuration, robustness checks, metric sensitivity, prompt templates, and task metrics. • Appendix D reports additional experimental results for ranking metrics, schema information, output integrity, and predictive tool usage. • Appendix E provides qualitative case studies and error analysis.

B. Dataset Details and Datasheet In this section, we provide granular statistics of the TopBench dataset, a detailed list of source tables, and the specific methodologies used for data synthesis. B.1. Expanded Dataset Statistics While the main paper outlines the distribution across domains and tasks, here we provide a detailed breakdown of the tabular context scale. As shown in Figure 3, TopBench challenges models with highly varying context lengths. Historical Context Lengths. We categorize the 35 source tables into Short (< 1k rows), Medium (1k–10k rows), and Long (> 10k rows). Table 5 presents the detailed distribution of rows and tokens for each domain. Table 5. Detailed Statistics of Historical Table Scales. We report the row count range and the average token count (using the Tiktoken tokenizer) for tables in each domain.

Domain

Table Length Distribution (# Tables)

Scale Metrics

Short (< 1k)

Med. (1k-10k)

Long (> 10k)

Max Rows

Avg. Tokens

Daily Consulting Finance Healthcare

1 3 5

8 1 3

4 8 2

239671 6778266 1020922

6368066 7755376 11257450

Total

6778266

8118243

Ranking Candidate Scales. For the Ranking and Filtering task, the difficulty is determined by the size of the candidate list that the model must process in the current context. • Short List (< 50 candidates): 17 queries (approx. 16%). • Medium List (50–100 candidates): 17 queries (approx. 16%). • Long List (100–200 candidates): 183 queries (approx. 68%). This distribution confirms that TopBench predominantly evaluates the capability to handle long-context inputs in agentic workflows. B.2. Table Shape and Scale Analysis To make the scale variation explicit, we also summarize query-level table contexts. Counting the table context attached to each query, T OP B ENCH spans 292 table instances. Row counts range from 160 to 6,778,266, with a median of 2,968 and a mean of 85,222.3. Column counts range from 5 to 55, with a median of 13 and a mean of 17.7. Overall, 47.3% of table contexts contain missing values, with an average missing-cell ratio of 2.85%. Categorical features appear in 78.4% of contexts. 14

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering Table 6. Query-Level Table Structure. Column buckets are computed over query-level table contexts. Statistic

Min

Median

Mean

Max

Miss. Tbl.

Cat. Tbl.

Rows Columns

160 5

2,968 13

85,222.3 17.7

6,778,266 55

47.3% -

78.4%

Table 7. Distribution by Table Shape. Row buckets are Short (≤1k), Medium (1k–100k), and Long (>100k). Column buckets are Narrow (≤10), Mid (11–30), and Wide (>30). Shape Axis Share

Row Scale

Column Width

Short

Medium

Long

Narrow

Mid

Wide

32.2%

54.5%

13.4%

33.6%

52.4%

14.0%

B.3. Truncation Ablation For text-based reasoning, 136 of 779 queries (17.5%) require truncation. Restricting to the three predictive reasoning tasks, 136 of 565 queries (24.1%) require truncation: 64/274 for Single-Point Prediction, 42/186 for Decision Making, and 30/105 for Treatment Effect Analysis. We compare three row-retention strategies while always preserving the table header: retaining head rows, random retention, and stratified retention across row chunks. B.4. Multi-Stage Data Synthesis Pipeline To ensure that the generated queries reflect real-world predictive intent rather than simple fact retrieval, we designed a rigorous three-stage synthesis pipeline. The workflow, illustrated in Figure 6, transitions from logic-driven sampling to dual-perspective generation and concludes with a hybrid validation mechanism. Stage 1: Foundation Curation and Logic-Driven Sampling. Unlike traditional benchmarks that sample rows randomly, we employ a logic-driven strategy to maximize difficulty and realism. First, we perform semantic feature engineering to identify high-cardinality columns (e.g., specific names or complex IDs), marking them for special handling to prevent token exhaustion. For the Decision Making task, we utilize a “Hard Negative Sampling” approach. Instead of randomly pairing options, we compute the similarity of target values between candidates and select pairs with minimal ground-truth differences. This forces the model to rely on precise predictive modeling rather than rough heuristics. Similarly, for Ranking and Filtering, we curate high-noise candidate pools where only a small fraction of entries satisfy the implicit predictive criteria, testing the model’s ability to filter massive datasets. Stage 2: Task Construction. To avoid the mechanical tone often found in synthetic datasets, we developed a dual-perspective prompting framework. • User Persona: For single-point prediction and decision-making, prompts simulate a non-technical individual facing a real-world dilemma. The generator is strictly instructed to weave feature values into a coherent narrative (e.g., describing a patient’s symptoms or a car’s condition) without using technical column names or JSON formatting. This ensures the query retains the ambiguity and casualness of natural human speech. • Data Holder Persona: For complex filtering tasks, the persona shifts to a domain expert (e.g., a Hiring Manager or Risk Officer). These queries explicitly reference historical archives and articulate business-centric goals (e.g., “identifying top candidates for high-risk profiles”), requiring the model to bridge the gap between business language and statistical operations. Stage 3: Hybrid Validation with Reward Modeling. The final stage ensures the validity and solvability of the generated queries through a two-step verification process. First, we deploy a specialized LLM Reward Model to audit the generated text. This model scores samples based on four criteria: completeness of feature inclusion, numerical accuracy (allowing for minor formatting variations), absence of hallucinations, and naturalness of expression. We enforce a strict quality threshold (Score > 5.0); samples falling below this score trigger a Fallback & Simplification mechanism, which regenerates the query with reduced complexity (e.g., fewer comparison options). Finally, samples passing the automated audit undergo expert human review (Human-In-The-Loop) to confirm that the implicit predictive intent is logically solvable given the provided history. 15

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering Table 8. Average Performance by Row Scale. Values are averaged over per-model means. Metric

Short

Med.

Long

Single-Point, text-based Acc. Single-Point, with-tool Acc. Decision Making, text-based Dec. Decision Making, with-tool Dec. Treatment Effect, text-based Trend Treatment Effect, with-tool Trend

0.5645 0.5624 0.5910 0.4801 0.4703 0.4497

0.5009 0.5091 0.5019 0.4624 0.6221 0.5907

0.5858 0.5162 0.5377 0.4927 0.5333 0.5412

Table 9. Average Performance by Column Width. There is no wide-column Treatment Effect subset in the current benchmark. Metric

Narrow

Mid

Wide

Single-Point, text-based Acc. Single-Point, with-tool Acc. Decision Making, text-based Dec. Decision Making, with-tool Dec. Treatment Effect, text-based Trend Treatment Effect, with-tool Trend

0.5255 0.4922 0.4769 0.3860 0.5396 0.4778

0.5298 0.5440 0.5253 0.4766 0.5613 0.5884

0.5990 0.5146 0.6778 0.6256 -

B.5. Human Authenticity Audit To check whether generated queries preserve realistic predictive intent, five annotators audited 48 sampled queries. Each query was rated on authenticity, naturalness, and plausibility with respect to the paired table and task setup. Table 13 reports the average scores.

C. Detailed Evaluation Protocols This section provides the rigorous definitions of the metrics used in T OP B ENCH and details the multi-stage verification protocols employed in our LLM-as-a-Judge pipeline. C.1. Hallucination Verification Mechanism To ensure that the values extracted by the Judge LLM are faithful to the model’s original free-form response, we implement a strict Hallucination Verification Pipeline. As illustrated in Figure 7, this mechanism filters out “hallucinated” extractions where the Judge might infer a value that the model did not explicitly state. The verification process consists of three hierarchical layers designed to balance precision and flexibility: 1. Text Standardization and Hybrid Surface Matching: Before comparison, raw responses undergo aggressive normalization: stripping LaTeX formatting (e.g., converting \text{1.5k} to 1.5k), unifying Unicode symbols, and mapping natural language numerals (e.g., “three”) to digits via a lookup table. The system then attempts to verify the extracted proof quote using a dual-strategy approach: • Spliced Matching: Handles citations with ellipses (e.g., “The value... is high”) by verifying the sequential existence of split segments. • Fuzzy Token Alignment: Uses a token subsequence algorithm (threshold ≥ 0.8) to tolerate minor morphological variations such as tense changes (“increase” vs. “increased”) or pluralization, provided the core semantic tokens align. 2. Numeric Parsing and Interval Logic: For regression tasks, we deploy a specialized NumberParser to verify mathematical equivalence between the extracted structure and the text. This module: • Unit Conversion: Maps domain-specific suffixes (e.g., “1.5k” → 1500, “20%” → 0.2, “5bn” → 5 × 109 ). • Interval Derivation: Validates implied intervals. For instance, if the text states “10 ± 2”, the system dynamically computes the range [8, 12] to verify against the extracted bounds. • Boundary Checks: Enforces strict distinction between lower and upper bounds (e.g., “more than 100” confirms 100 as a lower bound but rejects it as an upper bound). 3. Semantic Entailment (NLI Fallback): If structural and numeric matching fail—often due to high-level summarization—we activate a secondary LLM agent to perform Natural Language Inference (NLI). This agent determines if the 16

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering Table 10. Model-by-Row-Scale Cross-Tabulation for Text-Based Reasoning. Entries are Short/Medium/Long scores. Model DeepSeek-V3.2-Ins DeepSeek-V3.2-Think Gemini3.0-Flash GPT-5.2 Qwen3-Ins Qwen3-Think TableLLM-8B TableLLM-13B

Single

Decision

Treatment

0.63/0.53/0.65 0.61/0.50/0.68 0.73/0.59/0.71 0.54/0.54/0.58 0.62/0.54/0.57 0.52/0.40/0.43 0.36/0.36/0.43 0.22/0.27/0.36

0.57/0.58/0.55 0.67/0.59/0.57 0.74/0.57/0.60 0.63/0.52/0.60 0.59/0.51/0.50 0.43/0.37/0.41 0.57/0.38/0.48 0.39/0.37/0.43

0.47/0.70/0.57 0.50/0.72/0.50 0.56/0.70/0.57 0.41/0.56/0.57 0.47/0.67/0.47 0.47/0.47/0.43 0.47/0.61/0.47 0.34/0.30/0.43

Table 11. Truncation-Affected Subset. Entries are Head/Random/Stratified. Scores are reported only on queries whose serialized table exceeds the context budget. Model DeepSeek-V3.2-Ins DeepSeek-V3.2-Think Qwen3-Ins GPT-5.2

Single

Decision

Treatment

0.56/0.63/0.65 0.59/0.57/0.56 0.55/0.58/0.58 0.58/0.53/0.59

0.55/0.57/0.57 0.60/0.48/0.62 0.55/0.52/0.48 0.64/0.60/0.60

0.60/0.60/0.53 0.40/0.67/0.67 0.50/0.53/0.53 0.50/0.50/0.60

extracted claim is logically entailed by the source text. This step allows for valid semantic paraphrasing (e.g., equating “skyrocketed” with “increased significantly”) while rigorously rejecting unsupported inferences or hallucinated data points not present in the original response. Structured Extraction Schema. To standardize evaluation across diverse tasks, the Judge extracts predictions into specific JSON schemas. Figure 8 illustrates the extraction formats for Single-Point Prediction, Decision Making, and Treatment Effect Analysis. Note that for complex reasoning tasks, we extract both the final conclusion (e.g., “trend”) and the intermediate scenario-specific predictions to verify the chain of thought. C.2. Judge Configuration, Robustness, and Prompts The main judge used for structured extraction and logic scoring is DeepSeek-V3.2-Instruct. In the public release, this is configured through the DeepSeek chat endpoint and can be overridden by DEEPSEEK JUDGE MODEL ID. We use GPT-5.2 as an alternate judge for robustness checks. Prompt templates and decoding settings are fixed by task and mode. For Ranking and Filtering, evaluation is deterministic and file-based, so no judge model is used. The following are the core prompt templates exposed in the release under src/topbench/prompts. We include the text-based inference prompts, the agentic workflow prompts, and the main extraction prompt; task-specific judge schemas are shown in Figure 8. Text-Based Reasoning Prompt # System prompt Here is the preview/content of the history data: $history_table # User prompt, single-CSV tasks $query # User prompt, dual-CSV tasks $query $current_table

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering Table 12. Full Text-Based Benchmark Under Different Truncation Policies. Entries are Head/Random/Stratified. Effects are bounded because most queries do not require truncation. Model DeepSeek-V3.2-Ins DeepSeek-V3.2-Think Qwen3-Ins GPT-5.2

Single

Decision

Treatment

0.57/0.58/0.59 0.56/0.53/0.54 0.56/0.57/0.56 0.53/0.53/0.55

0.56/0.57/0.57 0.56/0.51/0.56 0.54/0.53/0.52 0.56/0.55/0.56

0.59/0.59/0.57 0.54/0.62/0.62 0.54/0.55/0.55 0.50/0.50/0.57

Table 13. Human Authenticity Audit. Scores are averaged over a five-annotator audit of 48 sampled queries. Dimension

Average Score

Authenticity Naturalness Plausibility

4.00 / 5 4.00 / 5 4.75 / 5

Agentic Workflow Prompt (Single CSV) The history data file is located at: history.csv (Note: The file is mounted in your environment, use ’history.csv’ directly) Here are the columns of the file: [$column_preview] IMPORTANT: You MUST use the ’CodeRunner’ tool to read the file to inspect the data content. You need to give the answer within $max_iterations rounds.

Agentic Workflow Prompt (Dual CSV) You have access to two csv files in your environment: 1. ’history.csv’ - Columns: $history_columns 2. ’current.csv’ - Columns: $current_columns Note: These files are mounted, use their filenames directly. The data provided above are ONLY column names. DO NOT hallucinate data rows. You MUST use the CodeRunner tool to read the files (e.g., pd.read_csv) to inspect the actual data content. CRITICAL REQUIREMENT: 1. You MUST process the data and save the final results into a file named ’result.csv’. 2. The ’result.csv’ MUST contain the exact columns matching history.csv format. 3. Do not just print the result, you must save it to ’result.csv’ using pandas to_csv().$prompt_extras You need to give the answer within $max_iterations rounds.

Judge Regression Extraction Prompt You are an expert evaluator for a tabular data prediction task (REGRESSION). Input Data: [Query]: {query} [Model Response]: {response} [Ground Truth]: {gt_str} [Dataset Metadata]: {dataset_meta_str} --### YOUR TASKS

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

STAGE 1: Foundation Curation and Logic-Driven Sampling Outlier and Noise Samples Choices with Similar Target Values

Diverse Selected Tables

Examples with Most Similar Input Features

Semantic Annotation

Candidates with Diverse Distributions

STAGE 2: Task Construction User Perspective

Data Holder Perspective

Prompt: You MUST naturally mention every

Prompt: You MUST mention that you are sharing your historical logs to help it make a decision...

single key-value pair from the `Current Profile`...

Intent-Rich Query Synthesis Single-Point Prediction

Decision Making

Treatment Effect Analysis

Ranking and Filtering

Role & Scenario: Describing a specific real-world situation to an assistant Input Data: Single Profile {Age: 19...} Requirement: Ask a direct question about the target (classification or regression)... Output: A numbered list of 5 messages.

Role & Scenario: Comparing multiple options to make a decision Input Data: The Features of Options Requirement: Create a scene with motivation. Avoid asking "Which is X?"; instead, use imperatives.... Output: A numbered list of 5 messages.

Role & Scenario: Considering a change to their current situation Input Data: Current Profile and Changed Features Requirement: Describe the current state and a justified plan for change Output: A numbered list of 5 messages.

Role & Scenario: A professional filtering a new batch of records. Input Data: Only Semantic Info Requirement: Each must apply a Hard Filter (Feature Condition) and a Top-K Prediction Ranking Output: 8 queries with conditions

STAGE 3: Hybrid Validation Process No hallucinations

LLM Auditor (Consistency Check)

Score > 5.0

Reward Model (Quality Scoring)

Human Review

TopBench Dataset

Figure 6. The Multi-Stage Data Synthesis Pipeline. The process begins with Foundation Curation, where logic-driven sampling selects challenging data points (e.g., hard negatives with similar feature values). In Task Construction, we employ a dual-perspective prompting strategy—simulating both non-technical users and professional data holders—to generate intent-rich queries across four sub-tasks. Finally, the Hybrid Validation phase filters samples using an LLM-based reward model followed by expert human verification to ensure solvability and alignment.

1. Prediction Extraction (STRICT): - Extract the final numerical prediction or interval. - CRITICAL RULE FOR VAGUE NUMBERS: - If the text says "2 million+", "over 500k", or "approx 10%", you must extract the visible number (e.g., 2000000, 500000, 10). - DO NOT make up a precise number to represent the "+". - Unless the precise number is explicitly stated in another part of the text. - CRITICAL RULE FOR INTERVALS: Do NOT narrow down or calculate. Extract the EXACT boundaries mentioned. - If the prediction value does not exist, set as null. {SYMBOL_RULE} 2. Proof Extraction (CRITICAL): - You MUST copy the proof_quote VERBATIM from the [Model Response]. - The numbers in predicted_interval MUST be visibly identical to the numbers in this quote. - For intervals, your quote MUST contain the text for BOTH the lower and upper bounds. - DO NOT add property names, keys, or prefixes. {ANTI_HALLUCINATION_EXAMPLES} {logic_block} --### OUTPUT JSON FORMAT {

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

PHASE 1: String Matcher From Ori. Response

INPUT PHASE

Exact & Spliced Match Model Response

A ... B

A [text] B

Text Normalizer

Valid PASS

SequenceMatcher > 0.8 Text siting is 28y into.. Text siting is 28y in the..

Fuzzy Token Match {

Classification

"predicted_category": "bad", "proof_quote": "your apple is more likely to be BAD ..." }

{

Regression

"predicted_value": 10535.0, "predicted_interval": [ 9500.0, 11500.0 ], "proof_quote_value": "Predicted Amount: $10,535 (from the regression model)", "proof_quote_interval": "Most Likely Range: $9,500 - $11,500" }

No Match Remove LaTeX Unicode Fix Word2Digit

PHASE 2: Numberic & Logic Quote Verifier 1.5k

1500

FeedBack Retry

NumberParser IntervalCheck

Regression

(three →> 3)

Proof Quote and Category Existence Check

Classification

LLM NLI Check

Figure 7. The Hallucination Verification Pipeline. To ensure data integrity, the extraction process employs a cascade of verification modules. Phase 1 performs aggressive text normalization followed by surface-level string matching. If direct matching fails, Phase 2 activates deep logic verifiers, including a numerical parser for unit conversion and an NLI agent to confirm semantic entailment.

"prediction_payload": { "predicted_value": number or null, "predicted_interval": [min, max] or null, "proof_quote_value": "exact substring from response containing the value. Don’t add any other words", "proof_quote_interval": "exact substring from response containing the interval. Don’t add any other words" }, "logic_assessment": { "logic_score_raw": 0-5, "detected_flaws": ["List strings from STEP 1 (e.g. ’Self-Contradiction’) or empty []"], "reasoning": "Brief justification." } } Output RAW JSON only.

C.3. Metrics for Predictive Reasoning For tasks involving natural language outputs, we assess performance using a combination of numerical precision and logical coherence. Normalized Mean Absolute Error (NMAE). To compare regression performance across datasets with vastly different scales (e.g., percentage rates vs. financial volumes), we normalize the error by the full value range of the dataset. For a predicted value ŷ and ground truth y: |y − ŷ| NMAE = (3) max(Y ) − min(Y ) 20

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

// Single-Point Prediction (Regression) { "task_type": "regression", "prediction_payload": { "predicted_value": 10535.0, "predicted_interval": [9500.0, 11500.0], "proof_quote_value": "Predicted Amount: $10,535", "proof_quote_interval": "Most Likely Range: $9,500 - $11,500" } } // Decision Making (Classification) { "task_type": "classification", "scenarios_extraction": { "001": { "predicted_category": "good", "proof_quote": "Apple 1967.0 would be the better choice..." }, "002": { "predicted_category": "bad", "proof_quote": "Apple 3037.0 is more likely to be the ’bad’ apple." } }, "final_decision_extraction": { "predicted_winner_id": "001", "proof_quote": "For your demo, Apple 1967.0 would be the better choice..." } } // Treatment Effect Analysis (Regression Trend) { "task_type": "regression", "scenario_002_extraction": { // The counterfactual scenario "predicted_value": 4978.34, "predicted_interval": null, "proof_quote_value": "Predicted charges: **$4,978.34**" "proof_quote_interval": null }, "trend_extraction": { "predicted_trend": "higher", "proof_quote": "Moving... will significantly INCREASE your medical charges." } }

Figure 8. Standardized JSON Schemas for Predictive Reasoning Tasks. The Judge extracts structured payloads containing point estimates, intervals, and verbatim proof quotes. For B2 and B3, predictions are extracted per scenario to verify comparative reasoning.

where max(Y ) and min(Y ) represent the maximum and minimum observed values of the target column in the historical dataset. If the range is negligible, we default to the absolute magnitude of the ground truth. Interval Intersection over Union (IoU) and Width Penalty. For probabilistic predictions where models output a confidence interval [ŷmin , ŷmax ], we calculate the Intersection over Union (IoU) with a constructed ground truth region. Since historical records typically provide single-point outcomes, we define the ground truth interval as [y − 0.5σ, y + 0.5σ], where σ is the standard deviation of the target variable derived from the dataset statistics. To prevent models from gaming the metric by predicting overly broad ranges (e.g., [−∞, ∞]), we apply an exponential Width Penalty: Wpred Pwidth = exp −α · max 0, −τ Wgt

(4)

where Wpred is the predicted width, Wgt = σ is the reference width, and τ = 2.0 is the tolerance threshold. This ensures that precision is not sacrificed for coverage. Logic Score. Beyond accuracy, we evaluate the reasoning process itself. The Judge LLM assigns a scalar score (0-5) based on a rubric that penalizes logical fallacies such as Self-Contradiction (text conflicts with prediction), Circular Reasoning, and False Causality (linking irrelevant IDs to outcomes). 21

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering Table 14. Judge Robustness Across Model Families. We rescore the same outputs with a GPT-family judge and report task-score differences relative to the DeepSeek judge. Setting

DS Judge

GPT Judge

Gap

SP, text SP, tool DM, text DM, tool TE, text TE, tool

0.5696 0.5901 0.5699 0.5672 0.5524 0.6095

0.4934 0.5147 0.5780 0.5618 0.5905 0.6810

-0.0763 -0.0754 +0.0081 -0.0054 +0.0381 +0.0714

Overall

0.5763

0.5502

-0.0262

Table 15. Manual Audit of Judge Extraction. Human annotators checked whether extracted structured values faithfully matched the model output. The only mismatch occurred because the response ambiguously provided multiple options, while the strict judge selected only one classification label. The two judge families also agree on the low-logic flag in 95.1% of cases with logic score below 0.4. Task

Audited Cases

Extraction Agreement

Single-Point Prediction Decision Making Treatment Effect Analysis

24 24 24

100.0% 95.8% 100.0%

Overall

98.6%

C.4. Metrics for Structured Ranking For the Ranking and Filtering task, which outputs structured CSV files, we employ deterministic set-based and rank-aware metrics. Set Retrieval Metrics. We treat the filtering task as a retrieval problem. Let Sgt be the set of ground truth candidates satisfying the implicit conditions, and Spred be the set of candidates returned by the model. We calculate Precision, Recall, and F1 Score to measure the model’s ability to strictly adhere to feature constraints. Rank-Aware Metrics. For queries requiring prioritized lists (e.g., “top 10 highest risk”), we evaluate the ordering quality using NDCG@k (Normalized Discounted Cumulative Gain). This metric rewards models for placing high-relevance items at the top of the list, discounting correct items that appear lower down. Kendall’s Tau (τ ). To assess the model’s ability to capture the correct relative order of candidates—even if specific values are imprecise—we compute Kendall’s Rank Correlation Coefficient (τ ) between the predicted ranking and the ground truth ranking of the matched pairs. Batch NMAE. To verify that the model is performing genuine batch inference rather than simple sorting, we calculate the NMAE averaged exclusively over the correctly retrieved (matched) candidates. This decouples the retrieval performance from the predictive precision.

D. Extended Experimental Results This section presents a granular analysis of model performance across three dimensions. First, we examine the quantitative precision and ranking quality in the structured Ranking and Filtering task. We analyze the structural integrity of the generated outputs, highlighting the trade-offs between reasoning depth and format compliance. Finally, we investigate the agentic behaviors of models by profiling their preference for predictive tools across different task types and information settings. D.1. Performance on Ranking and Filtering In this section, we provide a granular analysis of the Ranking and Filtering task. Unlike the natural language reasoning tasks, this scenario requires models to act as data processors, generating structured CSV outputs that satisfy complex filtering conditions and predictive ranking criteria. We evaluate performance across three dimensions: predictive precision, the impact of enhanced schema context, and structural output integrity. 22

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering Table 16. Metric Sensitivity on Single-Point Prediction. We sweep point weight in {0.2,0.3,...,0.8}, width factor in {1.5,2.0,3.0}, and width decay in {0.5,1.0,2.0}. Statistic

Text

Tool

Top-1 unchanged across all profiles Mean Spearman agreement vs. default Mean pairwise order agreement

1.0000 0.9960 0.9900

1.0000 0.9913 0.9833

D.1.1. C ORE P REDICTIVE P ERFORMANCE We first analyze the performance of General LLMs and Reasoning-Enhanced Models in the standard setting without auxiliary schema descriptions. Table 17 details the metrics for regression ranking and classification filtering. In the context of regression-based ranking, Gemini 3 Flash and DeepSeek-V3.2-Instruct demonstrate the highest consistency. Gemini 3 Flash achieves the leading NDCG score of 0.5046 and the highest Kendall’s τ of 0.2740, indicating superior capability in ordering high-value candidates correctly relative to the ground truth. DeepSeek-V3.2-Instruct closely follows in ranking quality and distinguishes itself with the highest absolute precision, achieving the lowest NMAE of 0.2639 and a median NMAE of 0.0448. Claude Sonnet 4.5 exhibits a strong retrieval capability with a Recall of 0.5242, comparable to the top performers, but records a higher NMAE (0.3149), suggesting that while it effectively identifies the correct set of candidates, its estimation of their specific attribute values is less precise. For categorical filtering tasks, Gemini 3 Flash maintains the lead with an F1 score of 0.5751, followed by Claude Sonnet 4.5 at 0.5456. In contrast, the Tabular Specialists, specifically TableLLM-8B and TableLLM-13B, fail to adapt to this complex agentic setting. Their near-zero scores across all metrics highlight the significant gap between domain-specific tuning on short contexts and the long-context reasoning required by TopBench. Table 17. Core Performance Metrics for Ranking and Filtering. We report detailed metrics for Regression (Recall, NDCG, NMAE, Kendall’s τ ) and Classification (F1). Reg NMAE Med denotes the median Normalized Mean Absolute Error, providing a robust measure of error unaffected by outliers.

Model

Regression Metrics NDCG NMAE NMAE (Med)

Kendall

Classification Metrics F1 Recall

0.6000 0.6000 0.6125 0.4000 0.6000 0.4000 0.0000

0.5046 0.5045 0.4964 0.4149 0.4624 0.4096 0.2791

0.3031 0.2639 0.3149 0.4058 0.3527 0.4550 0.6057

0.0489 0.0448 0.0517 0.1028 0.0697 0.1465 1.0000

0.2740 0.2148 0.2351 0.2031 0.1802 0.1838 0.0557

0.5751 0.4838 0.5456 0.3825 0.3842 0.4191 0.2782

0.6787 0.5671 0.6694 0.4372 0.4584 0.5186 0.4402

0.0000 0.0000

0.0040 0.0000

0.9951 1.0000

1.0000 1.0000

0.0000 0.0000

0.0001 0.0000

0.0171 0.0000

Recall

Recall (Med)

Gemini 3 Flash DeepSeek-V3.2-Instruct Claude Sonnet 4.5 GPT-5.2 DeepSeek-V3.2-Thinking Qwen3-Thinking Qwen3-Instruct

0.5331 0.5393 0.5242 0.4561 0.4977 0.4289 0.3069

TableLLM-8B TableLLM-13B

0.0028 0.0000

D.1.2. I MPACT OF E NHANCED S CHEMA C ONTEXT We investigate whether providing rich schema metadata (column definitions and value distributions) improves performance. Table 18 contrasts the Standard models with their With Info Mode variants. The provision of metadata yields significant gains for models with weaker initial performance. Qwen3-Instruct observes a substantial boost, with Regression Recall increasing from 0.3069 to 0.3693 and Median NMAE improving markedly from 1.0 to 0.8309. Similarly, GPT-5.2 sees consistent improvements, particularly in classification tasks where its F1 score rises from 0.3825 to 0.4760. Conversely, top-tier frontier models such as Gemini 3 Flash and DeepSeek-V3.2-Instruct show minimal performance variance or diminishing returns. This stability suggests that these advanced models possess robust internal capabilities to infer schema semantics directly from raw tabular data, rendering explicit metadata less critical for their reasoning processes compared to smaller models. 23

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering Table 18. Impact of Contextual Schema Information. Comparison of key metrics between standard models and models provided with auxiliary schema info (w/ Info). Metrics shown are Regression Recall, Regression NMAE, and Classification F1.

Base Model DeepSeek-V3.2-Instruct Gemini 3 Flash GPT-5.2 Qwen3-Instruct

Standard Mode Reg Recall Reg NMAE 0.5393 0.5331 0.4561 0.3069

0.2639 0.3031 0.4058 0.6057

Cls F1

With Info Mode Reg Recall Reg NMAE

0.4838 0.5751 0.3825 0.2782

0.5490 0.5291 0.4851 0.3693

0.2398 0.3072 0.3542 0.5466

Cls F1 0.5462 0.5659 0.4760 0.3339

D.1.3. S TRUCTURAL I NTEGRITY AND C OMPLIANCE A NALYSIS Given that Ranking tasks necessitates the generation of valid CSV files, we evaluate the structural integrity of the output in Table 19. A notable trade-off emerges between reasoning depth and format compliance. DeepSeek-V3.2-Thinking, while competitive in predictive metrics, exhibits a high CSV Parse Error Rate of 16.82% and a corresponding Missing Columns Rate. This suggests that models optimized for extensive Chain-of-Thought reasoning may struggle to separate internal monologue from the strict formatting requirements of structured file generation. In contrast, Gemini 3 Flash achieves flawless structural integrity with 0.00% error rates across all categories. Regarding logical adherence, the Filter Compliance Rate remains consistently high (> 98%) across all generalist models, confirming that the primary challenge in TopBench is the implicit prediction of target variables rather than the execution of explicit filtering instructions. Table 19. Structural Integrity and Error Analysis. We report the rate of CSV parsing failures, schema hallucinations (Extra/Missing Columns), and the logical Filter Compliance Rate. Lower error rates and higher compliance indicate better instruction following.

Model

CSV Parse Error

Missing Cols

Extra Cols

Empty Result

Filter Compliance

Gemini 3 Flash DeepSeek-V3.2-Thinking GPT-5.2 Qwen3-Thinking Claude Sonnet 4.5 Qwen3-Instruct DeepSeek-V3.2-Instruct

0.00% 16.82% 0.00% 0.00% 1.40% 1.40% 0.47%

0.00% 16.82% 0.00% 0.93% 1.40% 5.14% 0.47%

0.00% 0.47% 0.47% 0.93% 0.47% 0.93% 0.93%

0.00% 0.00% 0.00% 0.00% 0.47% 0.47% 0.00%

100.00% 100.00% 99.53% 99.53% 99.07% 99.07% 98.60%

TableLLM-8B

45.33%

47.66%

6.54%

0.00%

98.13%

D.2. Analysis of Predictive Tool Usage Behaviors We provide a granular analysis of how different Large Language Models (LLMs) select and utilize computational tools across the four predictive tasks. By comparing the tool invocation distributions in the standard setting against the semantic-enhanced setting, we isolate the impact of intent recognition on modeling strategy. Baseline Tool Usage Patterns. In the standard setting, distinct behavioral profiles emerge among the evaluated models. As illustrated in Figures 9 through 15, DeepSeek-V3.2-Instruct consistently exhibits a strong preference for rigorous predictive modeling, employing algorithms such as Random Forest in over 50% of cases across Single Point Prediction and Treatment Effect Analysis tasks. This suggests an intrinsic alignment with the predictive nature of the queries. In contrast, Qwen3-Instruct demonstrates a marked tendency towards heuristic solutions, predominantly relying on pandas for data filtering or performing direct arithmetic calculations. This behavior indicates that without explicit guidance, the model often misinterprets the implicit predictive intent as a retrieval or simple aggregation task. Notably, in the Ranking and Filtering task (Figure 15), we observe a universal increase in the complexity of chosen algorithms, with models frequently deploying gradient boosting frameworks to handle the batch scoring requirements. Impact of Semantic Disambiguation. The introduction of explicit semantic information—specifying target columns and task types—triggers a significant shift in agentic behavior. Comparing the baseline distributions with their enhanced 24

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

counterparts (Figures 10 to 16) reveals a convergence towards formal statistical modeling. The most dramatic transformation is observed in Qwen3-Instruct. In the Single Point Prediction task, its utilization of machine learning libraries surges, effectively bridging the strategic gap with DeepSeek and GPT-5.2. This behavioral correction confirms that the model’s prior reliance on simple heuristics stemmed primarily from intent ambiguity rather than a lack of coding capability. Furthermore, for the Decision Making task, while the shift in tool usage is evident, the accompanying performance gains are less pronounced, highlighting that correct tool invocation is a necessary but not sufficient condition for resolving fine-grained trade-offs.

9% 45%

54%

11% 17%

31% 28%

DeepSeekV3.2 Instruct

85%

92%

Qwen3 Instruct

Qwen3 Thinking

DeepSeekV3.2 Thinking 6%

40%

54%

10%

58%

23%

93%

Claude Sonnet-4.5

Gemini3 Flash

GPT-5.2

Predictive Models Random Forest Logistic Regression Linear Regression Gradient Boosting Others Without Modeling

Figure 9. Tool Usage Distribution for Single Point Prediction (Standard). DeepSeek actively employs predictive models, while Qwen3 relies heavily on non-modeling approaches.

Integrity of the Modeling Pipeline. Merely invoking a machine learning library does not guarantee a valid prediction; the raw tabular data must be rigorously prepared. We further investigate the completeness of the generated code by measuring the rate of data preprocessing—specifically, the implementation of categorical encoding or missing value imputation—given that a predictive model was instantiated. As shown in Figure 17, there is a strong correlation between model capability and pipeline integrity. Advanced agents not only select the correct algorithms but also autonomously recognize the need to transform raw string-based features into numerical formats compatible with standard libraries. In contrast, despite correctly identifying the need for prediction, weaker models frequently attempt to feed raw data directly into regressors. This observation highlights that true agentic intelligence extends beyond simple intent recognition to the comprehensive emulation of a data scientist’s workflow, encompassing both algorithm selection and essential data hygiene.

E. Qualitative Case Studies and Error Analysis In this section, we provide a qualitative examination of model behaviors, contrasting successful execution patterns in the agentic framework with typical failure modes observed in the text-based setting. We begin by analyzing the correlation between response verbosity and reasoning effectiveness across different model families. E.1. Analysis of Response Length and Verbosity We investigate the distribution of generated token counts across different inference modes to understand how tool availability influences model expressiveness. Figure 18 illustrates the average response length for each model, comparing the Text-Based Reasoning mode (No Tool) against the Agentic Workflow (With Tool). The data reveals a counter-intuitive phenomenon where “verbosity”often correlates inversely with “reasoning quality,”a trend that varies significantly across model types. In the Text-Based setting, open-weights models such as Qwen3-Instruct and 25

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

9% 54%

11%

9% 10% 58%

31% 85%

93%

Qwen

DeepSeek

23%

Gemini

GPT

24% 37%

40%

51%

38%

59%

41%

63% 34%

Qwen with Info

DeepSeek with Info Random Forest

Logistic Regression

Gemini with Info Linear Regression

Others

GPT with Info Without Modeling

Figure 10. Tool Usage Distribution for Single Point Prediction (With Semantic Info). The addition of semantic metadata prompts a significant increase in modeling frequency for Qwen3.

DeepSeek-V3.2 exhibits the most inflated response lengths. Lacking the internal capacity for precise arithmetic, these models frequently resort to generating extensive, convoluted reasoning chains to justify heuristic estimates. This “hallucination by verbosity”suggests an attempt to simulate logical depth through length. However, upon enabling the Agentic Workflow, these specific models show the most dramatic reduction in token usage. By offloading complex calculations to the interpreter, they transition from verbose rhetoricians to efficient dispatchers, converging towards the naturally more concise output patterns observed in proprietary models like GPT-5.2. This indicates that for open-weights models, tool access serves as a critical regularizer, effectively pruning unnecessary textual generation in favor of deterministic executable logic. E.2. The Retrieval Trap in Text-Based Reasoning A pervasive limitation observed across all evaluated models in the text-based setting is the tendency to reduce predictive problems to heuristic retrieval tasks. Lacking the computational tools to fit a statistical function f (x) → y, models rely on identifying “nearest neighbors”within the truncated context window to infer the target value. Heuristic Range Estimation vs. Model Bias. As illustrated in Figure 19, when presented with a query profile in the text-only mode, the model locates historical rows with similar feature values (e.g., matching region or age). Instead of producing a precise point estimate, it aggregates these retrieved values to construct a fuzzy confidence interval (e.g., “between $9,630 and $10,000”). While this approach provides a conservative estimate, it lacks the mathematical precision to capture complex non-linear dependencies. In contrast, Figure 20 demonstrates how the Agentic Workflow enables the model to explicitly train a regressor. However, methodological superiority does not guarantee numerical precision. While the agent successfully transitions from retrieval to modeling, the predicted range ($13,500–$14,000) still deviates significantly from the ground truth ($9,391). This error stems from the model’s tendency to default to simple algorithms (e.g., unoptimized Linear Regression) within the zero-shot sandbox, which often underfit complex real-world distributions characterized by non-linear interaction terms (e.g., the interplay between BMI and age). Thus, while the Agentic framework corrects the process, the accuracy remains bound by the sophistication of the generated code. Superficial Feature Matching in Decision Making. This retrieval-based bias is equally detrimental in decision-making tasks. Figure 21 shows that without tools, models attempt to predict the better candidate by searching for historically similar 26

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

8% 58%

13%

12% 42%

20%

21% 91%

23%

DeepSeekV3.2 Instruct

DeepSeekV3.2 Thinking

39%

55%

27%

91%

Claude Sonnet-4.5

Qwen3 Thinking

Qwen3 Instruct

8% 57%

99%

Gemini3 Flash

Predictive Models Random Forest Logistic Regression Linear Regression Others Without Modeling

GPT-5.2

Figure 11. Tool Usage Distribution for Decision Making (Standard). Models show a mixed strategy, balancing between comparison logic and predictive modeling.

instances. However, this often leads to “superficial feature matching,”where the model latches onto salient but non-causal features to justify its choice. Conversely, the Agentic approach shown in Figure 22 allows the model to quantify the expected outcome for both candidates independently using a trained predictor, transforming the decision from a qualitative guess into a quantitative comparison argmax(ypred ), though subject to the same regression precision limitations noted above. E.3. Failure Mode: The Exhaustive Retrieval Loop Beyond the general tendency for retrieval, we observe a specific, catastrophic failure mode in reasoning-enhanced models, most notably Qwen3-Thinking. We term this the “Exhaustive Retrieval Loop.” As depicted in Figure 23, this error occurs when the model fundamentally misinterprets the implicit prediction task as a strict database lookup. Faced with a query describing an unobserved profile (e.g., a hypothetical fruit or patient), the model operates under the false assumption that an exact match exists within the historical log. Consequently, it initiates a brute-force linear search, mechanically iterating through the serialized table string row by row. The reasoning trace degenerates into a repetitive cycle of element-wise verification (e.g., “Checking entry 1... mismatch. Checking entry 2... mismatch.”), consuming the entire context window. Since the query target is a future state rather than a historical record, this search is inherently futile. The model eventually exhausts its maximum token limit or hallucinates a match to break the loop. This behavior highlights a critical cognitive gap: despite strong logical capabilities, the model lacks the tabular awareness to distinguish between querying the past and modeling the future.

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

13% 58%

55%

21% 91%

91%

Qwen

DeepSeek

Gemini

12%

GPT

20%

29%

45% 41%

37%

Qwen with Info Random Forest

19%

38%

71%

77%

DeepSeek with Info

27%

Gemini with Info

Logistic Regression

Linear Regression

Others

GPT with Info Without Modeling

Figure 12. Tool Usage Distribution for Decision Making (With Semantic Info). Explicit task definition encourages models to adopt more formal comparative analysis techniques.

7% 42%

20%

44%

80%

21%

DeepSeekV3.2 Thinking 6%

14%

11% 50%

Claude Sonnet-4.5

87%

Qwen3 Instruct

43% 40%

14%

30%

DeepSeekV3.2 Instruct

6% 11%

13%

6% 7% 11% 23%

83%

Gemini3 Flash

GPT-5.2

Qwen3 Thinking Predictive Models Linear Regression Random Forest Logistic Regression OLS linregress Others Without Modeling

Figure 13. Tool Usage Distribution for Treatment Effect Analysis (Standard). Causal reasoning scenarios drive a higher baseline usage of regression models across all agents.

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

6% 11%

20%

42%

11%

7% 11%

50% 80%

30%

Qwen

DeepSeek

23%

83%

Gemini

28%

38%

GPT

37% 50%

47%

33%

10%

62%

22% 23%

31%

Qwen with Info

DeepSeek with Info Random Forest Linear Regression

Gemini with Info

Logistic Regression Ridge

OLS Gradient Boosting

GPT with Info

linregress NearestNeighbors

Others Without Modeling

Figure 14. Tool Usage Distribution for Treatment Effect Analysis (With Semantic Info). Enhanced context further solidifies the preference for causal inference methods over heuristic estimation.

7% 27%

61%

10%

11% 28%

54%

29%

DeepSeekV3.2 Instruct

DeepSeekV3.2 Thinking

23%

29%

48% 39%

66%

Qwen3 Thinking

Qwen3 Instruct

Predictive Models

72%

Claude Sonnet-4.5

30%

37%

70%

30%

Gemini3 Flash

GPT-5.2

Random Forest Logistic Regression Linear Regression Gradient Boosting Others Without Modeling

Figure 15. Tool Usage Distribution for Ranking and Filtering (Standard). The complexity of batch processing naturally leads to a higher adoption of robust algorithms like Random Forest.

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

29%

28%

30%

37%

27%

61%

66%

70% 30%

Qwen

DeepSeek

Gemini

GPT

14%

17%

23%

42% 44%

52% 71%

37%

86%

Qwen with Info

DeepSeek with Info Random Forest

Logistic Regression

Gemini with Info Linear Regression

GPT with Info

Others

Without Modeling

Figure 16. Tool Usage Distribution for Ranking and Filtering (With Semantic Info). Semantic clarity assists models in selecting more appropriate feature sets for batch ranking algorithms.

Model Usage and Data Preprocessing Capabilities 70

Model Usage (Total) Data Preprocessing (Ratio in Usage)

60.3%

55.4%

53.9%

49.3%

Rate (%)

50 40 30

26.3% 19.5%

21.6%

10 0 DeepSeekV3.2 Instruct

DeepSeekV3.2 Thinking

Qwen3 Instruct

Qwen3 Thinking

Claude Sonnet-4.5

Gemini3 Flash

GPT-5.2

Figure 17. Ratio of Data Preprocessing in Predictive Workflows. The chart quantifies the conditional probability that a model implements necessary feature engineering steps (e.g., encoding, imputation) when employing machine learning algorithms. Higher ratios indicate more robust and executable code generation.

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Avg Response Length (Chars)

6000

No Tool With Tool

83% 5104

5000

4000

3000

2000

15% 2243 1909

45% 2686

1474 1000

DeepSeek Instruct

DeepSeek Thinking

44% 1955

43% 1677 1341 848

950

Qwen3 Instruct

Qwen3 Thinking

Claude Sonnet4.5

43% 1740

1092

995

Gemini3 Flash

GPT-5.2

Figure 18. Comparison of Average Response Lengths. We contrast the token usage between Text-Based Reasoning (No Tool), where models rely solely on internal parameters, and the Agentic Workflow (With Tool), where models utilize Python execution. The lengths reported include the generated reasoning text and code blocks.

Single-Point Prediction Text-Based

Ground Truth: "charges": 9391.346

Query: Just had my annual checkup, and the doctor reviewed everything with me. So, being a 51-year-old man living in the southwest, he noted my BMI is 39.7. I've got one child on my insurance plan, and while I'm not a smoker, he did express some concern about my weight. With all that in mind, what kind of medical insurance bill should I be preparing for this year?

Context: age,sex,bmi,children,smoker,region,charges 23,male,31.73,3,yes,northeast,36189.1017 19,male,28.7,0,no,southwest,1253.936 19,male,20.9,1,no,southwest,1832.094 19,male,33.1,0,no,southwest,23082.95533 18,male,31.68,2,yes,southeast,34303.1672 48,female,28.88,1,no,northwest,9249.4952 57,male,28.975,0,yes,northeast,27218.43725 25,male,24.985,2,no,northeast,23241.47453 31,female,38.095,1,yes,northeast,58571.07448

Response From DeepseekV3.2-Instruct: Based on the historical insurance data you provided, for a 51-year-old male in the southwest with a BMI of 39.7, one child, and who is a non-smoker, the estimated annual medical insurance charges would be approximately **$9,630 to $10,000**. Here’s a breakdown of the key factors influencing this estimate: * * * * *

**Age (51):** This is a significant factor, as premiums generally increase with age. **BMI (39.7):** This falls into the obese category (BMI ≥ 30), which typically leads to higher premiums due to associated health risks. **Region (Southwest):** Costs can vary by region, and the southwest shows premiums in this range for similar profiles. **Smoker Status (Non-Smoker):** This keeps your costs lower than if you were a smoker. **Children (1):** Adding one dependent child increases the premium.

**Important Disclaimer:** This is an estimate based on historical data patterns. Your actual premium will be determined by the specific insurance plan you choose (e.g., deductible, co-pays, network), your exact location within the southwest, the insurance provider, and your complete medical history. **Recommended Next Steps:** 1. **Get Official Quotes:** Contact insurance providers directly or use the Health Insurance Marketplace for personalized quotes. 2. **Consider Plan Details:** Look beyond the premium at deductibles, out-of-pocket maximums, and covered services. 3. **Discuss with Your Doctor:** His concern about weight is a health prompt. Addressing that could improve your long-term health and potentially lower future insurance costs.

Figure 19. Single-Point Prediction (Text-Based). The model resorts to retrieving similar historical rows and estimating a vague range based on neighbor values. While semantically plausible, this approach fails to capture the precise, non-linear mapping required for the ground truth target.

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Single-Point Prediction Agentic-Workflow

Ground Truth: "charges": 9391.346

Response From DeepseekV3.2-Instruct: [Code] from sklearn.linear_model import LinearRegression from sklearn.preprocessing import LabelEncoder Based on my analysis of the insurance data, here's what you should prepare for in terms of medical insurance costs: ## **Predicted Annual Insurance Bill: $13,500 - $14,000** ### **Key Factors Driving Your Costs:** 1. **Age (51 years)**: Adds about $255 per year of age 2. **High BMI (39.7)**: Your BMI is in the 92nd percentile, adding approximately $330 per BMI unit 3. **Non-smoker status**: This is actually saving you about **$23,400 per year** compared to smokers! 4. **One dependent child**: Adds about $519 to your premium 5. **Southwest region**: Slightly lower costs than some other regions ### **Comparison Data:** - **Most similar profiles** (men 45-57, southwest, non-smokers, 1 child, high BMI): $8,068 - $11,576 - **Your predicted range**: $13,500 - $14,000 (slightly higher due to your specific age and exact BMI) - **All non-smokers average**: $8,452 - **All smokers average**: $31,876 ......

Figure 20. Single-Point Prediction (Agentic Workflow). Leveraging the sandbox, the model trains a Linear Regression model to generate a numerical estimate. Note that while the approach is correct (modeling vs. retrieval), the result ($13.5k) still deviates from the ground truth ($9.3k), highlighting the limitation of simple default algorithms in zero-shot code generation.

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Decision Making Text-Based

Ground Truth: "charges": "1725.5523" "charges": "1720.3537" ✅ "charges": "1727.54"

Query: I'm trying to help my younger cousin get a sense of his potential health insurance costs as he's looking at plans... The first person is an 18-year-old man from the Southeast. He's not a smoker, has a Body Mass Index of 33.77, and has one child covered on his plan. The second record is also... Finally, the third option... With all these similar but slightly different profiles, I'm a bit stuck on figuring out which one of these three would likely have the lowest medical costs billed by insurance. Can you help me sort that out?

Response From DeepseekV3.2-Instruct: Let's break this down. You have three 18-year-old male non-smokers from the Southeast, each with one child, and the only difference is their BMI: 1. **Person A**: BMI = 33.77 2. **Person B**: BMI = 30.03 3. **Person C**: BMI = 35.2 From the dataset, I can find records that closely match these profiles to estimate their insurance charges. ### **Matching records from the dataset** ...... ### **General trend from data** Looking at 18-year-old males in Southeast, non-smoker, 0 children (since 1-child records are rare for this age in dataset): ...... ### **Estimated order of cost (lowest to highest)** 1. **Person B (BMI 30.03)** — lowest BMI → lowest base charge. 2. **Person A (BMI 33.77)** — middle BMI → middle charge. 3. **Person C (BMI 35.2)** — highest BMI → highest charge. **Answer**: **Person B** (BMI 30.03) would likely have the lowest medical costs, followed by Person A, then Person C

Figure 21. Decision Making (Text-Based). The model relies on qualitative comparisons with retrieved historical samples. The reasoning is fragile, often basing the decision on superficial feature similarities (e.g., region matches) rather than calculated risk factors.

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Decision Making Agentic-Workflow

Ground Truth: "charges": "1725.5523" "charges": "1720.3537" ✅ "charges": "1727.54"

Response From DeepseekV3.2-Instruct: [Code] from sklearn.linear_model import LinearRegression Based on my analysis of the health insurance data, I can provide you with a clear answer: ## **Summary of Findings** After analyzing the insurance data and building predictive models, here's what I found for your three scenarios: ### **Predicted Insurance Charges:** 1. **Second person (BMI 30.03)**: $2,211.46 2. **First person (BMI 33.77)**: $2,286.33 3. **Third person (BMI 35.20)**: $2,314.95 ...... ### **Final Recommendation:** Your cousin should expect that **lower BMI generally correlates with lower insurance costs**. Among the three scenarios presented, the profile with **BMI 30.03** would likely have the most affordable insurance premiums. The good news is that all three scenarios are relatively close in predicted costs (within about $100 of each other annually), and all are significantly below the average for non-smokers ($8,451), primarily due to the young age and non-smoking status.

Figure 22. Decision Making (Agentic Workflow). The model quantifies the decision by predicting exact scores for both candidates. This quantitative comparison enables a more rigorous trade-off analysis compared to the fuzzy logic of the text-based baseline.

T OP B ENCH: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering

Single-Point Prediction Text-Based Qwen Error Ground Truth: "charges": 9391.346 Query: Just had my annual checkup, and the doctor reviewed everything with me. So, being a 51-year-old man living in the southwest, he noted my BMI is 39.7. I've got one child on my insurance plan, and while I'm not a smoker, he did express some concern about my weight. With all that in mind, what kind of medical insurance bill should I be preparing for this year?

Response From Qwen3-Thinking: No Result [Thinking Process / Chain of Thought]: Let me check if there are any entries with BMI around 39.7, non-smoker, in the southwest region: 60,male,39.9,0,yes,southwest,48173.361 (smoker) 55,male,38.28,0,no,southeast,10226.2842 (different region) 50,male,39.6,0,no,southwest,8457.818 (0 children) Actually, I found this entry: 51,male,39.7,0,no,southwest,10713.644 This is very close to what we're looking for, except it has 0 children instead of 1. Let me check if there are any entries with 1 child in the southwest region: I don't see any exact matches, but let me check more carefully. Actually, I found this entry: 48,male,37.05,1,yes,southwest,24603.04837 This has 1 child and is in the southwest region, but the person is a smoker (yes). Let me check if there are any entries with 1 child, non-smoker, in the southwest region: I don't see any exact matches...

Figure 23. The Exhaustive Retrieval Loop (Qwen3-Thinking). Misinterpreting the prediction task as a database lookup, the model enters an infinite loop of row-by-row verification. It attempts to find an exact match for a hypothetical profile, eventually exhausting the context window without producing a valid answer.

Record · ID 146925 · SHA-256 cc6953358c20be10
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.