TAB V ERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs Momina Ahsan1 , Sarfraz Ahmad1 , Ming Shan Hee1 , Roy Ka-Wei Lee2 , Preslav Nakov1 1 Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) 2 Singapore University of Technology and Design (SUTD) {momina.ahsan, preslav.nakov}@mbzuai.ac.ae Project õ TAB V ERSE § Code Leaderboard Abstract
arXiv:2606.09578v1 [cs.AI] 8 Jun 2026
Large Language Models (LLMs) and VisionLanguage Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation remains under-explored. In practice, the same table content may appear in different structural formats, such as HTML, Markdown, and LaTeX, or as rendered images. However, existing evaluations often let content, format, layout, and modality vary together, making it difficult to isolate representation effects. We introduce TAB V ERSE, a controlled multimodal table benchmark that aligns the same table content across multiple structural formats and rendered images, with question category and difficulty tags. This design enables systematic evaluation of representation effects while holding table content fixed. We evaluate LLMs and VLMs across three tasks: Question Answering (QA), Structural Understanding Capability (SUC), and Structure Reconstruction (SR). Our results show that representation choice substantially affects table understanding. Models generally perform better with structured text than with rendered images, but the size of this gap depends on the task, model, and format. HTML is often the most robust text format, while row-sensitive structural tasks and syntactically usable LaTeX reconstruction remain challenging. These findings show that table representation is a key factor in reliable table evaluation.
1
Introduction
Tables are widely used to present structured information in scientific documents, reports, and web content. This makes table understanding critical for AI systems that interpret and verify real-world data (Smock et al., 2022). Despite strong progress in Large Language Models (LLMs) and VisionLanguage Models (VLMs), table comprehension remains challenging (Brown et al., 2020; Touvron et al., 2023; Bubeck et al., 2023).
Unlike plain text, tables require models to interpret both content and structure, including headers, merged cells, row and column boundaries, and relevant cells (Deng et al., 2024; Sui et al., 2024a; Kim et al., 2024). The same table content can also be presented in different ways i.e., as HTML, LaTeX, or Markdown, or as a rendered image in a PDF or screenshot. These representations expose different cues. Structured text provides markup and delimiters, while images provide visual layout, so model performance can change even when the underlying table content is identical. Recent work has introduced table-specialized models (Zhang et al., 2024; Deng and Mihalcea, 2025) and shown that table reasoning is sensitive to serialization, prompting, and modality choices (Deng et al., 2024; Sui et al., 2024a,b; Singha et al., 2023). However, many benchmarks and evaluation pipelines let table content, format, layout, and modality vary together, making it difficult to isolate the effect of representation itself. We introduce TAB V ERSE, a benchmark for controlled cross-format and cross-modality table evaluation. TAB V ERSE aligns identical tables across three structural formats (HTML, LaTeX, Markdown) and their rendered images, enabling comparison while holding table content fixed. Built from held-out evaluation splits of FEVEROUS, H YBRID QA, TAB FACT, SQA, and W IKI TABLE Q UESTIONS, it includes a full tagged pool and a 700-sample balanced evaluation set balanced by question category and difficulty. We evaluate LLMs and VLMs on three complementary tasks: Question Answering (QA), Structural Understanding Capability (SUC), and Structure Reconstruction (SR). QA measures answer prediction under different table representations; SUC probes structure understanding through boundary detection, size estimation, and index-based retrieval; and SR measures whether VLMs can reconstruct tables from rendered images.
Our contributions are as follows: • We formulate cross-format and cross-modality table understanding as a controlled evaluation problem, where table content is fixed while structural format and input modality vary under matched pipelines. • We introduce TAB V ERSE, an aligned multimodal table benchmark with HTML, LaTeX, and Markdown representations, corresponding rendered images, category and difficulty tags, and a 700-sample balanced evaluation set from five TableQA sources. • We benchmark LLMs and VLMs across matched text-only and image-based table inputs on QA, SUC, and SR, revealing how format and modality choices change model behavior across tasks and question groups, and how SR errors separate into table reconstruction quality and output usability. Our experiments show that representation matters. Structured text often outperforms rendered images, HTML is often the most robust text format, and usable LaTeX reconstruction remains challenging.
2
Related Work
Table 1 summarizes related resources, which cover many table reasoning tasks but rarely isolate representation effects because content, format, layout, and modality often vary together. Table reasoning benchmarks: Early benchmarks established table QA over semi-structured tables, covering lookup, filtering, aggregation, and simple symbolic operations (Pasupat and Liang, 2015; Zhong et al., 2017). Later datasets expanded to sequential QA, fact verification, multi-hop reasoning over tables and text, open-domain and multitable QA, and table-grounded generation (Iyyer et al., 2017; Chen et al., 2020a,b; Aly et al., 2021; Chen et al., 2021a; Wu et al., 2025a; Parikh et al., 2020; Nan et al., 2022). Other resources target numerical reasoning (Chen et al., 2021b; Zhu et al., 2021), hierarchical tables (Cheng et al., 2022), long-context cell retrieval (Wang et al., 2026), and complex or multilingual table understanding (Zhu et al., 2025). These benchmarks provide important testbeds, but most evaluate a fixed representation or task setting; TAB V ERSE instead tests the same table-question pairs across aligned textual formats and rendered images.
Representation and multimodal table evaluation: Prior work shows that table reasoning depends strongly on serialization, prompting, segmentation, and modality. Table Meets LLM and TAP4LLM study prompting, sampling, augmentation, and structural decomposition (Sui et al., 2024a,b), while tables-as-text-versus-image comparisons show that representation choice can substantially change performance (Deng et al., 2024). LongTableBench and RealHiTBench evaluate long or hierarchical tables under multiple input formats (Li et al., 2025; Wu et al., 2025b); related studies examine source-sensitive table understanding, table-image modeling, and cross-domain evaluation behavior (Yang et al., 2025; Chen et al., 2023; Borisova et al., 2025); and multimodal benchmarks cover visual QA, semi-structured tables, rendered table images, and table-image retrieval (Kim et al., 2024; Singh et al., 2025; Mathur et al., 2024; Zheng et al., 2024; Titiya et al., 2025; Talmor et al., 2021; Lompo and Haraoui, 2025; Li et al., 2026; Xu et al., 2026). These works motivate format-aware and image-based table evaluation, but often vary table source, layout, visual complexity, and representation together across different experimental settings; TAB V ERSE holds table content fixed while systematically varying structural format and input modality. Table reconstruction and table-focused modeling: Beyond QA, table reconstruction and tablestructure recognition studies extract structured representations from rendered tables or document images (Roberts et al., 2025; Li et al., 2020). This is related to our SR task, but prior work usually focuses on recognition accuracy or one target representation rather than reconstruction across aligned input and output formats. Table-specialized pretraining and instruction tuning have been proposed for table manipulation, reasoning, and generation (Herzig et al., 2020; Gong et al., 2020; Zhang et al., 2024; Zha et al., 2023; Li et al., 2024; Su et al., 2024; Zhang et al., 2025; Deng and Mihalcea, 2025); recent systems also combine OCR-style transcription with LLM reasoning for table VQA (Guo et al., 2025) or add supervision through code-driven reasoning traces and structureaware guidance (Nguyen and Okatani, 2026; Zhu et al., 2026). These efforts are complementary to TAB V ERSE, which evaluates LLMs and VLMs under controlled QA, SUC, and SR settings across aligned textual and visual table representations.
Representations Literature
Text
Images
Visual Renders HTML LaTeX Markdown Others Q-diff Q-cat Aligned
Table Reasoning and Multimodal Benchmarks TableVQA-Bench (Kim et al., 2024) ✓ MTabVQA (Singh et al., 2025) ✗ MMTabQA (Mathur et al., 2024) ✓ MMTabQA (Mathur et al., 2024) ✗ NeedleInATable (Wang et al., 2026) ✓ TableVLM (Chen et al., 2023) ✓
✓ ✓ ✓ ✓ ✓ ✓
✓ ✗ ✗ ✗ ✗ ✓
✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗
✓ ✓ ✓ ✓ ✓ ✗
✗ ✗ ✗ ✗ ✗ ✗
✗ ✓ ✗ ✓ ✗ ✗
✓ ✗ ✗ ✗ ✓ ✓
Evaluation Frameworks / Controlled Studies Tables as Texts or Images (Deng et al., 2024) ✓ RealHiTBench (Wu et al., 2025b) ✓ LongTableBench (Li et al., 2025) ✓ Image2Struct (Roberts et al., 2025) ✗
✓ ✓ ✗ ✓
✗ ✗ ✗ ✓
✗ ✗ ✗ ✓
✗ ✗ ✗ ✗
✓ ✓ ✗ ✗
✗ ✗ ✗ ✗
✗ ✓ ✗ ✗
✓ ✓ ✗ ✗
✓
✓
✓
✓
✓
-
✓
✓
✓
TAB V ERSE (Ours)
Table 1: Table understanding resources related to TAB V ERSE. Under Visual Renders, HTML, LaTeX, and Markdown indicate whether the work provides or evaluates tables rendered from those source formats, while Others covers other visual styles or image sources. Q-diff and Q-cat denote question difficulty and category annotations. Aligned denotes paired textual tables and images with the same table content.
3
TAB V ERSE
Difficulty is estimated from zero-shot QA on rendered table images. GPT-5.2 and Gemini-3Flash-Preview answer each question using the three aligned image renders produced from HTML, Markdown, and LaTeX, giving six correctness indicators per question. Questions scoring 0–3 are labeled Hard; those scoring 4–6 are labeled Easy.
TAB V ERSE is constructed from filtered held-out splits of five TableQA datasets: FEVEROUS (Aly et al., 2021), TAB FACT (Chen et al., 2020a), SQA (Iyyer et al., 2017), H YBRID QA (Chen et al., 2020b), and WikiTableQuestions (WIKITQ) (Pasupat and Liang, 2015). We keep only singletable questions answerable from the table alone and check for overlap with the corresponding training splits where identifiers are available (Appendix B). This yields a full pool of 6,097 question–table pairs from 4,434 unique tables. After category and difficulty tagging, we select a 700-sample balanced evaluation set covering 629 unique tables (Table 2).
We balance the final evaluation set by difficulty and question category. It contains 700 question–table pairs: 350 Easy and 350 Hard questions, with 50 examples per category within each difficulty level. These questions reference 629 unique tables.
3.1
3.4
Aligned Formats and Rendered Images
For each table, we create HTML, Markdown, and LaTeX representations and render one image from each. We adapt conversion utilities from Sui et al. (2024a) and extend them for dataset artifacts such as missing values and special characters. Standardized font size, padding, and width keep textual and visual versions aligned by construction. Implementation details are in Appendix B. 3.2
Question Category and Difficulty
Each question is tagged with a question category and a binary difficulty label. The seven categories are Simple Lookup, Conditional Lookup, Multiitem Lookup, Aggregation/Arithmetic, Comparison/Extremum, and Binary Verification (singlestep and multi-hop). Gemini-3-Flash-Preview assigns the initial category tags, which are manually reviewed and corrected where needed.
3.3
Balanced Evaluation Set
Supported Tasks
TAB V ERSE supports three tasks: QuestionAnswering (QA), Structural Understanding Capability (SUC), and Structure Reconstruction (SR). QA: Given a question and a table as structured text or a rendered image, the model predicts an answer following the source dataset conventions. SUC: SUC probes table structure through boundary detection, size estimation, and index-based retrieval. We adapt the templates from Sui et al. (2024a) and extend them with probes for table grounding and document-style tables. Prompt templates appear in Appendix C. SR: Given a rendered table image, the model reconstructs the table in HTML, Markdown, or LaTeX.
DATASET
(FEVEROUS, HybridQA, SQA, Tabfact, Wikitq)
Structural formats
RAW (JSON) Complexity Tagging
(HTML/Markdown/LaTeX) 629 unique files for each format
Task Difficulty Tagging
Balanced Data (JSON) 700 QA pairs
LLM Pipeline
{Input: Prompt + Text}
VLM+Text Pipeline {Input: Prompt + Text}
Rendering
Aligned PNG images 629 images for each format
VLM Pipeline
{Input: Prompt + Image}
SUC
629 × 3 formats (structural files & images)
QA
700 × 3 formats (structural files & images)
SR
629 × 3 formats (images)
SUC_GT
QA_GT SR_GT
Figure 1: Overview of TAB V ERSE: From the balanced evaluation set, each table is represented in three structural formats (HTML, Markdown, LaTeX) with corresponding rendered images. These aligned multimodal pairs enable evaluation on QA, SUC, and SR tasks across VLMs and LLMs for cross-format and cross-modality analysis.
Split
Question–table pairs Unique tables
Full tagged pool Balanced set
6,097 700
4,434 629
Table 2: Dataset statistics for TAB V ERSE. The full tagged pool contains all filtered question-table pairs after category and difficulty tagging. The balanced evaluation set is used for all experiments, with equal coverage across difficulty and question category.
4
Experimental Settings
We evaluate how structural format and input modality affect table understanding while keeping table content fixed. Using TAB V ERSE, we vary the table format (HTML, Markdown, LaTeX) and input modality (structured text vs. rendered image), allowing performance differences to be attributed to representation rather than content variation. 4.1
Evaluation Pipelines
We evaluate LLMs and VLMs on TAB V ERSE using three different pipelines: VLM-Image: The VLM receives each question with a rendered table image. To measure visual format effects, we use three aligned image renderings per instance, rendered from HTML, Markdown, and LaTeX sources, while keeping the question and table content unchanged. VLM-Text: The VLM receives the question prompt with a structured text-based table in one of the three formats, without an image. Comparing VLM-Text with VLM-Image isolates the impact of visual input within the same model. LLM-Text: The LLM receives the question prompt with a structured text-based table in one of the three formats. Comparing LLM-Text with VLM-Text highlights differences between language-only and multimodal models on identical text inputs.
4.2
Models
We evaluate several LLMs and VLMs, including general-purpose and table-specialized models. For LLMs, we use Qwen2.5-7B-Instruct (Yang et al., 2024), Qwen3-30B-A3B-Instruct (Team, 2025b), TableGPT2-7B1 (Su et al., 2024), and TAMA-QWen31 (Xing et al., 2025). For VLMs, we use SmolVLM2-2.2B-Instruct (Marafioti et al., 2025), Gemma-3-12B-IT, Gemma-3-27BIT (Team, 2025a), InternVL3.5-14B, InternVL3.530B-A3B (Wang et al., 2025), Qwen3-VL-8BInstruct, Qwen3-VL-30B-A3B-Instruct (Team, 2025b), Ministral-3-14B-Instruct (Liu et al., 2026), LLaVA-1.6-7B and LLaVA-1.6-13B (Liu et al., 2024), TableLLaVA-v1.5-7B1 (Zheng et al., 2024), GPT-5.2 (OpenAI, 2025), and Gemini-3-FlashPreview (Google, 2024). 4.3
Evaluation Protocol
All experiments follow a uniform zero-shot setup across the three tasks (Appendix C). Models generate outputs using greedy decoding (temperature=0, top_p=1) with task-specific output limits: short for SUC, medium for QA, and long for SR. We apply minimal output normalization, including removing common answer prefixes such as the answer is and normalizing whitespace and casing where appropriate. For QA and SUC, we report Exact-Match (EM) accuracy following the dataset conventions after light normalization. For SUC, we additionally report Field Accuracy and Relaxed Accuracy for pipe-separated structured answers. Field Accuracy compares each gold field with the prediction at the same position, while Relaxed Accuracy checks whether each gold field appears anywhere in the prediction. These are diagnostic metrics; exact match remains the primary SUC metric. 1
Table-specialized model for table understanding.
Table Image Render Model
Table Text Format
Params. HTML LaTeX Markdown HTML LaTeX Markdown
Language Models (text-only) Qwen2.5-IT (Yang et al., 2024) 7B Qwen3-IT (Team, 2025b) 30B (A3B) TableGPT2 (Su et al., 2024) 7B TAMA-QWen3 (Xing et al., 2025) – Vision-Language Models SmolVLM2-IT (Marafioti et al., 2025) 2.2B Gemma-3-IT (Team, 2025a) 12B Gemma-3-IT (Team, 2025a) 27B InternVL3.5 (Wang et al., 2025) 14B InternVL3.5 (Wang et al., 2025) 30B (A3B) Qwen3-VL-IT (Bai et al., 2023) 8B Qwen3-VL-IT (Bai et al., 2023) 30B (A3B) Ministral-3-IT (Liu et al., 2026) 14B LLaVA-1.6∗ (Liu et al., 2023) 7B ∗ LLaVA-1.6 (Liu et al., 2023) 13B ∗ TableLLaVA-v1.5 (Zheng et al., 2024) 7B Proprietary Models GPT-5.2 (OpenAI, 2025) – Gemini-3-Flash-Preview (Google, 2024) –
– – – –
– – – –
– – – –
44.57 51.14 44.43 18.29
42.71 48.43 41.57 19.14
45.43 46.57 42.14 20.71
29.71 38.86 46.14 48.57 47.86 50.29 41.14 44.43 31.86 25.14 1.29
28.71 39.57 45.29 48.14 50.00 49.29 42.14 39.14 31.43 23.71 1.00
25.86 38.57 45.43 48.00 48.29 49.71 41.43 42.71 32.00 25.00 4.00
21.57 50.29 53.43 47.14 45.86 53.43 45.29 40.00 27.37 23.91 23.61
17.63 49.00 51.29 47.29 45.71 52.14 43.57 35.43 29.55 22.13 27.37
15.75 48.57 53.14 44.86 47.00 53.29 39.71 36.57 26.50 24.89 28.40
54.57 65.43
54.52 65.14
56.14 65.43
57.43 65.71
57.29 65.00
58.00 65.43
Table 3: TaskQA results (700 questions). EM accuracy (%) across three aligned table representations (HTML/LaTeX/Markdown). Text-only models (LLM-Text) operate on structured table text (right block). Vision-language models are evaluated both on rendered table images (left block) and on structured table text (right block), which isolates modality effects while keeping the underlying table content identical.
For SR, we report G RI TS (Smock et al., 2023) using both GriTS-Topology and GriTS-Content. GriTS-Topology measures structural similarity between the reconstructed and reference tables, while GriTS-Content measures cell-text fidelity. We also measure syntactic usability for each requested target format using HTML parse success, Markdown render/parse success, and LaTeX compilation success. To connect usability with reconstruction quality, Appendix D.3 reports two usability-aware variants of G RI TS. Valid-only G RI TS averages scores only over syntactically usable outputs when at least one output is usable: PN GriTSvalid =
i=1 ui · GriTSi PN i=1 ui
(1)
where ui = 1 if the generated output is syntactically usable and ui = 0 otherwise. Zero-penalized G RI TS instead assigns unusable outputs a score of zero before averaging: N
GriTS0 =
1 X ui · GriTSi N
(2)
i=1
5
Results
This section evaluates model performance across different table formats, input modalities, and table comprehension tasks.
5.1
Question Answering
Table 3 reports EM accuracy on 700 QA questions across three aligned table representations. Overall performance: Gemini-3-Flash-Preview obtains the highest scores across all formats and modalities, followed by GPT-5.2. Among openweight VLMs, Qwen3-VL-8B-IT is strongest under strict EM, while Qwen3-30B-A3B-IT is the strongest text-only LLM. Larger models are not always better under strict EM: the 8B Qwen3-VL variant outperforms the 30B-A3B variant across modalities and formats in Table 3. However, Appendix 10 shows that the 30B-A3B variant recovers substantially under relaxed matching, especially in VLM-Text. This suggests that part of the strict EM gap comes from answer-formatting behavior rather than answer retrieval alone. InternVL3.5 shows a milder version of the same pattern. Modality and format effects: Structured table text does not consistently outperform rendered images. Gemma-3 benefits from text inputs, SmolVLM2 and InternVL3.5 perform similarly or better on images, and Gemini-3-Flash-Preview remains nearly unchanged across modalities. At the format level, rendered-image scores are generally similar across HTML, LaTeX, and Markdown, while text pipelines show larger gaps, suggesting that symbolic table format affects text-based reasoning more than rendered-image reasoning.
Models
Formats
T.P. F.C. L.C. S.D. # Rows # Cols C.Lu. R.Lu. Co.Rt. Ro.Rt. Overall
Open Models HTML LaTeX Markdown HTML InternVL3.5-14B LaTeX Markdown HTML Qwen3-VL-30B-A3B-IT LaTeX Markdown HTML Ministral-3-14B-IT LaTeX Markdown HTML LLaVA-1.6-13B LaTeX Markdown Proprietary Models HTML GPT-5.2 LaTeX Markdown HTML Gemini-3-Flash-Preview LaTeX Markdown Gemma-3-27B-IT
29.6 28.3 30.2 35.3 38.0 31.6 50.2 45.6 40.1 36.1 36.1 30.7 0.2 0.2 0.5
63.9 59.6 59.0 86.8 88.6 83.0 92.5 92.7 87.9 40.5 41.8 34.3 31.5 32.9 35.6
52.3 52.0 51.5 68.4 71.9 65.3 85.2 82.2 78.7 61.5 53.3 49.4 27.5 29.6 31.6
21.0 23.5 27.3 22.3 16.4 32.3 32.9 34.2 32.0 22.7 38.2 38.5 4.5 3.8 4.3
39.9 39.7 44.4 55.3 28.6 51.0 44.2 40.2 41.7 35.1 35.6 35.6 19.4 20.0 19.1
50.1 50.2 55.8 89.8 83.1 86.6 91.9 87.4 87.4 46.4 36.1 52.5 28.0 20.5 22.6
22.6 22.7 24.2 15.6 18.9 20.3 21.1 22.3 21.6 26.6 26.9 26.6 1.4 2.2 2.1
32.6 32.6 33.7 15.9 22.7 23.5 24.3 27.3 25.1 39.6 39.4 38.8 6.4 5.6 6.2
70.9 70.3 74.2 84.6 90.1 88.9 71.1 83.8 88.9 79.7 72.7 74.2 37.5 36.7 36.6
10.5 9.1 11.3 4.1 6.4 5.4 0.2 4.0 1.1 10.7 11.1 10.7 0.0 0.2 0.0
39.3 38.8 41.2 47.8 46.5 48.8 51.4 52.0 50.4 39.9 39.1 39.1 15.6 15.2 15.9
93.0 85.4 87.4 91.7 86.1 87.9
97.1 93.3 93.6 97.0 94.8 94.8
94.9 32.9 88.1 57.7 91.6 80.3 88.7 0.2 87.4 0.0 87.7 0.2
78.4 85.7 91.3 0.5 6.5 1.9
98.1 96.5 98.9 94.9 99.2 99.7
2.7 3.5 3.0 0.6 1.6 1.3
15.7 19.2 20.8 14.8 15.1 14.9
95.2 93.5 96.0 94.4 95.7 97.3
1.7 2.5 5.1 0.3 0.8 0.3
61.0 62.5 66.8 48.3 48.7 48.6
Table 4: SUC results for the VLM-Image pipeline. EM accuracy (%) across ten structure-oriented subtasks, including table partitioning (T.P.), boundary detection (F.C., L.C.), size estimation (S.D., #Rows, #Cols), coordinate lookup (C.Lu., R.Lu.), and index-based retrieval (Co.Rt., Ro.Rt.). Models receive rendered table images derived from HTML, LaTeX, and Markdown sources.
∆ (Field Acc. - EM)
∆EM (Explicit – Implicit)
Model
T.P.
S.D.
C.Lu. Ro.Rt.
Model
Gemma-3-27B-IT InternVL3.5-14B Qwen3-VL-30B-A3B-IT Ministral-3-14B-IT LLaVA-1.6-13B GPT-5.2 Gemini-3-Flash-Preview
+25.7 +30.8 +28.3 +18.2 +6.2 +4.9 +7.3
+21.3 +29.5 +30.1 +25.5 +17.2 +21.0 +48.9
+20.9 +30.2 +27.0 +20.2 +7.8 +42.1 +35.2
Gemma-3-27B-IT +13.2 +28.4 +22.0 +7.0 +9.7 InternVL3.5-14B +17.0 +65.8 +18.0 +15.9 +26.7 Qwen3-VL-30B∗ +33.1 +79.6 +21.7 +9.8 +5.1 Ministral-3-14B-IT -8.2 +6.6 +24.4 +23.6 +0.3 LLaVA-1.6-13B +0.2 -21.1 +1.6 -0.7 +3.2 GPT-5.2 +80.5 +83.8 +1.4 -52.7 +16.8 ∗ Gemini +88.3 +94.8 +0.7 -15.4 -0.7
+13.7 +11.8 +20.6 +11.7 +3.6 +13.1 +13.4
Table 5: Field-level gaps on selected SUC subtasks. Values report ∆ = Field Accuracy − EM, averaged over HTML, LaTeX, and Markdown image renders. Larger gaps indicate subtasks where models often recover part of the structured answer but fail exact match.
Task factors: Beyond modality and format, QA performance varies by question category and difficulty. Verification questions tend to be easier, while multi-item lookup and aggregation/counting questions remain challenging, as illustrated by the category-wise profiles in Figure 8. Accuracy also drops substantially from Easy to Hard questions. We provide the full category and difficulty breakdowns in Appendix D.1. 5.2
Structural Understanding Capability
Tables 4–6 evaluate SUC across ten structureoriented subtasks defined in Section 3.4. Tables 4 and 11 report VLM-Image EM results, while Tables 13 and 14 report VLM-Text and LLM-Text results.
T.P. F.C. C.Lu. R.Lu. Co.Rt. Ro.Rt. +4.1 +1.7 +0.0 +3.6 +0.1 -1.9 +0.5
Table 6: Effect of prompt explicitness on SUC performance. Values report ∆EM = EMexplicit − EMimplicit for VLMs, averaged across HTML, LaTeX, and Markdown renders. Positive values favor explicit prompts, while negative values favor implicit prompts. Qwen3-VL-30B∗ denotes Qwen3-VL-30B-A3B-IT, and Gemini∗ denotes Gemini-3-Flash-Preview.
Overall performance: GPT-5.2 obtains the highest overall VLM-Image scores, while Qwen3-VL30B and InternVL3.5-14B are the strongest openweight VLMs in Table 4. Gemini-3-Flash performs well on boundary and column-oriented subtasks but struggles with size detection, cell lookup, and row retrieval, making SUC performance highly taskdependent. Column counting and column retrieval are among the easiest subtasks, whereas row retrieval, cell lookup, table partitioning, and size detection remain difficult, as shown in Figure 11. This gap is most visible in row-oriented reasoning, where even strong models fail to retrieve the correct indexed row.
Format sensitivity is modest in the VLM-Image setting. For most models, overall scores vary by only a few points across HTML, LaTeX, and Markdown renders, suggesting that visual table structure largely dominates source-format differences after rendering. GPT-5.2 is the main exception, improving from 61.0 on HTML to 66.8 on Markdown, driven mainly by stronger size detection and rowcount estimation. Boundary-related subtasks (F.C., L.C.) are generally solved reliably, with GPT-5.2, Gemini-3-Flash, and Qwen3-VL-30B exceeding 85% on most formats, whereas coordinate-based reasoning remains much harder. Row retrieval stays below 12% for all open models and below 6% even for the strongest proprietary models, highlighting a persistent gap between recognizing table structure and accurately navigating row-level coordinates. Field Accuracy reveals partial structural recovery: Field Accuracy shows that low EM often reflects incomplete or shifted localization rather than entirely wrong table understanding. In Table 5, the largest gains occur on row/column-sensitive subtasks: InternVL3.5-14B gains +30.8 on table partitioning and +30.2 on cell lookup, while Qwen3VL-30B gains +30.1 on size detection. These gaps are especially informative for row-sensitive outputs, where models often recover part of the structure but miss exact localization. Header and indexing cues affect SUC scores: Table 6 compares the explicit SUC prompt, which states header exclusion and 0-indexed row/column coordinates, with an implicit prompt that removes these details. The effect is strongest on indexdependent subtasks. For first-cell detection, the explicit prompt gives much higher EM for GPT-5.2 (+83.8) and Gemini-3-Flash (+94.8), with similar gains for the strongest open models; table partitioning shows the same trend for GPT-5.2 (+80.5) and Gemini-3-Flash (+88.3). The effect is much weaker for last-cell detection, suggesting that many firstcell errors come from treating the header as the first row. In contrast, reverse lookup improves under the implicit prompt for GPT-5.2 (-52.7) and Gemini-3Flash (-15.4). Thus, SUC also tests whether models follow the intended row-indexing and headerinclusion convention. Pipeline and format effects: Tables 4, 11, 13, and 14 show that SUC is generally stronger with structured table text than with rendered images.
The gains are clearest for GPT-5.2 and Gemini3-Flash, while open-weight models improve less uniformly. The largest image-to-text improvements occur on row-boundary and header-sensitive tasks. Row retrieval and cell lookup remain the primary bottlenecks across pipelines, despite strong column counting and retrieval, though several VLMs improve on these tasks in the VLM-Text setting (Figure 12). Table 14 shows that text alone does not solve SUC for LLMs. Text-only Qwen3-30B performs strongly on HTML but drops on LaTeX and Markdown, while smaller and table-specialized LLMs remain weak on coordinate lookup and row retrieval, indicating format effects that are present but not universal. HTML is often the safest text format for text-input pipelines, while rendered-image results show smaller and less consistent format differences. Overall, SUC depends on input modality, table format, and row/column indexing behavior. 5.3
Structure Reconstruction
Table 7 evaluates SR, where models reconstruct rendered table images into a given text format. We report G RI TS-Topology, G RI TS-Content, and output usability (Tables 8 and 16). Appendix D.3 also reports valid-only and zero-penalized G RI TS to distinguish usability failures from reconstruction errors. SR errors reflect both structure and content: Across models and formats, G RI TS-Topology consistently exceeds G RI TS-Content, showing that models recover table layout more reliably than exact cell text. Strong VLMs such as Qwen3, InternVL3.5, and GPT-5.2 obtain high topology scores across most source and target formats, but content scores drop more often, especially for LaTeX targets. This indicates that SR failures stem from both structural errors and cell-text degradation during reconstruction. Usability exposes syntax-level failures: Tables 8 and 16 show that high G RI TS does not always imply usable output. Strong open VLMs usually produce valid HTML and Markdown, but LaTeX usability is less stable. For example, Qwen3-VL-30B reaches perfect usability for HTML and Markdown targets, while its LaTeX usability ranges from 0.77 to 0.95. This separates unusable syntax from usable but inaccurate reconstructions. We therefore report zero-penalized G RI TS in Table 15, where unusable outputs receive zero before averaging.
HTML image
Markdown image
LaTeX image
GriTS-Topology GriTS-Content GriTS-Topology GriTS-Content GriTS-Topology GriTS-Content Models
HTML Md TeX HTML Md TeX HTML Md TeX HTML Md TeX HTML Md TeX HTML Md TeX
Open Models SmolVLM2-2.2B Gemma3-12B Gemma3-27B InternVL3.5-14B InternVL3.5-30B Qwen3-VL-8B Qwen3-VL-30B Ministral3-14B LLaVA1.6-7B LLaVA1.6-13B
0.87 0.94 0.97 0.99 0.98 0.99 0.98 0.98 0.70 0.66
0.86 0.81 0.95 0.91 0.97 0.94 0.99 0.96 0.99 0.95 1.00 0.95 0.99 0.84 0.95 0.95 0.66 0.05 0.81 0.23
0.74 0.79 0.86 0.95 0.95 0.98 0.98 0.94 0.43 0.46
0.74 0.69 0.79 0.76 0.85 0.82 0.94 0.92 0.94 0.92 0.98 0.92 0.98 0.80 0.90 0.92 0.40 0.02 0.51 0.15
0.66 0.95 0.98 0.99 0.99 0.99 0.99 0.98 0.70 0.62
0.79 0.78 0.96 0.92 0.97 0.94 0.98 0.96 0.99 0.96 1.00 0.97 1.00 0.97 0.95 0.95 0.54 0.04 0.75 0.22
0.57 0.80 0.85 0.93 0.93 0.97 0.97 0.92 0.46 0.45
0.68 0.68 0.81 0.77 0.85 0.81 0.93 0.91 0.94 0.92 0.98 0.93 0.98 0.93 0.90 0.90 0.35 0.02 0.51 0.15
0.79 0.94 0.97 0.96 0.96 0.98 0.97 0.95 0.71 0.65
0.85 0.57 0.95 0.90 0.96 0.94 0.97 0.95 0.97 0.95 0.98 0.97 0.98 0.96 0.93 0.93 0.58 0.05 0.76 0.27
0.65 0.78 0.83 0.93 0.93 0.95 0.95 0.87 0.44 0.46
0.69 0.47 0.77 0.75 0.82 0.81 0.91 0.93 0.92 0.92 0.95 0.95 0.95 0.95 0.86 0.88 0.37 0.03 0.49 0.19
Proprietary Models GPT-5.2 0.98 0.98 0.81 0.97 0.94 0.78 0.98 0.99 0.95 0.96 0.97 0.91 0.98 0.97 0.93 0.94 0.94 0.89 Gemini-3-Flash 0.05 0.96 0.65 0.05 0.94 0.64 0.86 0.97 0.51 0.85 0.96 0.51 0.65 0.93 0.58 0.63 0.91 0.57 Table-specialised Models TableLLaVA-7B 0.73 0.71 0.68 0.33 0.33 0.31 0.73 0.72 0.70 0.43 0.43 0.41 0.58 0.57 0.54 0.29 0.29 0.28
Table 7: SR from table images. Models reconstruct table images rendered from HTML, Markdown, or LaTeX into HTML, Markdown, or LaTeX. We report GriTS-Topology (structure) and GriTS-Content (cell text); higher scores indicate better reconstruction. Best scores per column are shown in bold.
HTML image Markdown image LaTeX image Models
HTML Md TeX HTML Md TeX HTML Md TeX
Open Models Gemma3-27B-IT Qwen3-VL-30B-A3B-IT LLaVA1.6-Vicuna-13B TableLLaVA-v1.5-7B
1.00 1.00 0.99 1.00
1.00 0.89 1.00 0.77 0.90 0.00 0.96 0.71
1.00 1.00 0.93 1.00
1.00 0.88 1.00 0.89 0.84 0.00 0.97 0.74
Proprietary Models GPT-5.2 0.99 1.00 0.94 0.99 1.00 0.96 Gemini-3-Flash-Preview 0.06 0.99 0.76 0.91 1.00 0.80
1.00 1.00 0.99 1.00
1.00 0.87 1.00 0.95 0.90 0.01 0.98 0.80
1.00 1.00 0.98 0.72 0.99 0.75
Table 8: Output usability for SR. Values report the fraction of syntactically usable reconstructed outputs across source render formats and target output formats. Best scores per column are shown in bold.
LaTeX and content preservation remain the main bottlenecks: LaTeX reconstruction is difficult in two ways: models must recover the table structure and also produce compilable syntax. This is most visible for weaker VLMs such as LLaVA variants, where LaTeX usability is near zero despite non-trivial HTML or Markdown usability. Even strong models show a consistent drop on LaTeX targets, with topology and content scores often lower than corresponding HTML or Markdown outputs. Among open models, Qwen3-VL-8B is highly competitive and often exceeds larger models on both topology and content, while TableLLaVA remains much weaker. Overall, modern VLMs can recover table topology reliably from images, but exact content preservation and syntactically usable LaTeX generation remain challenging.
6
Conclusion and Future Work
We introduce TAB V ERSE, a controlled multimodal table benchmark for studying representation effects in table understanding. TAB V ERSE provides aligned HTML, Markdown, and LaTeX representations with corresponding rendered images, category and difficulty tags, and a 700-sample balanced evaluation set drawn from five TableQA sources. This setup supports matched evaluation of QA, SUC, and SR across text and image inputs while keeping table content fixed. Our results show that table representation strongly affects model behavior. Structured text often outperforms rendered images, especially for structure-sensitive tasks, while HTML is often the most robust format for text inputs and LaTeX remains challenging. In QA, verification questions are easier than Multi-Item Lookup and Aggregation/Arithmetic. In SUC, models handle column-oriented subtasks better than row and cell indexing, and header/indexing conventions can substantially shift answers. In SR, strong VLMs recover broad table layout reliably, but exact cell content and syntactically usable LaTeX generation remain difficult. Overall, scaling alone does not guarantee better table understanding. Future work can extend TAB V ERSE to more realistic settings, including PDFs with noisy layouts and multiple tables, and explore methods for improving crossformat consistency and representation-aware decoding.
Limitations TABVERSE targets controlled cross-format and cross-modality evaluation, so we make a few scope choices. We build the benchmark from five established English TableQA sources (FEVEROUS, H YBRID QA, SQA, TAB FACT, WIKITQ) and focus on single-table questions answerable from the table alone; extending coverage to additional languages, scripts, and document-level settings is a natural next step. For VLM-Image, we render table images from clean markup under a standardized layout. This keeps the image inputs aligned across HTML, Markdown, and LaTeX and helps us isolate representation effects, but it does not capture noise from scanned or photographed documents.
Ethical Statement and Broad Impact TAB V ERSE is constructed from publicly available datasets: FEVEROUS, H YBRID QA, SQA, TAB FACT, and WIKITQ. The benchmark is intended for academic research on multimodal and crossformat table understanding. We use released tablequestion pairs and do not add any private or proprietary data. Because TAB V ERSE builds on existing datasets, it may inherit domain, linguistic, or annotation biases present in the original sources. TAB V ERSE aims to support transparent and reproducible evaluation of LLMs and VLMs, and is not designed for commercial deployment or decisionmaking in sensitive domains.
References Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. Preprint, arXiv:2308.12966. Ekaterina Borisova, Fabio Barth, Nils Feldhus, Raia Abu Ahmad, Malte Ostendorff, Pedro Ortiz Suarez, Georg Rehm, and Sebastian Möller. 2025. Table understanding and (multimodal) LLMs: A crossdomain case study on scientific vs. non-scientific data. In Proceedings of the 4th Table Representation
Learning Workshop, pages 109–142, Vienna, Austria. Association for Computational Linguistics. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. Preprint, arXiv:2303.12712. Leiyuan Chen, Chengsong Huang, Xiaoqing Zheng, Jinshu Lin, and Xuanjing Huang. 2023. TableVLM: Multi-modal pre-training for table structure recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2437–2449, Toronto, Canada. Association for Computational Linguistics. Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021a. Open question answering over tables and text. In International Conference on Learning Representations. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020a. TabFact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations, ICLR’20. Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020b. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics, EMNLP’20, pages 1026–1036, Online. Association for Computational Linguistics. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021b. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022. HiTab: A Hierarchical Table dataset for question answering and natural language
generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1094–1110, Dublin, Ireland. Association for Computational Linguistics. Naihao Deng and Rada Mihalcea. 2025. Rethinking table instruction tuning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21757–21780, Vienna, Austria. Association for Computational Linguistics. Naihao Deng, Zhenjie Sun, Ruiqi He, Aman Sikka, Yulong Chen, Lin Ma, Yue Zhang, and Rada Mihalcea. 2024. Tables as texts or images: Evaluating the table reasoning ability of LLMs and MLLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 407–426, Bangkok, Thailand. Association for Computational Linguistics. Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu, and Ting Liu. 2020. TableGPT: Few-shot table-to-text generation with table structure reconstruction and content matching. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1978–1988, Barcelona, Spain (Online). International Committee on Computational Linguistics. Google. 2024. Gemini 3 Flash model card. Google. Yutong Guo, Wanying Wang, Yue Wu, Zichen Miao, and Haoyu Wang. 2025. TALENT: Table VQA via augmented language-enhanced natural-text transcription. In 2025 IEEE International Conference on Data Mining Workshops (ICDMW), pages 1404– 1410. IEEE. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320–4333, Online. Association for Computational Linguistics. Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for Sequential Question Answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1821– 1831, Vancouver, Canada. Association for Computational Linguistics. Yoonsik Kim, Moonbin Yim, and Ka Yeon Song. 2024. TableVQA-Bench: A visual question answering benchmark on multiple table domains. ArXiv preprint, abs/2404.19205. Da Li, Keping Bi, Jiafeng Guo, Wei Yuan, Fan Yang, Tingting Gao, and Xueqi Cheng. 2026. Beyond textonly: Towards multimodal table retrieval in openworld. In The Fourteenth International Conference on Learning Representations. Liyao Li, Jiaming Tian, Hao Chen, Wentao Ye, Chao Ye, Haobo Wang, Ningtao Wang, Xing Fu, Gang
Chen, and Junbo Zhao. 2025. LongTableBench: Benchmarking long-context table reasoning across real-world formats and domains. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11927–11965, Suzhou, China. Association for Computational Linguistics. Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, and Zhoujun Li. 2020. TableBank: Table benchmark for image-based table detection and recognition. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1918– 1925, Marseille, France. European Language Resources Association. Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. 2024. Table-GPT: Table fine-tuned GPT for diverse table tasks. Proc. ACM Manag. Data, 2(3). Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Sooriyarachchi, Baptiste Bout, and 101 others. 2026. Ministral 3. Preprint, arXiv:2601.08584. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286–26296. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Thirtyseventh Conference on Neural Information Processing Systems. Boammani Aser Lompo and Marc Haraoui. 2025. Visual-TableQA: Open-domain benchmark for reasoning over table images. Preprint, arXiv:2509.07966. Andrés Marafioti, Orr Zohar, Miquel Farré, Merve noyan, Elie Bakouch, Pedro Manuel Cuenca Jiménez, Cyril Zakka, Loubna Ben allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. 2025. SmolVLM: Redefining small and efficient multimodal models. In Second Conference on Language Modeling. Suyash Vardhan Mathur, Jainit Sushil Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, and Dan Roth. 2024. Knowledge-aware reasoning over multimodal semistructured tables. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14054–14073, Miami, Florida, USA. Association for Computational Linguistics.
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, and Dragomir Radev. 2022. FeTaQA: Freeform table question answering. Transactions of the Association for Computational Linguistics, 10:35–49. Van-Quang Nguyen and Takayuki Okatani. 2026. CoReTab: Improving multimodal table understanding with code-driven reasoning. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6498–6523, Rabat, Morocco. Association for Computational Linguistics. OpenAI. 2025. GPT-5 system card. OpenAI. Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP’20, pages 1173–1186, Online. Association for Computational Linguistics. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470– 1480, Beijing, China. Association for Computational Linguistics. Josselin Somerville Roberts, Tony Lee, Chi Heem Wong, Michihiro Yasunaga, Yifan Mai, and Percy Liang. 2025. Image2Struct: Benchmarking structure extraction for vision-language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. Anshul Singh, Chris Biemann, and Jan Strich. 2025. MTabVQA: Evaluating multi-tabular reasoning of language models in visual space. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 19866–19891, Suzhou, China. Association for Computational Linguistics. Ananya Singha, José Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. 2023. Tabular representation, noisy operators, and impacts on table structure understanding tasks in LLMs. In Proceedings of the Table Representation Learning Workshop at Neural Information Processing Systems (NeurIPS) 2023. Brandon Smock, Rohith Pesala, and Robin Abraham. 2022. PubTables-1M: Towards comprehensive table extraction from unstructured documents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4634–4642.
Brandon Smock, Rohith Pesala, and Robin Abraham. 2023. GriTS: Grid Table Similarity metric for table structure recognition. In Document Analysis and Recognition - ICDAR 2023: 17th International Conference, San José, CA, USA, August 21–26, 2023, Proceedings, Part V, page 535–549, Berlin, Heidelberg. Springer-Verlag. Aofeng Su, Aowen Wang, Chao Ye, Chen Zhou, Ga Zhang, Gang Chen, Guangcheng Zhu, Haobo Wang, Haokai Xu, Hao Chen, Haoze Li, Haoxuan Lan, Jiaming Tian, Jing Yuan, Junbo Zhao, Junlin Zhou, Kaizhe Shou, Liangyu Zha, Lin Long, and 14 others. 2024. TableGPT2: A large multimodal model with tabular data integration. Preprint, arXiv:2411.02059. Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024a. Table Meets LLM: Can large language models understand structured table data? A benchmark and etabmpirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM ’24, page 645–654, New York, NY, USA. Association for Computing Machinery. Yuan Sui, Jiaru Zou, Mengyu Zhou, Xinyi He, Lun Du, Shi Han, and Dongmei Zhang. 2024b. TAP4LLM: Table provider on sampling, augmenting, and packing semi-structured data for large language model reasoning. In Findings of the Association for Computational Linguistics, EMNLP’24, pages 10306–10323, Miami, Florida, USA. Association for Computational Linguistics. Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. MultiModalQA: Complex question answering over text, tables and images. In International Conference on Learning Representations. Gemma Team. 2025a. Gemma 3 Technical Report. Qwen Team. 2025b. Qwen3 Technical Report. Preprint, arXiv:2505.09388. Prasham Yatinkumar Titiya, Jainil Trivedi, Chitta Baral, and Vivek Gupta. 2025. MMTBENCH: A unified benchmark for complex multimodal table reasoning. Preprint, arXiv:2505.21771. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971. Lanrui Wang, Mingyu Zheng, Hongyin Tang, Zheng Lin, Yanan Cao, Jingang Wang, Xunliang Cai, and Weiping Wang. 2026. NeedleinATable: Exploring long-context capability of large language models towards long-structured tables. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. Jian Wu, Linyi Yang, Dongyuan Li, Yuliang Ji, Manabu Okumura, and Yue Zhang. 2025a. MMQA: Evaluating LLMs with multi-table multi-hop complex questions. In The Thirteenth International Conference on Learning Representations. Pengzuo Wu, Yuhang Yang, Guangcheng Zhu, Chao Ye, Hong Gu, Xu Lu, Ruixuan Xiao, Bowen Bao, Yijing He, Liangyu Zha, Wentao Ye, Junbo Zhao, and Haobo Wang. 2025b. RealHiTBench: A comprehensive realistic hierarchical table benchmark for evaluating LLM-based table analysis. In Findings of the Association for Computational Linguistics, ACL’25, pages 7105–7137, Vienna, Austria. Association for Computational Linguistics. Junjie Xing, Yeye He, Mengyu Zhou, Haoyu Dong, Shi Han, Lingjiao Chen, Dongmei Zhang, Surajit Chaudhuri, and H. V. Jagadish. 2025. MMTU: A massive multi-task table understanding and reasoning benchmark. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Zhuoyan Xu, Haoyang Fang, Boran Han, Bonan Min, Bernie Wang, Cuixiong Hu, and Shuai Zhang. 2026. Efficient table retrieval and understanding with multimodal large language models. In Findings of the Association for Computational Linguistics: EACL 2026, pages 4327–4340, Rabat, Morocco. Association for Computational Linguistics. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2024. Qwen2.5 technical report. ArXiv preprint, abs/2412.15115. Bohao Yang, Yingji Zhang, Dong Liu, André Freitas, and Chenghua Lin. 2025. Does table source matter? benchmarking and improving multimodal scientific table understanding and reasoning. Preprint, arXiv:2501.13042. Liangyu Zha, Junlin Zhou, Liyao Li, Rui Wang, Qingyi Huang, Saisai Yang, Jing Yuan, Changbao Su, Xiang Li, Aofeng Su, and 1 others. 2023. TableGPT: Towards unifying tables, nature language and commands into one GPT. arXiv preprint arXiv:2307.08674. Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. 2024. TableLlama: Towards open large generalist models for tables. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies (Volume 1: Long Papers), pages 6024–6044, Mexico City, Mexico. Association for Computational Linguistics. Xiaokang Zhang, Sijia Luo, Bohan Zhang, Zeyao Ma, Jing Zhang, Yang Li, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, Jifan Yu, Shu Zhao, Juanzi Li, and Jie Tang. 2025. TableLLM: Enabling tabular data manipulation by LLMs in real office usage scenarios. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10315–10344, Vienna, Austria. Association for Computational Linguistics. Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weiping Wang. 2024. Multimodal table understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9102–9124, Bangkok, Thailand. Association for Computational Linguistics. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. Preprint, arXiv:1709.00103. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and TatSeng Chua. 2021. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3277–3287, Online. Association for Computational Linguistics. Junnan Zhu, Jingyi Wang, Bohan Yu, Xiaoyu Wu, Junbo Li, Lei Wang, and Nan Xu. 2025. TableEval: A real-world benchmark for complex, multilingual, and multi-structured table question answering. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7126– 7146, Suzhou, China. Association for Computational Linguistics. Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Youcheng Pan, Xiaoqiang Zhou, and Min Zhang. 2026. Decoupling skeleton and flesh: Efficient multimodal table reasoning with disentangled alignment and structure-aware guidance. Preprint, arXiv:2602.03491.
A
Implementation and Evaluation Details
A.5
A.1
Model list
For QA, we report exact-match accuracy after light normalization. Single-answer questions require an exact normalized match, while multi-item lookup questions require the normalized predicted set to match the gold set. We also report a relaxed QA metric that counts a prediction as correct when the gold answer appears as a complete normalized span. Numeric answers are compared after extracting and normalizing numeric strings. For SUC, exact match is the primary metric, with Field Accuracy and Relaxed Accuracy reported as diagnostics for structured-answer errors. Field Accuracy compares pipe-separated fields position-wise, giving partial credit when only some fields are correct:
We evaluate a set of text-only language models (LLMs) and vision–language models (VLMs), including both general-purpose and table-oriented models. The evaluated models are: Language Models (text-only): Qwen2.5-7BInstruct, Qwen3-30B-A3B-Instruct, TableGPT27B, TAMA-QWen3. Vision–Language Models: SmolVLM2-2.2BInstruct, Gemma-3-12B-IT, Gemma-3-27B-IT, InternVL3.5-14B, InternVL3.5-30B-A3B, Qwen3VL-8B-Instruct, Qwen3-VL-30B-A3B-Instruct, Ministral-3-14B-Instruct, LLaVA-1.6-7B∗ , LLaVA1.6-13B∗ , TableLLaVA-v1.5-7B∗ , GPT-5.2, Gemini-3-Flash-Preview. A.2
A.3
Decoding and post-processing
All evaluations use zero-shot prompting with greedy decoding (temperature=0, top_p=1) and task-specific output limits. We apply minimal normalization for scoring consistency (e.g., stripping boilerplate prefixes, whitespace normalization, and simple label extraction for verification tasks). Prompt templates and exact decoding limits are listed in Appendix C.
(3)
k=1
where Ki is the number of gold fields. Relaxed Accuracy measures how many gold fields appear somewhere in the model output: K
RelaxedAcci =
i 1 X ⊮[yi,k ∈ ŷi ] Ki
(4)
k=1
Context length and coverage
Structural-text evaluation includes the full table markup in the prompt and can require long context. When a prompt exceeds a model’s supported context length (or fails to run reliably), we mark that instance as out of coverage for that model– configuration and compute metrics over the remaining evaluable instances. We report per-model coverage statistics alongside results. We ran open-weight model experiments on NVIDIA A100 SXM GPUs, using GPU execution for inference and supervised fine-tuning. Closed-model experiments were conducted through the OpenAI and Google Gemini APIs. A.4
K
i 1 X FieldAcci = ⊮[ŷi,k = yi,k ] Ki
Pipelines and configurations
We evaluate: (i) LLM (structural-text) for text-only models, (ii) VLM-Image (rendered table images) for VLMs, and (iii) VLM-Text (structural-text) for VLMs when the model interface supports text-only operation. For starred models (∗ ), we report results only for the configurations that are supported reliably by the model interface and context window.
Additional Metric Details
These diagnostics distinguish fully incorrect predictions from outputs containing the correct values in the wrong format or order. For SR, output usability is evaluated separately by target format: HTML must yield a parsable table, Markdown must render to a recoverable table, and LaTeX must compile successfully with a fixed wrapper. No repair is applied to malformed outputs.
B
Dataset Details
B.1 Source splits, filtering, and overlap checks We construct the raw pool from the official heldout split for each source dataset (test when available; otherwise dev) and filter to retain single-table, table-grounded instances. For datasets with mixed table/passage supervision (e.g., H YBRID QA and FEVEROUS), we drop instances whose gold evidence requires non-table context or multiple tables. We also check for overlaps against the corresponding training splits and remove duplicated question– table pairs when detected. Table 9 summarizes the resulting tagged pool.
Dataset FEVEROUS H YBRID QA TAB FACT SQA
Split
# Questions # Tables
WIKITQ
dev dev test test unseen tables
794 1608 1695 1000 1000
525 1608 1695 185 421
Total
–
6097
4434
C
Prompts
We use a dedicated prompt to assign each table– question pair to one of the predefined reasoning categories described in Section 3. The full classification prompt is shown in Figure 2. Prompt: Question Category Classification
Table 9: Composition of the tagged pool after filtering and normalization. #Questions denotes retained question instances, and #Tables denotes unique underlying tables.
System Prompt You are an expert at classifying table-question pairs into Question Categories. You must follow the rules exactly and output exactly ONE line in the required format.
B.2
Input TABLE (as JSON with header + rows): {table_json} QUESTION {query}
Tagged pool normalization
The source datasets use different schemas for tables and supervision. We normalize each example into a common JSON format (table content, question, answer/label, and metadata) and assign a stable table_id to each unique table so that multiple questions can reference the same underlying table. B.3 Format conversion to HTML, Markdown, and LaTeX We convert each table into three structural formats. We adapt conversion utilities released with Sui et al. (2024a) and extend them to (i) target our held-out splits, (ii) enforce consistent row/column ordering across formats, and (iii) produce syntactically valid outputs under dataset-specific artifacts (e.g., missing values and special characters). We generate HTML markup with standard <table>/<tr>/<th>/<td> tags, Markdown tables with pipe-delimited syntax, and compilable LaTeX tabular code with appropriate escaping. B.4
Rendering pipelines
We render a table image from each structural representation under a standardized layout (font size, padding, width). We render HTML tables in a controlled browser environment, convert Markdown to HTML before rendering, and compile LaTeX to PDF before converting to PNG. Because we render images from the generated markup, the rendered images are aligned with the textual tables by construction. B.5
Question category taxonomy
We use seven question categories for analysis and stratification (the same set used in the Results section). For each category, we provide a short definition and one example prompt in Appendix C.
TASK Assign exactly ONE Question Category using the definitions below. CRITICAL RULE (TABLE-REQUIREDNESS) A question can be assigned a predefined Question Category ONLY IF the table is REQUIRED to produce the final answer. External knowledge policy: You MAY use external knowledge to: • Map a description or condition to an entity present in the table (e.g., “country with population 67.02 million” → “France”) • Resolve aliases, real-world facts, or descriptive constraints ONLY for the purpose of identifying the correct row(s) You MUST NOT: • Output a final answer that does not come directly from table cells • Compute the final answer using non-table facts • Label as answerable if the table is not necessary If the final answer can be obtained without using the table at all, output: None of the above — Not table-required If the required answer value does not exist in or cannot be computed from the table cells, output: None of the above — Missing answer attribute in table PROCEDURE Step A — Table-Requiredness Check 1. Identify what the question asks for: • single value • list/set • number • yes/no 2. Identify which table cell(s) must be read or aggregated to produce the final answer. 3. Verify that the final answer is directly read from or computed using ONLY table cells.
4. If the table is not needed to produce the final answer → None of the above — Not table-required 5. If the answer value is not present or cannot be computed from table cells → None of the above — Missing answer attribute in table
For Structure Reconstruction (SR), models are instructed to generate a complete table representation in the requested target format. The reconstruction prompts are shown in Figure 4. Prompt: Table Structure Generation
Otherwise, proceed to classification. QUESTION CATEGORIES (Choose EXACTLY ONE) • Simple Lookup Identify ONE row and read ONE cell. (No filtering beyond locating that row.) • Conditional Lookup Apply one or more conditions to select row(s), then read ONE resulting value. • Multi-Item Lookup Return multiple values/rows from the table (a list/set). • Aggregation / Counting / Arithmetic Compute a number from table values (count/sum/avg/difference/ratio/percent/etc.). • Comparison & Extremum Choose max/min/earliest/latest by comparing table values. • Single-step Binary Verification Verify one statement directly using the table. • Multi-hop Binary Verification Verify a statement requiring multiple reasoning steps across the table. OUTPUT FORMAT (STRICT) Output exactly ONE line and nothing else: Question Category: <CATEGORY NAME> OR (if not answerable under rules): Question Category: None of the above
Figure 2: Prompt used to classify table-question pairs into structured reasoning categories.
For QA evaluation, we use a minimal answer-only prompt and a separate binary-verification variant for yes/no statements. The prompts are shown in Figure 3. Prompt: Table Question Answering General QA Prompt Look at the given table and answer the following question directly. Do not include introductions, explanations, or extra text. Provide only the exact, precise final answer. {query} Binary Verification Prompt Look at the given table and answer the following question with only a single digit: 1 if the statement is true, 0 if the statement is false. Do not include any explanations or extra text. {query}
Figure 3: Prompts used for table-based QA and binary verification tasks.
HTML Generation Prompt Generate the complete HTML code that exactly represents this image. Provide only the code without any explanations. LaTeX Generation Prompt Generate the complete LaTeX code that exactly represents this image. Provide only the code without any explanations. Markdown Generation Prompt Generate the complete Markdown code that exactly represents this image. Provide only the code without any explanations.
Figure 4: Prompts used for generating structured table representations from images in HTML, LaTeX, and Markdown formats.
For Structure Understanding and Cell-level Tasks (SUC), we use task-specific prompts covering boundary detection, table size estimation, coordinate lookup, and row/column retrieval. The complete prompt set is shown in Figure 5. Prompt: Table Structure and Cell-Level Tasks table_partition What is the **first cell value** (not including headers) of the given table? What is the **last cell value** (not including headers) of the given table? Answer questions one by one and use | to split the answer. Answer the question without having any introduction or explanations. table_first_cell What is the **first cell value** (not including headers) of the given table? Answer the question without having any introduction or explanations. table_last_cell What is the **last cell value** (not including headers) of the given table? Answer the question without having any introduction or explanations. size_detection How many rows in the table? How many columns in the table? Answer the questions one by one and use | to split the answer. Answer the question without having any introduction or explanations. number_of_rows How many rows in the table? Answer the question without having any introduction or explanations. number_of_columns How many columns in the table? Answer the question without having any introduction or explanations. cell_lookup
Row/column indices start at 0 (top-left is 0|0). What is the position of the cell value {cell_value}? Use row index and column index to answer. Use | to split the answer. Answer the question without having any introduction or explanations. reverse_lookup Row/column indices start at 0 (top-left is 0|0). What is the cell value of row index {reverse_lookup_row}, column index {reverse_lookup_col} ? Only output the cell value without other information. Answer the question without having any introduction or explanations. column_retrieval Row/column indices start at 0 (top-left is 0|0). What is the column name with the index {column_idx} of the given table image? Only give the column name without any explanation. Answer the question without having any introduction or explanations. row_retrieval Row/column indices start at 0 (top-left is 0|0). What are the cell values of the {row_idx} row in following table? Only list the cell values one by one using | to split the answers. Answer the question without having any introduction or explanations.
Figure 5: Prompts used for evaluating structural understanding
D
Result Discussion
D.1
TaskQA: Additional Analyses
This appendix provides additional TaskQA analyses that complement the main results. We examine modality gaps, strict-versus-relaxed matching, question-category performance, and Easy/Hard difficulty breakdowns to better understand model behavior across table formats and reasoning types. Modality gap: Figure 6 reports ∆ = VLM-Text avg − VLM-Image avg, averaged over HTML, LaTeX, and Markdown. Positive values mean that structured table text helps more than rendered images, while negative values mean that rendered images help more. The direction of the gap depends on the model family. Gemma-3 shifts toward structured text, with gains of +10.3 points for the 12B model and +7.0 points for the 27B model. In contrast, SmolVLM2 and InternVL3.5 shift toward rendered images. Gemini-3-Flash-Preview stays near zero, which is consistent with its stable performance across modalities in Table 3. These results suggest that modality preferences are model-dependent rather than a universal property of VLM-based table reasoning.
Figure 6: TaskQA modality gap. ∆ accuracy (pp) = VLM-Text avg − VLM-Image avg, averaged over HTML/LaTeX/Markdown. Negative means images help more than text.
Strict vs relaxed matching: Table 10 reports both strict exact-match and relaxed accuracy. Relaxed accuracy counts a prediction as correct when the normalized gold answer appears as a complete answer span within a longer response. This diagnostic separates answer retrieval from answeronly formatting rather than replacing strict EM. For example, Qwen3-VL-8B-IT remains stronger under strict EM, while Qwen3-VL-30B-A3B-IT improves substantially under relaxed matching, especially on structured table-text inputs, suggesting that some EM errors stem from verbose formatting rather than answer retrieval failures. Question categories: Figure 7 reports accuracy by question category, averaged over models and formats. Verification-style questions are easier across pipelines, while multi-item lookup and aggregation/counting questions are more difficult. This suggests that models handle binary or localized evidence better than questions requiring multiple retrieved items, counting, or arithmetic operations. The category averages in Figure 7 mask differences among the strongest models. Figure 8 shows that Gemini-3-Flash performs consistently well across most categories, while Qwen3-30B-A3B is strongest on multi-hop binary verification. The two Gemini-3-Flash variants perform similarly, indicating limited modality effects. In contrast, multi-item lookup and aggregation/counting remain among the weakest categories across pipelines, highlighting the difficulty of retrieval and composition.
Table Image Render Model
HTML EM Rel.
LaTeX EM Rel.
Language Models (text-only) Qwen2.5-IT – – – Qwen3-IT – – – TableGPT2 – – – TAMA-QWen3 – – – Vision-Language Models SmolVLM2-IT 29.71 31.57 28.71 Gemma-3-IT 12B 38.86 41.86 39.57 Gemma-3-IT 27B 46.14 50.29 45.29 InternVL3.5 14B 48.57 52.57 48.14 InternVL3.5 30B-A3B 47.86 54.29 50.00 Qwen3-VL-IT 8B 50.29 55.43 49.29 Qwen3-VL-IT 30B-A3B 41.14 53.29 42.14 Ministral-3-IT 44.43 55.29 39.14 LLaVA-1.6 7B 31.86 35.71 31.43 LLaVA-1.6 13B 25.14 27.57 23.71 Table-specialised Vision-Language Models TableLLaVA-v1.5 1.29 20.86 1.00 Proprietary Models GPT-5.2 54.57 61.71 54.52 Gemini-3-Flash-Preview 65.43 72.00 65.14
Table Text Format
Markdown EM Rel.
HTML EM Rel.
LaTeX EM Rel.
Markdown EM Rel.
– – – –
– – – –
– – – –
44.57 51.14 44.43 18.29
50.29 61.43 57.29 52.00
42.71 48.43 41.57 19.14
48.43 60.86 56.00 53.71
45.43 46.57 42.14 20.71
51.43 62.57 58.57 52.71
30.71 43.29 48.57 52.43 55.43 54.71 54.29 49.29 35.29 26.43
25.86 38.57 45.43 48.00 48.29 49.71 41.43 42.71 32.00 25.00
30.00 43.00 49.43 52.57 54.71 55.14 53.29 53.00 35.71 27.43
21.57 50.29 53.43 47.14 45.86 53.43 45.29 40.00 27.37 23.91
39.80 55.57 59.14 55.86 57.57 59.14 62.86 50.86 43.91 42.56
17.63 49.00 51.29 47.29 45.71 52.14 43.57 35.43 29.55 22.13
34.39 54.14 56.57 54.29 56.14 57.43 60.86 46.86 41.05 43.67
15.75 48.57 53.14 44.86 47.00 53.29 39.71 36.57 26.50 24.89
33.67 53.71 58.86 55.14 61.00 59.14 63.00 46.57 42.17 44.07
19.71
4.00
23.71 23.61 36.24 27.37 37.41 28.40 38.80
61.12 56.14 63.29 57.43 66.00 57.29 65.29 58.00 66.57 71.16 65.43 71.57 65.71 72.29 65.00 71.86 65.43 72.86
Table 10: TaskQA strict and relaxed matching diagnostic. Exact-match accuracy (EM) and relaxed accuracy (Rel.) are reported across HTML, LaTeX, and Markdown inputs. Relaxed accuracy counts a prediction as correct when the normalized gold answer appears as a complete answer span inside a longer response. Underlined values indicate the highest score within each selected model-variant group and column. This diagnostic does not replace EM; it highlights cases where models retrieve the correct answer but fail the answer-only format required by strict exact match.
Figure 7: TaskQA category averages. Accuracy by question category, averaged over models and formats, shown per pipeline (VLM-Image / VLM-Text / LLMText).
Easy vs Hard split: Figure 9 reports Easy and Hard accuracy for each model and pipeline, averaged over HTML, LaTeX, and Markdown. All pipelines show a clear drop from Easy to Hard questions, confirming that the difficulty annotation captures increased reasoning or evidence-composition demands. The gap is particularly large for the strongest VLMs, whose Easy accuracy often exceeds 80–90% while Hard accuracy remains below 40%, indicating that multi-step reasoning remains a major challenge despite strong overall performance. We use this split as a diagnostic view of difficulty rather than a definition of reasoning complexity.
Figure 8: Category-wise Task QA accuracy for the strongest model from each pipeline.
Evaluation notes: TaskQA is scored with strict exact-match accuracy in the main results. We apply the same normalization and post-processing to all models. Strict EM is intentionally conservative: answers with extra explanatory text are counted as incorrect even when they contain the gold answer. For this reason, Table 10 provides a diagnostic relaxed-matching view.
Figure 9: TaskQA Easy vs Hard. Easy and Hard exact-match accuracy per model and pipeline (VLM-Image / VLM-Text / LLM-Text), averaged over HTML/LaTeX/Markdown.
Models marked with ∗ have shorter context windows; when they fail to return an answer on largetable cases, the output is counted as incorrect under the same scoring rule. This ensures consistent evaluation across architectures and context lengths. SUC Tasks
Partition and Boundary Detection
Lookup and Retrieval
Size Estimation
First Cell
No. of Rows
Cell Lookup
Last Cell
No. of Columns
Reverse Lookup
Table Partition
Size Detection
Row Retrieval Column Retrieval
Figure 10: Taxonomy of SUC tasks. Ten tasks are grouped into partitioning, size estimation, lookup, and retrieval.
Figure 11: SUC task difficulty. We average exactmatch accuracy over all models, pipelines, and formats for each SUC subtask. Higher is better.
D.2
Subtask difficulty: Figure 11 shows that SUC difficulty is highly subtask-dependent. Column counting and column retrieval are consistently among the easiest subtasks across pipelines. In contrast, row retrieval, cell lookup, table partitioning, and size detection remain difficult. This pattern shows that models are better at identifying global column structure than at recovering precise rowlevel or coordinate-based structure.
Structural Understanding Capability: Additional Analyses
This appendix provides additional SUC analyses that support Section 5.2. Figure 10 summarizes the SUC task taxonomy. We include subtask-level difficulty, pipeline comparisons, diagnostic metrics, prompt-sensitivity results, format effects, and evaluation notes.
Pipeline comparison: Figure 12 compares SUC accuracy across VLM-Image, VLM-Text, and LLM-Text. Structured text generally improves SUC, especially on subtasks that depend on row boundaries and header handling. This is also visible in Tables 4, 11, and 13. For example, several models improve on row retrieval, reverse lookup, and size detection when the table structure is provided as text. However, text input does not remove the bottleneck entirely. Cell lookup and row retrieval remain difficult across pipelines, showing that SUC requires more than access to explicit table markup. Field accuracy and relaxed accuracy diagnostics: Tables 5 and 12 show that strict EM can hide partial structural recovery. For multi-field tasks such as table partitioning, size detection, cell lookup, and row retrieval, models may partially recover the target structure. Field Accuracy captures this behavior by scoring individual fields, while Relaxed Accuracy captures outputs that contain the correct answer but fail strict string matching. The large gaps between EM and Field Accuracy show that many errors are incomplete or shifted structural predictions rather than completely unrelated answers. This pattern is consistent across both VLM-Image and VLM-Text pipelines, indicating that structural localization remains a common source of failure even when the correct information is partially recovered.
Models
Formats
T.P. F.C. L.C. S.D. # Rows # Cols C.Lu. R.Lu. Co.Rt. Ro.Rt. Overall
Open Models HTML LaTeX Markdown HTML Gemma-3-12B-IT LaTeX Markdown HTML InternVL3.5-30B-A3B LaTeX Markdown HTML Qwen3-VL-8B-IT LaTeX Markdown HTML LLaVA-1.6-7B LaTeX Markdown Table-specialised Models HTML TableLLaVA-v1.5-7B LaTeX Markdown SmolVLM2-2.2B-IT
0.2 0.0 0.0 10.3 13.8 10.7 0.0 0.0 0.0 23.8 20.8 24.6 0.0 0.3 0.2
48.6 49.4 19.6 37.5 41.2 40.1 88.9 89.2 84.9 93.3 93.2 87.8 14.3 13.8 14.1
31.6 34.5 17.0 45.8 48.2 46.7 79.0 80.6 76.5 80.9 78.9 78.4 14.5 18.1 18.1
2.1 0.2 0.5 13.7 13.5 18.1 8.7 12.6 25.1 8.1 14.3 14.6 3.2 1.3 1.9
26.9 18.9 2.7 30.0 30.2 32.8 39.9 44.5 56.1 25.6 36.1 31.8 18.9 20.5 17.6
8.9 8.4 0.6 35.0 37.2 42.0 88.6 83.5 85.5 89.3 88.7 90.1 30.2 23.5 23.8
1.1 1.1 0.5 10.3 11.0 14.1 17.8 21.5 24.5 27.0 32.4 28.0 1.7 1.9 1.6
6.2 7.0 6.2 14.5 15.1 15.4 9.2 15.9 13.0 26.6 30.0 34.2 1.3 2.9 3.3
35.0 30.4 5.6 56.3 62.8 66.8 53.7 69.8 64.9 90.1 89.5 89.8 29.6 28.3 29.9
0.0 0.0 0.0 2.9 3.0 2.4 2.2 4.0 4.5 1.0 3.0 2.2 0.0 0.0 0.0
16.1 15.0 5.3 25.6 27.6 28.9 38.8 42.1 43.5 46.6 48.7 48.2 11.4 11.1 11.1
0.0 0.0 0.0
6.5 2.2 6.7
4.3 4.3 9.2
0.0 0.0 0.0
0.0 2.1 13.0
0.3 2.9 14.5
0.0 0.0 0.0
1.6 1.7 2.7
1.0 3.7 9.2
0.0 0.0 0.0
1.4 1.7 5.5
Table 11: Additional SUC results for the VLM-Image pipeline. Exact-match accuracy (%) across ten structureoriented subtasks. Models receive rendered table images derived from HTML, LaTeX, and Markdown sources.
Figure 12: Pipeline comparison. We average exact-match accuracy per SUC subtask for VLM-Image, VLM-Text, and LLM-Text. Higher is better.
Prompt explicitness and header handling: Table 6 compares the explicit SUC prompt with an implicit prompt on selected VLM-Image subtasks. The explicit prompt states conventions such as excluding headers for first/last-cell tasks and using 0-indexed row/column coordinates for lookup and retrieval tasks, while the implicit prompt removes these details. The largest differences appear on index-dependent subtasks, especially firstcell detection and table partitioning, suggesting that rendered-table models often confuse header rows with table body rows and adopt different rowindexing conventions. The effect is weaker for last-cell detection, which is less affected by header counting. Reverse lookup shows the opposite trend for some models, where the implicit prompt performs better. These effects are particularly pronounced for first-row-sensitive tasks. This confirms that SUC is sensitive not only to visual structure recognition but also to how models interpret indexing and header conventions.
Format effects: Figure 13 and Tables 4, 11, 13, and 14 show that format effects are present but not uniform. For VLM-Image, differences across HTML, LaTeX, and Markdown renders are usually smaller than differences across models and subtasks. Figure 13 further shows that first-cell detection, column counting, row counting, and size detection are the most format-sensitive subtasks, while row and column retrieval vary little across formats. For text-input pipelines, format effects are more visible. HTML is often the safest structured-text format, especially for VLM-Text and LLM-Text, while LaTeX and Markdown can be less stable for some models. This suggests that SUC depends jointly on input modality, table representation, and row/column conventions. Evaluation notes: We report strict exact-match accuracy as the main SUC metric because the tasks require exact structured outputs, such as row– column coordinates, table size fields, or ordered cell sequences.
Model
Format
EM Field Acc. Relaxed Acc. ∆Field ∆Relaxed
VLM HTML LaTeX Markdown HTML InternVL3.5-30B-A3B LaTeX Markdown InternVL3.5-14B
47.8 46.5 48.8 38.8 42.1 43.5
57.5 57.1 59.2 52.8 55.4 56.8
58.3 57.8 59.8 53.7 56.4 57.8
+9.7 +10.6 +10.4 +14.0 +13.3 +13.3
+10.5 +11.3 +11.0 +14.9 +14.3 +14.3
53.9 48.9 47.9 46.9 42.8 42.1
66.0 61.2 60.8 64.2 57.3 57.5
67.5 62.4 62.5 72.0 60.4 65.8
+12.1 +12.3 +12.9 +17.3 +14.5 +15.4
+13.6 +13.5 +14.6 +25.1 +17.6 +23.7
VLM-TEXT HTML LaTeX Markdown HTML InternVL3.5-30B-A3B LaTeX Markdown InternVL3.5-14B
Table 12: Diagnostic comparison of InternVL3.5-14B and InternVL3.5-30B-A3B on SUC. Values report overall exact-match accuracy (EM), Field Accuracy, and Relaxed Accuracy across formats for the VLM-Image and VLM-Text pipelines. ∆Field = Field Accuracy − EM and ∆Relaxed = Relaxed Accuracy − EM. Model
Format
T.P. F.C. L.C. S.D. # Rows # Cols C.Lu. R.Lu. Co.Rt. Ro.Rt. Overall
Open Models HTML LaTeX Markdown HTML InternVL3.5-30B-A3B LaTeX Markdown HTML Qwen3-VL-30B-A3B-IT LaTeX Markdown HTML Qwen3-VL-8B-IT LaTeX Markdown HTML Gemma-3-12B-IT LaTeX Markdown HTML Gemma-3-27B-IT LaTeX Markdown HTML Ministral-3-14B-IT LaTeX Markdown HTML SmolVLM2-2.2B-IT LaTeX Markdown HTML LLaVA-1.6-13B∗ LaTeX Markdown HTML LLaVA-1.6-7B∗ LaTeX Markdown Table-specialised Models HTML TableLLaVA-v1.5-7B LaTeX Markdown Proprietary Models HTML GPT-5.2 LaTeX Markdown HTML Gemini-3-Flash-Preview LaTeX Markdown InternVL3.5-14B
8.4 11.1 1.4 0.0 0.0 0.0 49.9 29.4 26.4 26.9 18.6 10.7 18.9 14.5 4.3 53.7 33.2 16.4 20.0 14.1 1.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
95.5 94.1 76.3 91.6 85.1 64.4 94.6 95.5 87.8 93.5 92.1 68.5 68.5 53.1 31.6 82.5 62.6 45.3 95.5 79.0 69.0 11.3 0.5 8.3 21.5 24.7 12.5 27.5 38.1 16.1
78.7 76.0 71.4 75.5 75.8 75.4 76.8 69.6 64.9 80.0 76.3 64.5 81.4 76.5 67.4 82.2 76.2 70.6 57.2 62.3 49.4 9.2 0.2 5.9 5.2 7.6 4.9 2.7 2.7 2.9
8.4 11.0 14.5 0.0 0.0 0.0 68.5 46.9 62.3 27.8 15.7 30.2 64.9 30.5 51.4 72.2 60.4 59.5 22.4 30.2 6.5 0.0 0.0 0.0 1.0 0.3 0.0 0.0 0.0 0.0
50.6 44.2 65.7 11.6 9.1 15.7 73.4 64.5 84.6 69.0 36.2 66.9 77.9 35.6 73.3 73.8 60.1 83.3 51.2 40.5 44.7 6.8 0.0 17.9 12.3 1.0 8.1 0.0 10.8 1.8
98.4 99.4 85.9 99.5 99.4 97.1 96.7 97.5 71.7 98.3 100.0 92.8 90.1 98.7 67.2 98.1 99.5 76.5 99.0 99.4 80.0 0.2 0.0 1.8 31.2 1.6 5.5 0.0 8.4 0.3
50.9 36.9 43.6 42.9 28.8 32.4 40.5 29.7 29.6 45.3 30.2 28.1 31.3 19.1 19.4 45.0 42.0 40.5 41.2 19.2 23.8 0.2 0.0 0.0 2.5 2.3 1.6 0.0 1.0 0.2
44.2 22.3 33.2 31.0 21.0 28.5 37.5 19.7 20.8 47.9 17.5 18.8 28.9 18.4 20.7 55.6 32.3 38.5 37.7 17.0 21.3 2.6 2.7 4.5 0.7 2.4 1.3 1.0 0.8 0.6
95.2 87.1 83.9 87.0 86.8 86.2 89.7 83.0 72.2 84.9 74.1 69.0 85.1 76.9 76.3 94.9 89.0 91.6 88.4 68.5 62.0 2.9 2.6 4.2 16.0 19.4 12.5 14.4 13.6 3.9
8.6 6.8 3.3 30.2 21.9 21.5 14.3 8.6 5.9 3.2 5.6 8.3 5.6 3.7 2.5 16.2 13.2 11.6 22.7 12.2 11.1 0.0 0.0 0.0 0.2 0.2 0.0 0.0 0.0 0.0
53.9 48.9 47.9 46.9 42.8 42.1 64.2 54.5 52.6 57.7 46.6 45.8 55.3 42.7 41.4 67.4 56.9 53.4 53.5 44.3 37.0 3.3 0.6 4.3 9.1 5.9 4.6 4.6 7.5 2.6
0.0 19.7 0.0 33.8 0.0 21.3
2.8 5.7 3.9
0.0 0.0 0.0
7.0 23.9 24.5
0.5 9.5 3.1
0.2 0.5 0.0
1.7 1.5 1.0
7.3 7.8 5.4
0.0 0.0 0.0
3.9 8.3 5.9
94.9 90.5 88.6 92.7 90.6 88.9
95.5 92.5 93.3 90.0 89.3 89.0
98.4 29.3 87.3 86.5 8.9 75.8
97.3 84.4 96.0 97.8 89.8 95.1
100.0 100.0 99.8 100.0 100.0 100.0
53.3 7.0 14.8 16.9 0.6 15.9
86.0 17.2 34.8 79.0 19.4 67.1
94.8 97.6 96.2 97.3 97.5 97.5
84.4 3.5 25.1 59.9 5.9 49.8
90.4 62.0 73.1 81.7 59.9 77.5
99.4 98.4 94.9 97.3 97.1 95.7
Table 13: SUC results for the VLM-Text pipeline. Exact-match accuracy (%) across ten structure-oriented subtasks. Models receive table text extracted from HTML, LaTeX, and Markdown sources.
We use the same post-processing across models and formats. Field Accuracy and Relaxed Accuracy are diagnostic metrics for partial structural recovery and formatting errors, but do not replace EM.
For models with shorter context windows or weaker instruction following, incomplete outputs and formatting errors are counted as incorrect under strict EM to ensure consistent evaluation.
Models
Formats
T.P.
F.C.
L.C.
S.D.
# Rows
# Cols
C.Lu.
R.Lu.
Co.Rt.
Ro.Rt.
HTML LaTeX Markdown HTML LaTeX Markdown
16.7 11.1 5.9 45.0 25.6 23.4
95.1 44.4 55.6 94.8 91.3 83.5
57.7 54.8 53.7 68.0 60.9 56.0
20.5 3.3 15.1 72.2 45.5 57.6
52.3 34.3 62.5 81.1 51.2 75.5
80.0 99.0 43.7 89.8 85.4 44.4
25.4 7.3 8.1 44.7 28.3 28.9
14.3 5.2 9.2 25.0 9.4 9.2
75.4 61.4 52.6 82.2 66.9 63.4
5.7 5.4 2.7 4.8 2.2 1.4
HTML LaTeX Markdown HTML LaTeX Markdown
20.8 18.1 4.9 7.2 12.1 4.6
94.1 89.7 67.2 48.5 35.1 38.0
68.4 69.0 58.0 4.9 15.9 6.0
31.0 16.7 39.0 6.8 0.0 0.0
49.0 38.8 66.1 15.1 0.0 3.2
78.7 98.7 38.0 44.2 51.0 54.2
13.5 6.2 6.7 6.4 2.1 1.7
12.9 7.2 9.2 9.4 2.1 3.5
66.8 57.4 49.0 86.0 59.3 83.6
3.2 4.1 1.0 0.6 0.2 0.3
Open Models Qwen2.5-7B-Instruct Qwen3-30B-A3B-Instruct Table-specialised Models TableGPT2-7B TAMA-QWen3
Table 14: SUC results for the LLM pipeline. Exact-match accuracy (%) across subtasks is reported.
Figure 13: Format sensitivity. We show mean exact-match accuracy by SUC subtask and format (HTML, LaTeX, Markdown) and the variation across formats. Higher is better.
Figure 14: Pipeline for SR. A ground-truth table x is rendered into an image, the model predicts a structure x′ , and evaluation compares x′ with x.
D.3
Structure Reconstruction: Additional Analyses
Figure 14 illustrates the SR evaluation pipeline, where a table representation is rendered as an image, reconstructed by the model, and compared against the original structure. The analyses below examine reconstruction fidelity, output usability, format-pair difficulty, and cross-format conversion behavior. Validity-adjusted SR scores: Table 15 reports zero-penalized G RI TS, where unusable outputs receive a score of zero before averaging. This combines the two SR failure modes: invalid target syntax and inaccurate reconstruction.
The gap between raw and zero-penalized scores is small for strong models on HTML and Markdown targets, showing that most remaining errors are fidelity errors rather than syntax failures. The gap is larger for LaTeX targets, especially for weaker VLMs and TableLLaVA, confirming that LaTeX failures often arise from invalid or non-compilable outputs rather than low structural similarity alone. Combined with the usability results in Table 16, this indicates that output validity remains a major source of error primarily for LaTeX generation across model families. Output usability across formats: Table 16 reports format-specific usability rates for representative open models. HTML and Markdown outputs are nearly always usable for strong models, with several systems achieving usability close to 100% across source formats. In contrast, LaTeX usability is consistently lower, even for strong models, and drops to zero for weaker LLaVA variants. These results reinforce the main finding that LaTeX reconstruction is challenging not only because of table structure recovery but also because models must produce syntactically valid target code.
HTML image Models
Markdown image
LaTeX image
Topology-Zero Content-Zero Topology-Zero Content-Zero Topology-Zero Content-Zero HTML Md TeX HTML Md TeX HTML Md TeX HTML Md TeX HTML Md TeX HTML Md TeX
Open Models SmolVLM2-2.2B-IT Gemma3-12B-IT Gemma3-27B-IT InternVL3.5-14B InternVL3.5-30B-A3B Qwen3-VL-8B-IT Qwen3-VL-30B-A3B-IT Ministral3-14B-Instruct LLaVA1.6-Vicuna-7B LLaVA1.6-Vicuna-13B
0.87 0.94 0.97 0.99 0.98 0.99 0.98 0.98 0.70 0.66
0.85 0.31 0.95 0.75 0.97 0.84 0.99 0.89 0.99 0.88 1.00 0.87 0.99 0.69 0.94 0.83 0.64 0.00 0.80 0.00
0.74 0.79 0.86 0.95 0.95 0.98 0.98 0.94 0.43 0.46
0.73 0.27 0.79 0.63 0.84 0.75 0.94 0.86 0.94 0.85 0.98 0.85 0.98 0.67 0.90 0.81 0.39 0.00 0.51 0.00
0.66 0.95 0.98 0.99 0.99 0.99 0.99 0.98 0.70 0.62
0.76 0.30 0.96 0.75 0.97 0.84 0.98 0.87 0.99 0.87 1.00 0.86 1.00 0.88 0.94 0.83 0.52 0.00 0.74 0.00
0.57 0.80 0.85 0.93 0.93 0.97 0.97 0.92 0.46 0.45
0.66 0.26 0.81 0.63 0.85 0.74 0.93 0.83 0.94 0.84 0.98 0.84 0.98 0.85 0.90 0.80 0.35 0.00 0.50 0.00
0.79 0.94 0.97 0.96 0.96 0.98 0.97 0.95 0.71 0.65
0.83 0.57 0.94 0.74 0.96 0.83 0.97 0.89 0.97 0.87 0.98 0.92 0.98 0.93 0.92 0.86 0.57 0.00 0.74 0.01
0.66 0.78 0.83 0.93 0.93 0.95 0.95 0.87 0.44 0.46
0.68 0.47 0.77 0.62 0.82 0.73 0.91 0.87 0.92 0.84 0.95 0.91 0.95 0.92 0.86 0.82 0.36 0.00 0.49 0.01
Proprietary Models GPT-5.2 0.98 0.98 0.78 0.97 0.94 0.75 0.98 0.99 0.91 0.96 0.97 0.88 0.98 0.97 0.91 0.94 0.94 0.88 Gemini-3-Flash-Preview 0.05 0.96 0.63 0.05 0.94 0.62 0.86 0.97 0.50 0.85 0.96 0.49 0.65 0.93 0.57 0.63 0.91 0.57 Table-specialised Models TableLLaVA-v1.5-7B 0.73 0.69 0.55 0.33 0.32 0.26 0.73 0.71 0.56 0.43 0.42 0.33 0.58 0.56 0.49 0.29 0.29 0.26
Table 15: Usability-aware SR scores. We report zero-penalized GriTS-Topology and GriTS-Content, where unusable outputs receive a score of zero before averaging, capturing both reconstruction fidelity and output usability. HTML image
Markdown image
LaTeX image
Models
HTML
Md
TeX HTML
Md
TeX HTML
Md
TeX
Open Models SmolVLM2-2.2B-IT Gemma3-12B-IT InternVL3.5-14B InternVL3.5-30B-A3B Qwen3-VL-8B-IT Ministral3-14B-Instruct LLaVA1.6-Vicuna-7B
0.97 1.00 1.00 1.00 1.00 1.00 0.98
0.90 0.99 1.00 1.00 1.00 0.97 0.76
0.35 0.81 0.91 0.89 0.88 0.84 0.00
0.81 1.00 0.99 1.00 1.00 0.97 0.68
0.37 0.81 0.89 0.88 0.87 0.84 0.00
0.92 0.99 0.99 1.00 1.00 0.98 0.73
0.60 0.80 0.91 0.89 0.95 0.88 0.00
0.75 1.00 1.00 1.00 1.00 1.00 0.99
0.92 1.00 1.00 1.00 1.00 1.00 0.99
Table 16: Additional SR output usability results. Values report the fraction of syntactically usable outputs across source and target formats. Best scores per column are shown in bold.
Format-pair difficulty: Figure 15 averages G RI TS-Topology and G RI TS-Content across all models for each input–output format pair. The heatmaps show that output format has a stronger effect than input render format. Markdown targets achieve the highest average scores, while LaTeX targets consistently achieve the lowest. The similarity of rows within each heatmap suggests that models are generally robust to source rendering. Thus, SR difficulty is driven more by the target than the input representation.
In several cases, cross-format performance is comparable to or slightly better than same-format reconstruction. Larger gaps appear for weaker models, where failures are dominated by unstable parsing, content degradation, or invalid output generation rather than format conversion itself. This suggests that SR errors primarily arise from table understanding and target-format generation, not from translating between representations.
Same-format versus cross-format reconstruction: Figure 16 compares same-format reconstruction with cross-format conversion for each model. Strong models such as Qwen3-VL and InternVL3.5 exhibit only small differences between the two settings, indicating that cross-format conversion introduces little additional difficulty once the table structure has been recovered.
Error analysis (TaskQA): Table 17 summarizes the most common TaskQA failure modes under exact-match evaluation. Most errors arise from incorrect table grounding, incomplete answer retrieval, or answer-format mismatches rather than completely unrelated predictions or random hallucinations. These patterns are consistent across models, formats, and input modalities.
E
Error Analysis
Figure 15: Format-pair difficulty (SR). We average GriTS-Topology and GriTS-Content over all models for each input→output format pair.
Figure 16: Same-format vs cross-format SR. We compare per-model averages for same-format reconstruction against cross-format conversion.
On lookup-style questions, many errors arise from selecting a plausible cell from the correct column but the wrong row, or from missing a filtering condition in Conditional Lookup. Similar failures occur in Comparison/Extremum questions when models compare values within the wrong subset. MultiItem Lookup introduces two distinct failure modes: missing or extra items, and formatting mismatches that break set matching after normalization. For Aggregation/Arithmetic, errors are typically small counting or summation mistakes caused by skipped or double-counted rows. Verification errors usually appear as 0/1 label flips, often involving negation or multi-row conditions. Exact match also penalizes answers containing extra text before the answer token, a failure mode that is more common for smaller models and explanation-oriented outputs. Error analysis (SUC): Table 18 summarizes the main SUC failure modes under strict exact match across all evaluated pipelines.
We inspected outputs from strong proprietary models (GPT-5.2, Gemini-3-Flash-Preview), strong open models (Qwen3-VL-8B), and a tablespecialized baseline (TableLLaVA-v1.5-7B) across VLM-Image, VLM-Text, and LLM-Text settings. We apply identical post-processing and exactmatch scoring to all models, making indexing, formatting, and convention mismatches visible. Header offset explains many indexing/lookup errors, especially in VLM-Image: Our gold labels define indices over data cells (headers excluded). Under a fixed coordinate convention, some models treat the header as part of the indexable grid, which produces a consistent +1 row shift. When this happens, size-related probes report one extra row (e.g., gold S.D. = 10|3 vs. prediction 11|3), coordinate probes return row indices that are one larger than gold, and row retrieval often returns the row one position above the gold row for the same queried index.
Error type
What it looks like
Common in
Wrong table cell
Prediction matches a plausible cell/header, but it comes from the wrong Lookup / Conditional / row/column (often satisfies only part of the condition). Comparison
Off-table answer
Prediction does not match any table cell and does not equal the gold VLM-Image (more) (e.g., generic “cannot determine” replies or free-form values).
Multi-item mismatch
Missing items, extra items, or mixed sets when the question expects a Multi-Item Lookup specific set of values.
Multi-item formatting
Gold items appear in the output, but separators/punctuation break set Multi-Item Lookup matching (e.g., commas that belong to an entity name).
Arithmetic near-miss
Wrong count/sum; off-by-one errors are common when one row is Aggregation/Arithmetic missed or double-counted.
Binary label flip (0/1)
Model outputs a valid 0/1 token but flips the label relative to gold.
Answer not isolated
Gold answer appears in the output but not as the first token (e.g., extra Smaller models; LLMprefix or short descriptor). Text
Verification
Table 17: Common TaskQA failure modes under our exact-match scoring (light normalization; set match for Multi-Item Lookup). Error type
What it looks like
Often affects
Header offset (+1 row)
Model treats the header row as part of the indexed grid (header becomes row 0), shifting row indices by 1. The queried value appears multiple times; the model returns a different valid coordinate or lists several coordinates. Extra text, multi-line answers, or a delimiter different from the required template (e.g., “Row = 3, Col = 2”). Row retrieval returns the right row content but with small formatting differences (e.g., leading/trailing ‘|‘, missing cells, or Markdown-style rows). Model outputs a full Markdown table or explanation instead of a single required answer span.
S.D., #Rows, C.Lu., Ro.Rt.
Repeated-value ambiguity Answer-template mismatch Row serialization drift Verbose structured outputs
C.Lu., R.Lu. C.Lu., S.D., #Rows, Ro.Rt. Ro.Rt. Mostly Ro.Rt. C.Lu./S.D.)
(also
Table 18: Common SUC error types under strict exact match.
This pattern makes models look strong on boundary and column probes while scoring poorly on C.Lu. and Ro.Rt. under exact match. We keep one coordinate convention across pipelines to surface this sensitivity rather than tuning prompts separately per modality or per format. Repeated values make coordinate probes harder than they appear. Many tables contain repeated values (especially short strings and common numbers), so more than one coordinate can look reasonable for C.Lu. and R.Lu. Models then either pick a different occurrence or return multiple coordinates. Exact match counts both cases as incorrect, even when the output remains consistent with the table content. Ro.Rt. often fails on output formatting, not only row selection. Ro.Rt. requires emitting the full row as a pipe-separated string. Models sometimes add leading/trailing bars, change spacing, omit empty cells, or output Markdown-style rows.
TableLLaVA also tends to produce multi-line structured outputs instead of a single row span. These differences fail exact match even when the chosen row is close. Verbosity and uniform post-processing. Some models (especially TableLLaVA, and occasionally smaller VLMs) generate verbose answers that include the correct information inside extra text or inside a larger structured block. We intentionally keep one uniform post-processing rule for all models to evaluate end-to-end, machine-readable reliability. This choice can undercount models that do not follow the requested answer template without using model-specific extraction rules.