The Table Says Otherwise: Testing LLMs with Counterfactual Relational Data Xinzhi Wang
Chunwei Liu
Purdue University [email protected]
Purdue University [email protected]
arXiv:2606.23667v1 [cs.DB] 22 Jun 2026
ABSTRACT Large language models (LLMs) are increasingly used to answer natural-language questions over structured data. However, when a table contains familiar real-world facts, it is unclear whether the model answers by reading the provided data or by recalling knowledge learned during pretraining. This distinction is important for database applications, where the provided tables should be the source of truth. In this paper, we introduce ContraTable, a paired original–counterfactual benchmark for evaluating whether LLMs ground their answers in relational tables. We build the benchmark with two aligned versions: an original database with real-world facts and a counterfactual database that preserves the same schemas, identifiers, and relationships while changing selected country, club, and player attributes. We design 214 matched questions across three levels: single-table lookup, multi-table lookup, and multi-table temporal reasoning. Experiments on commercial closed-source and open-source models show that strong instruction-tuned models can often handle direct lookup, but their reliability drops as questions require joins, comparison, and temporal reasoning. The gap between original and counterfactual accuracy reveals that models may fall back on prior knowledge when table evidence conflicts with familiar facts. These results suggest that table-QA evaluation should measure not only accuracy, but also faithfulness to the provided database. VLDB Workshop Reference Format: Xinzhi Wang and Chunwei Liu. The Table Says Otherwise: Testing LLMs with Counterfactual Relational Data. VLDB 2026 Workshop: NOVAS. VLDB Workshop Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/AuroraWXZ/LLM_understand_table.
1
INTRODUCTION
Large language models (LLMs) are increasingly used as naturallanguage interfaces to data. Prior work has studied question answering over free-form documents and images [1, 13, 15, 20], while recent data-management systems and benchmarks explore how LLMs can help users interact with structured and semi-structured data [3, 4, 7, 10, 12, 19, 23, 24, 26, 27]. A simple and flexible approach is to provide a small set of related tables and ask questions in natural language, avoiding the need to write SQL. However, this setting raises a key question: when an LLM answers a table-based question, This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected]. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment. ISSN 2150-8097.
does the answer come from the provided tables or from knowledge learned during pretraining [9, 21]? This distinction is important for database applications, where the provided data should be the source of truth. Table values may be private, local, recently updated, or hypothetical, and may intentionally differ from public knowledge in settings such as simulation, what-if analysis, data cleaning, or testing. If an LLM replaces table evidence with familiar facts from pretraining, it may produce an answer that sounds plausible but is wrong with respect to the database. Existing table question-answering benchmarks make this behavior hard to isolate, because many are built from real-world entities that may already appear in pretraining data [3, 4, 10, 12, 19, 27]. As a result, benchmark accuracy can mix two abilities: reasoning over the given tables and recalling world knowledge. A high score may therefore overestimate how faithfully a model follows the provided tables. To separate these two abilities, we draw inspiration from knowledgeediting research, where counterfactual facts are used to test whether a model follows newly provided information or returns a familiar real-world answer [5, 8, 16, 17]. We adapt this idea from isolated facts to relational databases. Instead of changing a single statement, we construct a counterfactual database that preserves the original schema, identifiers, and relational structure, while replacing selected values with valid conflicting values. This design lets us compare model behavior when the table agrees with real-world knowledge and when it conflicts with it. In this paper, we introduce ContraTable, a paired original– counterfactual benchmark built from a football transfer-market database. This domain is well suited for our goal: many facts about countries, clubs, and players are likely to be known by LLMs [9, 21], while the database also contains rich relational and temporal structure through players, clubs, countries, competitions, games, appearances, and transfers. The original database contains real-world facts. The counterfactual database keeps the same schemas and relationships, but changes selected attributes such as country capitals, continents, confederations, club countries, domestic competitions, stadiums, player citizenship, birth information, preferred foot, height, and date of birth. The reference answer always follows the provided database. We organize questions into three levels, from single-table lookup to joins and temporal reasoning, to test how grounding changes with reasoning difficulty. We evaluate both commercial closed-source models and opensource models. Our results show that strong instruction-tuned models can often handle direct lookup questions, even when the table contains counterfactual values. However, as questions require joins, comparison, and temporal reasoning, the gap between original and counterfactual accuracy becomes larger. This counterfactual gap suggests that models become less faithful to the provided database
when the reasoning path is harder and the table conflicts with familiar knowledge. Stronger models and instruction tuning improve overall accuracy, but they do not fully remove this effect. In summary, this paper makes the following contributions: • We introduce ContraTable, a paired original–counterfactual benchmark for testing whether LLMs follow tables or prior knowledge. • We build a consistent counterfactual football database that preserves structure while changing selected real-world facts. • We design questions spanning lookup, joins, and temporal reasoning to study grounding under increasing complexity. • We show across commercial and open-source LLMs that table grounding weakens as reasoning paths become more complex.
2
1.1 million game events, together with country and competition information. Our benchmark focuses on transfers from 2023 through 2025 and the related players, clubs, countries, competitions, games, and events. From this subset, we build two aligned databases: an original database containing real-world facts and a counterfactual database in which selected facts are replaced with values that conflict with real-world knowledge. Both versions preserve the same schemas, identifiers, and relational structure.
3.1
Using the relations available in the dataset, we divide the benchmark into three levels. The level is determined by the database operation required to obtain the answer, rather than only by the number of tables involved. This design connects each natural-language question to a familiar database action and lets us study whether counterfactual grounding changes as the required operation becomes more complex. Level 1 corresponds to single-table lookup. The model only needs to identify one row in one table and read the requested column. Some Level 1 questions include simple row-level computation, such as extracting the year from a date or computing a player’s age from date of birth. These questions are similar to a selection followed by projection in a database query. They test whether the model can locate the correct evidence and return the table value, rather than answering from memorized facts. Level 2 corresponds to join-based lookup. The answer cannot be read from a single table, but the relevant entity or event is directly specified in the question. The model must follow one or more join paths, such as connecting a player to their citizenship country, a club to its country, or a club to its domestic competition. The reasoning is still lookup-based: after the joins identify the target row, the answer is obtained by reading an attribute. This level tests whether the model can preserve table evidence across relational links. Level 3 corresponds to multi-table temporal reasoning. The model must first derive which rows are relevant before reading or comparing their attributes across tables. These questions require operations such as joining transfer records with player, club, or country tables, ordering transfers by date, selecting the latest record before a target date, counting rows, comparing values, or aggregating over a set of records. In database terms, this level goes beyond direct lookup and requires joins together with filtering, ordering, grouping, comparison, or interval reasoning. The increase in difficulty is central to our evaluation. In a direct lookup, the relevant evidence is close to the requested answer. As joins and intermediate operations are added, the model must carry the provided values through a longer reasoning path. The example questions for each level are shown in table 1. This design allows us to test whether the model continues to rely on the table or falls back on familiar pre-trained knowledge, especially when asked to produce a direct answer. To ensure that this comparison reflects reasoning difficulty, all questions use clear, schema-aligned wording, and every included table contributes to a traceable reasoning path. We next describe how we introduce counterfactual facts while preserving the database’s internal consistency.
RELATED WORKS
Prior work on table question answering evaluates capabilities such as cell lookup, fact verification, numerical operations, and compositional reasoning over structured data. Recent studies directly serialize tables for general-purpose LLMs and show that they can perform competitively on established table-QA benchmarks, particularly with few-shot or chain-of-thought prompting [2]. However, performance is sensitive to factors such as table format, row order, prompt design, and table size [22]. Moreover, because these benchmarks commonly contain real-world entities and facts, answer accuracy may conflate reasoning over the supplied table with factual knowledge recalled from pretraining. Related QA research distinguishes knowledge stored in model parameters from evidence supplied in the input. Longpre et al. [14] show that models may prefer memorized answers when contextual evidence conflicts with their parametric knowledge . DisentQA further uses counterfactual passages to separate answers derived from these two sources [18]. These studies motivate conflict-based evaluation as a test of grounding, but primarily consider unstructured textual contexts rather than relational tables. Knowledge-editing research provides the closest methodological precedent. ROME introduced CounterFact, which uses counterfactual fact associations to evaluate editing success, generalization, and specificity [16]. CounterFact+ subsequently strengthened the benchmark by dynamically testing unintended effects on related model outputs [11]. Later benchmarks examine more complex consequences: MQuAKE tests whether edited facts propagate through multi-hop questions, while RippleEdits evaluates logical and semantic ripple effects [25]. However, these benchmarks generally evaluate edited model parameters or isolated textual facts. Our work instead keeps the model fixed and modifies an external relational database while preserving its schema and relationships. This paired original–counterfactual design measures whether an LLM follows table evidence across lookup, joins, aggregation, comparison, and temporal reasoning rather than reverting to memorized world knowledge.
3
Question Design
DATASET CONSTRUCTION
We construct a paired table-QA benchmark from a structured football dataset collected from Transfermarkt [6]. The source data consists of joinable CSV files covering more than 37,000 players, 400 clubs, 80,000 games, 99,000 transfers, 1.8 million appearances, and 2
Table 1: The three question levels in our benchmark. Level
Definition
Example
Level 1: Single-table lookup The answer is obtained from one row in one table through selection and projection. A What capital city is listed for {country name}? (61 questions) question may also require a simple row-level calculation, such as extracting a year or computing an age. No joins or multi-row reasoning are needed. Level 2: Multi-table lookup The answer requires one or more joins across tables. The target entity or event is directly What capital city is listed for {player name}’s citizenship (79 questions) identified, so the model follows the join path and reads the requested attribute. No aggre- country? gation or temporal interval inference is required. Level 3: Multi-table temporal The model must first derive the relevant rows through joins and operations such as On {target date}, was {player name}’s club country on reasoning filtering, ordering, aggregation, comparison, or temporal interval reasoning. The answer the same continent as the player’s citizenship country? (74 questions) therefore requires multi-step database reasoning rather than direct lookup.
3.2
Counterfactual Design
setting. The question wording and reasoning path stay the same, while the table values may differ between the original and counterfactual databases. This design lets us test whether a model follows the provided table evidence or falls back on its prior knowledge. To keep inputs comparable across questions, we limit the number of rows provided to the model. Ground-truth evidence rows are always included, and the remaining rows are sampled as distractors from the required tables.
The counterfactual database preserves the original schema, identifiers, and relational structure while replacing selected real-world facts with valid conflicting values. The evaluated model is not told which database it receives, and the reference answer always follows the provided tables. We modify attributes that are likely to be known by LLMs without breaking database relations. These include country capitals, continents, and confederations; club countries, domestic competitions, and stadiums; and player birth information, citizenship, date of birth, preferred foot, and height. Internal consistency is central to the design [5, 8]. Categorical values are reassigned through one-to-one chain swaps, with no selfmapping or repeated replacement. Related fields, such as country names and country codes, are updated together. Numeric and date fields receive small valid perturbations, while structural records, including transfers and game events, remain unchanged. With random seed 0, we modify values in the country, club, and player tables. With the original and counterfactual databases aligned, we next generate matched questions that share the same wording and reasoning logic but reflect their respective data.
3.3
4.2
Evaluation
For each question, the evaluated model generates an answer together with a brief explanation. We evaluate the answer and explanation as one complete response, rather than judging only the final answer string. This is important because a model may sometimes produce the correct answer while giving an explanation that does not match the table evidence. We use GPT-4o as a binary evaluator. Given the model response and the reference answer with its supporting evidence, GPT-4o determines whether the response is correct or incorrect. The same evaluation procedure is used for both the original and counterfactual settings.
4.3
Question Generation
Results
Table 2 reports model accuracy across the three question levels on both the original and counterfactual databases. Since each counterfactual question keeps the same wording and reasoning path as its original version, the accuracy drop from the original database to the counterfactual database is the main signal we study. We refer to this drop as the counterfactual gap. A larger gap means that the model becomes less reliable when the table evidence conflicts with familiar real-world facts. We first examine how this gap changes as questions move from direct lookup to more complex reasoning.
We first choose the target attributes, join paths, and question level. GPT-5.5 drafts candidate templates from these specifications. We then review each template manually, refine its reasoning path, and select entities for its placeholders. A deterministic program follows the approved path to compute the reference answer and collect the supporting rows. GPT-5.5 is used only to draft the question wording. Each template is instantiated independently against the original and counterfactual databases. This keeps the answer, explanation, placeholders, and provenance consistent with each data setting while preserving the same question and reasoning structure.
Counterfactual gaps reveal reliance on pre-training knowledge. Level 1 usually has the smallest counterfactual gap because it only requires direct lookup. The gap is nearly zero for strong models: GPT-5.4-Mini has a 0-point gap, and Qwen3.5-9B has a 1.64-point gap. This suggests that when the task is simply to read one cell, models can often follow the provided table even when the value is counterfactual. However, once questions require joins or longer reasoning, the gap becomes much larger. GPT-5.4-Mini increases to a 10.13-point gap at Level 2 and an 18.92-point gap at Level 3. Qwen3.5-9B and Gemma-4-E2B-it also show large Level 2 gaps of 20.25 and 31.65 points, respectively. This pattern supports our main assumption: as table reasoning becomes harder, models are more likely to fall back on familiar
4 RESULT 4.1 Experimental Overview We evaluate both commercial and open-source models on ContraTable. The commercial models include Gemini-3.1-Flash-Lite and GPT-5.4-Mini. In addition, we include open-source models because many table-based applications involve private data, where users may prefer to run the model locally instead of sending tables to an external API. The open-source models include Gemma-4-E2B-it, Gemma-4-E4B-it, Qwen3.5-2B, Qwen3.5-9B, Llama-3.1-8B, Llama3.1-8B-Instruct, Llama-3.2-3B-Instruct, and Llama-3.2-1B-Instruct. Each question is evaluated under a paired original–counterfactual 3
Table 2: Accuracy (%) on the original and counterfactual databases.
Level 1 Model Gemini-3.1-Flash-Lite GPT-5.4-Mini Qwen3.5-9B Gemma-4-E4B-it Gemma-4-E2B-it Llama-3.1-8B-Instruct Qwen3.5-2B Llama-3.2-3B-Instruct Llama-3.2-1B-Instruct Llama-3.1-8B
Original 98.36 100.00 98.36 96.72 91.80 88.52 80.33 73.77 45.90 18.03
Level 2
Counter. Original 100.00 100.00 96.72 98.36 86.89 68.85 65.57 57.38 34.43 1.64
Overall
Counter.
Original
Counter.
Original
Counter.
88.61 84.81 69.62 72.15 43.04 35.44 34.18 16.46 7.59 1.27
94.59 77.03 66.22 59.46 45.95 40.54 43.24 29.73 28.38 0.00
89.19 58.11 55.41 44.59 33.78 31.08 35.14 28.38 22.97 0.00
95.79 90.19 84.11 82.24 69.63 68.22 61.21 42.52 26.64 5.14
92.06 79.91 72.43 70.09 52.34 43.46 43.46 32.24 20.56 0.93
94.94 94.94 89.87 92.41 74.68 78.48 63.29 30.38 10.13 0.00
knowledge instead of strictly following the database. The accuracy drop is therefore not only a difference between two datasets. It also reflects a change in how the model appears to make decisions. We manually inspected the generated explanations for changed predictions and found that, in many cases, the model starts to justify its answer using real-world knowledge rather than the counterfactual values shown in the table. This behavior is especially clear in Level 3, where the model must combine evidence across multiple rows or reason over time. In these cases, the table still contains the required evidence, but the model is less faithful to it when the evidence conflicts with what the model already knows. The trend is not perfectly monotonic for all models. For weaker models, original accuracy can already be low, leaving less room for a further drop on the counterfactual database. Therefore, the counterfactual gap is most meaningful when the model first performs reasonably well on the original database. Gemini-3.1-Flash-Lite is a small Level 1 outlier, where counterfactual accuracy is slightly higher than original accuracy. We manually checked this case and found that the only original error comes from wording ambiguity rather than counterfactual reasoning: the model finds the correct cell value, but interprets the question as asking whether the literal word “confederation” appears, and answers “No.”
Level 3
Level 3 original accuracy from 43.24% to 66.22%, and counterfactual accuracy from 35.14% to 55.41%. Similarly, Gemma-4-E4B-it improves over Gemma-4-E2B-it on both original and counterfactual Level 3 questions. These results support our assumption from another angle: model strength improves table reasoning, but it does not fully prevent the model from being influenced by prior knowledge when the table contains counterfactual facts. We next isolate another factor that may affect this behavior: instruction tuning. Instruction tuning matters. To isolate the effect of instruction tuning, we compare Llama-3.1-8B with Llama-3.1-8B-Instruct. Instruction tuning improves overall original accuracy from 5.14% to 68.22%, showing that the model needs to follow the task instruction and use the provided tables. However, the counterfactual accuracy of Llama-3.1-8B-Instruct is still much lower than its original accuracy. This suggests that instruction tuning helps models read and use tables, but is not sufficient to remove the influence of prior knowledge.
5
CONCLUSION
This paper studies whether LLMs answer table-based questions by following the provided database or relying on prior knowledge. We introduce ContraTable, a paired original–counterfactual benchmark. This setup lets us compare model behavior when table evidence agrees with real-world knowledge and when it conflicts with it. Our results show that models handle direct lookup well, but become less faithful as questions require joins, comparison, and temporal reasoning. Stronger models and instruction tuning improve accuracy, but do not fully remove this behavior. In future work, we plan to extend the benchmark with more reasoning-intensive questions, including richer temporal conditions and longer reasoning paths. We also plan to study step-by-step answer generation, where the model is asked to produce intermediate reasoning before giving the final answer. Finally, we will go beyond end-to-end evaluation by analyzing text-to-SQL pipelines, separating question understanding, SQL generation, execution, and answer generation, and comparing them with direct LLM table answering in both accuracy and running time.
Stronger models help, but do not remove the gap. We next ask whether stronger models are better at staying faithful to the table when reasoning becomes harder. Level 3 is the most useful setting for this comparison, as it requires multi-table, comparison, or temporal reasoning. Gemini-3.1-Flash-Lite performs best at this level, reaching 94.59% accuracy on the original database and 89.19% on the counterfactual database. GPT-5.4-Mini also performs well on the original Level 3 questions, with 77.03% accuracy, but drops to 58.11% in the counterfactual setting. Among open-source models, Qwen3.5-9B and Gemma-4-E4B-it are the strongest on Level 3, reaching 66.22% and 59.46% on the original database, and 55.41% and 44.59% on the counterfactual database. This shows that stronger models are better at following longer table reasoning paths, but the counterfactual gap remains. The same pattern appears within model families. Moving from Qwen3.5-2B to Qwen3.5-9B improves
4
REFERENCES
processing. 7052–7063. [15] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2200–2209. [16] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems 35 (2022), 17359–17372. [17] Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229 (2022). [18] Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2023. Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 10056–10070. [19] Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 1470–1480. [20] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 conference on empirical methods in natural language processing. 2383–2392. [21] Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model?. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). 5418–5426. [22] Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 645–654. [23] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 conference on empirical methods in natural language processing. 3911–3921. [24] Victor Zhong, Caiming Xiong, and Richard Socher. [n.d.]. Seq2SQL: Generating structured queries from natural language using reinforcement learning (2017). arXiv preprint arXiv:1709.00103 ([n. d.]). [25] Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. 2023. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 15686–15702. [26] Xuanhe Zhou, Zhaoyan Sun, and Guoliang Li. 2024. DB-GPT: Large Language Model Meets Database. Data Science & Engineering 9, 1 (2024), 102. [27] Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (volume 1: long papers). 3277–3287.
[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision. 2425–2433. [2] Wenhu Chen. 2023. Large language models are few (1)-shot table reasoners. In Findings of the association for computational linguistics: EACL 2023. 1120–1130. [3] Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020. 1026–1036. [4] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. 2021. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 3697–3711. [5] Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics 12 (2024), 283–298. [6] David Cariboo. [n.d.]. Football Data from Transfermarkt. https://www.kaggle. com/datasets/davidcariboo/player-scores. Accessed: 2026-06-21. [7] Raul Castro Fernandez, Aaron J Elmore, Michael J Franklin, Sanjay Krishnan, and Chenhao Tan. 2023. How large language models will disrupt data management. Proceedings of the VLDB Endowment 16, 11 (2023), 3302–3309. [8] Peter Hase, Thomas Hofweber, Xiang Zhou, Elias Stengel-Eskin, and Mohit Bansal. [n.d.]. Fundamental problems with model editing: How should rational belief revision work in llms?, 2024. URL https://arxiv. org/abs/2406.19354 ([n. d.]). [9] Benjamin Heinzerling and Kentaro Inui. 2021. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 1772–1791. [10] Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pretraining. In Proceedings of the 58th annual meeting of the association for computational linguistics. 4320–4333. [11] Jason Hoelscher-Obermaier, Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. 2023. Detecting edit failures in large language models: An improved specificity benchmark. In Findings of the Association for Computational Linguistics: ACL 2023. 11548–11559. [12] Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1821–1831. [13] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7 (2019), 453–466. [14] Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 conference on empirical methods in natural language
5