Official Review · ICLR 2024

This paper proposes a benchmarking dataset for temporal reasoning named TRAM. The TRAM benchmark evaluation is exemplified with BERT-style pretrained-finetuned models and GPT style prompting-based LLMs. The authors intended to provide TRAM as a comprehensive benchmark to spur the LLM research progress in temporal reasoning capabilities. • A comprehensive benchmark covers various temporal reasoning abilities: ordering, frequency, duration, typical time, ambiguity, arithmetic, relation, temporal NLI, causality, storytelling. • The overall size of the dataset is big, being 526,068 problems for benchmarking. • Pretraining-finetuning and prompting paradigms of LLMs are both evaluated using the benchmarking providing reasoning evaluation conclusions. It is a good starting point from which the community can evolve the LLM techniques or other LLM alternatives for temporal reasoning. • The benchmark currently is only in the form of multi-choice questions. • The sizes of different categories of problems are imbalanced. For example, causality is of only 600 problems. This might render the benchmarking evaluation results misguiding. Especially for the pretraining-finetuning paradigms. • The texts are mostly from existing datasets. Latest LLMs might have seen them through the pretraining phrase crawled dataset. It might make the benchmarking results over-estimate the performance of LLMs in the temporal abilities. It is an issue beyond just the temporal reasoning abilities extending to all other LLM benchmarking datasets. It calls for more organic benchmarking approaches for LLMs and their iteration which can be pretrained with all kind of available data in human world including benchmarking data. 1. For the terms "commonsense", "analogy inference", "comparison" and so on, it would be better to have a formal definition and ensure that the datasets and the reasoning follow the formal definitions with verifiable criteria (automatically verifiable would be even better). 2. Page 7, it would be better to include a more comprehensive list of the settings for human expert annotators evaluation. For example, how the experts are drawn from the population, how to ensure they are capable experts or their level of expertise. How will an ordinary person performance comparing to experts? Such a systematic study of human experiments might also provides hints for comparing human performance variation with LLM performance variation. 3. Page 18, appendix B, for self-contained please detail a little bit more for SP with examples. The whole section is essentially about CoT with little information regarding how SP is constructed.