Official Review · ICLR 2025

This paper tackles the problem of benchmarking temporal reasoning using multimodal foundation models. First, the paper establishes some principles that temporal reasoning benchmarks must follow in order to test temporal reasoning capabilities, and shows that existing benchmarks fail to cater to these principles. Second, a new manually annotated benchmark, comprising six question types with some counterfactual modifications, is proposed. Third, several existing models are benchmarked demonstrating a big difference between human performance and best model's performance. And finally, some ablation studies are presented that show how one can improve models capabilities on this benchmark. The paper is written well and easy to follow. It addresses an important area in video understanding research, where several benchmarks are known to have shortcomings in actually requiring all the frames of a video [1]. - The benchmark is manually curated in a three-stage process, potentially yielding very high-quality QA annotations - Several models including proprietary ones lag significantly behind humans. This verifies that the benchmark is indeed difficult for existing models. - A commendable effort is made in conceptualizing the principles defining how to check whether a benchmark requires temporal reasoning, and its shown that TVBench is way ahead of other benchmarks in adhering to these principles as shown in Tables 1-4 - Some of the ablations / analysis about existing models' capabilities are interesting, i.e. the plateauing at 8 frames compared with humans, exploiting common sense shortcuts etc. [1]: Revealing Single Frame Bias for Video-and-Language Learning The main weakness in my opinion is that the defined principles and ways of measuring benchmarks' adherence to these principles is not very rigorous. 1: Single frame insufficiency: - Why is the definition limited to a single frame? What about QA pairs that can be answered in just a few frames, say 2 frames? In the abstract, it's argued that such a scenario is bad for temporal reasoning benchmark, however, this principle will consider such a question as good. - In the second implementation, a single hand-picked frame is compared with uniformly sampled 16 frames. This seems a bit unfair to the 16 frame case. Shouldn't we compare hand-picked 16 frames with hand-picked 1 frame to get a true sense of single frame insufficiency? 2: Frame order sensitivity: The definition assumes that frame order sensitivity is essential for a good temporal reasoning benchmark. However, a powerful model human can easily figure out the temporal order in a lot of cases. So, I don't know if this needs to be a hard requirement. 3: Frame contribution parity: Implementation-wise, again why only a single frame is considered? The proposed implementation will not give any signal for a QA pair requiring 2 frames. For example, a benchmark that simply contains state transition questions requiring exactly 2 frames will get perfec