Official Review · ICLR 2026

This paper introduces VidText, a new benchmark designed to comprehensively evaluate video text understanding in Large Multimodal Models (LMMs). The authors identify a significant gap in existing video understanding benchmarks. The paper evaluates 18 LMMs on VidText. The study also validates the benchmark's design through ablations, showing that performance drops when key components are masked. * **Comprehensive Benchmark Design:** The hierarchical (video/clip/instance) and paired (perception/reasoning) task structure is a major strength, enabling a much more nuanced evaluation than previous benchmarks. * **Extensive Empirical Analysis:** The evaluation of 18 models provides a valuable snapshot of the field's capabilities and limitations. The ablation studies effectively validate the benchmark's design choices. * **Outdated Model Comparison:** A significant weakness is the omission of the very latest flagship models (e.g., Gemini 2.5 Pro, GLM-4V). This quickly diminishes the paper's relevance and the persuasiveness of its conclusions about the current state-of-the-art. * **Limited Discussion on Data Biases:** While the dataset is diverse, there is no discussion of potential biases in the video sources (e.g., geographic or cultural biases from YouTube) or the annotation process that might affect model generalization. * **Question on Human Evaluation:** Please provide a detailed description of the human evaluation protocol for the 89.5% baseline. What were their backgrounds, and what was the inter-annotator agreement? **Suggestion:** Strengthen this section with rigorous methodology to make the human benchmark a reliable point of comparison. * **Suggestion on Generalization:** Perform an analysis or discuss potential biases within the VidText dataset itself. Could the performance gaps be partly due to specific types of videos or text that models struggle with? This would add depth to the analysis of model shortcomings.