Official Review · ICLR 2026

This work presents IndicVisionBench, a large-scale benchmark designed to evaluate Vision-Language Models (VLMs) on culturally grounded and multilingual tasks focused on the Indian subcontinent. The benchmark comprises 5,000 unique images and over 37,000 question-answer pairs spanning 13 cultural topics across English and 10 Indian languages. The evaluation framework encompasses three primary tasks: Visual Question Answering (VQA) with six question types, Optical Character Recognition (OCR), and Multimodal Machine Translation (MMT). The authors evaluate models, including both proprietary systems (Gemini-2.5, GPT-4o) and open-source variants, revealing substantial performance gaps particularly for low-resource languages and culturally specific content.​ - **Comprehensive Multi-task Framework**: The benchmark's tri-modal evaluation approach (VQA, OCR, MMT) provides a holistic assessment of VLM capabilities beyond simple question-answering. The inclusion of adversarial questions represents a good approach to probing deeper cultural knowledge beyond surface-level recognition. - **Linguistic Coverage** : The benchmark covers 10 Indic languages with diverse scripts. - **Limited Methodological Novelty**: While the cultural focus is valuable, the benchmark construction methodology largely follows established paradigms without introducing novel evaluation frameworks. The reliance on synthetic question generation using commercial models (Gemini-1.5-Flash, Gemini-2.5-Flash) raises concerns about potential biases inherited from these models. - **Self-Preferential Bias and Evaluation Circularity**: The most severe methodological flaw is the systematic use of Gemini models throughout the benchmark construction pipeline, followed by its evaluation on the same benchmark. Gemini-1.5-Flash and Gemini-2.5-Flash generate synthetic captions and QA pairs​ and translations across languages​. This creates a circular evaluation where Gemini is assessed on data it helped generate, violating fundamental evaluation principles established in contamination literature. The consistently superior performance of Gemini-2.5 across all tasks becomes suspect given this methodological flaw.​ - **Absence of Design Choice Validation**: The paper provides no systematic benchmarking for the choice of Gemini variants. While mentioning a "small pilot study and cost considerations," no details, quality assessments, or comparative analysis with alternative generation models are provided. Table 6 shows only cost comparisons without quality metrics. The benchmark's coverage appears skewed toward certain linguistic groups (Hindi at 26.8% of QA pairs) without adequate justification for this distribution. - **Complete Lack of Quality Control Statistics**: Caption Quality: No validation of synthetic caption accuracy against image content using multimodal embeddings or human verification statistics​. Human Refinement Rates: The claim that "Human reviewers refined all output