Official Review · ICLR 2025

The paper proposes Face-Human-Bench, a hierarchical benchmarking framework aimed at evaluating the ability of multi-modal large language models (MLLMs) to understand and interpret faces and human figures. The framework categorizes tasks across multiple levels of comprehension, from facial attributes and actions to complex reasoning like social relationships and spatial relationships. It comprises 900 development set problems and 1,800 test problems, supporting both English and Chinese. By leveraging a semi-automatic data pipeline, the authors collect and annotate images from multiple datasets to form this comprehensive benchmark. Evaluations on 25 MLLMs reveal insights on model performance relative to the task type, the relative positioning of target objects, and the effectiveness of Chain-of-Thought (CoT) prompting. Results suggest areas where specialist models surpass MLLMs, emphasizing the benchmark’s role as a holistic tool to gauge MLLM performance on face and human understanding. - **Comprehensive Benchmarking Framework**: The face-human bench spans many abilities, providing a holistic evaluation of multimodal assistants’ capabilities in the face and human understanding. - **New Metrics and Evaluation Protocols**: The paper introduces RPSS to measure sensitivity to the relative position of targets and percentile recall to assess retrieval in large galleries. These metrics provide nuanced insights, aiding model development. - **Multi-Language Support**: The benchmark ensures broader applicability across language barriers by supporting both English and Chinese. - **Empirical Findings on MLLM Performance**: The evaluation of 25 MLLMs is thorough, providing insights into model performance across diverse tasks and the potential utility of Chain-of-Thought prompting. - **Limited Discussion on Dataset Biases**: Although the benchmark includes diverse tasks, the paper could expand on potential biases in the benchmark datasets, especially considering the variability in demographic representations in face and human recognition tasks. - **Generalizability to Other Tasks**: The applicability of Face-Human-Bench to tasks beyond face and human recognition remains unclear. Expanding on how these benchmarks might generalize to other domains would add depth. - **Impact of CoT Prompting is Limited for Open-Source Models**: While Chain-of-Thought prompting improves closed-source models, the performance gain is minimal for open-source counterparts, indicating limitations in the broader applicability of CoT. 1. Dataset Composition and Diversity: Could you provide more details on the demographic diversity of the benchmark datasets, particularly for face recognition tasks? 2. Transferability Across Domains: Do you envision Face-Human-Bench or its methodologies applying to multi-modal tasks outside of face and human recognition? 3. Insights on CoT Prompting Performance: Do you hypothesize that the limited effectiveness of CoT prompting on open-source models is