I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning Shibo Gao∗

arXiv:2608.07417v1 [cs.CV] 7 Aug 2026

Beijing Jiaotong University Beijing, China HUJING Digital Media & Entertainment Group Beijing, China MAIS Institute of Automation, Chinese Academy of Sciences Beijing, China

Chongxiao Wang∗ Chenglong Huang Jie Ma Haolin Shi Fei Ding Jing Li

Qiang Lyu

Yangyang Liu

School of Computer Science and Technology University of Chinese Academy of Sciences Beijing, China

MAIS Institute of Automation, Chinese Academy of Sciences Beijing, China

HUJING Digital Media & Entertainment Group Beijing, China

Yang Liu

Jun Liu

Linlin Huang

Peipei Yang✉

College of Electronic and Information Engineering Tongji University Shanghai, China

School of Computing and Communications Lancaster University Lancaster, United Kingdom

Beijing Jiaotong University Beijing, China

MAIS Institute of Automation, Chinese Academy of Sciences Beijing, China

Abstract Real-world video reasoning often involves multimodal, multi-source inputs, whereas existing video reasoning tasks typically assume a simplified video-text setting, limiting identity matching and personcentric reasoning. To bridge this gap, we introduce the Identityconditioned Queries (ICQ) task, in which models are required to jointly associate and interpret an input video and a reference image of a person, and leverage this conditioning to address identity grounding, behavior understanding, and temporal reasoning, among other challenges. Building on ICQ, we present ISYV (I Seek You in Videos), a systematic solution comprising three components: (1) ISYV-Bench, a challenging evaluation benchmark with 1,377 real-world complex videos and 1,377 question–answer pairs, organized into six difficulty levels spanning capabilities from identity recognition to causal reasoning; (2) ISYV-75K, a large-scale training set of 75K high-quality samples constructed via automated annotation, multi-stage verification, and manual review; and (3) ISYV-Framework, containing an ICQ-oriented model and training strategy for learning to exploit informative video shots without additional shot-level annotations. Extensive experiments show that both mainstream closed-source and open-source MLLMs struggle on ISYV-Bench, especially in cross-domain identity matching and longhorizon tracking. ISYV-Model outperforms strong baselines and in some aspects approaches closed-source performance. Overall, ISYV ∗ Both authors contributed equally to this research.

This work is licensed under a Creative Commons Attribution 4.0 International License. MM ’26, Rio de Janeiro, Brazil. © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2213-4/2026/11 https://doi.org/10.1145/3767308.3835378

provides a unified task definition, scalable datasets/benchmarks, and modeling insights for person-centric video reasoning.

CCS Concepts • Computing methodologies → Computer vision tasks.

Keywords MLLM, Video Reasoning, Post-training, Benchmark, Framework ACM Reference Format: Shibo Gao, Chongxiao Wang, Chenglong Huang, Jie Ma, Haolin Shi, Fei Ding, Jing Li, Qiang Lyu, Yangyang Liu, Yang Liu, Jun Liu, Linlin Huang, and Peipei Yang. 2026. I Seek You in Videos: Identity-Conditioned Queries for PersonCentric Video Reasoning. In Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/ 3767308.3835378

1

Introduction

With the rapid progress of Multimodal Large Language Models (MLLMs) in recent years, video reasoning has become a key task for evaluating models’ higher-order cognitive capabilities and has emerged as an active research area [3–5, 23, 50]. Existing work on video reasoning spans major sub-tasks such as video captioning, video question answering, and video grounding [16, 21, 24, 45, 49]. However, most studies still follow the traditional “video–text” bimodal paradigm, where models only process video content and generate textual outputs [18, 38, 58]. The conventional setting overlooks the heterogeneous, multi-source inputs commonly encountered in real applications. For example, users often issue queries conditioned not only on the video itself but also on reference images

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil.

Figure 1: Comparison of the ICQ task with traditional video reasoning. Notably, the ICQ’s different levels are anchored to the progression of human cognitive abilities.

and specific object descriptions. This gap between simplified formulations and practical demands limits the applicability of current models in complex real-world scenarios. Identity-conditioned Queries (ICQ) is a challenging person-centric video reasoning task. Unlike conventional video reasoning [44, 55], ICQ requires models to reason over both a complex video and a reference image of a target person. Since the reference image may come from a different source, large appearance gaps in clothing, lighting, makeup, and other factors make identity matching difficult. Models must therefore identify the same person across these variations while handling tasks such as localization, action recognition, temporal grounding, and causal reasoning. Overall, ICQ presents three key challenges: (i) integrating heterogeneous inputs, (ii) localizing and tracking the target person in complex scenes, (iii) cross-domain identity matching with identity-aware understanding. These abilities are important for real-world applications such as video retrieval, personalized analytics, film production, and intelligent surveillance. Although some prior works explore related directions [1, 20, 28, 31], they focus mainly on image-level interactions and do not address video-image-text reasoning or provide high-quality benchmarks. To this end, we propose ISYV (I Seek You in Videos), a systematic solution for the ICQ task, comprising three components: ISYVBench, an ICQ-specific evaluation benchmark; ISYV-75K, a largescale training set; and ISYV-Framework, a dedicated model with an ICQ-oriented training strategy, detailed through our contributions below. The comprehensive experiments on ISYV-Bench show that existing mainstream MLLMs perform far from satisfactorily on the ICQ task. As shown in Fig. 2, existing closed-source models still exhibit a substantial gap to human performance on ISYV-Benchmark, while mainstream open-source models perform even worse, with overall accuracy below 40%. Under the 7B setting, our proposed model increases the average accuracy to 57%, yielding a significant improvement over the base model and achieving performance that

Shibo Gao et al.

is broadly comparable to closed-source models. Detailed ablation studies and case analyses reveal key bottlenecks of current MLLMs in cross-domain identity matching and long-horizon tracking, providing important insights for future development in this area. In summary, our main contributions are as follows: • We formulate Identity-Conditioned Queries (ICQ), a practical person-centric task requiring joint reasoning over a video, a reference image of the target person, and a text query. • We build a comprehensive data suite for ICQ: ISYV-Bench, a 1,377-sample evaluation benchmark organized into a sixlevel cognitive hierarchy [6], and ISYV-75K, a 75K-sample training set spanning diverse scenes, identities, and question types. • We propose ISYV-Framework, featuring an ICQ Module for reference-image abstraction and an ICQ-oriented training strategy with a novel ESR reward that selects identityrelevant evidence without shot-level annotations. With 7B parameters, it rivals closed-source MLLMs. • We comprehensively evaluate mainstream MLLMs on ISYVBench, revealing key bottlenecks in cross-domain identity matching and long-horizon tracking, with ablations offering insights for future research.

2 Related Works 2.1 Video Reasoning Benchmarks Current mainstream video reasoning benchmarks primarily encompass multiple sub-tasks, including Video Question Answering (VQA) [55], Video Grounding [10], and Video Captioning [53], aimed at comprehensively evaluating models’ understanding and analytical capabilities on individual videos [49]. Early benchmarks such as TVQA [21] assess models’ video comprehension through closed- or open-ended questions on video clips. Subsequently, works like NExT-QA [45] and LongVideoBench [44] introduce spatialtemporal reasoning challenges, further increasing evaluation complexity. Recently, benchmarks such as MVBench [24] and VideoMME [9] have expanded the diversity of task types. However, these benchmarks uniformly adopt a "video-text" dual-modal input paradigm and rarely address videos containing complex shot transitions. Beyond general-purpose video reasoning, domain-specific video analysis has been studied extensively, ranging from video anomaly detection [12–15] to long-term multi-object tracking [11], where associating and following specific subjects over time is essential; these methods, however, operate over fixed label spaces rather than open-ended, identity-conditioned reasoning. IDA-VLM [20], PLVM [28] and others [1, 31] attempt to incorporate person images as references to assist models in understanding images containing specific individuals. Nevertheless, this work remains confined to "image-text" paired tasks, without exploring multi-source input scenarios involving "complex videos, cross-shot person references, and textual queries."

2.2

RL-based Fine-tuning

With the rapid advancement of MLLMs, numerous studies have significantly enhanced models’ reasoning capabilities through RL approaches [8, 37, 51, 57]. The recently proposed Group Relative Policy Optimization (GRPO) [30] algorithm stabilizes the training

I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil.

process and improves training efficiency by computing group wise normalized rewards, achieving substantial performance gains in image and video understanding tasks. Furthermore, building upon the principles of GRPO, researchers have proposed various improved training strategies, such as DAPO [48], SRPO [40], GFPO [32] and Multi-GRPO [25], the latter extending group-relative estimation to multiple reward signals, further advancing the field. In parallel, a complementary line of work reduces the deployment cost of large models through compression, quantization, and distillation [34, 35]. However, to date, no study has systematically validated the effectiveness of GRPO and its variants on ICQ or similar tasks.

3 ISYV-75K & ISYV Benchmark 3.1 Overview The ISYV-75K dataset comprises 24,150 unique video clips and 74,578 question-answer pairs, with each video accompanied by a corresponding character patch to guide the model in answering questions. The ISYV Benchmark consists of 1,377 high-quality video clips, 1,377 corresponding character patches, and 1,377 questionanswer pairs. To ensure data quality, we collected over 100,000 video clips from authentic film and television productions, encompassing a wide range of video durations and varying degrees of visual complexity. During the video selection process, we emphasized shot transition complexity, scene diversity, and character distinctiveness, ensuring that the resulting dataset is sufficiently challenging while remaining suitable for the ICQ task.

3.2

Figure 2: Radar chart of model performance on ISYVBenchmark across the six levels. The ∗ indicates the model has been trained.

ISYV-75K Training Dataset

3.2.1 ICQ Task Definition. As aforementioned, the Identity Conditioned Queries (ICQ) task represents a further extension of current mainstream Video Reasoning in the domain of human-centric cognition. Specifically, the model is required to simultaneously process a long-form video containing complex shot transitions and a reference image of a person, where the latter serves to guide the model in focusing on a specific character when answering questions. The model must comprehend the relationship between the character in the reference image and the video content, thereby accurately answering related questions. Fig. 1 illustrates the distinction between our ICQ task and mainstream Video Reasoning tasks. 3.2.2 Hierarchical Tasks. We position the ICQ task as an extension of Video Reasoning capabilities within human-centric scenarios. Consequently, the ICQ task must encompass as many sub-task types as possible to comprehensively evaluate model performance. Based on this, we align the design philosophy of the ICQ task with the developmental trajectory of human cognitive abilities [6]. Specifically, we devise six hierarchical task levels with progressively increasing difficulty, where each level comprises several further subdivided question types, and each level corresponds to a specific human cognitive capability. Fig. 3 presents exemplar tasks across different levels. Furthermore, Table. 2 elaborates the correspondence between each level and its associated human cognitive ability. Through this design, ICQ enables comprehensive evaluation of model performance and establishes connections between MLLMs and human cognitive science.

Figure 3: Schematic diagram of the hierarchical level design in ICQ.

3.2.3 End-to-end automated construction process. ISYV-Benchmark is constructed through manual curation and verification based on ISYV-75K. We therefore first describe the semi-automatic pipeline for building ISYV-75K. We collected over 100,000 video clips from diverse audiovisual sources, including films, TV series, and anime. During preliminary filtering, clips containing no visible characters were removed. The retained clips were then analyzed in depth by Gemini-2.5-Pro [5]. Specifically, we first applied TransNet V2 [36] to segment each video with timestamps. Gemini-2.5-Pro then generated detailed descriptions for each shot, covering environmental context and ID-tagged character actions. Finally, these per-shot analyses were aggregated, and the model was invoked again to produce comprehensive structured annotations encompassing detailed shot descriptions, character appearance attributes, dialogue and narration, and an overall narrative analysis of the video. Since character bounding boxes are produced by the VLMs, we employed Qwen3-VL-32B [3] to perform multiple rounds of quality checks on each reference image, discarding samples that

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil.

Shibo Gao et al.

Table 1: Comparison of our ISYV-Benchmark with mainstream video reasoning datasets. Our benchmark highlights the importance of multimodal input integration and a cognitively inspired hierarchical task design. Benchmark

#Video Num

#QA Pairs

#Duration(s)

Task Num

Annotation

Complex transitions

Multi-input

Human-Aligned Cognition

TVQA [21] MVBench [24] ActivityNet-QA [49] NExT-QA [45] LongVideoBench [44] MMVU [55] Video-MME [9] MLVU [56] Ego-Exo4D [17] EgoExoLearn [19] CrossVid [23]

2,179 3,641 5,800 5,440 3,763 1,529 900 1,730 5,035 747 5,331

15,253 4,000 58,000 52,044 6,678 3,000 2,700 3,102 / / 9,012

11 16 180 44 473 51 1,017 930 156 / 215

3 20 4 2 17 27 12 9 4 4 10

M A M M M M M M+A M M M+A

✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓ ✓ ✓ ✓

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

ISYV-Benchmark

1,377

1,377

50

17

M+A

✓

✓

✓

Table 2: Hierarchical levels of the ICQ task and their corresponding human cognitive abilities. Description

Cognitive Analogy

L1 - Who, what, and which shot? L2 - Same person, different outfit or new location L3 - Subtle details L4 - Gaze, dialogue, exchange, and social dynamics L5 - Movement path within and across shots L6 - Inferring ‘why’ and understanding hidden intent

Basic Perception (BP) Object Permanence (OP) Procedural Observation (PO) Social Cognition (SC) Spatial Memory (SM) Causal Reasoning (CR)

contain multiple persons, are blurry, or lack any visible character. Drawing on mainstream video reasoning datasets and established human cognitive ability frameworks, we then designed the aforementioned task categories along with corresponding QA templates [26, 42]. Qwen3-Max [47] was subsequently used to generate question–answer pairs conditioned on the annotations and QA templates, while descriptions of the reference images were produced by Qwen3-VL-32B. All resulting data underwent a plausibility check via Qwen3-32B [47], after which Qwen3-Max was invoked again to generate CoT rationales and to perform leakage detection. To generate outfit-changed and cross-shot character images, we adopted two complementary strategies. The first leverages the Qwen-Image-Edit [43] to selectively modify visual attributes of a character (e.g., clothing, accessories, hair color, and pose) while preserving the background, thereby producing diverse appearance variants of the same identity. The second exploits established face ReID techniques [2, 29] to retrieve additional appearances of the same character from the same source material, selecting reference images that exhibit maximal differences in pose and background. To ensure identity consistency after editing or replacement, we used Qwen3-VL-32B [3] and Qwen3-32B [47] to independently describe, compare, and verify each image pair, retaining only samples that passed both visual and textual cross-checks. Fig. 4 (a)-(c) illustrate the construction pipeline of ISYV-75K in detail. To ensure the high quality of data obtained through the automated annotation pipeline, we incorporated multiple quality checks after each critical step. The specific locations of these checks are marked as <checkX>. We will present a more comprehensive description of the pipeline, along with the settings and prompts for the annotation process, in the appendix.

Figure 4: A schematic illustration of our pipeline for constructing ISYV-Benchmark and ISYV-75K. (a), (b), and (c) demonstrate the complex yet effective automated pipeline for constructing the foundational training set. We integrate multi-round checking mechanisms <check X> into the automated data construction pipeline. Detailed information for each check is provided in the appendix.

3.3

ISYV Benchmark

3.3.1 Data Selection and Manual Verification. To construct a highquality benchmark, we employ Qwen3-VL-32B [3] to perform multiple rounds of inference on the candidate dataset. After filtering out samples that are excessively simple, we engage a professional annotation team to manually validate the category-balanced filtered data. This process yields a benchmark comprising 1,377 high-quality samples, each consisting of a video, character reference images, and corresponding QA pairs. Notably, since ISYV-75K has undergone

I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil.

multiple rounds of automated quality inspection during its construction, the processing pipeline for ISYV-Benchmark is considerably streamlined, which further corroborates our dual quality assurance for both the training and evaluation sets. Notably, we enforced strict quality control throughout the manual annotation process. Specifically, each sample was cross-validated, error-analyzed, and revised by at least three annotators, followed by a second-round spot check after the completion of the full review process to ensure the accuracy and consistency of the annotations. 3.3.2 Comparison with Existing Benchmarks. We conduct a comprehensive comparison between mainstream Video Reasoning benchmarks and our proposed ISYV Benchmark, with results summarized in Table. 1. In contrast to the prevalent "video-text" input paradigm, ISYV Benchmark innovatively introduces the ICQ mechanism, which requires models to simultaneously process video content and character reference images, thereby enabling personcentric fine-grained video reasoning. ISYV significantly extends the scope of cognitive hierarchy design, question type diversity, and application scenario complexity. Specifically, while existing benchmarks primarily focus on general video reasoning capabilities, ISYV concentrates on personcentric long-form video reasoning, with particular emphasis on cross-shot and cross-scene character tracking and behavior analysis. Our proposed six-level cognitive hierarchy—ranging from basic character recognition to high-level intent inference—aligns with the human cognitive development process, filling the gap in cognitive science-oriented evaluation within current benchmarks. Furthermore, the challenging samples in ISYV Benchmark, including complex shot transitions, character costume changes, and cross-scene tracking, pose more rigorous requirements for models’ spatiotemporal reasoning capabilities and fine-grained visual understanding abilities.

4

ISYV Framework

In this section, we present a detailed description of the ISYV Framework, which we have developed specifically for ICQ task. We begin by introducing the ICQ Module, a component that leverages shorter learnable token sequences to capture key representations of reference images, thereby enhancing the model’s capability for multimodal input alignment. Subsequently, we elaborate on our training strategy and the specifically designed reward function.

4.1

ICQ Module

ICQ extends conventional video reasoning by introducing an additional person reference image, making the effective processing of the reference image and its alignment with the video content a central challenge. Although mainstream multimodal large language models support joint video–image inputs, directly reusing the original visual encoder and simply concatenating video tokens with image tokens leads to two notable issues. First, the model often fails to correctly capture the semantic relationship between the reference image and the video, frequently misinterpreting the reference image as the final frame of the video sequence, thereby conflating the logical relations among inputs. Second, the token sequence produced by the image encoder is typically excessively long, which not only introduces irrelevant noise (e.g., background

Figure 5: A schematic illustration of our ICQ Module architecture and its integration with MLLM.

textures and illumination variations) but also incurs unnecessary computational overhead. To address these limitations, we propose the ICQ Module, a simple yet effective architecture. Its core mechanism lies in compressing the reference image into a compact representation through learnable tokens and facilitating interaction with image features via attention mechanisms, thereby extracting person-specific features that are maximally relevant to the video content [22, 52]. Concretely, the workflow is as follows. We first initialize a short sequence of learnable tokens and apply self-attention to model their internal dependencies: (1) (0) (0) (0) (0) T𝑖𝑐𝑞 = SelfAttn(T𝑖𝑐𝑞 , T𝑖𝑐𝑞 , T𝑖𝑐𝑞 ), T𝑖𝑐𝑞 ∈ R𝑁𝑞 ×𝑑 ,

(1)

(0) where T𝑖𝑐𝑞 represents the initialized learnable tokens sequence, SelfAttn(·) denotes the self-attention mechanism, 𝑁𝑞 is the number of learnable tokens, and 𝑑 is the hidden dimension. Next, we use these learnable tokens as Queries and the image token sequence output by the original image encoder as Keys and Values, performing cross-attention to obtain an initial representation of the reference image: (2) (1) T𝑖𝑐𝑞 = CrossAttn(T𝑖𝑐𝑞 , T𝑖𝑚𝑎𝑔𝑒 , T𝑖𝑚𝑎𝑔𝑒 ), T𝑖𝑚𝑎𝑔𝑒 ∈ R𝑁𝑖 ×𝑑 ,

(2)

where CrossAttn(·) denotes the cross-attention mechanism, T𝑖𝑚𝑎𝑔𝑒 represents the original image token sequence, 𝑁𝑖 is the number of image tokens from original vision encoder. Overall, ICQ Module compresses the reference image into a compact representation using learnable tokens, alleviating the semantic confusion and computational inefficiency caused by naive video–image token concatenation. A schematic illustration of our proposed ICQ-Module and the overall ISYV-Model architecture is provided in Fig. 5.

4.2

Two Step Training Strategy

Regarding the training strategy, we follow mainstream post-training methodologies and adopt a two-stage pipeline [39, 41, 54]. We first perform supervised fine-tuning as a cold-start phase, enabling the model to acquire foundational knowledge and learn the target output format. Subsequently, we conduct reinforcement fine-tuning using the GRPO algorithm to further enhance model performance.

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil.

Shibo Gao et al.

To construct data suitable for different training stages, we first perform quality filtering on the chain-of-thought samples in ISYV75K. Specifically, we deploy Qwen3-32B [47] locally to score each CoT sample on a scale from 0 to 10 based on reasoning quality and label leakage risk, where a score of 10 indicates no label leakage with sound and complete reasoning, while a score of 0 indicates evident label leakage, such as explicitly referencing the original video annotations during the reasoning process. Based on these scores, we partition the data into SFT and RFT subsets with an intra-class ratio of 6:4. Since the SFT stage imposes explicit supervision on the reasoning component, leaked information in CoT may directly contaminate the supervision signal. Therefore, we avoid allocating low-scoring samples to the SFT.

4.3

Reward Function

Mainstream video reasoning methods typically employ answer accuracy reward 𝑟 𝑎𝑐𝑐 and format accuracy reward 𝑟 𝑓 𝑚𝑡 as signals for reinforcement learning [7, 27]. However, such approaches are far less effective on long videos with complex scene transitions, and especially on ICQ: the model struggles to locate question-relevant shots and instead processes the whole video, introducing substantial noise. Yet annotating effective shots is labor-intensive and highly subjective. To address this, beyond the above signals we design an Effective Shot Re-Reasoning (ESR) reward on top of GRPO and a caption reward for the reference image. The ESR reward triggers secondary re-reasoning on the initial sampling results, teaching the model which shots are effective without ground-truth shot annotations. Specifically, our RFT process employs the following four reward functions: Format Reward. To facilitate more intuitive evaluation and training for the ICQ task, we decompose it into four components, corresponding to the four required output fields. Specifically, <c aption>...</caption> summarizes the target task based on the reference image, <candidate>...</candidate> identifies the relevant shots for the query, and <think>...</think> and <answ er>...</answer> denote the reasoning process and final prediction, respectively. This decomposition enables the model to learn reference alignment, evidence localization, reasoning, and answer generation, thereby improving task performance while mitigating answer hacking. The format reward is defined as:   1.0,    𝑟 𝑓 𝑚𝑡 = 0.9,  Í   𝑡 𝑤𝑡 · 1𝑡 ,

all tags and in correct order all tags but in wrong order , otherwise

(3)

where 𝑤𝑐𝑎𝑝 = 0.2, 𝑤𝑐𝑎𝑛𝑑 = 0.15, 𝑤𝑡ℎ𝑖𝑛𝑘 = 0.15, 𝑤 𝑎𝑛𝑠 = 0.5, and 1𝑡 denotes the indicator function for the presence of tag 𝑡. Accuracy Reward. We employ a binary score for all task types, evaluating the consistency between the answer 𝐴1 and the groundtruth answer 𝐴𝑔𝑡 : ( 𝑟 𝑎𝑐𝑐 =

1.0, 0.0,

if 𝐴1 = 𝐴𝑔𝑡 . otherwise

(4)

Caption Semantic Reward. We employ an LLM M 𝑗𝑢𝑑𝑔𝑒 as a judge to evaluate the semantic consistency between the modelgenerated caption 𝐶𝑝𝑟𝑒𝑑 and the ground-truth reference image description 𝐶𝑔𝑡 , with particular emphasis on identifying obvious conflicts. The judge model outputs a score ranging from 0 to 10, which is mapped to a reward through a step function: 𝑟𝑐𝑎𝑝 = 𝜔 × ⌊𝑠/2⌋, 𝑠 = M 𝑗𝑢𝑑𝑔𝑒 (𝐶𝑝𝑟𝑒𝑑 , 𝐶𝑔𝑡 ) ∈ [0, 10],

(5)

where 𝜔 is the Hyperparameter. This reward facilitates better understanding of the reference image content related to the question, preventing the model from exploiting shortcuts by analyzing only the video and text while ignoring the reference image. Furthermore, this approach implicitly supervises the think process to ensure more reasonable reasoning. Effective Shot Re-Reasoning Reward. During the GRPO rollout phase, the policy model generates an output containing the effective shot sequence 𝑆 = {𝑠 1, 𝑠 2, ..., 𝑠𝑘 }, chain-of-thought 𝑇 , and final answer 𝐴1 . Let 𝑁𝑟𝑒 𝑓 denote the total number of shots containing the reference person in the video, and 𝑘 = |𝑆 | denote the number of shots output by the model. If the initial answer is correct and the shot sequence can be successfully parsed, we clip and concatenate the shots mentioned by the model according to timestamp annotations, reduce the overall sampling frame rate, and perform re-reasoning under the same settings to obtain the re-reasoning answer 𝐴2 . The re-reasoning reward is defined as: 𝑟𝑒𝑠𝑟 = 1[𝐴1 = 𝐴𝑔𝑡 ] · 1[Parse(𝑆)] · 1[𝐴2 = 𝐴𝑔𝑡 ] · (0.5 + 𝑟𝑒𝑥𝑡𝑟𝑎 ), (6) where the extra reward 𝑟𝑒𝑥𝑡𝑟𝑎 decreases linearly with the number of output shots:   0.5, if 𝑘 = 1     0, if 𝑘 ≥ 𝑁𝑟𝑒 𝑓 . 𝑟𝑒𝑥𝑡𝑟𝑎 = (7)  𝑁𝑟𝑒 𝑓 −𝑘    0.5 × 𝑁𝑟𝑒 𝑓 −1 , otherwise  For certain types of questions, there exist cases where 𝑁𝑟𝑒 𝑓 = None. In such cases, the ESR Reward degenerates to a binary score consistent with the accuracy reward. This mechanism enables the model to learn how to identify effective shots and achieve localization and acquisition of key information without ground-truth effective shot annotations. In summary, the overall reward function is: 𝑟 = 𝜃 𝑎𝑐𝑐 · 𝑟 𝑎𝑐𝑐 + 𝜃 𝑓 𝑚𝑡 · 𝑟 𝑓 𝑚𝑡 + 𝜃𝑐𝑎𝑝 · 𝑟𝑐𝑎𝑝 + 𝜃 𝑒𝑠𝑟 · 𝑟𝑒𝑠𝑟 ,

(8)

where 𝜃 𝑎𝑐𝑐 , 𝜃 𝑓 𝑚𝑡 , 𝜃𝑐𝑎𝑝 and 𝜃 𝑒𝑠𝑟 are hyperparameters. We will discuss these in the appendix. It is worth noting that no effective shot annotations are included during the data construction phase. Therefore, in the SFT stage, we train the model to output all shots containing the reference person, and this output will be gradually optimized to effective shots that contribute to answering the question during the RFT stage.

5

Experiments and Analysis

This section presents a systematic experimental evaluation and analysis of current mainstream open-source and closed-source MLLMs, as well as our specifically designed ISYV-Model for the ICQ task, on the ISYV-Benchmark. We first introduce the experimental setup, then analyze the performance of each model on the benchmark,

I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil.

Table 3: Evaluation results of mainstream closed-source models, open-source models, and related works on ISYV-Benchmark. 𝑂.𝐴𝑣𝑔 represents the accuracy of correct answers only, while 𝐼𝐶𝑄 𝑄.𝐴𝑣𝑔 represents the accuracy under ICQ’s unique requirements where both image descriptions and correct answers are verified. The blue values reflect the gap between the two. † denotes full adherence to the original training setup, ‡ denotes the use of our training setup with the model architecture unchanged. ★ The evaluation was conducted blindly by three annotators. Models

Query Format

Think

𝑂.𝐴𝑣𝑔

𝐼𝐶𝑄 𝑄.𝐴𝑣𝑔

Level 1 - BP

Level 2 - OP

Level 3 - PO

Level 4 - SC

Level 5 - SM

Level 6 - CR

Human★

Image

/

95.13%

/

96.30%

95.02%

96.88%

96.11%

90.81%

96.97%

GPT-5.2-Global [33] Gemini-2.5-Pro [5] Qwen3-VL-235B-A22B [3]

Image Image Image

Yes Yes No

45.53% 67.10% 39.80%

45.53% (-0.00%) 67.10% (-0.00%) 39.80% (-0.00%)

49.63% 85.93% 70.37%

37.55% 65.90% 45.59%

46.25% 66.25% 37.81%

66.67% 80.56% 27.78%

22.61% 40.99% 21.91%

65.66% 82.32% 51.01%

Qwen2.5-VL-7B [4] Qwen2.5-VL-32B [4] Qwen3-VL-8B [3]

Text Text Text

No No No

22.44% 38.56% 28.61%

/ / /

42.96% 55.56% 55.56%

24.90% 38.70% 33.33%

21.88% 39.69% 20.00%

19.44% 33.89% 6.11%

8.48% 19.43% 28.62%

28.79% 56.57% 38.38%

Qwen2.5-VL-7B [4] Qwen2.5-VL-32B [4] Qwen3-VL-8B [3] GLM-4.1V-9B-Thinking [18] Mimo-VL-7B-RL [46] InternVL3-8B [58] VideoLLaMA3-7B [50]

Image Image Image Image Image Image Image

No No No Yes Yes No No

24.76% 39.29% 31.45% 24.47% 18.74% 28.98% 37.25%

6.75% (-18.01%) 33.99% (-5.30%) 23.09% (-8.36%) 22.80% (-1.67%) 10.09% (-8.65%) 28.90% (-0.08%) 3.34% (-33.92%)

38.52% 57.78% 44.44% 34.07% 20.00% 29.63% 34.07%

33.33% 36.78% 32.18% 25.67% 17.24% 29.89% 35.25%

22.50% 41.56% 28.75% 25.62% 22.81% 36.25% 47.81%

20.00% 36.67% 17.22% 30.56% 20.00% 35.56% 40.00%

7.77% 18.37% 23.67% 11.31% 10.60% 8.13% 16.61%

36.36% 58.59% 50.00% 27.78% 23.74% 39.39% 52.02%

IDA-VLM [20] † PLVM [28] † Video-R1 [8] † Qwen2.5-VL-7B-SFT [4] ‡ Qwen2.5-VL-7B-RFT [4] ‡ ISYV-Model-SFT(Ours) ISYV-Model-RFT(Ours)

Image Image Image Image Image Image Image

No No Yes No Yes No Yes

25.05% 10.97% 47.57% 28.03% 54.22% 33.55% 57.01%

24.84% (-0.21%) 5.16% (-5.81%) 47.57% (-0.00%) 28.03% (-0.00%) 54.22% (-0.00%) 33.40% (-0.15%) 57.01% (-0.00%)

31.11% 5.93% 37.78% 37.04% 59.25% 49.63% 62.22%

31.03% 6.90% 40.61% 36.40% 51.72% 46.74% 61.69%

15.94% 14.06% 50.62% 19.38% 53.12% 20.31% 60.00%

5.56% 30.00% 45.56% 8.33% 34.44% 8.33% 37.22%

36.75% 5.65% 44.88% 37.46% 49.82% 43.82% 50.18%

28.79% 5.05% 64.14% 29.29% 69.19% 34.85% 70.20%

and finally conduct ablation studies and case studies to investigate the impact of key factors in depth.

5.1

and configurations. For Qwen2.5-VL-7B-SFT/RFT, we concatenated the video and image tokens before feeding them into the LLM, and enabled all reward functions during the RFT stage.

Experimental Setup

We evaluated multiple mainstream MLLMs on the ISYV-Benchmark, encompassing closed-source models (e.g., GPT-5.2-global [33], Gemini2.5-Pro [5]), open-source models (e.g., Qwen2.5-VL [4], Qwen3VL [3], InternVL3 [58]), and our proposed ISYV-Model, with parameter scales ranging from 7B to hundreds of billions. For the video component in multi-source inputs, we uniformly distributed the total number of sampled frames across the entire video duration and standardized the frame sampling to 32 frames for all evaluations. Regarding the organization of multi-source inputs, we constructed joint video-image-text inputs according to the specifications provided by the transformers library or the official documentation of each model, ensuring that all modality information was fed into the model within a single conversational turn. We uniformly required all models to produce outputs following the standard format of the ICQ task: first generating a description of the reference image, then providing a candidate sequence of valid shots, and finally outputting the thinking process and the final answer separately. Models supporting the thinking mode were required to output the think component, while those without this capability only needed to output the answer component. Inference for open-source models was conducted following their official implementations, while closed-source models were accessed through official APIs. Accuracy was adopted as the primary evaluation metric, and additional implementation details, evaluation settings, and training configurations are provided in the appendix. For IDAVLM, PLVM, and Video-R1, we retained their original architectures

5.2

Evaluation on the ISYV-Benchmark

As shown in Table. 3, we present the evaluation results of several MLLMs on ISYV-Benchmark, including accuracy across different levels, overall average accuracy, and 𝐼𝐶𝑄 𝑄.𝐴𝑣𝑔. Notably, 𝐼𝐶𝑄 𝑄.𝐴𝑣𝑔 in the table represents the accuracy of responses that are both correct and satisfy the ICQ task requirements (i.e., containing at least an analysis of the reference image and the answer). This metric is necessary because some models tend to completely ignore the input reference image and attempt to hack the answer using only the video and question. We employ rule-based methods or LLM outputs to parse responses and determine whether they contain descriptions of the reference image; detailed settings are provided in the appendix. Based on these results, we derive several key observations: The ICQ task poses significant challenges for existing MLLMs. Current MLLMs exhibit a substantial performance gap compared to humans on ICQ and similar complex tasks. Even the best-performing proprietary model, Gemini-2.5-Pro [5], falls considerably short of human performance. Humans demonstrate clear advantages across all difficulty levels, particularly in Level 2 tasks involving disguised character analysis. This reveals a critical limitation of existing MLLMs in jointly analyzing and comprehending complex multi-source inputs. Closed-source MLLMs significantly outperform open-source models and better adapt to the ICQ task. All closed-source MLLMs achieve higher overall average accuracy than open-source

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil.

Shibo Gao et al.

Table 4: Ablation studies examining the impact of different training strategies and reward function combinations on model performance. Columns are cumulative: each RFT column adds the listed reward to the previous one. Strategy

SFT

SFT-𝐺𝑇𝑐𝑎𝑛𝑑

RFT 𝑟 𝑓 𝑚𝑡 , 𝑟 𝑎𝑐𝑐

+𝑟𝑐𝑎𝑝

+𝑟𝑒𝑠𝑟

𝑂.𝐴𝑣𝑔

33.75%

35.62%

52.06%

55.44%

57.01%

MLLMs, with their advantages being particularly pronounced on several critical tasks. Furthermore, closed-source models are less inclined to attempt hacking answers and instead tend to correlatively understand the video and reference image. In contrast, VideoLLaMA3-7B [50] achieves a reasonable 𝑂.𝐴𝑣𝑔 but exhibits catastrophic performance on 𝐼𝐶𝑄 𝑄.𝐴𝑣𝑔. Our analysis of the model outputs reveals that this model fails to comprehend the ICQ task entirely and tends to provide brief responses that appear "random" or attempt to hack an answer. Fig. 6 presents a concrete example of answer hacking from Qwen2.5-VL-32B [4], additional analyses are provided in the appendix. Compared to multi-turn input and text-substituted image approaches, the ICQ task is irreplaceable. As previously discussed, the ICQ task cannot be effectively addressed through intuitive alternative methods, such as employing multi-turn dialogue or performing reasoning after substituting images with textual descriptions. Multiple models exhibit significant performance degradation when using text-substituted image inputs. Therefore, the ICQ task represents a necessary attempt to extend the boundaries of the video reasoning domain.

5.3

Ablation Studies

We first conduct ablation studies on the training strategies and the designed reward functions, with results presented in Table. 4. The results indicate that omitting the learning of <candidate> ...</candidate> during the SFT stage yields better immediate performance but constrains the model’s ability to learn effective shots in the subsequent RFT stage. During the RFT stage, in addition to the standard 𝑟 𝑓 𝑚𝑡 and 𝑟 𝑎𝑐𝑐 , both 𝑟𝑐𝑎𝑝 and 𝑟𝑒𝑠𝑟 further enhance model performance. The 𝑟𝑐𝑎𝑝 facilitates better association between reference image descriptions and the reasoning process, while 𝑟𝑒𝑠𝑟 enables the model to more effectively identify key shot sequences in the video that are relevant to solving the problem. We require the model to output the <candidate>...</candidate> field during the SFT stage only when 𝑟𝑒𝑠𝑟 is enabled. Furthermore, we conducted ablation studies on the length of the learnable token sequence in the ICQ Module and the number of input frames, with results shown in Table. 5. The results indicate that 32 is the optimal token length: too few tokens fail to adequately represent the reference person image, while too many may introduce noise. Regarding the number of input frames, since 32-frame sampling was used during training, increasing the number of frames did not improve performance, whereas using fewer frames led to performance degradation due to insufficient information.

Table 5: Ablation studies investigating the effect of varying the length of learnable token sequences in the ICQ Module. Token Nums

16 Token

32 Token

64 Token

𝑂.𝐴𝑣𝑔

46.93%

57.01%

53.31%

Frame Nums

16 Frames

32 Frames

64 Frames

𝑂.𝐴𝑣𝑔

34.50%

57.01%

56.56%

Figure 6: The case study of different models on a sample shows that our model completed the task very well, while other models all exhibited some errors to varying degrees.

5.4

Case Study

Fig.6 shows the responses of different models to a level-5 sample from ISYV-Benchmark. As shown, all baseline models exhibit varying issues. GLM4.1V-9B[18] and Qwen2.5-VL-32B[4] misinterpreted the reference image, Qwen3-VL-8B[3] incorrectly claimed that the referenced person was absent, and InternVL3-8B [58] failed to follow the required format. Although Qwen2.5-VL-32B selected the correct option, this may be considered a form of hacking behavior because its image description was inaccurate. In contrast, our model accurately described the reference person and answered correctly.

6

Conclusion

To address the diverse input scenarios in real-world video reasoning, we introduce ICQ, the person-centric multi-input video reasoning task. To be specific, we develop a comprehensive solution encompassing: a manually annotated high-quality evaluation benchmark ISYV-Benchmark (spanning 6 difficulty levels and 17 subtasks), an automatically constructed large-scale training set ISYV-75K, and a dedicated model architecture with corresponding training methodology. Extensive evaluation reveals that the ICQ task poses significant challenges for current MLLMs, with even the best-performing proprietary models exhibiting substantial gaps

I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

compared to human-level performance. Through comprehensive experiments and analysis, we uncover fundamental limitations of existing models. We anticipate that the ICQ task will serve as a valuable resource for advancing the boundaries of video reasoning and promoting the development of MLLMs. In the future, we will further explore and expand video reasoning tasks to improve MLLMs’ capabilities in more complex and realistic scenarios.

Acknowledgments This work was supported in part by the Key-Area Research and Development Program of Guangdong Province (2024B0101040008).

References [1] Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel CohenOr. 2024. Myvlm: Personalizing vlms for user-specific queries. In European Conference on Computer Vision. Springer, 73–91. [2] Xiang An, Jiangkang Deng, Jia Guo, Ziyong Feng, Xuhan Zhu, Yang Jing, and Liu Tongliang. 2022. Killing Two Birds with One Stone: Efficient and Robust Training of Face Recognition CNNs by Partial FC. In CVPR. [3] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025). [4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2. 5-VL Technical Report. arXiv preprint arXiv:2502.13923 (2025). [5] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025). [6] Roy G d’Andrade. 1995. The development of cognitive anthropology. Cambridge University Press. [7] Bo Fang, Yuxin Song, Qiangqiang Wu, Haoyuan Sun, Wenhao Wu, and Antoni B Chan. 2025. ViSS-R1: Self-Supervised Reinforcement Video Reasoning. arXiv preprint arXiv:2511.13054 (2025). [8] Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. 2025. Videor1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776 (2025). [9] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2025. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 24108–24118. [10] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision. 5267–5275. [11] Shibo Gao, Jinmeng Gong, Peipei Yang, Chao Liang, and Linlin Huang. 2023. A Stable Long-Term Tracking Method for Group-Housed Pigs. In Proceedings of the International Conference on Image and Graphics (ICIG). 238–249. [12] Shibo Gao, Peipei Yang, Haiyang Guo, Yangyang Liu, Yi Chen, Shuai Li, Han Zhu, Jian Xu, Xu-Yao Zhang, and Linlin Huang. 2025. The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM. arXiv preprint arXiv:2507.21649 (2025). [13] Shibo Gao, Peipei Yang, and Linlin Huang. 2024. Scene-Adaptive SVAD Based on Multi-Modal Action-Based Feature Extraction. In Proceedings of the Asian Conference on Computer Vision (ACCV). 329–346. [14] Shibo Gao, Peipei Yang, and Linlin Huang. 2025. Semi-Supervised Video Anomaly Detection with Compact Deformable 3D Convolution. In ICASSP 2025 – 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. [15] Shibo Gao, Peipei Yang, and Linlin Huang. 2025. Suvad: Semantic understanding based video anomaly detection using mllm. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5. [16] Shibo Gao, Peipei Yang, Yangyang Liu, Yi Chen, Han Zhu, Xu-Yao Zhang, and Linlin Huang. 2026. Vagu & gts: Llm-based benchmark and framework for joint video anomaly grounding and understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 4167–4175. [17] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. 2024. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19383–19400.

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil.

[18] Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. 2025. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006 (2025). [19] Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang, Limin Wang, et al. 2024. Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22072–22086. [20] Yatai Ji, Shilong Zhang, Jie Wu, Peize Sun, Weifeng Chen, Xuefeng Xiao, Sidi Yang, Yujiu Yang, and Ping Luo. 2024. Ida-vlm: Towards movie understanding via id-aware large vision-language model. arXiv preprint arXiv:2407.07577 (2024). [21] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. Tvqa: Localized, compositional video question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing. 1369–1379. [22] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. PMLR, 19730–19742. [23] Jingyao Li, Jingyun Wang, Molin Tan, Haochen Wang, Cilin Yan, Likun Shi, Jiayin Cai, Xiaolong Jiang, and Yao Hu. 2026. CrossVid: A Comprehensive Benchmark for Evaluating Cross-Video Reasoning in Multimodal Large Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 6244–6252. [24] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2024. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22195–22206. [25] Qiang Lyu, Zicong Chen, Chongxiao Wang, Haolin Shi, Shibo Gao, Ran Piao, Youwei Zeng, Jianlou Si, Fei Ding, Jing Li, Chun Pong Lau, and Weiqiang Wang. 2025. Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewards. arXiv preprint arXiv:2512.00743 (2025). [26] Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. 2025. Drvideo: Document retrieval based long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference. 18936– 18946. [27] Jinyoung Park, Jeehye Na, Jinyoung Kim, and Hyunwoo J Kim. 2025. Deepvideor1: Video reinforcement fine-tuning via difficulty-aware regressive grpo. arXiv preprint arXiv:2506.07464 (2025). [28] Chau Pham, Hoang Phan, David Doermann, and Yunjie Tian. 2025. PLVM: A Tuning-Free Approach for Personalized Large Vision-Language Model. In Proceedings of the Computer Vision and Pattern Recognition Conference. 3632– 3641. [29] Xingyu Ren, Alexandros Lattas, Baris Gecer, Jiankang Deng, Chao Ma, and Xiaokang Yang. 2023. Facial Geometric Detail Recovery via Implicit Representation. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG). [30] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024). [31] Liang Shi, Wei Li, Kevin M Beussman, Lin Chen, and Yun Fu. 2026. IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Models. arXiv preprint arXiv:2601.14188 (2026). [32] Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. 2025. Sample more to think less: Group filtered policy optimization for concise reasoning. arXiv preprint arXiv:2508.09726 (2025). [33] Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267 (2025). [34] Siqing Song, Chuang Wang, Yong Lang, Yi Yang, and Xu-Yao Zhang. 2026. LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 35470–35484. doi:10.18653/v1/2026.acl-long.1640 [35] Siqing Song, Chuang Wang, Rui-Qi Wang, Yi Yang, and Xu-Yao Zhang. 2025. Achieving Binary Weight and Activation for LLMs using Post-Training Quantization. In Findings of the Association for Computational Linguistics: ACL 2025. 8782–8795. [36] Tomáš Souček and Jakub Lokoč. 2020. TransNet V2: An effective deep network architecture for fast shot transition detection. arXiv preprint arXiv:2008.04838 (2020). [37] Zhaochen Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, et al. 2025. Thinking with images for multimodal reasoning: Foundations, methods, and future frontiers. arXiv preprint arXiv:2506.23918 (2025). [38] Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. 2025. Kimi-vl

MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil.

technical report. arXiv preprint arXiv:2504.07491 (2025). [39] Shulin Tian, Ruiqi Wang, Hongming Guo, Penghao Wu, Yuhao Dong, Xiuying Wang, Jingkang Yang, Hao Zhang, Hongyuan Zhu, and Ziwei Liu. 2025. Ego-r1: Chain-of-tool-thought for ultra-long egocentric video reasoning. arXiv preprint arXiv:2506.13654 (2025). [40] Zhongwei Wan, Zhihao Dou, Che Liu, Yu Zhang, Dongfei Cui, Qinjian Zhao, Hui Shen, Jing Xiong, Yi Xin, Yifan Jiang, et al. 2025. Srpo: Enhancing multimodal llm reasoning via reflection-aware reinforcement learning. arXiv preprint arXiv:2506.01713 (2025). [41] Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, et al. 2025. Time-r1: Post-training large vision language model for temporal video grounding. arXiv preprint arXiv:2503.13377 (2025). [42] Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. 2025. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference. 3272–3283. [43] Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Shengming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. 2025. Qwen-image technical report. arXiv preprint arXiv:2508.02324 (2025). [44] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems 37 (2024), 28828–28857. [45] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9777–9786. [46] LCT Xiaomi and Core Team. 2025. Mimo-vl technical report. arXiv preprint arXiv:2506.03569 1, 2 (2025), 5. [47] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025). [48] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. 2025. Dapo: An opensource llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 (2025). [49] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via

Shibo Gao et al.

question answering. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 9127–9134. [50] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. 2025. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106 (2025). [51] Haoji Zhang, Xin Gu, Jiawen Li, Chixiang Ma, Sule Bai, Chubin Zhang, Bowen Zhang, Zhichao Zhou, Dongliang He, and Yansong Tang. 2025. Thinking with videos: Multimodal tool-augmented reinforcement learning for long video reasoning. arXiv preprint arXiv:2508.04416 (2025). [52] Qiming Zhang, Jing Zhang, Yufei Xu, and Dacheng Tao. 2024. Vision transformer with quadrangle attention. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 5 (2024), 3608–3624. [53] Shi-Xue Zhang, Hongfa Wang, Duojun Huang, Xin Li, Xiaobin Zhu, and XuCheng Yin. 2026. Vcapsbench: A large-scale fine-grained benchmark for video caption quality evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 12726–12734. [54] Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. 2025. Tinyllava-videor1: Towards smaller lmms for video reasoning. arXiv preprint arXiv:2504.09641 (2025). [55] Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al. 2025. Mmvu: Measuring expert-level multi-discipline video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference. 8475–8489. [56] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, et al. 2025. Mlvu: Benchmarking multitask long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13691–13701. [57] Yuan Zhou, Litao Hua, Shilong Jin, Wentao Huang, and Haoran Duan. 2026. ReaSon: Reinforced Causal Search with Information Bottleneck for Video Understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 13791–13799. [58] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025).

Related documents

• Companion-style QA Assistance in Ego-Vision#974816
• CAM: Question Answering on Entity-Centric Videos with Continuous Extraction and Adaptive Querying#974075
• ICM-Bench: Person-Level Identity Reasoning in Multimodal Agents with Long-Term Memory#985276
• Target-Checked Reliability Score Refinement for Video Question Answering#1071092
• Question Answering for Diagram-Rich Technical Meeting Videos#363337