Towards One-to-Many Temporal Grounding
Qi Xu * 1 Yue Tan * 2 Shihao Chen 1 Jiahao Meng 2 Anna Wang 3 Shunping Ji 1 Hao Fei 4 Jason Li 3
Abstract
One-to-Many Temporal Grounding
Strong OMTG Capability Ours Doubao Seed 1.8
Caption Reward (+2.97) Temporal Reward (+5.87)
SFT (+34.60)
W ea kO MT GC ap ab ilit
y
Temporal Grounding (TG) aims to localize video segments corresponding to a textual query. Prior research predominantly focuses on singlesegment retrieval. Real-world scenarios, however, often require localizing multiple disjoint segments for a single query—a setting we term One-toMany Temporal Grounding (OMTG).
arXiv:2606.06294v1 [cs.CV] 4 Jun 2026
EtF1
Gemini 2.5 Pro
Gemini 3 Pro
OMTG Dataset
No OMTG Capability
Previous state-of-the-art MLLMs, optimized for one-to-one settings, struggle in this context, often yielding near-zero scores due to a lack of event cardinality perception.
One-to-One Temporal Grounding
Qwen Series
Figure 1. The Road to One-to-Many Temporal Grounding. Left: Leveraging our OMTG dataset, we empower the model to evolve from One-to-One to One-to-Many through SFT and RL, underpinned by novel temporal and caption rewards. Right: The capability landscape on the proposed OMTG Bench.
To bridge this gap, we present a systematic solution with three key contributions. First, we establish the first comprehensive OMTG benchmark, introducing Count Accuracy (C-Acc) and Effective Temporal F1 (EtF1) as evaluation metrics.
As a fundamental task in video understanding, it has witnessed significant advancements driven by Multi-modal Large Language Models (MLLMs) (Lin et al., 2023; Li et al., 2025c; Wang et al., 2025; Zeng et al., 2024; Ren et al., 2024; Li et al., 2025b).
Second, we curate a high-quality OMTG dataset comprising 56k samples through a sophisticated construction pipeline.
However, conventional research has predominantly focused on the one-to-one correspondence between queries and temporal segments. In real-world scenarios, video content is inherently dynamic and repetitive, with a single semantic action (e.g., "a person clapping") recurring at multiple distinct intervals. This characteristic gives rise to the One-to-Many Temporal Grounding problem, which requires identifying all disjoint time segments semantically consistent with a query. Accurately retrieving the complete set of occurrences, rather than a single instance, is essential for a comprehensive understanding of complex video narratives.
Third, we develop novel temporal and caption reward functions specifically designed for OMTG. In particular, the caption reward leverages Chainof-Thought reasoning over dense video captions to explicitly guide policy optimization toward both preciseness and completeness. Extensive experiments show our model achieves a new state-of-the-art EtF1 of 43.65% on OMTG Bench, outperforming Gemini 2.5 Pro and Seed1.8 by 15.85% and 15.61%, respectively.
To bridge this gap, we formally define One-to-Many Temporal Grounding (OMTG) as a set generation task within the MLLM framework. Recognizing that standard metrics for one-to-one grounding (e.g. tIoU, R@1) are illsuited for this setting, we introduce a rigorous evaluation suite: Temporal F1-Score (tF1) to balance precision and recall, Count Accuracy (C-Acc) to assess event cardinality perception, and Effective Time F1 (EtF1) to strictly penalize incomplete retrieval and hallucinations.
1. Introduction Temporal Grounding aims to localize specific temporal segments within a video that semantically correspond to a given natural language query. 1
Wuhan University 2 Peking University 3 Nanyang Technological University 4 National University of Singapore. Correspondence to: Jason Li <[email protected]>.
Furthermore, we establish the first comprehensive benchmark tailored for OMTG. Our extensive evaluation of state-
Preprint. June 5, 2026.
1
OMTG: Towards One-to-Many Temporal Grounding
of-the-art open-source and proprietary MLLMs, as illustrated in Figure 1, reveals a critical capability gap: existing open-source models and traditional TG experts struggle significantly in the OMTG task, often yielding near-zero EtF1 scores, advanced proprietary models (e.g., Gemini series (Comanici et al., 2025), Seed-1.8 (Bytedance Seed Team, 2025)) demonstrate weak-OMTG-capability, and our model significantly outperforms all baselines, reaching the strong-OMTG-capability zone. This stark contrast underscores the urgency of exploring this new direction.
et al., 2017; Lei et al., 2021; Krishna et al., 2017; Zhang et al., 2025b) and, critically, a rigid one-to-one formulation that fails to capture recurring or overlapping events. This limitation extends to training data: despite scaling efforts via MLLM generation (Bao et al., 2024; Wang et al., 2024b) or large-scale collection (Qian et al., 2024; Huang et al., 2024), current datasets rarely provide one-to-many supervision. Reinforcement Learning for Video MLLMs. Reinforcement learning has proven effective in improving the visual and cross-modal reasoning capabilities of MLLMs through verifiable or preference-based rewards (OpenAI, 2023; Zhou et al., 2025; Zhan et al., 2025; Deng et al., 2025; Liu et al., 2025b; Yang et al., 2025b; Zhang et al., 2025a). More recently, RL has been applied to video MLLMs to better model spatio-temporal structure and long-range dependencies. Many methods (Meng et al., 2025a; Feng et al., 2025; Li et al., 2025b; Yan et al., 2025; Meng et al., 2025b) introduce rule-based or perception-aware rewards and test-time scaling to significantly improve the model’s temporal reasoning ability. However, existing RL-based approaches primarily optimize for a one-to-one temporal grounding task. In contrast, we present the sophisticated RL pipeline for One-to-Many Temporal Grounding.
To tackle these challenges, we devise a sophisticated data pipeline to construct 56k high-quality training samples. Leveraging this data, we propose a two-stage training strategy that synergizes Supervised Fine-Tuning (SFT) with subsequent Reinforcement Learning (RL). We employ two complementary rewards: caption rewards that leverages dense video captions with Chain-of-Thought reasoning to comprehend complex event structures, and temporal rewards that directly supervises temporal boundaries for precise localization. Notably, we observe that RL training on the OMTG task also improves standard one-to-one temporal grounding performance. Extensive experiments demonstrate the superiority of our approach. Our model surpasses both leading open-source and proprietary models, achieving an EtF1 score of 43.65% on the OMTG Bench. This performance sets a new stateof-the-art, outperforming the previous best proprietary models Gemini 2.5 Pro and Seed-1.8 by significant margins of 15.85% and 15.61%, respectively.
3. One-to-Many Temporal Grounding 3.1. Problem Formulation We formulate One-to-Many Temporal Grounding as a generative task under the MLLM framework. Given an input video V = {ft }Tt=1 consisting of T visual frames and a textual query Q = {wl }L l=1 , the objective is to localize multiple temporal segments in the video that correspond to repeated semantic occurrences of the query.
2. Related Work MLLMs for Video Temporal Grounding. MLLMs extend LLMs to visual modalities by unifying language, image, and video understanding within a single reasoning framework (Liu et al., 2023; Li et al., 2023; Alayrac et al., 2022; Zhang et al., 2023; Li et al., 2025a). This task has also been reshaped by these models. Early TG methods relied on visual encoders with task-specific heads (Zhang et al., 2019; Moon et al., 2023; Liu et al., 2022), while recent approaches directly leverage MLLMs’ cross-modal reasoning capabilities via instruction tuning, causal event modeling, and hierarchical reasoning (Huang et al., 2024; Ren et al., 2024; Guo et al., 2024; Qian et al., 2024; Liu et al., 2025a). The training paradigms span supervised fine-tuning (Yu et al., 2023; Lu et al., 2024), reinforcement learning (Wang et al., 2025; Li et al., 2025b), and training-free methods (Zheng et al., 2024; Qin et al., 2025). Despite these advances, existing methods largely inherit a one-to-one supervision assumption, limiting their ability to handle complex real-world scenarios.
Specifically, we learn a mapping function Fθ , parameterized by an MLLM, which directly generates a natural language response: Y = Fθ (V, Q) (1) The generated sequence Y encodes a structured description of temporal intervals associated with the query events. A deterministic parsing function ϕ(·) is applied to extract a set of predicted temporal segments P = ϕ(Y ) = {(ŝm , êm )}M m=1
(2)
where ŝm and êm denote the predicted start and end timestamps of the m-th instance, and M is the number of predicted segments. The ground-truth annotations are given as a set of temporal intervals G = {(sk , ek )}K (3) k=1
Video Temporal Grounding Benchmarks and Datasets. Existing TG benchmarks suffer from annotation noise (Gao
where K ≥ 1 denotes the number of semantic occurrences of the query in the video. The learning objective is to gen2
OMTG: Towards One-to-Many Temporal Grounding
Question: Find the video segment that corresponds to the given textual query 'a man speaking to the camera in a video message' and determine its start and end seconds. If there are multiple segments, please output the start and end time for each one separately.
Question: Find the video segment that corresponds to the given textual query 'A brownish, silt filled river cuts through a landscape covered in trees.' and determine its start and end seconds. If there are multiple segments, please output the start and end time for each one separately.
Ground Truth Answer: The given query happens in <time>3 - 32 seconds</time>, <time>33 - 55 seconds</time>, <time>56 - 64 seconds</time> and <time>65 - 86 seconds</time>.
Ground Truth Answer: The given query happens in <time>0 - 30 seconds</time>, and <time>61 - 126 seconds</time>. #Segments: 2
#Segments: 4 Gemini 3 Pro Answer: <time>00:01 - 00:30 seconds</time>, <time>00:32 - 00:35 seconds</time>, <time>01:01 - 02:05 seconds</time> and <time>02:10 - 02:15 seconds</time> #Segments: 4
Gemini 3 Pro Answer: <time>00:01 - 01:26 seconds</time>. #Segments: 1 tIoU: 0.94
C-Acc: 0
EtF1: 0
tIoU: 0.90
High tIoU but Bad Answer!
[email protected]: 0.5 C-Acc: 0
EtF1: 0
High tIoU but Hallucinatory Answer!
Our Answer: The given query happens in <time>3 - 31 seconds</time>, <time>32 - 55 seconds</time>, <time>56 - 64 seconds</time> and <time>65 - 85 seconds</time>.
Our Answer: The given query happens in <time>0 - 31 seconds</time> and <time>61 - 127 seconds</time>.
#Segments: 4
#Segments: 2
tIoU: 0.96
[email protected]: 1.0 C-Acc: 1.0 EtF1: 1.0
tIoU: 0.98
[email protected]: 1.0 C-Acc: 1.0 EtF1: 1.0
Figure 2. The deceptiveness of tIoU in One-to-Many Temporal Grounding. Gemini 3 pro achieves high tIoU (> 0.9) in both examples despite event counting failures: under-segmentation (Left) and over-segmentation (Right). tIoU metrics fail to capture these counting errors. In contrast, our proposed EtF1 strictly penalizes count mismatches, providing a rigorous evaluation that highlights our method’s superior precision in both event counting and localization.
erate a response Y such that the extracted predictions P closely match G in terms of both cardinality (i.e., M = K) and temporal boundaries.
a single continuous segment, the resulting tIoU can remain deceptively high (e.g., 0.9) due to dominant overlap, despite the model failing to distinguish multiple occurrences and producing incorrect cardinality.
3.2. Metrics
Therefore, a holistic evaluation of OMTG requires decoupling instance-level coverage and prediction correctness, explicitly measuring both precision and recall, and employing the F1-score to jointly assess temporal localization quality and instance-level fidelity. Furthermore, we introduce Effective Temporal F1-score which conditions F1-score on event cardinality to provides a rigorous evaluation of OMTG task.
One-to-One Temporal Grounding benchmarks (Gao et al., 2017; Krishna et al., 2017; Lei et al., 2021; Zhang et al., 2025b) predominantly adopt Recall@1 (R@1) with a temporal IoU (tIoU) threshold as the evaluation metric. In the One-to-One setting, where each query is associated with a unique ground-truth segment, this metric is sufficient, as a correct retrieval simultaneously satisfies both precision and recall.
We formulate the evaluation as a bipartite matching problem between the predicted segments P and the ground-truth segments G. Given an IoU threshold ξ, we apply the Hungarian algorithm to compute the optimal one-to-one matching that maximizes the total temporal IoU. Based on this matching, we define the following evaluation metrics:
In contrast, One-to-Many Temporal Grounding poses fundamentally different evaluation challenges, since a single query corresponds to a set of ground-truth segments. As illustrated in Fig. 2, the distinction between precision and recall becomes non-trivial. A high recall score only indicates that some ground-truth instances are covered, but fails to penalize redundant or hallucinated predictions. Moreover, the tIoU metric, while effective in measuring temporal overlap, is insensitive to the structural composition of events. For example, if a model incorrectly merges two semantically distinct events separated by a short temporal gap into
Temporal IoU (tIoU). To assess the overall temporal coverage between predictions and ground truth, we compute the Intersection over Union (IoU) between their temporal unions. Specifically, for each sample i, let ∪Pi and ∪Gi denote the unions of all predicted and ground-truth segments,
3
OMTG: Towards One-to-Many Temporal Grounding
respectively. The dataset-level tIoU is defined as N 1 X length (∪Pi ) ∩ (∪Gi ) tIoU = N i=1 length (∪Pi ) ∪ (∪Gi )
tions Temporal F1-Score on count consistency: N
2 · tPi @ξ · tRi @ξ 1 XX 1(Mi = Ki ) · EtF1 = N · |Ξ| tPi @ξ + tRi @ξ i=1
(4)
ξ∈Ξ
(8) Here, the indicator 1(Mi = Ki ) acts as a gating function that assigns zero score to samples with incorrect predicted cardinality, and Ξ = {0.3, 0.5, 0.7} denotes the set of IoU thresholds. By explicitly coupling instance-level precision–recall with event-count correctness, EtF1 provides a rigorous and holistic evaluation of OMTG task.
where length(·) measures the total duration of a set of temporal intervals and N denotes the number of samples. Temporal Precision and Recall. Based on the optimal bipartite matching under IoU threshold ξ, we define instancelevel Temporal Precision and Temporal Recall to explicitly characterize prediction correctness and coverage in the Oneto-Many setting. Temporal Precision measures the fraction of predicted segments that correctly match ground-truth instances, penalizing redundant or hallucinated predictions, while Temporal Recall measures the fraction of groundtruth instances that are successfully localized. Formally, for sample i, they are defined as tPi @ξ =
T Pi @ξ , Mi
tRi @ξ =
T Pi @ξ Ki
3.3. OMTG Benchmark We construct a high-quality benchmark consisting of 340 manually curated samples spanning diverse domains, including sports, cooking, and news. We randomly sampled and manually curated videos from the test sets of Charades (Gao et al., 2017), ActivityNet (Krishna et al., 2017), QVHighlights (Lei et al., 2021), VTimeLLM (Huang et al., 2024), and Moment10m (Qian et al., 2024), ensuring that the video sources of our benchmark have no overlap with the training set. Each sample is annotated with precise boundaries and verified by independent experts with a consistency rate exceeding 90%.
(5)
where T Pi @ξ denotes the number of matched prediction–ground-truth pairs whose IoU exceeds ξ, Mi is the number of predicted segments, and Ki is the number of ground-truth segments.
The benchmark presents a diverse and challenging distribution. The number of ground truth segments per query ranges from 2 to 20; while the majority (62.2%) involve 2-3 instances, a significant portion (15%) contains more than 6 occurrences, posing a severe test for counting ability. Regarding temporal duration, the videos span from 21 seconds to over 17 minutes (avg. 221.6s), ensuring robust evaluation across both short clips and long-form narratives. Additional detailed statistics and examples of our benchmark are provided in the Appendix G.
Temporal F1-Score (tF1). To jointly evaluate temporal precision and temporal recall in One-to-Many Temporal Grounding, we report the Temporal F1-Score at an IoU threshold ξ: N
tF 1@ξ =
tPi @ξ · tRi @ξ 1 X 2· N i=1 tPi @ξ + tRi @ξ
(6)
Count Accuracy (C-Acc). To explicitly evaluate the model’s ability to perceive the correct number of event occurrences—a core challenge in One-to-Many Temporal Grounding—we introduce the Count Accuracy metric. It measures the percentage of test samples where the number of predicted segments exactly matches the number of ground truth segments.
4. Method 4.1. Constructing High-Quality OMTG Dataset To facilitate the training of robust OMTG models, we construct the OMTG Dataset, a high-quality instruction tuning dataset comprising approximately 56k samples. The raw videos are sourced from diverse public datasets, including Cosmos-Cap(Wang et al., 2024a), Moment-10M(Qian et al., 2024), and VTimeLLM(Huang et al., 2024). As shown in Fig 3, to transform these raw videos into precise one-tomany supervision signals, we design a rigorous four-stage automated pipeline leveraging state-of-the-art MLLMs.
N
C-Acc =
1 X 1(Mi = Ki ) N i=1
(7)
where N is the total number of samples in the dataset, Mi is the predicted count, Ki is the ground truth count, and 1(·) is the indicator function. A higher C-Acc indicates that the model has learned to count the occurrences of an event in the video.
Stage 1: Repetitive Event Discovery. We employ the powerful Qwen3-VL-235B model as the event discoverer. The model scans the raw videos to identify salient events that occur multiple times. For each identified repetitive event, the model generates a descriptive query, serving as
Effective Temporal F1-Score (EtF1). To jointly enforce accurate localization and correct instance counting, we propose the Effective Temporal F1-Score (EtF1), which condi4
OMTG: Towards One-to-Many Temporal Grounding
event1: man driving event3: front view while driving
event1: <time>15 Ð 61 seconds</time>, <time>83 Ð 126 seconds</time> and <time>135 Ð 142 seconds</time>
Stage1: Repetitive Event Discovery
Stage2: Initial One-to-Many Temporal Grounding
event2: road signs appear
<time>0.0 - 4.0 seconds</time>: The video opens with a wide-angle view of a desert landscape under a bright blue sky dotted with fluffy white clouds. <time>4.0 - 7.0 seconds</time>: The camera angle shifts to show the side of a white van driving on a two-lane road. <time>7.0 - 13.0 seconds</time>: The view cuts to the interior of the van, focusing on a man with a beard, wearing a black beanie and sunglasses, driving. ÉÉ
Stage5: Query-Guided Dense Captioning
discard
query: man in sunglasses is driving. answer: <time>15 Ð 60 seconds</time>, <time>82 Ð 125 seconds</time> and <time>135 Ð 140 seconds</time>
Stage4: Recall Check and Query Refinement
retain Stage3: Visual Verification
Figure 3. Overview of our data construction pipeline: The annotation pipeline includes repetitive event discovery, initial one-to-many grounding, strict visual verification, recall check and query refinement. Temporal Reward
Video&Query GT Segments Policy Model
KL Referenc Model
Outouts GRPO
the surviving samples undergo a semantic refinement phase using Gemini 2.5 Pro. We feed the video, the query, and the verified timestamps back to the model to perform a dual check: (1) Recall Check: Identifying if any valid segments were missed in the previous stages; (2) Query Refinement: Polishing the query text to ensure it is unambiguous and accurately describes the visual commonality of all segments.
Reward Function
tIoU Score C-Acc Score
Caption Reward Pred.; Segments Pred.; Caption
Caption Quality Score Coverage
Precision
Discrimina bility
Caption Guided Grounding Score Caption Guided tF1 Score Grounding
Stage 5: Query-Guided Dense Captioning. Finally, we generate comprehensive, fine-grained captions using Qwen3VL-235B. The model is prompted to identify all distinct activity events in the video. Crucially, the refined queries from Stage 4 serve as mandatory guidance: the model must incorporate their information through detailed elaboration. This yields dense, semantically precise captions that contextualize the repetitive events within the full activity stream. Leveraging these captions, we construct Chain-of-Thought (CoT) and design a caption reward upon this foundation to better guide policy optimization and enhance temporal grounding accuracy.
Length Penalty
Figure 4. Composite reward function optimized via GRPO. The framework combines rule-based rewards for temporal precision with an LLM-as-a-judge mechanism for caption quality evaluation to improve one-to-many temporal grounding.
the initial prompt for the subsequent stages. Stage 2: Initial One-to-Many Grounding. Using the generated queries, we prompt Gemini 2.5 Pro to perform finegrained temporal grounding. The model is instructed to scan the video and return a set of precise start and end timestamps for all occurrences of the event. This step transforms the semantic query into preliminary temporal annotations.
This pipeline results in 56k high-fidelity training samples with dense, verified annotations. We split the dataset into 46k samples for SFT training and 10k samples for RL training. Detailed prompts and additional implementation details are provided in the Appendix A.
Stage 3: Strict Visual Verification. To eliminate hallucinations and inaccurate boundaries, we implement a strict visual verification protocol. We temporally crop the video segments based on the timestamps from Stage 2. Each cropped clip is then fed back into Qwen3-VL-235B to verify whether the visual content strictly aligns with the textual query. We adopt an "All-or-Nothing" filtering strategy: if any single segment within a sample fails the verification (i.e., the model judges it as a mismatch), the entire data sample is discarded. This rigorous filtering ensures that the final dataset maintains an exceptionally high precision rate.
4.2. Achieving Preciseness and Completeness OMTG We conduct SFT based on the OMTG Dataset. SFT stage facilitates the integration of fine-grained temporal localization details into the CoT reasoning process with dense video caption, from which the model can deduce the final grounding results. While SFT provides a strong initialization with chain-of-
Stage 4: Recall Check and Query Refinement. Finally, 5
OMTG: Towards One-to-Many Temporal Grounding Table 1. Main results on the OMTG Bench. We conduct a comprehensive assessment of representative open-source and proprietary MLLMs to establish a comprehensive baseline for the OMTG task. Metrics include Count Accuracy (C-Acc), Temporal F1-Scores ([email protected]/0.5/0.7), average temporal IoU (tIoU), and Effective Temporal F1-Score (EtF1) are reported. The benchmark reveals a critical capability gap: standard open-source models (e.g., Qwen2.5-VL series) yield 0% C-Acc, failing to capture the one-to-many complexity. Model
C-Acc
tIoU
EtF1
Seed-1.8 (Bytedance Seed Team, 2025) Gemini-2.5-Pro (Comanici et al., 2025) Gemini-3-Pro (Comanici et al., 2025)
38.12 50.94 30.63
67.13 55.72 58.30
54.67 43.57 47.75
38.79 27.97 29.89
56.81 43.24 47.63
28.04 27.80 21.30
Qwen2.5-VL-3B (Bai et al., 2025b) Qwen2.5-VL-7B (Bai et al., 2025b) Qwen2.5-VL-32B (Bai et al., 2025b) Qwen2.5-VL-72B (Bai et al., 2025b) Qwen3-VL-4B (Bai et al., 2025a) Qwen3-VL-8B (Bai et al., 2025a) Qwen3-VL-30B (Bai et al., 2025a) Qwen3-VL-235B (Bai et al., 2025a) VideoChat-R1-7B (Li et al., 2025b) VideoChat-R1.5-7B (Yan et al., 2025) Time-R1-7B (Wang et al., 2025) UniTime (Li et al., 2025c) Timelens-8B (Zhang et al., 2025b) OMTG-4B
0.00 0.00 0.00 0.00 0.31 0.00 0.00 0.31 0.00 0.31 0.00 0.00 0.00 55.63
15.17 21.04 16.81 21.16 37.07 37.73 37.03 34.66 32.07 28.41 28.94 35.27 39.14 73.46
7.01 12.08 9.66 12.20 26.75 27.02 25.98 25.25 19.70 15.53 18.73 30.15 32.76 65.40
2.86 7.14 4.76 6.88 17.93 18.70 17.52 16.45 10.42 9.85 10.00 23.58 22.58 48.96
11.60 20.35 18.32 20.02 30.42 30.62 32.36 25.56 24.93 27.96 24.11 37.12 32.38 61.24
0.00 0.00 0.00 0.00 0.21 0.00 0.00 0.21 0.00 0.10 0.00 0.00 0.00 43.65
Table 2. Performance gain of our method on OMTG Bench.
Table 3. Ablation on different reward functions on OMTG Bench.
Model C-Acc [email protected] [email protected] [email protected] tIoU EtF1
Reward Functions
Base
0.31
37.07
26.75
17.93
30.42 0.21
SFT RL
44.06 55.63
69.57 73.46
61.23 65.40
45.63 48.96
56.94 34.81 61.24 43.65
RtIoU +0.31 RtIoU + RC-Acc +7.50 RtIoU + RC-Acc + RCaption +11.57
C-Acc [email protected] [email protected] [email protected] tIoU EtF1 +2.01 +4.69 +3.89
+1.82 +3.86 +4.17
+0.28 +3.27 +3.33
+2.61 +0.74 +4.03 +5.87 +4.30 +8.84
thought reasoning capabilities, including generating descriptive captions before final predictions, it often struggles to balance the trade-offs between retrieval completeness and localization precision. To address this, we design a composite reward function optimized via the GRPO (Shao et al., 2024) algorithm:
Caption Reward. To encourage the model to generate informative intermediate reasoning during chain-of-thought, we introduce a caption quality reward RCaption that evaluates the descriptive captions produced before final predictions. We employ a two-part LLM-as-Judge evaluation framework using Qwen3-30B-A3B (Yang et al., 2025a) as the reward model.
R = λ1 RtIoU + λ2 RC-Acc + λ3 RCaption + λ4 RLength
The first part computes a Caption Quality Score (Scq ), which evaluates three dimensions with access to ground truth: coverage (Scov ), measuring whether all ground truth segments are matched; precision (Sprec ), assessing boundary alignment accuracy; and discriminability (Sdisc ), determining whether captions provide unique contextual information:
(9)
where RtIoU and RC-Acc are defined following the metric formulations in Section 3.1. We set λ1 = λ2 = λ3 = 0.5 and λ4 = −0.3 to balance temporal localization quality, counting completeness, caption quality, and response conciseness. Figure 4 intuitively presents the overall design of our proposed reward function.
Scq = µ1 · Scov + µ2 · Sprec + µ3 · Sdisc
Temporal Reward. In Eq. 9, RtIoU serves as a foundational component for refining temporal boundaries, whose effectiveness has been extensively validated in prior works (Wang et al., 2025; Li et al., 2025b; 2024; Yan et al., 2025). Complementing this, we introduce RC-Acc as a strict constraint that activates only when the predicted segment count exactly matches the ground truth, explicitly correcting the model’s perception of event cardinality.
(10)
The second part computes a Caption Guided Grounding Score (Scgg ), where the judge attempts to localize event timestamps by reading only the generated captions without access to the video. The predicted intervals are then compared against ground truth using tF1 scores to measure localization accuracy. This ensures that captions contain sufficient semantic information to independently support event localization. 6
OMTG: Towards One-to-Many Temporal Grounding Table 4. Main results on One-to-One Temporal Grounding (OOTG) benchmarks. We benchmark the performance of various state-of-the-art proprietary and open-source models on TimeLens-Bench. Charades ActivityNet QVHighlights [email protected] [email protected] [email protected] tIoU [email protected] [email protected] [email protected] tIoU [email protected] [email protected] [email protected] tIoU
Model Proprietary Models GPT-4o (OpenAI, 2023) GPT-5 (OpenAI, 2023) Gemini-2.5-Pro (Comanici et al., 2025)
60.6 59.3 74.1
44.5 42.0 61.1
23.5 22.0 34.0
41.8 40.5 52.8
55.2 57.4 72.3
41.4 44.9 64.2
25.8 30.4 47.1
40.4 42.9 58.1
69.0 72.4 84.1
54.8 60.4 75.9
38.5 46.4 61.1
52.1 56.8 70.4
Open-Source Models VideoChat-Flash-7B (Li et al., 2024) VideoChat-R1-7B (Li et al., 2025b) Time-R1-7B (Wang et al., 2025) Qwen2.5-VL-7B (Bai et al., 2025b) TimeLens-7B (Zhang et al., 2025b) Qwen3-VL-4B (Bai et al., 2025a) Qwen3-VL-8B (Bai et al., 2025a) OMTG-4B (SFT) OMTG-4B (RL)
60.2 51.9 57.9 59.7 70.5 67.1 69.2 70.4 72.0
37.9 30.8 32.0 37.8 55.6 47.5 53.4 56.9 58.3
17.8 11.7 16.9 16.6 28.4 24.1 27.5 31.2 32.0
39.7 33.7 36.6 39.3 48.8 45.6 48.3 50.0 50.5
35.5 35.0 44.8 44.1 62.8 62.9 62.1 59.5 64.7
21.8 23.9 31.0 31.0 51.0 51.0 51.2 50.2 54.6
10.5 11.3 19.0 16.1 32.6 34.3 34.4 34.4 37.0
24.8 25.0 33.1 31.4 46.2 47.2 46.8 45.8 46.8
45.2 29.3 65.8 41.5 74.1 77.6 74.2 76.5 80.9
30.6 19.1 51.5 27.8 62.7 66.0 64.6 65.4 70.4
16.7 9.4 36.1 15.2 43.1 50.9 49.3 50.8 53.5
32.7 21.5 49.2 31.6 56.0 61.7 59.4 61.5 62.2
The final caption reward combines these two components: RCaption = α · Scq + (1 − α) · Scgg
the refined version of the TimeLens (Zhang et al., 2025b) dataset. Following standard protocols, we report Recall@1 (R@1) at IoU thresholds of 0.3, 0.5, and 0.7, alongside tIoU.
(11)
For training, we construct task-specific data mixtures. In the SFT stage, we utilize a high-quality mixture comprising 46k samples from our OMG-TG Dataset and 32k samples from TimeLens-100k (Zhang et al., 2025b). In the subsequent RL stage, we exclusively utilize a subset of 10k samples from the OMG-TG Dataset to focus on alignment with complex grounding objectives.
Detailed formulations, coefficient settings and prompt templates are provided in Appendix B. Length Penalty. Excessively long captions introduce irrelevant details, dilute query-relevant temporal cues, and degrade localization performance. We therefore adopt a soft length penalty RLength that progressively penalizes responses exceeding predefined thresholds, preventing the model from being distracted from the core temporal grounding task. The specific formulation of the length penalty is detailed in the Appendix B.
Implementation Details. Our primary experiments use Qwen3-VL-4B (Bai et al., 2025a) as the backbone. During the SFT stage, the model is trained for 1 epoch on 16 NVIDIA H100 GPUs (∼5 hours) using the AdamW(Loshchilov & Hutter, 2017) optimizer. The learning rate is set to 1e-5 with a cosine scheduler and a 0.03 linear warmup. We use a global batch size of 64 with 4 gradient accumulation steps.
We also explored alternative reward combinations such as RtIoU + RC-Acc . Through comprehensive ablation studies (see Section 5.3), we find that the combination of RtIoU + RC-Acc + RCaption + RLength achieves strong performance for One-to-Many Temporal Grounding. By jointly optimizing these components via GRPO, we achieve a holistic alignment that ensures both accurate event counting and precise temporal localization.
During the RL stage, we employ Group Relative Policy Optimization (GRPO) for 308 steps on 16 NVIDIA H100 GPUs (∼30 hours).
5. Experiment
We perform 8 rollouts per prompt with a global batch size of 64. DeepSpeed (Rasley et al., 2020) ZeRO-2 and Flash Attention (Dao et al., 2022) are utilized for optimization.
5.1. Experiments Setup
5.2. Main Results
Datasets and Metrics. We conduct a comprehensive evaluation across two distinct settings: One-to-Many Temporal Grounding (OMTG) and One-to-One Temporal Grounding (OOTG). For the OMTG task, we evaluate models on our proposed OMTG Bench. We report a holistic set of metrics, including Count Accuracy (C-Acc), Time F1-Score ([email protected], 0.5, 0.7), our proposed Effective Time F1 (EtF1), and the traditional tIoU. For the OOTG task, we evaluate on
Results on OMTG Task. As shown in Table 1, we conduct a comprehensive evaluation on our proposed OMTG Bench. The results reveal a critical capability gap in existing MLLMs: open-source models and traditional TG experts struggle significantly in the OMTG task, often yielding near-zero EtF1 scores; advanced proprietary models (e.g., Gemini series (Comanici et al., 2025), Seed-1.8 (Bytedance
7
OMTG: Towards One-to-Many Temporal Grounding Several newspapers are drawn out across a wooden floor
A man is in a bathroom talking to the camera
Video
Video
GT Gemini3Pro Seed 1.8 Qwen3VL Ours SFT Ours RL
GT Gemini3Pro Seed 1.8 Qwen3VL Ours SFT Ours RL Family photos of Disney World are shown
A man stands by a lake an talks to the camera
Video
Video
GT Gemini3Pro Seed 1.8 Qwen3VL Ours SFT Ours RL
GT Gemini3Pro Seed 1.8 Qwen3VL Ours SFT Ours RL
Figure 5. Qualitative visualization of One-to-Many Temporal Grounding. We compare the predicted temporal segments from our model against state-of-the-art baselines across four diverse datasets. The green bars denote the Ground Truth. Baselines: Existing MLLMs (e.g., Gemini 3 Pro, Qwen3-VL) typically fail to capture the repetitiveness nature of events. They often retrieve only a single segment (e.g., Gemini 3 Pro in the top-left) or incorrectly merge distinct segments into a continuous span (e.g., Qwen3-VL, Seed-1.8). Ours: In contrast, our model (Ours RL) accurately localizes all disjoint event occurrences, demonstrating superior capability in both event cardinality perception and boundary precision.
5.3. Ablation Studies
Seed Team, 2025)) demonstrate weak OMTG capability. In contrast, our OMTG-4B achieves state-of-the-art performance across all metrics, attaining an EtF1 of 43.65, which outperforms the best proprietary baselines by over 15.61%.
Ablation on Training Strategy. As shown in Table 2, the Qwen3VL-4B base model demonstrates minimal capability on the OMTG task, achieving near-zero performance (EtF1: 0.21). However, SFT with our OMTG dataset fundamentally enables OMTG ability, significantly improving EtF1 to 34.81. Subsequently, the RL stage further improves this capability, boosting EtF1 to 43.65 (+8.84). Crucially, this progressive improvement confirms that while SFT establishes foundational OMTG ability, reinforcement learning with our proposed temporal and caption rewards provides additional alignment beyond supervised training alone.
Qualitative comparisons in Figure 5 further illustrate the typical failure modes of existing models, including undersegmentation and over-segmentation, while demonstrating our method’s superior ability to accurately identify all event occurrences with precise temporal boundaries. Qualitative comparisons in Figure 5 further illustrate the typical failure modes of existing models, including undersegmentation and over-segmentation, while demonstrating our method’s superior ability to accurately identify all event occurrences with precise temporal boundaries.
Ablation on RL Reward Design. As shown in Table 3, adding RtIoU yields consistent improvements in localization metrics. Adding RC-Acc further improves counting accuracy and EtF1, indicating that a direct cardinality signal benefits event perception. Incorporating RCaption achieves the best performance, improving C-Acc by 11.57 and EtF1 by 8.84 over the SFT baseline. We attribute this to dense captioning, which enforces fine-grained temporal perception and requires the model to explicitly distinguish each event occurrence. Note that RCaption is used in conjunction with RLength , which prevents excessively verbose outputs that could dilute query-relevant temporal cues. Based on these findings, we adopt tIoU + C-Acc + Caption + Length as our default configuration.
Results on One-to-One Temporal Grounding. To verify that our approach does not compromise performance on conventional single-segment grounding, we evaluate on the TimeLens (Zhang et al., 2025b) benchmark (Table 4). Our OMTG-4B consistently improves upon both the base model and domain-specific baselines across all three datasets. Notably, the RL stage, trained exclusively on OMTG data without any one-to-one supervision, yields further gains over SFT across all benchmarks, which suggests that the one-tomany formulation cultivates more generalizable temporal grounding capabilities.
8
OMTG: Towards One-to-Many Temporal Grounding
6. Conclusion
Zhou, J., Zhu, Y., and Zhu, K. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025a.
In this paper, we identify and formalize the task of One-toMany Temporal Grounding (OMTG), addressing the critical gap between current one-to-one paradigms and dynamic real-world scenarios.
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b.
We reveal that existing state-of-the-art MLLMs, despite their success in standard settings, struggle significantly to perceive event cardinality and localize disjoint segments.
Bao, P., Kong, C., Shao, Z., Ng, B. P., Er, M. H., and Kot, A. C. Vid-morp: Video moment retrieval pretraining from unlabeled videos in the wild. arXiv preprint arXiv:2412.00811, 2024.
To bridge this gap, we curate 56k high-quality one-to-many training samples via a sophisticated data pipeline and conduct SFT+RL training that incorporates our temporal and caption rewards, achieving state-of-the-art results.
Bytedance Seed Team. Seed1.8 Model Card: Towards Generalized Real-World Agency. Preprint, Bytedance Seed, dec 2025.
Our study establishes a strong baseline for this novel OMTG setting and facilitates future research in this direction.
Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Limitations and Future Works. Our current approach incurs high training costs and faces scalability challenges with extremely long videos. Future works will explore OMTG with memory in long videos setting.
Impact Statement
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344–16359, 2022.
This paper presents work aimed at advancing the field of Machine Learning, specifically in fine-grained video understanding and retrieval. Our proposed One-to-Many Temporal Grounding framework has the potential to positively impact society, such as enhancing video search efficiency, automating video editing workflows, and improving content accessibility. However, we acknowledge that advancements in precise temporal localization could potentially be misused in surveillance or privacy-intrusive applications. We explicitly condemn any use of our technology that violates individual privacy or human rights. We encourage the community to prioritize data privacy and responsible deployment when applying these technologies to sensitive real-world scenarios.
Deng, H., Zou, D., Ma, R., Luo, H., Cao, Y., and Kang, Y. Boosting the generalization and reasoning of visionlanguage models with curriculum reinforcement learning. arXiv preprint arXiv:2503.07065, 2025. Feng, K. et al. Video-r1: Reinforcing video reasoning in multimodal large language models. arXiv preprint arXiv:2503.21776, 2025. Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24108–24118, 2025.
References Alayrac, J.-B. et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022.
Gao, J., Sun, C., Yang, Z., and Nevatia, R. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pp. 5267–5275, 2017.
Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, J., Tu, J., Wan, J., Wang, P., Wang, P., Wang, Q., Wang, Y., Xie, T., Xu, Y., Xu, H., Xu, J., Yang, Z., Yang, M., Yang, J., Yang, A., Yu, B., Zhang, F., Zhang, H., Zhang, X., Zheng, B., Zhong, H., Zhou, J., Zhou, F.,
Guo, Y., Liu, J., Li, M., Liu, Q., Chen, X., and Tang, X. Trace: Temporal grounding video llm via causal event modeling. arXiv preprint arXiv:2410.05643, 2024. Huang, B., Wang, X., Chen, H., Song, Z., and Zhu, W. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14271–14280, 2024. 9
OMTG: Towards One-to-Many Temporal Grounding
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., and Carlos Niebles, J. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp. 706–715, 2017.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. Lu, W., Li, J., Yu, A., Chang, M.-C., Ji, S., and Xia, M. Llava-mr: Large language-and-vision assistant for video moment retrieval. arXiv preprint arXiv:2411.14505, 2024.
Lei, J., Berg, T. L., and Bansal, M. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34: 11846–11858, 2021.
Meng, J., Li, X., Wang, H., Tan, Y., Zhang, T., Kong, L., Tong, Y., Wang, A., Teng, Z., Wang, Y., et al. Open-o3 video: Grounded video reasoning with explicit spatiotemporal evidence. arXiv preprint arXiv:2510.20579, 2025a.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
Meng, J., Sun, S., Tan, Y., Qi, L., Tong, Y., Li, X., and Wen, L. Cyberv: Cybernetics for test-time scaling in video understanding. arXiv preprint arXiv:2506.07971, 2025b.
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., and Qiao, Y. Videochat: Chat-centric video understanding. Science China Information Sciences, 68 (10):200102, 2025a.
Moon, W., Hyun, S., Park, S., Park, D., and Heo, J.-P. Querydependent video representation for moment retrieval and highlight detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23023–23033, 2023.
Li, X., Wang, Y., Yu, J., Zeng, X., Zhu, Y., Huang, H., Gao, J., Li, K., He, Y., Wang, C., et al. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574, 2024.
OpenAI. Gpt-4 technical report. arXiv:2303.08774, 2023.
Li, X., Yan, Z., Meng, D., Dong, L., Zeng, X., He, Y., Wang, Y., Qiao, Y., Wang, Y., and Wang, L. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958, 2025b.
arXiv preprint
Qian, L., Li, J., Wu, Y., Ye, Y., Fei, H., Chua, T.-S., Zhuang, Y., and Tang, S. Momentor: advancing video large language model with fine-grained temporal reasoning. In Proceedings of the 41st International Conference on Machine Learning, pp. 41340–41356, 2024.
Li, Z., Di, S., Zhai, Z., Huang, W., Wang, Y., and Xie, W. Universal video temporal grounding with generative multi-modal large language models. arXiv preprint arXiv:2506.18883, 2025c.
Qin, H., Xiao, J., and Yao, A. Question-answering dense video events. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 884–894, 2025.
Lin, K. Q., Zhang, P., Chen, J., Pramanick, S., Gao, D., Wang, A. J., Yan, R., and Shou, M. Z. Univtg: Towards unified video-language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2794–2804, 2023.
Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, pp. 3505–3506, 2020. doi: 10.1145/3394486.3406703.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023. Liu, Y., Li, S., Wu, Y., Chen, C.-W., Shan, Y., and Qie, X. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3042–3051, 2022.
Ren, S., Yao, L., Li, S., Sun, X., and Hou, L. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14313–14323, 2024.
Liu, Y., Lin, K. Q., Chen, C. W., and Shou, M. Z. Videomind: A chain-of-lora agent for long video reasoning. arXiv preprint arXiv:2503.13444, 2025a.
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Liu, Z., Sun, Z., Zang, Y., Dong, X., Cao, Y., Duan, H., Lin, D., and Wang, J. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025b. 10
OMTG: Towards One-to-Many Temporal Grounding
Wang, A. J., Li, L., Lin, K. Q., Wang, J., Lin, K., Yang, Z., Wang, L., and Shou, M. Z. Cosmo: Contrastive streamlined multimodal model with interleaved pre-training. arXiv preprint arXiv:2401.00849, 2024a.
Zhang, J., Wang, T., Ge, Y., Ge, Y., Li, X., Shan, Y., and Wang, L. Timelens: Rethinking video temporal grounding with multimodal llms. arXiv preprint arXiv:2512.14698, 2025b.
Wang, Y., Meng, X., Liang, J., Wang, Y., Liu, Q., and Zhao, D. Hawkeye: Training video-text llms for grounding text in videos, 2024b.
Zheng, M., Cai, X., Chen, Q., Peng, Y., and Liu, Y. Trainingfree video temporal grounding using large-scale pretrained models. In European Conference on Computer Vision, pp. 20–37. Springer, 2024.
Wang, Y., Wang, Z., Xu, B., Du, Y., Lin, K., Xiao, Z., Yue, Z., Ju, J., Zhang, L., Yang, D., et al. Time-r1: Posttraining large vision language model for temporal video grounding. arXiv preprint arXiv:2503.13377, 2025.
Zhou, H., Li, X., Wang, R., Cheng, M., Zhou, T., and Hsieh, C.-J. R1-zero’s “aha moment” in visual reasoning on a 2b non-sft model. arXiv preprint arXiv:2503.05132, 2025.
Yan, Z., Li, X., He, Y., Yue, Z., Zeng, X., Wang, Y., Qiao, Y., Wang, L., and Wang, Y. Videochat-r1. 5: Visual test-time scaling to reinforce multimodal reasoning by iterative perception. arXiv preprint arXiv:2509.21100, 2025. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. Yang, Y., He, X., Pan, H., Jiang, X., Deng, Y., Yang, X., Lu, H., Yin, D., Rao, F., Zhu, M., et al. R1-onevision: Advancing generalized multimodal reasoning through crossmodal formalization. arXiv preprint arXiv:2503.10615, 2025b. Yu, S., Cho, J., Yadav, P., and Bansal, M. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36:76749–76771, 2023. Zeng, X., Li, K., Wang, C., Li, X., Jiang, T., Yan, Z., Li, S., Shi, Y., Yue, Z., Wang, Y., et al. Timesuite: Improving mllms for long video understanding via grounded tuning. arXiv preprint arXiv:2410.19702, 2024. Zhan, Y., Zhu, Y., Zheng, S., Zhao, H., Yang, F., Tang, M., and Wang, J. Vision-r1: Evolving human-free alignment in large vision-language models via vision-guided reinforcement learning. arXiv preprint arXiv:2503.18013, 2025. Zhang, D., Dai, X., Wang, X., Wang, Y.-F., and Davis, L. S. Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1247–1257, 2019. Zhang, H., Li, X., and Bing, L. Video-llama: An instructiontuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023. Zhang, J., Huang, J., Yao, H., Liu, S., Zhang, X., Lu, S., and Tao, D. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937, 2025a. 11
OMTG: Towards One-to-Many Temporal Grounding
Appendix Overview • Section A: gives more details on training data pipeline building process. • Section B: presents more details on our reward function designs. • Section C: shows the results on Video MME benchmark. • Section D: shows the results across different model size. • Section E: give the details of OMTG benchmarking process. • Section F: presents zero-shot OOD evaluation on longer in-the-wild videos. • Section G: analyzes the statistics details. • Section H: analyzes failure cases • Section I: shows the annotation process of OMTG and the human check process.
12
OMTG: Towards One-to-Many Temporal Grounding
A. More Details of Training Data Pipeline A.1. Prompt Templates In this section, we provide the detailed prompt templates for each stage in our proposed data pipeline. Table 5. Prompt template for Stage 1: Repetitive Event Discovery. Stage 1: Repetitive Event Discovery The task is Repetitive Event Discovery. You need to scan the raw video to identify salient events that occur multiple times (repetitive events). Task Based on the content of the video, generate descriptive queries for these repetitive events. Place queries in <query></query> tags. If no salient repetitive events exist, enter “No” in <judge></judge>; otherwise enter “Yes”. Requirements (1) Repetition Requirement The identified event must occur at least twice as distinct instances. It should NOT be a continuous state lasting the entire video. Example: Do not select “a man standing” if he stands there the whole time. (2) Format Requirement Output must be descriptive: phrase or short sentence. Format: “subject + action (+ object/environment)” Examples: “a person jumping over a fence”, “a dog catching a frisbee” (3) Salience Requirement Query should correspond to main content or significant actions. Must serve as meaningful entry for event localization. Output Format Examples No repetitive events: <judge>No</judge> With repetitive events: <judge>Yes</judge><query>a person jumping over a fence</query> <query>basketball player shooting a three-pointer</query>
Table 6. Prompt template for Stage 2: Initial One-to-Many Grounding. Stage 2: Initial One-to-Many Grounding Given a textual query, determine when the described content occurs in the video. Input Textual Query: “{query}” Task Identify all temporal segments where the queried event occurs. Return timestamps in seconds. If the specified query occurs multiple times, output multiple relevant time segments. Output Format Return start and end timestamps for each occurrence.
13
OMTG: Towards One-to-Many Temporal Grounding Table 7. Prompt template for Stage 3: Strict Visual Verification. Stage 3: Strict Visual Verification Verify whether the video segment satisfies the conditions described in the textual query. Input Textual Query: “{query}” Video Segment: [extracted segment from Stage 2] Task Determine if the content in the video segment perfectly and completely satisfies ALL conditions described in the textual query. Decision Criteria Answer “Yes” if and only if ALL conditions are met. Answer “No” otherwise. Output Format Binary response: “Yes” or “No” Table 8. Prompt template for Stage 4: Recall Check and Query Refinement. Stage 4: Recall Check and Query Refinement. You are an expert video temporal grounding annotator with exceptional attention to detail. Your task is to verify and refine temporal annotations for a specific query in a video. Context Query: “{query}” Dense Video Caption: {dense_caption} Previous Prediction: {previous_prediction} Task Carefully watch the video and identify ALL segments where the query occurs. 1. Verify existing predictions: Check if previously annotated segments contain the queried event 2. Find missing segments: Identify any additional occurrences that were missed 3. Refine boundaries: Adjust start/end times to precisely capture event timing 4. Remove false positives: Exclude segments that don’t match the query 5. Refine query if needed: If original query doesn’t match but similar action occurs, refine it Critical Guidelines • Watch the ENTIRE video carefully before making annotations • The query may appear multiple times — find ALL occurrences • Be precise with timestamps — round to the nearest second • Only include segments where the query is CLEARLY happening • Consider semantic equivalence (e.g., “cleaning carpet” includes scrubbing, vacuuming) • Do NOT include segments where someone talks about the action without performing it • Continuous actions should be ONE segment, not multiple 1-second segments • Segments should have meaningful duration (typically at least 2–3 seconds) Query Refinement Guidelines • If exact query doesn’t appear, check for SIMILAR action around predicted timestamps • Refined query should be concise and descriptive • Examples: “person peeling egg” → “person breaking egg with hands” • Only refine if action is genuinely similar/related Output Format Respond with ONLY a JSON object: {“original_query”: “...”, “query_refined”: true/false, “refined_query”: “...”, “segments”: [{“start”: int, “end”: “reasoning”: “...”}
14
int}, ...],
OMTG: Towards One-to-Many Temporal Grounding
Table 9. Prompt template for Stage 5: Query-Guided Dense Captioning. Stage 5: Query-Guided Dense Captioning Analyze the given video and capture all distinct activity events occurring within it. For each event, provide a clear, descriptive label and specify its exact time interval. Input Video: [input video] Reference Queries: “{query}” Important Requirements (1) Fine-grained Timestamps Break down long continuous activities into smaller, meaningful segments whenever possible. (2) Perceptible Changes Each segment should represent a perceptible change in action, behavior, or context. (3) Temporal Continuity Events should collectively cover the entire video without gaps or overlaps. (4) Query Integration Include information from the given queries; describe these in detail within the content. Note: Do NOT directly copy queries — they require more granular refinement. (5) Precision over Brevity Prioritize precision and semantic relevance over brevity. (6) Detailed Captions Make generated captions as detailed as possible. Avoid overly broad or prolonged time intervals. Guideline: No single event should span more than 10–15 seconds unless clearly justified. Output Format For each event: descriptive label + time interval as “start – end seconds”
15
OMTG: Towards One-to-Many Temporal Grounding
A.2. Quality Control To ensure the semantic consistency of our dataset, we implement a Strict Visual Check mechanism following the initial grounding (Stage 2). Given the query Q and the set of predicted segments S = {s1 , s2 , ..., sN } from Gemini 2.5 Pro, we employ a powerful open-source MLLM, Qwen3-VL-235B, as the verifier. Mechanism. As outlined in Algorithm 1, the process operates on a "one-vote veto" principle. For a sample to be retained, every individual segment si must pass the visual verification against the query Q. If any segment is deemed irrelevant by the verifier, the entire sample is discarded. Algorithm 1 Strict Visual Check Pipeline Input: Video V , Query Q, Segments S = {s1 , ..., sN } Model: Verifier M (Qwen3-VL-235B) Output: Boolean (Keep or Discard) for i = 1 to N do vi ← CropVideo(V, si ) {Extract video clip} result ← M(vi , Q) {Verify alignment} if result is Negative then return Discard {One-vote veto} end if end for return Keep Theoretical Proof of Quality Gain We define the Quality Gain as the relative improvement of the sample validity probability after passing the visual check compared to the raw probability. Formulation. Let θ be the prior probability of a segment mismatch, and p be the verifier’s error rate. Based on the independence assumption: • Prior Validity (No Check): P (Valid) = (1 − θ)N
• Posterior Validity (Passed Check): P (Valid|Pass) =
(1−θ)(1−p) (1−θ)(1−p)+θp
N
Quantifying the Improvement. We measure the improvement using the Relative Lift Quality Gain (L), defined as the ratio of the posterior to the prior: N P (Valid|Pass) 1−p L(N ) = = (12) P (Valid) 1−θ−p 1−p Let the base term be β = 1−θ−p . As proved previously, if 0 < p < 1 and 0 < θ < 1 and 0 < 1 − θ − p < 1, then β > 1. Consequently, the lift L(N ) = β N grows exponentially with N . This implies that the visual check is significantly more effective at filtering noise for complex samples (higher N ) than for simple ones.
Numerical Analysis (N = 2 vs. N = 4). Based on our statistics, let θ ≈ 0.5 (raw data noise, we roughly assume this equals to 1 − C-Acc of Gemini 2.5 pro) and p ≈ 0.2 (verifier error). • Base Term β: β=
1 − 0.2 = 2.67 1 − 0.5 − 0.2
(13)
• Improvement for N = 2: L(2) = 2.672 ≈ 7.13× 16
(Quality boosted by 7 times)
(14)
OMTG: Towards One-to-Many Temporal Grounding
• Improvement for N = 4: L(4) = 2.674 ≈ 50.82×
(Quality boosted by 50 times)
(15)
The calculation demonstrates that the quality improvement for N = 4 is substantially higher than for N = 2. Given this and based on our experimental results, we directly accept samples with N ≥ 4 without further processing. Conversely, for samples with fewer segments (N = 2, 3), we employ Stage 4: Recall Check and Query Refinement to further boost data quality. Experiments in Tab. 10 have shown that this strategy is very effective in improving data quality. Table 10. Performance comparison between using data w/ or w/o quality control. Model
C-Acc [email protected] [email protected] [email protected] tIoU EtF1
SFT data w/o quality control 39.17 RL data w/o quality control 40.25
66.32 65.41
59.43 60.01
40.65 41.72
53.82 29.73 54.14 32.55
SFT data w/ quality control RL data w/ quality control
69.57 73.46
61.23 65.40
45.63 48.96
56.94 34.81 61.24 43.65
44.06 55.63
B. More Details of Reward Functions Design In this section, we provide the detailed prompt templates for caption reward evaluation and the formulation of the length penalty. B.1. Caption Reward Prompts The caption reward RCaption employs a two-part LLM-as-Judge evaluation framework using Qwen3-30B-A3B as the reward model. Both parts are computed in parallel to assess complementary aspects of caption quality. Part 1: Caption Quality Score. The Caption Quality Score (Scq ) evaluates captions with access to ground truth annotations across three dimensions. Coverage (Scov ) measures what fraction of ground truth segments are matched by corresponding captions with appropriate descriptions. Precision (Sprec ) assesses how accurately the temporal boundaries of captions align with ground truth intervals, penalizing both undershooting and overshooting. Discriminability (Sdisc ) determines whether each caption provides unique contextual information (e.g., who, what, when, where) to distinguish different occurrences of the same event. The composite score is computed as: Scq = µ1 · Scov + µ2 · Sprec + µ3 · Sdisc
(16)
where µ1 = 0.5, µ2 = 0.3, and µ3 = 0.2 to emphasize coverage completeness. The prompt template is shown in Table 11. Part 2: Caption Guided Grounding Score. The Caption Guided Grounding Score (Scgg ) evaluates whether the generated captions contain sufficient information for event localization. Given only the text query and generated captions (without access to the video), the judge identifies all segments where the queried event likely occurs by matching caption descriptions to the query semantics. The predicted intervals are then compared against ground truth using tF1 scores at IoU thresholds of 0.3 and 0.5: [email protected] + [email protected] Scgg = (17) 2 This text-only grounding evaluation ensures that captions are semantically informative rather than merely temporally co-occurring with ground truth. The prompt template is shown in Table 12. The final caption reward combines the two components: RCaption = α · Scq + (1 − α) · Scgg
(18)
where α = 0.5 balances quality assessment and grounding consistency. B.2. Length Penalty Excessively long responses can introduce irrelevant details, dilute query-relevant temporal cues, and degrade localization performance. We adopt a soft length penalty that progressively penalizes responses exceeding predefined thresholds. 17
OMTG: Towards One-to-Many Temporal Grounding
Soft Overlong Punishment. For a given text length L, we define the soft overlong penalty function as: 0 if L ≤ Lsoft P (L; Lsoft , Lhard , α) = α · L − Lsoft if Lsoft < L ≤ Lhard Lhard − Lsoft α if L > Lhard
(19)
where Lsoft is the soft threshold below which no penalty is applied, Lhard is the hard threshold beyond which the maximum penalty is reached, and α is the penalty factor. Total Length Penalty. The total length penalty consists of two components. Thinking Content Penalty penalizes overly verbose reasoning in the <think> block: Pthink = P (Lthink ; 2000, 5000, 1.0)
(20)
where Lthink is the character count of the thinking content. Caption Length Penalty is applied for excessively long captions. The average caption length penalty across all N captions is: N
Pcap =
1 X P (L(i) cap ; 100, 200, 0.5) N i=1
(21)
(i)
where Lcap is the character count of the i-th caption. The final length penalty is computed as: RLength = Pthink + Pcap
(22)
B.3. Temporal Rewards Design Choices To identify the optimal supervision for the temporal branch, we conduct an ablation study on different combinations of temporal rewards: RtIoU , RtF1 , and RC-Acc . The definitions of these rewards strictly follow the metrics defined in Section 3.2. We report the performance gains over the SFT baseline in Tab 13. Critical Role of Cardinality Supervision. As shown in Table 3, using only the boundary-aware reward (RtIoU ) yields negligible improvement in Count Accuracy (+0.31 C-Acc). This indicates that standard overlap-based objectives encourage the model to refine local boundaries but fail to rectify the number of predicted segments (e.g., merging two distinct events or splitting one event). However, incorporating the cardinality-aware reward (RtIoU + RC-Acc ) results in a substantial performance leap, particularly in C-Acc (+9.06) and the comprehensive metric EtF1 (+7.91). This confirms that explicit supervision on event counts is indispensable for One-to-Many Temporal Grounding, as it forces the model to discern the discrete nature of multiple occurrences. Redundancy in Dense Temporal Rewards. We further investigate the effect of adding the Temporal F1 reward (RtF1 ). Surprisingly, the combination of all three rewards (RtIoU + RtF1 + RC-Acc ) leads to a performance degradation compared to the simpler RtIoU + RC-Acc setting. We hypothesize that RtIoU and RtF1 provide overlapping supervision signals regarding localization quality. Optimizing these redundant, dense objectives simultaneously may dilute the gradient signal from the sparse cardinality reward RC-Acc . Consequently, we adopt RtIoU + RC-Acc as our final temporal reward design choice.
C. More Results on Video MME To assess whether our specialized training for One-to-Many Temporal Grounding compromises the model’s general video understanding capabilities, we evaluated our models on the VideoMME (Fu et al., 2025) benchmark. We report results under the setting without subtitles (w/o sub), sampling 128 frames per video. The results are summarized in Tab 14. As expected, domain-specific fine-tuning typically incurs a trade-off in general capabilities. The naive SFT model (w/o CoT) exhibits a performance drop compared to the backbone Qwen3-VL (62.1 vs. 66.7). However, our proposed strategies effectively mitigate this issue: 18
OMTG: Towards One-to-Many Temporal Grounding Table 11. Prompt template for Scq evaluation. Part 1: Scq Evaluation Prompt You are a STRICT evaluator for Video Temporal Grounding caption quality. Context Query: “{query}” Ground Truth: {num_gt_intervals} segment(s) at {gt_intervals_str} Video duration: approximately {video_duration}s Model’s Captions {caption_list_str} Evaluation Task Step 1: Map each GT to captions For each GT segment, find the BEST matching caption (if any). A match requires: (1) temporal overlap, AND (2) caption describes “{query}” Step 2: Score STRICTLY using these rules Scov (0–10): What fraction of GT segments are matched? 10 = ALL {num_gt_intervals} GT matched with clear “{query}” descriptions 8 = ALL matched, but one has weak description 6 = approx. 70% matched 2 = Only one matched
4 = approx. 50% matched 0 = None matched
Note: If ANY GT is missing, score at most 8 Sprec (0–10): How close are boundaries? 10 = ALL within 1s of GT 4 = Off by 5–10s
8 = Most within 2s
6 = Within 3–5s
2 = Off by more than 10s
Note: Captions much WIDER than GT count as imprecise Sdisc (0–10): Can occurrences be distinguished? 10 = Each has a unique context (who/what/when/where) 7 = Good context for most
4 = Generic
0 = Impossible to distinguish
Output Format After analysis, output ONLY valid JSON: {“coverage”:
int 0-10, “precision”:
int 0-10, “discriminability”:
int 0-10}
BE STRICT: Average captions score 4–6, not 8–10.
Impact of CoT: Incorporating Chain-of-Thought (CoT) data during SFT significantly recovers general performance (+2.3% Overall), particularly in Short videos where it matches the backbone (77.6). This suggests that enhancing reasoning capabilities benefits both temporal grounding and general video understanding. Impact of RL with Caption Reward: The RL stage further improves the performance. Notably, including the Caption Reward is crucial; it boosts the Overall score to 65.1, narrowing the gap with the backbone to a minimal margin. This indicates that the Caption Reward helps the model maintain high-quality semantic representations while optimizing for grounding metrics. In summary, our final OMTG-4B model evolves into a specialist in temporal grounding while remaining a robust generalist in video understanding. 19
OMTG: Towards One-to-Many Temporal Grounding Table 12. Prompt template for Scgg evaluation. Part 2: Scgg Evaluation Prompt You are predicting video timestamps from text captions ONLY (no video access). Query: “{query}” Captions {caption_list_str} Task Find ALL segments where “{query}” occurs based on the captions. Rules 1. Look for captions that DESCRIBE or IMPLY “{query}” 2. Use the caption’s timestamp as your prediction 3. If multiple captions match, list all of them 4. If caption text is vague but likely refers to the query, include it 5. Output format: one segment per line as “start – end” Example Output 10.5 - 15.0 32.0 - 37.0 Your predictions (list ALL matching segments): Table 13. Ablation on different temporal reward combinations on OMTG Bench. All results are reported as absolute improvements over the SFT baseline (Row 1). RtIoU + RC-Acc yields the best balance between localization and cardinality. Reward Functions
C-Acc tF1@30 tF1@50 tF1@70 tIoU EtF1
RtIoU RtIoU + RtF1 RtIoU + RC-Acc RtIoU + RtF1 + RC-Acc
+0.31 +1.88 +9.06 +5.94
+2.01 +3.65 +5.55 +2.94
+1.82 +3.15 +4.86 +3.62
+0.28 +1.71 +3.64 +3.29
+2.61 +0.74 +3.12 +2.96 +5.23 +7.91 +3.21 +5.50
D. Performances across Different Model Sizes In this section, we present a analysis of how model capacity affects performance on OMTG Bench. As illustrated in Tab. 15.
E. Implementation Details for OMTG Benchmarking In this section, we present the implementation details for evaluating existing MLLMs on our OMTG evaluation suite, yielding the results reported in Tab. 1. Proprietary Models. We evaluated the Gemini series (Gemini 2.5 Pro and Gemini 3 Pro) via their official Video Understanding API. Notably, the inputs for these models incorporated both visual and audio modalities to maximize information intake. For Seed-1.8, we accessed the model via the Volcano Engine API. We uploaded the complete video files and used the default sampling configuration, extracting frames at 2.0 FPS. Open-Source Models. For our OMTG-4B model and the Qwen series (including Qwen3-VL and Qwen2.5-VL), we employed the sglang engine as the inference backend to ensure efficiency. For other open-source baselines, we utilized the standard transformers library for inference. To ensure a fair comparison regarding visual information, we imposed consistent resolution constraints across all open-source models (including ours), setting fps=2, min_pixels=2048, and total_pixels=8388608. For UniTime, we follow it’s default adaptive frame scaling strategy setting. Prompts. To ensure reproducibility, we strictly standardized the prompts used for evaluation: 20
OMTG: Towards One-to-Many Temporal Grounding Table 14. Results on VideoMME (w/o sub, 128 frames). We compare our OMTG-4B variants against the backbone Qwen3-VL-4B to analyze the impact of our training strategies on general video understanding. Model
Overall Short Medium Long
Qwen3-VL-4B (Base)
66.7
77.6
65.8
56.8
Ablation on SFT Training Data OMTG-4B (SFT w/o CoT) OMTG-4B (SFT w/ CoT)
62.1 64.4
72.4 77.6
62.0 64.0
52.0 51.8
Ablation on Reward Functions OMTG-4B (RL w/o Cap. Reward) OMTG-4B (RL w/ Cap. Reward)
62.5 65.1
72.7 77.3
62.3 65.7
52.4 52.3
Table 15. Performances across Different Model Sizes Model Size C-Acc tF1@30 tF1@50 tF1@70 tIoU EtF1 OMTG-2B OMTG-4B
46.56 55.63
68.32 73.46
61.06 65.40
44.10 48.96
55.97 35.48 61.24 43.65
• All open-source models (including our OMTG-4B) and Seed-1.8 utilized the unified prompt detailed in the Tab 16. • The Gemini series utilized the specific prompt detailed in the Tab 17. Table 16. Prompt template for open-source models and Seed-1.8 evaluation. Open-Source Models and Seed-1.8 Evaluation Prompt Find the video segment that corresponds to the given textual query ’{query}’ and determine its start and end seconds. If there are multiple segments, please output the start and end time for each one separately.
F. In-the-Wild Generalization To further validate the generalization capability beyond the benchmark domain, we collect an additional out-of-domain (OOD) test set from recent videos on Bilibili and YouTube. Using the same data pipeline and human verification protocol as the main benchmark, we annotate 60 samples across 52 videos, covering diverse real-world content including travel vlogs, gaming, sports, news, and anime. These videos are entirely out-of-domain from any training source. The average duration is 422.87s (max 1419.93s). Despite the limited sample size, this set serves as a challenging OOD test for zero-shot evaluation. Table 18 reports the zero-shot performance. Our model demonstrates strong generalization to longer, truly in-the-wild videos, significantly outperforming baselines across all metrics.
G. Statistics Details of OMTG Benchmark In this section, we present more statistics of our OMTG Benchmark.
H. Failure Cases Study In this section, we present typical failure cases and provide analyses. On hard cases like Fig. 7 ("person moves the fridge door"), the model exhibits complete temporal misalignment when action semantics are ambiguous or visually subtle. The query "moves the fridge door" creates interpretive ambiguity—models struggle to distinguish between the transitional motion of opening/closing (discrete actions) and the sustained state of the door being open (static condition), often defaulting to coarse-grained segmentation (e.g., 06-24s encompassing the entire interaction) or fixating on visually salient but semantically irrelevant frames (e.g., the person walking toward the fridge rather than the hand manipulating the handle). This reveals a critical vulnerability: when action boundaries lack sharp visual 21
OMTG: Towards One-to-Many Temporal Grounding Table 17. Prompt template for Gemini series evaluation. Gemini Series Evaluation Prompt Find the video segment that corresponds to the given textual query ’{query}’ and determine its start and end seconds. Format your response as: ‘<time>start - end seconds</time>‘ Where: - start = starting second - end = ending second Example: ‘<time>40 - 49 seconds</time>‘. If there exists multiple segments, separate them with a comma, e.g., ‘<time>10 13 seconds</time>, <time>27 - 29 seconds</time>‘. Table 18. Zero-shot OOD evaluation on in-the-wild videos. Model
tIoU
C-Acc
EtF1
Gemini-2.5-Pro Gemini-3-Pro Qwen3-VL-4B
12.99 14.22 18.91
15.00 21.67 0.13
16.35 16.60 22.11
12.17 12.68 15.72
7.15 6.53 11.09
4.81 3.29 0.09
OMTG-4B (Ours)
40.51
35.00
50.39
44.21
30.55
22.10
≥6 100
15.0%
8.1%
41.6%
14.7%
5
Number of Videos
2
4
80
60
40
20
20.6%
0
3
(a) Distribution of #ground truth segments.
900+
-60
0
60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960
Video Duration (seconds)
(b) Distribution of video durations (in seconds).
Figure 6. OMTG Benchmark Statistics. Left: The distribution of the number of temporal segments in the ground truth. Right: The histogram of video durations in the benchmark.
distinctive features, the model prioritizes scene context over fine-grained motion semantics, leading to predictions that either dilute the precise temporal boundaries or completely drift away from the actual motion event.
I. Annotation Interface and Manual Check For OMTG Benchmark To construct a high-quality One-to-Many Temporal Grounding (OMTG) benchmark, we developed a custom web-based annotation tool designed to facilitate the precise labeling of disjoint temporal segments. This section details the interface design, the annotation workflow, and the strict quality criteria provided to the annotators. Annotation Tool Design: Our annotation interface is a lightweight, local web application based on Python. It is designed to handle the complexity of multi-event video retrieval, allowing annotators to mark multiple non-contiguous time segments for a single textual query. The interface, illustrated in Figure 8. Annotation Workflow: The annotation process is standardized to ensure consistency across different annotators. The workflow proceeds as follows: 1. Review and Labeling. For each video, the annotator reviews the pre-generated textual queries. They watch the video to identify all occurrences of the described action or event. • If the query accurately describes events in the video, the annotator marks the start and end times for every instance of that event. • If the pre-generated query is inaccurate, the annotator modifies it. 22
OMTG: Towards One-to-Many Temporal Grounding
Question: Find the video segment that corresponds to the given textual query Õperson moves the fridge door' and determine its start and end seconds. If there are multiple segments, please output the start and end time for each one separately.
Our Answer: <think>Let me carefully sort out the events in the video. <time>0.0 - 5.0 seconds</time>: A man in a kitchen, wearing a blue vest over a white long-sleeve shirt, dark pants, and blue slip-on shoes, stands near a countertop. He holds a white plastic bag with colorful vertical stripes and the word ÒlifestyleÓ printed on it. He looks around the kitchen, which features black countertops, wooden cabinets, and a water dispenser. A white bucket sits on the floor beneath the counter.
Ground Truth Answer: The given query happens in <time>9 - 12 seconds</time>, and <time>22 - 24 seconds</time>. #Segments: 2 Gemini 3 Pro Answer: The segment where the person interacts with and moves the fridge door corresponds to the timeframe when they open the refrigerator, place an item inside, and then close it.\n\n- Starts at **00:06** as the person reaches for the handle and begins to pull the door open.\n- Ends at **00:24** when the person pushes the door shut and steps away.\n\n<time>00:06 00:24 seconds</time>
The CoT bundles the entire interaction into a entire event. #Segments: 1
Gemini 2.5 Pro Answer: <time>6 - 24 seconds</time>
Similar to Gemini3, only output 1 segment. #Segments: 1 Seed 1.8 Answer: Got it, let's check the video. The person moves the fridge door starting around 3 seconds, when they reach for the door and open it, up to 11 seconds maybe? Wait no, looking at the timestamps: the action of opening the fridge door is from 3 to 11 seconds? Wait no, the first movement towards the door is at 4 seconds, then opening it. Wait the video shows the person walking to the fridge, opening the door. So the segment where the person moves the fridge door is 3 to 11 seconds? Wait no, let's check: at 5 seconds, they start opening the door, and the door is open by 8 seconds. Wait the correct timestamps are 3 to 11 seconds? Wait no, the user asked for when the person moves the fridge door. Let's see: the person reaches for the door at around 4 seconds, opens it, so the segment is 3 to 11 seconds? Wait no, looking at the video: the action of moving the fridge door (opening) is from 4 to 11 seconds. Wait the video: at 4, he turns, walks to the door, opens it. So the start is 3, end is 11 seconds.
<time>5.0 - 8.0 seconds</time>: The man turns and walks toward a refrigerator on the left side of the frame. He reaches out with his left hand and opens the refrigerator door, revealing shelves stocked with bottles, jars, and vegetables. He continues holding the bag in his right hand. <time>8.0 - 13.0 seconds</time>: Standing in front of the open refrigerator, the man bends slightly and reaches into the bottom shelf with his left hand, appearing to retrieve or adjust something. He keeps the bag in his right hand, occasionally shifting its position. The interior light of the refrigerator illuminates the contents. <time>13.0 - 18.0 seconds</time>: The man continues to interact with the refrigerator, bending further and reaching deeper into the lower compartment. He appears to be organizing or retrieving items, possibly placing the bag inside or adjusting its contents. His focus remains on the refrigerator interior. <time>18.0 - 22.0 seconds</time>: He pulls out a small item from the refrigerator, possibly a container or bottle, and examines it briefly. He then places it back or adjusts its position. He remains bent over, still holding the bag, and appears to be organizing the contents of the refrigerator. <time>22.0 - 25.0 seconds</time>: The man straightens up slightly and begins to close the refrigerator door. He uses his left hand to push the door shut while still holding the bag in his right hand. The refrigerator door closes with a soft click. <time>25.0 - 29.0 seconds</time>: After closing the refrigerator, the man turns away from it and walks back toward the center of the kitchen. He places the bag on the floor briefly, then picks it up again. He looks around the kitchen, appearing to assess his next move, before the video ends. Based on the step-by-step analysis, the event 'person moves the fridge door.' is clearly identified in 1 distinct segments: <time>22 24 seconds</time>. Therefore, I should output 1 time segments. </think> The given query happens in <time>22 - 24 seconds</time>."
After multiple attempts, Seed1.8 ultimately failed.
The CoT correctly identifies two distinct interactions but only output 1 after reasoning.
#Segments: 1
#Segments: 1
Figure 7. Failure case from video T42IZ.mp4
• If the video contains distinct events not covered by the list, the annotator adds a new query and labels its occurrences. 2. Submission. Annotations are automatically saved to a local json file. Once a batch is completed, this file is collected for quality verification. Annotation Quality Criteria: To ensure the benchmark rigorously evaluates a model’s ability to handle the OMTG task, we enforcedstrict guidelines. Annotators were instructed to adhere to the following four rules: 1. Completeness (No Missing Segments): The core requirement of OMTG is to find all instances. Annotators must ensure that every single occurrence of the query within the video is marked. Missing a segment is considered a critical error. 2. Temporal Tightness (Boundary Precision): Annotations must be as tight as possible. The start timestamp should mark the 23
OMTG: Towards One-to-Many Temporal Grounding
exact beginning of the action, and the end timestamp should mark its immediate conclusion. In cases of ambiguity (e.g., slow transitions), annotators should adopt a "conservative" approach, shrinking the window to include only the frames where the action is clearly visible, rather than including ambiguous transition frames. 3. Human-Perceived Clarity: Quality is prioritized over quantity. If a video is low-resolution, heavily occluded, or if the events are too ambiguous to define clearly (e.g., "someone might be smiling"), the video should be discarded using the Discard Video function. Ground truth is established solely based on clear human perception. 4. Preference for High Cardinality: Since existing datasets are dominated by single-segment events, our benchmark aims to fill the gap for multi-segment retrieval. Annotators are explicitly instructed to prioritize and retain queries that correspond to multiple time segments (e.g., "a person jumps" happening three times) over single-occurrence events.
Figure 8. Illustration of our annotation interface. The tool allows annotators to watch videos and label multiple disjoint time segments for a given query. Pre-generated Gemini queries are provided as hints but require manual verification and adjustment.
24