The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding Jiameng Zhang University of Zurich [email protected]

Srikanth Madikeri University of Zurich [email protected]

Abstract

systems must identify both the temporal extent and spatial location of query-relevant events at scale. A critical deployment reality has received insufficient attention in prior work. Industrial video systems are constrained to sparse inputs of 8 to 16 frames per video to satisfy latency and throughput requirements, yet state-of-the-art multimodal large language models (MLLMs) are pretrained on dense sequences of hundreds of frames. The resulting mismatch between training and deployment conditions causes severe performance degradation that makes existing models impractical for real-world use without targeted adaptation. Prior work on video temporal grounding has largely focused on designing new model architectures and visual aggregation modules (Wasim et al., 2024; Liu et al., 2024b; Xiong et al., 2024; Gu et al., 2024), answering the question of how to build a better grounding model from scratch. We instead address a complementary and more practical question: given a strong pretrained MLLM, what training strategy is most effective for sparseframe spatial-temporal grounding? This question directly reflects the situation faced by practitioners, who must adapt existing models to deployment constraints rather than design new architectures. We investigate this question through systematic experiments on Qwen3-VL models and CharadesSTA (Gao et al., 2017; Sigurdsson et al., 2016). Since Charades-STA provides only temporal annotations, we construct a spatial annotation extension using GroundingDINO (Liu et al., 2023), yielding 3,951 bounding box annotations with 85% acceptance rate under manual inspection. These annotations provide pseudo-labeled spatial supervision rather than exhaustively verified gold labels. Our experiments compare vision encoder versus language model adaptation, joint versus single-task training, and multiple frame sampling strategies under realistic sparse-input constraints. Our findings challenge common assumptions

arXiv:2607.24570v1 [cs.CV] 27 Jul 2026

Large-scale video platforms process millions of uploads hourly, requiring moderation systems that can localize when and where policy violations occur within each video. Processing every frame is infeasible at scale, so systems are constrained to sparse inputs of 8 to 16 frames per video. (Miech et al., 2019; Tang et al., 2024) Yet state-of-the-art multimodal large language models (MLLMs) are pretrained on dense sequences of hundreds of frames, creating a fundamental mismatch between training and deployment conditions. This mismatch causes severe performance collapse: the Qwen3-VL 8B model drops from 56.0% to 22.3% temporal mIoU when frames are reduced to 16, a 60.2% relative degradation. We present a systematic empirical study of training strategies to close this gap for spatialtemporal video grounding. Our results suggest that visual feature extraction is the dominant bottleneck under sparse-frame inputs. Adapting only the final three ViT layers, 4% of total parameters, achieves 68.8% temporal mIoU and surpasses a zero-shot 8B model using dense inputs by 12.8 points. Language model finetuning, by contrast, offers negligible or negative returns. A boundary-aware sampling strategy, H YBRID 16, further improves temporal mIoU by 26 points over uniform sampling when temporal boundaries are available. We conclude that for sparse-frame video grounding, training strategy dominates model scale: a fine-tuned 2B model consistently outperforms a zero-shot 8B model, with or without dense frame access.

Introduction

Spatial-temporal video grounding requires models to localize when and where actions occur in videos given natural language queries (Gao et al., 2017; Hendricks et al., 2017). This capability is central to industrial applications such as content moderation, video retrieval, and automated surveillance, where 1

Figure 1: Overview of the train-dense/infer-sparse mismatch and our proposed solutions. Left: the Qwen3-VL 8B model drops from 56.0% to 22.3% temporal mIoU when frames are reduced from hundreds to 16. Center: our four key findings. Right: a fine-tuned 2B model outperforms a zero-shot 8B model on both temporal and spatial grounding.

about MLLM adaptation. Fine-tuning language model attention layers via DoRA (Liu et al., 2024a) degrades performance below zero-shot baselines, while adapting only the final three ViT layers achieves 68.8% temporal mIoU and surpasses a zero-shot 8B model using dense inputs by 12.8 points. This suggests that the pretrained language model already provides substantial temporal-query understanding, while sparse visual feature extraction is the main bottleneck for sparse-frame grounding. A boundary-aware sampling strategy that reallocates frames from existing temporal annotations to action segments further improves temporal mIoU by 26 points over uniform sampling in annotationaware settings.

bust Performance Scores for comprehensive spatialtemporal assessment.

Related Work

Video Temporal Grounding. Early approaches to temporal grounding relied on cross-modal attention mechanisms to align natural language queries with video features (Gao et al., 2017; Hendricks et al., 2017). Subsequent work focused on designing specialized architectures, including proposalbased methods that generate candidate temporal segments and rank them by query relevance, and proposal-free methods that directly regress temporal boundaries. More recent work has explored integrating temporal grounding with spatial localization. Wasim et al. (2024) extend GroundingDINO to spatio-temporal video grounding in an openvocabulary setting, while Xiong et al. (2024) propose structured video-language modeling with joint temporal grouping and spatial grounding. Liu et al. (2024b) address efficient image-to-video transfer learning for temporal grounding. In contrast to these architecture-focused approaches, we investigate training strategy choices for adapting a pretrained MLLM to sparse-frame grounding, a deployment constraint largely overlooked in prior work.

These findings yield four contributions. We identify the training and deployment mismatch as a critical but underaddressed challenge for industrial video grounding and provide the first systematic analysis of training strategies to close this gap. We show that adapting only 4% of model parameters on a single GPU suffices to outperform models three times larger under zero-shot dense inference. We propose H YBRID 16, a boundaryaware sampling strategy for pre-annotated archives and re-inspection pipelines. We contribute a spatial annotation extension for Charades-STA and a paired evaluation framework using Peak and Ro2

Sparse Frame Processing for Video Grounding. Industrial-scale video platforms process millions of uploads requiring real-time analysis, making dense frame processing computationally prohibitive (Miech et al., 2019). Prior work has examined how frame sampling strategies influence temporal grounding performance (Zhu et al., 2023), and recent training-free approaches demonstrate that large pretrained models can localize moments without task-specific supervision (Zheng et al., 2024a). However, both lines of work assume access to dense frame sequences at inference time. The performance collapse that arises when models pretrained on dense inputs are deployed with sparse frames remains unaddressed. We provide the first systematic study of training strategies specifically designed to close this train-deploy gap.

guage model attention layers degrades performance below zero-shot baselines, while selective vision encoder fine-tuning achieves substantially stronger results, pointing to visual feature extraction rather than language reasoning as the primary bottleneck.

Dataset and Annotation

3.1

Charades-STA

Charades-STA (Gao et al., 2017) is a standard benchmark for video temporal grounding, containing 12,408 training and 3,720 test query-segment pairs drawn from 9,849 indoor activity videos. Each query consists of a natural language description paired with temporal boundary annotations marking the relevant action segment. Video durations average approximately 30 seconds, providing sufficient temporal context while remaining computationally tractable for sparse-frame processing. Charades-STA is particularly suitable for this study because its controlled indoor environment and dense temporal annotations allow focused evaluation of spatial-temporal grounding under realistic computational constraints.

Multimodal Large Language Models for Video. MLLMs have emerged as a unified framework for vision-language tasks, combining pretrained vision encoders with large language models through lightweight connectors (Alayrac et al., 2022; Li et al., 2023). Video-LLaMA (Zhang et al., 2023; Maaz et al., 2024) and Video-ChatGPT (Maaz et al., 2024) extend this paradigm to video inputs through instruction tuning on visual and auditory content. Recent models such as Qwen3-VL (Qwen Team, 2025) demonstrate strong performance on video understanding benchmarks through multi-level visual feature fusion and extended context windows. Recent work specifically targeting temporal grounding with video LLMs, such as VTimeLLM (Huang et al., 2024), further demonstrates the importance of boundary-aware training for precise temporal localization. However, these models are pretrained on dense frame sequences, creating a fundamental mismatch with industrial deployment constraints that limit inputs to 8–16 frames per video. We directly address this train-deploy mismatch through targeted adaptation strategies.

3.2

Spatial Annotation Pipeline

Since Charades-STA provides only temporal annotations, we extend the dataset with spatial bounding box annotations to enable joint spatial-temporal training and evaluation. We use GroundingDINO (Liu et al., 2023), an open-vocabulary object detection model, to automatically generate spatial annotations from the existing natural language queries. The pipeline processes 131,000 frames extracted at 1 fps from Charades-STA videos, using the natural language query associated with each video as the detection prompt. For example, given the query “person opens refrigerator,” GroundingDINO detects “person” and “refrigerator” in relevant frames and outputs bounding box coordinates with confidence scores. Detections with confidence below 0.3 are filtered out. To prevent redundant training examples, we retain one bounding box per queryvideo pair, selected by detection confidence and temporal proximity to the annotated action segment. Overrepresented object categories are subsampled to prevent class imbalance. This process yields 3,951 spatial annotations covering diverse indoor object categories. To assess annotation quality, we manually inspect a random sample of 100 annotations. An annotation is accepted if the predicted bounding box

Parameter-Efficient Fine-Tuning. PEFT methods reduce the cost of adapting large pretrained models by updating only a small subset of parameters. LoRA (Hu et al., 2021) decomposes weight updates into low-rank matrices, while DoRA (Liu et al., 2024a) further separates magnitude and direction components for more expressive adaptation. These methods have been widely applied to language model adaptation, but their effectiveness for video grounding remains underexplored. Our experiments demonstrate that applying DoRA to lan3

reasonably localizes the object referred to in the query. This inspection yields an 85% acceptance rate, indicating that the pipeline produces useful spatial supervision for training and diagnostic evaluation. Because the full spatial split is not exhaustively human verified, we interpret spatial metrics as evaluation against high-confidence pseudo labels rather than fully gold spatial annotations. 3.3

The 3,951 spatial annotations are combined with temporal samples to form five training configurations, as detailed in Section 4. Spatial instances use single frames at 480×360 resolution with bounding box annotations in [xmin , ymin , xmax , ymax ] format. Temporal instances use 16 frames with ground truth start and end timestamps. Two non-overlapping test sets of 1,000 instances each are constructed from the Charades-STA test split using Even16 and Hybrid16 sampling respectively, ensuring no overlap with training data.

Experimental Setup

4.1

Model and Training Strategy

Trainable

Text Transformer Vision Encoder (Blocks 0–20) Vision Encoder (Blocks 21–23) Merger Modules Embedding Layers

1.5B 180M 45M 20M 5M

0 0 45M 20M 5M

Total

1.75B

70M (4%)

ciency. FPS1 extracts one frame per second, yielding approximately 30 frames per video. This provides dense temporal coverage but results in variable sequence lengths, and is used only for spatial annotation instances. Even16 selects 16 frames uniformly across the entire video duration, ensuring fixed input size and consistent memory usage. This strategy reflects common practice in video understanding systems but may undersample action regions when the target event occupies a small portion of the video timeline.

We use Qwen3-VL 2B (Qwen Team, 2025) as our foundation model. The architecture consists of a Vision Transformer encoder (Dosovitskiy et al., 2021) (24 layers, approximately 300M parameters), a text Transformer decoder (28 layers, approximately 1.5B parameters), and a DeepStack multimodal connector that fuses features from multiple encoder depths. The model has been pretrained on large-scale vision-language data including video understanding tasks. Our training strategy freezes the text Transformer entirely to preserve pretrained language understanding, and also freezes early vision encoder layers (blocks 0–20) to retain general visual features such as edges, textures, and local patterns. We make trainable only the final three ViT blocks (21– 23), the Merger and DeepStack Merger projection modules, and the patch embedding and positional encoding layers. This yields approximately 70M trainable parameters, or 4% of the total 1.75B parameters, enabling training on a single NVIDIA H100 (80GB GPU). Table 1 summarizes the parameter distribution. 4.2

Total

Table 1: Parameter distribution across model components.

Training Data

Component

Hybrid16 addresses this limitation by allocating 8 frames to the annotated action segment and 8 frames uniformly across the full video. This design concentrates coverage at temporal boundaries where state transitions are most informative, while retaining global context, at zero additional annotation cost beyond the temporal labels already present in Charades-STA. Hybrid16 sampling requires temporal annotations at inference time and is therefore suited to pre-annotated video archives and re-inspection pipelines; for deployment scenarios where no temporal annotations are available, Even16 provides a fully annotation-free alternative with stable performance. Table 2 illustrates the difference between strategies for a representative video.

Frame Sampling Strategies

Strategy

In Segment

Outside

Total

FPS1 Even16 Hybrid16

8 3 10

23 13 6

31 16 16

Table 2: Frame distribution for a 30-second video with action segment at 11.2–16.7s.

We compare three frame sampling strategies that balance temporal coverage and computational effi4

Figure 2: Comparison of H YBRID 16 and Even16 sampling strategies for a 30-second video with action segment at 11.2–16.7s. H YBRID 16 concentrates 8 frames within the action segment (dark green) while distributing 8 frames uniformly (mint green). Even16 distributes all 16 frames uniformly across the video.

4.3

Training Configurations

grounding is evaluated using spatial mIoU and [email protected]. To assess joint spatial-temporal capability, we propose two complementary metrics. The Peak Performance Score averages the best temporal mIoU across both test sets with spatial mIoU, measuring achievable performance under favorable conditions. The Robust Performance Score averages temporal mIoU across both test distributions with spatial mIoU, quantifying stability under distribution shift. These paired metrics address the limitation of evaluating spatial and temporal grounding in isolation, which fails to capture trade-offs between the two dimensions.

We design five training configurations to evaluate the impact of data composition and sampling strategy. Each configuration is trained independently on Qwen3-VL 2B and evaluated on both Even16 and Hybrid16 test sets to assess generalization across frame distributions. Table 3 summarizes the configurations. Config

Size

Sampling

Ann.

BBx-only Even-only Hybrid-only Evenbbx Hybridbbx

3,951 7,000 7,000 6,500 6,500

FPS1 Even16 Hybrid16 FPS1 + Even16 FPS1 + Hybrid16

Spatial Temporal Temporal Both Both

Table 3: Training configurations. Joint configurations (Evenbbx, Hybridbbx) combine 2,500 spatial and 4,000 temporal instances.

All models are trained for 3 epochs using AdamW with learning rate 5 × 10−6 , BF16 mixed precision, gradient checkpointing, and cosine annealing with 10% linear warmup. Images are resized to fit within 256×256 to 512×512 pixels while preserving aspect ratio. Training was conducted on a single NVIDIA H100 GPU using the LLaMA-Factory framework (Zheng et al., 2024b). 4.4

Config

Spatial

Peak

Robust

Hybridbbx Evenbbx Hybrid-only Even-only BBx-only

0.60 0.55 0.00 0.01 0.60

0.64 0.51 0.37 0.25 0.33

0.55 0.50 0.24 0.34 0.33

Table 4: Joint spatial-temporal evaluation using Peak and Robust Performance Scores. Peak = (best temporal + spatial mIoU) / 2. Robust = (average temporal + spatial mIoU) / 2.

Evaluation Metrics

Temporal grounding performance is measured using mean Intersection over Union (mIoU) and Recall at IoU thresholds R@{0.3, 0.5, 0.7}. Spatial 5

Results

5.1

Vision Encoder vs. Language Model Adaptation

which sparse frames contain action-relevant information. Fine-tuning reverses this relationship entirely. As shown in Table 6, the fine-tuned 2B model achieves 68.8% temporal mIoU using 16 frames, surpassing the zero-shot 8B model under dense processing by 12.8 points and under sparse processing by 46.5 points. At stricter evaluation thresholds, the advantage becomes more pronounced: the finetuned model achieves 64.0% [email protected] on Hybrid16 compared to 4.0% for zero-shot 8B. These results demonstrate that targeted adaptation can outweigh model scale under sparse-frame constraints.

We first investigate which model component is most critical to adapt for sparse-frame grounding. As a baseline, we apply DoRA (Liu et al., 2024a) with rank 64 to all language model attention layers while keeping the vision encoder frozen. This language-only adaptation degrades temporal mIoU to 15–16%, below the zero-shot baseline of 21– 22%, demonstrating that modifying language components disrupts pretrained temporal query understanding without compensating benefits. The pretrained language model already handles temporal queries adequately; further adaptation introduces noise rather than improvement. Adapting the vision encoder produces the opposite effect. Fine-tuning only the final three ViT layers achieves 68.8% temporal mIoU on matched test distributions and over 55% spatial mIoU, more than three times higher than language-only adaptation. Table 5 summarizes the comparison. Strategy

Params

Zero-shot 8B DoRA (LM only) Vision-focused

None LM attention ViT 21–23

Even16

Hybrid16

0.21 0.15 0.47

0.22 0.16 0.69

Frames

mIoU

[email protected]

Zero-shot 8B Zero-shot 8B Zero-shot 2B Fine-tuned 2B†

All frames 16 (Hyb16) 16 (Hyb16) 16 (Hyb16)

0.56 0.22 0.09 0.69

– 0.13 0.03 0.74

– 0.04 0.01 0.64

Table 6: Temporal grounding performance comparing dense zero-shot inference against sparse fine-tuned models. † Hybridbbx configuration.

5.3

Frame Sampling Strategy

Sampling strategy creates strong dependencies between training and test distributions. Models perform best when test conditions match training conditions. Hybridbbx achieves 68.8% mIoU on Hybrid16 test but drops to 33.6% on Even16 test, a 51% relative decline. Evenbbx shows the opposite pattern, achieving 47.4% on Even16 and 42.8% on Hybrid16, a more stable 10% decline under distribution shift. This asymmetry reflects different learned representations. Hybrid16 training concentrates frames near action boundaries, teaching the model to recognize fine-grained temporal transitions. These specialized features prove highly effective under aligned test conditions but transfer poorly when frame distributions change. Even16 training exposes the model to frames from all temporal positions, developing more general representations that maintain consistent performance across distributions at the cost of peak capability. For spatial grounding, sampling strategy has a much smaller effect. Hybridbbx achieves 59.7% spatial mIoU compared to 55.3% for Evenbbx, a 4.4 point difference, versus the 26 point gap observed for temporal grounding on matched distributions. Spatial grounding depends on object localization within individual frames rather than temporal

Table 5: Temporal mIoU for language model adaptation vs. vision encoder adaptation. Vision-focused training uses the Evenbbx and Hybridbbx configurations on their respective matched test sets.

This asymmetry is further supported by zeroshot scaling results. Expanding the language model from 28 to 36 layers (2B to 4B, same vision encoder) yields only 5.1 points improvement on Hybrid16. Expanding the vision encoder from 24 to 27 layers (4B to 8B, same LM depth) yields 8.0 points despite adding fewer layers. Vision encoder capacity contributes more to grounding performance than language model capacity, identifying visual feature extraction as the primary bottleneck. 5.2

Model

Sparse Fine-tuning vs. Dense Zero-shot

Zero-shot models rely on dense frame processing to achieve competitive performance. The Qwen3VL 8B model achieves 56.0% temporal mIoU using all available frames but drops to 22.3% when limited to 16 frames, a 60% relative collapse. Without task-specific training, models cannot identify 6

frame ordering, explaining its relative insensitivity to sampling strategy. 5.4

edge of temporal concepts and action semantics during pretraining. They already understand phrases such as “person starts to drink” or “someone picks up a phone” and can reason about temporal relationships between events. Adapting these representations with DoRA does not address the actual challenge, which is connecting linguistic concepts to visual observations in sparse frame sequences. Modifying language components instead disrupts existing capabilities, explaining the performance degradation below zero-shot baselines.

Joint vs. Specialized Training

Specialized models achieve the highest scores on their target tasks. Hybrid-only reaches 72.7% temporal mIoU on Hybrid16, outperforming Hybridbbx by 3.9 points. BBx-only achieves 60.0% spatial mIoU, exceeding Hybridbbx by 0.3 points. However, specialized models fail completely on the complementary task: Hybrid-only achieves 0.34% spatial mIoU and BBx-only achieves 6.0% temporal mIoU. To assess overall joint capability, we apply the Peak and Robust Performance Scores introduced in Section 4. As shown in Table 4, Hybridbbx achieves the highest scores on both metrics (0.64 peak, 0.55 robust), demonstrating that joint training provides balanced capability across both tasks. The small gap between peak and robust scores (0.09) indicates consistent performance across test conditions. Specialized configurations score poorly on both metrics despite strong single-task performance, as near-zero performance on one dimension drags down the combined score. The performance trade-off from joint training is modest: 5.4% relative loss for temporal grounding and 0.5% for spatial grounding, a reasonable cost for maintaining full dual-task capability within a single model.

Hybrid sampling and action boundaries. Human actions have natural boundaries with concentrated visual transitions at their start and end points. Hybrid16 exploits this structure by allocating half its frame budget to the annotated action segment, providing dense coverage of precisely the frames that carry the most discriminative temporal information. The large performance gap between Hybrid16 and Even16 on matched distributions (68.8% vs. 42.8%) demonstrates the value of boundary-aware sampling, while the robustness gap (51% vs. 10% decline under distribution shift) reveals the trade-off between specialization and generalization. For deployment scenarios where temporal boundaries are already available or can be supplied by an upstream system, Hybrid16 is the stronger annotation-aware option. For scenarios with unknown frame distributions, Even16 provides more stable performance.

Discussion

Practical implications. These findings provide concrete guidance for practitioners adapting MLLMs to video grounding tasks. Vision encoder fine-tuning should be considered before language model adaptation, as our experiments indicate that it addresses the dominant bottleneck while preserving pretrained linguistic capabilities. Hybrid frame sampling is applicable to datasets and workflows with temporal annotations, such as curated archives or re-inspection pipelines. Joint spatial-temporal training is recommended over specialized models for most deployment scenarios, as the modest 3–4 point performance trade-off is outweighed by the operational benefit of maintaining a single model for both tasks.

Why vision encoder adaptation succeeds. Temporal grounding requires recognizing subtle visual cues that indicate action boundaries, state changes, and temporal transitions. General-purpose vision encoders trained on image-text alignment do not capture these patterns well. The final ViT layers encode high-level semantic features that are most amenable to task-specific adaptation, while early layers capture low-level visual primitives that transfer well across tasks and benefit from being frozen. Training only 4% of parameters is sufficient because the bottleneck is concentrated in these upper layers, not distributed across the entire model. This finding aligns with concurrent observations in image understanding (Bi et al., 2024; Chen et al., 2024), suggesting it reflects a structural property of MLLM architectures rather than a dataset-specific artifact.

Implications for the research community. Much prior work on video temporal grounding emphasizes cross-modal interaction mechanisms, attention-based fusion modules, and temporal reasoning architectures (Wasim et al., 2024; Xiong

Why language model adaptation fails. Modern large language models acquire extensive knowl7

et al., 2024). These approaches address how to better combine visual and linguistic information, implicitly assuming that the visual representations themselves are adequate. Our results challenge this assumption: in our setting, language adaptation provides negligible gains, while vision encoder adaptation yields over 200% relative improvement. This suggests that visual representation learning for sparse-frame grounding deserves greater attention as a research direction in its own right, complementing work on cross-modal fusion. Targeted adaptation of the vision encoder, rather than scaling model size or refining fusion mechanisms, may offer a more efficient path to deployment-ready spatial-temporal grounding.

human validation would further strengthen the reliability of spatial grounding evaluation. Third, the choice to fine-tune the final three ViT layers was based on preliminary experiments rather than systematic ablation; other layer selection strategies may yield further improvements. Fourth, the comparison between joint and specialized training is confounded by differences in training set size, as joint configurations use 6,500 instances while specialized models use 3,951–7,000; future work should control for this variable to isolate the effect of task composition from data quantity. We leave cross-dataset validation, layer selection ablation, and controlled joint training experiments to future work.

Conclusion

We have presented a systematic empirical study of training strategies for spatial-temporal video grounding under the sparse-frame constraints of industrial deployment. Our evidence indicates that sparse visual feature extraction, rather than language-model adaptation, is the dominant bottleneck in this setting. Adapting only the final three ViT layers surpasses language model fine-tuning by a factor of three and outperforms a zero-shot model four times larger. A boundary-aware frame sampling strategy further improves temporal grounding by 26 points under matched annotation-aware test conditions, and joint spatial-temporal training achieves balanced capability across both tasks with minimal trade-off. Together, these results show that targeted adaptation can be more effective than model scaling for sparse-frame video grounding, offering a practical path to high-performance deployment without requiring large models or dense frame processing.

Ethical Considerations

This work studies video grounding methods that may be used in content moderation, archive search, and monitoring workflows. Such systems can affect privacy and fairness when applied to people in videos, especially if model predictions are used for enforcement or surveillance without human review. We therefore view the proposed methods as decision-support tools rather than standalone systems for high-stakes decisions. The spatial annotations used in this study are generated with an automatic detector and are not exhaustively human verified, so downstream uses should account for annotation noise and potential detector bias.

References Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Ian Barr, Yossi Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, and 8 others. 2022. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198.

Limitations

Our study has four primary limitations. First, all experiments are conducted on Charades-STA, which features short indoor activity videos with controlled environments; whether the findings generalize to long-form videos, egocentric recordings, or outdoor scenes remains an open question. Second, spatial annotations are automatically generated by GroundingDINO and validated on a sample of 100 annotations (85% acceptance rate). As a result, spatial results should be interpreted as evaluation against high-confidence pseudo labels rather than fully human-verified gold annotations; larger-scale

Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2024. LLaVA steering: Visual instruction tuning with 500x fewer parameters through modality linear representationsteering. arXiv preprint arXiv:2412.17339. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2024. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of

Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 12585–12602.

the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24185–24198. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16×16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR).

Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2630– 2640.

Jiyang Gao, Chen Sun, Jia Yang, and Ram Nevatia. 2017. TALL: Temporal activity localization via language query. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 5267–5275.

Qwen Team. 2025. Qwen3-VL: Large language model for vision-language understanding with DeepStack connector. Technical report, Alibaba Group.

Xianyi Gu, Houwen Fan, Yutong Huang, Tianyu Luo, and Linghao Zhang. 2024. Context-guided spatiotemporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18330–18339.

Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision (ECCV), pages 510–526.

Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 5803–5812.

Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Zitao Lin, Rongyi Zheng, Yiyang Zhu, Chao Xu, Shengkui Huang, Lijun Ding, Chenglin Xu, Guanhong Luo, Yonghe Dong, and Yiwei Bao. 2024. Video understanding with large language models: A survey. arXiv preprint arXiv:2312.17432.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Syed Tahoon Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. 2024. VideoGrounding-DINO: Towards open-vocabulary spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18909–18918.

Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. 2024. VTimeLLM: Empower LLM to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14271–14280. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597.

Yuanhao Xiong, Long Zhao, Boqing Gong, Ming-Hsuan Yang, Florian Schroff, Ting Liu, Cho-Jui Hsieh, and Liangzhe Yuan. 2024. Structured video-language modeling with temporal grouping and spatial grounding. In International Conference on Learning Representations (ICLR).

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. DoRA: Weightdecomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML), pages 32100–32121.

Hang Zhang, Xin Li, and Lidong Bing. 2023. VideoLLaMA: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP), pages 543–553.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. 2023. Grounding DINO: Marrying DINO with grounded pretraining for open-set object detection. arXiv preprint arXiv:2303.05499.

Minghang Zheng, Xinhao Cai, Qingchao Chen, Yuxin Peng, and Yang Liu. 2024a. Training-free video temporal grounding using large-scale pre-trained models. In Proceedings of the European Conference on Computer Vision (ECCV). Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024b. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL).

Yifan Liu, Jing He, Wei Li, Jaehong Kim, Dennis Wei, Hanspeter Pfister, and C. Chen. 2024b. R2 -Tuning: Efficient image-to-video transfer learning for video temporal grounding. In European Conference on Computer Vision (ECCV).

Jingkang Zhu, Dong Liu, Peng Zhou, Xin Di, Yiming Cheng, and 1 others. 2023. Rethinking the video sampling and reasoning strategies for temporal sentence grounding. arXiv preprint arXiv:2301.00514.

Prompt Templates

All prompts follow the Qwen3-VL conversation format with interleaved image and text tokens. A.1

Temporal Grounding Prompt

The temporal grounding prompt provides the model with a list of frame timestamps and the corresponding 16 video frames, followed by a natural language action query. The model is instructed to output only the start and end time in seconds. The timestamps reflect the sampling strategy used during training or evaluation: Hybrid16 timestamps concentrate near action boundaries, while Even16 timestamps are distributed uniformly across the video duration (see Section 4 for details).

Figure 3: Training loss curves for all five configurations. Hybrid-only training was stopped early at epoch 2.28 following loss plateau.

tions and test distributions.

User: Timestamps: [t_1, t_2, ..., t_16]. <image> <image> ... <image> (16 frames) Locate the following action: {query}. Only output the start and end time in seconds. Assistant: {start_time}, {end_time}

Table 9 summarizes architectural differences across Qwen3-VL model sizes, and Table 10 presents zero-shot performance scaling results.

A.2

Spatial Grounding Prompt

The spatial grounding prompt provides the model with a single video frame and instructs it to output bounding box coordinates for the queried objects. Coordinates are expressed in pixel space as [xmin , ymin , xmax , ymax ].

Implementation Details

All models are trained using the LLaMA-Factory framework on a single NVIDIA H100 GPU (80GB) with PyTorch 2.9.1 and Hugging Face Transformers. Table 12 provides complete hyperparameter specifications.

Training Loss Curves

Figure 3 presents training loss curves for all five configurations. All configurations converge smoothly within three epochs. BBx-only reaches a lower final loss than temporal configurations, reflecting the simpler single-frame spatial grounding objective. Hybrid-only was stopped early at epoch 2.28 following loss plateau. Dashed vertical lines indicate epoch boundaries.

Spatial Annotation Details

Table 11 shows the top object categories in the generated spatial annotations before subsampling. Door and person are heavily overrepresented, reflecting the indoor household activity focus of Charades-STA. Subsampling to 3,951 annotations preserves category diversity while preventing class imbalance during training.

User: <image> Provide bounding box coordinates for the following objects: {query}. Only output coordinates as: [x_min, y_min, x_max, y_max]. Assistant: {object} <bbox> x_min y_min x_max y_max </bbox>

Model Scaling Analysis

Spatial-Temporal Coupling Analysis

Table 13 illustrates an interaction between spatial and temporal grounding on video 50E06, which contains multiple queries involving small objects. When spatial localization is accurate (eats food, IoUs =0.84), temporal grounding is also strong (IoUt =1.00). When spatial accuracy degrades (drinks from cup, IoUs =0.45), temporal performance also suffers (IoUt =0.59), suggesting the two tasks interact through shared visual representations.

Complete Experimental Results

Tables 7 and 8 present complete temporal and spatial grounding results across all training configura11

Configuration

Test Set

mIoU

[email protected]

Zero-shot Dense (All Frames) Qwen3-VL-2B All frames Qwen3-VL-4B All frames Qwen3-VL-8B All frames

0.55 0.56 0.56

– – –

Zero-shot Sparse (16 Frames) Qwen3-VL-2B Even16 Qwen3-VL-2B Hybrid16 Qwen3-VL-4B Even16 Qwen3-VL-4B Hybrid16 Qwen3-VL-8B Even16 Qwen3-VL-8B Hybrid16

0.13 0.09 0.15 0.14 0.21 0.22

0.15 0.13 0.21 0.20 0.33 0.36

0.04 0.03 0.06 0.07 0.12 0.13

0.01 0.01 0.02 0.03 0.05 0.04

Fine-tuned 2B (16 Frames) BBx-only Even16 BBx-only Hybrid16 Even-only Even16 Even-only Hybrid16 Hybrid-only Even16 Hybrid-only Hybrid16 Evenbbx Even16 Evenbbx Hybrid16 Hybridbbx Even16 Hybridbbx Hybrid16

0.06 0.06 0.51 0.49 0.24 0.73 0.47 0.43 0.34 0.69

0.08 0.09 0.72 0.70 0.38 0.81 0.70 0.61 0.53 0.78

0.03 0.05 0.57 0.54 0.19 0.77 0.52 0.46 0.33 0.74

0.01 0.02 0.36 0.31 0.08 0.68 0.30 0.27 0.13 0.64

Table 7: Complete temporal grounding results across all configurations and test distributions.

Configuration

mIoU

[email protected]

Zero-shot Baselines Qwen3-VL-2B 0.03 Qwen3-VL-4B 0.03 Qwen3-VL-8B 0.03

0.00 0.00 0.00

Fine-tuned 2B BBx-only Even-only Hybrid-only Evenbbx Hybridbbx

0.67 0.00 0.00 0.61 0.67

0.52 0.00 0.00 0.44 0.47

0.60 0.01 0.00 0.55 0.60

Model

[email protected]

Even16

Hybrid16

Spatial

Qwen3-VL-2B Qwen3-VL-4B Qwen3-VL-8B

0.13 0.15 0.21

0.09 0.14 0.22

0.03 0.03 0.03

2B→4B gain 4B→8B gain

+0.02 +0.07

+0.05 +0.08

0.00 +0.01

Table 10: Zero-shot performance scaling across model sizes. The 2B→4B transition expands only the language model (28→36 layers); the 4B→8B transition also expands the vision encoder (24→27 layers), yielding larger gains despite fewer added layers.

Table 8: Complete spatial grounding results.

Model Qwen3-VL-2B Qwen3-VL-4B Qwen3-VL-8B

ViT Layers

LM Layers

Params

24 24 27

28 36 36

∼2.5B ∼4.5B ∼8.5B

Table 9: Qwen3-VL model architecture specifications. The 2B and 4B models share identical vision encoders; only the language model depth differs.