Small Vision-Language Models are Smart Compressors for Long Video Understanding Junjie Fei1,2,∗,† , Jun Chen1 , Zechun Liu1 , Yunyang Xiong1 , Chong Zhou1 , Wei Wen1 , Junlin Han1 , Mingchen Zhuge1,2 , Saksham Suri1 , Qi Qian1 , Shuming Liu1,2 , Lemeng Wu1 , Raghuraman Krishnamoorthi1 , Vikas Chandra1,† , Mohamed Elhoseiny2,† , Chenchen Zhu1,† 1
arXiv:2604.08120v1 [cs.CV] 9 Apr 2026
∗
Meta AI, 2 King Abdullah University of Science and Technology (KAUST) Work done at Meta, † Project lead
Adapting Multimodal Large Language Models (MLLMs) for hour-long video understanding is severely bottlenecked by context window limits. Dense visual streams quickly saturate input token budgets and exacerbate the lost-in-the-middle phenomenon. Existing efficiency heuristics, such as sparse sampling or query-agnostic uniform pooling, blindly sacrifice fidelity. They frequently discard transient decisive moments, blur fine-grained evidence, and waste representational bandwidth on irrelevant backgrounds. In this paper, we propose Tempo, an efficient, query-aware framework that compresses long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) to act as a local temporal compressor. It casts visual token reduction as an early cross-modal distillation process, generating compact, intent-aligned video representations in a single forward pass. To enforce strict inference budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM’s inherent zero-shot relevance prior and empirical semantic front-loading, ATA acts as a training-free, O(1) dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments demonstrate that our compact 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5–16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual token budget, outperforming proprietary baselines such as GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames pushes performance to 53.7. Crucially, empirical profiling reveals that Tempo frequently compresses hour-long videos to token counts substantially below theoretical computational limits, proving that true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows. Date: April 10, 2026 Correspondence: [email protected], [email protected] Code: https://github.com/FeiElysia/Tempo Project & Demo: https://FeiElysia.github.io/tempo-page/
1
Introduction
The advancement of Multimodal Large Language Models (MLLMs) has significantly transformed visual understanding, empowering systems to perform complex semantic analysis over images and short video clips Liu et al. (2023, 2024a); Zhu et al. (2023); Li et al. (2024a); Bai et al. (2025); Li et al. (2025); Zhang et al. (2024b, 2023). However, scaling these capabilities to hour-long videos remains challenging. The core difficulty lies in the structural mismatch between the massive, continuous visual stream of long videos and the rigidly bounded context windows of downstream LLMs. As temporal duration expands, raw visual tokens quickly overwhelm the input capacity, severely diluting attention mechanisms and causing models to fail at retrieving sparse evidence buried within extensive contexts Liu et al. (2024c). To fit long video understanding into limited contexts, existing methods typically force one of two compromises. A common approach is sparse frame sampling Xu et al. (2024a); Li et al. (2025); Lin et al. (2024a), which reduces compute but inevitably risks skipping the transient yet decisive moments required to answer a specific query. Alternatively, methods retain more frames but apply query-agnostic compression, such as uniform
1
Ours: Tempo
(6B, ~6 toks/f)
55
Existing Methods (Query-Agnostic)
LVBench Accuracy (%)
50 45
Sparse
Sampling
Ours: Tempo
(6B, ~4 toks/f)
40
Missed Transient Moment
35
20
Visual Tokens / Frame
Temporal Anchors (~0.5 tok/f)
...
ow many green mugs were on ...
H
Smart
Compressor
LLaMA-VID
(13B)
1 2
GPT-4o(2024-05-13) Baseline (30.8%) Marker Size = Params 6B
7B
8B
13B
4 6 8 16 64 256 2K Average Visual Tokens per Frame (Log Scale)
(c) Efficiency vs. Performance (LVBench)
Blurred Fine-grained Evidence Ours: Tempo with ATA (Query-Aware)
Qwen2.5-VL
(7B)
Kangaroo
(8B) Gemini 1.5 Pro Baseline (33.1%)
30 25
niform
Pooling
U
VideoLLaMA3
(7B, ~91toks/f)
VideoChat-Flash
(7B, 16toks/f)
16 toks
Fine-grained Details
~0.5 toks
Temporal Anchors
Video Temporal Sequence (Frames)
Fine-grained Details (16 tok/frame)
(a) Existing Query-Agnostic Methods vs. Ours (Query-Aware)
(b) Adaptive Token Allocation Visualized over Time
Figure 1 Tempo achieves SOTA long video understanding via query-aware Adaptive Token Allocation (ATA). (a) Motivation:
Query-agnostic methods either miss transient moments (sparse sampling) or blur details (uniform pooling). Tempo instead utilizes a small vision-language model as a smart compressor for query-aware cross-modal distillation. (b) Mechanism: ATA dynamically allocates high bandwidth (16 tokens/frame) to relevant segment for fine-grained details, while compressing redundant contexts into minimal temporal anchors (∼0.5 tokens/frame) to maintain causality. (c) Result: Leading performance on LVBench. Tempo-6B achieves superior accuracy at extreme compression rates (i.e., 4 or 6 tokens/frame), outperforming open-source models and proprietary baselines with a fraction of the context budget.
spatiotemporal pooling Maaz et al. (2024); Jiang et al. (2025) or token merging Bolya et al. (2022); Li et al. (2024b); Jin et al. (2024). By compressing without knowing what the user will ask, these heuristics often blur fine-grained evidence in query-critical segments while wasting representational bandwidth on irrelevant backgrounds. In essence, most existing pipelines reduce visual evidence before interacting with the language model, preventing the dynamic allocation of bandwidth to query-critical segments. Even pioneering query-aware approaches (e.g., LongVU Shen et al. (2024)) rely on disjoint auxiliary feature-matching modules, thereby decoupling the routing mechanism from the end-to-end multimodal pipeline. We introduce Tempo, an efficient query-aware framework for long video understanding that natively learns to compress videos for downstream text generation tasks. As its name suggests, Tempo acts as an intelligent temporal compressor that dynamically distributes the rhythm of the video: it allocates high token bandwidth to semantic beats relevant to the query while swiftly fast-forwarding through redundant contexts. Rather than treating visual compression as a purely visual, query-agnostic operation Jiang et al. (2025); Li et al. (2024b), Tempo casts this reduction as an early cross-modal semantic distillation process. Concretely, Tempo leverages a Small Vision-Language Model (SVLM) as a local compressor, seamlessly bridging it with an LLM for global understanding and response generation. By prepending the user query to the SVLM input, Tempo performs a preliminary cross-modal distillation pass that produces compact video memory tokens aligned with the user’s intent and is trained end-to-end with standard auto-regressive objectives. A practical challenge is enforcing a strict token budget at inference time (e.g., representing a 1024-frame video under an 8K visual token budget) without sacrificing either fine-grained evidence or global causal structure. To this end, we propose Adaptive Token Allocation (ATA), a training-free inference strategy guided by two key empirical properties of the Tempo architecture. (i) Zero-shot relevance prior and temporal anchors. Inheriting from the base model’s extensive multimodal pre-training, the local compressor exhibits a zero-shot ability to estimate query-video relevance without auxiliary supervision. ATA exploits this prior to allocate budgets segment-wise, enabling an aggressive dynamic compression range (0.5–16 tokens per frame). Crucially, instead of hard pruning, which breaks causality, ATA preserves dense representational bandwidth for relevant segments while compressing redundant contexts into minimal temporal anchors (i.e., 4 tokens) to maintain the global storyline. (ii) Semantic front-loading driven by causal attention. Our ablations empirically reveal 2
that under the SVLM’s causal attention, salient visual semantics natively concentrate into the earliest video memory tokens. Consequently, a simple head truncation effectively isolates high-value evidence, avoiding lossy spatial blurring with zero overhead. In summary, our contributions are: • Tempo: an end-to-end, query-aware compression framework for long video understanding. It directly addresses the context window bottleneck by unifying an SVLM-based local compressor and an LLM-based global decoder, performing query-conditioned cross-modal distillation in a single forward pass. • ATA: a training-free, budget-aware inference strategy leveraging the local compressor’s inherent zero-shot relevance prior and semantic front-loading. ATA dynamically dictates the optimal token allocation, preserving fine-grained details for query-critical moments while compressing redundancies into minimal temporal anchors to maintain global causal structure. • Scaling Behaviors: an empirical analysis revealing that optimal resource allocation varies with the task and video duration. While a 4K visual token budget acts as a sweet spot for standard long video tasks (e.g., Video-MME Long, 30–60 mins), restrictive budgets ultimately limit performance on extreme-long videos (e.g., LVBench, >1 hour). Scaling to larger capacities unlocks new performance peaks. Notably, in practice we observe that Tempo allocates tokens largely based on semantic necessity, often compressing hour-long videos far below the available token budget. • Leading Performances: despite being a compact 6B model, Tempo sets a new state-of-the-art across long video benchmarks. On challenging LVBench, it scores 52.3 under a 8K budget, outperforming proprietary baseline (e.g., GPT-4o, Gemini 1.5 Pro) and open-source counterparts (e.g., VideoChat-Flash). Scaling to 2048 frames with a 12K budget further pushes performance to 53.7, demonstrating robust hour-long video understanding of our proposed Tempo.
2
Related Work
2.1
Multimodal Large Language Models for Videos
The rapid evolution of MLLMs has established a dominant paradigm: aligning pre-trained visual encoders with powerful LLMs. Recent state-of-the-art models, such as VideoChat2 Li et al. (2025), VILA Lin et al. (2024b), LLaVA-OneVision Li et al. (2024a), VITA-1.5 Fu et al. (2025b), Kimi-VL Team et al. (2025), InternVL3.5 Wang et al. (2025b), Molmo2 Clark et al. (2026), and the Qwen-VL series Bai et al. (2025), demonstrate exceptional capability in short video understanding. They typically map sampled video frames directly into the LLM’s context window. While effective for short-horizon tasks, extending this dense representation to hour-long videos results in a linear explosion of visual tokens. This quickly overwhelms the maximum context length of the LLM, leading to prohibitive computational costs and exacerbating the lost-in-the-middle phenomenon Liu et al. (2024c), where models fail to retrieve pivotal evidence buried in extensive multimodal contexts.
2.2
Context Extension and Token Reduction
To comprehend extended temporal horizons, recent efforts generally bifurcate into two directions. The first direction focuses on context extension via algorithmic extrapolation, architectural innovations, or system-level parallelization to natively support massive token sequences. For instance, LongVA Zhang et al. (2024a) extrapolates the context window to comprehend extensive visual tokens, LongVILA Chen et al. (2024b) introduces sequence parallelism for long-context training, and LongLLaVA Wang et al. (2024) employs a hybrid Mamba-Transformer architecture to mitigate memory constraints. While these approaches successfully preserve visual fidelity and push the context boundaries, they strictly rely on processing dense visual streams. Consequently, ingesting hundreds of thousands of visual tokens per video still incurs exorbitant memory footprints and computational overhead, rendering them highly resource-intensive for routine inference. The second, more prevalent direction relies on query-agnostic token reduction. Drawing inspiration from image-level token pruning and merging techniques like FastV Chen et al. (2024a) and ToMe Bolya et al. (2022), video MLLMs typically employ spatiotemporal pooling or fixed-rate sparse sampling Maaz et al. (2024); Li et al. (2025); Jin et al. (2024); Li et al. (2024b); Jiang et al. (2025). For instance, VideoChat-Flash Li et al. (2024b)
3
Inference-Only Bypass
The Local Compressor
Training-Free ATA
Micro View - Causal Base
n t i eo Segment Si
I pu V d
Zero-shot Relevance
hirel
ATA
Controller
Vision Encoder
all Vision-Language Model
Sm
stem
Prompt
Sy
Visual Tokens Xi
Intercept hirel
User Query
Q
Segment 1: High Score si
Memory Tokens
M (kmax=128)
usal Attention: M
e s to all conditions
Ca
Tokens (kmax)
Global Budget
Bmax
...
<t=2.0s>
...
<t=4.0s>
...
User Query
Q
Large Language Model (e.g., Qwen3-4B) ...
Keep all details (ki = 128)
lobal Self-Attention ...
G
lobal Self-Attention
G
Segment 2: Low Score si ...
att nd
Full Memory
<t=0.0s>
(SVLM)
(e.g., SigLIP)
The Global Decoder
Macro View - Time-Aware Sequence
Keep temporal anchors (ki = 4)
Final Answer
Figure 2 Overview of the Tempo framework. Our unified architecture casts long video understanding as an end-to-end, query-aware compression process. The Local Compressor (Left). For each segment, a Small Vision-Language Model
(SVLM) acts as a semantic temporal compressor. Under causal attention, learnable memory tokens M inherently distill the preceding visual tokens Xi and user query Q. Inference-Only Bypass (Middle). During a single forward pass, an Adaptive Token Allocation (ATA) controller intercepts the hidden state hrel to compute a zero-shot relevance i score si . This enables an O(1) dynamic head truncation, allocating dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to strictly satisfy a global budget Bmax . The Global Decoder (Right). The compressed memory tokens are assembled into a highly sparse, time-aware sequence using explicit temporal tags (e.g., <t=2.0s>). A global LLM synthesizes this condensed multimodal context to generate the final response.
leverages visual redundancy to hierarchically compress tokens, while Storm Jiang et al. (2025) applies temporal and spatial pooling to fit tight token budgets. However, because these heuristics are completely agnostic to the user’s textual query, they risk blurring semantic boundaries and discarding transient, fine-grained segments that may be critical to the downstream question.
2.3
Hierarchical and Query-Aware Video Architectures
To overcome uniform processing limits, models like SLOWFAST-LLAVA, LLaVA-Videoslowf ast , and KeyeVL-1.5 deploy dual pathways to balance spatial and temporal resolutions Feichtenhofer et al. (2019); Xu et al. (2024b); Zhang et al. (2024b); Yang et al. (2025). However, whether utilizing static sampling or dynamic inter-frame similarity, their resource allocation remains purely vision-driven and fundamentally detached from the user’s textual intent. Recent works have begun to explore query-aware processing Li et al. (2024c); Islam et al. (2025); Shen et al. (2024). BIMBA Islam et al. (2025) introduces an optional query-conditioned token selection mechanism. LongVU Shen et al. (2024) leverages cross-modal attention for selective spatial compression, yet still depends on disjoint auxiliary modules that decouple the routing mechanism from the end-to-end multimodal decoding process. Tempo fundamentally advances this trajectory by natively employing an SVLM as an active, query-conditioned temporal compressor in a single forward pass. Furthermore, our ATA mechanism exploits the SVLM’s inherent zero-shot relevance prior to dynamically dictate the video’s rhythm. This preserves dense, high-fidelity tokens for critical segments while compressing irrelevant backgrounds into minimal temporal anchors, achieving causal-preserving sequence assembly with zero routing overhead.
3
TEMPO
3.1
Overall
We target the fundamental bottleneck in long video MLLMs: the downstream LLM can only attend to a limited number of visual tokens, while hour-long videos produce a massive, continuous stream. Tempo resolves this mismatch by turning visual token reduction into an early cross-modal distillation problem.
4
Problem Setup. Given a long video V and a user query Q, we uniformly partition V into N temporal segments S = {S1 , . . . , SN }. Our goal is to convert each Si into a compact set of query-conditioned video memory tokens, with the total sequence bounded by a global inference budget Bmax , enabling the downstream LLM to process the entire video and generate the final answer efficiently. Architecture. Tempo constitutes a two-level generative hierarchy (Fig. 2): (1) an SVLM-based local compressor Cϕ , and (2) an LLM-based global decoder Dθ . Concretely, the SVLM’s native vision encoder maps segment Si to dense visual tokens Xi . Its causal attention then performs query-conditioned distillation, integrating Xi and query Q into learnable memory tokens M. This yields a fixed-capacity representation Hi of exactly kmax tokens. A linear projector maps Hi into the LLM’s embedding space as H̃i . Finally, the global LLM Dθ consumes all memory tokens {H̃i }N i=1 alongside Q to auto-regressively decode the answer. Training vs. Inference. Tempo is trained with a fixed per-segment capacity kmax to learn a strong query-aware local compressor Cϕ . At inference, we additionally enforce a global budget Bmax . We therefore introduce ATA, a training-free strategy that uses a zero-shot relevance prior extracted from the same SVLM forward pass to allocate per-segment budgets ki ∈ [kmin , kmax ], followed by a constant-time head truncation.
3.2
Query-Aware Visual Compression
We cast segment compression as a query-driven sequence-to-sequence transformation. An explicit information bottleneck forces the Cϕ to discard visual redundancies and distill semantic evidence relevant to user intent. SVLM Input Construction. For each segment Si , the SVLM constructs a single causal sequence comprising: (i) a system prompt, (ii) visual tokens Xi (extracted via its native vision encoder), (iii) user query Q, and (iv) learnable memory tokens M. Placing M last is critical: under causal attention, each memory token inherently attends to all preceding visual and textual contexts. This conditions the SVLM to distill query-aligned evidence into M. Extracting their final-layer hidden states yields the compressed representation Hi ∈ Rkmax ×ds . Sequence Assembly & Temporal Grounding. To preserve temporal identity and causal order across the entire video, we prepend an explicit textual timestamp (e.g., <t=2.0s>) to each segment when assembling the global context. In practice, these temporal tags significantly stabilize long-range attribution (i.e., which evidence comes from where) within the downstream global LLM. End-to-End Learning. Let the ground-truth answer be A = {at }Tt=1 . The global Dθ receives all projected segment memories {H̃i }N i=1 in temporal order, optimized via standard auto-regressive next-token prediction: LAR (θ, ϕ) = −
T X
log pθ at | a<t , Q, {H̃i }N i=1
(1)
t=1
Crucially, we do not impose auxiliary compression losses, routing networks, or heuristic token-dropping regularizations during training. The fixed capacity of kmax memory tokens acts as a hard structural bottleneck. The gradients back-propagated from LAR naturally compel the compressor Cϕ to discard query-irrelevant backgrounds and pack the most predictive visual evidence into this bounded space.
3.3
Zero-Shot Relevance Prior
A core insight driving Tempo is that modern multimodal foundation models (e.g., Qwen3-VL Bai et al. (2025)) inherently possess a robust zero-shot capability to evaluate semantic alignment between a visual sequence and a text query (Refer to Sec. 4.3 – D). We harness this foundational prior to extract a highly accurate relevance signal without introducing or training any auxiliary routing modules. Logit-Based Relevance Score. To explicitly elicit this prior during inference, we slightly augment our training system prompt. Following the standard compression instruction, we append a strict binary directive: “Now, ds before compressing, answer exactly ‘Yes’ or ‘No’: is this segment relevant to the Query?” Let hrel be i ∈R 5
Algorithm 1: Adaptive Token Allocation (ATA) at inference N Input: Segment memories {Hi }N i=1 , relevance scores {si }i=1 , budget Bmax , bounds kmin , kmax ATA N Output: Budgeted memories {Hi }i=1
Normalize {si } → {ŝi } by min-max scaling; Compute kiideal via Eq. (3); P ideal if ≤ Bmax then i ki ki ← kiideal ; else
ki ← Eq. (4) with Bbase = N kmin ; P Discretize {ki } to integers s.t. i ki ≤ Bmax ; HATA ← Hi [1:ki ], ∀i ∈ {1, . . . , N }; i return {HATA }N i i=1 ;
the final hidden state immediately preceding the model’s binary response. Using the SVLM’s frozen language modeling head weights for the vocabulary tokens Yes (wyes ) and No (wno ), we compute a continuous relevance probability si via logit difference Li et al. (2026): si = σ (wyes − wno )⊤ hrel ∈ (0, 1), (2) i where σ(·) is the Sigmoid function. This O(1) projection avoids auto-regressive decoding overhead while yielding a highly stable ranking signal. Single-Pass Design. The score si and the compressed memory tokens Hi are extracted within a single forward pass of Cϕ . As illustrated in the Inference-Only Bypass (Fig. 2), we simply intercept the hidden state hrel i to compute the zero-shot score, and then seamlessly continue the forward pass to extract Hi . This architectural elegance guarantees that both the relevance routing signal and the compressed representations are rigorously conditioned on the exact same multimodal context, achieving adaptive evaluation with effectively zero latency.
3.4
Adaptive Token Allocation (ATA)
At inference, the total visual context provided to the global LLM must strictly satisfy a bounded capacity Bmax . As illustrated in Fig. 2, the ATA controller translates the zero-shot scores {si } into dynamic per-segment token budgets ki , executing the physical compression via zero-overhead head truncation. Stage 1: Contrastive Linear Allocation. To guarantee causal continuity across the entire video sequence, we enforce a minimal temporal anchor for every segment, regardless of its relevance. We first normalize the raw scores via Min-Max scaling: ŝi = (si − min(s))/(max(s) − min(s) + ϵ). To maximize the contrast between query-critical events and irrelevant backgrounds, we linearly map these normalized scores to a target capacity: (3)
kiideal = kmin + ⌊(kmax − kmin ) · ŝi ⌋.
Stage 2: Capacity-Aware Protection. Let Bbase = N ·kmin represent the foundational cost Prequired to maintain the global temporal anchors. If the sum of ideal allocations satisfies the global limit ( i kiideal ≤ Bmax ), we directly adopt {kiideal } to maximize sparsity. Otherwise, we distribute the residual budget Bres = Bmax − Bbase proportionally based on the normalized scores: % $ ŝi ki = kmin + Bres · PN . (4) j=1 ŝj + ϵ We then discretize {ki } and distribute any fractional remainders to strictly ensure
6
P
i ki ≤ Bmax (Alg. 1).
Head Truncation: Zero-Overhead Token Selection. Once the dynamic budget ki is allocated, we compress the segment by simply slicing the memory sequence, i.e., HATA = Hi [1:ki ]. Driven by the auto-regressive i nature of the SVLM’s causal attention, we empirically observe a semantic front-loading phenomenon: the local compressor packs the most salient global evidence into the earliest generated memory tokens (Refer to Sec. 4.3 – C). Consequently, this O(1) tensor slice naturally isolates high-value semantics without introducing lossy spatiotemporal pooling. The final global sequence {H̃ATA }N i i=1 strictly conforms to Bmax , rendering memory footprints entirely predictable even for hour-long reasoning.
4
Experiments
4.1
Experimental Setup
Architecture & Implementation. Tempo’s local SVLM is initialized from Qwen3-VL-2B-Instruct, while the global LLM uses Qwen3-LM-4B. A linear projector bridges the SVLM’s memory space to the LLM, yielding a compact 6B-parameter architecture. We extract frames at 2 FPS via Decord, applying uniform subsampling if limits are exceeded. During training, continuous videos are partitioned into 4-frame segments, each compressed by the SVLM into kmax = 128 memory tokens. During inference, we expand the segment window to 8 frames. ATA (Sec. 3.4) strictly enforces a global visual budget Bmax (4K or 8K) via head truncation. Models are trained on a 64-GPU cluster with FSDP. Additional hyper-parameters are provided in the Appendix C. Progressive Training Curriculum. We adopt a rigorous four-stage progressive training curriculum to ensure stable optimization and context extrapolation: • Stage 0 (Modality Alignment): We freeze both the SVLM and the LLM, exclusively optimizing the linear projector on the standard LCS-558K dataset Liu et al. (2023). This establishes the fundamental vision-language alignment, bridging the SVLM’s visual representations with the LLM’s text embedding. • Stage 1 (Pre-training): We unfreeze the entire architecture and optimize it on a large-scale, curated multimodal corpus comprising ∼2M images, ∼1.38M videos, and ∼143K pure text samples. During this phase, videos are sparsely sampled at 8 frames, endowing the model with initial temporal perception. • Stage 2 (Broad Supervised Fine-Tuning): To develop robust instruction-following and semantic-aware temporal reasoning capabilities, we perform comprehensive SFT using a highly diverse data mixture (∼0.93M images, ∼2.25M videos, and ∼71K text samples). In this stage, the temporal context is systematically expanded, with the maximum number of sampled frames per video strictly capped at 128. • Stage 3 (Long-Context SFT): To effectively extrapolate the context window, we freeze the SVLM and exclusively fine-tune the global LLM on a high-quality subset of ∼384K samples from Stage 2. Here, the maximum frame limit is extended to 384, enabling the LLM to handle long temporal sequences. To curate our training data, we primarily follow the data mixtures established by VideoChat-Flash Li et al. (2024b) and LLaVA-OneVision-1.5 An et al. (2025). All training datasets utilized throughout our progressive curriculum are publicly accessible, ensuring full reproducibility. Evaluation Benchmarks & Baselines. To evaluate Tempo’s long video understanding, we conduct comprehensive experiments across four prominent benchmarks, i.e., LongVideoBench Wu et al. (2024), MLVU Zhou et al. (2025), Video-MME Fu et al. (2025a), LVBench (extreme-long video) Wang et al. (2025a), spanning standard long-form tasks to hour-long stress tests. We benchmark Tempo against widely adopted proprietary baselines (e.g., GPT-4o, Gemini Pro 1.5), general open-weight MLLMs (e.g., InternVL, Qwen-VL), and specialized long-video MLLMs (e.g., VideoChat-Flash, LongVA). All evaluations are conducted using the lmms-eval.
4.2
Long Video Understanding
Tab. 1 summarizes the evaluation of Tempo (capped at 1024 frames) against state-of-the-art MLLMs across four major benchmarks. Despite a compact 6B-parameter architecture and aggressive token compression (0.5–16 tokens/frame), Tempo achieves state-of-the-art performance. While larger open-weight models (e.g., Qwen3-VL 8B, Molmo2) yield strong absolute scores via exorbitant visual token consumption, Tempo operates 7
Table 1 Comparison with state-of-the-art MLLMs on long video benchmarks, highlighting Tempo’s superior accuracy and extreme token efficiency. Bold and underline denote the best and second-best results among specialized long video MLLMs. “-”
indicates unavailable results. * indicates the average tokens per frame are dynamically adjusted. For our model, we report the theoretical dynamic range (0.5–16) alongside the actual empirical average tokens per frame ( gray rows ), demonstrating that Tempo inherently operates substantially below the maximum budget limits in practice. Tokens
LongVideoBench
MLVU
per frame
(473s)
(651s)
Overall (1010s)
Long (2386s)
(4101s)
-
-
66.7 64.0
64.6 -
71.9 75.0
65.3 67.4
30.8 33.1
7B 7B 7B 7B 8B 8B 7B 2B 8B
72 196 676 ≤ 91 256 83 1924 ≤ 640 ≤ 640
56.4 58.2 59.8 62.1 67.5 56.0 -
47.9 64.7 70.8 73.0 70.2 70.2 68.3 78.1
45.3 58.2 63.3 66.2 66.0 69.9 65.1 61.9 71.4
39.8 54.9 -
45.3 52.8 45.3 47.4 58.0
7B 7B 8B A13B 7B 7B 7B 7B 7B
2 144 256 144 196 64 64 36 16
54.8 53.5 57.1 60.5 59.5 64.7
33.2 56.3 61.0 65.4 72.9 71.4 74.7
25.9 52.6 56.0 53.8 60.1 60.6 63.4 64.7 65.3
46.2 46.7 46.4 47.0 59.5 53.4 55.4
23.9 (13B) 39.4 48.2
6B
0.5–16
64.5 2.8
75.6
67.8
57.8
52.7
2.8
3.6
3.4
2.9
0.5–16
65.1
,→ actual avg. toks/frame
3.1
75.2 3.3
67.7 4.3
57.0 4.1
52.3 3.5
Model
Size
Video-MME (w/o sub.)
LVBench
Proprietary Models GPT-4o Hurst et al. (2024) Gemini 1.5 Pro Team et al. (2024) General Open-Source MLLMs VideoChat2-HD Li et al. (2025) LLaVA-OneVision Li et al. (2024a) LLaVA-Video Zhang et al. (2024b) VideoLLaMA3* Zhang et al. (2025) InternVL3.5 Wang et al. (2025b) Molmo2 Clark et al. (2026) Qwen2.5-VL Bai et al. (2025) Qwen3-VL* Bai et al. (2025) Qwen3-VL* Bai et al. (2025) Specialized Long Video MLLMs LLaMA-VID Li et al. (2024c) LongVA Zhang et al. (2024a) Kangaroo Liu et al. (2024b) LongLLaVA Wang et al. (2024) LongVILA Chen et al. (2024b) LongVU Shen et al. (2024) Storm Jiang et al. (2025) BIMBA Islam et al. (2025) VideoChat-Flash Li et al. (2024b) Tempo* (4K Budget)
,→ actual avg. toks/frame Tempo* (8K Budget)
6B
under extreme efficiency. By routing evidence through ATA, Tempo strictly bounds visual tokens to 4K or 8K budgets. In practice, ATA dynamically distributes bandwidth so efficiently that the actual consumption falls well below these limits (e.g., 2.9 tokens/frame on LVBench under the 4K budget). Remarkably, its comparative advantage over specialized long video MLLMs amplifies as the temporal span extends. Dominance in Ultra-Long Video Understanding. The most notable results emerge on the extreme-long benchmark LVBench, a rigorous stress test for long-term memory and evidence retrieval. Operating strictly within a 4K visual budget, Tempo achieves 52.7, outperforming the strongest 7B specialized MLLM, VideoChatFlash (48.2), by 4.5 points. Impressively, despite its compact capacity, Tempo eclipses proprietary systems in this ultra-long setting, surpassing GPT-4o (30.8) and Gemini 1.5 Pro (33.1) by massive margins. This proves that explicit query-aware compression is vastly superior to blindly feeding raw frames into expansive LLM context windows, which often suffer from attention dilution. Robustness Across Varied Temporal Contexts. This dominance consistently extends across other benchmarks. On Video-MME, Tempo secures 67.8 under the 4K budget, exceeding VideoChat-Flash (65.3) and showing massive improvement over its base model Qwen3-VL-2B (61.9). On the challenging Video-MME Long subset (2386s), Tempo achieves 57.8. Similarly, Tempo delivers SOTA-level performances on MLVU (75.6) and LongVideoBench (65.1 under 8K), asserting its robust generalization across diverse temporal scales and tasks.
8
Table 2 Ablation studies on Tempo’s core components. We decompose our framework across five dimensions: (A)
progressive training curriculum, (B) segment-level budget allocation, (C) intra-segment token reduction scheme, (D) relevance scoring source, and (E) temporal continuity. Unless otherwise specified, all variants process videos uniformly sampled at 2 FPS up to a maximum of 1024 frames, strictly bounded by an 8K visual token budget for fair comparison. The default Tempo configuration is highlighted in gray . LongVB denotes LongVideoBench.
Ablation Setting
A. Progressive Training Curriculum w/o Stage 3 - Long-Context SFT (16K Budget) w/o Adaptive Token Allocation (16K Budget) Tempo Default (8K Budget) B. Segment-Level Budget Allocation Uniform Subsampling (Equal tokens per segment) Random Drop (Uniform random segment selection) Adversarial Routing (Keep lowest-scoring segments) Hard Top-K Routing (Keep highest-scoring segments) ATA (Adaptive token allocation, Alg. 1) C. Intra-Segment Token Reduction Scheme Uniform Tail Truncation (Fixed 64 tokens) Uniform Head Truncation (Fixed 64 tokens) Token Merging (Merge visual features to ki tokens) Dynamic Tail Truncation (Keep last ki tokens) Dynamic Head Truncation (Keep first ki tokens)
LongVB
MLVU
(473s)
(651s)
61.4 62.8
67.2 73.5
Video-MME (w/o sub.) Overall
Long
(1010s)
(2386s)
66.1 67.0
56.3 56.2
LVBench
(4101s) 47.3 51.1
65.1
75.2
67.7
57.0
52.3
61.9 59.3 50.7 63.5
74.0 70.9 59.3 73.9
66.3 63.6 52.4 66.7
55.2 55.2 47.8 56.2
49.9 49.8 36.9 52.7
65.1
75.2
67.7
57.0
52.3
59.5 63.2 63.6 61.9
71.6 73.4 74.9 73.4
64.1 66.9 66.3 64.8
54.8 56.2 55.4 54.2
41.8 51.5 53.0 50.5
65.1
75.2
67.7
57.0
52.3
64.6 65.7 64.3 64.1
75.1 76.3 75.4 75.4
67.2 67.6 67.2 67.2
56.1 57.3 57.0 57.0
52.6 52.7 51.8 53.4
D. Relevance Scoring Source & Zero-Shot Prior Base Model Prior (Standard prompt) Base Model Prior (Explicit routing prompt) External Dense Retriever (Qwen3-VL Reranker) Tempo SVLM Prior (Standard prompt) Tempo SVLM Prior (Explicit routing prompt)
65.1
E. Temporal Continuity (Minimum Token Guarantee) Hard Pruning (0 tokens for irrelevant segments) Minimal Temporal Anchors (kmin = 4)
65.1
63.9
75.2
74.8
75.2
67.7
67.4
67.7
57.0
56.3
57.0
52.3
52.3
52.3
The “Less is More” Phenomenon. Crucially, Tempo’s performance under the 4K budget frequently matches or exceeds the 8K budget (e.g., 52.7 vs. 52.3 on LVBench; 57.8 vs. 57.0 on Video-MME Long subset). This counter-intuitive phenomenon powerfully validates our ATA strategy. Enforcing a stricter information bottleneck filters out background distractors, forcing the LLM to focus purely on high-value semantic beats. This actively mitigates the lost-in-the-middle phenomenon without requiring additional inference compute. Qualitative Analysis. We provide comprehensive qualitative results in Appendix B to further analyze Tempo’s adaptive behavior. We contrast localized queries (requiring pinpoint accuracy) with global queries (requiring holistic understanding). These comparisons explicitly demonstrate how Tempo dynamically shifts its compression rhythm—allocating high-fidelity bandwidth to query-critical moments while applying extreme sparsity to irrelevant backgrounds. For overarching queries lacking singular salient events, ATA gracefully defaults to a smooth, low-variance token allocation, ensuring global temporal comprehension remains intact.
4.3
Ablation Studies
We systematically decompose Tempo’s core components in Tab. 2. Unless specified, all variants process videos uniformly sampled at 2 FPS for a maximum of 1024 frames, strictly bounded by an 8K visual token budget. A. Progressive Training Curriculum. We first evaluate our training stages (Tab. 2A). Stopping after Stage 2 (w/o Long-Context SFT) yields sub-optimal performance on extreme-long benchmarks (e.g., 47.3 on
9