arXiv CSopen access

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference Tinghao Wang1,2,∗ , Yichen Guo1,3,∗ , Rui Huang2,∗,† , Zheng Lu2 , Qizhe Zhang1 , Chenxi Li4 , Yuan Zhang1 , Jiajun Cao1 , Zhirong Shen2 , Yaosong Du2 , Guangyan Gan3 , Wenya Wang3 , Lin William Cong3 , Shanghang Zhang1,‡ 1

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University

2 3

University of Electronic Science and Technology of China

Nanyang Technological University, 4 Beijing Academy of Artificial Intelligence (BAAI)

arXiv:2606.27161v1 [cs.AI] 25 Jun 2026

(a)

(b)

Figure 1: (a) Qualitative comparison of pruning methods. On detail-sensitive VQA questions, single-criterion pruning methods, including attention-based, diversity-based, and coverage-based methods, often fail to answer, whereas the multi-stage TOPS module helps model preserve key visual evidence and produce the correct answers. (b) Performance comparison on four mainstream MLLMs. We validate TOPS across four architectures. TOPS consistently covers the largest area, demonstrating superior performance across all models and benchmarks.

Abstract Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead. Visual token pruning offers a natural solution, yet existing methods are imperfect: attention-based criteria tend to retain redundant tokens, while diversity-based criteria are often agnostic to user instructions. Even methods that combine multiple criteria still lack a principled formulation of the intrinsic objective of token pruning. In this paper, we revisit visual token pruning from a first-principles perspective and formulate it as constructing Token Optimal Preservation Sets. Through a top-down informationtheoretic analysis, we identify three fundamen*

Equal contribution. † Project leader. ‡ Corresponding author.

tal principles for effective token selection: Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs. Extensive experiments on 7 MLLM backbones and 14 benchmarks demonstrate that TOPS outperforms prior methods under diverse pruning settings. Notably, on LLaVA-NeXT, TOPS removes 77.8% of visual tokens while preserving 100.0% and 100.6% performance on its 7B and 13B models, respectively, suggesting that pruning redundant visual tokens can sometimes mitigate hallucination and inspire future lightweight MLLM design.

1

Introduction

Large language models (LLMs) (Achiam et al., 2023; Hurst et al., 2024; Singh et al., 2025; Yang et al., 2025a; Team, 2026; Touvron et al., 2023a,b;

Grattafiori et al., 2024; Team et al., 2023, 2024; Comanici et al., 2025; Team et al., 2026, 2025) have achieved remarkable success in language understanding and reasoning. Building on these capabilities, multimodal large language models (MLLMs) (Liu et al., 2023, 2024b; Li et al., 2024; Zhang et al., 2024b; Bai et al., 2025b,a; Chen et al., 2024c; Zhu et al., 2025) have made rapid progress in multimodal understanding. However, their efficiency is limited by numerous visual tokens, which are processed through all transformer layers (Chen et al., 2024a; Zhang et al., 2024a, 2025c,a). Since self-attention scales quadratically with sequence length, these tokens introduce substantial computational and memory overhead, especially for multiimage and high-resolution inputs. Therefore, reducing visual tokens while preserving performance is a critical challenge. Previous methods (Chen et al., 2024a; Yang et al., 2025b; Wang et al., 2026; Cao et al., 2026) have attempted to reduce visual tokens to lower the inference cost of MLLMs. Existing pruning approaches can be broadly categorized into three types. Attention-based methods (Zhang et al., 2025c; Xing et al., 2024; Zhang et al., 2024a) identify token importance via cross-modal attention or cls token attention, but often retain highly similar tokens, resulting in redundancy. Diversity-based methods (Alvar et al., 2025; Wen et al., 2025) encourage semantic dispersion, yet are typically agnostic to user instructions and may discard taskcritical evidence. Other methods combine multiple criteria or incorporate coverage-based objectives (Song et al., 2025; Shang et al., 2025; Zhang et al., 2025c; Baek et al., 2026; Zhang et al., 2025b) to model the representativeness of selected subsets. However, despite these advances, existing methods mostly treat token pruning as a scoring problem and rank tokens based on heuristic criteria, without a principled justification for why such criteria are appropriate or sufficient for constructing an optimal token subset. Fundamentally, these approaches do not start from the intrinsic objective of pruning, but rely on heuristic scoring schemes. To address these challenges, we move beyond conventional heuristic scoring schemes (Rao et al., 2021; Liang et al., 2022; Bolya et al., 2022; Chen et al., 2024a; Shang et al., 2025) and revisit token pruning from a first-principles perspective. Instead of designing new scoring heuristics, we rethink the core objective of token selection, conduct a top-down analysis using information theory, and

identify three fundamental principles for effective pruning—Task Relevance, Information Coverage, and Semantic Diversity. Based on these principles, we formulate token pruning as an optimal subset selection problem and propose TOPS, which constructs a compact yet sufficient token subset that satisfies the proposed properties and can be applied at any pruning point during MLLM inference. We further implement TOPS as a two-stage pipeline for fine-grained token reduction, where Stage I removes coarse visual redundancy and Stage II performs text-aware refinement. As shown in Figure. 1(a), TOPS better preserves task-critical visual evidence under aggressive pruning. As a simple yet effective solution, TOPS does not depend on any specific visual encoder or language model, which means it can be readily implemented across any token-based MLLM. Extensive experiments across various MLLMs demonstrate the effectiveness and efficiency of TOPS, surpassing existing methods (Figure. 1(b)). For instance, on LLaVA-v1.5-7B (Liu et al., 2023) and LLaVANeXT-7B (Liu et al., 2024b), TOPS removes nearly 90% of visual tokens while retaining 97.1% and 99.1% of the original performance. Overall, our main contributions are as follows: • We revisit token pruning from a first-principles perspective, conduct a top-down analysis using information theory and identify three criteria that govern effective token selection. • We propose TOPS, a training-free and modelagnostic pruning module, considering the fundamental criteria and dynamically constructing token optimal preservation sets in MLLMs. • We conduct extensive experiments across various MLLMs and benchmarks, demonstrating TOPS consistently achieves state-of-the-art performance across different reduction ratios.

2

Related Work

Multimodal large language models. Multimodal large language models (MLLMs) (Liu et al., 2024a; Bai et al., 2023b; Chen et al., 2024c; Li et al., 2024; Team et al., 2023; Hurst et al., 2024; Team et al., 2024) extend large language models (LLMs) (Brown et al., 2020; Achiam et al., 2023; Bai et al., 2023a; Yang et al., 2024; Touvron et al., 2023a; Peng et al., 2023; Bi et al., 2024) to multimodal understanding by encoding visual inputs as token sequences and processing them together with text tokens. However, visual tokenization introduces substantial computational overhead, since

Figure 2: Logit fidelity of pruning methods across token budgets (128/64/32) on 200 MME samples. We report ∆L = Lpruned − Lvanilla , where lower values indicate smaller output distortion. TOPS consistently achieves the lowest loss increase, demonstrating stronger fidelity under aggressive pruning.

visual tokens are often far more numerous than text tokens and are propagated through all LLM layers. For example, LLaVA-1.5 (Liu et al., 2024a) represents a 336 × 336 image with 576 tokens, while LLaVA-NeXT (Liu et al., 2024b) can produce up to 2, 880 tokens for high-resolution inputs. The problem becomes more severe in video understanding (Lin et al., 2024; Kondratyuk et al., 2023), where long frame sequences lead to long visual token sequences and expensive inference. Therefore, effective token reduction is essential for scalable MLLM inference. Visual token reduction. Visual token reduction aims to improve MLLM efficiency by removing redundant visual tokens (Jin et al., 2025). Existing training-free methods can be broadly categorized by their selection criteria. Attention-based methods estimate token importance from attention signals (Chen et al., 2024a; Xing et al., 2024; Zhang et al., 2024a; Zhang et al.; Yang et al., 2025b), such as CLS-to-patch attention in VisionZip (Yang et al., 2025b) or text-guided cross-modal attention in FastV (Chen et al., 2024a). While effective for retaining salient tokens, they often preserve redundant tokens with similar semantics. Diversitybased methods (Alvar et al., 2025; Wen et al., 2025) reduce redundancy by encouraging semantic dispersion, but are usually instruction-agnostic and may discard task-critical evidence. More recent methods combine importance, diversity, saliency, coverage, or progressive pruning strategies (Song et al., 2025; Shang et al., 2025; Zhang et al., 2025c,d; Baek et al., 2026; Zhang et al., 2025b; Tan et al., 2026; Wang et al., 2026; Deng et al., 2025; Liu et al., 2024c; Zhang et al., 2025a). Despite improved performance, they still rely on heuristic

scoring schemes and lack a principled formulation of the intrinsic objective of token pruning.

3

Motivation

Modern MLLMs typically consist of a vision encoder fv , a multimodal projector g, and a language model fϕ . Given an image Xv and a textual query Q, the model produces visual tokens V = g(fv (Xv )) ∈ Rn×d . Since n is typically much larger than the number of text tokens, visual token pruning seeks a subset S ∗ = arg minS⊂V, |S|=K D(fϕ (S, Q) ∥ fϕ (V, Q)), where D(·∥·) measures the output divergence between the pruned and full models. 3.1 First-Principles Pruning Formulation Stepping beyond the conventional visual token pruning paradigm, we revisit the problem from a first-principles perspective. Given the full visual token set V and the textual query Q, the goal of token pruning is to retain a subset S ⊆ V such that reasoning based on (S, Q) remains consistent with that based on (V, Q). We formalize this as an information-theoretic objective: maxS⊆V, |S|≤K I(S; V, Q) (1) where I(·; ·) denotes mutual information. This defines the first-principles of visual token pruning: the optimal subset is one that maximally preserves information about both V and Q. 3.2 Decomposition of the First-Principles By looking inside the mutual information objective, we can decompose it via the chain rule: I(S; V, Q) = I(S; Q) + I(S; V | Q) (2) | {z } | {z } task relevance

information coverage

= I(S; Q) + H(V | Q) − H(V | S, Q).

Figure 3: Overview of TOPS. Left: TOPS is a plug-and-play pruning module that can be applied at multiple stages during MLLM inference. Right: at each pruning point, TOPS constructs the optimal token preservation set by greedily selecting tokens that jointly maximize task relevance, information coverage, and semantic diversity—the three criteria derived from our first-principles formulation.

The two terms reflect two key properties of an optimal subset: Task Relevance I(S; Q) measures how informative S is for the query, while Information Coverage I(S; V | Q) encourages S to preserve sufficient information about the original visual set V by minimizing H(V | S, Q). Let Sk = {t1 , . . . , tk } be the subset selected after k steps. The total mutual information can be decomposed into per-token contributions: k X I(Sk ; V, Q) = I(ti ; V, Q | Si−1 ). (3)

for semantic diversity, and SCOPE (Deng et al., 2025) for information coverage. We measure the inference loss increase over the unpruned model to quantify pruning degradation. As shown in Figure 2, relevance-based pruning performs better at low pruning ratios, while diversity- and coveragebased methods become more effective under aggressive pruning. This suggests that different principles dominate under different budgets, and combining them in TOPS yields the best overall performance. Additional results are reported in Table 5.

i=1

This means each token’s value depends on what has already been selected. Specifically, the marginal contribution of adding token ti to the current set Si−1 is the conditional mutual information: ∆i = I(Si ; V, Q) − I(Si−1 ; V, Q) (4) = I(ti ; V, Q | Si−1 ). Furthermore, this per-step gain admits a natural upper bound governed by the conditional entropy of the candidate token: 0 ≤ I(ti ; V, Q | Si−1 ) ≤ H(ti | Si−1 ) ≤ H(ti ). (5) The upper bound H(ti | Si−1 ) is tight when ti carries novel information beyond the current set, and diminishes as ti becomes redundant with already selected tokens. This reveals that, under a fixed budget, maximizing mutual information favors candidates that are semantically distinct from existing selections, yielding Semantic Diversity as a necessary condition for efficient and non-redundant subset construction. 3.3 Principle Instantiation and Analysis We empirically evaluate three principles using representative methods: FastV (Chen et al., 2024a) for task relevance, DivPrune (Alvar et al., 2025)

4

Method

4.1 Token Optimal Preservation Set Utilizing the first-principles derived in Section 3.2, we design the TOPS construction procedure. At each pruning point, we dynamically select a set of text raters Tr and precompute a pairwise similarity matrix F over the visual tokens. For each candidate token i, given the current selected subset S, we denote the remaining visual tokens as U = V P \ S and compute its task relevance ri = |T1r | t∈Tr Attn(t → i) via text-rater attention, and its information coverage ci (U ) = P j∈U max(0, sim(hi , hj ) − maxk∈S sim(hj , hk )). After min-max normalization, denoted by r̃i and c̃i (U ), these two scores are combined into an information preservation score: g i = r̃i + λ c̃i (U ). info (6) Following the diversity principle derived before, we further incorporate a semantic diversity score di (S) = 1 − maxj∈S sim(hi , hj ) to select. The subset is then expanded greedily as:    g ˜ St+1 = St ∪ arg max infoi + α di (St ) . i∈V \St

(7)

Table 1: Comparison of pruning methods across LLaVA series. All numbers report Rel. (%), the ratio of pruned model accuracy to baseline. Red : attention-based. Green : attention&diversity. Blue : diversity-based. Cyan : coverage-based. Purple : ours. “–”: not available. Detailed per-benchmark results are provided in Appendix D. Method

LLaVA-1.5-7B (2023) 576 tokens

Compress Ratio Remain Token

↓77.8% ↓88.9% ↓94.4% ↓77.8% ↓88.9% ↓94.4% ↓77.8% ↓88.9% ↓94.4% ↓77.8% ↓88.9% ↓94.4% 128 64 32 128 64 32 Up. 640 Up. 320 Up. 160 Up. 640 Up. 320 Up. 160

SparseVLM (ICML25) 96.0% 86.2% – VisionZip (CVPR25) 96.8% 93.0% 86.8% DivPrune (CVPR25) 96.7% 93.7% 90.2% SCOPE (NeurIPS25) 97.8% 96.0% 93.5% TOPS (Ours) 98.3% 97.1% 94.6%

LLaVA-1.5-13B (2023) LLaVA-NeXT-7B (2024a) LLaVA-NeXT-13B (2024a) 576 tokens Upper(Up.) 2880 tokens Upper(Up.) 2880 tokens

98.2% 93.0% – 98.3% 93.2% 96.9% 93.2% 86.5% 99.4% 95.3% 96.8% 94.2% 90.5% 98.4% 96.0% 97.7% 96.4% 93.3% 99.8% 97.8% 98.9% 97.3% 94.7% 100.0% 99.1%

where α, λ are balance factors. We expand the subset until it reaches the target size K. The construction is initialized as S1 = {arg maxi ri }. Since coverage and diversity share the same maxk∈S sim(·, hk ) term, it is incrementally maintained, introducing negligible overhead. The complete algorithm is in Algorithm 2 in Appendix. 4.2

Multi-Stage TOPS Implementation

TOPS can be applied at multiple pruning points during MLLM inference. To fully exploit its flexibility, we implement a multi-stage pipeline for fine-grained pruning, as illustrated in Figure. 3. Stage I applies TOPS after multimodal projector, before visual tokens enter LLM. Let V (0) = g(fv (Xv )) denote the projected visual token set. Since text-rater attention is unavailable at this stage, we use CLS attention to replace it:   V (1) = TOPS V (0) , rcls . (8) where rcls denotes CLS-based relevance scores. This coarse reduction removes clearly redundant tokens before LLM processing. Stage II applies TOPS inside the LLM at a set of designated layers P = {p1 , . . . , pL }. At each layer pl ∈ P, for l = 1, . . . , L, we perform:   V (l+1) = TOPS V (l) , r(pl ) . (9) where r(pl ) denotes text-rater relevance at layer pl , enabling TOPS to leverage LLM’s text-to-visual attention at deeper layers for useful token selection.

5

Experiments

5.1

Experimental Setup

Model Architectures. We validate TOPS across multiple MLLM architectures, including LLaVA1.5 (Liu et al., 2024a) for image understanding, LLaVA-NeXT (Liu et al., 2024b) for highresolution inputs, and LLaVA-Video (Zhang et al.,

– 99.7% 96.4% – 89.3% 99.9% 96.4% 91.8% 92.4% 98.1% 96.3% 93.9% 94.4% 99.4% 98.2% 95.8% 96.4% 100.6% 99.1% 96.6%

2024b) for video tasks. We also evaluate on advanced models Qwen2.5-VL-7B-Instruct (Bai et al., 2025b) and InternVL3-8B (Zhu et al., 2025). More experiments are provided in Appendix D. Evaluation Benchmarks. We conduct experiments across diverse multimodal benchmarks. For image-based evaluation, we select 8 general VQA benchmarks: GQA (Hudson and Manning, 2019), ScienceQA-IMG (Lu et al., 2022), TextVQA (Singh et al., 2019), POPE (Li et al., 2023), MME (Fu et al., 2023), MMBench-EN, MMBench-CN (Liu et al., 2024d), and MMVet (Yu et al., 2023). Additionally, we evaluate on MMStar (Chen et al., 2024b), AI2D (Kembhavi et al., 2016), and HallusionBench (Guan et al., 2024). For video understanding, we benchmark on MLVU (Zhou et al., 2025), LongVideoBench (Wu et al., 2024), and Video-MME (Fu et al., 2025). Comparison Methods. We compare TOPS with recent methods, including FastV (Chen et al., 2024a), PyramidDrop (Xing et al., 2024), SparseVLM (Zhang et al., 2024a), DivPrune (Alvar et al., 2025), DART (Wen et al., 2025), VisionZip (Yang et al., 2025b), TRIM (Song et al., 2025), PruMerge+ (Shang et al., 2025), SCOPE (Deng et al., 2025), and CDPruner (Zhang et al., 2025d). 5.2

TOPS for LLaVA and LLaVA-NeXT

We first evaluate TOPS on LLaVA-1.5 and LLaVANeXT, widely adopted for benchmarking token pruning (Table 1; full per-benchmark results in Appendix D.). On LLaVA-1.5-7B, TOPS retains 98.3% of the original performance at 77.8% compression, surpassing SCOPE by 0.5%. At 64 tokens, attention-based methods degrade by over 25%, while TOPS only decreases by 1.2%. Even at 32 tokens (5.6% retained), TOPS maintains 94.6%,

Table 2: Performance comparison of different pruning methods on advanced VLM architectures across 8 benchmarks. Acc. denotes the average percentage of baseline performance maintained. Red : attention-based. Green : attention&diversity. Blue : diversity-based. Purple : ours.

Method

AI2D POPE Hall MME MMBEN MMBCN MMStar SQA Acc.

Rel.

Qwen2.5-VL-7B-Instruct — Upper Bound, All 1296 Tokens (100%) Baseline

84.9

87.7 55.9 2301.8

84.8

82.9

65.5

86.8 73.1 100.0%

55.5 57.7 57.9 59.5

83.6 67.8 92.7% 84.1 68.9 94.3% 84.8 68.8 94.1% 87.2 70.4 96.3%

43.7 53.8 52.3 53.9

79.6 57.0 78.0% 82.5 65.5 89.6% 82.8 65.5 89.6% 85.3 67.9 92.9%

Retain 256 Tokens (↓ 80.2%) FastV (ECCV24) 78.4 CDPruner (NeurIPS25) 82.2 DivPrune (CVPR25) 81.2 TOPS (Ours) 81.5

83.0 83.5 85.3 86.1

49.1 2169.3 45.3 2231.9 46.6 2167.3 50.3 2284.6

80.5 81.5 81.8 81.4

78.8 80.1 80.9 80.5

Retain 128 Tokens (↓ 90.1%) FastV (ECCV24) 69.9 CDPruner (NeurIPS25) 79.5 DivPrune (CVPR25) 75.9 TOPS (Ours) 78.9

67.7 80.5 83.9 83.9

41.0 1596.8 41.3 2033.2 44.5 2044.3 46.8 2217.3

66.4 78.6 79.2 80.2

68.8 78.7 78.6 78.8

InternVL3-8B — Upper Bound, All 1280 Tokens (100%) Baseline

85.1

90.4 49.4 2369.1

85.7

85.1

68.3

97.9 74.9 100.0%

61.2 55.7 55.1 58.9 62.8

93.3 71.2 95.1% 90.2 67.4 90.0% 90.7 67.8 90.5% 91.8 69.3 92.5% 95.5 72.4 96.7%

47.3 51.5 49.2 52.0 57.1

83.8 60.4 80.6% 85.6 62.8 83.8% 81.7 60.3 80.5% 87.5 64.6 86.2% 91.8 68.8 91.9%

Retain 256 Tokens (↓ 80.0%) FastV (ECCV24) 80.5 CDPruner (NeurIPS25) 78.8 VisionZip (CVPR25) 76.0 DivPrune (CVPR25) 80.3 TOPS (Ours) 81.3

88.7 89.1 85.6 89.4 89.3

44.0 2289.4 41.5 2130.0 41.0 2148.8 43.0 2178.5 45.8 2302.0

83.6 79.2 82.0 81.9 84.8

83.7 78.3 81.2 80.5 84.6

Retain 128 Tokens (↓ 90.0%) FastV (ECCV24) 68.4 CDPruner (NeurIPS25) 73.1 VisionZip (CVPR25) 68.5 DivPrune (CVPR25) 74.2 TOPS (Ours) 76.1

73.8 86.9 77.9 88.0 87.5

38.9 1806.9 37.4 1952.6 33.6 1864.7 37.6 2051.0 41.4 2252.0

outperforming SCOPE by 1.1%. On LLaVA-NeXT with 2,880 visual tokens, TOPS achieves 100.0% performance at 77.8% compression, and retains 99.1% and 96.4% at 88.9% and 94.4% reduction, outperforming SCOPE by 1.3% and 2.0%. 5.3

TOPS for Qwen2.5-VL and InternVL3

To verify generalizability, we further evaluate TOPS on Qwen2.5-VL-7B-Instruct and InternVL38B (Table 2), two architectures with different visual encoders and fusion strategies.At ∼80% pruning, TOPS preserves 96.3% and 96.7% of the original performance, surpassing the best competing methods by 2.0% and 1.6%, respectively. At ∼90% pruning, TOPS retains 92.9% and 91.9% accuracy

73.7 75.3 74.7 78.3 82.0

73.5 73.5 73.5 75.7 80.5

while its advantage amplifies, reaching +3.3% over CDPruner/DivPrune on Qwen2.5-VL and +5.7% over DivPrune on InternVL3. Notably, the best baseline differs across architectures (CDPruner on Qwen2.5-VL vs. FastV on InternVL3), yet TOPS consistently ranks first, demonstrating architectureagnostic effectiveness. TOPS achieves the best scores across all settings On HallusionBench, indicating stronger resistance to hallucination. 5.4

TOPS for LLaVA-Video

Video understanding is highly redundant because multi-frame inputs introduce many visual tokens. We apply TOPS to LLaVA-Video with up to 64 frames at 384×384 resolution, producing over 10K

Table 3: Performance comparison of different methods on LLaVA-Video-7B with 64 frames per video. Acc. denotes average accuracy across 8 metrics of 3 benchmarks. Red : attention-based. Blue : diversity-based. Purple : ours.

Method Metric

MLVU LongVideoBench Video-MME Acc. m-avg val perception relation w/o sub short medium long

Rel.

Upper Bound, All 64 × 169 Tokens (100%) Baseline

67.7

59.0

65.0

53.8

63.6

76.6

61.2

53.1 62.5 100.0%

73.6 73.0 73.0 72.9 75.3

59.3 58.8 59.9 59.3 61.4

52.7 60.0 96.0% 51.2 59.8 95.7% 51.9 60.5 96.8% 51.2 60.6 97.0% 52.3 61.5 98.4%

63.8 69.8 67.3 69.9 72.3

55.9 56.9 57.1 57.9 58.8

48.4 55.1 88.2% 50.3 57.3 91.7% 50.0 57.0 91.2% 50.2 58.6 93.8% 52.6 60.1 96.2%

55.0 53.8 64.8 67.7 70.9

50.0 49.3 52.9 54.2 57.4

45.0 49.1 78.6% 46.3 49.3 78.9% 48.1 54.2 86.7% 48.2 55.3 88.5% 49.8 57.7 92.3%

Retain 64 × 64 Tokens (↓ 62.1%) FastV (ECCV24) SparseVLM (ICML25) DART (EMNLP25) DivPrune (CVPR25) TOPS (Ours)

63.9 65.5 64.1 64.1 66.4

56.1 56.0 57.5 58.6 57.6

60.6 61.0 62.1 64.2 64.6

52.1 51.7 53.5 53.7 51.4

61.9 61.0 61.6 61.1 63.0

Retain 64 × 32 Tokens (↓ 81.1%) FastV (ECCV24) SparseVLM (ICML25) DART (EMNLP25) DivPrune (CVPR25) TOPS (Ours)

58.5 60.7 61.1 61.5 64.3

52.4 53.7 54.1 56.4 56.9

57.0 58.1 57.8 62.1 63.5

48.5 49.9 50.8 51.4 51.1

56.0 59.0 58.1 59.3 61.2

Retain 64 × 16 Tokens (↓ 90.5%) FastV (ECCV24) SparseVLM (ICML25) DART (EMNLP25) DivPrune (CVPR25) TOPS (Ours)

52.8 52.0 56.7 58.6 60.7

46.6 47.6 51.8 52.1 54.3

48.8 53.0 56.8 57.6 60.8

44.7 42.8 47.5 47.2 48.6

Table 4: Efficiency analysis on LLaVA-NeXT-7B (POPE benchmark). Latency in ms; Memory in GB. Method

#Tok FLOPs (T) Lat. (ms) Mem. (GB) F1

Baseline

2880

41.7

265

16.7

86.8

FastV (ECCV24) PDrop (CVPR25) SparseVLM (ICML25) PruMerge+ (ICCV25) VisionZip (CVPR25) TOPS (Ours)

320 320 320 320 320 320

4.4 (×9.5) 4.4 (×9.4) 4.4 (×9.5) 4.2 (×9.9) 4.2 (×9.9) 4.2 (×9.9)

77 67 101 54 60 85

15.6 15.6 18.6 14.8 14.8 14.8

49.5 60.8 85.3 79.5 82.3 86.3

visual tokens (Table 3). TOPS remains robust under aggressive compression, retaining 98.4% and 96.2% performance at 62.1% and 81.1% token reduction, respectively. Even with only 16 tokens per frame, TOPS maintains 92.3%, while FastV drops to 78.6%. 5.5

Computational Efficiency

To demonstrate the efficiency of TOPS, we conduct a comparative analysis against other methods in terms of FLOPs, CUDA latency, GPU memory and F1 score on LLaVA-NeXT-7B. Experiments are performed on a single NVIDIA A800-80GB GPU. We use POPE for evaluating inference efficiency, as

50.0 49.8 55.3 56.7 59.4

it contains questions of similar length and involves only one prefill and one decode stage. As shown in Table 4, when the number of visual tokens is reduced from 2,880 to 320, TOPS achieves nearly a ×10 reduction in FLOPs. In terms of runtime latency, TOPS reduces prefill time and decode time by ×3.12 and ×1.05, significantly improving real-world inference efficiency. In addition to latency, TOPS reduces GPU memory usage by 1.9GB. Compared to other methods, TOPS achieves the best performance (86.3 vs. 85.3) while maintaining comparable or even better efficiency. 5.6

Ablation Studies

We conduct a series of ablation studies to analyze the key design choices of TOPS on LLaVA-1.57B. Table 6 examines contribution of each stage in the two-stage pipeline, showing that TOPS’s full two-stage design yields the best results. We further evaluate all combinations of Relevance (R), Diversity (D), and Coverage (C) at 32 tokens (Table 5), where TOPS achieves the best performance. As shown in Figure 5, TOPS consistently out-

Table 5: Ablation of token selection criteria.

GQA SQA TVQA POPE MME MMBEN MMBCN MMVet Acc.

Criteria

Rel.

Relevance only 53.5 Diversity only 55.0 Coverage only 56.6

69.2 67.5 68.9

53.9 53.1 51.4

77.5 84.7 83.2

1347.4 1355.4 1358.1

60.6 58.1 59.5

54.9 52.4 51.6

24.5 27.5 24.8

57.5 91.1% 58.3 92.4% 58.0 91.9%

R+D R+C D+C

55.6 56.4 55.7

69.0 69.1 68.4

54.4 54.7 53.0

81.7 82.1 84.4

1365.0 1371.6 1368.6

60.2 61.0 58.7

55.2 55.6 52.4

27.8 28.0 26.2

59.0 93.5% 59.4 94.1% 58.4 92.6%

TOPS

56.7

68.8

54.9

83.5

1384.7

59.5

55.1

29.7

59.7 94.6%

VQAText

53.6

7 71.2 3.8

68.6

72.57

60.9

0.5

1.0

1.5

.7 65 8.1 6

2.0

51.22

.1

0.5

1.0

1.5

58.1

.9 42 15 4.2 160 665.5 6.8 1 72 1

1411.9

60.7

2.0

1354.9 1297.9

62.81

63.3

60.36

.6

53.47

65.25

1469.0

68

55.97

59

55.71

67.69

1526.0

13

58.31

70.13

1583.1

148 1.7

57.95

66.0

.9

60.66

.3

60.19

1542

60.9

56.0

58.4

63.01

63

.2 66

2.0

62.43

1640.1

66.0 .6 68

1.5

65.35

53.62

77.45 75.01

1697.2

2.9

1.0

.7 68 1.3 7

1754.2

66.92 64.68

.9 60 .3 63

0.5

63.7

.4

154

61.2 .7 63 .2 66

0.5

56.1

1 1 726 160 665.5 .8 4.2

6 63.3 65 8.1 .7

66.2 63.7 58.7

1.0

20

1811.3

.0 66

67.70

14

MMBench 1.7

69.16

70.04

1.5

71.40

148

72.39

MME .9

74.73

60

.7

63

7 68.7 1.3

GQA 2.0

0.5

1.0

66.0

.2 71 3.8 7

1.5

2.0

57.92 55.48

Figure 4: Hyperparameter sensitivity of α and λ. Contour plots across seven (α, λ) configurations at 64 tokens on LLaVA-1.5-7B. Star: optimal; white dots: other configurations. The optimal (α, λ) generally falls within [0.5, 1]. Table 6: Ablation of two stages.

Setting GQA POPE MME MMBEN MMBCN S1-only 59.2 S2-only 59.1

86.2 86.5

1444.0 1412.6

61.1 61.7

56.5 56.0

TOPS

86.8

1482.7

62.5

57.2

60.5

performs FastV, DivPrune, and SCOPE across all five token budgets, and its advantage widens under more aggressive compression. Figure 4 visualizes the sensitivity of α (diversity weight) and λ (coverage weight) across 8 benchmarks at 64 tokens. Across seven (α, λ) configurations, the optimal values consistently fall within [0.5, 1] for both parameters, with the exception of POPE, where a larger α=2 is preferred due to its binary question format that benefits from stronger diversity. The overall performance variation remains within 1–2% across all configurations, confirming that TOPS is robust to hyperparameter choices. Additional ablations on pruning layer positions are provided in Appendix E.3.

6

Conclusion

In this work, we revisit visual token pruning from a first-principles perspective and identify its fundamental objective. Based on an informationtheoretic analysis, we derive three key principles— task relevance, information coverage, and semantic diversity—and propose TOPS, a plug-and-play

Figure 5: Robustness across token budgets. Relative performance (%) of FastV, DivPrune, SCOPE and TOPS at five budgets on LLaVA-1.5-7B.

pruning method that constructs compact yet informative token subsets. Extensive experiments across multiple MLLMs and benchmarks demonstrate that TOPS consistently achieves superior performance under aggressive token reduction while maintaining strong generalization across model architectures and tasks. These results show that effective token pruning should not rely solely on heuristic importance scores, but should jointly preserve taskrelevant, representative, and non-redundant visual information. We believe that our work provides a principled foundation and offers useful guidance for future research on token pruning and efficient multimodal inference.

Limitations

References

While TOPS demonstrates consistent improvements across diverse LVLMs and benchmarks, several limitations remain.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Greedy construction overhead. TOPS constructs the preserved token set greedily, updating diversity and coverage scores after each selection step. Although both scores share the same maxsimilarity structure and can be maintained incrementally, the greedy loop still introduces O(KN ) additional operations per pruning layer (where K is the target budget and N is the current token count), which is non-negligible at very aggressive budgets or when pruning is applied at many layers. Future work could explore approximate or parallel variants to further reduce this overhead. Attention-based relevance signal. The task relevance score in TOPS relies on cross-modal attention weights as a proxy for query-conditioned importance. While this signal is readily available in standard transformer architectures, it may be less reliable in models that use alternative attention mechanisms (e.g., linear attention or sparse attention), or in very early layers where text-visual attention has not yet matured. The Stage I relevance estimation similarly relies on CLS attention, which may not generalize equally well to vision encoders that lack a dedicated CLS token. Fixed hyperparameters across layers and tasks. The balance coefficients α (diversity) and λ (coverage) are set globally and kept fixed across all pruning layers and all tasks. In practice, the optimal trade-off between relevance, coverage, and diversity may vary with pruning depth, token budget, and task type. An adaptive scheme that adjusts these weights per layer or per query could further improve performance, particularly under extreme compression ratios. Evaluation scope. Our experiments focus on standard vision-language benchmarks covering image understanding, video understanding, and OCRheavy tasks. Performance in highly specialized domains (e.g., medical imaging, remote sensing, or dense captioning with hundreds of objects) has not been systematically evaluated, and the generalization of the three-criterion framework to such settings remains an open question.

Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. 2025. Divprune: Diversitybased visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9392–9401. Changwoo Baek, Jouwon Song, Sohyeon Kim, and Kyeongbo Kong. 2026. Agilepruner: An empirical study of attention and diversity for adaptive visual token pruning in large vision-language models. arXiv preprint arXiv:2603.01236. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023a. Qwen technical report. arXiv preprint arXiv:2309.16609. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 1(2):3. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025a. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, and 1 others. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2022. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. Jiajun Cao, Qizhe Zhang, Peidong Jia, Xuhui Zhao, Bo Lan, Xiaoan Zhang, Xiaobao Wei, Sixiang Chen,

Liyun Li, Xianming Liu, and 1 others. 2026. Fastdrivevla: Efficient end-to-end driving via plug-andplay reconstruction-based token pruning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 2571–2579.

Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14375–14385.

Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024a. An image is worth 1/2 tokens after layer 2: Plug-andplay inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer.

Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and 1 others. 2024b. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056–27087.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024c. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Jinhong Deng, Wen Li, Joey Tianyi Zhou, and Yang He. 2025. Scope: Saliency-coverage oriented token pruning for efficient multimodel llms. arXiv preprint arXiv:2510.24214. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and 1 others. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, and 1 others. 2025. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24108–24118. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, and 1 others. 2024.

Yizhang Jin, Jian Li, Tianjun Gu, Yexin Liu, Bo Zhao, Jinxiang Lai, Zhenye Gan, Yabiao Wang, Chengjie Wang, Xin Tan, and 1 others. 2025. Efficient multimodal large language models: A survey. Visual Intelligence, 3(1):27. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In European conference on computer vision, pages 235–251. Springer. Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, and 1 others. 2023. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024. Llavaonevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 292– 305. Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. 2022. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 5971–5984.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. Llavanext: Improved reasoning, ocr, and world knowledge. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916. Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. 2024c. Multistage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.10803. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, and 1 others. 2024d. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, KaiWei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in neural information processing systems, 35:2507–2521. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277. Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949. Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2025. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22857–22867. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, and 1 others. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326.

Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael X Guan, and Benyou Wang. 2025. Less is more: A simple yet effective token reduction method for efficient multi-modal llms. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7614–7623. Yifan Tan, Yifu Sun, Shirui Huang, Hong Liu, Guanghua Yu, Jianchen Zhu, and Yangdong Deng. 2026. Idpruner: Harmonizing importance and diversity in visual token pruning for mllms. arXiv preprint arXiv:2602.13315. Gemini Team, Rohan Anil, Sebastian Borgeaud, JeanBaptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, and 1 others. 2026. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276. Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, and 1 others. 2025. Kimi-vl technical report. arXiv preprint arXiv:2504.07491. Qwen Team. 2026. Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Yahong Wang, Juncheng Wu, Zhangkai Ni, Chengmei Yang, Yihang Liu, Longzhen Yang, Yuyin Zhou, Ying Wen, and Lianghua He. 2026. Entropyprune: Matrix entropy guided visual token pruning for multimodal large language models. arXiv preprint arXiv:2602.17196. Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. 2025. Stop looking for important

tokens in multimodal language models: Duplication matters more. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9972–9991. Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. Longvideobench: A benchmark for longcontext interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828–28857. Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and 1 others. 2024. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2025b. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19792–19802. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490. Ce Zhang, Kaixin Ma, Tianqing Fang, Wenhao Yu, Hongming Zhang, Zhisong Zhang, Yaqi Xie, Katia Sycara, Haitao Mi, and Dong Yu. 2025a. Vscan: Rethinking visual token reduction for efficient large vision-language models. arXiv preprint arXiv:2505.22654. Hao Zhang, Mengsi Lyu, Chenrui He, Yulong Ao, and Yonghua Lin. 2025b. Towards adaptive visual token pruning for large multimodal models. arXiv e-prints, pages arXiv–2509. Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. 2025c. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20857–20867.

Qizhe Zhang, Mengzhen Liu, Lichen Li, Ming Lu, Yuan Zhang, Junwen Pan, Qi She, and Shanghang Zhang. 2025d. Beyond attention or similarity: Maximizing conditional diversity for token pruning in mllms. arXiv preprint arXiv:2506.10967. Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and 1 others. 2024a. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417. Yuan Zhang, Junpeng Ma, Qizhe Zhang, Chun-Kai Fan, Wenzhao Zheng, Kuan Cheng, Jiwen Lu, and Shanghang Zhang. Sparsevlm+: Visual token sparsification with improved text-visual attention pattern. Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024b. Llava-video: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713. Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, and 1 others. 2025. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13691– 13701. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, and 1 others. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479.

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference Appendix Appendix A provides comprehensive details about model architectures, evaluation benchmarks, and baseline methods. Appendix B presents the complete TOPS algorithm and theoretical foundations. Appendix C describes implementation details including hyperparameters and pruning schedules. Appendix D reports additional results on LLaVA1.5-7B, LLaVA-NeXT-7B, LLaVA-1.5-13B, and LLaVA-NeXT-13B. Appendix E provides additional ablation studies. Appendix F presents additional empirical analyses supporting the motivation of TOPS. Appendix G presents qualitative comparisons of token selections. Appendix H provides per-benchmark radar visualizations across all models and compression ratios.

A

Experimental Setup Details

A.1

Model Architectures

LLaVA-1.5 (Liu et al., 2024a). We evaluate the LLaVA-1.5 architecture, which combines a CLIP ViT-L/14 vision encoder with Vicuna-7B/13B language models through a two-layer MLP projector. For our experiments, we process images at 336 × 336 resolution, yielding 576 visual tokens (24 × 24 spatial grid). We conduct experiments on both 7B and 13B model scales. LLaVA-NeXT (Liu et al., 2024b). This iteration introduces adaptive resolution handling to accommodate higher-quality visual inputs. The model dynamically partitions high-resolution images into multiple tiles and processes each tile through the vision encoder independently. For controlled evaluation, we standardize the input to 672 × 672 resolution, generating 2,880 visual tokens. LLaVA-Video (Zhang et al., 2024b). We extend our evaluation to this video-specialized variant that processes temporal sequences of frames. The architecture employs SigLIP as the vision backbone and samples 64 frames per video clip at 384 × 384 resolution, producing an initial token count of 10,816 visual tokens. InternVL3-8B (Zhu et al., 2025). We evaluate InternVL3, adopting a ViT-MLP-LLM design

with native multimodal pre-training. For our experiments, we configure the input resolution to 448 × 448, producing 1,280 visual tokens. Qwen2.5-VL-7B-Instruct (Bai et al., 2025b). We assess Qwen2.5-VL, featuring a redesigned Vision Transformer with window attention, SwiGLU activations, and RMSNorm, built upon the Qwen2.5 language model. We evaluate the 7BInstruct variant using its default dynamic resolution settings. A.2

Evaluation Benchmarks

GQA (Hudson and Manning, 2019). A visual reasoning benchmark grounded in scene graphs. We report accuracy on the balanced test split with 12,578 questions. ScienceQA-IMG (Lu et al., 2022). A multimodal multiple-choice benchmark covering diverse scientific subjects. TextVQA (Singh et al., 2019). Evaluates reading and reasoning about text in images. The validation set has 5,000 questions. POPE (Li et al., 2023). Polling-based Object Probing Evaluation assesses object hallucination using binary yes/no questions. We use the adversarial split with 3,000 questions. MME (Fu et al., 2023). A comprehensive benchmark evaluating both perception and cognition abilities, containing 14 subtasks. MMBench (Liu et al., 2024d). A systematicallydesigned benchmark covering 20 ability dimensions. We report results on both English and Chinese test sets. MM-Vet (Yu et al., 2023). Focuses on integrated multimodal capabilities across 6 core abilities with 218 curated examples. MMStar (Chen et al., 2024b). A comprehensive vision-language benchmark evaluating diverse capabilities including coarse and fine-grained perception. AI2D (Kembhavi et al., 2016). A diagram QA benchmark with over 5,000 science diagrams and 15,000 questions. HallusionBench (Guan et al., 2024). Evaluates vision-language models’ susceptibility to language hallucination and visual illusion.

MLVU (Zhou et al., 2025). A multi-task long video understanding benchmark. We report the mean average score (m-avg) across all subtasks. LongVideoBench (Wu et al., 2024). A longcontext video-language benchmark with interleaved video-language inputs of up to one hour. Video-MME (Fu et al., 2025). A comprehensive video multimodal evaluation benchmark. We report results in the no-subtitle setting across short, medium, and long video splits. A.3

Baseline Methods

We compare TOPS against 10 recent training-free visual token pruning methods. FastV (Chen et al., 2024a). An in-LLM pruning method that ranks visual tokens by text-to-visual cross-modal attention scores at layer 2 and removes those with the lowest scores. PyramidDrop (Xing et al., 2024). A progressive in-LLM method that drops a fixed fraction of visual tokens at the end of each decoder stage based on attention importance. SparseVLM (Zhang et al., 2024a). A text-aware in-LLM method that identifies high-quality text rater tokens and uses their cross-modal attention patterns to guide visual token sparsification. PruMerge+ (Shang et al., 2025). A pre-LLM pruning-and-merging method that computes token importance via attention sparsity and merges similar tokens using k-nearest-neighbor matching. TRIM (Song et al., 2025). A pre-LLM method that leverages CLIP-based vision-text similarity to score and retain visual tokens with the highest cross-modal relevance. VisionZip (Yang et al., 2025b). A pre-LLM method that selects dominant tokens based on CLSto-patch attention scores and additionally identifies contextual tokens through clustering. DART (Wen et al., 2025). A diversity-based inLLM method that iteratively selects the most diverse tokens by choosing candidates with the lowest similarity to already-selected ones. DivPrune (Alvar et al., 2025). A diversity-based pre-LLM method that reformulates token selection as a max-min diversity problem (MMDP).

SCOPE (Deng et al., 2025). A hybrid method combining CLS-based saliency scoring with a submodular coverage term that penalizes semantically redundant tokens. CDPruner (Zhang et al., 2025d). A conditionaldiversity method that defines similarity between visual tokens conditioned on the user instruction.

B

Algorithm and Theoretical Analysis

B.1

Complete Algorithm

Algorithm 1 provides the vision-side pruning procedure of TOPS, which is applied immediately after the multimodal projector and before the LLM. Algorithm 2 provides the in-LLM pruning procedure at a designated pruning layer, which is applied progressively across selected LLM layers. B.2

Theoretical Analysis of Coverage and Diversity

We formally establish the theoretical properties of the coverage and diversity criteria used in TOPS. Proposition 1 (Submodularity of Coverage). The coverage function X F (S) = max sim(vi , s) vi ∈V

s∈S

is submodular. Proof. For any A ⊆ B ⊆ V and x ∈ V \ B, define mA (vi ) = max sim(vi , a), a∈A

mB (vi ) = max sim(vi , b). b∈B

Since A ⊆ B, we have mA (vi ) ≤ mB (vi ) for every vi ∈ V . The marginal gain of adding x to A for token vi is
∆x (vi | A) = max sim(vi , x) − mA (vi ), 0 , and likewise
∆x (vi | B) = max sim(vi , x) − mB (vi ), 0 . Since mA (vi ) ≤ mB (vi ), it follows that ∆x (vi | A) ≥ ∆x (vi | B) for every vi ∈ V . Summing over all tokens: F (A ∪ {x}) − F (A) ≥ F (B ∪ {x}) − F (B), which is the submodularity condition. Remark 1 (Diminishing Marginal Contribution of Diversity). The diversity score di (S) = 1 − maxj∈S sim(hi , hj ) satisfies the following diminishing marginal contribution property: for any S ⊆ T and i ∈ / T , di (S) ≥ di (T ). Since S ⊆

T , maxj∈T sim(hi , hj ) ≥ maxj∈S sim(hi , hj ), hence di (T ) = 1 − maxj∈T sim(hi , hj ) ≤ di (S). This guarantees that as the retained set grows, each new token contributes progressively less diversity, providing the same intuitive justification for greedy construction as submodularity does for coverage. Algorithm 1 TOPS Stage I — Vision-Side Token Pruning (0)

Require: Projected visual tokens V (0) = {hi }N i=1 , CLS-to-patch attention scores ai , target budget M0 , balance factors α1 , λ1 Ensure: Coarsely pruned visual token set V (1) 1: Precompute similarity matrix F(0) from (0) projected visual tokens {hi }N i=1 2: S ← {i0 } where i0 = arg maxi ai {seed: highest vision-side relevance} 3: Initialize maxsimj ← F(0) [j, i0 ] for all j ∈ /S 4: while |S| < M0 do 5: for each token i ∈ V (0) \ S do (0) 6: divi ← 1 − maxj∈S F(0) [i, j] X (0) 7: covi ← (0)

j∈V \S
max 0, F(0) [i, j] − maxk∈S F(0) [j, k]

8: 9:

end for Normalize each criterion by its mean: (0) f (0) , cg ãi , div ov i

through validation on a small held-out subset of POPE, MME, and GQA. Table 7 reports the (α, λ) pairs used for LLaVA-1.5 and LLaVA-NeXT at each pruning ratio. Across all settings, α = 0.5 is kept fixed, while λ varies slightly between 0.4 and 1.0 depending on the compression level. For Stage I we uniformly use α1 = 0.5, λ1 = 0.5 across all models. C.3 Pruning Schedule For all models, Stage I applies TOPS immediately after the multimodal projector to reduce the initial token count before entering the LLM; Stage II then applies two successive TOPS passes at designated LLM layers to reach the final budget. C.4 Hardware and Evaluation Protocol All experiments are conducted on NVIDIA A80080GB GPUs. Inference is performed with batch size 1 to ensure fair latency comparison across methods. We follow the standard evaluation protocol for each benchmark, using greedy decoding without sampling for generative tasks.

i

10:

(0) f (0) scorei ← ãi + α1 div i (0) +λ1 cg ovi

11: 12: 13:

i⋆ ← arg maxi∈V (0) \S scorei

(0)

⋆

S ← S ∪ {i } Update maxsimj ← max(maxsimj , F(0) [j, i⋆ ]) for all j ∈ /S 14: end while 15: V (1) ← S 16: return V (1)

Algorithm 2 TOPS Stage II — In-LLM LayerWise Token Pruning (l)

Require: Hidden states {hi } at pruning layer l, attention weights A(l) , current visual tokens V (l) , target budget Ml , balance factors α2 , λ2 Ensure: Updated visual token set V (l+1) 1: Compute dynamic text rater set Q(l) via text-visual relevance thresholding

2: Compute text-guided relevance ri(l) for all i ∈ V (l)

C

Implementation Details

C.1

Codebase

We implement TOPS on top of the official LLaVA codebase1 for image-based LLaVA models (LLaVA-1.5-7B and 13B). For LLaVA-NeXT and its high-resolution variants, we build on the LLaVANeXT codebase2 . For LLaVA-Video, we adopt the same LLaVA-NeXT codebase (Zhang et al., 2024b) and use lmms-eval3 for video benchmark evaluation. For advanced architectures (Qwen2.5-VL and InternVL3), we integrate TOPS via VLMEvalKit4 to enable unified evaluation across all benchmarks. C.2

Hyperparameters

Unless otherwise specified, we set the balance factors α (diversity weight) and λ (coverage weight) 1

https://github.com/haotian-liu/LLaVA https://github.com/LLaVA-VL/LLaVA-NeXT 3 https://github.com/EvolvingLMMs-Lab/ lmms-eval 4 https://github.com/open-compass/VLMEvalKit 2

using Q(l) 3: Precompute similarity matrix F(l) from (l) {hi }i∈V (l)

4: S ← {i0 } where i0 = arg maxi ri(l) {seed: highest task relevance}

5: Initialize maxsimj ← F(l) [j, i0 ] for all j ∈ /S

6: while |S| < Ml do 7: for each token i ∈ V (l) \ S do (l) 8: divi ← 1 − maxj∈S F(l) [i, j] X (l) 9: covi ← (l)

j∈V \S
max 0, F(l) [i, j] − maxk∈S F(l) [j, k]

10: 11:

end for Normalize each criterion by its mean: (l) (l) f (l) , cg r̃ , div ov

12:

f scorei ← r̃i + α2 div i (l) +λ2 cg ovi

13: 14: 15:

i⋆ ← arg maxi∈V (l) \S scorei

i

i

(l)

i

(l)

(l)

(l)

S ← S ∪ {i⋆ } Update maxsimj ← max(maxsimj , F(l) [j, i⋆ ]) for all j ∈ /S 16: end while 17: Rebuild hidden sequence: (l) H (l) ← [Hsys ; {hi }i∈S ; Htext ] 18: Rebuild attention_mask: set 1 for retained positions and 0 for pruned visual positions 19: Rebuild position_ids: re-index retained positions contiguously from 0 20: V (l+1) ← S 21: return V (l+1)

Table 7: Hyperparameter settings (α, λ) used for each model and pruning ratio. α is the diversity weight and λ is the coverage weight.

Pruning Ratio 77.8% 88.9% 94.4%

LLaVA-1.5-7B

LLaVA-1.5-13B

LLaVA-NeXT-7B

LLaVA-NeXT-13B

(0.5, 0.5) (0.5, 1.0) (0.5, 1.0)

(0.5, 0.4) (0.5, 0.4) (0.5, 0.4)

(0.5, 0.5) (0.5, 0.4) (0.5, 0.5)

(0.5, 0.5) (0.5, 0.5) (0.5, 0.5)

Table 8: Hyperparameter settings (α, λ) for Qwen2.5-VL-7B, InternVL3-8B, and LLaVA-Video.

Qwen2.5-VL-7B

LLaVA-Video-7B

InternVL3-8B

Pruning Ratio

(α, λ)

80.2% 90.1%

(0.5, 0.4) 80.0% (0.5, 0.4) 90.0%

Pruning Ratio

(α, λ)

Pruning Ratio

(α, λ)

62.1% (0.2, 0.2) 81.1% (0.3, 0.2) 90.5%

(0.5, 0.5) (0.5, 0.5) (0.5, 0.5)

Table 9: TOPS pruning schedule for LLaVA-1.5 and LLaVA-NeXT at three token budgets (T). Stage I reduces visual tokens before the LLM to 2T ; Stage II applies two successive TOPS passes at designated LLM layers to reach the final token count. Target T

Stage I

Stage II Layers

Stage II Budgets

LLaVA-1.5-7B

128 64 32

576 → 256 576 → 128 576 → 64

(L12, L24) (L12, L24) (L12, L24)

(256 → 128, 128 → 32) (128 → 64, 64 → 16) (64 → 32, 32 → 8)

LLaVA-1.5-13B

128 64 32

576 → 256 576 → 128 576 → 64

(L15, L30) (L15, L30) (L15, L30)

(256 → 128, 128 → 32) (128 → 64, 64 → 16) (64 → 32, 32 → 8)

LLaVA-NeXT-7B

640 320 160

2880 → 1280 2880 → 640 2880 → 320

(L12, L24) (L12, L24) (L12, L24)

(1280 → 640, 640 → 160) (640 → 320, 320 → 80) (320 → 160, 160 → 40)

LLaVA-NeXT-13B

640 320 160

2880 → 1280 2880 → 640 2880 → 320

(L15, L30) (L15, L30) (L15, L30)

(1280 → 640, 640 → 160) (640 → 320, 320 → 80) (320 → 160, 160 → 40)

Model

Table 10: TOPS pruning schedule for Qwen2.5-VL-7B (initial: 1296 tokens) and InternVL3-8B (initial: 1280 tokens). Stage I reduces tokens before the LLM; Stage II applies two successive TOPS passes at designated LLM layers. Model

Stage I

Stage II Layers

Stage II Budgets

Qwen2.5-VL-7B

1296 → 512 1296 → 256 1296 → 128

(L12, L16) (L12, L16) (L12, L16)

(512 → 281, 281 → 77) (256 → 139, 139 → 39) (128 → 71, 71 → 19)

InternVL3-8B

1280 → 512 1280 → 256 1280 → 128

(L12, L16) (L12, L16) (L12, L16)

(512 → 281, 281 → 77) (256 → 139, 139 → 39) (128 → 71, 71 → 19)

D

Experiments on More MLLMs

To verify that TOPS generalizes across model scales, we additionally evaluate it on LLaVA-1.57B/13B and LLaVA-NeXT-7B/13B. As shown in Tables 11–14, TOPS consistently outperforms all baselines across three compression levels.

E

Additional Ablation Studies

E.1

Hyperparameter Sensitivity

Table 15 analyzes sensitivity to α (diversity weight) and λ (coverage weight). Even a small coverage weight (λ ≈ 0.1) provides consistent improvements, while a moderate diversity weight (α ≈ 0.5) yields the most stable results.

E.2

Effect of Dynamic Text Rater

Table 16 compares different text rater strategies across three pruning ratios. The dynamic rater consistently outperforms last_token and all_mean by focusing on text tokens most engaged with visual information at each layer.

Table 16: Ablation of text rater strategy. Pruning 77.8% Strategy

MME

Pruning 88.9%

MMBench GQA

MME

MMBench GQA

all_mean 1480.5 last_token 1488.9

62.4 62.4

60.3 60.2

1441.7 1401.8

61.2 61.0

58.7 58.4

Ours

62.5

60.5

1442.7

60.9

58.7

1482.7

F

Additional Empirical Study

F.1

Logit Fidelity on an Additional Dataset

To further validate the empirical observations reported in Section 3.3, we extend the logit fidelity analysis to TextVQA. As shown in Figure 7, we measure ∆L = Lpruned − Lvanilla across token budgets of 128, 64, and 32 on 200 TextVQA samples. The pattern closely mirrors that observed on MME: at low pruning ratios, relevance-based pruning (FastV) incurs smaller logit distortion; as the budget decreases, diversity and coverage methods exhibit lower degradation. Across all budgets, TOPS consistently achieves the smallest logit increase, confirming that the complementary advantage of combining all three principles generalizes across datasets. F.2

A key motivation for TOPS’s multi-stage progressive pruning design is that token importance varies substantially across LLM layers. Figure 8 measures the mean Jaccard similarity between the top-R=128 token sets selected independently at each pair of layers, computed over 1000 POPE samples on LLaVA-1.5-7B. Near-zero offdiagonal similarities indicate that token selection is highly layer-dependent, motivating progressive multi-stage pruning rather than relying on a single fixed pruning layer. F.3

Effect of Pruning Layers

Table 17 varies the pruning layer positions. Middle layers {12, 24} offer the best balance: early enough for computational savings, yet late enough for sufficient text–visual interaction.

Token Selection Spatial Frequency

Figure 9 visualizes the spatial selection frequency heatmaps for representative baselines—FastV, DivPrune, DART, and SCOPE—averaged over 9000 POPE samples at a token budget of 128. FastV exhibits pronounced positional bias toward bottom rows due to attention shift in shallow LLM layers. Figure 10 further shows the per-token selection probability of TOPS across pruning stages, confirming that TOPS maintains spatially balanced token selection.

G E.3

Cross-Layer Token Selection Instability

Visualization of TOPS

Figure 11 compares TOPS against the Vanilla (unpruned) model. For each example, we show the original image, the token selection heatmap, and the generated answer. Under aggressive compression, TOPS focuses on task-relevant regions—text, key objects, and fine-grained details—while discarding redundant background.

Table 11: Performance comparison of different pruning methods on LLaVA-1.5-7B. Rel. denotes the ratio of pruned accuracy to baseline accuracy. Red: attention-based; Green: attention&diversity; Blue: diversity-based; Cyan: coverage-based; Purple: ours.

Method

GQA SQAIM G VQAText POPE MME MMBEN MMBCN MMVet Acc. Rel. Upper Bound: All 576 tokens (100%)

Baseline

61.9

69.5

58.2

85.9 1506.5

64.7

58.1

31.3

63.1 100.0%

63.0 62.3 62.6 61.8 63.0 62.1 60.7 61.5 62.7 62.5

55.9 55.3 56.9 55.8 52.3 57.0 57.3 54.8 57.0 57.2

27.0 27.6 29.7 30.4 29.9 31.6 30.9 30.6 31.3 30.0

57.8 91.6% 59.9 94.9% 60.6 96.0% 60.4 95.7% 60.1 95.2% 61.1 96.8% 60.4 95.7% 61.0 96.7% 61.7 97.8% 62.0 98.3%

50.1 48.0 58.3 59.6 60.9 60.1 59.5 60.1 61.0 60.9

42.1 36.6 49.6 52.1 48.2 55.4 54.0 52.3 56.0 56.5

18.9 17.7 24.4 28.0 24.8 29.4 26.5 28.1 30.5 30.6

45.4 71.9% 44.5 70.5% 54.4 86.2% 57.3 90.8% 57.8 91.6% 58.7 93.0% 57.6 91.3% 59.1 93.7% 60.6 96.0% 61.3 97.1%

55.1 57.7 57.0 58.5 57.6 60.7 59.5

45.9 40.1 50.3 50.0 49.1 52.5 55.1

24.7 20.5 25.3 25.0 26.3 29.8 29.7

53.0 84.0% 54.5 86.4% 54.8 86.8% 55.1 87.3% 56.9 90.2% 59.0 93.5% 59.7 94.6%

Retain 128 Tokens (↓ 77.8%) FastV (ECCV24) 54.0 PDrop (CVPR25) 57.1 SparseVLM (ICML25) 57.3 PruMerge+ (ICCV25) 58.2 TRIM (COLING25) 58.4 VisionZip (CVPR25) 57.6 DART (EMNLP25) 57.9 DivPrune (CVPR25) 59.4 SCOPE (NeurIPS25) 59.4 TOPS (Ours) 60.5

69.2 70.1 69.0 69.1 68.6 68.7 69.1 68.6 68.5 68.2

56.4 56.7 56.3 54.0 52.2 56.9 56.3 55.9 57.1 57.0

68.2 77.5 83.1 83.1 85.3 83.3 80.4 87.0 85.9 86.8

1368.9 1444.1 1399.3 1408.1 1413.4 1436.9 1408.7 1405.1 1440.5 1482.7

Retain 64 Tokens (↓ 88.9%) FastV (ECCV24) 46.0 PDrop (CVPR25) 46.1 SparseVLM (ICML25) 52.0 PruMerge+ (ICCV25) 55.4 TRIM (COLING25) 56.6 VisionZip (CVPR25) 55.1 DART (EMNLP25) 54.7 DivPrune (CVPR25) 57.5 SCOPE (NeurIPS25) 58.3 TOPS (Ours) 58.7

70.1 68.8 69.2 69.5 69.0 69.0 69.3 68.0 68.7 68.6

51.6 49.2 52.1 52.0 49.7 55.5 54.7 54.5 56.5 56.2

35.5 40.8 69.7 75.7 85.9 77.0 73.8 85.5 84.1 86.5

973.5 982.2 1190.4 1316.8 1350.9 1365.2 1365.1 1334.7 1399.6 1442.7

Retain 32 Tokens (↓ 94.4%) PruMerge+ (ICCV25) TRIM (COLING25) VisionZip (CVPR25) DART (EMNLP25) DivPrune (CVPR25) SCOPE (NeurIPS25) TOPS (Ours)

52.9 54.5 51.8 52.9 54.9 56.2 56.7

67.9 68.1 69.1 69.3 68.6 69.4 68.8

49.2 47.6 53.1 52.2 52.9 54.8 54.9

66.7 84.9 69.4 69.1 81.5 80.2 83.5

Figure 12 extends the comparison to multiple baselines (FastV, DivPrune, SCOPE) across diverse questions. Green text denotes correct answers; red denotes incorrect ones.

H

Per-Benchmark Radar Visualization

To provide a more intuitive view of per-benchmark performance across all compression ratios and model variants, we present radar charts covering LLaVA-1.5 (7B and 13B), LLaVA-NeXT (7B and 13B), Qwen2.5-VL-7B and InternVL3-8B. Each axis corresponds to one benchmark; the outer boundary on each axis is set by the highest-scoring method. TOPS (red) consistently covers the largest area across all settings.

1236.6 1251.8 1251.2 1273.3 1284.9 1371.6 1384.7

I

Assets, Licenses, and Intended Use

Our experiments use publicly released models, codebases, and benchmarks solely for noncommercial academic research and evaluation. We build on official open-source codebases and toolkits, including LLaVA, LLaVA-NeXT, lmms-eval, and VLMEvalKit, and follow the licenses and usage terms of the corresponding model providers for LLaVA-1.5, LLaVA-NeXT, LLaVA-Video, InternVL3-8B, and Qwen2.5-VL-7B-Instruct. We evaluate on public multimodal benchmarks, some of which impose non-commercial or academiconly restrictions, such as ScienceQA, MLVU, LongVideoBench, MM-Vet, and Video-MME. We do not redistribute third-party model weights, datasets, annotations, or videos; users should ob-

Table 12: Performance comparison of different pruning methods on LLaVA-NeXT-7B. Rel. represents the ratio of pruned model’s Acc. to the baseline’s Acc. Red : attention-based. Green : attention&diversity. Blue : diversity-based. Cyan : coverage-based. Purple : ours.

Method

GQA SQAIM G VQAText POPE MME MMBEN MMBCN MMVet Acc.

Rel.

Upper Bound: All 2880 tokens (100%) Baseline

62.5

67.5

60.3

86.8 1511.8

65.8

57.3

40.0

64.5 100.0%

53.5 55.2 58.6 57.3 55.8 58.1 57.1 57.3 58.2 59.0

39.5 36.7 36.1 32.7 37.8 38.9 36.9 38.0 39.7 36.4

61.3 95.0% 62.3 96.6% 63.4 98.3% 62.2 96.4% 63.1 97.8% 64.1 99.4% 63.2 98.0% 63.5 98.4% 64.4 99.8% 64.5 100.0%

42.5 44.7 56.7 55.6 51.0 55.6 55.7 55.7 57.6 58.4

20.0 24.0 32.8 31.4 32.7 35.8 35.7 34.8 36.1 37.4

48.6 75.3% 51.2 79.4% 60.1 93.2% 60.3 93.5% 60.3 93.5% 61.5 95.3% 61.5 95.3% 61.9 96.0% 63.1 97.8% 63.9 99.1%

48.9 45.2 50.4 53.6 53.7 56.4 57.1

29.3 29.6 32.3 32.2 32.0 32.4 36.5

55.6 86.2% 56.7 87.9% 57.6 89.3% 58.6 90.9% 59.6 92.4% 60.9 94.4% 62.2 96.4%

Retain 640 Tokens (↓ 77.8%) FastV (ECCV24) 58.9 PDrop (CVPR25) 60.0 SparseVLM (ICML25) 61.2 PruMerge+ (ICCV25) 60.8 TRIM (COLING25) 62.1 VisionZip (CVPR25) 61.2 DART (EMNLP25) 61.3 DivPrune (CVPR25) 61.9 SCOPE (NeurIPS25) 62.0 TOPS (Ours) 62.0

67.4 66.7 67.6 67.8 66.9 68.1 68.0 67.8 68.0 69.3

58.1 57.8 59.7 54.9 54.8 59.9 59.5 57.0 60.1 60.0

79.5 83.8 85.3 85.3 86.9 86.0 85.0 86.9 86.7 87.6

1412.6 1475.9 1456.8 1480.2 1471.8 1493.4 1450.2 1469.7 1485.1 1527.6

63.1 64.1 65.9 64.6 66.8 65.8 64.9 65.8 66.2 65.7

Retain 320 Tokens (↓ 88.9%) FastV (ECCV24) 49.8 PDrop (CVPR25) 50.4 SparseVLM (ICML25) 57.9 PruMerge+ (ICCV25) 58.8 TRIM (COLING25) 59.9 VisionZip (CVPR25) 58.9 DART (EMNLP25) 59.5 DivPrune (CVPR25) 61.1 SCOPE (NeurIPS25) 60.9 TOPS (Ours) 61.2

66.6 66.7 67.2 68.1 66.2 67.5 67.5 67.7 68.0 68.5

52.2 49.0 56.5 54.0 50.2 58.8 57.6 56.2 58.3 58.4

49.5 60.8 76.9 79.5 86.5 82.3 81.0 84.7 85.0 86.3

1099.0 1171.5 1386.1 1444.3 1443.8 1397.1 1419.5 1423.3 1477.0 1500.1

53.4 55.5 63.1 63.0 63.5 63.3 64.2 63.9 65.0 65.8

Retain 160 Tokens (↓ 94.4%) PruMerge+ (ICCV25) TRIM (COLING25) VisionZip (CVPR25) DART (EMNLP25) DivPrune (CVPR25) SCOPE (NeurIPS25) TOPS (Ours)

56.2 57.4 55.2 56.8 59.3 59.8 59.6

66.9 65.5 67.9 67.8 67.1 67.1 67.7

50.3 45.8 55.0 54.9 54.1 56.8 56.7

71.1 84.8 74.9 75.3 80.0 81.3 83.6

tain these assets from their official sources and comply with their original licenses. We do not collect new human-subject data. Since our experiments use publicly released benchmarks, we rely on their original curation procedures and manually inspect examples used in qualitative visualizations to avoid displaying personally identifying or offensive content.

J

Broader Impact

This work presents TOPS, a training-free visual token pruning method for efficient MLLM infer-

1289.6 1275.8 1327.8 1325.4 1356.6 1402.2 1446.8

58.0 61.6 58.6 62.0 62.9 63.3 64.2

ence. TOPS lowers energy consumption and hardware requirements of deploying MLLMs, democratizing access to capable vision-language models on resource-constrained devices and reducing carbon emissions per query. As a training-free, architecture-agnostic method, TOPS does not introduce new biases through retraining. Users should be aware that token pruning may degrade accuracy on inputs requiring fine-grained spatial reasoning under aggressive compression, and should validate pruning configurations on their target task before deployment in safety-critical settings.

Table 13: Performance comparison of different pruning methods on LLaVA-1.5-13B. Rel. represents the ratio of pruned model’s Acc. to the baseline’s Acc. Red : attention-based. Green : attention&diversity. Blue : diversity-based. Cyan : coverage-based. Purple : ours.

Method

GQA SQAIM G VQAText POPE MME MMBEN MMBCN MMVet Acc.

Rel.

Upper Bound: All 576 tokens (100%) Baseline

63.3

72.8

61.2

86.0 1531.2

68.5

63.5

36.2

66.0 100.0%

66.1 67.5 68.4 66.3 67.1 67.4 65.4 66.3 66.7 66.6

62.3 62.8 62.6 61.2 58.4 62.5 62.2 60.7 62.9 62.4

32.8 35.1 35.2 33.6 35.1 36.0 34.8 34.4 36.6 38.1

62.6 94.8% 64.4 97.6% 64.8 98.2% 63.0 95.4% 63.2 95.7% 64.0 96.9% 62.9 95.3% 63.9 96.8% 64.5 97.7% 65.3 98.9%

59.2 63.1 65.2 65.0 65.0 64.9 64.7 64.1 65.9 66.8

55.1 56.6 60.3 59.3 52.7 61.3 60.6 59.8 62.6 62.3

26.9 21.9 32.9 30.3 27.8 33.4 32.8 29.3 34.5 37.0

54.9 83.2% 56.6 85.8% 61.4 93.0% 60.2 91.2% 60.5 91.7% 61.5 93.2% 60.8 92.1% 62.2 94.2% 63.6 96.4% 64.2 97.3%

61.1 63.1 61.2 61.9 61.7 63.5 65.2

53.5 45.4 55.8 56.2 57.2 60.1 61.9

28.7 26.4 29.3 29.4 27.8 34.0 35.1

56.5 85.6% 57.6 87.3% 57.1 86.5% 57.6 87.3% 59.7 90.5% 61.6 93.3% 62.5 94.7%

Retain 128 Tokens (↓ 77.8%) FastV (ECCV24) 58.3 PDrop (CVPR25) 61.0 SparseVLM (ICML25) 59.6 PruMerge+ (ICCV25) 58.3 TRIM (COLING25) 59.4 VisionZip (CVPR25) 57.9 DART (EMNLP25) 57.7 DivPrune (CVPR25) 59.2 SCOPE (NeurIPS25) 59.1 TOPS (Ours) 60.2

74.2 73.3 74.3 73.3 72.4 73.8 74.2 72.8 73.8 72.9

58.6 60.2 59.3 56.1 55.0 58.9 58.7 58.0 58.9 59.4

75.5 83.6 85.0 82.7 86.8 82.7 80.4 86.8 86.0 86.6

1460.6 1489.5 1487.9 1445.9 1426.9 1449.2 1395.0 1457.7 1440.2 1522.8

Retain 64 Tokens (↓ 88.9%) FastV (ECCV24) 51.9 PDrop (CVPR25) 54.1 SparseVLM (ICML25) 55.9 PruMerge+ (ICCV25) 56.3 TRIM (COLING25) 57.9 VisionZip (CVPR25) 56.2 DART (EMNLP25) 55.7 DivPrune (CVPR25) 57.9 SCOPE (NeurIPS25) 58.6 TOPS (Ours) 58.6

73.1 73.1 73.0 73.5 72.0 74.2 73.8 71.7 73.6 73.8

53.4 55.3 57.1 54.4 52.0 57.4 57.4 57.3 58.2 56.6

56.9 66.1 77.9 75.7 86.5 75.7 72.8 84.5 83.1 85.4

1246.4 1247.0 1374.3 1338.2 1406.2 1379.6 1380.0 1454.2 1445.2 1462.2

Retain 32 Tokens (↓ 94.4%) PruMerge+ (ICCV25) TRIM (COLING25) VisionZip (CVPR25) DART (EMNLP25) DivPrune (CVPR25) SCOPE (NeurIPS25) TOPS (Ours)

54.1 55.6 52.7 53.9 56.2 57.2 56.1

71.7 70.4 72.9 73.2 70.9 72.4 73.3

52.4 49.6 55.2 55.1 54.6 57.2 57.7

67.4 85.8 66.8 66.9 79.3 77.6 78.2

1269.1 1284.7 1257.7 1282.8 1405.2 1413.4 1442.2

Table 14: Performance comparison of different pruning methods on LLaVA-NeXT-13B. Rel. represents the ratio of pruned model’s Acc. to the baseline’s Acc. Red : attention-based. Green : attention&diversity. Blue : diversity-based. Cyan : coverage-based. Purple : ours.

Method

GQA SQAIM G VQAText POPE MME MMBEN MMBCN MMVet Acc.

Rel.

Upper Bound: All 2880 tokens (100%) Baseline

64.4

73.1

63.2

85.3 1539.5

68.5

61.2

45.0

67.2 100.0%

59.9 60.8 64.0 61.9 61.2 62.6 61.9 62.9 63.2 63.4

43.8 39.7 41.3 39.4 42.3 46.8 45.5 39.0 40.9 44.4

64.8 96.4% 65.8 97.9% 67.0 99.7% 64.6 96.1% 66.1 98.4% 67.1 99.9% 66.5 99.0% 65.9 98.1% 66.8 99.4% 67.6 100.6%

54.4 55.3 63.5 60.6 57.4 62.3 61.9 61.9 63.1 63.0

30.2 29.5 39.3 36.5 33.1 41.1 41.4 39.2 42.8 42.2

56.6 84.2% 59.7 88.8% 64.8 96.4% 62.7 93.3% 62.9 93.6% 64.8 96.4% 64.2 95.5% 64.7 96.3% 66.0 98.2% 66.6 99.1%

57.1 51.6 60.0 60.8 60.9 62.9 62.8

30.6 27.8 35.9 35.0 37.4 37.8 40.4

58.9 87.6% 59.9 89.1% 61.7 91.8% 61.4 91.4% 63.1 93.9% 64.4 95.8% 64.9 96.6%

Retain 640 Tokens (↓ 77.8%) FastV (ECCV24) 60.9 PDrop (CVPR25) 62.8 SparseVLM (ICML25) 62.7 PruMerge+ (ICCV25) 62.8 TRIM (COLING25) 63.1 VisionZip (CVPR25) 62.9 DART (EMNLP25) 62.7 DivPrune (CVPR25) 63.5 SCOPE (NeurIPS25) 63.7 TOPS (Ours) 64.1

71.7 71.7 72.5 70.6 71.2 70.8 71.0 72.2 71.7 72.8

60.7 62.1 62.8 56.2 57.6 62.1 61.3 59.2 62.4 62.5

80.2 84.4 85.6 83.7 87.3 85.8 85.2 86.5 86.5 86.7

1516.7 1559.1 1562.7 1497.3 1554.6 1549.2 1542.4 1526.1 1573.1 1560.8

65.5 66.6 68.8 67.4 68.7 68.1 67.6 67.5 67.6 69.0

Retain 320 Tokens (↓ 88.9%) FastV (ECCV24) 54.6 PDrop (CVPR25) 57.7 SparseVLM (ICML25) 60.9 PruMerge+ (ICCV25) 61.1 TRIM (COLING25) 61.3 VisionZip (CVPR25) 60.7 DART (EMNLP25) 60.9 DivPrune (CVPR25) 61.8 SCOPE (NeurIPS25) 62.7 TOPS (Ours) 63.2

70.5 72.1 70.9 70.7 69.9 70.2 69.8 72.3 71.0 71.5

55.4 56.2 60.0 55.9 52.8 60.7 59.7 57.6 60.8 61.1

63.6 74.6 81.5 79.1 87.2 82.3 81.1 85.2 85.2 85.9

1279.0 1386.3 1491.6 1426.5 1476.6 1487.3 1457.4 1473.0 1509.3 1569.7

59.8 62.8 68.0 66.6 67.3 66.5 65.9 65.9 66.6 67.6

Retain 160 Tokens (↓ 94.4%) PruMerge+ (ICCV25) TRIM (COLING25) VisionZip (CVPR25) DART (EMNLP25) DivPrune (CVPR25) SCOPE (NeurIPS25) TOPS (Ours)

57.9 58.9 57.8 58.7 60.0 61.2 61.1

70.1 69.1 69.7 70.1 71.4 71.2 70.6

52.8 49.2 58.6 57.2 56.3 59.2 59.7

72.1 87.0 76.8 75.7 81.9 82.7 83.9

1345.9 1392.3 1393.9 1389.3 1436.7 1473.7 1480.9

63.2 65.7 64.8 64.6 65.1 66.2 66.6

SQAIMG

62.43

63.01

73.57

60.19

70.88

0.5

1.0

1.5

2.0

57.92 55.48

61

.5 63

2.0

66.0

.6 68 .2 71 .8 73

58.7 .1

0.5

1.0

1.5

53.61

66.0 .4 68

51.34

1 101 04.8 .1 97.5

93.8

86.5 36.3 33.8 35.1 32.6

2.0

.0

.5

63

28.84

.2 30

0.5

1.0

1.5

27.71 26.58

33 .8

1297.9

63.3

29.96 29

148

1354.9

55.88

31.09

31.4

60.36 60.7

32.22

.6

58.15

31.4

53.8 56.2

62.81

33.34 30.2

6 66.0 8.4

60.42

.6

1.5

61.1

66.0

65.25

68

1.0

63.5

7 71.2 3.8 68.6

1 1 726 160 665.5 .8 4.2

.9 1542

1.7

1526.0

.0

0.5

34.47

64.95 62.68

66

.9

42

15

4.2 160 65.5 16 26.8 17

35.60

67.22

67.69

1411.9 2.9 154

36.72

69.49

1583.1

58.1

79.13

.2 30

70.13

82.55

MM-Vet 71.75

72.57

1469.0

93.8

27.8

5

13

51.22

.5 97

9.1

85.96

.5 97 1.1 10 04.8 1

90.1 .8

75.01

89.38

53.47

.1

1640.1

.3 63 65.7 .1 68

61

77.45

1697.2

1.0

0.5

60.9

.3

.0

1811.3

92.80

82.8

MMBCN 66

.7

.4

20

1754.2

1.5

.9

MMBench 81 14

14

62.81

58.4 60

63

.3 77

MME 2.0

74.4

96.21

55.71

65.50

.3 77 .2 80 .0 83

99.63

93

53.62

71.5 .4

63.7

55.97

103.05

57.95

53.6

68.19

74

.7 63 .2 66

.2 66 .7 68 .3 71

106.46

32

65.7

.1

61.2

6 65.7 8.1

76.26

58.31

0.5

60.9

65.35

60.66

56.1

64.68

56.0

74.4

78.95

109.88

66.92

61

1.0

69.16

81.65

68.6

67.70

58.7

1.5

63.7

70.04

71.40

63.3

8 80.2 3.0

77.3

66.2

84.34

.8 93

87.03

POPE .9 60

72.39

.4

74.73

VQAText 74

.7 63

2.0

7 68.7 1.3

GQA

2.0

Figure 6: Full hyperparameter sensitivity across all 8 benchmarks. Contour plots of per-benchmark performance across seven (α, λ) configurations at 64 tokens on LLaVA-1.5-7B. Star: optimal configuration; white dots: other tested configurations.

Table 15: Ablation study of hyper-parameters α (diversity weight) and λ (coverage weight) on LLaVA-1.5-7B. Acc. is averaged over benchmarks. Bold and underline denote the best and second results per column within each group.

α

λ

GQA SQAIM G VQAText POPE MME MMBEN MMBCN MMVet Acc. Retain 64 Tokens

0.5 0.5 1 0.5 0.5 1 1 1 2 1 1 2 2 2

58.9 58.7 58.7 58.5 58.6 58.9 58.8

68.6 68.3 68.6 68.8 68.6 68.6 68.6

56.4 56.3 56.2 56.1 56.2 55.8 56.1

86.5 86.2 86.5 86.5 86.8 86.4 86.4

1441.7 1436.4 1442.7 1436.8 1428.2 1417.6 1390.5

61.3 60.5 60.9 60.7 60.7 61.0 60.5

56.9 56.6 56.5 56.4 56.5 56.0 55.8

28.6 26.6 30.6 30.5 29.1 28.6 27.4

61.2 60.6 61.3 61.2 61.0 60.8 60.4

60.2 60.2 59.5 60.2 60.5 59.5 59.4

55.5 55.2 55.1 54.8 53.6 54.0 54.6

28.9 26.6 29.7 26.4 27.4 28.4 27.2

59.3 59.2 59.7 59.4 59.3 59.6 58.5

Retain 32 Tokens 0.5 0.5 1 0.5 0.5 1 1 1 2 1 1 2 2 2

55.9 56.8 56.7 56.9 56.8 56.8 55.6

69.1 68.7 68.8 69.3 68.7 68.7 69.0

55.5 55.2 55.1 54.8 54.4 54.0 54.4

81.6 83.4 83.5 84.5 84.8 84.6 81.7

1362.6 1357.0 1384.7 1361.8 1370.3 1388.9 1327.9

Table 17: Ablation of pruning layer configurations in Stage 2 for LLaVA-1.5-7B. All variants apply Stage 1 (576→256) identically. Avg. is mean score across seven benchmarks; Rel. is relative to unpruned baseline (63.1). Stage-2 Layers

MME MMB MMBCN SQA MMVet TVQA GQA POPE Avg Rel (%)

L2 (256→119) 1421.1 L10 (256→70) 1472.0 L2+L14 (256→128→114) 1439.1 L12+L24 (TOPS) (256→64→32) 1482.7

60.5 61.6 60.9 62.5

54.8 57.5 55.4 57.2

68.4 68.4 68.0 68.2

30.3 28.7 31.3 30.0

57.0 56.9 57.2 57.0

58.9 60.0 59.1 60.5

85.9 86.0 86.3 86.8

60.9 61.6 61.3 62.0

96.5 97.6 97.1 98.3

Figure 7: Logit fidelity comparison across pruning methods and token budgets on 200 TextVQA samples. TOPS consistently achieves the smallest logit distortion across all budgets.

Figure 8: Cross-layer token selection stability via mean Jaccard similarity (R=128, N =1000 POPE samples, LLaVA-1.5-7B). Left to right: attention-based, diversity-based, and coverage-based criteria. Near-zero off-diagonal values confirm that cross-layer inconsistency is universal, justifying TOPS’s multi-stage design.

Figure 9: Spatial selection frequency heatmaps for FastV, DivPrune, DART, and SCOPE (9000 POPE samples, budget= 128). FastV shows strong positional bias toward bottom rows due to attention shift; other methods achieve roughly uniform spatial coverage.

Figure 10: Per-token selection probability of TOPS across three pruning stages (budget= 128, 9000 POPE samples, LLaVA1.5-7B). All stages maintain low Gini (≤ 0.290) and high normalized entropy (≥ 0.975), confirming spatially uniform token selection.

Figure 11: Qualitative comparison of visual token selections between the Vanilla model (no pruning) and TOPS across multiple examples. Despite using far fewer tokens, TOPS selects task-relevant regions and produces correct answers.

Figure 12: Comprehensive qualitative comparison of visual token selections by FastV, DivPrune, SCOPE, and TOPS across diverse real-world questions. Green text indicates a correct answer; red indicates an incorrect answer.

GQA 97.74%

GQA 94.83% SQA

MMVet

95.85%

95%

IMG

98.13%

SQA

MMVet

97.76%

95%

90%

MMBCN

97.94%

MMBEN

POPE

96.60%

101.05%

MMBCN

SCOPE

96.56%

MMBEN

POPE

94.13%

95%

100.70%

MMBCN

(a) LLaVA-1.5-7B — 128 tokens (↓77.8%)

VisionZip

DivPrune

SCOPE

VQAText

94.33%

MMBEN

POPE

91.96%

97.21% MME 91.92%

LLaVA-1.5-7B ( 88.9%)

TOPS (Ours)

98.99%

85%

94.84%

MME 95.77%

LLaVA-1.5-7B ( 77.8%) DivPrune

SQAIMG

MMVet

94.89% 90%

VQAText

85%

97.25%

MME 98.42%

VisionZip

98.71%

90%

VQAText

85%

98.45%

GQA 91.60% IMG

LLaVA-1.5-7B ( 94.4%)

TOPS (Ours)

(b) LLaVA-1.5-7B — 64 tokens (↓88.9%)

VisionZip

DivPrune

SCOPE

TOPS (Ours)

(c) LLaVA-1.5-7B — 32 tokens (↓94.4%)

Figure 13: Radar charts for LLaVA-1.5-7B at three compression levels. GQA 95.10%

GQA 92.58% SQAIMG

MMVet

105.25%

95%

100.14%

102.21%

95%

90%

MMBCN

97.06%

MMBEN

POPE

97.23%

100.70%

MMBCN

DivPrune

92.48%

MMBEN

POPE

97.52%

95%

99.30%

MMBCN

(a) LLaVA-1.5-13B — 128 tokens

PruMerge+

DART

DivPrune

VQAText

94.28%

MMBEN

POPE

95.18%

90.93% MME 94.19%

LLaVA-1.5-13B ( 88.9%)

TOPS (Ours)

100.69%

85%

97.48%

MME 95.49%

LLaVA-1.5-13B ( 77.8%) DART

SQAIMG

96.96% 90%

VQAText

85%

98.11%

MME 99.45%

PruMerge+

101.37%

MMVet

90%

VQAText

85%

98.27%

GQA 88.63% SQAIMG

MMVet

LLaVA-1.5-13B ( 94.4%)

TOPS (Ours)

(b) LLaVA-1.5-13B — 64 tokens

PruMerge+

DART

DivPrune

TOPS (Ours)

(c) LLaVA-1.5-13B — 32 tokens

Figure 14: Radar charts for LLaVA-1.5-13B at three compression levels. GQA 99.20% SQAIMG

MMVet

91.00%

95%

102.67%

SQAIMG

MMVet

93.50%

95%

90%

MMBCN

99.50%

MMBEN

POPE

99.85%

100.92%

MMBCN

DivPrune

96.85%

MMBEN

POPE

100.00%

95%

99.42%

MMBCN

(a) LLaVA-NeXT-7B — 640 tokens

PruMerge+

DART

DivPrune

VQAText

94.03%

MMBEN

POPE

97.57%

96.31% MME 95.70%

LLaVA-NeXT-7B ( 94.4%)

LLaVA-NeXT-7B ( 88.9%)

TOPS (Ours)

100.30%

85%

99.65%

MME 99.23%

LLaVA-NeXT-7B ( 77.8%) DART

SQAIMG

MMVet

91.25% 90%

VQAText

85%

101.92%

MME 101.05%

PruMerge+

101.48%

90%

VQAText

85%

102.97%

GQA 95.36%

GQA 97.92%

TOPS (Ours)

(b) LLaVA-NeXT-7B — 320 tokens

VisionZip

DivPrune

SCOPE

TOPS (Ours)

(c) LLaVA-NeXT-7B — 160 tokens

Figure 15: Radar charts for LLaVA-NeXT-7B at three compression levels. GQA 99.53%

GQA 98.14% SQA

MMVet

98.67%

95%

IMG

99.59%

SQA

MMVet

93.78%

95%

90%

MMBCN

98.89%

MMBEN

POPE

100.73%

101.64% MME 101.38%

LLaVA-NeXT-13B ( 77.8%) PruMerge+

TRIM

DivPrune

97.81%

SQAIMG

MMVet

95.28%

95%

90%

VQAText

85%

103.59%

GQA 96.52% IMG

TOPS (Ours)

(a) LLaVA-NeXT-13B — 640 tokens

MMBCN

90%

VQAText

85%

102.94%

MMBEN

96.68%

POPE

98.69%

100.70% MME 101.96%

LLaVA-NeXT-13B ( 88.9%) PruMerge+

TRIM

DivPrune

99.58%

TOPS (Ours)

(b) LLaVA-NeXT-13B — 320 tokens

MMBCN

VQAText

85%

100.96%

MMBEN

94.91%

POPE

99.11%

97.22% MME 95.54%

LLaVA-NeXT-13B ( 94.4%) PruMerge+

TRIM

DivPrune

TOPS (Ours)

(c) LLaVA-NeXT-13B — 160 tokens

Figure 16: Radar charts for LLaVA-NeXT-13B at three compression levels.

MLVU 98.08%

MLVU 94.98%

Long

Long

LVB

98.49%

97.63%

95%

Medium

99.38%

Short

95%

LVB-R

Short

94.98%

SparseVLM

(a) LLaVA-Video-7B — 64 tok/frame

DART

DivPrune

90.33% VMME 93.40%

LLaVA-Video-7B ( 95.1%)

TOPS (Ours)

SparseVLM

(b) LLaVA-Video-7B — 32 tok/frame

DART

DivPrune

AI2D 96.00%

AI2D 92.93%

SQA

POPE

98.18%

95%

SQA

POPE

98.27% 90%

Hall

85%

90.84%

89.98%

MMBCN

MME

97.10%

99.25%

MMStar

MMBCN

MMB 95.99%

DivPrune

MMB 94.58%

TOPS (Ours)

FastV

CDPruner

(b) Qwen2.5-VL-7B — 128 tokens

98.78%

SQA

POPE

93.77% 90%

Hall

85%

91.95%

MMBCN

92.71%

MME

99.41%

97.17%

MMStar

MMBCN

VisionZip

TOPS (Ours)

(c) InternVL3-8B — 256 tokens

95.06% MMB 95.68% EN

InternVL3-8B ( 80.0%) DivPrune

83.81%

MME

94.59%

MMB 98.95%

Hall

85%

83.60%

EN

CDPruner

96.79%

95%

90%

MMStar

TOPS (Ours)

DivPrune

AI2D 89.42% POPE

95%

96.33%

Qwen2.5-VL-7B ( 90.1%)

AI2D 95.53%

97.55%

MME EN

(a) Qwen2.5-VL-7B — 256 tokens SQA

83.72%

95.05%

Qwen2.5-VL-7B ( 80.2%) CDPruner

Hall

85%

82.29%

EN

FastV

95.67%

95%

90%

MMStar

InternVL3-8B ( 90.0%) CDPruner

TOPS (Ours)

(c) LLaVA-Video-7B — 16 tok/frame

Figure 17: Radar charts for LLaVA-Video-7B at three compression levels.

100.46%

93.54%

LVB-R

92.56%

LLaVA-Video-7B ( 90.1%)

TOPS (Ours)

LVB-P

85%

93.79%

VMME 96.23%

LLaVA-Video-7B ( 80.2%) DivPrune

Medium

97.69%

94.39%

VMME 99.06%

92.03%

90%

Short

95.54%

DART

LVB

93.79%

LVB-P

85%

96.08%

LVB-R

98.30%

SparseVLM

96.44%

95% 90%

LVB-P

85%

100.33%

Long

LVB

99.06%

90%

Medium

MLVU 89.66%

DivPrune

VisionZip

TOPS (Ours)

(d) InternVL3-8B — 128 tokens

Figure 18: Radar charts for Qwen2.5-VL-7B and InternVL3-8B.