Conceptio › Archive › arXiv CS

arXiv CSopen access

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Yunkai Dang* 1 Yizhu Jiang* 1 Yifan Jiang 1 Qi Fan 1 Yinghuan Shi 1 Wenbin Li† 1 Yang Gao 1

Multimodal Large Language Models (MLLMs) suffer from substantial computational overheads, driven by the massive redundancy of visual token sequences. To mitigate such redundancy, existing works typically rely on single-layer ViT features and static pruning strategies. However, these fixed configurations often render them brittle across diverse instructions. To address these limitations, we present class-adaptive layer fusion and dual-stage pruning (CLASP), a plugand-play token reduction framework. Specifically, we construct a category-specific visual representation via multi-layer vision feature fusion. Then we perform dual-stage pruning that allocates the token budget between attention-salient pivots (relevance) and redundancy-aware completion tokens (coverage). By class-adaptively pruning, our method shows that prompt-conditioned feature fusion and budget allocation enable aggressive yet robust visual token pruning. Experiments show that our method achieves superior performance across various benchmarks, pruning ratios, and MLLM architectures compared to existing methods. Code will be available at https://github.com/Yunkaidang/CLASP.

28.8

28.0

28.2

28.6

28.5

27.8

28.4

28.3

28.7

28.3

28.9

28.3

27.1

28.8

16.2

27.8

28.2

28.1

28.2

28.4

28.6

25.8

28.5

14.0

28.1

28.9

28.4

28.7

28.0

29.1

28.3

29.5

28.8

28.5

28.2

28.3

27.9

28.4

28.6

27.9

28.1

28.0

28.2

28.6 28.4 28.2

Score

Layer mixture strategy

MMVet (layer mixture strategy, R=192)

28.0 27.8 29.6

30.0

29.9

29.0

29.8

29.2

0.3

29.4

31.0

30.6

29.7

30.1

30.3

29.0

30.1

30.0

0.5

29.9

29.7

30.1

31.0

29.4

30.0

28.8

30.1

30.4

30.00

0.7

30.2

30.4

29.5

29.7

29.2

30.5

28.6

30.2

30.5

29.75

0.9

29.3

31.2

29.5

29.9

30.4

29.6

30.2

30.5

30.7

1.0

29.8

Te C2 xt

at C4 ia l

Co C un 5 t

ct C6 io n

In C te 7 nt

Score

30.25

29.50

th C8 er

29.25

30.4

30.2

Sc C en 3 e

29.6

A C1 ttr

30.50

0.1

C O 0 bj

MMVet (mixture ratio, R=192)

Mixture ratio

arXiv:2604.12767v1 [cs.CV] 14 Apr 2026

Abstract

Question type category

Figure 1. Impact of hyperparameter settings on MMVet dataset performance (LLaVA-v1.5-7B, 192 retained tokens). The heatmaps illustrate the score distribution across different question categories under varying conditions. Top: Evaluation of five representative layer-fusion strategies (A–E), ordered by an increasing proportion of weights assigned to deeper layers (i.e., shifting from shallow in A to deep in E). Bottom: Sweeping the attention–similarity mixing coefficient (higher: more attention; lower: more similarity).

primary bottleneck stems from the substantial computational overhead associated with processing visual tokens. In these architectures, images are typically serialized into long, highdensity sequences of patch-level tokens (Jin et al., 2025). This representation causes the computational cost of selfattention to scale quadratically with the input resolution. For instance, increasing the input resolution from 336 × 336 in LLaVA-1.5 (Liu et al., 2023a) to 672 × 672 in LLaVANeXT (Liu et al., 2024b) raises the token count from 576 to a staggering 2, 880. Such a proliferation of tokens imposes excessive demands on both memory footprint and inference latency. Ultimately, these overheads create a major barrier to efficient inference and deployment (Kong et al., 2025).

1. Introduction Multimodal Large Language Models (MLLMs) extend the instruction-following and reasoning capabilities of LLMs to the visual domain (?). These models typically align pre-trained vision encoders with autoregressive language decoders through an alignment module (Wu et al., 2023). Despite their impressive performance across various multimodal benchmarks (Liu et al., 2024d; Fu et al., 2025), MLLMs face significant challenges in practical utility. The * Equal contribution † Corresponding author. 1 School of Artificial Intelligence Science and Technology, Nanjing University. Correspondence to: Wenbin Li <[email protected], [email protected]>.

To reduce the inference cost of MLLMs, recent studies (Xing et al., 2024; Chen et al., 2024a; Zhang et al., 2024b; Bolya et al., 2022; Wen et al., 2025a; Zou et al., 2025b) propose token reduction approaches to prune visual tokens. Existing approaches can be broadly grouped

Preprint. April 15, 2026.

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

into two primary directions. The first line leverages crossmodal attention scores (Chen et al., 2024a; Zhang et al., 2024b; Zhan et al., 2024; Arif et al., 2025; Zhang et al., 2025a) to identify and retain instruction-relevant visual regions, primarily focusing on token relevance. The second line employs similarity-based pruning (Bolya et al., 2022; Xing et al., 2024; Yang et al., 2025; Alvar et al., 2025; Zhang et al., 2025b) to remove redundant patches by merging or dropping low-saliency tokens, emphasizing the coverage of visual information. While existing approaches have achieved promising results from various perspectives, most methods still rely on category-insensitive and fixed strategies for both visual feature extraction and token pruning. Specifically, these methods rely on a single or a fixed set of ViT layers to derive visual representations. These static representations serve as both pruning signals and decoder inputs, yet they fail to capture task-specific nuances. Building on this static foundation, existing methods (Chen et al., 2024a; Zhang et al., 2024b; Wen et al., 2025a; Zou et al., 2025b) then apply a rigid pruning strategy with predefined parameters, regardless of the instruction category. However, such methods overlook a critical nuance: different intents emphasize distinct levels of visual abstraction and impose varying requirements on spatial relevance and coverage.

dynamically fuses visual features and prunes tokens. In particular, we first introduce class-adaptive layer fusion to optimize the visual representation. Instead of relying on a single ViT layer, this module adaptively integrates visual features from multiple layers, conditioned on the input instructions. This process preserves category-relevant local details while effectively removing redundant information. Building on these optimized representations, we implement a class-adaptive dual-stage pruning strategy. The pruning process consists of a relevance-preserving stage to protect key evidence. It is followed by a coverage-oriented stage to maintain the necessary context while further reducing redundancy. We evaluate our method across a comprehensive suite of eight image benchmarks and three video benchmarks. The results demonstrate that our approach consistently surpasses existing state-of-the-art token pruning methods. Notably, our method preserves approximately 94.7% of the original performance even with a token reduction of up to 88.9%. Besides, our method is plug-and-play and easily integrable into various MLLM architectures, which makes our framework well-suited for efficient practical deployment.

Based on these insights, we contend that both visual feature extraction and token pruning should be dynamically adjusted according to input question categories. To verify this, we evaluate the sensitivity of specific instruction categories to different configurations under a fixed token budget (MM-Vet, LLaVA-v1.5-7B, R = 192). We first conduct a preliminary analysis on layer-mixture strategies to optimize visual feature extraction (Fig. 1, top). Specifically, we employ multiple schemes (A→E) that shift the fusion emphasis from shallow to deep ViT layers. The results indicate that model accuracy varies significantly depending on the layer depth of the integrated features. For instance, the performance of the Count category degrades as the fusion emphasis shifts from hybrid to deep layers (29.5 vs.27.9). To further examine the impact of pruning criteria, we investigate the limitations of relying on a fixed balance between attention and similarity (Fig. 1, bottom). We implement a hybrid strategy that transitions the pruning logic between attention-led and similarity-led modes. The findings show that a single, static mixing ratio fails to achieve optimal performance for every category. Specifically, text-based categories reach peak accuracy at lower ratios (≈ 0.3), whereas counting categories require higher ratios (0.7–0.9) in MMVet. These results confirm that effective token reduction requires class-aware adaptation of both layer-wise feature integration and the balance between relevance and coverage.

• We reveal the limitations of visual feature extraction from fixed ViT layers and the reliance on fixed attention or fixed similarity mechanisms.

In summary, our main contributions are as follows:

• We propose class-adaptive layer fusion to extract more category-relevant visual representations. • We introduce a class-adaptive dual-stage pruning strategy to balance the relevance and coverage requirements of visual tokens. • We evaluate our method across various benchmarks, demonstrating consistent improvements in the accuracy–efficiency trade-off over existing approaches.

2. Related Work Vision-language Models (VLMs). Multimodal large language models (MLLMs) enable visual reasoning by coupling vision encoders with LLMs through alignment interfaces (?), such as projectors (Liu et al., 2023a; 2024a) or query-based connectors (Li et al., 2023b; Dai et al., 2023). To enhance these capabilities, subsequent systems have focused on scaling vision backbones, training data, and instruction-tuning recipes (Zhu et al., 2023; Wang et al., 2024; Chen et al., 2024b; Zhu et al., 2025; Yang et al., 2026). However, as these models scale, the serialization of redundant visual inputs into long token sequences imposes a significant computational burden. This burden becomes particularly acute when handling high-resolution images, where the resulting explosion in sequence length leads to

Motivated by this evidence, we propose class-adaptive layer fusion and dual-stage pruning (CLASP), a framework that 2

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

prohibitive inference latency (Liu et al., 2023a; 2024b; An et al., 2025). Beyond efficiency, current MLLMs still struggle with fine-grained visual perception, often leading to perception gaps and hallucinations (Li et al., 2023c; Yu et al., 2024; Dang et al., 2025). These perceptual shortcomings indicate a pressing need for more faithful visual representations and scalable context handling (Zou et al., 2025b; Wen et al., 2025b). A primary structural bottleneck is that most projector-based models extract tokens from only a single, static and often late, vision-encoder layer. By relying on such a fixed extraction point, these models fail to dynamically adapt the level of visual abstraction to the specific requirements of diverse instructions.

token indices. The vision encoder produces layer-wise patch (l) (l) dv token representations {Zn }L lies in l=1 , where zn,t ∈ R the vision-encoder embedding space: (l)

Z(l) n ≜ {zn,t }t∈Vn ,

l = 1, . . . , L.

(1)

The interface module (e.g., a projector) maps the selected/fused visual representations into the decoder embedding space, yielding decoder-space visual tokens Z̃n , which are concatenated with the text embeddings of the prompt xn . The decoder generates yn autoregressively: Y p(yn | xn , Z̃n ) = p yn,i | yn,<i , xn , Z̃n , (2) i

Visual Token Compression and Pruning. Visual token compression and pruning have emerged as pivotal techniques to alleviate the quadratic attention cost of long sequences, particularly as MLLMs increasingly handle highresolution inputs. To achieve such efficient reduction, various methodologies have been developed, differing in their importance signals and optimization goals, such as instruction relevance versus redundancy. Attention-based approaches, like FastV (Chen et al., 2024a) and VTW (Lin et al., 2025b), typically prune tokens using cross-modal attention patterns, while PyramidDrop (Xing et al., 2024) operates on intra-modal attention inside the vision backbone. Instruction-guided methods like SparseVLM (Zhang et al., 2024b) further strengthen the role of textual instructions to condition these attention signals. However, attention-based criteria are often noisy and tend to retain highly duplicated tokens. To address this, similarity-driven methods (e.g., ToMe (Bolya et al., 2022), VisionZip (Yang et al., 2025), HiRED (Arif et al., 2025), VisPruner (Zhang et al., 2025a)) leverage intrinsic visual cues to manage redundancy and maintain representation diversity. Nonetheless, these approaches are fundamentally limited by a fixed-layer visual readout, which fails to provide the adaptive level of visual abstraction required by diverse tasks. Furthermore, they employ a rigid pruning policy that cannot balance instruction relevance and visual coverage according to the specific intent category. In contrast, CLASP leverages the inferred instruction category to dynamically guide both multi-layer visual feature fusion and hybrid attention-similarity pruning.

Attention Mechanism for Token Relevance. Long dense visual token sequences are a primary efficiency bottleneck in MLLMs. A common training-free strategy is to use attention weights as a proxy for token relevance and perform token reduction accordingly (e.g., by retaining the most attended tokens) (Zhang et al., 2024b; Wen et al., 2025b; Zhang et al., 2025a). Given visual token representations Z̃n (as part of the input sequence) participating in an attention block (e.g., a vision-encoder or multimodal-decoder layer), selfattention is: QK⊤ Attn(Q, K, V) = softmax √ V, (3) dk with Q, K, V denoting the query/key/value projections and dk the key√dimension (per head). Let An = softmax(QK⊤ / dk ) be the corresponding attention matrix (mean over heads in practice), where An,i,t denotes the attention mass assigned by query token i to key token t. To score a visual token t ∈ Vn , we measure the attention it receives from a compact reference set S (e.g., instruction tokens or designated summary/query tokens): ϕn,t =

1 X An,i,t , |S|

t ∈ Vn .

(4)

i∈S

Given a retention budget K, attention-based selection keeps the top-K highest-scoring tokens: Pn = TopK {ϕn,t }t∈Vn , (5) where TopK (·) returns indices of the K largest values.

3. Preliminary

Similarity Mechanism for Coverage. Attention is a strong relevance cue, yet attention-only pruning may retain many near-duplicate tokens and can be unstable under attention shift/dispersion (Zhang et al., 2025a; Wen et al., 2025b). Therefore, several approaches incorporate similarity-based signals to ensure broader coverage (Zhang et al., 2025a).

Architecture of MLLMs. We consider a multimodal LLM consisting of (i) a vision encoder with L layers, (ii) a lightweight vision–text interface (e.g., a projector), and (iii) a causal text decoder. Specifically, the vision encoder extracts visual features from input images, which are then aligned with the text embedding space to guide the generation process. For sample n, let Vn denote the set of visual

Given token vectors {z̃n,t }t∈Vn , define normalized features un,t = z̃n,t /∥z̃n,t ∥2 so that sim(t, t′ ) = u⊤ n,t un,t′ is cosine 3

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Visual Tokens Text Tokens Class

What color is the ski suit?

classifier

0 1 2 ...... 7 8

R·a

R·(1-a)

ViT Layers

layer 0

Fusion Weights

Data Flow

layer 1

......

Weight

Tokens Link

Stage2

Text

Chosen Class

Class Link

Stage1

Text Tokenizer

layer 23

Image

Input

Attention Map

2 Pruning Ratio (a) Top-k Visual Tokens

Class-Adaptive Layer Fusion

Decoder

cluster 3

“The color is red.”

Class-Adaptive Pruning

Output

Figure 2. Overview of our method. The framework utilizes a prompt-to-class router to condition visual processing on textual intent. (i) Class-Adaptive Layer Fusion: ViT features are aggregated from multiple layers using class-specific mixture weights to capture an appropriate level of visual abstraction. (ii) Class-Adaptive Pruning: The projected visual token budget R is dynamically split (ratio a) between attention-based selection (Stage 1) and similarity-based clustering (Stage 2) to balance instruction relevance with visual coverage.

similarity. Let Pn be the attention-selected pivot set in Eq. (5). For any candidate token t ∈ Vn \ Pn , we define its worst-case redundancy with respect to the pivots as: ρn,t = max u⊤ n,t un,j . j∈Pn

pruning depends on the instruction category: detail-centric instructions (e.g., OCR, counting) benefit from retaining mid/shallow cues, whereas holistic instructions (e.g., scene understanding) prefer deeper abstractions. Therefore, instead of extracting visual representations from a fixed depth, we construct a prompt-conditioned depth mixture that adaptively balances fine-grained and semantic representations.

(6)

A smaller ρn,t indicates that token t is weakly covered by the pivot set (low cosine overlap) and thus provides more complementary information, which is preferred for enhancing contextual coverage.

Prompt-to-class Routing. We assume a discrete category space C = {0, 1, . . . , C − 1} defined by a text taxonomy (e.g., benchmark question types or a user-defined schema). A lightweight router Route(·) is employed to map the input prompt xn to a category index:

4. Method Overview. Given a prompt xn and an input image In , our goal is to reduce the effective number of visual tokens processed by the multimodal decoder to a target budget R, while preserving instruction-relevant content and maintaining token coverage. Following SparseVLM (Zhang et al., 2024b), we adopt a three-stage progressive pruning schedule and sparsify the visual token set at intermediate decoder layers {2, 6, 15}. The stage-wise retention budgets are configured to match the target effective budget R. As illustrated in Figure 2, our method introduces two classadaptive components driven by a prompt-to-class router: (i) class-adaptive layer fusion, which forms a categoryconditioned mixture of multi-layer vision representations, and (ii) class-adaptive pruning, which allocates each stage budget between attention-salient pivots (relevance) and lowredundancy completion tokens (coverage). The complete procedure is summarized in Algorithm 1.

cn ∈ C.

cn = Route(xn ),

(7)

The text-only router runs before token reduction, incurring negligible overhead compared to multimodal decoding. Category-conditioned Layer Mixture. To encode category-specific layer preferences, we maintain a score matrix W ∈ RC×L , where the c-th row wc assigns unnormalized importance to each vision layer (detailed in B). For sample n, we convert wcn to mixture weights via a temperature-controlled softmax αn = softmax(τ wcn ): αn = {αn,l }L l=1 ,

L X

αn,l = 1,

(8)

l=1

where τ > 0 controls the sharpness of the mixture (small τ : uniform averaging; large τ : near one-hot selection). This formulation strictly generalizes fixed-layer heuristics: choosing a single layer corresponds to a one-hot αn .

4.1. Class-Adaptive Layer Fusion Motivation and Design Principle. Vision backbones in MLLMs (e.g., ViT-style encoders) exhibit a well-known depth hierarchy: shallower layers preserve local texture, edges, and fine spatial layouts, while deeper layers become increasingly semantic and invariant. Our study (Sec. 1, Fig. 1) indicates that the optimal feature granularity for

(l)

Token-wise Fusion. Let Zn = {zn,t }t∈Vn be layer-wise patch representations as in Eq. (1). Directly mixing intermediate activations across layers can be sensitive to scale differences. We then fuse multi-layer features via adaptive 4

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Algorithm 1 Class-adaptive layer fusion and pruning.

tokens (useful for detail-centric prompts), while small an allocates more slots to coverage completion (useful for holistic prompts). Given total budget R, we compute:

(l) layer-wise tokens {Zn }L l=1 with

Require: prompt xn ; indices Vn ; final budget R; progressive schedule {R(2) , R(6) , R(15) }; router Route(·); layer-score matrix W; split ratios a; temperature τ ; projector fproj ; reference set S 1: cn ← Route(xn ) 2: αn ← softmax(τ P wcn ) (l) L 3: Z̃n ← fproj α Z n n,l l=1 4: V ← Vn 5: for (ls , Rs ) ∈ {(2, R(2) ), (6, R(6) ), (15, R(15) )} do 6: Compute relevance {ϕn,t }t∈V at layer ls 7: K1 ← ⌊acn Rs ⌋; K2 ← Rs − K1 8: P ← TopK1 ({ϕn,t }t∈V ) (Eq. (12)) Initialize centers using BottomK2 ({ρn,t (P)}) 9: (Eq. (13)) 10: Refine centers via Spherical K-Means (T iters) (Eqs. (14), (15)) 11: Q ← Select cluster medoids (Eqs. (16), (17)) 12: V ←P ∪Q 13: end for 14: return {z̃n,t }t∈V

K1 = ⌊an R⌋ ,

Z̄n ≜ {z̄n,t }t∈Vn ,

z̄n,t =

(l) αn,l zn,t .

This stage is intentionally aggressive in preserving queryor summary-attended evidence and acts as an “anchor” set that protects small but critical local visual regions (e.g., text glyphs, counting targets). Appendix C formalizes the optimality of TopK1 for additive relevance. Stage II: Coverage Completion via Redundancy-aware Clustering. Attention pivots alone can be redundant: multiple high-attention tokens may correspond to near-identical patches. To complement them, we allocate K2 slots to a clustering-based completion stage that explicitly improves coverage, consistent with the overview depicted in Fig. 2. Let us define Un = Vn \ Pn and the normalized features un,t = z̃n,t /∥z̃n,t ∥2 . We first measure redundancy of a candidate token in the pivot set as: ρn,t (Pn ) = max u⊤ n,t un,j ,

t ∈ Un .

j∈Pn

(9)

l=1

(13)

Instead of directly taking BottomK2 , we use ρn,t to construct a deterministic, redundancy-minimizing initialization for the clustering phase: we choose the K2 least redundant tokens as seeds and set their features as initial centers. We then run T iterations of spherical K-means (Hornik et al., 2012) on {un,t }t∈Un computed with cosine similarity:

The convexity is deliberate: it provides stability and interpretability (see Appendix C for formal properties). Finally, the original frozen MLP projector is employed to map the fused features into the decoder embedding space: Z̃n = fproj (Z̄n ),

(11)

Stage I: Attention Pivots for Relevance Preservation. We select K1 pivot tokens that maximize total relevance: Pn = TopK1 {ϕn,t }t∈Vn . (12)

weights by a convex combination: L X

K2 = R − K1 .

Z̃n = {z̃n,t }t∈Vn , z̃n,t ∈ Rd . (10)

(r)

sn,t = arg

While we assume ViT-style token alignment, hierarchical backbones are supported via a simple token-alignment mapping to a common grid, and fusion adds only an O(L |Vn | dv ) overhead (Appendix B.2 and Appendix B.6).

P (r) µn,k =

(r−1)

max

k∈{1,...,K2 } (r)

un,t

(r)

un,t

t:sn,t =k

u⊤ n,t µn,k ,

t:sn,t =k

4.2. Class-Adaptive Pruning

(14)

(15)

for r = 1, . . . , T . Finally, we select one representative (medoid) token per resulting cluster:

Given aligned visual tokens Z̃n = {z̃n,t }t∈Vn , we retain exactly R tokens. Conceptually, we want a subset that (i) preserves instruction-critical evidence (relevance) and (ii) avoids wasting budget on redundant patches (coverage). A single scalar scoring rule cannot reliably satisfy both criteria across heterogeneous instruction types. We therefore instantiate pruning as an explicit two-stage procedure, and make the relevance–coverage budget split category-dependent.

(T )

qn,k = arg max u⊤ n,t µn,k ,

(16)

2 Qn = {qn,k }K k=1 .

(17)

(T ) t:sn,t =k

Geometrically, the redundancy-aware seeding enforces angular separation from the pivot set (low ρn,t ), while the clustering refinement reduces mutual duplication among completion tokens by encouraging them to cover multiple modes on the unit sphere. We use T =5 iterations by default following the ablation in Table 13.

Category-dependent Split Ratio. We maintain a classwise ratio vector a ∈ [0, 1]C and set an ≜ acn . Intuitively, large an allocates more slots to attention-salient 5

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

4.3. Retained Tokens and Decoder Inference.

LLaVA-1.5-7B across seven diverse and challenging imageunderstanding benchmarks. For a fair comparison, we report the raw task scores and additionally normalize the vanilla 576-token model to a 100% upper bound, using the resulting normalized average to compare methods. Across three retention budgets (192/128/64 tokens; ↓66.7%/↓77.8%/↓88.9%), our method consistently achieves the best normalized average (98.4%/97.0%/94.7%). With 192 tokens, we retain 98.4% overall and obtain the strongest GQA result (60.4) together with competitive performance on the remaining benchmarks, while improving over classical reduction baselines by large margins (e.g., +9.9% and +10.6% over ToMe (Bolya et al., 2022) and FastV (Chen et al., 2024a)). At 128 tokens, we maintain 97.0% overall and reach the best POPE accuracy (85.2), suggesting improved faithfulness under tight budgets. Even with 64 tokens, our method preserves 94.7% overall, outperforming the similarity-based method DART (Wen et al., 2025b) by +1.7%, while further extending the margin over attention-based methods (+22.0% over PDrop (Xing et al., 2024) and +23.6% over FastV (Chen et al., 2024a)). Overall, these results show that our method preserves critical visual evidence more effectively, especially in the extreme low-token setting.

The final retained index set is constructed as the union Pn ∪ Qn , thereby yielding a total of exactly R tokens: {z̃n,t | t ∈ Pn ∪ Qn },

Pn ∪ Qn = R.

(18)

We feed the retained visual tokens as a compact prefix to the decoder. Conditioned on xn and the retained tokens, the decoder generates the response yn autoregressively: p yn | xn ,{z̃n,t }t∈Pn ∪Qn Y = p yn,i | yn,<i , xn , {z̃n,t }t∈Pn ∪Qn . i

(19) This hybrid formulation ensures that the language decoder receives a compressed yet representative visual context, where Pn safeguards task-critical relevance while Qn maintains global representational coverage.

5. Experiment Benchmarks. We evaluate multimodal capability on ten established benchmarks spanning general visual understanding, compositional reasoning, OCR-centric reasoning, realworld robustness, object hallucination, and video comprehension. For image understanding, we use GQA (Hudson & Manning, 2019), MMBench (Liu et al., 2024d), MME (Fu et al., 2025), POPE (Li et al., 2023c), VQAv2 (Goyal et al., 2017), ScienceQA (Lu et al., 2022), and TextVQA (Singh et al., 2019). To assess temporal reasoning and video capability, we further include TGIF (Li et al., 2016) for animated GIF description, MSVD (Chen & Dolan, 2011) for video captioning, and MSRVTT (Xu et al., 2016) for open-domain video description. We follow the official dataset splits and evaluation protocols, reporting the standard metrics for each benchmark to ensure reproducibility and fair comparison.

Main Results on Higher Resolution. For further comprehensive evaluation, we report results on LLaVA-NeXT 7B when reducing the visual tokens from 2880 to 320 (a ↓ 88.9% token reduction) in Table 2. Following the same evaluation protocol as Table 1, we report raw task scores and a normalized average where the 2880-token model is set to 100% for cross-benchmark comparison. Under this aggressive budget, our method achieves the best normalized average of 95.2%, improving over the similarity-based pruning method DART (Wen et al., 2025b) by 1.3%, and outperforming recent competitors such as HiRED (Arif et al., 2025) by an even larger margin (+1.9%). In addition to the overall ranking, our method attains the best scores on four out of seven benchmarks, including GQA (62.7), MME (1723), POPE (85.8), and TextVQA (61.7), indicating stronger retention of task-critical visual evidence under severe token reduction. These results show that our pruning strategy scales well to higher-resolution token settings and remains highly effective, preserving accuracy while substantially shrinking the visual context length.

Models and Comparison Methods. We apply our method to various MLLM architectures, including the LLaVA series: LLaVA-1.5 (Liu et al., 2024a) for image understanding, LLaVA-NeXT (Liu et al., 2024b) for high-resolution inputs, and Video-LLaVA (Zhang et al., 2024c) for video understanding, as well as the open-source model Qwen2.5VL (Bai et al., 2025). For efficiency-oriented comparisons, we benchmark against state-of-the-art token reduction methods for MLLMs, including ToMe (Bolya et al., 2022), LLaVA-PruMerge (Shang et al., 2025), FastV (Chen et al., 2024a), HiRED (Arif et al., 2025), PDrop (Xing et al., 2024), Multi-Stage Vision Token Dropping (Liu et al., 2024c), SparseVLM (Zhang et al., 2024b), VisionZip (Yang et al., 2025), and DART (Wen et al., 2025b). More details regarding the model architectures and additional experimental results are provided in Appendix A and Appendix D.

Results on Qwen Architecture. Beyond LLaVA, we extend our method to Qwen2.5-VL-7B (Bai et al., 2025) to validate its generalizability, evaluating it under three token pruning rates (66.7%, 77.8%, and 88.9%). As shown in Table 3, our method consistently retains a larger fraction of the upper-bound performance than SparseVLM (Zhang et al., 2024b) at every budget, and the advantage widens as pruning becomes more aggressive. We improve the normalized average from 94.1% to 96.5% at 66.7% pruning (+2.4%), from 90.8% to 94.4% at 77.8% pruning (+3.6%), and from 82.9%

Main Results. In Table 1, we compare our method with representative token merging and dropping methods on 6

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Table 1. Performance comparison of various methods on LLaVA-v1.5-7B across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue. Methods

GQA

MMB

MME

POPE

SQA

VQAV2

VQAText

Average

Upper Bound, 576 Tokens

61.9

64.7

1862

85.9

69.5

78.4

58.2

100.0%

52.1 52.5 56.5 54.3 56.1 55.7 47.4 57.3 56.1 57.4 57.6

88.5% 87.8% 96.6% 90.2% 96.0% 95.4% 93.9% 98.1% 95.9% 98.1% 98.4%

LLaVA-1.5 7B ToMe (ICLR23)(Bolya et al., 2022) FastV (ECCV24)(Chen et al., 2024a) MustDrop (2024.11)(Liu et al., 2024c) LLaVA-PruMerge (ICCV25)(Shang et al., 2025) PDrop (CVPR25)(Xing et al., 2024) FiCoCo-V (2025.03)(Han et al., 2024) HiRED (AAAI25)(Arif et al., 2025) VisionZip (CVPR25)(Yang et al., 2025) SparseVLM (ICML25)(Zhang et al., 2024b) DART (EMNLP25)(Wen et al., 2025b) CLASP (ours)

54.3 52.7 58.2 54.3 57.1 58.5 58.7 59.3 57.6 58.9 60.4

60.5 61.2 62.3 59.6 63.2 62.3 62.8 64.5 62.5 63.6 61.3

Retain 192 Tokens (↓ 66.7%) 1563 72.4 65.2 68.0 1612 64.8 67.3 67.1 1787 82.6 69.2 76.0 1632 71.3 67.9 70.6 1766 82.3 68.8 75.1 1732 82.5 67.8 74.4 1737 82.8 68.4 74.9 1767 86.4 68.9 76.8 1721 83.6 69.1 75.6 1856 82.8 69.8 76.7 1848 85.6 69.6 77.1

52.4 49.6 56.9 53.3 56.0 57.6 57.2 57.6 56.0 57.9 58.9

53.3 56.1 61.1 58.1 61.1 61.1 61.5 63.4 60.0 63.2 60.7

Retain 128 Tokens (↓ 77.8%) 1343 62.8 59.6 63.0 1490 59.6 60.2 61.8 1745 78.7 68.5 74.6 1554 67.2 67.1 68.8 1644 82.3 68.3 72.9 1711 82.2 68.3 73.1 1710 79.8 68.1 73.4 1768 84.7 68.8 75.6 1696 80.5 67.1 73.8 1845 80.1 69.1 75.9 1790 85.2 69.0 76.7

49.1 50.6 56.3 54.3 55.1 55.6 46.1 56.8 54.9 56.4 56.7

80.4% 81.2% 94.6% 87.9% 93.6% 94.6% 91.9% 96.8% 93.3% 96.7% 97.0%

48.6 46.1 53.1 51.9 41.9 52.4 54.6 55.1 52.7 55.9 57.0

43.7 48.0 60.0 55.3 33.3 60.3 60.2 60.1 56.2 60.6 59.1

Retain 64 Tokens (↓ 88.9%) 1138 52.5 50.0 57.1 1256 48.0 51.1 55.0 1612 68.0 63.4 69.3 1549 65.3 68.1 67.4 1092 55.9 68.6 69.2 1591 76.0 68.1 71.3 1599 73.6 68.2 69.7 1690 77.0 69.0 72.4 1505 75.1 62.2 68.2 1765 73.9 69.8 72.4 1709 82.8 69.8 75.2

45.3 47.8 54.2 54.0 45.9 53.6 44.2 55.5 51.8 54.4 55.2

70.1% 71.1% 88.1% 86.5% 72.7% 90.4% 88.0% 92.8% 86.5% 93.0% 94.7%

to 89.0% at 88.9% pruning (+6.1%). These results show that our pruning strategy transfers effectively to Qwen2.5VL. The gains are most pronounced in high-compression settings, where preserving diverse yet instruction-relevant visual evidence is critical, highlighting strong generalizability on advanced MLLM architectures.

repetitive or non-informative tokens, our method likely reduces noise and allows the model to focus more effectively on core temporal semantics. Overall, these findings confirm that our relevance-diversity pruning mechanism is highly effective for video understanding. Efficiency Analysis. Table 5 compares the computational cost and performance of our method against the full-token upper bound and the baseline SparseVLM (Zhang et al., 2024b) on the POPE benchmark. Our approach consistently achieves a superior trade-off between efficiency and accuracy. With a token budget of R = 192, we attain a 1.5× end-to-end speedup while retaining 99.6% of the upper-bound performance, significantly outperforming SparseVLM (1.2× speedup, 97.3% accuracy) in both throughput and fidelity. Notably, in the highly aggressive compression setting (R = 58), our method demonstrates robust resilience, delivering a 2.1× speedup with only a 4.6% relative performance drop, whereas SparseVLM suffers a sharp degradation to 92.3%. These results confirm that the syn-

Main Results on Video Benchmarks. To verify the versatility of our framework beyond static images, we extend our evaluation to the video domain using Video-LLaVA. As presented in Table 4, we compare CLASP against the fulltoken Upper Bound and the baseline SparseVLM on three representative benchmarks (TGIF, MSVD, and MSRVTT). Remarkably, CLASP achieves the highest average score of 52.95, surpassing not only SparseVLM (51.22) but also the unpruned Upper Bound (52.23). This performance gain over the full-token baseline is particularly evident on TGIF (45.65 vs. 43.47) and MSRVTT (51.58 vs. 51.28). We attribute this to the high temporal and spatial redundancy inherent in video streams. By aggressively filtering out

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Table 2. Performance comparison of various methods on LLaVA-NeXT-7B across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue. Methods

GQA

MMB

MME

POPE

SQA

VQAV2

VQAText

Average

Upper Bound, 2880 Tokens

64.2

67.4

1851

86.5

70.1

81.8

64.9

100.0%

61.6 61.3 63.4 62.8 61.6 64.2 60.6 65.3 61.2

Retain 320 Tokens (↓ 88.9%) 1661 71.7 62.8 71.9 1534 60.8 66.4 69.7 1663 77.6 67.5 73.5 1641 82.1 68.0 73.7 1701 83.6 66.5 74.0 1690 83.3 66.7 75.7 1533 82.4 66.1 71.5 1710 84.1 68.4 79.1 1723 85.8 67.0 78.0

55.7 50.6 54.4 59.9 56.5 58.8 58.4 58.7 61.7

88.0% 85.6% 90.9% 92.2% 91.1% 93.3% 89.6% 93.9% 95.2%

LLaVA-NeXT 7B FastV (ECCV24)(Chen et al., 2024a) LLaVA-PruMerge (ICCV25)(Shang et al., 2025) PDrop (CVPR25)(Xing et al., 2024) MustDrop (2024.11)(Liu et al., 2024c) FasterVLM(ICCV25)(Zhang et al., 2024a) HiRED (AAAI25)(Arif et al., 2025) SparseVLM (ICML25)(Zhang et al., 2024b) DART (EMNLP25)(Wen et al., 2025b) CLASP (ours)

55.9 53.6 56.4 57.3 56.9 59.3 56.1 61.7 62.7

Table 3. Performance comparison on Qwen2.5-VL-7B across widely-used benchmarks. Best results in blue. MME

POPE

SQA

VQAText

MMB

Avg

Upper Bound

2308

86.1

78.0

77.8

82.2

100%

Qwen2.5-VL-7B SparseVLM CLASP (ours)

2165 2197

TokenPruningRate = 66.7% 81.7 73.8 71.5 78.4 85.1 75.4 74.7 78.6

94.1% 96.5%

Qwen2.5-VL-7B SparseVLM CLASP (ours)

2086 2154

TokenPruningRate = 77.8% 77.7 72.8 68.7 75.6 83.0 74.5 73.1 76.2

90.8% 94.4%

Qwen2.5-VL-7B SparseVLM CLASP (ours)

1813 2010

TokenPruningRate = 88.9% 66.7 72.0 61.2 72.1 76.6 72.5 68.0 72.7

82.9% 89.0%

MSRVTT

Average

61.93

51.28

52.23

SparseVLM CLASP (ours)

44.67 45.65

59.29 61.62

49.69 51.58

51.22 52.95

Acc.

Upper Bound

100.0%

SparseVLM CLASP (ours)

40:51 33:08

15.8G 17.6G

97.3% 99.6%

1.2× 1.5×

Upper Bound

Retain 58 Tokens (↓ 89.9%) 49:41 0.5ms 0.334s 19.0G

100.0%

SparseVLM CLASP (ours)

31:28 24:10

14.6G 17.6G

92.3% 95.4%

1.6× 2.1×

Layer 2 (R=303)

Layer 6 (R=110)

Layer 15 (R=36)

0.6ms 0.2ms

0.6ms 0.1ms

Layer 6 (R=110)

Layer 15 (R=36)

0.251s 0.216s

0.212s 0.155s Original

SparseVLMs

MSVD

43.47

Latency

Ours

TGIF

Upper Bound

Mem.

Retain 192 Tokens (↓ 66.7%) 49:41 0.5ms 0.334s 19.0G

Ours

Table 4. Performance comparison on Video-LLaVA-7B across video benchmarks (TGIF, MSVD, MSRVTT). Best results in blue. Method

Time

Layer 2 (R=303)

Prefill

∆

Method

Original SparseVLMs

Methods

Table 5. Efficiency analysis on POPE at different retention levels. ∆ denotes the speedup ratio in Time.

ergy of class-adaptive layer fusion and dual-stage pruning effectively identifies and preserves the most informationdense tokens, enabling substantial latency reduction while maintaining high accuracy.

Figure 3. Example visualization of the original image and the corresponding token-retention map.

Token Pruning Visualization. Complementing the quantitative results, Fig. 3 visualizes the spatial distribution of retained tokens across samples. Unlike baselines that often fail to retain salient regions, our method consistently preserves critical areas. We further categorize the retained visual regions: blue regions represent tokens selected based on attention scores, indicating high relevance, while red regions represent tokens selected based on similarity metrics, providing contextual coverage. This distinction emphasizes that our method balances different visual aspects: blue regions capture the primary foreground, while red regions ensure background completeness. As pruning rates increase, our method robustly retains these essential cues, effectively filtering redundancy while preserving the core semantics needed for reasoning. Overall, this qualitative evidence confirms that our pruning strategy preserves pivotal infor-

mation for visual understanding, ensuring robust alignment for downstream tasks.

6. Conclusion In this paper, we proposed CLASP, a framework synergizing class-adaptive layer fusion with dual-stage pruning to balance token relevance and spatial coverage. Our results demonstrate that dynamic reduction minimizes redundancy without compromising fine-grained perception, marking a fundamental shift from static to conditional visual encoding. Future work should aim to transition from manual heuristic priors to end-to-end learnable routing policies, alongside tighter hardware-aware co-design to fully realize efficient multimodal intelligence on edge and embodied systems. 8

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Impact Statement

Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185–24198, 2024b.

This work reduces compute, memory, and latency of MLLM inference by pruning redundant visual tokens without retraining, which can lower cost and energy per query and enable deployment on resource-constrained hardware. Improved efficiency may broaden access to multimodal systems in assistive and on-device settings. However, lowering the marginal cost of multimodal inference can also facilitate harmful uses, including privacy-invasive analytics, large-scale surveillance, and high-volume content generation. CLASP does not add new capabilities beyond faster inference, but responsible deployment remains important: practitioners should follow data minimization and consent practices, apply secure storage and access control for visual inputs, monitor misuse, and validate robustness under aggressive pruning before use in sensitive domains.

Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P. N., and Hoi, S. Instructblip: Towards generalpurpose vision-language models with instruction tuning. Advances in neural information processing systems, 36: 49250–49267, 2023. Dang, Y., Wang, D., Yang, J., Jiang, Y., Zhu, M., Yang, Y., Wang, C., Fan, Q., Li, W., and Gao, Y. Fuse-rsvlm: Feature fusion vision-language model for remote sensing. arXiv preprint arXiv:2512.24022, 2025.

Alvar, S. R., Singh, G., Akbari, M., and Zhang, Y. Divprune: Diversity-based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9392–9401, 2025.

Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.

An, X., Xie, Y., Yang, K., Zhang, W., Zhao, X., Cheng, Z., Wang, Y., Xu, S., Chen, C., Zhu, D., et al. Llavaonevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661, 2025.

Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904–6913, 2017.

Arif, K. H. I., Yoon, J., Nikolopoulos, D. S., Vandierendonck, H., John, D., and Ji, B. Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.

Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

References

Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608–3617, 2018.

Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.

Han, Y., Liu, X., Zhang, Z., Ding, P., Chen, J., Wang, D., Chen, H., Yan, Q., and Huang, S. Filter, correlate, compress: Training-free token reduction for mllm acceleration. arXiv preprint arXiv:2411.17686, 2024.

Chen, D. and Dolan, W. B. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 190–200, 2011.

Hornik, K., Feinerer, I., Kober, M., and Buchta, C. Spherical k-means clustering. Journal of statistical software, 50: 1–22, 2012.

Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large visionlanguage models. In European Conference on Computer Vision, pp. 19–35. Springer, 2024a.

Huang, Y., Ma, F., Shao, Y., Guo, J., Yu, Z., Cui, L., and Tian, Q. N\" uwa: Mending the spatial integrity torn by vlm token pruning. arXiv preprint arXiv:2602.02951, 2026. 9

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700– 6709, 2019.

Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023a. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023b.

Jin, Y., Li, J., Gu, T., Liu, Y., Zhao, B., Lai, J., Gan, Z., Wang, Y., Wang, C., Tan, X., et al. Efficient multimodal large language models: A survey. Visual Intelligence, 3 (1):27, 2025.

Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296–26306, 2024a.

Kong, Z., Li, Y., Zeng, F., Xin, L., Messica, S., Lin, X., Zhao, P., Kellis, M., Tang, H., and Zitnik, M. Token reduction should go beyond efficiency in generative models– from vision, language to multimodality. arXiv preprint arXiv:2505.18227, 2025.

Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llavanext: Improved reasoning, ocr, and world knowledge, 2024b.

Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.

Liu, T., Shi, L., Hong, R., Hu, Y., Yin, Q., and Zhang, L. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.10803, 2024c.

Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730–19742. PMLR, 2023b.

Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216–233. Springer, 2024d.

Li, K., Chen, X., Gao, C., Li, Y., and Chen, X. Balanced token pruning: Accelerating vision language models beyond local optimization. arXiv preprint arXiv:2505.22038, 2025.

Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022.

Li, Y., Song, Y., Cao, L., Tetreault, J., Goldberg, L., Jaimes, A., and Luo, J. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4641–4650, 2016.

Shang, Y., Cai, M., Xu, B., Lee, Y. J., and Yan, Y. Llavaprumerge: Adaptive token reduction for efficient large multimodal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22857– 22867, 2025.

Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large visionlanguage models. arXiv preprint arXiv:2305.10355, 2023c.

Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317–8326, 2019.

Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 conference on empirical methods in natural language processing, pp. 5971–5984, 2024.

Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024.

Lin, J., Chen, H., Fan, Y., Fan, Y., Jin, X., Su, H., Fu, J., and Shen, X. Multi-layer visual feature fusion in multimodal llms: Methods, analysis, and best practices. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4156–4166, 2025a.

Wen, Z., Gao, Y., Li, W., He, C., and Zhang, L. Token pruning in multimodal large language models: Are we solving the right problem? arXiv preprint arXiv:2502.11501, 2025a.

Lin, Z., Lin, M., Lin, L., and Ji, R. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5334–5342, 2025b.

Wen, Z., Gao, Y., Wang, S., Zhang, J., Zhang, Q., Li, W., He, C., and Zhang, L. Stop looking for important tokens in 10

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

multimodal language models: Duplication matters more. arXiv preprint arXiv:2502.11494, 2025b.

Zhang, Q., Liu, M., Li, L., Lu, M., Zhang, Y., Pan, J., She, Q., and Zhang, S. Beyond attention or similarity: Maximizing conditional diversity for token pruning in mllms. arXiv preprint arXiv:2506.10967, 2025b.

Wu, J., Gan, W., Chen, Z., Wan, S., and Yu, P. S. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pp. 2247–2256. IEEE, 2023.

Zhang, Y., Fan, C.-K., Ma, J., Zheng, W., Huang, T., Cheng, K., Gudovskiy, D., Okuno, T., Nakata, Y., Keutzer, K., et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417, 2024b.

Xing, L., Huang, Q., Dong, X., Lu, J., Zhang, P., Zang, Y., Cao, Y., He, C., Wang, J., Wu, F., et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247, 2024.

Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., and Li, C. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024c.

Xu, J., Mei, T., Yao, T., and Rui, Y. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288–5296, 2016.

Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.

Yang, J., Chen, A., Dang, Y., Fan, Q., Wang, C., Li, W., Miao, F., and Gao, Y. Annotation-free visual reasoning for high-resolution large multimodal models via reinforcement learning. arXiv preprint arXiv:2602.23615, 2026.

Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.

Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., and Jia, J. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19792–19802, 2025.

Zou, X., Lu, D., Wang, Y., Yan, Y., Lyu, Y., Zheng, X., Zhang, L., and Hu, X. Don’t just chase" highlighted tokens" in mllms: Revisiting visual holistic context retention. arXiv preprint arXiv:2510.02912, 2025a.

Yu, T., Zhang, H., Yao, Y., Dang, Y., Chen, D., Lu, X., Cui, G., He, T., Liu, Z., Chua, T.-S., et al. Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220, 2(3): 8, 2024.

Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. Zhan, Z., Kong, Z., Gong, Y., Wu, Y., Meng, Z., Zheng, H., Shen, X., Ioannidis, S., Niu, W., Zhao, P., et al. Exploring token pruning in vision state space models. Advances in Neural Information Processing Systems, 37:50952– 50971, 2024. Zhang, Q., Cheng, A., Lu, M., Zhuo, Z., Wang, M., Cao, J., Guo, S., She, Q., and Zhang, S. [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv e-prints, pp. arXiv–2412, 2024a. Zhang, Q., Cheng, A., Lu, M., Zhang, R., Zhuo, Z., Cao, J., Guo, S., She, Q., and Zhang, S. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20857–20867, 2025a. 11

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

A. Detailed Experiment Settings. Benchmarks and Metrics. We conduct experiments on a comprehensive suite of benchmarks to evaluate our model’s multimodal capabilities across both static and dynamic scenarios. For image understanding, we assess general perception, compositional reasoning, OCR, and hallucination using eight widely used datasets in our main evaluation: GQA (Hudson & Manning, 2019), MMBench (MMB) (Liu et al., 2024d), MME (Fu et al., 2025), POPE (Li et al., 2023c), ScienceQA (SQA) (Lu et al., 2022), VQAv2 (Goyal et al., 2017), TextVQA (Singh et al., 2019), and MMVet (Yu et al., 2023). Additionally, to further demonstrate the robustness of our approach, we provide supplementary evaluations on LLaVA-Bench (LLaVA-B) (Liu et al., 2023b), VizWiz (Gurari et al., 2018), MMBench-Chinese (MMB-CN) (Liu et al., 2024d), and SEED-Bench (SEED) (Li et al., 2023a) in the appendix. Furthermore, to verify that our pruning method maintains temporal reasoning and video comprehension capabilities, we evaluate on three standard video benchmarks: TGIF (Li et al., 2016), MSVD (Chen & Dolan, 2011), and MSRVTT (Xu et al., 2016). Unless otherwise specified, we report accuracy following the official evaluation protocols to ensure a fair comparison. GQA. GQA (Hudson & Manning, 2019) evaluates scene understanding and compositional reasoning. It is built upon images, structured scene graphs, and questions designed to probe fine-grained object attributes, spatial relations, and multistep reasoning. Generated from the Visual Genome dataset, GQA contains over 22 million questions across approximately 113K images. A key characteristic of GQA is its rigorous balancing of the answer distribution for each question group, which significantly mitigates language priors and forces models to rely on visual evidence rather than statistical correlations. In the context of model compression, GQA provides a diagnostic view of a model’s ability to jointly perceive and reason over complex visual scenes, serving as a stress test for preserving multi-hop reasoning capabilities after pruning. MMBench. MMBench (Liu et al., 2024d) provides a multi-dimensional evaluation framework organized in a three-level hierarchy of capabilities. Level-1 targets the two core abilities of perception and reasoning; Level-2 expands these into six sub-abilities; and Level-3 further refines the assessment into 20 fine-grained ability dimensions. Constructed from various sources, MMBench contains approximately 3,000 multiple-choice questions covering diverse domains. A distinguishing feature of MMBench is its CircularEval strategy, which inputs the same question with shifted options multiple times to the model. This mechanism effectively mitigates the model’s sensitivity to option ordering and random guessing. For VLM pruning, MMBench serves as a critical benchmark to verify that the compressed model maintains robust instruction-following and reasoning capabilities across a broad spectrum of tasks, rather than overfitting to specific patterns. MME. MME (Fu et al., 2025) is a comprehensive and quantitative benchmark designed to evaluate multimodal LLMs across 14 distinct subtasks. These subtasks are structurally categorized into two primary axes: Perception (e.g., coarsegrained recognition, OCR, color) and Cognition (e.g., commonsense reasoning, numerical calculation). MME employs concise instruction–answer pairs, predominantly in a "Yes/No" format, to minimize the influence of prompt engineering and reduce potential data leakage. With manually annotated samples, it facilitates a reliable measurement of multimodal performance. For pruning studies, MME is particularly useful for diagnosing whether parameter reduction disproportionately impacts basic visual perception or higher-order cognitive reasoning. POPE. POPE (Li et al., 2023c) is a specialized benchmark focusing on evaluating object hallucination in VLMs. It reformulates hallucination assessment as a series of binary "Yes/No" questions about the presence of specific objects in an image (based on MSCOCO validation sets). Crucially, POPE evaluates models under three distinct sampling settings to disentangle visual perception from language priors: Random (objects not in the image), Popular (frequent objects in the dataset), and Adversarial (objects that often co-occur with present objects but are absent). By reporting Accuracy, Recall, Precision, and F1 across these settings, POPE provides a robust quantification of hallucination tendencies. For pruned models, maintaining high performance on the Adversarial setting is a strong indicator that the compression has not compromised the model’s ability to ground answers in actual visual evidence. ScienceQA. ScienceQA (Lu et al., 2022) spans a wide array of domains, including natural, social, and language sciences. Questions are hierarchically categorized into 26 topics, 127 categories, and 379 skills, providing a diverse and comprehensive testbed for evaluating multimodal understanding and multi-step reasoning. Distinctively, ScienceQA includes annotated lectures and explanations, facilitating the assessment of Chain-of-Thought (CoT) capabilities. In the context of VLM pruning, ScienceQA is essential for evaluating whether the compressed model retains domain-specific knowledge and the ability to perform interpretable reasoning, ensuring that parameter reduction does not compromise the model’s capacity to 12

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

handle complex, knowledge-intensive tasks. VQAv2. VQAv2 (Goyal et al., 2017) is a standard open-ended VQA benchmark covering diverse real-world images and questions derived from the MS COCO dataset. A critical feature of VQAv2 is its design to minimize language bias through balanced pairs: for every question, the dataset includes complementary images that result in different answers (e.g., "Yes" vs. "No"). This structure compels the model to rely on visual evidence rather than exploiting statistical language correlations. With over 1.1 million questions and multiple human annotations per question, VQAv2 serves as a large-scale testbed for general visual question answering. For pruning, it provides a fundamental baseline to ensure that the compressed model retains generalized visual recognition capabilities and aligns visual features correctly with textual queries. TextVQA. TextVQA (Singh et al., 2019) evaluates reasoning over embedded text in images, addressing a specific capability often overlooked by general VQA benchmarks. Built upon images from the Open Images dataset, it comprises 45,336 questions that require the model to detect, recognize, and reason about text appearing in diverse scenes (e.g., signboards, book covers). Successful performance demands a tight integration of optical character recognition (OCR) and semantic reasoning, as the model must often transcribe specific text strings from the image to formulate the answer. For VLM pruning, TextVQA serves as a rigorous test of fine-grained feature preservation, verifying that the compressed model retains sufficient resolution and local attention to process small, symbol-rich visual elements without degradation. MM-Vet. MM-Vet (Yu et al., 2023) targets complex multimodal problem solving by defining six core vision–language capabilities: Recognition, Knowledge, OCR, Spatial Awareness, Language Generation, and Math. Beyond testing these in isolation, MM-Vet evaluates 16 distinct integrations of these capabilities (e.g., recognizing an object and then answering a knowledge-based question about it). The benchmark employs an LLM-based evaluation system (typically GPT-4) to score open-ended responses, offering a nuance that rigid string-matching metrics lack. For pruning, MM-Vet is instrumental in verifying that the compressed model retains the synergy required to chain multiple reasoning steps, ensuring that the removal of parameters does not sever the functional connections between different cognitive modules. LLaVA-Bench. LLaVA-Bench (Liu et al., 2023b) is designed to evaluate the capability of MLLMs in handling complex, open-ended visual instructions in real-world scenarios. Comprising a diverse set of “in-the-wild” images and detailed queries, it relies on GPT-4 as an impartial judge to score the model’s generated responses against reference answers. Unlike standard objective benchmarks, LLaVA-Bench captures the nuances of conversational fluency, helpfulness, and visual grounding. In the context of token pruning, it is crucial for verifying that aggressive visual compression does not degrade the model’s capacity for rich, free-form multimodal generation and its adherence to unpredictable human instructions. VizWiz. VizWiz (Gurari et al., 2018) originates from authentic visual questions asked by blind and visually impaired individuals. Consequently, the dataset is characterized by images with severe real-world noise, such as poor lighting, blurriness, and occlusions, coupled with occasionally unanswerable queries. This benchmark rigorously tests a model’s robustness and zero-shot generalization under suboptimal visual conditions. For VLM pruning, VizWiz serves as a unique diagnostic tool: it assesses whether the pruning algorithm remains robust when visual cues are inherently scarce or distorted, ensuring that the token reduction process does not disproportionately discard critical, albeit noisy, visual information. MMBench-Chinese. MMBench-Chinese (MMB-CN) (Liu et al., 2024d) is the bilingual extension of the MMBench evaluation framework, translating the rigorous, multi-dimensional assessment into the Chinese linguistic context. Retaining the CircularEval strategy to mitigate option-selection bias, it evaluates the same broad spectrum of perception and reasoning capabilities but fundamentally requires robust cross-lingual alignment. Within our pruning study, MMB-CN is utilized to confirm that the class-adaptive layer fusion and token reduction mechanisms generalize effectively across diverse linguistic spaces, ensuring that cross-modal semantic alignment is not structurally impaired when evaluated in non-English languages. SEED-Bench. SEED-Bench (Li et al., 2023a) is a comprehensive multimodal benchmark specifically designed to evaluate fine-grained visual understanding and spatial reasoning. It comprises thousands of multiple-choice questions with meticulously crafted distractors, targeting specific dimensions such as object attributes, instance locations, and spatial relations. The precise nature of these queries forces models to rely on exact visual evidence rather than holistic semantic guessing. In the evaluation of pruned MLLMs, SEED-Bench acts as a stringent stress test for fine-grained token retention; it reveals whether our dual-stage pruning strategy successfully preserves the crucial, highly localized visual tokens required to distinguish between subtle, misleading options. 13

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

TGIF. TGIF (Li et al., 2016) is a dataset focusing on animated GIFs, which captures motion semantics and temporal dynamics. It contains over 100K animated images sourced from social media, paired with natural language descriptions. Unlike static image datasets, TGIF challenges models to reason about actions, repetitions, and state transitions over time. In the context of VLM pruning, TGIF serves as a critical benchmark to assess whether the compressed model preserves temporal consistency. Since pruning often involves reducing redundancy in visual tokens, performance on TGIF indicates whether the method effectively distinguishes between essential motion cues and redundant background frames. MSVD. MSVD (Chen & Dolan, 2011) constitutes a fundamental benchmark for video question answering, consisting of approximately 1,970 short video clips collected from YouTube. Unlike the captioning task, MSVD requires the model to provide precise answers to questions regarding video content based on spatio-temporal reasoning. For pruning studies, MSVD acts as a baseline sensitivity test. High performance here ensures that the reduction in model parameters has not destroyed the basic alignment between spatio-temporal visual features and the language decoder, serving as a sanity check for general video-to-text alignment. We report the accuracy following the official evaluation protocol to ensure consistency with other video benchmarks. MSRVTT. MSRVTT (Xu et al., 2016) is a dataset designed for open-domain video question answering, containing 10,000 video clips categorized into 20 distinct classes (e.g., music, sports, news). This diversity requires models to handle a wide vocabulary and complex visual scenes to answer questions regarding fine-grained events. In the context of model compression, MSRVTT is particularly useful for diagnosing "catastrophic forgetting" in the video domain. Maintaining high accuracy on MSRVTT implies that the pruned model retains robust feature representations across diverse categories and is not overfitting to specific visual patterns. Consistent with the official settings, we report accuracy as the primary metric. Models. We instantiate and evaluate CLASP on top of several representative open-source multimodal large language models (MLLMs). For image understanding, we primarily build on the LLaVA family, including the standard LLaVA1.5 (Liu et al., 2024a) and the stronger LLaVA-NeXT (Liu et al., 2024b). We specifically adopt LLaVA-NeXT to validate performance under high-resolution visual inputs, following the official inference settings. To demonstrate the generalization of our method across different architectures, we further include Qwen2.5-VL (Bai et al., 2025). As a state-of-the-art model, it introduces Naive Dynamic Resolution mechanisms to handle arbitrary aspect ratios, serving as a rigorous testbed for our pruning strategy on variable-length visual tokens. Finally, to assess capabilities in the temporal domain, we extend our evaluation to Video-LLaVA (Lin et al., 2024). This model unifies image and video feature alignment, allowing us to verify whether CLASP can effectively reduce temporal redundancies without compromising motion understanding. ToMe. ToMe (Bolya et al., 2022) accelerates vision transformers by merging similar tokens within transformer layers using lightweight token matching. Unlike traditional pruning methods that discard "unimportant" tokens, ToMe employs a bipartite matching algorithm based on feature similarity (typically using the Key or Query projections) to identify redundant tokens. It then aggregates these tokens via weighted averaging, progressively reducing the sequence length across the network depth. Since it requires no additional parameters or retraining, ToMe serves as a highly efficient, plug-and-play baseline for evaluating the trade-off between inference speed and information preservation in the VLM’s visual encoder. LLaVA-PruMerge. LLaVA-PruMerge (Shang et al., 2025) introduces an adaptive hybrid strategy that combines the benefits of both pruning and merging. It first evaluates the importance of visual tokens based on the attention scores from the special [CLS] token to image patches, effectively identifying regions most relevant to the global semantic context. Based on these scores, the method divides tokens into subsets: tokens with low importance scores are directly pruned to remove background noise, while highly correlated foreground tokens are merged based on key similarity. This approach significantly reduces the sequence length of the visual encoder without retraining, ensuring that critical visual details are preserved while spatial redundancy is efficiently eliminated before the tokens enter the LLM. FastV. FastV (Chen et al., 2024a) identifies the inefficiency of visual tokens within Large Language Models (LLMs) and performs early-stage token pruning. It is grounded in the observation that while visual tokens are crucial in the initial transformer layers, the attention mechanism in deeper layers tends to ignore the vast majority of them. Leveraging this, FastV ranks visual tokens based on their average attention weights in the early layers (e.g., the second layer) and discards the least significant ones for all subsequent layers. This strategy significantly reduces the KV-cache memory footprint and FLOPs during inference, allowing the model to maintain high performance while processing significantly fewer tokens in the computation-heavy deep layers. 14

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

HiRED. HiRED (Arif et al., 2025) introduces a spatially-aware pruning strategy designed to mitigate the "tunnel vision" problem often observed in global ranking methods. Instead of competing all visual tokens in a single pool, HiRED divides the image into spatial partitions. It then dynamically allocates a specific token budget to each partition based on the attention distribution of the [CLS] token. By selecting the most informative tokens within these localized budgets, HiRED ensures that the pruned model maintains broad spatial coverage across the image. This hierarchical approach allows for aggressive compression while preventing the complete suppression of background regions or secondary objects, which are essential for tasks requiring holistic scene understanding. PDrop. PDrop (Xing et al., 2024), or PyramidDrop, implements a progressive pruning strategy that mimics the hierarchical structure of CNNs within a standard Vision Transformer. Instead of maintaining a constant sequence length throughout the encoder, PDrop drastically reduces the token count at specific intermediate layers based on attention importance scores. This constructs a "pyramid-like" information flow, where high-resolution details are processed in shallow layers, while only the most semantically salient tokens are retained for the computation-heavy deep layers. By effectively funneling visual information, PDrop achieves a significant reduction in FLOPs and latency, serving as a representative baseline for structural pruning methods. Multi-Stage Vision Token Dropping (MustDrop). Multi-Stage Vision Token Dropping (Liu et al., 2024c) adopts a progressive compression strategy within the vision encoder to optimize computational efficiency. Instead of performing a single-step reduction, it executes token dropping at multiple intermediate layers (stages) based on attention significance or learned policies. This hierarchical approach allows the model to retain fine-grained low-level details in the shallow stages while aggressively reducing spatial redundancy in the deeper, more semantic layers. Consequently, it significantly lowers the FLOPs of the vision backbone and minimizes the sequence length passed to the multimodal projector, achieving a favorable balance between inference speed and task performance. SparseVLM. SparseVLM (Zhang et al., 2024b) presents a dynamic pruning framework that goes beyond uni-modal visual salience. It ranks token importance using cross-modal attention, ensuring that visual tokens are preserved based on their relevance to the specific textual query or instruction. Recognizing that different images contain varying amounts of information, SparseVLM introduces adaptive sparsity ratios, dynamically adjusting the token budget for each input. Furthermore, to mitigate the risks of aggressive pruning, it proposes a novel token recycling mechanism. Instead of permanently discarding pruned tokens, this mechanism aggregates or buffers them, allowing the model to retrieve context from these "discarded" regions if necessary. This significantly improves efficiency under varying input complexity without compromising the semantic integrity of the visual representation. VisionZip. VisionZip (Yang et al., 2025) addresses the spatial redundancy problem often found in attention-based pruning methods. While standard approaches simply select the top-k tokens with the highest attention scores, VisionZip observes that these high-scoring tokens tend to cluster around the same visual object, leading to repetitive information. To mitigate this, it implements a two-step pipeline: first, it evaluates token saliency via encoder attention to filter out background noise; second, it clusters the remaining high-saliency tokens based on key similarity. By selecting representative tokens from these clusters, VisionZip ensures that the final compressed sequence maintains high semantic diversity, covering various distinct objects and regions within the image for downstream multimodal reasoning. DART. DART (Wen et al., 2025b) proposes a Duplication-Aware Reduction Transformer strategy designed to maximize information diversity while maintaining hardware efficiency. Unlike magnitude-based methods that may select redundant high-norm tokens, DART explicitly filters out redundancy. It operates by selecting a small set of "pivot" tokens and computing the cosine similarity between these pivots and the remaining tokens. Tokens with high similarity (high duplication) are discarded, ensuring that the retained set covers a broad semantic range. Crucially, DART is designed to be compatible with FlashAttention and other efficient attention operators. By avoiding complex gather-scatter operations or irregular memory access patterns, it ensures that the theoretical reduction in FLOPs translates directly into significant wall-clock speedups during inference. Implementation details. For image-based benchmarks, we run experiments using the official LLaVA implementation. For video benchmarks, we build the model following the official LLaVA-NeXT codebase and conduct evaluation via lmms-eval. For more recent VLM architectures (e.g., Qwen2.5-VL), we rely on VLMEvalKit as the evaluation toolkit. All inference 15

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

evaluations were executed on NVIDIA A100 (80GB) GPUs. To ensure reproducibility, we detail the specific configurations for dataset construction, parameter validation, and model-specific architectures below. Dataset Categorization and Intent Mapping. To enable class-adaptive routing, we constructed a categorized dataset using the Qwen3-8B model to identify visual question answering intents. We designed a system prompt defining 15 fine-grained intents (e.g., arithmetic reasoning, temporal order) and instructed the model to output classifications in a strict JSON format. To enhance robustness, we applied deterministic heuristic overrides (e.g., mapping keywords like “who wrote” to Text/Symbol Recognition). These labels were merged into the 9 core categories (Table 6) to ensure sufficient sample density for routing. Hyperparameter Search and Optimization. To determine the optimal fusion weights and pruning ratios, we employed the Discrete Subspace Search Algorithm detailed in Appendix B. Specifically, we performed this calibration on a balanced dataset of 2,000 samples per category. Although fully unconstrained gradient-based optimization can risk overfitting on limited calibration data, we found that our approach—which restricts the search space to discrete, semantically meaningful layer prototypes—effectively prevents such overfitting. Crucially, our experiments indicated that this data-driven calibration yielded superior zero-shot generalization compared to task-specific manual heuristics. By automatically adapting to the level of visual abstraction required for each intent (e.g., balancing fine-grained details and high-level semantics), the search algorithm identifies more robust configurations. Consequently, all results reported in this paper utilize the calibrated configurations obtained via this search process, as provided in Table 6. Model-Specific Configurations. We devised distinct fusion strategies tailored to the architecture of the vision encoders. The LLaVA family (LLaVA-v1.5, LLaVA-NeXT, and Video-LLaVA) shares a unified set of fusion parameters, while Qwen2.5-VL employs a separate configuration adapted to its layer structure. The split ratios (ac ) for the two-stage pruning were kept consistent across all models. Pruning Schedule and Budgets. We adopt the progressive sparsification strategy from SparseVLM, executing token pruning at three designated intermediate layers of the multimodal decoder on the projected (decoder-space) visual tokens: Layer 2, Layer 6, and Layer 15. Following SparseVLM (Zhang et al., 2024b), we report an effective token budget R under progressive pruning; the stage-wise budgets [KL2 , KL6 , KL15 ] are chosen to match the same cumulative computational cost (FLOPs) as maintaining a constant sequence length of R throughout the decoder. Taking the standard input resolution (N = 576 projected visual tokens) as a baseline, the stepwise retention counts are configured as follows: (1) For the 192-token setting (R = 192), we retain [300, 200, 110] tokens respectively; (2) For the 128-token setting (R = 128), the schedule is set to [303, 110, 36]; (3) For the 64-token setting (R = 64), we employ an aggressive schedule of [66, 30, 17]. For high-resolution architectures (e.g., LLaVA-NeXT) processing N = 2880 tokens, these budgets are scaled proportionally. Relation to SparseVLM. For fair efficiency comparisons, we follow SparseVLM’s (Zhang et al., 2024b) progressive threestage pruning schedule in the multimodal decoder (Layer 2/6/15), while differing in how the prompt conditions the visual representation and how the token budget is allocated. SparseVLM ranks visual tokens using query-conditioned cross-modal attention and further mitigates aggressive sparsification via token recycling and adaptive sparsity ratios (Zhang et al., 2024b). In contrast, we adopt a lightweight text-only router and use its predicted category to drive two class-adaptive components: (i) class-adaptive multi-layer feature fusion, where token representations are formed by a temperature-controlled mixture over vision layers to match the level of visual abstraction required by the prompt; and (ii) attention–similarity adaptive-ratio pruning, where a fixed budget R is split into K1 = ⌊ac R⌋ attention-salient pivots (relevance) and K2 = R − K1 similaritybased completion tokens selected by minimizing redundancy under cosine similarity (diversity). This formulation explicitly combines relevance-driven attention selection (e.g., FastV (Chen et al., 2024a), HiRED (Arif et al., 2025), PyramidDrop (Xing et al., 2024)) with redundancy-aware retention strategies (e.g., ToMe (Bolya et al., 2022), VisionZip (Yang et al., 2025), DART (Wen et al., 2025b), DivPrune (Alvar et al., 2025), conditional-diversity pruning (Zhang et al., 2025b)), while keeping the backbone frozen and introducing negligible inference overhead.

B. Additional Method Details B.1. Calibration of W and a The adaptive parameters in our method are low dimensional (C × L layer scores and C split ratios) and can be set without finetuning any parameters of the underlying MLLM (vision encoder, projector, or LLM). We outline a practical calibration protocol to obtain stable per-category configurations. 16

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Table 6. Hyperparameters for class-adaptive layer fusion and pruning across MLLM backbones. Fusion weights are reported as {layer: coefficient}. The split ratio ac governs the prompt-conditioned budget allocation between attention-salient pivots for relevance and redundancy-aware completion tokens for coverage. ID

Category

GQA

MMB

MME

MMVet

POPE

SQA

VQAv2

VQAText

0 1 2 3 4 5 6 7 8

Object identification Attribute / breed identification Text / symbol recognition Scene understanding Spatial relations Counting Action / interaction Intention / function Default

2095 4015 75 2319 3632 9 200 43 190

272 268 111 1950 379 100 74 287 936

355 85 147 1209 70 87 2 79 340

21 19 12 58 18 5 4 9 72

2277 0 0 6633 0 0 0 0 0

178 410 465 521 193 54 1 8 2411

18263 19380 4368 35553 7378 10007 5850 3440 3155

239 1037 2856 388 39 53 8 121 259

cal Data and objective. Assume access to a small calibration set Dcal = {(xn , imgn , yn⋆ , cn )}N n=1 that contains (i) prompts xn , ⋆ (ii) images imgn , (iii) reference outputs yn (e.g., gold answers for QA / multiple-choice labels), and (iv) category labels cn ∈ Nc . The category labels can be obtained from benchmark taxonomies (e.g., question-type tags) or a lightweight manual/automatic annotation effort.

For a fixed effective token budget R, we select (W, a) to maximize the task score under pruning: max Score(Dcal ; R, W, a) W,a

s.t. ac ∈ [0, 1], ∀c.

(20)

Here Score(·) uses the same decoding and evaluation protocol as in the main experiments, but restricted to Dcal . Concretely, we write X 1 Eval(ŷn (R, W, a), yn⋆ ) , (21) Score(Dcal ; R, W, a) = |Dcal | ⋆ (xn ,imgn ,yn ,cn )∈Dcal

where ŷn (R, W, a) is the model prediction produced when applying our class-adaptive layer fusion and dual-stage pruning under budget R, and Eval(·) is the benchmark metric (e.g., exact-match accuracy for QA, multiple-choice accuracy for MMBench-style evaluation, or any dataset-specific scorer). Why calibration does not require MLLM finetuning. The calibration variables (W, a) are extremely low-dimensional and represent high-level architectural priors rather than dense feature mappings. Because the MLLM weights are frozen, the scoring function Score can be treated as a lightweight objective that does not require computing gradients through the large-scale backbone. While (W, a) could theoretically be optimized via forward-only black-box search or lightweight gradient updates, our empirical analysis reveals that the system is remarkably robust to these parameters. Specifically, we find that our discrete subspace search—which selects from structured prototypes tailored to the required level of visual abstraction (e.g., semantic-rich deep layers for identification vs. detail-oriented shallow layers for OCR)—consistently outperforms unconstrained continuous calibration in terms of zero-shot generalization. This inherent robustness allows CLASP to function as a true plug-and-play solution, bypassing the need for expensive calibration cycles or backward activations entirely. Per-class decomposition. Since cn selects the row wcn and ratio acn , the calibration naturally decomposes over categories: (c) each class c can be calibrated on the subset Dcal = {(xn , imgn , yn⋆ , cn ) ∈ Dcal : cn = c}. independently, which improves 17

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

stability and reduces search complexity. Algorithm 2 summarizes a simple per-class procedure. Efficient evaluation in practice. Calibration can be further accelerated by: (i) caching the frozen vision-encoder layer (l) outputs {Zn }L l=1 for each (xn , imgn ) once, so re-evaluating a candidate wc only requires a cheap weighted sum (fusion) plus the standard forward decode; (ii) using a small, balanced Dcal per category (hundreds to a few thousand samples typically suffice); and (iii) optionally using early rejection (evaluate candidates on a small subset first, and fully score only the top few). Impact of Calibration Set Size and Distribution. To rigorously justify our choices regarding the calibration dataset (Dcal ), we investigate how the size of the calibration set and its categorical distribution (sampling strategy) affect the final pruned model’s performance. We evaluate two data selection strategies: a Balanced distribution (enforcing an equal number of samples per intent category) and a Random distribution (naturally imbalanced based on the source dataset’s available distribution). We test total calibration sizes of 4.5k, 9k, and 18k samples. To account for statistical variance, we evaluate each configuration across five random seeds ([42, 43, 44, 45, 46]) during the redundancy-aware initialization and report the mean and standard deviation. As shown in Table 8, our offline search process is exceptionally sample-efficient. Performance strictly saturates around 9k samples, with negligible gains observed when doubling the size to 18k. Furthermore, in the data-scarce regime (4.5k), the Balanced sampling strategy significantly reduces variance compared to Random sampling (e.g., ±6 vs. ±15 on MME, and ±0.3 vs. ±0.6 on TextVQA). This empirically proves that maintaining a balanced category distribution is crucial for stabilizing the search space and avoiding overfitting to dominant task categories. Consequently, we confirm that using a moderately sized (around 9k), class-balanced calibration set is the optimal strategy to achieve high-performance configurations with minimal computational overhead. Table 8. Ablation on calibration set size and sampling strategy. Results are reported as mean ± standard deviation across 5 random seeds. The framework achieves optimal, low-variance performance with a balanced dataset of merely 9k samples, demonstrating excellent sample efficiency. Calibration Size

Sampling Strategy

MME

TextVQA

MMVet

4.5k 4.5k

Balanced (Avg) Random

1814 ± 6 1809 ± 15

56.6 ± 0.3 56.4 ± 0.6

32.0 ± 0.3 32.1 ± 0.4

9k 9k

Balanced (Avg) Random

1845 ± 3 1840 ± 5

57.5 ± 0.1 57.6 ± 0.2

33.2 ± 0.1 33.2 ± 0.2

18k 18k

Balanced (Avg) Random

1845 ± 2 1845 ± 3

57.5 ± 0.1 57.6 ± 0.1

33.3 ± 0.0 33.2 ± 0.1

Structured parameterization for fast search. To avoid overfitting and to reduce search complexity, we recommend a structured parameterization: (i) restrict wc to have support on a small candidate layer set (e.g., 3–5 layers spanning shallow/mid/deep), and (ii) restrict ac to a coarse grid (e.g., {0.2, 0.4, 0.6, 0.8} or a slightly denser grid when needed). This yields a compact search space while preserving the key adaptivity signals revealed by our motivation study. Optional continuous refinement (if desired). If one wants to refine beyond a discrete candidate set, one can keep the same constraints but allow non-uniform weights on the selected layers (still normalized by softmax), and use derivative-free optimizers (e.g., coordinate search / random search) on the low-dimensional parameters. This is optional and not required for stable gains. Empirical Cost of Offline Calibration and Online Routing. To quantitatively support the efficiency claims of our calibration procedure, we report the wall-clock time and peak GPU memory usage required for both the offline search and the online intent classification. The measurements were conducted on a single NVIDIA A100 (80GB) GPU. As detailed in Table 9, the offline calibration is highly resource-efficient. Thanks to the caching of frozen vision-encoder features (as discussed in our efficient evaluation strategy), the discrete subspace search for Layer Weights (W ) and Split Ratios (a) takes approximately 1.5 hours and 1.0 hour, respectively. The peak memory footprint remains well under 20GB, making this calibration completely feasible on consumer-grade hardware without requiring large-scale distributed clusters or gradient backpropagation. Furthermore, during online inference, the intent classifier introduces a negligible latency of less than 2ms and requires only 15.5GB of memory (which includes the loaded weights of the lightweight routing model), 18

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Algorithm 2 Discrete Subspace Search for Optimal Layer Prototypes and Pruning Ratios. Require: calibration set Dcal with labels cn ; candidate layer sets {L(m) }; candidate ratios A; budget R 1: for c ∈ Nc do 2: Initialize best score s⋆ ← −∞ 3: for layer candidate L(m) do 4: for a ∈ A do 5: Define wc supported on L(m) (e.g., uniform over L(m) , −∞ elsewhere) (c) 6: Evaluate Score on Dcal using (wc , a) ⋆ 7: if score > s then 8: Update (wc⋆ , a⋆c ) ← (wc , a); s⋆ ← score 9: end if 10: end for 11: end for 12: end for ⋆ 13: Return W = [w0⋆ ; . . . ; wC−1 ], a = [a⋆0 , . . . , a⋆C−1 ]

ensuring that dynamic class-adaptive routing does not become a system bottleneck. Table 9. Time and memory cost of offline calibration and online routing. The offline calibration of layer weights and split ratios takes only a few hours on a single GPU without requiring backpropagation. The online intent classifier introduces negligible latency. Component

Time Cost

Memory Usage

Intent Classifier (Online Inference) Calibration of Layer Weights (W ) (Offline) Calibration of Split Ratios (a) (Offline)

< 2 ms ≈ 1.5 h ≈ 1.0 h

15.5 GB 19.4 GB 17.6 GB

B.2. Fusion with hierarchical backbones Our exposition assumes token indices align across layers (a standard property of ViT-style encoders). For hierarchical encoders with resolution changes (e.g., feature pyramids), fusion can be supported by mapping each layer to a common token set before applying Eq. (9). Token alignment via spatial resampling. Let layer l output a grid of patch tokens with spatial resolution Hl × Wl (excluding any special token), and let the target resolution be H⋆ × W⋆ (typically the final layer resolution). We define an (l) (l) alignment operator Alignl (·) that maps Zn ∈ R(Hl Wl )×dv to Ẑn ∈ R(H⋆ W⋆ )×dv :   if Hl Wl < H⋆ W⋆ , bilinear/nearest interpolation (upsample) Ẑn(l) = Alignl Zn(l) , Alignl = average pooling / strided pooling (downsample) if Hl Wl > H⋆ W⋆ , (22)   identity if Hl Wl = H⋆ W⋆ . In practice, we reshape tokens back to a feature map of size Hl × Wl × dv , apply standard 2D resampling, and flatten back to a sequence. If the backbone includes a special token (e.g., [CLS]), we either (i) keep it separate and fuse it with the same mixture weights, or (ii) drop it when it is not used by the downstream projector. (l)

After alignment, layer fusion proceeds exactly as in Eq. (9) by mixing the aligned tokens {Ẑn }L l=1 . The computational overhead of fusion remains linear in the number of visual tokens (and linear in the number of fused layers); see Appendix B.6 for a step-wise complexity breakdown.

C. Additional Theoretical Analysis C.1. Notation and setup (l)

For sample n, let Mn ≜ |Vn | denote the number of visual patch tokens. Layer-wise tokens are zn,t ∈ Rdv for l ∈ {1, . . . , L} and t ∈ Vn . Class-adaptive fusion produces z̄n,t ∈ Rdv and aligned tokens z̃n,t = fproj (z̄n,t ) ∈ Rd . 19

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Selection operators. For a collection of real-valued scores {st }t∈I indexed by I, we denote by TopK ({st }t∈I ) the index set of the K largest scores, and by BottomK ({st }t∈I ) the index set of the K smallest scores. When ties occur, any deterministic tie-breaking rule suffices and does not affect the statements below. Simplex notation.

We denote the probability simplex by ∆

L−1

L n o X L γl = 1 . ≜ γ ∈ R : γl ≥ 0 (∀l),

(23)

l=1

Unit-normalization for cosine similarity.

Whenever cosine similarity is used, we work with ℓ2 -normalized features un,t ≜

z̃n,t ∈ Rd . ∥z̃n,t ∥2

(24)

This normalization isolates directional information and makes inner products u⊤ n,t un,j equal cosine similarity, which is the natural notion of redundancy/novelty on the unit sphere. Consequently, Euclidean distance on this manifold becomes strictly monotonic with respect to angular separation, simplifying the geometric proofs in Section C.4. C.2. Properties of class-adaptive layer fusion Fusion equations in execution order.

For clarity, we restate the fusion module as a short sequence of equations.

1. Routing and mixture weights. Given the prompt xn , a text-only router predicts a class cn , which selects a row wcn ∈ RL from the layer-score matrix W. We then convert it into a probability distribution over layers: cn ← Route(xn ),

αn ≜ softmax(τ wcn ) ∈ ∆L−1 .

(25)

Here τ > 0 is a temperature that controls how “peaky” the layer preference is. 2. Token-wise convex fusion across layers. For each token index t ∈ Vn , we fuse its layer-wise representations by a weighted sum. This dynamically integrates features from varying levels of visual abstraction: z̄n,t =

L X

(l)

αn,l zn,t .

(26)

l=1

3. Projection into the decoder space. The fused token is mapped to the decoder embedding space through fproj : z̃n,t = fproj (z̄n,t ) ∈ Rd . Lemma C.1 (To keep fused features inside the layer-wise convex hull, we fuse by a convex combination) For (l) sample n and token index t ∈ Vn , the fused token in Eq. (26) lies in the convex hull of {zn,t }L l=1 .

(27) any

PL Proof C.1 By Eq. (25), αn = softmax(τ wcn ) satisfies αn,l ≥ 0 and l=1 αn,l = 1, hence Eq. (26) is a convex combination. Geometrically, the fused vector lies inside the layer-wise feature polytope. Interpretation. Eq. (26) prevents “out-of-manifold” extrapolation across layers: fusion interpolates between representations at different depths but cannot invent directions outside their convex span. This is desirable when fusion weights are controlled by a text router, ensuring the resulting representation remains within the valid semantic latent space. Lemma C.2 (To interpolate between uniform averaging and hard layer selection, we analyze the temperature limits) Let α(τ ) = softmax(τ w) for any fixed w ∈ RL . (i) As τ → 0, α(τ ) → (1/L)1. (ii) As τ → ∞, α(τ ) converges to a one-hot distribution supported on arg maxl wl when the maximizer is unique. Proof C.2 (i) softmax(0 · w) is uniform; continuity gives the limit. (ii) For large τ , exp(τ wl ) is dominated by the largest wl , yielding the standard softmax one-hot asymptotics, effectively selecting the single maximal layer. 20

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Assumption 1 (To enable norm-based perturbation bounds, we assume bounded layer-wise token norms) There ex(l) ists B > 0 such that for all n, t, l, ∥zn,t ∥2 ≤ B, uniformly bounding the feature magnitude. Lemma C.3 (To control how routing-score noise changes fusion weights, we show softmax is ℓ1 -Lipschitz) Let α(u) = softmax(u). Then for any u, v ∈ RL , representing arbitrary input logit vectors, ∥α(u) − α(v)∥1 ≤

1 ∥u − v∥1 . 2

(28)

Consequently, with α(τ w) = softmax(τ w), applied to the temperature-scaled inputs, ∥α(τ w) − α(τ w′ )∥1 ≤

τ ∥w − w′ ∥1 . 2

(29)

Proof C.3 We make the dependence explicit in three steps, mirroring the execution logic. (a) Jacobian form. Let J(u) = ∇u α(u) with entries Jij = αi (δij − αj ). (b) Column-wise ℓ1 bound. For any column j, summing the absolute values of entries, X X X |Jij | = |αj (1 − αj )| + |− αi αj | = αj (1 − αj ) + αj αi = 2αj (1 − αj ) ≤ 12 , i

i̸=j

since maxx∈[0,1] 2x(1 − x) = 1/2. (c) Mean value theorem. The induced operator norm satisfies ∥J(u)∥1→1 ≤ 1/2 for all u. Therefore, ∥α(u) − α(v)∥1 ≤ sup ∥J(ξ)∥1→1 ∥u − v∥1 ≤ 12 ∥u − v∥1 . ξ

Eq. (29) follows by substituting u = τ w and v = τ w′ . Proposition 1 (To guarantee robustness to routing-score perturbations, we bound fused-token drift linearly in τ ) Under Assumption 1, for any two layer-score vectors w, w′ ∈ RL and fixed τ , ∥z̄n,t (w) − z̄n,t (w′ )∥2 ≤ If fproj is Lproj -Lipschitz, then ∥z̃n,t (w) − z̃n,t (w′ )∥2 ≤

Bτ ∥w − w′ ∥1 . 2

(30)

Bτ Lproj ∥w − w′ ∥1 . 2

Proof C.4 By applying the triangle inequality with the norm bound B and Lemma C.3, we have: ∥z̄(w) − z̄(w′ )∥2 ≤ B∥α(τ w) − α(τ w′ )∥1 ≤

Bτ ∥w − w′ ∥1 . 2

The projector bound follows from Lipschitzness of fproj , scaling the error by the constant. Practical reading. Eq. (30) makes the role of τ explicit: larger τ yields sharper (more class-distinct) depth preferences (Lemma C.2), but also linearly amplifies sensitivity to score perturbations. This justifies conservative τ (or annealing) to effectively dampen the amplification of routing noise when router uncertainty is non-negligible. Corollary 1 (To upper bound misrouting sensitivity, we relate class-to-class drift to inter-row distance in W) Let c and c′ be two classes with score vectors wc and wc′ . Under Assumption 1, the fused-token drift caused by misrouting c → c′ , which corresponds to the feature shift induced by the incorrect selection, satisfies ∥z̄n,t (wc ) − z̄n,t (wc′ )∥2 ≤

Bτ ∥wc − wc′ ∥1 . 2

Proof C.5 Apply Proposition 1 by substituting the specific class vectors w = wc and w′ = wc′ . 21

(31)

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

C.3. Pruning as an explicit relevance–coverage optimization with redundancy-aware seeding and clustering We analyze the class-adaptive two-stage pruning under a fixed budget split K1 = ⌊an R⌋ and K2 = R − K1 where an ≜ acn . Stage I selects relevance pivots using saliency scores ϕn,t , and Stage II selects diversity completion tokens from the non-pivot pool Un = Vn \ Pn , designed to capture semantic information missed by the saliency-based selection. Pruning equations in execution order.

We restate the pruning pipeline as a numbered “recipe” for easier reference.

1. Attention matrix and saliency (relevance score). Given an attention block with query/key/value Q, K, V , define QK ⊤ Attn(Q, K, V ) = softmax √ V, dk

QK ⊤ An ≜ softmax √ , dk

(32)

where An is the attention matrix (typically averaged over heads in practice). For a compact reference index set S (e.g., visual [CLS] or selected text instruction tokens), we score a visual token t ∈ Vn by ϕn,t ≜

1 X An,i,t , |S|

t ∈ Vn .

(33)

i∈S

Here An,i,t measures how much the reference token i attends to visual token t; larger ϕn,t indicates higher instruction relevance, effectively highlighting the visual regions most aligned with the user prompt. 2. Budget split into relevance pivots vs. diversity completion. Given a class-dependent ratio an ∈ [0, 1] and total budget R, determining the specific allocation size for the relevance and diversity stages: K1 = ⌊an R⌋,

K2 = R − K1 .

(34)

3. Stage I: pivots by top-K1 saliency. We keep the K1 most salient tokens as pivots: Pn = TopK1 {ϕn,t }t∈Vn .

(35)

This stage is “relevance-first”: it aggressively preserves tokens strongly queried by the instruction. 4. Pivot-relative redundancy. Let Un = Vn \ Pn be the non-pivot pool. Using unit features un,t (Eq. (24)), define redundancy of a candidate token t ∈ Un as ρn,t ≜ max u⊤ n,t un,j , j∈Pn

t ∈ Un .

(36)

Because un,t are unit vectors, ρn,t ∈ [−1, 1] is cosine similarity. Large ρn,t means t is highly duplicated by some pivot; small ρn,t means t lies in a direction poorly covered by pivots. 5. Deterministic redundancy-aware seeding for Stage II. We view ρn,t as a pivot-overlap cost: it measures how much a candidate token t ∈ Un resembles the already-selected pivot set Pn (worst-case cosine overlap). To make our seeding rule explicit, for any seed set C ⊆ Un with |C| = K2 , we define the total pivot-relative redundancy as X D(C | Pn ) ≜ ρn,t . (37) t∈C

Since Eq. (37) is additive over tokens, the global minimizer under |C| = K2 is obtained by selecting the K2 tokens with the smallest ρn,t , yielding a deterministic initialization that starts from directions complementary to Pn . Note that D targets seed-to-pivot redundancy; redundancy among completion tokens is then reduced by the subsequent K-means refinement. 6. Bottom-K2 seeding (optimal for D). The minimizer of Eq. (37) is obtained by selecting the K2 least redundant tokens, since the sum is minimized by the smallest individual terms: Cn(0) = BottomK2 {ρn,t }t∈Un . (38) (0)

(0)

We use the corresponding unit features as initial spherical K-means centers, i.e., set µn,k = un,ck for ck ∈ Cn . 22

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

7. Spherical K-means refinement (coverage). With cosine similarity, spherical K-means iterates between: (a) Assignment (nearest-center by cosine). s(r) (t) = arg

max

k∈{1,...,K2 }

(r−1)

u⊤ n,t µn,k ,

t ∈ Un .

(39)

k ∈ {1, . . . , K2 }.

(40)

(b) Update (normalized mean direction). P

t:s(r) (t)=k un,t

(r) µn,k =

t:s(r) (t)=k

un,t

, 2

We run r = 1, . . . , T iterations, where small T (e.g., 5) is typically sufficient in practice. 8. Discrete completion tokens via cluster medoids. After T iterations, we select one representative per cluster: qn,k = arg

max t:s(T ) (t)=k

(T )

u⊤ n,t µn,k ,

2 Qn = {qn,k }K k=1 ⊆ Un .

(41)

Eq. (41) returns token indices (not continuous centers), so the retained set is directly usable by the decoder. Proposition 2 (To initialize clustering away from pivots, Bottom-K2 seeding minimizes total seed redundancy) Fix (0) Pn and let Un = Vn \ Pn . Among all seed sets C ⊆ Un with |C| = K2 , the minimizer of D(C | Pn ) in Eq. (37) is Cn in Eq. (38), providing the global optimum for the additive redundancy cost function. Proof C.6 Eq. (37) is a sum of independent scalar costs ρn,t over the chosen indices. Thus the minimum over all K2 -subsets is achieved by selecting the K2 smallest ρn,t values, i.e., Eq. (38). A coverage objective for spherical K-means.

Define the directional coherence (coverage) objective X 2 J ({µn,k }K max u⊤ n,t µn,k . k=1 ) ≜ t∈Un

k∈{1,...,K2 }

(42)

Each summand in Eq. (42) rewards a token by its cosine similarity to the nearest center. Maximizing J therefore encourages the centers to cover multiple modes in Un rather than collapsing to a single region. Proposition 3 (To monotonically improve coverage of Un , spherical K-means performs coordinate ascent on J ) The alternating updates in Eqs. (39)–(40) do not decrease J in Eq. (42). Moreover, since J is bounded above, the procedure converges to a stationary point (a local optimum) of J . Proof C.7 We verify monotonicity in the same two-step order as the algorithm. (a) Assignment improves J for fixed centers. For fixed {µn,k }, choosing s(r) (t) = arg maxk u⊤ n,t µn,k maximizes each summand in Eq. (42), guaranteeing a non-decreasing objective value, hence cannot decrease J . (b) Update improves J for fixed assignments. For fixed assignments, the contribution of cluster k equals P u⊤ µ subject to ∥µn,k ∥2 = 1. By Cauchy–Schwarz, this is maximized by setting µn,k proportional to Pt:s(r) (t)=k n,t n,k t:s(r) (t)=k un,t and normalizing, yielding the optimal directional center, i.e., Eq. (40). Since each iteration is non-decreasing and J ≤ |Un | (each term ≤ 1), monotone ascent implies convergence. C.4. Geometric diversity guarantees induced by redundancy-aware seeding Lemma C.4 (To translate cosine redundancy into geometry, we relate it to Euclidean distance on the unit sphere) For any unit vectors u, v, we directly have ∥u − v∥22 = 2(1 − u⊤ v). Proof C.8 Expand ∥u − v∥22 = ∥u∥22 + ∥v∥22 − 2u⊤ v = 2 − 2u⊤ v. 23

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Corollary 2 (To ensure seeds start outside pivot neighborhoods, separation is bounded by redundancy thresholds) (0) Let Cn be the seed set in Eq. (38). Define the (deterministic) redundancy threshold δn ≜ max ρn,t ,

(43)

(0)

t∈Cn

(0)

which is equivalently the K2 -th smallest value among {ρn,t }t∈Un . Then for every seed c ∈ {un,t : t ∈ Cn }, which are chosen to minimize the redundancy with the pre-selected pivots, and any pivot j ∈ Pn , p c⊤ un,j ≤ δn and ∥c − un,j ∥2 ≥ 2(1 − δn ). (44) (0)

(0)

Proof C.9 By construction of Cn , each selected seed index t ∈ Cn satisfies ρn,t ≤ δn . Since ρn,t = maxj∈Pn u⊤ n,t un,j , ⊤ we have un,t un,j ≤ ρn,t ≤ δn for all pivots j. Applying Lemma C.4 yields the Euclidean bound. Interpretation. Eq. (44) formalizes what redundancy-aware seeding enforces geometrically: initial centers lie outside a cosine-similarity “cap” around each pivot, guaranteeing a minimum Euclidean distance to the pivot set. Spherical K-means then refines these centers to better represent modes of Un without relying on random initialization. C.5. Computational complexity Let Mn = |Vn |, embedding dimension be d, and K1 = ⌊an R⌋, K2 = R − K1 . We summarize the additional overhead incurred by pruning beyond a standard forward pass in the same step-wise style. 1. Layer fusion (Eq. (26)). Computing {z̄n,t }t∈Vn costs O(L Mn dv ): a single weighted sum over cached layer outputs. 2. Saliency top-K1 (Eq. (35)). Selecting TopK1 costs O(Mn log K1 ) via partial sort / heap. 3. Redundancy computation for seeding (Eq. (36)). Naively, computing ρn,t = maxj∈Pn u⊤ n,t un,j for all t ∈ Un costs O((Mn − K1 )K1 d). This can be implemented as a matrix multiplication between UU ∈ R(Mn −K1 )×d and UP ∈ RK1 ×d followed by a row-wise max, yielding the required redundancy values. 4. Bottom-K2 seeding (Eq. (38)). Selecting BottomK2 costs O((Mn − K1 ) log K2 ). 5. Spherical K-means refinement (Eqs. (39)–(40)). Each iteration costs O((Mn − K1 )K2 d) for similarity evaluation/assignment plus O((Mn − K1 )d) for accumulating cluster sums and normalization. Over T iterations, the refinement cost is O(T (Mn − K1 )K2 d), which remains efficient for small iteration counts. 6. Medoid selection (Eq. (41)). Computing similarities of each token to its final cluster center costs O((Mn − K1 )d). Memory overhead is dominated by storing normalized token matrices, e.g., O((Mn − K1 )d) if UU is materialized. Since K2 ≤ R and T is a small constant (e.g., T =5), the refinement term remains lightweight compared to the quadratic attention cost of processing long visual sequences in the decoder, typically dominating the total runtime.

D. More Experiment Ablation of Core Components. To isolate and quantify the individual contributions of our proposed modules, we conduct a component-wise ablation study on LLaVA-v1.5-7B under a fixed retention budget of R = 192. Table 10 presents the performance of the baseline pruning model, the model augmented solely with Class-Adaptive Layer Fusion (+Fusion), the model utilizing only the Category-Dependent Split Ratio (+Ratio), and the complete CLASP framework combining both (+Both). The results demonstrate that each component independently provides robust improvements over the baseline. Introducing the class-adaptive layer fusion yields substantial gains (e.g., +93 points on MME and +2.0 on GQA), highlighting the critical importance of dynamically matching the visual abstraction depth to the user’s intent. Similarly, adjusting the relevance-coverage pruning budget based on the prompt category (+Ratio) consistently boosts performance across all tasks. Ultimately, the integration of both components achieves the highest accuracy across all metrics (e.g., MME reaching 1848 and GQA reaching 60.4), proving that optimal visual representation extraction and adaptive budget allocation are highly complementary. 24

Performance

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

MME

1400

1600

GQA

MMVet

0.5 0.75 0.95

70 55

VQAText

50 0

0.5 0.75 0.95 Pruning ratio

VQAV2

0.5 0.75 0.95 Pruning ratio SparseVLM PDrop

SQA

0.5 0.75 0.95

POPE

1200

0 Performance

1800

MMB

0.5 0.75 0.95 Pruning ratio Our Baseline

0.5 0.75 0.95 Pruning ratio

Figure 4. Per-benchmark performance under increasing token pruning. We plot performance as a function of pruning ratio on eight evaluation suites, comparing our method with representative pruning baselines (SparseVLM and PDrop). The dashed line denotes the unpruned model performance. Our method exhibits the slowest degradation and stays consistently closest to the unpruned upper bound, particularly at aggressive pruning ratios. Table 10. Ablation study of core components. We evaluate the individual and synergistic effects of Class-Adaptive Layer Fusion (+Fusion) and Category-Dependent Split Ratio (+Ratio) on LLaVA-v1.5-7B (R = 192). Both components contribute significantly, and their combination yields the best overall performance. Config

MME

TextVQA

GQA

POPE

Baseline + Fusion + Ratio + Both (CLASP)

1721 1814 1787 1848

56.1 56.9 56.6 57.6

57.6 59.6 58.4 60.4

83.6 85.0 84.7 85.6

Sensitivity analysis of pruning ratios across diverse benchmarks. Figure 4 provides a comprehensive visualization of performance trajectories under varying degrees of pruning ratio (ranging from 0% to 95%). We compare our proposed method against two representative state-of-the-art baselines: SparseVLM and PDrop. Three key observations can be drawn from these results. First, our method (orange line) consistently forms the upper envelope of the performance curves across all eight distinct evaluation suites, demonstrating superior token retention capabilities compared to the baselines. Second, we observe a phenomenon of graceful degradation: while competing methods, particularly PDrop (green), suffer from catastrophic performance collapse in the high-pruning regime (pruning ratio > 0.75), our approach maintains remarkably robust efficacy. For instance, on the MME and GQA benchmarks, even when discarding 95% of visual tokens, our method retains a significant portion of the unpruned baseline accuracy, whereas PDrop approaches severely degraded performance. Third, in challenging reasoning tasks such as MMVet (note that PDrop results are unavailable for this benchmark) and SQA, our method exhibits high resilience, significantly widening the performance gap against SparseVLM as the compression rate increases. These findings corroborate that our similarity-based selection strategy effectively isolates and preserves semantically critical visual regions, ensuring model reliability even under extreme computational constraints. Performance analysis on a larger language model. To further evaluate the robustness of our method under extreme token scarcity, we compare CLASP against SparseVLM across four distinct pruning levels (retaining 192, 128, 64, and 32 tokens). Table 11 details these results. CLASP consistently outperforms SparseVLM across all compression ratios, demonstrating superior information retention capabilities. Specifically, in the high-retention regime (192 tokens), CLASP achieves an average performance retention of 98.2%, closely matching the upper bound. At the aggressive pruning level of 64 tokens (88.9% reduction), CLASP maintains 92.5% of the original performance, significantly surpassing SparseVLM’s 87.3%. Even under the extreme constraint of 32 tokens, where the baseline performance drops sharply to 80.3%, CLASP sustains a robust 87.5%. This trend is particularly evident in fine-grained tasks such as POPE and TextVQA, suggesting that 25

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Table 11. Performance comparison of various methods on LLaVA-v1.5-13B across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue.

Methods

GQA

MMB

MME

POPE

SQA

VQAText

Average

Upper Bound

63.3

68.9

1818

85.9

72.8

61.8

100.0%

LLaVA-1.5 13B SparseVLM CLASP (ours)

58.7 60.1

67.4 68.0

Retain 192 Tokens (↓ 66.7%) 1768 82.2 73.1 59.6 1785 86.2 72.8 59.6

96.9% 98.2%

LLaVA-1.5 13B SparseVLM CLASP (ours)

57.9 59.6

65.8 68.0

Retain 128 Tokens (↓ 77.8%) 1774 81.1 69.9 58.4 1789 85.7 73.3 59.0

95.0% 98.0%

LLaVA-1.5 13B SparseVLM CLASP (ours)

54.0 56.2

61.3 64.4

Retain 64 Tokens (↓ 88.9%) 1641 65.0 69.0 54.6 1675 77.2 72.8 55.6

87.3% 92.5%

LLaVA-1.5 13B SparseVLM CLASP (ours)

50.2 53.5

56.2 60.7

Retain 32 Tokens (↓ 94.4%) 1451 55.9 67.9 50.7 1592 68.7 71.9 52.6

80.3% 87.5%

Table 12. Performance comparison of various methods on LLaVA-NeXT-7B across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue.

Methods

GQA

MMB

MME

POPE

SQA

VQAText

Average

Upper Bound

64.2

67.4

1851

86.5

70.1

64.9

100.0%

59.7 61.8

94.4% 97.0%

LLaVA-NeXT 7B SparseVLM CLASP (ours)

61.2 63.1

62.2 62.8

Retain 640 Tokens 1697 85.3 67.6 1746 87.9 69.8

LLaVA-NeXT 7B SparseVLM CLASP (ours)

56.1 62.7

60.6 61.2

1533 1723

Retain 320 Tokens 82.4 66.1 85.8 67.0

58.4 61.7

90.0% 95.2%

LLaVA-NeXT 7B SparseVLM CLASP (ours)

55.8 62.1

56.9 59.8

1420 1699

Retain 160 Tokens 78.2 57.8 84.5 60.9

55.9 59.3

84.5% 92.2%

our method effectively preserves critical visual semantics. Performance analysis across varying token budgets. Tables 12 and 13 compare CLASP against SparseVLM on LLaVANeXT-7B and LLaVA-NeXT-13B under three budgets (640/320/160). Across all benchmarks, CLASP consistently surpasses the baseline and exhibits graceful degradation as the budget shrinks. With high retention (640 tokens), CLASP approaches the unpruned upper bound, reaching 97.0% (7B) and 97.2% (13B) average performance, improving over SparseVLM by 2.6 and 2.3 points, respectively. As sparsity increases, the advantage becomes more pronounced: at 320 tokens, CLASP attains 95.2% vs. 90.0% on 7B (+5.2) and 95.6% vs. 92.7% on 13B (+2.9); under the most aggressive setting (160 tokens), CLASP maintains 92.2% (7B) and 92.4% (13B), while SparseVLM drops to 84.5% and 88.5%, widening the gap to 7.7 and 3.9 points. Notably, CLASP yields consistent gains on grounding- and reasoning-sensitive benchmarks (e.g., GQA/MME) under heavy pruning, indicating that it better preserves critical visual evidence for multi-step inference. Performance analysis on InternVL2-26B. To further validate the scalability and robustness of our class-adaptive pruning framework on more recent and massive multimodal foundation models, we extended our evaluation to InternVL2-26B (Chen et al., 2024b). As shown in Table 14, we compared CLASP against existing token reduction baselines (FastV and ToMe) under a strict retention budget of R = 35%. Our method demonstrates exceptional information preservation capabilities at this massive scale. For example, on TextVQA and MMVet, CLASP retains high accuracy (81.7 and 63.2), substantially outperforming both FastV (75.6 and 45.0) and ToMe (75.7 and 52.5). Notably, on GQA, CLASP (65.0) even marginally exceeds the unpruned upper bound (64.9), suggesting that our redundancy-aware pruning effectively acts as a noise filter, benefiting complex compositional reasoning. These supplementary results confirm that CLASP remains highly effective and generalizable, regardless of the underlying model’s parameter size. 26

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Table 13. Performance comparison of various methods on LLaVA-NeXT-13B across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue. Methods

GQA

MMB

MME

POPE

SQA

VQAText

Average

Upper Bound

65.4

70.0

1901

86.2

73.5

64.3

100.0%

LLaVA-NeXT 13B SparseVLM CLASP (ours)

62.7 64.2

62.0 63.2

1821 1867

Retain 640 Tokens 86.0 71.5 86.8 73.5

59.3 61.5

94.9% 97.2%

LLaVA-NeXT 13B SparseVLM CLASP (ours)

60.9 63.0

60.8 62.8

1798 1848

Retain 320 Tokens 83.5 70.6 85.2 71.1

57.2 60.8

92.7% 95.6%

LLaVA-NeXT 13B SparseVLM CLASP (ours)

59.4 61.8

56.5 59.8

1755 1801

Retain 160 Tokens 81.1 66.2 83.7 68.1

53.3 57.8

88.5% 92.4%

Table 14. Performance comparison on InternVL2-26B across widely-used benchmarks. Results are shown for an aggressive pruning ratio (retaining 35% of visual tokens), with accuracy highlighted. Best results in blue.

Methods

TextVQA

MME

GQA

MMVet

Upper Bound

82.5

2270

64.9

64.0

InternVL2 26B FastV ToMe CLASP (ours)

75.6 75.7 81.7

Retain 35% Tokens 2140 61.2 2178 63.6 2262 65.0

45.0 52.5 63.2

Supplementary Evaluations on Diverse Open Datasets. To further validate the generalization and robustness of our approach beyond standard academic benchmarks, we conduct supplementary evaluations on a diverse set of open datasets. These include open-ended conversational evaluation (LLaVA-Bench, MMVet), real-world noisy visual perception (VizWiz), cross-lingual multimodal understanding (MMBench-Chinese), and fine-grained spatial reasoning (SEED-Bench). As summarized in Table 16, we compare CLASP against a strong token reduction baseline, SparseVLM, across different token retention budgets (R = 192, 128, 64). While both methods perform comparably well at a higher budget (R = 192), CLASP demonstrates significantly superior robustness under aggressive pruning conditions. For instance, at an extreme sparsity level of R = 64, CLASP outperforms SparseVLM by a massive margin of +6.5% on LLaVA-Bench and +9.5% on MMB-CN. This substantial gap underscores that our class-adaptive layer fusion and dual-stage pruning mechanisms effectively preserve the critical visual tokens necessary for complex, open-ended generation and cross-lingual semantic alignment, whereas static pruning methods suffer severe degradation. Ablation study of layer-mixture strategies across different task categories. Table 15 and Figure 5 present the performance variations across nine distinct task categories under five representative layer-fusion strategies with a fixed token budget (R = 192). These strategies (A–E) vary the source of visual features, ranging from shallow-biased mixtures to deep-layer integration. As visualized in the heatmaps and detailed in the table, while deeper layers generally provide superior semantic abstraction, specific tasks exhibit distinct sensitivities to feature depth. For instance, in the MMVet benchmark, Class 4 (Spatial Relations) experiences a severe performance drop with the shallow-focused Strategy B (16.2) compared to the deep-focused Strategy E (28.6), highlighting the necessity of high-level features for spatial reasoning. Similarly, in TextVQA, Class 2 (Text/Symbol Recognition) performs poorly under Strategy C (12.03), whereas Strategy E achieves optimal accuracy (57.74). Notably, the hybrid Strategy D, which fuses early visual cues (L5 ) with deep semantics (L22 ), consistently achieves the highest average scores across benchmarks like MMVet (28.6) and MME (1787.6), demonstrating that a balanced integration of low-level detail and high-level semantics offers the most robust generalization. Ablation study of similarity and attention for different classes. Table 17 and Figure 6 present the performance variations across nine distinct task categories under varying attention selection ratios p, where p denotes the proportion of retained tokens based on attention scores (ranging from 0.1 for high sparsity to 1.0 for full attention). Our results indicate that the 27

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Table 15. Ablation on layer-mixture strategies across different task categories with R = 192. The columns correspond to specific task types: (0) Object Identification, (1) Attribute/Breed ID, (2) Text/Symbol Recognition, (3) Scene Understanding, (4) Spatial Relations, (5) Counting, (6) Action/Interaction, (7) Intention/Function, and (8) Default. Layer Mixture Legend: A: 0.2L2 + 0.3L6 + 0.5L11 ; B: 0.2L5 + 0.3L15 + 0.5L22 ; C: 0.2L12 + 0.3L15 + 0.5L19 ; D: 0.2L5 + 0.8L22 ; E: 0.2L20 + 0.8L22 . Best results are highlighted in blue, and the row with the highest average score in each dataset is shaded.

Layers

Class 0

Class 1

Class 2

Class 3

Class 4

Class 5

Class 6

Class 7

Class 8

Avg

A B C D E

28.0 28.9 28.4 28.7 28.2

28.2 28.3 28.6 28.0 28.3

28.6 27.1 25.8 29.1 27.9

MMVet (R = 192) 28.5 27.8 28.4 28.8 16.2 27.8 28.5 14.0 28.1 28.3 28.3 29.5 28.4 28.6 27.9

28.3 28.2 28.9 28.8 28.1

28.7 28.1 28.4 28.5 28.0

28.3 28.2 28.4 28.2 28.2

28.3 26.8 26.6 28.6 28.2

A B C D E

57.56 57.62 57.61 57.58 57.59

57.56 57.58 57.59 57.58 57.62

55.94 45.43 12.03 57.44 57.74

TextVQA (R = 192) 57.51 57.60 57.19 57.54 57.57 55.57 57.56 57.55 52.64 57.60 57.57 57.53 57.54 57.54 57.54

57.55 57.57 57.53 57.56 57.60

57.58 57.55 57.49 57.59 57.59

57.51 57.57 57.53 57.55 57.34

57.33 56.00 51.95 57.56 57.57

A B C D E

68.62 68.72 68.52 68.47 68.67

68.47 68.67 68.62 68.52 68.62

63.41 39.43 14.83 68.27 68.42

68.52 68.27 68.17 68.52 68.47

SQA (R = 192) 68.72 68.77 68.57 68.17 68.67 64.35 68.57 68.62 68.62 68.57

68.57 68.57 68.52 68.57 68.52

68.57 68.57 68.57 68.62 68.67

68.67 68.67 68.57 68.52 68.67

68.04 65.29 62.09 68.52 68.58

A B C D E

1793 1780 1786 1788 1782

1790 1783 1791 1792 1786

1702 731 1433 1806 1790

MME (R = 192) 1774 1778 1785 1787 1788 1777 1776 1785 1784 1785 1784 1779 1793 1783 1788

1788 1783 1788 1785 1778

1780 1781 1788 1784 1788

1778 1786 1777 1785 1794

1774.2 1666.2 1745.3 1787.6 1786.9

optimal attention ratio is highly task-dependent. First, we observe that the model maintains competitive performance even at lower ratios (e.g., p = 0.3 or p = 0.5) across benchmarks like MMVet and SQA, demonstrating the redundancy in standard visual tokens. More notably, reducing p can lead to performance gains over the baseline (p = 1.0). For instance, in the MME dataset, Class 3 (Scene Understanding) achieves a peak score of 1828 at p = 0.7, surpassing the full-attention score of 1814. Similarly, TextVQA shows improved accuracy in Class 5 (Counting) at p = 0.3. These results demonstrate that the optimal attention ratio is highly category-dependent, with specific tasks achieving peak performance at intermediate p values rather than the full-attention baseline. Clustering iterations. Under a fixed retention budget (R = 192), we ablate the number of clustering refinement steps used in the similarity-driven stage of our pruning (Stage II), where token affinity is computed by cosine similarity on ℓ2 -normalized aligned features, i.e., sim(t, t′ ) = u⊤ n,t un,t′ (Eq. (13)). As shown in Table 18, 5 iterations achieves the best overall trade-off, yielding the highest MME score (1848) and the best/near-best performance on GQA (60.44), MMVet (31.8), and TextVQA (57.59), while leaving POPE/SQA essentially unchanged. With fewer iterations (e.g., 3), clustering is underrefined and redundant tokens are insufficiently consolidated. With more iterations (≥ 7), performance drops consistently (e.g., MME decreases to 1823/1762/1757), indicating over-merging or over-smoothing that can erase fine-grained evidence. We therefore adopt 5 clustering iterations as the default setting. Qualitative Analysis. Figure 7 and Figure 8 present a qualitative comparison of visual token pruning. We visualize the results across different network layers and varied pruning ratios (R). In these visualizations, the red bounding boxes denote the ground-truth target regions. The retained tokens are visualized to illustrate the rationale behind their selection. Specifically, blue points indicate regions preserved primarily due to their attention significance. Meanwhile, red points 28

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Table 16. Supplementary evaluations on diverse open datasets using LLaVA-v1.5-7B under varying token budgets. We compare our CLASP framework against SparseVLM. The best results between the two pruning methods at each budget are highlighted in bold.

Budget

Method

LLaVA-B

MMVet

VizWiz

MMB-CN

MMB

SEED

Unpruned

LLaVA-v1.5-7B

66.8

30.9

50.0

58.1

64.7

66.2

R=192

SparseVLM CLASP (Ours)

66.1 66.7

33.1 33.3

50.5 52.1

53.7 57.9

62.5 61.3

64.2 65.4

R=128

SparseVLM CLASP (Ours)

62.7 65.6

29.0 30.0

51.4 51.9

51.1 57.1

60.0 60.7

63.6 63.0

R=64

SparseVLM CLASP (Ours)

57.5 64.0

24.9 26.2

50.1 51.6

46.1 55.6

56.2 59.1

56.8 58.5

represent regions selected based on similarity metrics. As clearly observed, our method demonstrates superior capability in semantic preservation compared to state-of-the-art baselines such as PDrop and SparseVLM. Methods like PDrop and SparseVLM often struggle to maintain the semantic structure. They tend to lose critical foreground information or retain excessive background noise. This limitation is particularly evident at a high pruning ratio of R = 88.9%. In contrast, our approach consistently aligns the retained tokens with the target object within the red boxes. This robust alignment is maintained across all depths, including Layers 2, 6, and 15. These results indicate that our method effectively filters redundancy. It successfully maintains a focused representation of the region of interest throughout the inference process. Search-derived Weights vs. Further Calibration. To assess the robustness of the learned parameters, we compare our Search-derived Weights (fixed W ) against a variant that starts from the same Search-derived Weights init but further calibrates W on the held-out calibration set (Search-derived Weights init + Calibrated W ), under a fixed token budget (R = 192). Table 19 shows that the fixed Search-derived Weights yield the strongest overall performance: it improves MMBench (61.34 vs. 60.13), MMVet (33.3 vs. 32.2), MME (1848 vs. 1759), and TextVQA (57.59 vs. 56.15), and provides consistent gains on GQA/SQA, while matching VQA-v2 and remaining comparable on POPE. Moreover, naive alternatives are clearly suboptimal: a penultimate-biased initialization underperforms, and uniform averaging across layers severely degrades all benchmarks, highlighting the effectiveness of our discrete search strategy. We hypothesize that this gap reflects a mismatch between continuous optimization behavior and what token pruning needs to preserve. Vision encoders exhibit a strong depth hierarchy: shallow layers retain high-frequency local evidence (e.g., edges, strokes, and fine layouts) that is crucial for OCR- and counting-heavy queries, whereas deep layers emphasize invariant semantics. Our Search-derived Weights effectively capture this inductive bias by finding an optimal allocation of mixture mass to shallow/mid layers, ensuring that fine-grained evidence is not washed out. In contrast, further continuous calibration of W on limited data can exhibit shortcut behavior: optimization is naturally attracted to deep-layer features that yield strong, easy-to-fit semantic signals, resulting in a deep-biased mixture that is locally optimal for coarse semantics but dilutes the precise visual cues required by detail-sensitive tasks (e.g., TextVQA). This explanation is consistent with common ablation patterns reported in recent VLM pruning studies, where shallow/early cues often matter disproportionately for fine-grained perception, and overly aggressive averaging or smoothing across layers harms OCR-centric performance. Comparison with Static Multi-Layer Fusion Baseline. To further validate the necessity and superiority of our classadaptive layer fusion strategy, we compare CLASP against a strong static multi-layer fusion baseline. Specifically, we adopt the optimal static layer combination [2, 17, 23] (0-indexed, corresponding to layers 3, 18, and 24) identified as the empirical best practice in recent literature (Lin et al., 2025a). As shown in Table 20, we evaluate both methods on LLaVA-v1.5-7B across various token retention budgets (R = 192, 128, 64). While the optimally searched static fusion provides a reasonable baseline, CLASP consistently outperforms it across all budgets and benchmarks. For instance, at a moderate budget of R = 192, CLASP achieves a significant +92 point improvement on MME and +3.0% on GQA compared to the static baseline. The performance gap widens further under extreme sparsity (R = 64), where CLASP retains 1709 on MME versus the static baseline’s 1614 (+95 points). These results strongly suggest that different instruction intents inherently require distinct levels of visual abstraction; a fixed combination of layers, even when globally optimized, is fundamentally limited compared to our dynamic, class-conditioned routing approach. 29

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Table 17. Ablation study on the sensitivity of different task categories to the attention selection ratio p (with fixed R = 192). The columns correspond to specific task types mapped as follows: (0) Object Identification, (1) Attribute/Breed ID, (2) Text/Symbol Recognition, (3) Scene Understanding, (4) Spatial Relations, (5) Counting, (6) Action/Interaction, (7) Intention/Function, and (8) Default. Best results are highlighted in blue, and the row with the robust average score in each dataset is shaded.

Class 0

Class 1

Class 2

Class 3

Class 5

Class 6

Class 7

Class 8

0.1 0.3 0.5 0.7 0.9 1.0

1815 1809 1799 1817 1816 1814

1810 1814 1813 1812 1813 1814

1815 1770 1791 1813 1809 1814

MME (R = 192) 1812 1815 1782 1814 1818 1802 1812 1811 1801 1828 1818 1812 1816 1817 1816 1814 1814 1814

1812 1805 1813 1812 1813 1814

1812 1809 1815 1814 1814 1814

1813 1804 1811 1813 1814 1814

0.1 0.3 0.5 0.7 0.9 1.0

29.6 29.4 29.9 30.2 29.3 29.8

30.2 31.0 29.7 30.4 31.2 29.8

30.4 30.6 30.1 29.5 29.5 29.8

MMVet (R = 192) 29.6 30.0 29.9 29.7 30.1 30.3 31.0 29.4 30.0 29.7 29.2 30.5 29.9 30.4 29.6 29.8 29.8 29.8

29.0 29.0 28.8 28.6 30.2 29.8

29.8 30.1 30.1 30.2 30.5 29.8

29.2 30.0 30.4 30.5 30.7 29.8

0.1 0.3 0.5 0.7 0.9 1.0

56.88 56.96 56.96 56.96 56.96 56.99

56.48 56.66 56.90 57.07 57.09 56.99

55.89 56.24 56.50 56.85 57.00 56.99

TextVQA (R = 192) 56.86 56.88 56.97 56.90 56.90 57.48 56.93 56.93 57.01 56.93 56.98 57.07 56.94 56.96 57.00 56.99 56.99 56.99

56.97 56.99 56.94 56.92 56.97 56.99

56.98 57.02 56.95 56.99 57.02 56.99

56.92 57.10 56.98 56.90 56.91 56.99

0.1 0.3 0.5 0.7 0.9 1.0

68.37 68.28 68.32 68.42 68.17 68.27

68.27 68.12 68.27 68.32 68.22 68.27

68.37 68.17 68.37 68.37 68.27 68.27

69.56 68.82 68.47 68.22 68.32 68.27

SQA (R = 192) 68.02 68.32 68.22 68.32 68.27 68.32 68.37 68.32 68.67 68.22 68.27 68.27

68.27 68.27 68.22 68.27 68.32 68.27

68.32 68.42 65.26 65.32 65.21 68.27

65.26 64.95 64.35 65.65 66.41 68.27

Class 4

Robustness Analysis across Random Seeds. To rigorously verify that the performance improvements achieved by CLASP are statistically significant and not artifacts of random noise or specific initialization configurations, we conducted a variance analysis on the video understanding benchmarks. Specifically, we evaluated our method under a 50% retention budget using five different random seeds ([42, 43, 44, 45, 46]) for the redundancy-aware clustering initialization. As shown in Table 21, the standard deviations across all three video benchmarks (TGIF, MSVD, MSRVTT) are extremely small (ranging from ±0.13 to ±0.21). Furthermore, the mean performance across all seeds consistently outperforms the baseline methods evaluated under similar computational constraints. This confirms that the dual-stage pruning mechanism—particularly the similarity-driven completion stage—is highly stable and introduces negligible variance, ensuring robust representation compression. Robustness of the Prompt-to-Class Router. To further evaluate the robustness of our intent classifier and understand the impact of potential routing errors, we conduct two simulated stress tests. First, we simulate classifier uncertainty by forcing a specific proportion of samples to be assigned to the “Default” category. Second, we simulate critical routing failures by randomly misclassifying a certain ratio of samples into incorrect non-default categories. As shown in Table 22, falling back to the “Default” category yields a highly graceful degradation. Even when 50% of the samples are forced to the default class, the model maintains competitive performance (e.g., MME merely drops to 1756, and POPE to 83.6). In contrast, random misclassification leads to severe performance penalties, with MME dropping to 1722 at 30

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Table 18. Impact of the number of clustering iterations on performance (R = 192).

Iterations

MME

GQA

MMVet

POPE

SQA

TextVQA

3 5 7 9 11

1828 1848 1823 1762 1757

60.41 60.44 59.08 59.02 58.61

31.1 31.8 31.3 30.5 30.1

85.57 85.55 85.48 84.33 84.41

69.58 69.56 68.32 68.37 68.22

56.83 57.59 56.81 56.92 56.94

Table 19. Effect of initialization under a fixed budget (R=192). We compare our search-derived weight configuration with three initialization schemes while keeping the training protocol and budget fixed. Evaluation covers three general multimodal benchmarks (MMBench, MMVet, MME), four VQA/QA benchmarks (TextVQA, VQA-v2, GQA, SQA), and the hallucination benchmark POPE. Higher is better for all metrics. Best and second-best results are bold and underlined, respectively. Methods Search-derived Weights (fixed W ) Search-derived Weights init + Calibrated W Penultimate-biased init + Calibrated W Uniform init + Calibrated W

MMBench

MMVet

MME

TextVQA

VQA-v2

GQA

SQA

POPE

61.34 60.13 57.7 17.70

33.3 32.2 33.2 12.8

1848 1759 1722 1127

57.59 56.15 56.05 30.28

77.08 77.08 77.14 62.43

60.44 60.22 59.78 43.13

69.56 69.39 68.69 47.63

85.55 85.58 85.67 78.03

a 50% error rate, and experiencing a catastrophic collapse to 1627 at a 100% error rate. These findings not only demonstrate the robustness of our routing mechanism but also strongly justify our design choice: when the router encounters ambiguous or out-of-distribution prompts, falling back to a generalized “Default” prototype effectively preserves a robust baseline representation. This strategy is significantly safer and more effective than making arbitrary categorical guesses. Detailed Latency and Overhead Breakdown. To provide a more granular view of the computational efficiency of our framework and to empirically validate the theoretical complexity discussed in Appendix C.5, we conduct a micro-benchmark analysis of the inference latency. Table 23 details the time allocation across different components of the pipeline, including Total Prefilling, the Prompt-to-Class Classifier, Multi-layer Weight Fusion, and Spherical K-means clustering, under varying token retention budgets (R). The results clearly demonstrate that the additional operations introduced by CLASP contribute negligibly to the overall inference time. Specifically, the text-only intent classifier executes in under 2ms, and the token-wise weight fusion takes less than 0.5ms across all settings. The K-means clustering refinement (Stage II pruning) requires only 27.2ms at R = 192, and its cost naturally scales down to 17.2ms at R = 64 as the number of candidate tokens decreases. Importantly, these minor overheads are heavily offset by the immense savings in the vision-language decoder. By aggressively reducing the sequence length, the Total Prefilling time drops from 125.2ms to 74.7ms, and the FLOPs are nearly halved (from 2.67T to 1.39T), leading to a highly favorable end-to-end speedup. This confirms that CLASP is an operationally lightweight plug-and-play module. Robustness to Router Model Choice. To demonstrate that the CLASP framework is robust and not heavily reliant on the specific instruction-following model used for routing, we conduct an ablation study comparing our default text-only router against alternative architectures. Table 24 details the downstream performance across diverse multimodal benchmarks (MME, TextVQA, SQA, MMVet, and POPE) when substituting the default Qwen3-8B router with Llama3-8B (Grattafiori et al., 2024) and a larger Qwen3-32B model. The results clearly demonstrate that while varying the router model introduces slight differences in the intent assignment distribution (indicated by the Count Variance), the overall multimodal performance remains remarkably stable. Specifically, the MME score fluctuates marginally between 1840 and 1848, and the POPE accuracy remains tightly bounded between 85.3% and 85.7% across all settings. Importantly, this stability indicates that our discrete subspace search strategy and category-conditioned fusion mechanisms are highly resilient to minor routing shifts. This confirms that CLASP is effectively model-agnostic with respect to the prompt-to-class router, ensuring reliable and consistent deployment in diverse environments. 31

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models Table 20. Performance comparison between the optimal Static Fusion baseline and our Class-Adaptive Fusion (CLASP) on LLaVA-v1.57B under varying token budgets. Best results are highlighted in bold.

Budget

Method

MME

GQA

POPE

TextVQA

Unpruned

LLaVA-v1.5-7B

1862

61.9

85.9

58.2

R=192

Static [3, 18, 24] CLASP (Ours)

1756 1848

57.4 60.4

83.9 85.6

57.6 57.6

R=128

Static [3, 18, 24] CLASP (Ours)

1701 1790

56.2 58.9

82.7 85.2

55.3 56.7

R=64

Static [3, 18, 24] CLASP (Ours)

1614 1709

54.1 57.0

78.2 82.8

51.9 55.2

Table 21. Variance analysis of CLASP on video benchmarks (TGIF, MSVD, MSRVTT). Results are reported as mean ± standard deviation across five random seeds (42, 43, 44, 45, 46) under a 50% retention budget. Retention Budget (R) 50%

TGIF

MSVD

MSRVTT

45.53 ± 0.18

61.72 ± 0.21

51.45 ± 0.13

Comparison with Concurrent State-of-the-Art Methods. To further contextualize our contributions against the most recent advancements in the field, we compare CLASP with several concurrent and newly introduced visual token reduction methods: Nuwa (Huang et al., 2026), Holov (Zou et al., 2025a), and BTP (Li et al., 2025). We conduct this evaluation on the LLaVA-v1.5-7B architecture under a retention budget of R = 192 tokens. As detailed in Table 25, CLASP maintains a strong competitive edge against these latest baselines. Most notably, CLASP achieves the highest performance on complex multi-step reasoning benchmarks, securing 1848 on MME and 33.3 on MMVet, which substantially outperforms the recent Nuwa model (1834 and 30.5, respectively). On OCR-centric and general scientific QA tasks like TextVQA and SQA, CLASP consistently ranks at or near the top (57.6 and 69.6), demonstrating that our dual-stage class-adaptive pruning preserves essential local visual cues better than fixed-strategy concurrent works. Even on hallucination metrics (POPE), our method remains extremely robust (85.6). These results affirm that CLASP represents a leading solution in the rapidly evolving landscape of efficient MLLM inference.

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Table 22. Ablation on router robustness and classification errors. We compare the impact of forcing a specific ratio of samples to the “Default” class (top) versus randomly misclassifying them (bottom). Results demonstrate that falling back to the default class under uncertainty is significantly safer than random assignment. Forced “Default” Ratio

MME

TextVQA

SQA

MMVet

POPE

5% 20% 50% 100%

1840 1799 1756 1715

57.4 55.0 55.1 53.7

69.5 68.7 68.0 66.7

33.3 32.8 31.3 29.6

85.5 84.7 83.6 82.2

Random Error Ratio

MME

TextVQA

SQA

MMVet

POPE

5% 10% 20% 50% 100%

1834 1819 1789 1722 1627

57.4 56.0 54.1 51.5 49.7

69.4 68.9 67.2 64.9 60.1

33.1 32.2 29.3 26.2 23.8

85.3 84.6 83.1 80.7 77.4

Table 23. Detailed latency and computational overhead breakdown. We report the end-to-end Total Time, Memory footprint, FLOPs, and the specific time consumed by distinct pipeline components (Total Prefilling, Classifier, Weight Fusion, and K-means) under varying retention budgets (R). The overhead of our proposed components is minimal compared to the overall prefilling savings. Retained Tokens (R)

Total Time

Memory

FLOPs

Total Prefill

Classifier Time

Weight Fusion

K-means

192 128 64

223.1 ms 201.7 ms 155.8 ms

17.62 GB 17.59 GB 17.45 GB

2.67 T 1.97 T 1.39 T

125.2 ms 109.6 ms 74.7 ms

< 2 ms < 2 ms < 2 ms

< 0.5 ms < 0.5 ms < 0.5 ms

27.2 ms 23.1 ms 17.2 ms

Table 24. Performance comparison of CLASP across different prompt-to-class router models. We report the Count Variance (measuring the intent assignment deviation relative to the default Qwen3-8B router) alongside task accuracy across five representative benchmarks. The downstream performance remains highly stable regardless of the chosen router. Router Model Qwen3-8B (Default) Llama3-8B Qwen3-32B

Count Variance

MME

TextVQA

SQA

MMVet

POPE

0 37.2 7.9

1848 1840 1848

57.6 57.5 57.7

69.6 69.4 69.4

33.3 32.9 33.1

85.6 85.7 85.3

Table 25. Comparison with concurrent state-of-the-art methods. Evaluation is performed on LLaVA-v1.5-7B with a token retention budget of R = 192. “-” indicates that the result is not reported by the respective authors.

Method

MME

TextVQA

SQA

MMVet

POPE

Nuwa (ICLR 2026) (Huang et al., 2026) Holov (NeurIPS 2025) (Zou et al., 2025a) BTP (NeurIPS 2025) (Li et al., 2025)

1834 1820 1816

57.4 57.4 -

68.2 69.8 69.1

30.5 29.1

86.4 85.6 85.6

CLASP (ours)

1848

57.6

69.6

33.3

85.6

CLASP: Class-Adaptive Layer Fusion and Dual-Stage Pruning for Multimodal Large Language Models

Record · ID 13147 · SHA-256 534da9941a586708
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.