Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference Zhaoyang Luo1 , Runmin Dong2* , Miao Yang4 , Fan Wei4 , Yushan Lai1 , Bin Luo1 , Haohuan Fu1,3,4* , 1 Tsinghua Shenzhen International Graduate School, Shenzhen, China 2 Sun Yat-sen University, Zhuhai, China 3 National Supercomputing Center in Shenzhen, Shenzhen, China 4 Tsinghua University, Beijing, China
arXiv:2606.31903v1 [cs.CV] 30 Jun 2026
Correspondence: [email protected]; [email protected]
Abstract
Multimodal large language models (MLLMs) (Liu et al., 2024a; An et al., 2025; Liu et al., 2024b; Bai et al., 2025b,a) have become a central paradigm for visual understanding, visual reasoning, and visual question answering. A typical MLLM encodes an image into visual tokens, projects them into the embedding space of a large language model *
Corresponding authors. The source code is available at: https://github.com/ zayan-l/Op-Skip. 1
Both Skip
Introduction
Skip FFN
1
Visual Token Skip Attn
Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation. Existing acceleration methods usually remove visual tokens or skip visual-token updates in entire layers, but these coarse strategies may discard fine-grained evidence or suppress useful operators together with redundant ones. In this paper, we study visual-token computation from an answer-observable perspective and find that late visual-token updates can remain large while having little effect on answer-token representations. Motivated by this answersilent redundancy, we decompose each Transformer layer into attention and FFN operators and show that useful visual computation is often operator-dominant and layer-dependent. We propose an operator-level visual-token skipping framework that preserves the full visualtoken sequence while selectively bypassing redundant attention, FFN, or both. Experiments across three MLLM architectures and 10 VQA benchmarks show that our method achieves strong efficiency-accuracy trade-offs, reducing 33.7% TFLOPs on Qwen3-VL while retaining 99.5% of the vanilla model performance.1
Textual Token
Token Pruning
Layer-wise Skipping
Operator-level Skipping
Figure 1: Overview of visual-computation reduction granularities for MLLM inference. Token pruning removes selected visual tokens, which disrupts spatial relationships and causes irreversible loss of visual evidence needed for reasoning. Layer-wise skipping preserves the token sequence but treats each Transformer layer as an indivisible unit, failing to fully exploit the distinct roles of attention and FFN operators. Our operator-level skipping selectively skips attention, FFN, or both while preserving the full visual-token sequence.
(LLM), and concatenates them with text tokens for autoregressive generation (Liu et al., 2024a; Bai et al., 2025b,a). To preserve fine-grained visual evidence, recent MLLMs increasingly adopt high-resolution inputs, image tiling, or dense visual encoders, expanding the visual sequence from hundreds to thousands of tokens (Li et al., 2023a; Liu et al., 2024b; Bai et al., 2025b,a). While this improves perceptual coverage, it also makes the prefill stage expensive, as the LLM backbone must process all visual and textual tokens simultaneously through self-attention and feed-forward transformations (Chen et al., 2025). A growing body of work therefore accelerates MLLMs by reducing visual computation, either by pruning, selecting, or merging visual tokens (Huang et al., 2026; Xing et al., 2025; Zhang et al., 2025; Yang et al., 2025), or by skipping visual-token updates in selected Trans-
former layers (Yuan et al., 2025; Lin et al., 2025; Ma et al., 2026a). These methods show that not all visual computation is equally necessary, but they rely on coarse notions of redundancy. Token-level methods may permanently discard fine-grained evidence, layer-level skipping removes attention and FFN computation together, and V-Skip (Ma et al., 2026a) assumes a fixed preference for preserving FFN while skipping visual attention. Such fixed granularities cannot decide whether attention, FFN, or both are redundant at each layer, and may suppress useful computation when only part of a layer is redundant. In this paper, we revisit visual redundancy in MLLMs by examining how visual tokens evolve inside the LLM backbone. We begin with a counterintuitive observation, in late layers, visual-token hidden states can still change substantially, even when these changes have little influence on the final answer. This indicates that update magnitude alone is not a reliable measure of useful visual computation. To distinguish representational movement from computation that actually affects answer generation, we introduce a set of answerobservable diagnostics, which measure how visualtoken updates propagate to answer-token hidden states. Across multiple MLLM architectures and VQA benchmarks, we consistently find that late visual tokens often undergo large hidden-state updates while exhibiting small answer-observable effects. These updates are therefore not inactive, but answer-silent, they consume computation while contributing little to the representations that determine the generated response. This observation suggests that visual redundancy is not merely a property of individual tokens, nor is it always aligned with entire Transformer layers. Instead, redundancy may emerge at a finer operator level within each layer. This view is supported by recent analyses of LVLM backbones (Xi et al., 2026), which reveal a functional decoupling between attention and FFN operators, attention primarily reconfigures information within a relatively preserved representation subspace, whereas FFNs expand the representation subspace and drive semantic transformation. If these two operators serve different representational roles, their importance for visual tokens should not be assumed to be coupled in every MLLM layer. Some layers may require attention to route visual evidence toward the question, some may rely more on FFN transformations to refine visual semantics, and others may
contribute little through either operator. Motivated by this observation, we decompose each Transformer layer into its attention and FFN operators and ask a finer question: which operator is actually necessary for visual-token computation in each layer? By separately measuring the answerobservable contribution of attention and FFN, we find strong layer-wise heterogeneity, some layers are attention-dominant, some are FFN-dominant, and others show little answer-observable contribution. Based on this observation, we propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while bypassing redundant visual computation. This design avoids the irreversible evidence loss of token pruning and releases visual skipping from the static assumption that either attention or FFN is always the important operator. Experiments across multiple MLLM architectures and 10 visual question answering benchmarks show that our framework substantially reduces visual computation during inference while maintaining answer quality, achieving up to 33.7% visual-computation reduction with only 0.5% average performance change. Beyond efficiency, our analysis suggests that useful visual computation in MLLMs is operator-dependent rather than block-aligned, and that efficient inference should be designed around answer-observable computation rather than raw update magnitude or fixed architectural boundaries. Our contributions are summarized as follows: • We identify answer-silent visual-token updates in late MLLM layers and introduce answer-observable diagnostics to measure whether visual computation actually propagates to answer-token representations. • We reveal that useful visual computation is operator-dominant and layer-dependent by decomposing each Transformer layer into attention and FFN operators, exposing visual redundancy beyond token- and layer-level views. • We propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while bypassing redundant visual computation, saving up to 33.7% visual computation while retaining 99.5% model performance across Qwen3VL and 10 VQA benchmarks.
Figure 2: Layer-wise visual update magnitude, answer-observable influence, and answer-observable efficiency. All curves are normalized within each model and dataset for visualization. Across both GQA and TextVQA, late layers exhibit increasing visual-token update magnitude but decreasing answer-observable influence and efficiency.
2
Answer-Silent Visual Updates in MLLM
We first examine when visual-token information is effectively used by the textual stream during the prefill stage. In mainstream decoder-only MLLMs (Liu et al., 2024a; Bai et al., 2025b,a), visual tokens are placed before the textual prompt and processed under causal masking. As a result, the last prompt token is the first position that can integrate both the visual prefix and the textual question. Its hidden state, which we refer to as the final prompt-token hidden state, is directly projected by the language-modeling head to produce the nexttoken logits. This motivates the following diagnostic question: Question 1. Do visual-token updates remain answer-observable across all layers? Let Hvl ∈ RNv ×d denote the visual-token hidden states at layer l, where Nv is the number of visual tokens and d is the hidden dimension. We first quantify the magnitude of visual-token state changes between consecutive layers: ∆Vl = Hvl+1 − Hvl ,
VUMl =
|∆Vl |F . maxj |∆Vj |F + ϵ
(1) We refer to this normalized quantity as visual update magnitude (VUM), where larger values indicate stronger visual-state changing. To measure whether such changing is observable from the answer side, we estimate its first-order influence on the final prompt-token hidden state. Let hlast denote the final prompt-token hidden state during prefill, and let Jl =
∂hlast ∂Hvl+1
(2)
be the Jacobian that maps perturbations of visualtoken states at layer l + 1 to changes in hlast . We define δhl = Jl [∆Vl ],
AOIl = |δhl |2 .
(3)
and we call AOIl answer-observable influence (AOI), where Jl [∆Vl ] is the Jacobian-vector product estimating how the visual update at layer l changes the final prompt-token hidden state. Finally, we normalize AOI by the size of the visual update: AOIl AOEl gl= AOE . ∥∆Vl ∥F + ϵ maxj AOEj + ϵ (4) We refer to this quantity as answer-observable efficiency (AOE). In summary, VUM measures the magnitude of visual-token changes, AOI measures how much these changes reach the final prompttoken hidden state, and AOE measures the answerobservable influence per unit of visual-token update. Figure 2 shows the layer-wise behavior of VUM, AOI, and AOE on GQA and TextVQA for LLaVA1.5 and Qwen3-VL. The curves reveal a consistent late-layer decoupling: visual-token states continue to change substantially, while their answerobservable influence decreases. In LLaVA-1.5, VUM rises sharply after layer 24, whereas both AOI and AOE continue to decline. Qwen3-VL exhibits the same pattern around its decoupling onset layer. Thus, raw update magnitude systematically overestimates the usefulness of late visual computation. This observation suggests a direct intervention. If late visual-token updates are weakly coupled to AOEl =
Figure 3: Layer-wise operator-risk analysis across Qwen3-VL, Qwen2.5-VL, and LLaVA-1.5. Each point denotes one Transformer layer, with the horizontal axis measuring the risk of skipping visual attention and the vertical axis measuring the risk of skipping the visual FFN. Layers above the diagonal are FFN-domain layers, layers below the diagonal are attention-domain layers, and low Raf late layers are assigned to the frozen domain, inset plots zoom in on low-risk regions where frozen layers concentrate. Table 1: Effect of freezing late visual-token states. We freeze visual-token hidden states after the decoupling 24 layer while keeping text-token computation unchanged. Model LLaVA-1.5-7B Qwen3-VL-8B
Method
GQA
POPE
TextVQA
Full
61.9
85.9
58.2
69.5
61.1↓0.8
86.4↑0.5
57.2↓1.0
69.6↑0.1
Frozen visual Full Frozen visual
SQA
61.6
89.1
80.0
94.4
61.3↓0.3
88.9↓0.2
80.1↑0.1
94.4
answer generation, freezing visual-token states after the decoupling onset layer should have limited impact on task performance. We therefore freeze visual-token hidden states after layer 24 while keeping all text-token computation unchanged. Concretely, for layers l > 24, we reuse the visual states from layer 24 and allow textual tokens to continue attending to these fixed visual representations. This removes late visual-state changes while preserving the visual evidence accessible to textual tokens. As shown in Table 1, freezing late visual-token states causes only marginal degradation on 4 benchmarks across the two MLLM architectures. This result indicates that repeatedly updating visual-token states is often not the computation that determines next-token prediction. The model can still access visual evidence through the textual stream, while the continued changes of late visual-token states become increasingly answer-silent. Finding 1: Late-layer visual tokens can change substantially while becoming increasingly answer-silent, freezing them removes considerable visual computation with limited performance loss. This finding motivates a finer-grained analysis of where useful visual computation resides within
each Transformer block. We therefore move beyond the block-level view and decompose each layer into its attention and FFN operators, analyzing which operator dominates answer-observable visual computation at different depths.
3
Methodology
3.1
Operator Risk Analysis
We first describe how to identify the layer-wise operator structure of the visual branch without retraining the model or using human-annotated answers. Given a decoder-only MLLM, let the hidden states at layer l be H l = [HVl ; HTl ].
(5)
where HVl ∈ RNv ×d denotes the visual-token hidden states and HTl denotes the textual hidden states, including the system prompt, question tokens, and possible answer-prefix tokens. We decompose the visual branch of a Transformer block into two residual updates: H̄Vl = HVl + Al (H l ) V , (6) HVl+1 = H̄Vl + F l (H̄ l ) V . where Al is the self-attention update and F l is the FFN update at layer l. The subscript V denotes the visual-token rows. This decomposition allows us to ask a more fine-grained question than whether an entire layer can be skipped: Question 2. Which operator dominates the visual-branch computation at each layer? For an input sample x, the full model produces a reference next-token distribution from the final prompt-token hidden state after prefill: p0 (·|x) = softmax(z0 (x)).
(7)
··· Feed-Forward Network
FFN-Dominant Layer
Feed-Forward Network
Feed-Forward Network
··· Multi-head Self Attention
Atten-Dominant Layer
···
K
FFN-Dominant Layer
K
Q
···
Linear
Multi-head Self Attention
V
K
Q Linear
Frozen Layer
Multi-head Self Attention
V
V
Q
Linear
Linear
Linear
Linear
Linear
Linear
Linear
··· Frozen Layer Textual Token
Answer
Visual Token
FFN-dominant Layer (Skip Atten)
Attention-dominant Layer (Skip FFN)
Frozen Layer (Skip Atten and FFN)
Figure 4: Overview of the proposed operator-level visual-token skipping policy. Each layer is assigned to an operator domain according to its visual-token contribution, FFN-domainant layers preserve visual-token FFN computation while skipping redundant visual-token attention, attention-domainant layers preserve visual-token attention while skipping redundant visual-token FFN computation, and frozen layers skip both visual-token attention and FFN updates. Textual tokens continue to follow the original Transformer computation, while the full visual-token sequence is retained throughout inference.
This reference distribution is obtained from the model itself and does not require ground-truth labels. We estimate layer-wise operator risks on a small calibration set D sampled from multiple benchmarks, using these samples only to probe how the output distribution changes under counterfactual visual-operator interventions. 1 X DKL p0 (·|x)∥p(l) (·|x) , s |D| x∈D
(8)
s ∈ {A, F , AF}. Here, RA(l), RF(l), and RAF (l) measure the output risk of removing visual attention, removing visual FFN, and freezing the visual branch at layer l, respectively. log
RA(l) + ϵ . RF(l) + ϵ
(9)
A positive Γl indicates an attention-dominant layer, while a negative Γl indicates an FFN-dominant layer. Layers with uniformly small risks, especially small RAF (l), are treated as frozen layers. Finding 2: Visual computation in MLLM Transformer layers exhibits layer-dependent operator dominance, suggesting that redundancy reduction should be guided by the dominant operator of each layer. 3.2
for simplification. These are layers where removing either visual attention or visual FFN individually induces only a small output shift. For each selected layer, we assign the least risky visual-branch policy. If RA(l) < RF (l), we apply FFN- ONLY, skipping visual attention while preserving visual FFN. If RF(l) < RA(l), we apply ATTENTION - ONLY, preserving visual attention while skipping visual FFN. If RAF(l) is sufficiently small, we apply the more aggressive F REEZE - UPDATE policy, skipping both visual operators and directly propagating HV l to the next layer. Unlike token pruning, our method preserves the full visual-token sequence and avoids irreversible evidence removal. Unlike layer-level skipping, it does not treat attention and FFN as an inseparable block, but preserves the dominant visual operator while bypassing redundant visual-branch computation. 3.3
Complexity Analysis
Let Nv be the number of visual tokens, d the hidden dimension, and dff the FFN intermediate dimension. Since our method modifies only visual rows and keeps text-token computation unchanged, the computational reduction comes from skipping visual-row attention, visual-row FFN, or both.
Operator-Aware Skipping
We convert the layer-wise risk profile into an inference-time skipping policy through two steps, budgeted layer selection and policy assignment. Given a compute budget B, we rank layers by RA(l)+RF (l) and select the B lowest-risk layers
∆CA = Θ Nv d2 + MV d , ∆CF = Θ (κNv ddff ) .
(10)
where κ = 3 for gated FFNs used in LLaMA- and Qwen-style backbones, and κ = 2 for standard
Table 2: Main comparison across three MLLM architectures and 10 multimodal benchmarks. Avg. Ret. denotes the mean score ratio to the corresponding vanilla baseline across benchmarks. Red and blue indicate the best and second-best results among methods within each architecture. Model
Method
TFLOPs
GQA
TextVQA
MME
MMB
MMMU
POPE
SQA
AI2D
OCRB
VizWiz
Avg. Ret.
LLaVA-1.5-7B
Vanilla VTW (K=16) ShortV (N=20) VSkip (N=20) VSkip+ (N=20) V2Drop APET Ours (N=20)
100% 55% 55% 76% 76% 66% 66% 66%
61.94 54.77 60.49 60.77 61.63 60.51 60.45 60.84
58.21 52.17 53.63 57.21 57.21 55.92 56.36 56.40
1866.15 1852.02 1831.26 1766.56 1840.62 1838.27 1841.29 1841.68
64.18 63.92 64.60 64.69 64.60 64.15 64.09 64.52
36.11 35.67 35.67 35.56 36.44 31.67 36.11 35.67
85.94 86.89 86.19 87.33 87.26 86.82 86.82 87.26
69.46 69.71 68.27 69.16 69.26 69.12 68.93 69.16
55.18 55.38 54.24 55.12 55.12 54.37 55.54 55.18
31.50 5.20 28.10 30.80 30.50 29.70 29.90 30.20
54.09 49.68 49.65 53.95 54.16 52.56 53.23 52.75
100.0 88.6 96.5 98.9 99.6 97.0 98.7 98.8
Qwen2.5-VL-7B
Vanilla VTW (K=16) ShortV (N=20) VSkip (N=16) V2Drop APET Ours (N=20)
100% 50% 50% 67% 55% 55% 55%
60.40 46.01 49.47 57.26 58.87 59.38 59.29
77.77 65.43 71.93 63.58 72.26 70.90 72.02
2515.82 1659.20 1772.94 2039.69 2253.59 2263.83 2263.38
83.25 68.21 72.25 80.15 80.29 81.70 80.67
50.00 43.56 41.89 47.00 49.89 50.11 46.89
87.62 70.60 58.91 84.38 87.34 86.70 87.32
87.51 80.32 78.38 85.72 87.06 87.51 85.52
82.42 63.12 65.06 78.04 81.54 81.54 79.95
84.10 42.39 45.50 69.40 56.40 60.00 76.90
70.81 52.86 36.36 63.85 68.68 69.02 65.48
100.0 76.9 75.7 91.0 93.8 94.5 95.0
Qwen3-VL-8B
Vanilla VTW (K=16) ShortV (N=20) VSkip (N=20) V2Drop APET Ours (N=20)
100% 58% 58% 76% 66% 66% 66%
61.60 43.72 60.59 60.67 60.06 60.96 60.70
80.07 51.16 76.08 75.98 76.44 78.36 78.45
2390.38 1808.02 2263.21 2226.98 2290.27 2332.50 2336.01
84.79 67.53 81.62 81.87 82.56 82.47 83.08
51.33 47.33 52.22 51.33 52.44 52.53 52.89
89.13 73.88 88.97 89.16 88.34 88.54 89.43
94.40 81.11 92.61 92.46 91.72 92.56 94.35
83.78 69.88 81.77 80.60 81.64 82.16 82.55
82.80 37.00 73.90 72.20 63.80 75.80 80.40
69.37 54.81 69.61 68.49 69.78 68.82 71.70
100.0 75.8 97.1 96.3 96.0 98.0 99.5
two-layer FFNs. X (l) ∆Cskip = ∆CA
ducted with lmms-eval under the official benchmark protocols.
l∈LFFN
+
X
(l) ∆CF
(11)
l∈LAttn
+
X
(l)
(l)
∆CA + ∆CF
.
l∈LFreeze
Here, FFN- ONLY skips visual attention, ATTENTION - ONLY skips visual FFN, and F REEZE - UPDATE skips both operators. In practice, wall-clock speedup also depends on kernel implementation, memory movement, and hardware utilization; therefore, we report both theoretical FLOPs and measured first-token latency.
4
Experiments
4.1
Experimental Setup
Models. We evaluate our method on three representative open-source MLLMs, LLaVA-1.57B (Liu et al., 2024a), Qwen2.5-VL-7B (Bai et al., 2025b), and Qwen3-VL-8B (Bai et al., 2025a). LLaVA-1.5-7B uses a fixed-resolution visual encoder and represents each image with 576 visual tokens, while Qwen2.5-VL-7B and Qwen3-VL8B adopt more recent Qwen-VL architectures with stronger visual encoders and dynamic-resolution image processing. This evaluation tests whether operator-level visual redundancy generalizes across different MLLM designs. All evaluations are con-
Benchmarks. We evaluate on 10 multimodal benchmarks covering general VQA, text-rich understanding, hallucination sensitivity, multimodal reasoning, and scientific QA, GQA (Hudson and Manning, 2019), TextVQA (Singh et al., 2019), MME (Fu et al., 2025), MMBench (Liu et al., 2025), MMMU (Yue et al., 2024), POPE (Li et al., 2023b), ScienceQA (Lu et al., 2022), AI2D (Kembhavi et al., 2016), OCRBench (Liu et al., 2024c), and VizWiz (Gurari et al., 2018). Baselines. We compare with training-free acceleration baselines including layer-wise visual skipping methods VTW (Lin et al., 2025), ShortV (Yuan et al., 2025), and V-Skip (Ma et al., 2026a), as well as token-reduction methods V2Drop (Chen et al., 2026) and APET (Ma et al., 2026b). Unlike these coarse-grained baselines, our method preserves all visual tokens and selectively skips attention, FFN, or both according to layer-wise operator dominance. Implementation details. For each model, we estimate layer-wise visual operator risks using a small calibration set, sampling 7 examples from each of GQA, POPE, TextVQA, MME, MMMU, and ScienceQA. These examples are used only for estimating RA(l), RF(l), and RAF (l). Based on the risks, each selected layer is assigned one of three policies, ATTENTION - ONLY, FFN- ONLY, or
Table 3: Sensitivity to the operator-skipping budget across the three evaluated MLLMs. Avg. Ret. is the mean relative to the vanilla model. Budget
GQA
TextVQA
MME
MMB
MMMU
POPE
SQA
AI2D
OCRB
VizWiz
Avg. Ret.
0 8 20 32
61.94 61.14 60.84 57.89
58.21 57.18 56.40 53.80
1866.15 1878.10 1841.68 1728.08
LLaVA-1.5-7B (32 Layers) 64.18 36.11 85.94 69.46 64.18 35.67 86.43 69.61 64.52 35.67 87.26 69.16 62.03 35.67 85.01 69.21
55.18 55.51 55.18 52.46
31.50 31.40 30.20 25.10
54.09 53.23 52.75 52.76
100.0 99.6 98.8 94.5
0 8 20 28
60.40 59.85 59.29 57.27
77.77 76.62 72.02 54.75
2515.82 2346.15 2263.38 1884.70
Qwen2.5-VL-7B (28 Layers) 83.25 50.00 87.62 87.51 82.56 49.11 87.29 87.41 80.67 46.89 87.32 85.52 69.07 39.33 86.29 77.94
82.42 82.58 79.95 62.08
84.10 81.80 76.90 42.30
70.81 68.25 65.48 54.81
100.0 98.2 95.0 79.2
0 12 20 28 36
61.60 60.98 60.70 59.97 58.51
80.07 79.93 78.45 75.58 69.91
2390.38 2352.81 2336.01 2158.87 1707.86
Qwen3-VL-8B (36 Layers) 84.79 51.33 89.13 94.40 84.62 52.44 88.83 94.70 83.08 52.89 89.43 94.35 80.93 51.00 88.42 85.72 75.34 48.22 87.42 81.85
83.78 83.65 82.55 76.39 72.38
82.80 81.90 80.40 71.20 63.80
69.37 70.78 71.70 66.86 63.77
100.0 100.0 99.5 94.0 87.7
F REEZE - UPDATE. The resulting layer-wise policy is fixed for all samples of the same model. We report relative FLOPs ratio, and first-token latency measured on NVIDIA RTX A800 GPUs.
retaining stronger answer quality. This shows that operator-level skipping provides a better accuracy– efficiency balance than freezing full visual-token updates.
4.2
Table 4: Efficiency comparison with ShortV on TextVQA. We report theoretical FLOPs, relative FLOPs ratio, first-token latency, and average score.
Main Results
Table 2 compares our method with representative layer-wise skipping and token-pruning baselines across three MLLM architectures and ten benchmarks. Our method achieves a strong accuracy– efficiency trade-off by preserving the full visualtoken sequence while selectively skipping redundant operators, leading to the highest average performance retention across the evaluated backbones. This robustness comes from keeping the dominant visual operator in each selected layer, which helps preserve fine-grained and text-rich evidence while removing answer-silent computation. Following the V-Skip paper, VSkip+ uses datasetspecific optimal settings, whereas the other methods use one configuration for all benchmarks on each model. Our single per-model policy remains competitive or superior under this comparison setting.
Model
FLOPs
Prefill
Latency
Avg.
LLaVA-1.5-7B ShortV Ours
9.057 T 5.039 T 6.475 T
59.1 ms 45.0 ms 49.3 ms
120.4 ms 108.2 ms 110.4 ms
100.0 96.5 98.8
Qwen3-VL-8B ShortV Ours
11.38 T 5.79 T 7.23 T
95.7 ms 45.8 ms 52.9 ms
465.1 ms 310.6 ms 344.8 ms
100.0 97.1 99.5
Figure 5 visualizes model responses under operator-aware visual-token skipping. Our method still produces accurate answers on both reasoning and recognition examples, suggesting that it preserves the operators responsible for key layer-wise visual reasoning functions and largely maintains the model’s answer quality. 4.4
4.3
Efficiency Comparison on TextVQA
We further compare efficiency on TextVQA against ShortV, a competitive layer-wise visualcomputation baseline. As shown in Table 4, ShortV reduces more FLOPs but causes larger accuracy degradation, whereas our method achieves comparable prefill efficiency and first-token latency while
Effect of Operator-Skipping Budget
We study how the operator-skipping budget affects performance across the three evaluated MLLMs. As shown in Table 3, moderate budgets simplify redundant visual computation through ATTENTION ONLY, FFN- ONLY, or F REEZE - UPDATE while preserving most benchmark scores, whereas overly aggressive budgets eventually degrade performance
Is the chandelier to the left of the clock made of metal or glass?
What is in front of the green bush?
ShortV
VTW
glass VSkip
tree
Ours
glass
VSkip
fence
What is written in the image? ShortV
VTW
caiongr
08/03/18
caiongr
VSkip
Ours
08/03/18
Ours
elephant
ShortV
2023-04-27
elephant
VSkip
metal
When was this receipt issued? Answer this question using the text in the image directly. VTW
ShortV
VTW
glass
Ours
caiongr
06/03/18 18:04
caiognr
Figure 5: Qualitative visualization of model responses under operator-aware visual-token skipping.
on fine-grained or text-rich tasks. This consistent trend across LLaVA-1.5-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B suggests that operator-level redundancy is broadly present, but the best budget remains model-dependent and should be treated as an efficiency–performance knob. 4.5
Ablation Study
To isolate the contribution of operator-aware policy assignment, we conduct a fixed-budget ablation on Qwen3-VL-8B with budget B = 28. All variants simplify the same number of layers, but use different operator actions, ATTN -S KIP always skips visual attention, FFN-S KIP always skips visual FFN, and F ROZEN skips both operators. As shown in Table 5, fixed rules are not uniformly reliable, while our layer-wise risk-based assignment better balances efficiency and accuracy by matching the skipped operator to each layer’s dominant visual operator. Table 5: Ablation on Qwen3-VL-8B under budget B = 28. All variants simplify the same number of layers. Method
GQA
POPE
MME
OCRB
VizWiz
Vanilla ATTN -S KIP FFN-S KIP F ROZEN Ours
61.6 53.6 59.6 26.8 60.0
89.1 78.6 89.3 57.1 89.2
2390 1506 2118 975 2159
82.8 11.2 61.7 20.0 76.4
69.4 58.8 66.2 52.3 69.1
4.6
Combination with Token Pruning
Finally, we study whether operator-aware skipping is complementary to token pruning. Token pruning reduces the visual sequence length, while our method removes redundant operator computation on the remaining tokens, so the two strate-
gies target different sources of redundancy. As shown in Table 6, applying our method on top of VisionZip (Yang et al., 2025) or V2Drop (Chen et al., 2026) further reduces computation while preserving answer quality better than more aggressive pruning. This suggests that, especially in the lowtoken regime, retaining more visual evidence and skipping redundant operators is preferable to further shortening the visual sequence. Table 6: Combining operator-aware skipping with token pruning on Qwen3-VL-8B. Method (Token Ret.)
TFLOPs
GQA
TextVQA
MME
MMB
POPE
VisionZip (10%) VisionZip (3%) VisionZip + Ours (10%)
1.67 1.33↓0.34 1.31↓0.36
49.8 42.0↓7.8 49.9↑0.1
61.0 46.6↓14.4 60.7↓0.3
1717.1 1412.9↓304.2 1723.7↑6.6
69.0 47.9↓21.1 66.3↓2.7
79.5 65.5↓14.0 80.3↑0.8
V2Drop (10%) V2Drop (2%) V2Drop + Ours (10%)
2.59 1.99↓0.60 1.31↓1.28
54.7 41.5↓13.2 55.6↑0.9
60.4 44.9↓15.5 60.9↑0.5
2012.6 1482.2↓530.4 1999.2↓13.4
77.7 44.1↓33.6 77.4↓0.3
86.5 67.5↓19.0 86.4↓0.1
4.7
Qualitative Comparison
Figure 5 shows that operator-aware visual-token skipping preserves accurate responses on examples requiring visual reasoning and object recognition. These qualitative results suggest that our policy maintains answer capability while removing redundant visual computation.
5
Conclusion
We demonstrated that visual redundancy in MLLMs is answer-observable and operatordependent, with late visual-token updates often contributing little to answers. Our operator-aware skipping preserves all visual tokens while bypassing redundant attention, FFN, or both, reducing visual computation across multiple backbones with little performance loss, without irreversible evidence loss from pruning.
References Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. 2025. Divprune: Diversitybased visual token pruning for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9392–9401. Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, Jie Yu, Xiyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, and 4 others. 2025. Llava-onevision-1.5: Fully open framework for democratized multimodal training. Preprint, arXiv:2509.23661. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. Preprint, arXiv:2511.21631. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. Preprint, arXiv:2502.13923. Junjie Chen, Xuyang Liu, Zichen Wen, Yiyu Wang, Siteng Huang, and Honggang Chen. 2026. Variationaware vision token dropping for faster large visionlanguage models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3489–3499. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2025. An image is worth 1/2 tokens after layer 2: Plug-andplay inference acceleration for large vision-language models. In Computer Vision – ECCV 2024, pages 19–35, Cham. Springer Nature Switzerland. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. 2025. Mme: A comprehensive evaluation benchmark for multimodal large language models. In Advances in Neural Information Processing Systems, volume 38. Curran Associates, Inc. Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Yihong Huang, Fei Ma, Yihua Shao, Jingcai Guo, Zitong Yu, Laizhong Cui, and Qi Tian. 2026. Nüwa:
Mending the spatial integrity torn by vlm token pruning. Preprint, arXiv:2602.02951. Drew A. Hudson and Christopher D. Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In Computer Vision – ECCV 2016, pages 235–251, Cham. Springer International Publishing. Jiwan Kim, Kibum Kim, Wonjoong Kim, Byung-Kwan Lee, and Chanyoung Park. 2026. Why and when visual token pruning fails? a study on relevant visual information shift in mllms decoding. Preprint, arXiv:2604.12358. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. BLIP-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19730–19742. PMLR. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023b. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, Singapore. Association for Computational Linguistics. Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. 2025. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. Proceedings of the AAAI Conference on Artificial Intelligence, 39(5):5334–5342. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. Llavanext: Improved reasoning, ocr, and world knowledge. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2025. Mmbench: Is your multi-modal model an all-around player? In Computer Vision – ECCV 2024, pages 216–233, Cham. Springer Nature Switzerland. Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, ChengLin Liu, Lianwen Jin, and Xiang Bai. 2024c. OCRBench: On the hidden mystery of OCR in large multimodal models. Science China Information Sciences, 67(12):220102.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, KaiWei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, volume 35, pages 2507– 2521. Curran Associates, Inc. Jie Ma, Zhike Qiu, Jiayi Ji, Xiaoshuai Sun, and Rongrong Ji. 2026a. Look less, reason more: Blockwise attention skipping for efficient multimodal llms. Preprint, arXiv:2606.08511. Qiankun Ma, Ziyao Zhang, Haofei Wang, Zhen Song, Jie Chen, and Hairong Zheng. 2026b. Apet: Approximation-error guided token compression for efficient vlms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26306–26316. Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2025. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22857– 22867. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Yahong Wang, Juncheng Wu, Zhangkai Ni, Longzhen Yang, Yihang Liu, Chengmei Yang, Ying Wen, Lianghua He, Xianfeng Tang, Hui Liu, and Yuyin Zhou. 2026. When token pruning is worse than random: Understanding visual token information in vllms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 31910–31919. Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. 2025. Token pruning in multimodal large language models: Are we solving the right problem? In Findings of the Association for Computational Linguistics: ACL 2025, pages 15537–15549, Vienna, Austria. Association for Computational Linguistics. Gongli Xi, Ye Tian, Mengyu Yang, Huahui Yi, Liang Lin, Xiaoshuai Hao, Kun Wang, and Wendong Wang. 2026. Large vision-language models get lost in attention. Preprint, arXiv:2605.05668. Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin. 2025. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. Preprint, arXiv:2410.17247. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2025. Visionzip: Longer is better but not necessary in vision
language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19792–19802. Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, and Le Sun. 2025. Shortv: Efficient multimodal large language models by freezing visual tokens in ineffective layers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 329–339. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9556–9567. Qizhe Zhang, Mengzhen Liu, Lichen Li, Ming Lu, Yuan Zhang, Junwen Pan, Qi She, and Shanghang Zhang. 2025. Beyond attention or similarity: Maximizing conditional diversity for token pruning in mllms. In Advances in Neural Information Processing Systems, volume 38, pages 25438–25468. Curran Associates, Inc.
A
Related Work
Layer-wise visual-computation skipping. As high-resolution inputs, image tiling, and dense visual encoders increase the length of visual prefixes, the LLM prefill stage has become a major bottleneck for MLLM inference (Chen et al., 2025; Yuan et al., 2025; Lin et al., 2025; Ma et al., 2026a). One line of work reduces this cost by modifying how visual tokens are processed across Transformer layers. VTW withdraws visual tokens from later layers to avoid repeatedly propagating visual states through the full LLM backbone (Lin et al., 2025), while ShortV freezes visual-token states in layers identified as visually ineffective (Yuan et al., 2025). V-Skip studies block-wise attention skipping and shows that visual attention computation can be redundant in selected layers (Ma et al., 2026a). Related structural acceleration methods also exploit layer-wise or block-wise redundancy to reduce visual computation without directly deleting all visual inputs (Chen et al., 2025; Ma et al., 2026a). Together, these studies show that visual computation can be redundant in a depth-dependent manner, but they differ in the granularity at which redundancy is modeled. VTW and ShortV primarily act on full visual-token states, whereas V-Skip focuses on attention computation. Our work is complementary, instead of treating a layer or a single operator family as the default unit of analysis, we compare the answer-observable risks of visual-row attention and visual-row FFN within each layer. This operatorlevel view allows the policy to preserve the operator that remains useful at a given depth while skipping the redundant one, or to freeze the visual update when both operators are answer-silent. Visual-token pruning and compression. Another major direction accelerates MLLM inference by reducing the number of visual tokens. Early and recent methods prune, merge, or select visual tokens according to attention scores, token similarity, spatial redundancy, diversity, or output sensitivity (Chen et al., 2025; Shang et al., 2025; Yang et al., 2025; Xing et al., 2025; Zhang et al., 2025; Alvar et al., 2025; Ma et al., 2026b; Chen et al., 2026). These approaches directly shorten the visual sequence and can reduce both attention and FFN costs in subsequent layers. However, token reduction is inherently destructive, once visual tokens are removed, merged, or prematurely withdrawn, the corresponding spatial evidence cannot be recovered by later textual reasoning. This limitation is espe-
cially pronounced in fine-grained recognition tasks, including OCR-centric VQA, chart or diagram understanding, small-object reasoning, and visually grounded question answering, where the decisive evidence may occupy only a small image region or become relevant only after interacting with the question. Recent studies further show that aggressive token pruning can damage spatial integrity, shift relevant visual information during decoding, and even underperform random selection in certain settings (Wen et al., 2025; Huang et al., 2026; Kim et al., 2026; Wang et al., 2026). In contrast, our method preserves the full visual-token sequence throughout inference and reduces computation at the operator level. This avoids irreversible loss of fine-grained visual evidence while still removing answer-silent visual updates, matching the motivation in the introduction: efficient MLLM inference should preserve visual information and reduce only computation that is not observable from the answer side.
B
Implementation Environment
All experiments were conducted on an internal GPU cluster equipped with NVIDIA A800-SXM4 GPUs. Each compute node contains two Intel(R) Xeon(R) Platinum 8358 CPUs at 2.60GHz, providing 128 logical CPU cores in total, approximately 1 TiB of system memory, and 8 NVIDIA A800-SXM4 GPUs with 80GB VRAM each. The NVIDIA driver version is 535.161.08. The system CUDA toolkit is CUDA 12.2, while the PyTorch binaries used in our conda environments are built with CUDA 12.1. Table 7: Conda environments used for different model families. Model family
Python
PyTorch
Torchvision
Transformers
Accelerate
lmms-eval
Qwen2-VL Qwen3-VL LLaVA-1.5 LLaVA-NeXT
3.12.12 3.12.12 3.12.9 3.10.20
2.4.1 2.4.1 2.4.1 2.1.2+cu121
0.19.1 0.19.1 0.19.1 0.16.2+cu121
4.51.3 5.5.4 4.37.2 4.40.0.dev0
1.13.0 1.13.0 0.21.0 0.29.3
0.6.1 0.6.1 0.6.1 0.6.1
We use the official lmms-eval framework for all reported multimodal benchmark results and follow its default evaluation protocols. Since different model families rely on different dependency constraints, we maintain separate conda environments for Qwen2-VL, Qwen3-VL, LLaVA-1.5, and LLaVA-NeXT. Model checkpoints are loaded from local copies of the corresponding public Hugging Face repositories. The evaluated benchmark suite includes MME, MMBench, OCRBench, POPE,
GQA, TextVQA, ScienceQA, MMMU, AI2D, and VizWiz when applicable. Table 8: Additional package versions used in each environment.
In practice, rk can be sampled from a standard Gaussian distribution or a Rademacher distribution. For each probe, we define gl,k =
∂(rk⊤ hlast )
, ∆Vl
.
Model family
Qwen2.5VL
Qwen3VL
Llava1.5
Llava-Next
∂Hvl+1
datasets tokenizers sentencepiece qwen-vl-utils llava
2.16.1 0.21.4 0.2.1 0.0.14 –
2.16.1 0.22.2 0.2.1 0.0.14 –
2.16.1 0.15.1 0.2.1 – 1.2.2.post1
5.0.0 0.15.2 0.2.1 – 1.7.0.dev0
This quantity can be computed with a single reverse-mode gradient call, without explicitly constructing the full Jacobian. By the chain rule, ∂(rk⊤ hlast )
C
Computation of Answer-Observable Metrics
In Section 2, we define answer-observable influence (AOI) using the exact Jacobian from visualtoken states to the final prompt-token hidden state. This definition is mathematically direct, but computing the full Jacobian is prohibitively expensive for modern MLLMs. We therefore estimate AOI with randomized probes. This section derives the estimator used in our implementation. Let Hvl+1 ∈ RNv ×d denote the visual-token hidden states after layer l, and let
∂vec(Hvl+1 )
Jl =
∂hlast ∈ Rd×Nv d ∂vec(Hvl+1 )
be the Jacobian mapping perturbations of visualtoken states at layer l + 1 to perturbations of the final prompt-token hidden state. The exact firstorder change induced by the visual update is δhl = Jl vl ,
= Jl⊤ rk .
Therefore, gl,k = ⟨Jl⊤ rk , vl ⟩ = rk⊤ Jl vl = rk⊤ δhl . Thus, gl,k is a random projection of the exact answer-side perturbation δhl . The squared projection gives an unbiased estimator of the squared AOI: h i 2 Erk [gl,k ] = Erk (rk⊤ δhl )2 ⊤ = δh⊤ l E[rk rk ]δhl
= ∥δhl ∥22
∆Vl = Hvl+1 − Hvl be the visual-token update. We write vl = vec(∆Vl ) for its flattened representation. Let hlast ∈ Rd be the final prompt-token hidden state after prefill, and let
F
= AOI2l . With K independent probes, we estimate AOI as K
dl = AOI
1 X 2 gl,k K
!1/2 .
k=1
This estimator is consistent for AOIl as K increases, while requiring only vector-Jacobian products rather than the full Jacobian. In our diagnostic figures, the plotted AOI is computed using this randomized estimator. After estimating AOI, we compute answerobservable efficiency (AOE) by normalizing the estimated answer-observable influence by the visual update magnitude:
and the exact answer-observable influence is AOIl = ∥δhl ∥2 = ∥Jl vl ∥2 . Directly forming Jl is infeasible because it has size d × Nv d. Instead, we estimate the norm of Jl vl through random projections. Let rk ∈ Rd be an independent random probe satisfying E[rk rk⊤ ] = Id .
dl= AOE
dl AOI . ∥∆Vl ∥F + ϵ
For visualization across layers, we further normalize AOE within each model and dataset: gl= AOE
dl AOE . d j +ϵ maxj AOE
Table 9: Llava-Next comparison across multimodal benchmarks. We report benchmark scores and average performance retention relative to the vanilla model. Model
Method
TFLOPs
GQA
TextVQA
MME
MMB
MMMU
POPE
SQA
AI2D
OCRB
VizWiz
Avg. Ret.
LLaVA-NeXT
Vanilla VTW (K=16) ShortV (N=20) VSkip+ (N=20) V2Drop APET Ours (N=20)
100% 51% 51% 74% 70% 70% 67%
64.30 55.41 62.83 63.31 62.58 62.97 63.01
61.36 47.71 56.50 59.74 59.83 59.98 60.65
1851.44 1852.56 1854.51 1759.67 1853.23 1840.33 1797.57
67.10 66.75 66.92 67.01 67.25 67.18 66.75
36.00 35.56 35.89 36.22 36.00 37.11 36.78
87.61 87.41 87.38 87.35 87.98 87.79 88.31
70.15 70.00 69.26 69.83 69.92 69.71 69.96
65.35 65.38 64.48 65.25 64.75 65.01 65.03
52.20 6.90 41.40 48.80 50.30 50.60 50.60
60.75 54.86 57.09 59.35 56.83 60.72 57.99
100.0 86.5 96.0 98.2 98.4 99.4 98.8
Therefore, VUM measures how much visualtoken states move, AOI estimates how much this movement reaches the final prompt-token hidden state, and AOE measures the answer-observable influence induced per unit visual update. In implementation, we first run the original model forward pass and cache Hvl , Hvl+1 , and hlast for each layer. For each layer l, we sample K ran⊤ dom probes {rk }K k=1 , compute the scalar rk hlast , backpropagate it to obtain the gradient with respect to Hvl+1 , and take its Frobenius inner product with ∆Vl . Model parameters are kept frozen, and no ground-truth answer labels are used. This makes the diagnostic label-free and applicable to arbitrary multimodal prompts.
D
Experiments on LLaVA-NeXT
To further test architectural generality, we additionally evaluate on LLaVA-NeXT. Compared with LLaVA-1.5, LLaVA-NeXT uses stronger visual processing and is commonly evaluated under higher-resolution or multi-image settings, making visual-token computation more expensive. This provides a useful stress test for operator-aware skipping because high-resolution visual inputs amplify the cost of repeatedly updating visual-token representations inside the LLM backbone. Table 9 shows that our method remains effective in this higher-resolution setting. With 67% TFLOPs, our method preserves 98.8% average performance and achieves the best results among nonvanilla methods on TextVQA and POPE, while also ranking second on GQA, MMMU, and SQA. It also matches the best OCRBench score. These results indicate that the operator-level redundancy identified in the main experiments is not limited to the three primary backbones, but also appears in LLaVA-NeXT. Even when the visual input is more expensive to process, selectively skipping redundant visual operators can reduce computation while maintaining strong benchmark performance.
E
Calibration Sample Size Study
We study how the number of calibration samples from each benchmark affects the analysis decisions made by our method. For each of the seven calibration benchmarks used in the main experiments, we randomly sample 8, 16, or 20 examples and estimate the layer-wise operator risks for LLaVA1.5-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B. The resulting risk profile determines the selected layers and their operator-skipping policies, and the same policy is then evaluated on the full benchmark suite. Table 11 reports the average retention under different layer budgets. The goal is to test whether using more calibration samples leads to a substantially more reliable policy. Since calibration is used only for risk estimation and does not require labels or model training, a small sample size is desirable if it already supports stable policy selection. Table 11: Effect of calibration sample size under different layer budgets. Each setting samples the same number of examples from each of seven calibration benchmarks. Model
Samples Per Benchmark
Avg. Ret. under Layer Budget 8
12
16
20
24
28
32
36
LLaVA-1.5-7B
8 16 20
99.58 99.62 99.74
99.25 99.37 99.55
99.15 99.28 99.52
98.76 98.81 99.01
98.46 98.52 98.57
97.37 97.49 97.60
94.48 94.32 94.08
– – –
Qwen2.5-VL-7B
8 16 20
98.16 98.18 97.92
97.42 97.51 97.76
96.14 96.17 96.65
94.97 95.10 95.19
92.66 92.68 92.63
79.23 79.26 79.31
– – –
– – –
Qwen3-VL-8B
8 16 20
– – –
99.98 99.99 100.06
100.04 100.01 100.10
99.45 99.32 99.15
97.55 97.56 97.16
95.73 94.91 93.60
91.93 91.49 88.42
87.67 86.75 84.22
The results show that increasing the number of calibration samples can bring improvements at some layer budgets, but the gains are not consistently large or monotonic. For example, more samples slightly improve several LLaVA-1.5-7B settings, while the trends for Qwen2.5-VL-7B and Qwen3-VL-8B vary across budgets. This suggests that 8 samples per benchmark are already sufficient to support the risk analysis used by our method. Adding more samples can provide stronger evidence in some cases, but it is not always necessary for obtaining stable operator-skipping decisions.
Table 10: Sensitivity to the operator-skipping budget across the four evaluated MLLMs. Avg. Ret. is the mean retention relative to the vanilla model. Budget
GQA
TextVQA
MME
MMB
SQA
AI2D
OCRB
VizWiz
Avg. Ret.
0 8 12 16 20 24 28 32
61.94 61.14 61.10 61.02 60.84 60.71 60.23 57.89
58.21 57.18 57.12 56.30 56.40 56.22 55.61 53.80
1866.15 1878.10 1874.85 1856.99 1841.68 1797.31 1784.78 1728.08
LLaVA-1.5-7B (32 Layers) 64.18 36.11 85.94 69.46 64.18 35.67 86.43 69.61 64.26 35.44 86.82 69.36 64.18 35.89 86.93 69.36 64.52 35.67 87.26 69.16 64.61 35.78 87.22 69.16 65.38 34.67 86.94 69.16 62.03 35.67 85.01 69.21
55.18 55.51 55.34 55.18 55.18 55.28 54.53 52.46
31.50 31.40 31.00 31.30 30.20 29.60 28.30 25.10
54.09 53.23 52.73 52.45 52.75 53.46 53.06 52.76
100.0 99.6 99.3 99.1 98.8 98.5 97.4 94.5
0 8 12 16 20 24 28 32
64.30 63.75 63.55 63.36 63.01 62.30 61.75 59.64
61.36 61.37 61.08 60.77 60.65 58.36 56.49 50.67
1851.44 1845.83 1837.72 1832.66 1797.57 1777.99 1744.20 1616.91
LLaVA-NeXT-7B (32 Layers) 67.10 36.00 87.61 70.15 67.10 36.33 87.54 70.15 67.01 35.67 87.57 70.10 67.10 36.33 87.57 69.96 66.75 36.78 88.31 69.96 66.15 36.00 88.20 69.26 64.18 35.00 86.87 68.67 59.54 34.00 85.26 66.48
65.35 65.48 65.32 65.19 65.03 63.76 63.02 61.27
52.20 51.40 51.30 51.20 50.60 45.70 39.50 26.30
60.75 61.05 60.06 59.45 57.99 58.21 59.73 58.67
100.0 99.9 99.4 99.3 98.8 96.7 94.3 87.9
0 8 12 16 20 24 28
60.40 59.85 59.60 59.38 59.29 58.75 57.27
77.77 76.62 74.52 73.41 72.02 70.28 54.75
2515.82 2346.15 2343.01 2281.94 2263.38 2159.23 1884.70
Qwen2.5-VL-7B (28 Layers) 83.25 50.00 87.62 87.51 82.56 49.11 87.29 87.41 82.65 49.33 87.08 87.06 81.70 48.22 86.43 86.91 80.67 46.89 87.32 85.52 79.38 46.56 86.52 82.94 69.07 39.33 86.29 77.94
82.42 82.58 82.22 81.41 79.95 78.82 62.08
84.10 81.80 80.50 78.40 76.90 70.60 42.30
70.81 68.25 66.77 66.16 65.48 64.84 54.81
100.0 98.2 97.4 96.1 95.0 92.7 79.2
0 12 16 20 24 28 32 36
61.60 60.98 61.04 60.70 60.24 59.97 59.59 58.51
80.07 79.93 79.64 78.45 77.01 75.58 72.67 69.91
2390.38 2352.81 2356.06 2336.01 2273.65 2158.87 1909.52 1707.86
Qwen3-VL-8B (36 Layers) 84.79 51.33 89.13 94.40 84.62 52.44 88.83 94.70 84.45 53.67 88.76 94.70 83.08 52.89 89.43 94.35 82.05 53.00 89.31 90.53 80.93 51.00 88.42 85.72 78.01 49.44 88.42 85.72 75.34 48.22 87.42 81.85
83.78 83.65 83.65 82.55 80.18 76.39 76.39 72.38
82.80 81.90 81.40 80.40 77.90 71.20 71.20 63.80
69.37 70.78 70.12 71.70 69.77 66.86 66.86 63.77
100.0 100.0 100.0 99.5 97.6 94.0 91.9 87.7
F
MMMU
Ablation on Risk-Based Layer Allocation
We compare our budgeted layer allocation with a simple last-layer heuristic. As described in Section 3.2, our policy first ranks layers by the summed single-operator risk RA (l) + RF (l) and allocates the skipping budget to the lowest-risk layers. This design assumes that layers with smaller summed risk can safely discard their non-dominant visual operator earlier, whereas layers with larger summed risk should be postponed because even their non-dominant operator may still affect the output.
POPE
Table 12: Ablation against a heuristic that skips visual computation in the last 20 layers on Qwen3-VL-8B. Method
GQA
TextVQA
MME
MMB
SQA
AI2D
OCRB
Last-20 heuristic Ours (N=20)
60.57 60.70
77.40 78.45
2282.61 2336.01
82.47 83.08
93.01 94.35
81.44 82.55
78.80 80.40
This ablation isolates the effect of this layerallocation rule. For both methods, we keep the per-layer operation type fixed; each selected layer uses the same skip-operator decision or the same frozen update determined by the operator-risk profile. The only difference is which layers are selected. Our method selects the 20 layers with the smallest RA (l) + RF (l), while the Last-20 heuristic directly selects the final 20 Transformer layers regardless of their risk scores. Table 12 reports
What is the frisbee about to hit?
What type of vehicle do you think is the pedestrian behind of? ShortV
VTW
person VSkip
car
Ours
pole
VSkip
grass
What's the name of the book on the top of the pile?
The Painted Veil VSkip
bus Ours
car
fire truck
What is the brand of the phone? ShortV
VTW
ShortV
VTW
pole
ShortV
VTW
O-TECN
Mortal Coils
VSkip
Ours
Mortal Coils
What time is on the clock?
Ours
O-TECN
Ariel
O-TECN
C-TECH
What is the word written in the bottom of the box? ShortV
VTW
11:00 VSkip
8:00
12:15
not assigned VSkip
07009
CPN-4-50
Ours
VSkip
ShortV
ShortV
VTW
no VSkip
07809
Hardcast
Is there a handbag in the image?
07009 Ours
CPN-4-50 Ours
FLEXIBLE DUCT CONNECTOR
11:45
What is the value for 'CODE ASSIGNED'? Answer this question using the text in the image directly. VTW
ShortV
VTW
no Ours
no
yes
Figure 6: Additional qualitative examples of operator-aware visual-token skipping.
the Qwen3-VL-8B comparison, testing whether risk-based allocation is preferable to choosing late layers purely by position. The results support our hypothesis. With the same number of selected layers and the same perlayer operator actions, risk-based allocation consistently outperforms the Last-20 heuristic across all reported benchmarks. This indicates that smallRA + RF layers are indeed safer places to discard non-dominant visual operators, while simply choosing the final layers may remove useful visual computation whose operator risk remains high.
G
Different Budget of Our Method
Table 10 studies how the operator-skipping budget affects performance across four MLLM backbones. Overall, the results show that our operatoraware policy is not tied to a particular architecture, LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL, and Qwen3-VL all maintain high average retention under moderate budgets, despite their different visual encoders, language backbones, and dependency
stacks. In particular, small and medium budgets usually preserve nearly all vanilla performance, and several individual benchmarks even improve over the budget-0 setting. This suggests that the skipped operators are often answer-silent rather than uniformly useful, supporting the reliability of the proposed answer-observable selection criterion. The budget trend also reveals a consistent efficiency–accuracy pattern. As the budget increases, average retention decreases smoothly rather than collapsing abruptly, indicating that the policy removes redundant visual computation in a controlled manner. This behavior is especially important for practical deployment, users can choose conservative budgets when accuracy is the priority, or larger budgets when higher efficiency is needed. Across all models, moderate budgets retain strong performance, while aggressive settings expose the trade-off when too much visual computation is skipped. These results show that operator-level visual skipping is broadly applicable and robust under different layer-level budgets.