VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models Huawei Ji1 , Yuanhao Sun1 , Yuan Jin1 , Cheng Deng2 , Jiaxin Ding1 * Luoyi Fu1 , Xinbing Wang1 1
Shanghai Jiao Tong University, Shanghai, China, 2 University of Edinburgh, Edinburgh, UK {sjtu3365981, h_iden, lemon0703, jiaxinding, yiluofu, xwang8}@sjtu.edu.cn
arXiv:2604.15188v1 [cs.CV] 16 Apr 2026
Abstract Visual token pruning methods effectively mitigate the quadratic computational growth caused by processing high-resolution images or long video frames in vision-language models (VLMs). However, existing approaches rely on predefined pruning configurations without determining whether they achieve computationperformance optimality. In this work, we introduce VisPCO, a novel framework that formulates visual token pruning as a Pareto configuration optimization problem to automatically identify optimal configurations. Our approach employs continuous relaxation and straightthrough estimators to enable gradient-based search, solved via the Augmented Lagrangian method. Extensive experiments across 8 visual benchmarks demonstrate that VisPCO effectively approximates the empirical Pareto frontier obtained through grid search and generalizes well across various pruning methods and VLM architectures. Furthermore, through learnable kernel functions, we investigate layerwise pruning patterns and reveal that multi-step progressive pruning captures VLMs’ hierarchical compression structure, achieving superior computation-performance trade-offs compared to single-layer approaches.
1
Introduction
Large-scale vision-language models (LVLMs) process both visual and textual features as input, enabling them to learn unified multimodal representations and perform cross-modal reasoning. Recent studies have shown that higher-resolution image inputs can effectively improve the model’s understanding and generation performance (Guo et al., 2024; An et al., 2025; Chen et al., 2024b). Meanwhile, tasks such as video understanding require models to process numerous frames to capture temporal continuity and dynamic semantics (Lin et al., 2024; Xu et al., 2025). Both scenarios significantly * Corresponding author
(a) Pareto Frontier
(b) Base Models
(c) Pruning Methods
(d) Different Datasets
Figure 1: Pareto frontiers across different configurations: (a) The Pareto frontier connects optimal pruning configurations. (b) Pareto frontiers across different VLMs. (c) Pareto frontiers across different pruning methods. (d) Pareto frontiers across different datasets.
increase the number of visual tokens, leading to quadratic growth in computational costs. To solve this problem, various visual pruning algorithms for VLMs have emerged. These methods mainly focus on designing different importance scoring mechanisms to prune redundant visual tokens at single or multiple layers. For instance, FastV (Chen et al., 2024a), DynamicLLaVA (Huang et al., 2024), VTW (Lin et al., 2025), and TOPV (Yang et al., 2025) prune visual tokens at a specific LLM layer using predefined pruning ratios. In contrast, ATP-LLaVA (Ye et al., 2025b), HiMAP (Zhou et al., 2024), and SparseVLM (Zhang et al., 2024) apply dynamic pruning ratios across multiple selected layers. All these works aim to reduce computational costs (e.g., FLOPs) while maintaining model performance. However, two critical questions remain unexplored in existing works. First, it is unclear whether current pruning configurations (i.e., pruning positions and ratios) achieve the optimal computation-
Visual Token Pruning
✂️ Re-rank & Filtering ��%
Forward
Backward
�
��’
=
LM Head
Block k
Block L
Pruned Layer L
Pruned Layer k
Block 1 Filter (2)
��+1 �2
��� ⊙ �1 ��
��’ = �� ⋅ �(�, 퐿) (1)
System Tokens
�1 �2
���’ ���
…
Ratio Predictor
��
R(B, L)
(b) Differentiable Search
…
…
���
��� �1
R(B, k)
…
Rank
…
…
Text Query
�� ��+1
��+1 �2
…
�1 �2
…
…
Image Input
R(B, 1)
Ratio Predictor
…
…
System Prompt
ℒ���푖
ℒ푑푖 �
��
+
Budget
Score
What is the man in the blue jacket trying to climb? Given the follow -ing options, choose the correct ...
Pruned Layer 1
��
Transformer Block k
You are a helpful, honest, and reliable AI assistant. You can understand both images and text....
Projector
…
Transformer Block k+1
��
Visual Encoder
Language Model Head
(a) VisPCO Training Pipeline
�
…
B. mountain
Visual Token Pruning Configuration Optimization
…
Generated Answer
Stop Grad ! ! ! Visual Tokens
ℒ퐹퐿�
(1) Soft Threshold �
��+1 �2 ~ (2) Soft Mask �푖 ~ �푖
Text Tokens
…
��� �1
�푖 − �
Trainable
Retained Tokens
…
��
⋅ Round
Figure 2: Illustration of our VisPCO framework. (Left) Overview of the visual token pruning process. After each transformer block, visual tokens are ranked by their importance scores and low-scoring tokens are filtered out. (Right) Upper panel: The overall architecture of VisPCO, where the trainable Ratio Predictor, a lightweight surrogate network, determines the pruning ratio to guide token compression at each layer. Lower panel: The gradient disconnection problem encountered during end-to-end training of the Ratio Predictor and our proposed solution.
performance trade-off. Second, how to adjust these configurations efficiently to reach the optimal remains an open problem. Addressing these two questions is crucial for the efficient deployment of VLMs in real-world applications, e.g., edge device deployment and mobile vision systems. To this end, we introduce the concept of Pareto optimality to characterize the optimal computationperformance trade-off (Arrow and Debreu, 2024). Pareto optimality is a classic concept in multiobjective optimization that describes a state where no further improvement can be made among multiple conflicting objectives. In this work, we define a point as Pareto-optimal if we cannot simultaneously reduce computational cost and improve performance. The curve connecting these points is called the Pareto frontier. In Figure 1, we visualize different pruning configurations in the computation-performance space and their corresponding Pareto frontier. Each point represents the experimental result of one configuration. Figure 1(a) reveals that under the same computational budget, different pruning configurations exhibit significant performance variations. For instance, at a computational budget of 1.7 TFLOPs, the performance gap between different configurations can reach up to 23.6%. Moreover, Paretooptimal frontiers are not fixed. Instead, they vary with VLM architectures, pruning methods and image complexity as illustrated in Figure 1(b-d). In
practice, to determine the best pruning configuration under a fixed computational budget, we typically need to perform grid sampling over the configuration search space. This involves conducting numerous experiments to measure performance and cost across different configurations, and then selecting the optimal one for deployment. However, this process is prohibitively time-consuming and resource-intensive. In this paper, we propose a computation budget-aware method for Visual token Pruning Configuration Optimization, termed VisPCO. This approach employs a learnable surrogate model to automatically predict pruning configurations on the Pareto frontier given a computational budget, thereby achieving optimal model performance. As shown in Figure 2, unlike traditional grid search methods, VisPCO employs efficient gradient descent for search, significantly reducing search costs. To address the discrete and non-differentiable nature of visual token pruning, we introduce continuous relaxation techniques and straight-through estimators for end-to-end optimization. For the optimization objective, we formulate it as a Pareto optimization problem with non-convex inequality constraints and solve it using the Augmented Lagrangian method (Nocedal and Wright, 2006). Furthermore, we investigate the layer-wise pruning patterns in VLMs. Specifically, we explore how the optimal pruning ratio varies across different layers
when progressively compressing visual tokens. To model this variation, we use learnable kernel functions to parameterize the pruning ratio distribution across layers. By evaluating how well different kernel functions approximate the Pareto frontier, we identify the intrinsic pruning patterns that lead to optimal computation-performance trade-offs. Our contributions can be summarized as follows: • We propose VisPCO, a differentiable framework that automatically finds Pareto-optimal configurations via gradient-based methods, eliminating the prohibitive cost of exhaustive grid search. • Experiments on 8 benchmarks demonstrate that VisPCO effectively approximates the empirical Pareto frontier and generalizes across various pruning methods and VLM architectures. • We reveal non-uniform visual token redundancy across layers via learnable pruning kernels, showing multi-step pruning is most effective under tight budgets. Our code is available at https: //github.com/JHW5981/VisPCO.
2
Related Work
2.1
Visual Token Pruning
Visual token pruning accelerates VLMs by reducing computational costs from processing hundreds of visual tokens. Single-layer approaches perform one-shot reduction at specific layers: FastV (Chen et al., 2024a) prunes tokens after layer 2 using attention scores, Dynamic-LLaVA (Huang et al., 2024) dynamically adjusts token retention ratios based on input characteristics, VTW (Lin et al., 2025) withdraws all tokens after sufficient absorption, and TopV (Yang et al., 2025) optimizes configurations at inference time. Multi-layer progressive methods distribute reduction across multiple layers: ATP-LLaVA (Ye et al., 2025b) adaptively prunes tokens at different depths with layer-specific ratios, SparseVLM (Zhang et al., 2024) progressively reduces tokens across layers with recycling mechanisms, and PyramidDrop (Xing et al., 2024) implements stage-wise pyramid reduction that preserves more tokens in shallow layers. These strategies achieve 40-90% computational savings while maintaining competitive performance. However, the pruning configurations in these methods are either predefined or heuristically determined, and it remains unclear whether they achieve the optimal computation-performance trade-off.
2.2
Pruning Configuration Optimization
While most pruning methods rely on predefined configurations, recent works explore adaptive strategies to optimize pruning ratios across layers. FitPrune (Ye et al., 2025a) employs binary search over attention statistics to minimize distribution divergence and generate layer-wise pruning recipes. G-Search (Zhao et al., 2025) combine greedy search with Bayesian-optimized sigmoid functions to approximate optimal retention ratios. ATP-LLaVA (Ye et al., 2025b) introduces learnable modules for training-based optimization of layer-specific sparsity. SparseVLM (Zhang et al., 2024) employs rank-based adaptive determination of per-layer sparsification ratios. More recent methods explore input-adaptive configurations: AIM (Zhong et al., 2025) develops schedulercontrolled pruning with adjustable parameters, and MADTP (Cao et al., 2024) utilizes learnable thresholds for instance-wise adaptive pruning. Despite these advances, existing approaches focus primarily on performance preservation rather than budgetaware optimization. They lack mechanisms to systematically identify near-optimal configurations under varying computational constraints.
3
VisPCO
3.1
Pareto Optimization
We formulate the visual pruning configuration optimization problem as finding the optimal layerwise pruning ratios r = [r1 , r2 , . . . , rL ] ∈ [0, 1]L , where L denotes the number of layers and ri represents the token retention ratio at layer i (relative to the original number of visual tokens). Our objective is to identify a configuration r̄ on the Pareto frontier that achieves optimal model performance under a given computational budget. To quantify performance degradation, we define the pruned VLM output logits as ˆl and the original output logits as l, and measure their discrepancy using KL divergence: Ldistill (r) = DKL softmax(ˆl) softmax(l) . (1) A smaller value of Ldistill indicates less impact of pruning on model performance. Meanwhile, we define the computational cost function as: F (r) =
L X 24(Nt + ri Nv )D2 + 4(Nt + ri Nv )2 D (2) i=1
where Nt is the number of text tokens, Nv is the number of visual tokens, and D is the hidden dimension. Detailed derivation of the FLOPs computation is provided in Appendix A. Therefore, the Pareto optimization problem for the computationperformance trade-off can be formulated as the following constrained optimization problem: min
Ldistill (r)
s.t.
F (r) ≤ B,
(3)
where B is the given computational budget. Considering that the objective function Ldistill is typically non-convex, we employ the Augmented Lagrangian Method for numerical iterative solving. Definition 1 (Augmented Lagrangian Method). Consider the equality-constrained optimization problem: min
j = 1, . . . , l
where f (x) : Rn → R is the objective function to be minimized, and hj (x) : Rn → R are the equality constraint functions. The augmented Lagrangian function is defined as: ϕ(x, v, λ) = f (x) −
l X
l
vj hj (x) +
j=1
λX 2 hj (x), 2 j=1
(4)
where v = [v1 , . . . , vl ] is the Lagrange multiplier vector and λ > 0 is the penalty parameter. Theorem 1 (Adapted from Bertsekas, 2014). Let x̄ and v̄ satisfy the second-order conditions for a local optimal solution of the problem. Then there exists λ′ ≥ 0 such that for all λ > λ′ , x̄ is a strict local minimizer of ϕ(x, v̄, λ). The proof of Theorem 1 is provided in Appendix B.1. Based on this theorem, we can develop an iterative algorithm with bounded λ, avoiding the ill-conditioning of quadratic penalty methods (Nocedal and Wright, 2006) and the convergence difficulties of standard Lagrangian methods (Bertsekas, 2014). Specifically, at iteration k, let x(k) be the minimizer of (4) with respect to x. The multiplier update rule is: (k+1)
vj
(k)
= vj
min
Ldistill (r)
s.t.
B − F (r) − y 2 = 0.
(6)
The corresponding augmented Lagrangian function is defined as: ϕ̃(r, y, w, λ) = Ldistill (r) − w B − F (r) − y 2 2 λ + B − F (r) − y 2 . 2
(7)
By completing the square with respect to y, we can eliminate the dependence on y and obtain the simplified augmented Lagrangian (see Appendix B.2): 1 z 2 − w2 , (8) 2λ where z = max 0, w − λ B − F (r) . The problem is thus transformed into minimizing the unconstrained objective ϕ(r, w, λ). Using the iterative algorithm in Algorithm 1, we can obtain the optimal pruning configuration r̄. ϕ(r, w, λ) = Ldistill (r) +
f (x)
hj (x) = 0,
s.t.
Returning to our optimization problem (3), we introduce an auxiliary variable y to convert the inequality constraint into an equality constraint:
− λhj (x(k) ),
j = 1, . . . , l (5)
Through this iterative update, we have v(k) → v̄ and x(k) → x̄, with convergence rate typically measured by ∥h(x(k) )∥/∥h(x(k−1) )∥.
3.2
Differentiable Configuration Search
Despite having the Pareto optimization objective Eq. (8) and iterative Algorithm 1, computing ∇r Ldistill (r) faces two critical non-differentiability challenges, as illustrated in the bottom right of Figure 2. First, the discretization of retained token counts introduces non-differentiability. For layer i with pruning ratio ri and Nv visual tokens, the retained token count ki = ⌊ri · Nv ⌋ involves a floor operation that causes vanishing gradients, preventing backpropagation-based updates of ri . Second, selecting the top-ki tokens based on importance scores involves discrete operations that block gradient flow. We propose the following two methods to address these challenges respectively: Continuous Relaxation. To address the first challenge, we adopt a continuous relaxation strategy. We retain the floating-point form k̃i = ri · Nv and design a soft interpolation method using a Gaussian kernel to estimate the selection threshold. Specifically, we first sort the importance scores of all visual tokens in descending order to obtain {si1 , si2 , . . . , siNv }, where sij denotes the score of the j-th token in layer i after sorting. Traditional hard thresholding directly uses the score at position
⌊k̃i ⌋ as the threshold, which leads to vanishing gradients. Instead, we employ Gaussian kernel-based soft interpolation to maintain differentiability: (j − k̃i )2 wij = exp − , 2σ 2
P Nv τi =
j=1 wij sij , PNv j=1 wij
(9)
where wij is the Gaussian weight for the j-th token in layer i. The parameter σ controls the kernel width, balancing between approximation accuracy and gradient stability. As σ → 0, the soft threshold τi converges to the hard threshold si⌊k̃i ⌋ . The gradient of τi with respect to ri can be expressed as: N
v X wij (j − k̃i ) ∂τi = Nv (sij − τi ), P v ∂ri σ2 N w il l=1
(10)
j=1
which remains well-defined for all ri ∈ (0, 1), enabling smooth gradient flow via backpropagation. Straight-Through Estimator. To address the second challenge, we employ the Straight-Through Estimator (STE) strategy. Inspired by GumbelSoftmax (Jang et al., 2016), we use discrete hard decisions in the forward pass and continuous soft approximations in the backward pass. Given the threshold τi , for each visual token j in layer i, we compute both a hard selection mask and a soft mask: sij − τi , mij = I[sij ≥ τi ], m̃ij = sigm T (11) where I[·] is the indicator function, sigm(·) is the sigmoid function, and T is the temperature parameter. The soft mask m̃ij provides a smooth approximation: as T → 0+ , m̃ij → mij . The STE combines both masks through: m̂ij = m̃ij + sg(mij − m̃ij ),
(12)
where sg(·) denotes the stop-gradient operation. This formulation ensures m̂ij = mij in the forward pass, while the backward gradient satisfies: X ∂L ∂ m̃ij X ∂L m̃ij (1 − m̃ij ) ∂L = · =− · , ∂τi ∂ m̂ij ∂τi ∂ m̂ij T j j (13)
where ∂∂L m̂ij denotes the upstream gradient propagated from subsequent layers and is well-defined since m̂ij is treated as a differentiable proxy of mij during backpropagation. These provide biased but low-variance gradient estimators that enable end-to-end optimization of both token scores and adaptive thresholds.
Algorithm 1 Training pipeline of VisPCO Initial configuration r(0) , initial Lagrange multiplier w(1) , penalty parameter λ, convergence threshold ϵ > 0, update coefficients α > 1, β ∈ (0, 1) 2: Output: Locally optimal configuration r̄ and optimal multiplier w̄ 3: Set k = 1 4: while True do 5: Starting from r(k−1) , solve optimization problem min ϕ(r, w, λ) 6: /* Train using gradient descent */ 7: Obtain solution r(k) 8: if ∥B − F (r(k) )∥ < ϵ then /* Constraint satisfied, training con9: verged */ 10: break 11: end if ∥B−F (r(k) )∥ ≥ β then 12: if ∥B−F (r(k−1) )∥ 13: /* Update penalty parameter */ 14: λ ← αλ 15: end if 16: /* Update Lagrange multiplier */ 17: w(k+1) ← w(k) − λ(B − F (r(k) )) 18: k ←k+1 19: end while 20: return r̄ = r(k) , w̄ = w (k) 1: Input:
3.3
Learnable Kernel Functions
With VisPCO, we can automatically search for configurations on the Pareto frontier in a differentiable manner. To further investigate how different pruning patterns affect the Pareto frontier, we impose structural constraints on the pruning configuration search space. In practice, visual token pruning exhibits a monotonically non-increasing pattern across layers: deeper layers tend to retain fewer tokens. We leverage this prior by introducing learnable kernel functions to parameterize the layer-wise pruning ratios. This design offers two key benefits: (1) it provides interpretability for visual token pruning behavior, and (2) it reduces the parameter search space, ensuring both optimization stability and computational efficiency. We consider two pruning scenarios: single-layer pruning and multi-layer pruning. For the first case, we employ a parameterized p-sigmoid kernel to model a sharp transition at layer k: Ks (i; k, r, γ) = 1 + (r − 1) · sigm(γ(i − k)),
(14)
where i is the layer index, k is the pruning position, r ∈ (0, 1] controls the final retention ratio, γ > 0 is the sharpness parameter, and sigm(·) is the sigmoid function. With sufficiently large γ, the transition approaches a step function retaining all tokens before layer k and applying retention ratio r thereafter, effectively approximating single-layer pruning while remaining differentiable. For the second case, we explore several kernel functions to capture diverse pruning patterns. First, inspired by the Ebbinghaus forgetting curve from cognitive psychology (Fuchs, 2000), we design an exponential decay kernel to investigate whether VLMs exhibit similar attention decay patterns for visual tokens across layers: Ke (i; k, r) = r · e−k·i .
(15)
Second, we consider a linear decay kernel that models uniform, gradual token reduction: Kl (i; k, r) = −k · i + r.
(16)
Third, motivated by findings in (Zhao et al., 2025) showing that attention score rankings remain similar across layers and follow a sigmoid-like curve, we adopt a gentle p-sigmoid kernel. Following Eq. (14), we use a smaller γ to capture smooth, progressive pruning transitions. Finally, inspired by hierarchical representation learning where deep networks perform feature extraction at different levels, we introduce a multi-step sigmoid kernel to model the hypothesis that VLMs compress information at multiple critical layers: M X (2j − 1)L 1−r Kms (i; k, r, M ) = 1− ·σ k i − , M 2M j=1 (17)
where M is the number of pruning steps, and the j-th step is centered at layer (2j−1)L 2M , which evenly distributes the steps across layers. This design creates M evenly-spaced decision points for progressive information compression across layers. Collectively, these learnable kernels capture a wide spectrum of pruning patterns, enabling the model to discover task-specific compression strategies. The parameters k and r are dynamically predicted by a lightweight surrogate neural network fθ . As illustrated in the top-right of Figure 2, the network takes as input the concatenated visual and textual embeddings along with the computational
budget B, and computes layer-wise retention ratios ri = K(i; k, r) according to the selected pruning pattern. All outputs ri are clipped to [0, 1]. Through gradient-based optimization, the surrogate network learns which layers and tokens are critical. This provides mechanistic insights into how VLMs prioritize visual information across layers.
4
Experiments
4.1
Implementation Details
We use Qwen2.5VL-3B (Bai et al., 2025) as the base model and construct our training set by downsampling 30K samples from LLaVAInstruct-150K (Liu et al., 2023). To mitigate the long-tail distribution of image resolutions, we apply resolution-based resampling to prevent performance degradation on rare image sizes. Our evaluation spans three categories of benchmarks: visual question answering (A-OKVQA (Schwenk et al., 2022), VizWiz (Bigham et al., 2010), SEEDBench (Li et al., 2023)), multimodal reasoning (MMBench (Liu et al., 2024a), MME (Fu et al., 2025)), and chart understanding (ChartQA (Masry et al., 2022), OCRBench (Liu et al., 2024b), TextVQA (Singh et al., 2019)). For single-layer pruning, we set the penalty parameter λ = 100, convergence threshold ϵ = 0.01, and update coefficients α = 2, β = 0.5. The Gaussian kernel width σ = 10 and temperature T = 0.1 control the continuous relaxation and straight-through estimator, respectively. We use the AdamW optimizer with a learning rate of 4 × 10−4 and batch size of 16. All experiments are conducted on 8 NVIDIA H20 GPUs (96GB each). Other training configurations are provided in Appendix C.1. 4.2
Pareto Frontier Approximation
We apply VisPCO to three representative pruning methods: FastV (Chen et al., 2024a), FitPrune (Ye et al., 2025a), and SparseVLM (Zhang et al., 2024), which employ different importance scoring mechanisms for visual tokens. Table 1 compares performance before and after applying VisPCO under different FLOPs budgets. For each method, results without VisPCO are obtained by sampling multiple configurations and averaging their performance, with standard deviations reported (± std). Figure 3(left) illustrates the empirical Pareto frontier obtained through grid search and the predicted Pareto frontier by VisPCO, with computational budget on the x-axis and average accuracy across 8
Table 1: Comparison of pruning methods with and without VisPCO on eight benchmarks under different budgets. Results without VisPCO are averaged over multiple sampled configurations meeting the budget constraint (± std). Method
AOKVQA
VizWiz
Qwen2.5VL-3B
90.2
75.1
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
88.2 ± 0.4 88.4 88.5 ± 0.3 88.6 89.1 ± 0.5 89.6
72.9 ± 0.9 73.8 73.1 ± 0.5 73.5 73.9 ± 0.4 74.1
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
74.7 ± 10.1 84.8 75.9 ± 9.8 85.2 77.1 ± 8.7 85.9
60.3 ± 9.6 69.4 62.6 ± 8.2 69.0 63.4 ± 7.7 69.4
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
33.3 ± 2.3 35.5 33.6 ± 2.1 35.5 33.8 ± 2.1 35.6
30.4 ± 1.6 31.7 31.2 ± 1.3 31.5 31.5 ± 1.1 31.6
SEED MMB MME† ChartQA Upper Bound, 100% Budget, ∼3.56 TFLOPs 75.6 79.8 84.2 64.1 Reduce FLOPs Budget to 90%, ∼3.20 TFLOPs 72.4 ± 0.9 76.4 ± 0.5 81.3 ± 0.5 62.2 ± 0.8 73.2 76.9 81.7 62.9 73.4 ± 0.4 76.9 ± 0.6 82.1 ± 0.3 62.2 ± 0.7 73.8 77.5 82.4 62.9 74.2 ± 0.5 77.6 ± 0.4 82.5 ± 0.6 63.1 ± 0.6 74.6 77.9 82.8 63.5 Reduce FLOPs Budget to 50%, ∼3.56 TFLOPs 61.5 ± 8.1 62.4 ± 9.3 68.8 ± 9.1 51.6 ± 9.9 67.6 71.2 77.1 58.1 63.1 ± 7.2 63.9 ± 8.6 69.9 ± 8.2 51.9 ± 8.3 68.1 71.9 77.6 58.4 63.9 ± 6.8 64.5 ± 8.2 70.8 ± 7.9 52.8 ± 8.1 68.4 72.4 77.9 58.8 Reduce FLOPs Budget to 10%, ∼0.36 TFLOPs 44.5 ± 2.7 33.0 ± 2.5 39.7 ± 1.4 29.8 ± 4.1 46.9 35.5 40.1 33.2 44.9 ± 2.5 33.9 ± 2.3 40.3 ± 1.1 30.5 ± 3.7 47.1 35.8 40.4 33.3 45.3 ± 2.4 34.2 ± 2.2 40.6 ± 1.0 30.9 ± 3.5 47.3 35.8 40.9 33.5
visual benchmarks on the y-axis. Performance gains at moderate budgets. As shown in Table 1, VisPCO’s benefits vary significantly across different computational budget regimes. At extreme budgets, configuration selection has limited impact. For example, at 90% budget, performance varies by less than 1 percentage point across different configurations, as resources are abundant enough that most configurations perform well. Similarly, at very low budgets, severe resource constraints limit all configurations. In contrast, moderate budgets (e.g., 50%) present a critical regime where configuration choice significantly impacts performance—different configurations can vary by up to 19 percentage points. This substantial performance gap demonstrates the importance of principled configuration optimization and validates the need for methods like VisPCO. Quality of frontier approximation. As shown in Figure 3(left), the predicted Pareto frontiers exhibit near-perfect alignment with empirical frontiers. This validates the effectiveness of our kernelbased approximation approach in capturing the true computation-performance trade-off landscape. Table 2 presents a comprehensive comparison with existing approaches, including predefined pruning configuration strategies, training-based methods, and random search baselines (Random-N denotes selecting the best from N random samples). We evaluate both search efficiency and performance. VisPCO outperforms all baseline
OCRB
TextVQA
Avg (%)
74.6
81.3
78.1
71.6 ± 0.7 72.3 71.9 ± 0.6 72.5 72.5 ± 0.5 72.9
79.1 ± 0.6 79.5 79.5 ± 0.6 80.0 79.9 ± 0.3 81.2
75.5 ± 0.7 76.1 76.0 ± 0.5 76.4 76.2 ± 0.5 77.1
59.2 ± 9.1 67.8 62.4 ± 6.9 67.9 63.3 ± 6.4 68.2
65.9 ± 10.8 75.9 66.6 ± 9.8 76.3 67.6 ± 9.4 76.6
63.1 ± 9.5 71.5 64.5 ± 8.4 71.8 65.4 ± 7.9 72.2
8.3 ± 2.1 10.1 9.1 ± 2.0 10.2 9.6 ± 1.9 10.4
33.7 ± 2.8 36.1 34.4 ± 2.2 36.3 34.6 ± 2.1 36.4
31.6 ± 2.4 33.6 32.2 ± 2.2 33.8 32.6 ± 2.0 33.9
Table 2: Comparison of configuration search methods. Time represents search time to identify optimal configurations (training time for VisPCO). Random-N denotes random sampling with N evaluations. All methods target similar computational budgets. Methods
FLOPs (T) ↓
Time (h) ↓
MMB ↑
SEED ↑
TQA ↑
VTW G-Search ATP-LLaVA MADTP AIM
2.34 2.58 2.23 3.91 2.33
1+ 48+ 6+ -
36.5 42.1 37.2 42.4 39.5
44.1 47.5 45.5 46.8 43.8
73.2 80.2 77.2 79.1 74.6
Random-40 Random-80 Random-160
2.18 2.20 2.20
12+ 24+ 48+
31.2 42.8 44.0
36.6 47.7 48.1
70.9 80.1 82.4
VisPCO
2.20
1+
43.6
47.5
81.3
methods including VTW (Lin et al., 2025), GSearch (Zhao et al., 2025), ATP-LLaVA (Ye et al., 2025b), MADTP (Cao et al., 2024), AIM (Zhong et al., 2025), and moderate random search variants, while requiring only 1 hour of training time. Notably, MADTP requires additional training of MAG and DTP modules (6+ hours) and incurs substantially higher FLOPs (3.91T). AIM is training-free with a fixed layer-wise pruning strategy; at comparable FLOPs (2.20T vs. 2.33T), VisPCO achieves clearly superior performance (MMB: 43.6 vs. 39.5; TQA: 81.3 vs. 74.6; SEED: 47.5 vs. 43.8). 4.3
Cross-Model Generalization
To validate the generalization capability of VisPCO across different VLM architectures, we apply it to Gemma3-4B (Team et al., 2025) and LLaVA†
MME scores are normalized to percentages.
Figure 3: Experimental results of VisPCO. (Left) Comparison between empirical and predicted Pareto frontiers. (Middle) Comparison between empirical and predicted Pareto frontiers across different VLM architectures. (Right) Comparison of Pareto frontiers among different pruning patterns. Table 3: Performance of different VLMs with and without VisPCO at 50% FLOPs budget (± std).
Table 5: Comparison of performance under different pruning patterns at 50% FLOPs budget.
Models
AOKVQA
MMBench
TextVQA
Avg (%)
Patterns
LLaVA-7B + VisPCO
50.3 ± 10.7 60.8
26.5 ± 11.5 38.0
64.2 ± 11.4 75.2
47.0 ± 11.2 58.0
Single-Layer
-
84.8
71.2
75.9
77.3
Gemma3-4B + VisPCO
41.2 ± 12.4 53.4
44.7 ± 12.9 57.3
33.6 ± 12.3 45.5
39.8 ± 12.5 52.1
74.7 ± 10.1 84.8
62.4 ± 9.3 71.2
65.9 ± 10.8 75.9
67.7 ± 10.1 77.3
Multi-Layer
Qwen2.5VL-3B + VisPCO
Linear Exponential P-Sigmoid Multi-Step
82.6 82.2 81.9 84.9
70.9 70.4 69.6 71.8
74.9 74.4 74.1 76.7
76.1 75.7 75.2 77.8
Table 4: Hardware performance measurements at 50% compute budget across models. TTFT: time to first token. Throughput: tokens generated per second. Avg. Perf.: average accuracy across benchmarks. Method
Budget
TTFT (ms) ↓
Throughput (tokens/s) ↑
Avg. Perf. ↑
Qwen2.5VL-3B + FastV + FastV + VisPCO
100% 50% 50%
83 ± 3 74 ± 2 76 ± 3
18 ± 2 20 ± 3 20 ± 2
78.1 63.1 71.5
LLaVA-v1.5-7B + FastV + FastV + VisPCO
100% 50% 50%
114 ± 12 96 ± 10 97 ± 8
14 ± 6 16 ± 6 16 ± 4
63.9 41.4 52.3
Gemma3-4B + FastV + FastV + VisPCO
100% 50% 50%
94 ± 6 81 ± 7 81 ± 6
16 ± 4 18 ± 5 18 ± 5
68.9 38.8 50.8
v1.5-7B (Liu et al., 2023). Table 3 presents representative results using FastV (Chen et al., 2024a) under a 50% FLOPs budget. Consistency across architectures. While Qwen2.5VL exhibits a wider performance range compared to Gemma3 and LLaVA, VisPCO consistently selects optimal configurations. Figure 3(middle) shows that predicted Pareto frontiers closely match the empirical frontiers across all architectures, demonstrating robust generalization. Notably, Qwen2.5VL’s frontier is positioned in the upper-left region, indicating superior accuracyefficiency characteristics. This advantage stems from its native image resolution, whereas Gemma3 and LLaVA resize inputs to fixed dimensions. This suggests preserving original dimensions with learned pruning is potentially more effective than aggressive preprocessing for efficient VLMs. Hardware efficiency. To verify that FLOPs
Kernels
AOK MMB TQA Avg (%)
reductions translate into practical speedups, we measure time-to-first-token (TTFT) and throughput on an NVIDIA H20 GPU. As shown in Table 4, at a 50% compute budget, FastV with VisPCO maintains the same hardware efficiency as FastV alone—comparable TTFT and throughput— while recovering substantial performance lost from pruning. For example, on Qwen2.5VL-3B, VisPCO improves average performance from 63.1% to 71.5% with negligible latency overhead (76 vs. 74 ms TTFT). Similar patterns hold across LLaVAv1.5-7B and Gemma3-4B, confirming that VisPCO preserves the underlying pruning method’s hardware efficiency while substantially improving task performance. 4.4
Analysis of Pruning Patterns
We investigate the performance differences between single-layer and multi-layer pruning strategies, as well as the impact of different kernel choices in multi-layer configurations. Table 5 presents the Pareto-optimal results for different pruning patterns under a 50% computational budget. Figure 3(right) illustrates and compares the Pareto frontiers across different pruning patterns. Strategic pruning pattern selection. As shown in Table 5, multi-layer pruning with the multi-step kernel achieves the best performance under a 50% budget, outperforming both single-layer pruning and other kernel variants (linear, exponential, sig-
moid). Figure 3(right) reveals that this advantage is budget-dependent. When computational budget exceeds 50%, all pruning patterns converge to comparable performance and closely approximate the empirical Pareto frontier, making strategy selection less critical. However, below 50% budget, notable differences emerge among pruning patterns, with the multi-step kernel showing clear superiority. Implications for VLM design. The multi-step kernel’s superior performance at low budgets reveals important architectural insights. Visual token redundancy emerges at specific layers rather than uniformly across the network. Certain layers introduce redundancy through attention or feature transformations, while others preserve essential representations. The multi-step kernel identifies these critical compression points, enabling targeted pruning while retaining key information. These findings provide practical guidance. When resources are sufficient (budget >50%), simple single-layer pruning achieves near-optimal performance. Under tight constraints (budget <50%), multi-step layer-wise pruning is recommended to better exploit VLMs’ hierarchical compression structure.
5
Conclusion
In this paper, we introduced VisPCO, a novel computation budget-aware framework for automatically optimizing visual token pruning configurations in vision-language models. By formulating the problem as Pareto optimization with continuous relaxation, VisPCO enables efficient end-toend gradient-based training to automatically identify optimal pruning configurations for any given computational budget. This approach significantly reduces search costs compared to traditional exhaustive grid search methods. Extensive experiments across 8 visual benchmarks demonstrate that our method generalizes well across various pruning strategies and VLM architectures. Furthermore, our investigation through learnable kernel functions reveals that progressive multi-step pruning consistently outperforms both single-layer and other multi-layer kernel approaches, providing valuable insights for efficient VLM design in resource-constrained deployment scenarios.
Limitations Although our framework demonstrates strong performance across diverse benchmarks and model architectures, several limitations remain to be ad-
dressed in future work. First, our experiments primarily focus on single-image tasks; further validation is needed to assess how effectively our optimized pruning configurations generalize to multiimage and video inputs, which involve more complex temporal and spatial redundancies. Second, our proposed kernel functions provide a structured and interpretable way to model pruning distributions. Future work could explore extending this approach to learn more flexible, non-parametric or input-adaptive patterns, potentially capturing even more nuanced task-specific pruning strategies.
Ethics Statement This work focuses on optimizing visual token pruning configurations for vision-language models to improve computational efficiency. Our method does not involve the collection or use of private or sensitive data; all experiments are conducted on publicly available benchmarks. We do not foresee direct negative societal impacts from this research. By reducing the computational cost of VLMs, our work may contribute to lowering energy consumption and carbon emissions associated with largescale model inference, thereby promoting more sustainable and accessible AI deployment.
Acknowledgments We sincerely thank the students and engineers at the Data Intelligence Research Center, Shanghai Jiao Tong University, for their assistance during the development of this work. This work was supported by NSF China under Grant No.T2421002, 92579104, 62525209, T2542021.
References Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Chunsheng Wu, and 1 others. 2025. Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661. Kenneth J Arrow and Gerard Debreu. 2024. Existence of an equilibrium for a competitive economy. In The Foundations of Price Theory Vol 5, pages 289–316. Routledge. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others.
2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Dimitri P Bertsekas. 2014. Constrained optimization and Lagrange multiplier methods. Academic press. Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and 1 others. 2010. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23rd annual ACM symposium on User interface software and technology, pages 333–342. Jianjian Cao, Peng Ye, Shengze Li, Chong Yu, Yansong Tang, Jiwen Lu, and Tao Chen. 2024. Madtp: Multimodal alignment-guided dynamic token pruning for accelerating vision-language transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15710–15719. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024a. An image is worth 1/2 tokens after layer 2: Plug-andplay inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024b. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971–5984. Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. 2025. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 5334–5342. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, and 1 others. 2024a. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer. Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, ChengLin Liu, Lianwen Jin, and Xiang Bai. 2024b. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and 1 others. 2025. Mme: A comprehensive evaluation benchmark for multimodal large language models. In The 39th Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the association for computational linguistics: ACL 2022, pages 2263– 2279.
Thomas Fuchs. 2000. Das gedächtnis des leibes. Phänomenologische Forschungen, 5(1):71–89.
Jorge Nocedal and Stephen J Wright. 2006. Numerical optimization. Springer.
Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. 2024. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision, pages 390–406. Springer. Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, Yao Hu, and Shaohui Lin. 2024. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.00876. Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162. Springer. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786.
Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and 1 others. 2024. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247. Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. 2025. Streamingvlm: Real-time understanding for infinite video streams. arXiv preprint arXiv:2510.09608. Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, and Bo Yuan. 2025. Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19803–19813. Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. 2025a. Fit and prune: Fast and training-free visual token pruning for multi-modal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 22128–22136. Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. 2025b. Atp-llava: Adaptive token pruning for large vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24972–24982.
A
FLOPs Computation
In this section, we provide a detailed derivation of the floating-point operations (FLOPs) computation for Transformer layers in vision-language models. For a standard Transformer layer, the primary computational costs come from the self-attention mechanism and the feed-forward network (FFN). Given a sequence length N and hidden dimension D, we compute the FLOPs for each component separately. A.1
Self-Attention Mechanism
The self-attention mechanism consists of the following operations: (1) Linear projections: Three projection matrices WQ , WK , WV ∈ RD×D map the input to Query, Key, and Value representations. Each matrix multiplication requires 2N D2 floating-point operations (multiplying input of size N × D with weight of size D × D), thus: FLOPsQKV = 3 × 2N D2 = 6N D2 .
(18)
(2) Attention score: Computing QKT
∈
RN ×N , where Q, K ∈ RN ×D : FLOPsScore = 2N 2 D.
(19)
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. 2024. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417.
(3) Attention Computing √ weighting: T Softmax(QK / D)V, i.e., multiplying RN ×N with RN ×D :
Shiyu Zhao, Zhenting Wang, Felix Juefei-Xu, Xide Xia, Miao Liu, Xiaofang Wang, Mingfu Liang, Ning Zhang, Dimitris N Metaxas, and Licheng Yu. 2025. Accelerating multimodal large language models by searching optimal vision token reduction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29869–29879.
Note: The FLOPs for the Softmax operation are relatively small and typically neglected. (4) Output projection: Projecting back to the original dimension through WO ∈ RD×D :
Yiwu Zhong, Zhuoming Liu, Yin Li, and Liwei Wang. 2025. Aim: Adaptive inference of multi-modal llms via token merging and pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20180–20192.
Therefore, the total FLOPs for the self-attention mechanism is:
Yi Zhou, Hui Zhang, Jiaqian Yu, Yifan Yang, Sangil Jung, Seung-In Park, and ByungIn Yoo. 2024. Himap: Hybrid representation learning for end-toend vectorized hd map construction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15396–15406.
FLOPsWeight = 2N 2 D.
FLOPsOutput = 2N D2 .
FLOPsAttention = 8N D2 + 4N 2 D.
(20)
(21)
(22)
A.1.1 Feed-Forward Network The standard FFN consists of two linear layers with an intermediate dimension Dffn : FFN(x) = W2 · GELU(W1 · x).
(23)
where W1 ∈ RD×Dffn and W2 ∈ RDffn ×D . The total FLOPs for the FFN is: FLOPsFFN = 4N Dffn D.
(24)
In standard Transformer architectures, Dffn = 4D, which gives:
where f, hj : Rn → R are twice continuously differentiable functions. The augmented Lagrangian function for this problem is given by Equation (4). Let x̄ be a local optimal solution of problem (29) that satisfies the second-order sufficient conditions. That is, there exists a Lagrange multiplier vector v̄ = [v̄1 , . . . , v̄l ]T such that:
FLOPslayer = 24N D2 + 4N 2 D.
∇f (x̄) − Av̄ = 0,
A.1.2 Total FLOPs per Layer Combining self-attention and FFN, the total FLOPs for a single Transformer layer is: FLOPslayer = 8N D2 + 4N 2 D + 4N Dffn D. (25)
(26)
We neglect relatively small computational costs such as LayerNorm and residual connections. A.2 Total FLOPs for Vision-Language Models For vision-language models, the input sequence consists of text tokens and visual tokens. Let Nt denote the number of text tokens and Nv denote the initial number of visual tokens. The total number of tokens at layer i is: Ni = Nt + ri Nv
(27)
where ri ∈ [0, 1] represents the retention ratio of visual tokens at layer i. For a Transformer model with L layers, the total computational cost is: F (r) =
L X
24(Nt + ri Nv )D2 + 4(Nt + ri Nv )2 D
i=1
(28)
where: • The first term 24(Nt + ri Nv )D2 corresponds to linear projections in self-attention and the FFN • The second term 4(Nt +ri Nv )2 D corresponds to the quadratic complexity of attention matrix computation • r = [r1 , r2 , . . . , rL ] is the vector of visual token retention ratios across layers This formula indicates that as visual tokens are pruned (ri decreases), the model’s computational cost is significantly reduced, especially the quadratic complexity term.
B
Theoretical Analysis
B.1
Proof of Theorem 1
f (x)
s.t.
hj (x) = 0,
j = 1, . . . , l,
(31)
and for every nonzero vector d satisfying dT ∇hj (x̄) = 0 for j = 1, . . . , l, we have: dT ∇2x L(x̄, v̄)d > 0,
(32)
A = [∇h1 (x̄), . . . , ∇hl (x̄)],
(33)
where
and L(x, v) = f (x) − vT h(x) is the standard Lagrangian function. By assumption, x̄ is a Karush-Kuhn-Tucker (KKT) point of problem (29), thus: ∇x ϕ(x̄, v̄, λ) = 0.
(34)
We now prove that the Hessian matrix ∇2x ϕ(x̄, v̄, λ) is positive definite at x̄ for sufficiently large λ. From Equation (4), we can derive: ∇2x ϕ(x, v̄, λ) = ∇2 f (x) −
l X
v̄j ∇2 hj (x)
j=1
+σ
l X
hj (x)∇2 hj (x) + λ
j=1
l X
∇hj (x)∇hj (x)T
(35)
j=1
= ∇2 f (x) −
l X (v̄j − λhj (x))∇2 hj (x) j=1
+λ
l X
∇hj (x)∇hj (x)T = Q + λAAT ,
j=1
where 2
Q = ∇ f (x) −
l X
(v̄j − λhj (x))∇2 hj (x), (36)
j=1
A = [∇h1 (x), . . . , ∇hl (x)].
Consider the equality-constrained optimization problem: min
hj (x̄) = 0,
(30)
j = 1, . . . , l,
(29)
(37)
At the point x̄, we have: ∇2x ϕ(x̄, v̄, λ) = Q̄ + λĀĀT , where Q̄ and Ā denote the evaluations at x̄.
(38)
Let rank(Ā) = r ≤ l, and let B ∈ Rn×r be an orthonormal basis matrix for Ā (i.e., BT B = Ir ), meaning the r columns of B form an orthonormal basis for the subspace spanned by the l columns of Ā. Thus, we have: Ā = BC,
(39)
where C = BT Ā has rank r. For any nonzero vector u ∈ Rn , we decompose it as: u = p + Bq, (40) where p satisfies BT p = 0. Clearly, ĀT p = 0, which implies: ∇hj (x̄)T p = 0,
j = 1, . . . , l.
(41)
Now, we can write uT ∇2x ϕ(x̄, v̄, λ)u as: uT ∇2x ϕ(x̄, v̄, λ)u = (p + Bq)T (Q̄ + λĀĀT )(p + Bq) T
T
T
(42)
T
= p Q̄p + 2p Q̄Bq + q B Q̄Bq + λqT CCT q.
B.2
Elimination of y via Quadratic Completion
Starting from the augmented Lagrangian function in Equation (7), we apply the technique of completing the square to eliminate the auxiliary variable y. Let g(r) = B − F (r) denote the constraint function. We can rewrite Equation (7) as: ϕ̃(r, y, w, λ) λ (49) = Ldistill (r) − w(g(r) − y 2 ) + (g(r) − y 2 )2 2 λ = Ldistill (r) + −w(g(r) − y 2 ) + (g(r) − y 2 )2 . 2
Completing the square with respect to y 2 , we have: λ − w(g(r) − y 2 ) + (g(r) − y 2 )2 2 h w i2 w 2 λ − (g(r) − y 2 ) − = 2 λ 2λ h i 2 λ 2 w w2 = y − g(r) − − . (50) 2 λ 2λ
Since x̄ is a local optimal solution of problem (29) satisfying the second-order sufficient conditions, there exists a constant α > 0 such that:
To minimize ϕ̃ with respect to y, we analyze the optimal value of y 2 . The term 2 λ w 2 is minimized when: 2 y − g(r) − λ
pT Q̄p ≥ α∥p∥2 .
w 1 = (λg(r) − w). (51) λ λ However, since y ∈ R, we must have y 2 ≥ 0. Therefore, the optimal value is: 1 2 (52) y = max 0, (λg(r) − w) . λ
(43)
Let b be the largest singular value of Q̄B, let e = ∥BT Q̄B∥2 , and let µ > 0 be the smallest eigenvalue of CCT . Then: uT ∇2x ϕ(x̄, v̄, λ)u ≥ α∥p∥2 − 2b∥p∥∥q∥ + (λµ − e)∥q∥2 . (44)
Since u ̸= 0, the vectors p and q cannot both be zero. Therefore, if we choose λ sufficiently large such that: b2 λµ − e − > 0, (45) α that is, b2 + αe λ> , (46) αµ then we always have: uT ∇2x ϕ(x̄, v̄, λ)u > 0.
(47)
b2 + αe . λ = αµ
This can be expressed equivalently as: ( 1 (λg(r) − w), if λg(r) − w ≥ 0, y2 = λ 0, if λg(r) − w < 0. (53) 2 Substituting the optimal y back into Equation (50), we obtain: λ 1 −w(g(r)−y 2 )+ (g(r)−y 2 )2 = z 2 − w2 , 2 2λ (54) where z = max {0, w − λ(B − F (r))}
Therefore, there exists: ′
y 2 = g(r) −
= max {0, w − λg(r)} . (48)
When the penalty parameter λ > λ′ , the matrix ∇2x ϕ(x̄, v̄, λ) is positive definite. Combined with Equations (34) and (47), we conclude that x̄ is a strict local minimizer of ϕ(x, v̄, λ). This completes the proof.
(55)
Therefore, the simplified augmented Lagrangian function, after eliminating y, is: 1 ϕ(r, w, λ) = Ldistill (r) + z 2 − w2 , (56) 2λ where z = max{0, w − λ(B − F (r))} and g(r) = B − F (r) represents the constraint satisfaction.
C
Experiment Details
C.1
Experiment Settings
C.1.1
Training Dataset
We find that directly training on the original dataset leads to suboptimal performance: VisPCO’s predicted Pareto frontier for high-resolution images concentrates on low computational budgets, diverging from the empirical frontier under high budget regimes. We analyze the training dataset and observe that the distribution of image areas exhibits significant skewness, heavily concentrated on smaller areas, as shown in the left panel of Figure 4. This imbalance is detrimental to learning appropriate pruning ratios, as it leads to poor generalization on high-resolution images. To address this issue, we preprocess the training images using histogram equalization to balance the area distribution. Specifically, we divide the image area range into uniform bins and apply stratified sampling to ensure balanced representation across all area intervals. For each bin, we either oversample images (for underrepresented bins) or subsample images (for overrepresented bins) to achieve approximately equal counts per bin. This rebalancing procedure ensures that the training distribution covers the full spectrum of image resolutions uniformly, enabling VisPCO to learn robust pruning configurations for both low and high-resolution images. The left panel of Figure 4 shows the original skewed distribution, while the right panel illustrates the balanced distribution after equalization. C.1.2
Evaluation Datasets
We utilize the evaluation datasets provided by VLMEvalKit, which includes curated questionanswer pairs and images from various visionlanguage benchmarks. Our evaluation spans three categories of tasks: visual question answering, multimodal reasoning, and chart understanding. For MME, to maintain comparability with other benchmarks, we report the ratio of correct answers to total questions, normalizing the final evaluation results to the range [0, 1]. An example evaluation case is shown in Figure 5. A-OKVQA (Schwenk et al., 2022) is a knowledge-based visual question answering dataset that requires models to leverage external commonsense and world knowledge beyond visual content. It contains 1,145 questions across diverse image types, challenging models to perform reasoning that combines visual understanding with factual
knowledge. VizWiz (Bigham et al., 2010) is a visual question answering dataset collected from blind users who took images and asked questions about them. The dataset contains over 4,319 image-question pairs with natural, real-world scenarios, often featuring challenging conditions such as poor image quality, blur, or unusual viewpoints, making it particularly valuable for evaluating model robustness. SEEDBench (Li et al., 2023) is a comprehensive benchmark for evaluating multimodal large language models across multiple dimensions. It includes 14,232 multiple-choice questions spanning nine evaluation dimensions including scene understanding, instance identity, spatial relation, and visual reasoning, providing a holistic assessment of model capabilities. MMBench (Liu et al., 2024a) (Multimodal Benchmark) is a systematically designed objective benchmark for evaluating various abilities of vision-language models. It covers 20 ability dimensions organized into three categories: perception (e.g., object localization, OCR), reasoning (e.g., social reasoning, physical commonsense), and knowledge (e.g., celebrity recognition, landmark identification). MME (Fu et al., 2025) (Multi-Modal Evaluation) is a comprehensive evaluation benchmark measuring both perception and cognition abilities. It consists of 14 subtasks including existence, count, position, color, posters, celebrity, scene, landmark, artwork, OCR, commonsense reasoning, numerical calculation, text translation, and code reasoning. We normalize scores to [0, 1] for consistency with other benchmarks. ChartQA (Masry et al., 2022) focuses on question answering about statistical charts and plots. The dataset contains over 2,000 human-written questions covering bar charts, line plots, and pie charts, requiring models to perform visual reasoning, data extraction, and numerical computation from chart images. OCRBench (Liu et al., 2024b) is a comprehensive benchmark for evaluating optical character recognition and text understanding capabilities in vision-language models. It includes diverse text recognition scenarios such as scene text, handwritten text, document text, and multilingual text, assessing both basic OCR accuracy and text-based reasoning abilities. TextVQA (Singh et al., 2019) requires models to read and reason about text in images to answer
Figure 4: Distribution histogram of image areas in the training dataset before and after applying histogram equalization to balance area diversity. The left panel shows the original distribution heavily concentrated on smaller image areas, while the right panel demonstrates the more balanced distribution after the equalization process.
User: "<image> What is the laptop on? Given the following options, choose the correct answer: A. counter
B. bed
C. island
D. table
Only give the correct choice:" Assistant: "The answer is D. table"
Figure 5: An example evaluation case from the VLMEvalKit benchmark. The figure demonstrates a typical question-answer pair with the corresponding image, showing how the model processes visual and textual inputs to generate responses for evaluation.
questions. The dataset contains 1,000 images from OpenImages, where answering questions necessitates reading and understanding scene text, making it essential for evaluating text-aware visual reasoning capabilities. C.1.3
Pruning Configuration Sampling
To identify the empirical Pareto frontier that serves as the ground truth for evaluating VisPCO, we employ a comprehensive sampling-based approach. Our methodology consists of three steps: (1) systematically sampling a large number of pruning configurations across the search space, (2) eval-
uating each configuration’s performance across multiple benchmarks and measuring its computational cost in FLOPs, and (3) extracting the Paretooptimal configurations from the evaluated results. Sampling Strategy. Our sampling strategy operates at the layer level to capture fine-grained pruning patterns. For a vision-language model with L Transformer layers, we independently sample the visual token retention ratio for each layer from layer 1 to layer L. Specifically, the retention ratio ri for layer i is sampled from the discrete set {0.01, 0.06, 0.11, . . . , 0.96, 0.99}, with a uniform step size of 0.05. This granularity balances comprehensive coverage of the configuration space with computational feasibility. For the Qwen2.5-VL-3B model with L=36 layers, this sampling scheme generates a total of 700 distinct pruning configurations spanning diverse computational budgets. Evaluation Protocol. For each sampled configuration, we perform a complete evaluation to obtain both its performance and computational cost. Performance is measured by averaging accuracy across our eight evaluation benchmarks, providing a comprehensive assessment of model capabilities. Computational cost is calculated using the FLOPs formula derived in Appendix A, accounting for both the attention mechanism and feed-forward network operations across all layers. Pareto Frontier Extraction. Given the set of evaluated configurations C = {(pi , fi )}N i=1 , where pi ∈ [0, 1] represents the normalized average performance (higher is better) and fi represents the computational cost in TFLOPs (lower is better) for configuration i, we identify the Pareto frontier us-
Table 6: Hyperparameters for main experiments comparing different methods across multiple VLMs. Model + Method
λ
α
ϵ
β
σ
T
lr
B
Qwen2.5-VL-3B + FastV Qwen2.5-VL-3B + SparseVLM Qwen2.5-VL-3B + FitPrune
100 100 100
5 5 5
0.005 0.005 0.005
0.5 0.5 0.5
10 10 10
0.1 0.1 0.1
1e-4 1e-4 1e-4
16 16 16
Gemma3-4B + FastV Gemma3-4B + SparseVLM Gemma3-4B + FitPrune
1 1 1
5 5 5
0.005 0.005 0.005
0.5 0.5 0.5
10 10 10
0.1 0.1 0.1
5e-4 5e-4 5e-4
16 16 16
LLaVA-v1.5-7B + FastV LLaVA-v1.5-7B + SparseVLM LLaVA-v1.5-7B + FitPrune
100 100 100
10 10 10
0.01 0.01 0.01
0.5 0.5 0.5
10 10 10
0.1 0.1 0.1
5e-5 5e-5 5e-5
16 16 16
Table 7: Hyperparameters for ablation studies on different pruning scheduling strategies. Model + Strategy
λ
α
ϵ
β
σ
T
lr
B
Qwen2.5-VL-3B + Linear Qwen2.5-VL-3B + Exponential Qwen2.5-VL-3B + P-sigmoid Qwen2.5-VL-3B + Multi-step
100 100 100 1
5 10 5 10
0.01 0.005 0.01 0.05
0.5 0.5 0.5 0.5
1 1 1 5
0.1 0.1 0.1 0.1
1e-4 5e-5 1e-4 1e-4
16 16 16 16
ing the Pareto dominance criterion. Formally, a configuration (pi , fi ) is said to dominate another configuration (pj , fj ) if and only if: pi ≥ pj
and
fi ≤ fj .
(57)
The Pareto frontier P consists of all nondominated configurations: P = {(pi , fi ) ∈ C | ∄(pj , fj ) ∈ C such that (pj , fj ) dominates (pi , fi )}.
Figure 6: Comparison of layer-wise pruning ratios for different kernel functions under 50% computational budget. Linear kernel produces gradual transitions across layers, Exponential concentrates pruning in later layers, P-Sigmoid creates smooth S-shaped curves, and Multi-Step generates progressive discrete transitions. The diversity of patterns enables comprehensive exploration of different pruning strategies.
(58)
These Pareto-optimal configurations represent the best achievable trade-offs between performance and computational efficiency, forming the empirical frontier against which we evaluate VisPCO’s predictions. This extensive sampling and evaluation process requires significant computational resources (approximately 48+ GPU hours for 700 configurations), highlighting the practical necessity of efficient optimization methods like VisPCO. C.1.4 Hyperparameter Settings We provide detailed hyperparameter configurations for our experiments in Tables 6 and 7. The key hyperparameters and their roles are as follows: λ denotes the penalty parameter in the augmented Lagrangian method, controlling the strength of constraint enforcement. ϵ is the convergence threshold that determines when the optimization terminates. α and β are the update coefficients for the Lagrangian multiplier and penalty parameter, respectively, governing the convergence dynamics. σ controls the Gaussian kernel width for continuous relaxation of discrete pruning decisions,
with larger values leading to smoother approximations. T is the temperature parameter for the straight-through estimator, balancing between gradient flow and discretization sharpness during training. lr denotes the learning rate for the AdamW optimizer, and B indicates the batch size (number of samples per training iteration). C.2
More Experiment Results
We provide more detailed experimental results in this section. First, we present the results of different pruning methods under various computational budgets with VisPCO in Table 8. Second, we show the results of applying VisPCO to different base VLMs in Table 9. Third, we report the results of VisPCO with different pruning patterns in Table 10. C.3
Case Studies of Predicted Pruning Configurations
We present case studies of pruning configurations predicted by VisPCO on Qwen2.5-VL-3B to provide insights into its behavior under different computational budgets. Figure 7 illustrates the layerwise pruning curves predicted by VisPCO under various budget constraints, along with the corresponding visual token retention patterns at different layers. These visualizations reveal how VisPCO adaptively adjusts its pruning strategy in response to varying resource constraints.
The visualization reveals several key observations. First, as the computational budget becomes more constrained, VisPCO adopts increasingly aggressive pruning strategies, with pruning occurring earlier in the network and achieving lower retention ratios. This demonstrates the model’s ability to adaptively allocate computational resources based on budget constraints. Second, the predicted configurations exhibit smooth transitions across layers, validating the effectiveness of our continuous relaxation approach. Additionally, Figure 6 presents a comparison of different kernel functions (Linear, Exponential, PSigmoid, Multi-Step) for multi-layer pruning under a 50% computational budget. The layer-wise pruning ratios reveal distinct patterns: linear kernels produce gradual transitions, exponential kernels concentrate pruning in later layers, p-sigmoid kernels create smooth S-shaped curves, and multi-step kernels generate progressive discrete transitions. These diverse patterns enable VisPCO to explore different trade-offs.
Figure 7: Layer-wise pruning configurations predicted by VisPCO under different computational budgets. The left panel shows the retention ratio curves across layers for budgets ranging from 10% to 90%. The right panel visualizes the actual visual token retention at selected layers (from top to bottom: 90%, 50%, 10% budgets), demonstrating how aggressive pruning (lower budgets) leads to earlier and more extensive token removal.
Table 8: Detailed comparison of pruning methods with and without VisPCO across eight vision-language benchmarks under different computational budgets. Results without VisPCO are averaged over multiple randomly sampled configurations that satisfy the budget constraint, with standard deviations reported in parentheses. Method
AOKVQA
VizWiz
Qwen2.5VL-3B
90.2
75.1
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
88.2 ± 0.4 88.4 88.5 ± 0.3 88.6 89.1 ± 0.5 89.6
72.9 ± 0.9 73.8 73.1 ± 0.5 73.5 73.9 ± 0.4 74.1
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
87.0 ± 1.6 88.3 87.1 ± 1.8 88.4 87.3 ± 2.2 89.3
71.6 ± 2.1 73.7 71.7 ± 2.4 73.9 72.3 ± 2.1 73.8
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
83.9 ± 4.5 88.0 84.1 ± 4.7 88.1 84.2 ± 4.6 88.2
69.2 ± 4.4 73.4 69.4 ± 4.6 73.6 69.5 ± 4.7 73.7
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
80.2 ± 5.9 86.0 80.3 ± 6.2 86.2 80.4 ± 6.1 86.3
68.4 ± 5.4 71.4 68.5 ± 5.5 71.6 68.6 ± 5.4 71.7
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
74.7 ± 10.1 84.8 75.9 ± 9.8 85.2 77.1 ± 8.7 85.9
60.3 ± 9.6 69.4 62.6 ± 8.2 69.0 63.4 ± 7.7 69.4
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
65.6 ± 12.2 77.6 66.8 ± 12.6 77.9 66.9 ± 12.5 78.4
50.3 ± 11.4 61.7 50.9 ± 11.8 61.9 50.7 ± 12.2 62.2
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
62.5 ± 8.1 70.2 62.6 ± 8.3 70.5 62.7 ± 8.2 70.6
46.4 ± 8.2 54.6 46.5 ± 8.3 54.8 46.7 ± 8.2 54.9
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
42.5 ± 4.1 46.6 42.6 ± 4.2 46.7 42.5 ± 4.3 46.8
39.4 ± 4.2 43.1 39.5 ± 4.3 43.2 39.4 ± 4.4 43.3
⌞ FastV + VisPCO ⌞ SparseVLM + VisPCO ⌞ FitPrune + VisPCO
33.3 ± 2.3 35.5 33.6 ± 2.1 35.5 33.8 ± 2.1 35.6
30.4 ± 1.6 31.7 31.2 ± 1.3 31.5 31.5 ± 1.1 31.6
SEED MMB MME† ChartQA Upper Bound, 100% Budget, ∼3.56 TFLOPs 75.6 79.8 84.2 64.1 Reduce FLOPs Budget to 90%, ∼3.20 TFLOPs 72.4 ± 0.9 76.4 ± 0.5 81.3 ± 0.5 62.2 ± 0.8 73.2 76.9 81.7 62.9 73.4 ± 0.4 76.9 ± 0.6 82.1 ± 0.3 62.2 ± 0.7 73.8 77.5 82.4 62.9 74.2 ± 0.5 77.6 ± 0.4 82.5 ± 0.6 63.1 ± 0.6 74.6 77.9 82.8 63.5 Reduce FLOPs Budget to 80%, ∼2.84 TFLOPs 71.2 ± 2.1 75.1 ± 1.8 80.1 ± 1.6 61.0 ± 2.2 73.1 76.9 81.6 62.8 71.4 ± 2.1 75.3 ± 2.1 80.2 ± 2.2 61.6 ± 2.2 73.5 77.4 82.1 62.6 72.5 ± 2.3 75.8 ± 2.2 80.6 ± 2.2 61.6 ± 2.5 74.3 77.4 82.6 63.3 Reduce FLOPs Budget to 70%, ∼2.50 TFLOPs 68.4 ± 4.8 73.1 ± 3.8 76.2 ± 4.5 59.3 ± 4.3 72.8 76.7 80.9 62.4 68.5 ± 5.1 73.3 ± 4.0 76.3 ± 4.5 59.5 ± 4.5 72.9 76.7 81.0 62.5 68.6 ± 5.2 73.5 ± 4.1 76.6 ± 4.5 59.4 ± 4.7 73.1 76.7 81.1 62.8 Reduce FLOPs Budget to 60%, ∼2.14 TFLOPs 66.6 ± 6.8 69.4 ± 5.3 74.5 ± 5.5 56.4 ± 5.3 70.8 74.7 78.9 60.4 66.7 ± 7.1 69.6 ± 5.6 74.8 ± 5.8 56.4 ± 5.6 70.9 74.9 79.2 60.6 66.8 ± 7.0 69.8 ± 5.8 74.9 ± 5.9 56.7 ± 5.9 71.0 75.1 79.4 60.9 Reduce FLOPs Budget to 50%, ∼1.78 TFLOPs 61.5 ± 8.1 62.4 ± 9.3 68.8 ± 9.1 51.6 ± 9.9 67.6 71.2 77.1 58.1 63.1 ± 7.2 63.9 ± 8.6 69.9 ± 8.2 51.9 ± 8.3 68.1 71.9 77.6 58.4 63.9 ± 6.8 64.5 ± 8.2 70.8 ± 7.9 52.8 ± 8.1 68.4 72.4 77.9 58.8 Reduce FLOPs Budget to 40%, ∼1.42 TFLOPs 52.6 ± 10.2 53.3 ± 11.3 61.4 ± 11.2 42.5 ± 11.7 62.6 64.4 72.6 54.1 53.2 ± 10.9 53.5 ± 11.8 62.3 ± 11.3 42.9 ± 11.9 62.7 64.9 72.9 54.4 53.0 ± 11.1 53.7 ± 11.9 62.7 ± 11.5 42.8 ± 12.2 62.8 65.0 73.1 54.6 Reduce FLOPs Budget to 30%, ∼1.06 TFLOPs 48.7 ± 7.6 49.4 ± 7.5 57.2 ± 8.9 38.8 ± 8.1 55.5 54.5 65.7 46.4 48.9 ± 7.7 49.5 ± 7.7 57.4 ± 9.0 38.9 ± 8.3 55.9 54.8 65.8 46.9 49.0 ± 7.6 49.7 ± 7.9 57.3 ± 9.1 39.1 ± 8.1 56.0 54.7 65.8 47.1 Reduce FLOPs Budget to 20%, ∼0.72 TFLOPs 46.7 ± 3.6 39.4 ± 4.5 47.2 ± 4.9 34.8 ± 4.1 50.1 43.9 51.2 38.9 46.9 ± 3.7 39.6 ± 4.6 47.4 ± 5.0 34.9 ± 4.2 50.2 43.9 51.3 39.1 46.7 ± 3.9 39.7 ± 4.7 47.6 ± 5.1 34.7 ± 4.1 50.4 44.1 51.4 39.2 Reduce FLOPs Budget to 10%, ∼0.36 TFLOPs 44.5 ± 2.7 33.0 ± 2.5 39.7 ± 1.4 29.8 ± 4.1 46.9 35.5 40.1 33.2 44.9 ± 2.5 33.9 ± 2.3 40.3 ± 1.1 30.5 ± 3.7 47.1 35.8 40.4 33.3 45.3 ± 2.4 34.2 ± 2.2 40.6 ± 1.0 30.9 ± 3.5 47.3 35.8 40.9 33.5
OCRB
TextVQA
Avg (%)
74.6
81.3
78.1
71.6 ± 0.7 72.3 71.9 ± 0.6 72.5 72.5 ± 0.5 72.9
79.1 ± 0.6 79.5 79.5 ± 0.6 80.0 79.9 ± 0.3 81.2
75.5 ± 0.7 76.1 76.0 ± 0.5 76.4 76.2 ± 0.5 77.1
70.2 ± 1.9 72.0 70.5 ± 2.1 72.4 70.6 ± 2.5 72.5
77.7 ± 2.2 79.4 78.3 ± 2.0 79.6 78.4 ± 1.6 79.9
74.2 ± 1.9 75.7 74.5 ± 2.1 76.2 74.9 ± 2.2 76.6
66.2 ± 4.9 71.1 66.3 ± 5.0 71.1 66.5 ± 5.2 71.0
75.5 ± 4.1 78.9 75.7 ± 4.2 79.1 75.8 ± 4.3 79.3
71.5 ± 4.4 75.5 71.6 ± 4.6 75.6 71.8 ± 4.7 75.7
65.2 ± 5.1 69.1 65.7 ± 5.5 69.3 65.9 ± 5.7 69.6
71.7 ± 5.1 76.9 71.8 ± 5.3 77.1 71.9 ± 5.3 77.1
69.2 ± 5.4 73.5 69.4 ± 5.7 73.7 69.5 ± 5.8 73.9
59.2 ± 9.1 67.8 62.4 ± 6.9 67.9 63.3 ± 6.4 68.2
65.9 ± 10.8 75.9 66.6 ± 9.8 76.3 67.6 ± 9.4 76.6
63.1 ± 9.5 71.5 64.5 ± 8.4 71.8 65.4 ± 7.9 72.2
51.4 ± 11.2 62.5 51.8 ± 11.3 62.6 51.7 ± 11.5 62.6
57.5 ± 12.7 69.9 58.1 ± 12.9 70.2 58.3 ± 12.6 70.4
54.3 ± 11.5 65.7 54.9 ± 11.8 65.9 55.0 ± 11.9 66.1
46.7 ± 8.3 54.4 46.9 ± 8.5 55.1 46.7 ± 8.6 55.3
53.6 ± 9.1 62.8 53.8 ± 9.3 63.4 53.9 ± 9.4 63.1
50.4 ± 8.2 58.0 50.6 ± 8.4 58.4 50.6 ± 8.4 58.4
12.7 ± 5.3 17.8 12.8 ± 5.4 17.9 12.6 ± 5.2 18.0
43.6 ± 5.1 48.5 43.7 ± 5.3 48.6 43.8 ± 5.4 48.7
38.3 ± 4.5 42.5 38.4 ± 4.6 42.6 38.3 ± 4.6 42.7
8.3 ± 2.1 10.1 9.1 ± 2.0 10.2 9.6 ± 1.9 10.4
33.7 ± 2.8 36.1 34.4 ± 2.2 36.3 34.6 ± 2.1 36.4
31.6 ± 2.4 33.6 32.2 ± 2.2 33.8 32.6 ± 2.0 33.9
Table 9: Performance of VisPCO applied to different base vision-language models across eight benchmarks under various computational budgets. The results demonstrate the generalizability and effectiveness of VisPCO across different model architectures and sizes. Method
AOKVQA
VizWiz
LLaVA-7B Gemma3-4B
72.3 80.1
93.1 61.2
LLaVA-7B + VisPCO Gemma3-4B + VisPCO
71.3 ± 0.5 71.8 78.8 ± 0.5 79.3
91.5 ± 0.8 92.3 59.8 ± 0.9 60.5
LLaVA-7B + VisPCO Gemma3-4B + VisPCO
70.1 ± 1.6 71.7 76.6 ± 1.4 77.8
89.7 ± 1.7 91.4 57.9 ± 1.7 59.5
LLaVA-7B + VisPCO Gemma3-4B + VisPCO
66.2 ± 4.7 70.7 70.2 ± 5.1 75.3
85.7 ± 4.5 89.3 51.7 ± 5.4 56.9
LLaVA-7B + VisPCO Gemma3-4B + VisPCO
62.1 ± 6.6 68.1 54.1 ± 6.1 60.2
82.9 ± 6.2 88.3 46.6 ± 6.3 52.9
LLaVA-7B + VisPCO Gemma3-4B + VisPCO
50.3 ± 10.7 60.8 41.2 ± 12.4 53.4
70.8 ± 10.3 80.5 33.5 ± 12.6 45.9
LLaVA-7B + VisPCO Gemma3-4B + VisPCO
48.4 ± 8.6 56.8 39.4 ± 8.1 47.5
68.5 ± 8.4 76.9 31.1 ± 9.4 40.1
LLaVA-7B + VisPCO Gemma3-4B + VisPCO
41.3 ± 6.7 47.8 31.4 ± 6.1 37.5
61.6 ± 6.1 67.7 24.1 ± 6.4 30.1
LLaVA-7B + VisPCO Gemma3-4B + VisPCO
40.8 ± 2.8 43.6 31.2 ± 2.2 33.4
60.9 ± 2.2 63.1 23.8 ± 2.4 26.2
LLaVA-7B + VisPCO Gemma3-4B + VisPCO
40.1 ± 0.7 40.7 30.1 ± 0.3 30.4
60.8 ± 0.8 61.5 22.8 ± 0.5 23.2
MMB MME† ChartQA Upper Bound, 100% Budget 52.1 48.2 50.4 42.3 69.9 72.3 79.3 53.8 Reduce FLOPs Budget to 90% 50.3 ± 0.6 47.3 ± 0.6 48.4 ± 0.8 40.9 ± 0.7 50.8 47.7 49.2 41.5 67.7 ± 0.7 70.5 ± 0.6 77.6 ± 0.9 51.8 ± 0.8 68.4 71.1 78.5 52.5 Reduce FLOPs Budget to 80% 48.7 ± 1.3 45.4 ± 1.3 46.2 ± 1.7 38.5 ± 1.6 49.9 46.7 47.6 40.0 65.6 ± 1.6 68.4 ± 1.6 75.3 ± 1.8 49.8 ± 1.8 67.2 69.9 77.0 51.5 Reduce FLOPs Budget to 70% 44.6 ± 4.3 41.3 ± 4.4 42.3 ± 4.6 34.6 ± 4.7 48.9 45.7 46.6 39.1 59.5 ± 5.5 62.3 ± 5.2 69.1 ± 4.7 43.7 ± 5.4 65.1 67.5 73.5 58.9 Reduce FLOPs Budget to 60% 41.7 ± 6.4 38.4 ± 5.6 38.1 ± 7.0 31.6 ± 6.5 47.5 43.6 44.1 37.0 54.4 ± 6.6 57.2 ± 7.3 65.1 ± 6.8 38.6 ± 6.2 60.9 64.1 71.1 44.3 Reduce FLOPs Budget to 50% 30.8 ± 11.3 26.5 ± 11.5 26.1 ± 11.4 21.3 ± 10.6 42.1 38.0 37.4 31.6 41.8 ± 11.5 44.7 ± 12.9 52.3 ± 11.8 25.7 ± 12.3 53.1 57.3 62.8 37.9 Reduce FLOPs Budget to 40% 29.7 ± 8.8 24.1 ± 9.4 24.2 ± 9.8 19.5 ± 8.6 38.4 33.5 34.0 27.9 39.8 ± 9.7 42.8 ± 8.9 50.4 ± 9.5 23.6 ± 8.4 48.7 51.5 59.9 31.4 Reduce FLOPs Budget to 30% 22.8 ± 6.5 17.2 ± 6.5 17.4 ± 6.9 14.6 ± 6.7 29.2 23.6 24.2 20.9 31.8 ± 6.7 36.8 ± 6.9 44.4 ± 6.5 20.6 ± 6.4 38.5 43.5 50.2 27.0 Reduce FLOPs Budget to 20% 21.7 ± 2.6 17.1 ± 2.9 16.9 ± 3.1 13.8 ± 2.5 24.3 19.9 19.8 16.3 31.9 ± 2.2 36.4 ± 2.5 44.6 ± 2.6 20.3 ± 2.6 34.0 38.9 47.2 22.9 Reduce FLOPs Budget to 10% 21.4 ± 0.6 17.2 ± 0.4 16.7 ± 0.2 13.5 ± 0.3 22.0 17.6 16.8 13.8 27.4 ± 0.6 33.3 ± 0.8 41.5 ± 0.7 19.2 ± 0.6 27.9 34.1 42.2 19.7 SEED
OCRB
TextVQA
Avg (%)
64.3 64.2
88.2 70.3
63.9 68.9
62.8 ± 0.7 63.5 62.3 ± 0.7 62.9
86.2 ± 0.5 86.5 58.8 ± 0.5 59.2
62.3 ± 0.7 62.9 65.9 ± 0.7 66.6
60.6 ± 1.7 62.3 60.2 ± 1.6 61.8
84.1 ± 1.6 85.6 56.6 ± 1.5 57.9
60.2 ± 1.9 61.9 63.8 ± 1.6 65.3
56.3 ± 4.5 60.7 54.3 ± 5.4 59.7
80.3 ± 4.1 84.4 50.6 ± 6.2 56.7
56.4 ± 4.5 60.7 57.7 ± 5.4 64.2
53.5 ± 6.1 59.6 49.4 ± 6.5 54.9
76.1 ± 6.3 81.3 45.6 ± 7.2 51.3
53.1 ± 6.3 58.7 53.1 ± 6.6 57.5
41.2 ± 11.6 52.8 37.6 ± 12.7 50.3
64.2 ± 11.4 75.2 33.6 ± 12.3 45.5
41.4 ± 11.1 52.3 38.8 ± 12.3 50.8
39.3 ± 9.4 48.7 35.8 ± 8.9 44.2
62.7 ± 9.9 71.7 31.7 ± 8.3 39.9
39.6 ± 9.1 48.5 36.8 ± 8.9 45.4
36.4 ± 6.3 42.5 31.8 ± 6.9 38.3
60.8 ± 8.2 68.9 25.7 ± 6.3 31.9
34.0 ± 6.7 40.6 30.8 ± 6.5 37.1
34.6 ± 2.5 37.1 31.7 ± 2.3 34.0
60.1 ± 2.3 62.4 25.6 ± 2.3 27.9
37.2 ± 2.6 39.8 30.7 ± 2.4 33.1
34.3 ± 0.9 35.0 30.6 ± 0.6 31.2
60.2 ± 0.5 60.6 23.5 ± 0.4 23.9
36.8 ± 0.6 33.5 28.6 ± 0.6 29.1
Table 10: Performance comparison of VisPCO with different pruning patterns across vision-language benchmarks. The table shows detailed results for single-layer pruning and various multi-layer pruning strategies including Linear, Exponential, P-Sigmoid, and Multi-Step patterns. Method
Kernels
Qwen2.5VL-3B
-
Single-Layer
Linear Exponential P-Sigmoid Multi-Step
Multi-Layer
Single-Layer Multi-Layer
Single-Layer Multi-Layer
Single-Layer Multi-Layer
Single-Layer Multi-Layer
Single-Layer Multi-Layer
Single-Layer Multi-Layer
Single-Layer Multi-Layer
Single-Layer Multi-Layer
Linear Exponential P-Sigmoid Multi-Step Linear Exponential P-Sigmoid Multi-Step Linear Exponential P-Sigmoid Multi-Step Linear Exponential P-Sigmoid Multi-Step Linear Exponential P-Sigmoid Multi-Step Linear Exponential P-Sigmoid Multi-Step Linear Exponential P-Sigmoid Multi-Step Linear Exponential P-Sigmoid Multi-Step
AOKVQA VizWiz SEED MMB MME† ChartQA Upper Bound, 100% Budget, ∼3.56 TFLOPs 90.2 75.1 75.6 79.8 84.2 64.1 Reduce FLOPs Budget to 90%, ∼3.20 TFLOPs 88.4 73.8 73.2 76.9 81.7 62.9 88.2 73.1 72.5 76.4 81.3 61.8 87.9 72.8 72.3 76.4 81.3 61.7 87.5 72.3 72.1 76.1 81.0 61.3 88.5 73.9 73.3 76.9 81.9 62.9 Reduce FLOPs Budget to 80%, ∼2.84 TFLOPs 88.3 73.7 73.1 76.9 81.6 62.8 87.7 73.2 72.6 76.3 81.4 62.4 87.4 72.7 72.2 75.9 81.0 62.2 87.1 72.3 71.8 74.9 80.2 61.3 88.5 73.9 73.4 77.2 81.9 63.2 Reduce FLOPs Budget to 70%, ∼2.50 TFLOPs 88.0 73.4 72.8 76.7 80.9 62.4 87.4 72.8 72.2 76.3 80.5 62.1 87.2 72.5 72.0 76.1 80.4 61.8 86.8 72.3 71.8 75.7 80.1 61.6 87.7 72.9 72.5 76.6 80.8 62.0 Reduce FLOPs Budget to 60%, ∼2.14 TFLOPs 86.0 71.4 70.8 74.7 78.9 60.4 84.5 69.7 68.9 72.7 77.1 58.1 83.2 68.5 67.7 71.4 76.3 56.8 82.9 68.1 66.5 70.1 75.3 56.2 86.3 71.7 71.1 74.9 79.2 60.6 Reduce FLOPs Budget to 50%, ∼1.78 TFLOPs 84.8 69.4 67.6 71.2 77.1 58.1 82.6 67.7 65.4 70.9 76.2 57.5 82.2 67.3 65.1 70.4 75.3 57.1 81.9 67.0 64.8 69.6 74.7 56.8 84.9 69.5 68.6 71.8 77.9 59.2 Reduce FLOPs Budget to 40%, ∼1.42 TFLOPs 77.6 61.7 62.6 64.4 72.6 54.1 76.2 59.4 60.3 62.6 70.3 52.7 76.1 59.2 59.6 62.3 69.6 52.3 73.3 56.3 56.7 60.5 66.9 49.1 77.8 62.1 62.9 64.8 73.3 55.3 Reduce FLOPs Budget to 30%, ∼1.06 TFLOPs 70.2 54.6 55.5 54.5 65.7 46.4 65.5 49.6 50.8 49.3 61.3 41.8 65.3 49.5 50.6 49.2 60.9 41.6 60.9 45.3 47.2 44.7 55.9 38.3 71.3 55.7 56.2 55.6 66.2 47.6 Reduce FLOPs Budget to 20%, ∼0.72 TFLOPs 46.6 43.1 50.1 43.9 51.2 38.9 44.8 41.7 48.2 41.4 48.7 36.4 43.9 40.2 47.2 40.4 47.4 35.8 42.1 39.2 46.1 39.7 46.6 34.3 46.7 43.2 50.5 43.2 51.4 38.8 Reduce FLOPs Budget to 10%, ∼0.36 TFLOPs 35.5 31.7 46.9 35.5 40.1 33.2 35.2 31.4 46.7 35.2 39.8 32.9 34.4 30.9 46.4 34.9 39.1 31.9 34.6 31.3 46.8 35.2 39.4 32.3 35.4 31.1 46.3 35.3 39.7 32.8
OCRB
TextVQA
Avg (%)
74.6
81.3
78.1
72.3 72.0 71.9 71.8 72.2
79.5 79.1 79.0 78.6 79.8
76.1 75.6 75.4 75.1 76.2
72.0 71.3 70.9 69.9 72.4
79.4 78.8 78.8 77.7 79.7
75.7 75.5 75.1 74.4 76.3
71.1 70.8 70.6 70.4 70.6
78.9 78.3 78.2 78.0 78.7
75.5 75.1 74.9 74.6 75.2
69.1 68.1 67.5 66.5 69.3
76.9 74.4 73.1 72.4 77.3
73.5 71.7 70.6 69.8 73.8
67.8 66.5 65.5 65.2 68.5
75.9 74.9 74.4 74.1 76.7
71.5 70.2 69.7 69.3 72.1
62.5 61.3 50.7 47.5 62.6
69.9 67.7 67.4 65.7 66.1
65.7 63.8 62.2 59.5 65.7
54.4 49.4 49.3 45.9 55.6
62.8 57.8 57.7 52.4 63.2
58.0 53.2 53.0 48.8 58.9
17.8 15.3 14.5 13.8 18.6
48.5 46.4 45.4 44.5 47.9
42.5 40.4 39.4 38.3 42.5
10.1 9.9 9.3 9.8 10.0
36.1 35.7 35.2 35.5 35.8
33.6 33.6 32.8 33.1 33.3