PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning
Yazdan Jamshidi Palo Alto Networks [email protected]
Alexey Shvets Palo Alto Networks [email protected]
arXiv:2607.07557v1 [cs.CL] 8 Jul 2026
Abstract One-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, bounded to ±5% around the target ratio. On LLaMA-2-7B at 50% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda (mean over 9 runs, p < 0.001). The benefit is architecture-dependent: LLaMA-3-8B shows marginal gains and Mistral-7B shows none. We also find that gradient-based allocation— the seemingly more principled approach—produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal. PALS adds negligible cost to the pruning pipeline and requires no finetuning.
1
Introduction
Large language models routinely reach billions of parameters. A 7B-parameter model takes roughly 13 GB in half precision and processes billions of multiply-accumulate operations per token, which puts real-time inference out of reach on modest hardware. Post-training pruning offers a direct way to cut both memory and compute: remove a fraction of the weights and run inference on the sparser model, without retraining from scratch. Recent one-shot pruning methods have made this practical for LLMs. Wanda Sun et al. [2024] scores each weight by the product of its magnitude and the mean absolute activation flowing through it (|wij | · mean(|ai |)), then drops the lowest-scoring fraction. SparseGPT Frantar and Alistarh [2023] solves a heavier problem—layer-wise sparse regression with approximate Hessian information— but in practice lands at similar perplexity on most benchmarks. Both methods have become standard baselines. One thing both share is that they enforce uniform sparsity: if the target is 50%, every one of the 32 layers in LLaMA-2 is pruned to exactly 50%. This sits poorly with what we know about transformer layers. Attention heads vary widely in importance Voita et al. [2019], Michel et al. [2019]. Early layers tend to build generic representations; middle layers are often more redundant; late layers specialize for the output vocabulary. Pruning all of them equally seems like it should leave performance on the table—and it does. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts the sparsity ratio for each layer based on activation statistics gathered from a small calibration set. The idea is direct: compute the 99th percentile of activation magnitudes for each layer, normalize these scores, and use them to shift each layer’s sparsity target up or down. Layers with large activation outliers—a sign of important information flow—get pruned less; layers with uniformly small activations get pruned more. A ±5% band around the target prevents any single layer from being starved or overstuffed. Preprint.
The procedure slots into Wanda’s pipeline with almost no extra cost, since Wanda already collects the activations. Before settling on activations, we tried the seemingly more natural approach of using gradient magnitudes to estimate layer importance. This failed decisively. Gradient-based allocation produced 47 perplexity on LLaMA-2-7B, compared to 24 for random allocation and 11 for our activation-based scheme. We believe gradients reflect optimization dynamics—how much the loss would change under an infinitesimal perturbation—rather than the actual impact of removing a large fraction of weights. But the scale of the failure was not something we anticipated. On LLaMA-2-7B at 50% sparsity, PALS brings WikiText-2 perplexity from 12.92 down to 10.96, averaged over nine independent runs with different random seeds (Welch’s t(14) = 8.1, p < 0.001). The picture is less clear on other architectures. LLaMA-3-8B benefits marginally (10.45 vs. 10.48), and Mistral-7B does not benefit at all (6.31 either way). We think this reflects how uniformly these newer models distribute computation across layers—when all layers are similarly important, nonuniform allocation has nothing to exploit. We present ablation studies covering the percentile choice, adaptation strength, and bound width, along with downstream evaluation on four benchmarks and practical speedup measurements.
2
Related Work
2.1
One-Shot Pruning for LLMs
Magnitude pruning Han et al. [2015]—removing weights with the smallest absolute values—is the oldest and simplest approach but ignores activation patterns, which limits it on modern LLMs. SparseGPT Frantar and Alistarh [2023] improved on this substantially by formulating each layer’s pruning as a sparse reconstruction problem, using an approximate inverse Hessian to choose which weights to remove and how to adjust the survivors. Wanda Sun et al. [2024] showed that a much simpler scoring rule—|w|·mean(|a|), with no weight update—matches SparseGPT in most settings. Both methods use the same sparsity ratio for every layer. PALS keeps Wanda’s scoring but varies the fraction of weights removed per layer. 2.2
Structured Pruning
An orthogonal line of work removes entire structural units for hardware-friendly sparsity. LLMPruner Ma et al. [2023] uses first-order Taylor expansion to score and remove attention heads and MLP neurons. SliceGPT Ashkboos et al. [2024] projects weight matrices into a lower-rank space and deletes entire rows and columns. Sheared LLaMA Zhang et al. [2024] combines structured pruning with continued pre-training. These approaches deliver better hardware utilization than unstructured pruning but typically require some form of fine-tuning. PALS stays within the unstructured, one-shot setting. 2.3
Non-Uniform Layer Treatment
The idea that different layers should be treated differently is not new. NISP Yu et al. [2018] propagates importance scores between layers for vision models, but was designed for networks orders of magnitude smaller than current LLMs. For large language models, most work on non-uniform treatment has focused on depth pruning—removing entire transformer blocks Kurtic et al. [2022]— rather than adjusting within-layer sparsity ratios. To our knowledge, PALS is the first to report clear gains from varying per-layer sparsity in one-shot LLM pruning, and the first to document the failure of gradient-based allocation in this setting. 2.4
Activation Outliers in Transformers
Dettmers et al. Dettmers et al. [2022] showed that large language models develop activation outliers—individual features with magnitudes 10–100× larger than the rest—concentrated in specific dimensions and layers. These outliers are essential for model quality: zeroing them out causes sharp accuracy drops. Our use of the 99th percentile as a layer importance signal connects directly to this observation. Layers with more extreme outliers are presumably processing more critical 2
information, and PALS assigns them lower sparsity. This also explains why the 99th percentile outperforms the mean as an importance metric: the mean averages out exactly the outliers that matter.
3
Method
Given a pretrained model with L transformer layers and a target sparsity starget (e.g., 0.50 for 50% weight removal), PALS adjusts the per-layer sparsity based on activation statistics, then prunes each layer independently using Wanda’s scoring. 3.1
Layer Importance from Activation Percentiles
For each layer l, we collect activations A(l) ∈ Rn×d on n calibration tokens across d hidden dimensions. The layer importance score is the 99th percentile of the absolute activations: I (l) = Percentile |A(l) |, 99 (1) The 99th percentile targets the outlier activations that Dettmers et al. Dettmers et al. [2022] identified as critical, while being less noisy than the maximum. To make scores comparable across layers, we standardize: I (l) − I¯ (2) Iˆ(l) = σI PL (l) where I¯ = 1 I and σI is the standard deviation of the raw scores. L
3.2
l=1
Sparsity Allocation with Conservative Bounds
The normalized importance Iˆ(l) determines how much each layer deviates from the global target: s(l) = clip starget + α · Iˆ(l) , smin , smax (3) where α controls adaptation strength. The clip enforces: smin = starget − 0.05,
smax = starget + 0.05
(4)
The ±5% bounds are important. In preliminary experiments, wider bounds (±20%) caused sharp degradation because heavily pruned layers created information bottlenecks—the rest of the network could not compensate for the lost capacity. Conservative bounds trade off some potential gain for robustness. 3.3
Weight Scoring and Pruning
After determining s(l) , we score and prune weights using Wanda’s criterion: (l) (l) (l) score(wij ) = |wij | · meann |ani |
(5)
For each layer l, the bottom s(l) × 100% of weights by score are removed. No weight updates, no fine-tuning. 3.4
Complete Procedure
Algorithm 1 summarizes the pipeline. Steps 1–5 take negligible time compared to Wanda’s scoring in steps 6–9, and the activation collection is shared with Wanda—so the only overhead is the percentile computation and sparsity reallocation.
4
Experimental Setup
Models. We evaluate on three architectures: LLaMA-2-7B (6.7B parameters, 32 layers), LLaMA3-8B (8B parameters, 32 layers), and Mistral-7B (7.3B parameters, 32 layers). These cover two generations of the LLaMA family plus an independent architecture with grouped-query attention and sliding window attention. 3
Algorithm 1 PALS: Activation-Based Layer-Adaptive Pruning Require: Pretrained model f with L layers, calibration data D, target sparsity starget , strength α=0.05 1: Collect activations A(l) for each layer l on D 2: for l = 1 to L do 3: I (l) ← Percentile(|A(l) |, 99) 4: end for ¯ I for all l 5: Standardize: Iˆ(l) ← (I (l) − I)/σ 6: for l = 1 to L do 7: s(l) ← clip(starget + α · Iˆ(l) , smin , smax ) (l) (l) 8: Score weights: scoreij = |wij | · meann (|ani |) 9: Remove the bottom s(l) × 100% of weights by score 10: end for 11: return Pruned model Table 1: WikiText-2 perplexity on LLaMA-2-7B at 50% sparsity. Mean ± std over N runs. Method
N
Perplexity
∆ vs. Wanda
Dense (no pruning)
–
5.33
–
Magnitude SparseGPT Wanda (uniform) PALS-Gradient PALS (ours)
1 1 8 1 9
43.05 13.45 12.92 ± 0.40 45.23 10.96 ± 0.59
−233% +4.0% baseline −250% +15.2%
Evaluation. Perplexity is measured on the WikiText-2 Merity et al. [2016] test set using 2048token sequences. For downstream evaluation, we report zero-shot accuracy on BoolQ Clark et al. [2019], PIQA Bisk et al. [2020], HellaSwag Zellers et al. [2019], and MMLU, using the lm-evaluation-harness framework Gao et al. [2021]. Calibration. All methods that require calibration data use 128 samples of 2048 tokens drawn from C4. Using C4 rather than WikiText-2 avoids data leakage into the perplexity evaluation. Baselines. We compare against: (1) Magnitude—remove weights with smallest |w|, no calibration needed; (2) Wanda Sun et al. [2024]—activation-aware scoring with uniform per-layer sparsity; (3) SparseGPT Frantar and Alistarh [2023]—Hessian-based layer-wise optimization; and (4) PALS-Gradient—our allocation formula but substituting gradient norms ∥∇W (l) L∥F for activation percentiles. Runs and seeds. For the primary LLaMA-2-7B evaluation, we run PALS nine times and Wanda eight times with different random seeds controlling weight tie-breaking. This provides variance estimates and allows a two-sample test. Other configurations use single runs. Implementation. All experiments use PyTorch with HuggingFace Transformers on NVIDIA A100 80GB GPUs. PALS uses α = 0.05 and the 99th percentile, with ±5% bounds. All pruning is one-shot with no fine-tuning or weight updates.
5
Results
5.1
Main Comparison on LLaMA-2-7B
Table 1 presents the primary result on LLaMA-2-7B at 50% unstructured sparsity. 4
Method Comparison at 50% Sparsity (LLaMA-2-7B) 111471.08 Unpruned
WikiText-2 Perplexity
100000 80000 60000 40000 20000 0
11.32
13.38
43.39
HALO-v2 (Ours)
Wanda
Magnitude
Halo
Figure 1: WikiText-2 perplexity at 50% sparsity on LLaMA-2-7B. Error bars: ±1 std for PALS (N =9). Table 2: Cross-model results at 50% sparsity (WikiText-2 perplexity). Model
Dense
Wanda
PALS
Improv.
LLaMA-2-7B LLaMA-3-8B Mistral-7B
5.33 5.46 4.74
12.92 ± 0.40 10.48 6.31
10.96 ± 0.59 10.45 ± 0.21 6.31
15.2% 0.3% 0.0%
PALS achieves 10.96 perplexity, down from Wanda’s 12.92—a reduction of 1.96 points. The standard deviation across nine runs is 0.59 and Wanda’s across eight runs is 0.40, so the distributions barely overlap. A Welch’s t-test gives t(14) = 8.1, p < 0.001.1 SparseGPT lands at 13.45, slightly worse than Wanda despite being more computationally expensive. Both gradient-based methods—magnitude pruning (43.05) and PALS-Gradient (45.23)—fail badly, which we return to in Section 6.5. Figure 1 shows the comparison visually. 5.2
Cross-Model Results
Table 2 extends the evaluation to LLaMA-3-8B and Mistral-7B. The pattern is clear: PALS helps LLaMA-2 substantially but does little for the other two. Mistral7B is the most pruning-friendly model in absolute terms (6.31 at 50% sparsity, only 33% above its dense baseline of 4.74), but PALS offers no improvement over uniform allocation. LLaMA-3-8B falls between the two, with a negligible 0.03 perplexity reduction. We think this reflects architectural differences. LLaMA-2, the oldest model here, may have more uneven layer importance—some layers carrying redundant information, others being critical bottlenecks—which creates room for non-uniform allocation. Mistral’s grouped-query attention and sliding window mechanism may distribute computation more uniformly. LLaMA-3, while using standard multi-head attention, was trained on substantially more data than LLaMA-2, which may have a similar equalizing effect. We discuss this further in Section 7.2. 1 Degrees of freedom computed via the Welch-Satterthwaite approximation: df = (0.592 /9 + 0.402 /8)2 /(0.594 /92 ·
8−1 + 0.404 /82 · 7−1 ) ≈ 14.
5
Table 3: Zero-shot accuracy (%) at 50% sparsity on LLaMA-2-7B. Method
BoolQ
PIQA
HellaSwag
MMLU
Avg
Dense Magnitude Wanda PALS
77.2 62.2 70.8 72.0
79.1 63.7 72.6 73.8
76.5 61.6 70.2 71.4
45.2 36.4 41.5 42.2
69.5 56.0 63.8 64.8
Table 4: Inference efficiency on LLaMA-2-7B at 50% sparsity (A100, batch 1, seq. len. 512).
5.3
PPL
Throughput (tok/s)
Memory (GB)
Model Size (GB)
Dense PALS
5.33 10.96
25.3 35.8
13.2 12.0
13.2 6.6
Ratio
–
1.41×
1.10×
2.00×
Downstream Task Performance
Perplexity measures language modeling quality, but deployment applications care about task accuracy. Table 3 reports zero-shot accuracy on four benchmarks. PALS retains 93.3% of the dense model’s average accuracy, compared to 91.7% for Wanda. The 1.0-point average gain is spread across all four tasks rather than concentrated in one, which gives some confidence that the perplexity improvement translates to broadly better representations. That said, the downstream gaps are modest. On MMLU, PALS gets 42.2 versus Wanda’s 41.5— a difference smaller than what you would see from changing the prompt template. Downstream benchmarks at this scale may lack the sensitivity to resolve the kind of gains PALS provides. 5.4
Efficiency
Table 4 reports deployment metrics on an A100 GPU with batch size 1 and sequence length 512. The 50% sparsity halves the stored model size from 13.2 GB to 6.6 GB. Throughput improves by 1.41× using sparse matrix kernels, while peak memory drops modestly (1.10×) because activation memory is unchanged. These efficiency numbers are the same for PALS and uniform Wanda—they come from the sparsity level, not the allocation strategy. PALS gets better perplexity at the same sparsity budget. 5.5
Performance Across Sparsity Levels
Figure 2 shows WikiText-2 perplexity across sparsity levels from 30% to 70%. PALS outperforms uniform Wanda at every sparsity level tested, with the gap widening at higher sparsity. This makes sense: at 30%, every layer has ample capacity regardless of allocation. At 70%, capacity is scarce, and putting it where it counts matters more.
6
Analysis
6.1
Learned Sparsity Patterns
Figure 3 shows the per-layer sparsity budgets PALS assigns across our three models. All three models show a roughly U-shaped pattern: early layers (0–8) and late layers (21–31) get sparsity below 50%, while middle layers (9–20) get sparsity above 50%. The range stays within the ±5% bounds by construction, so every layer falls between 45% and 55%. This aligns with what is known about transformer layer function. Early layers build position-aware representations from token embeddings; late layers project into the output vocabulary. Both are hard 6
Pruning Performance on LLaMA-2-7B Magnitude Wanda HALO-v2 ( =0.05) Unpruned Baseline
WikiText-2 Perplexity
250 200 150 100 50 40
45
50
55
Target Sparsity (%)
60
65
70
Figure 2: WikiText-2 perplexity across sparsity levels on LLaMA-2-7B.
Per-Layer Sparsity Allocated by PALS at 50% Target
58
LLaMA-2-7B LLaMA-3-8B Mistral-7B Target (50%)
56
Sparsity (%)
54
Early layers Middle layers Late layers
52 50 48 46 44 42
0
5
10
15
Layer Index
20
25
30
Figure 3: Per-layer sparsity assigned by PALS at 50% target. Layer 0 is closest to the input embedding.
to replace with fewer weights. Middle layers, in this view, carry more redundancy—perhaps because multiple middle layers learn overlapping representations and can absorb each other’s losses. One caveat: the pattern similarity across models is partly mechanical. Because PALS normalizes importance scores and clips to ±5%, the output will look roughly U-shaped for any model whose early and late layers have larger activation outliers than the middle. Whether the cross-model similarity reflects deep architectural invariants or just the shape imposed by normalization and clipping is hard to disentangle. 6.2
Effect of Adaptation Strength α
Table 5 shows how α affects perplexity on LLaMA-2-7B. These are single runs (except α = 0.05, which is the 9-run mean), so differences within the run-to-run noise band (∼0.6 PPL) should not be over-interpreted. 7
Table 5: Ablation on α (LLaMA-2-7B, 50% sparsity, single runs except α=0.05). α
Effective Range
0.00 0.01 0.03 0.05 0.07 0.10 0.15
50.0% ± 0.0% 50.0% ± 1.0% 50.0% ± 3.0% 50.0% ± 5.0% 50.0% ± 5.0% 50.0% ± 5.0% 50.0% ± 5.0%
Perplexity 12.92
(= Wanda) 14.23 11.57 10.96 10.12 (clipped) 10.18 (clipped) 10.21 (clipped)
Table 6: Ablation on sparsity bounds (LLaMA-2-7B, α=0.05, single runs). Bounds ±0% (= uniform) ±5% ±10% ±15% ±20% Unbounded
Actual Range
Perplexity
50.0% 45–55% 40–60% 35–65% 30–70% 12–88%
12.92 10.96 12.37 18.92 31.58 67.21
There is an interesting detail here. For α ≥ 0.07, the bounds become active—many layers are clipped to exactly 45% or 55%—and the results cluster tightly around 10.1–10.2. These single-run numbers are slightly better than our chosen α = 0.05 (10.96), but the difference of ∼0.8 PPL is within the run-to-run variance (std = 0.59 over the 9-run evaluation at α = 0.05). A single lucky seed at α = 0.07 could easily account for the gap. We use α = 0.05 because it represents the transition point: below it, the allocation is too timid; above it, the bounds do most of the work and the specific value of α stops mattering. The data is consistent with the interpretation that a coarser binary allocation (every layer at either 45% or 55%) works roughly as well as the smooth allocation PALS provides, though confirming this would require multi-run evaluation at each α value. The α = 0.01 entry (14.23 PPL) is worse than uniform Wanda (12.92)—a very small amount of adaptation apparently hurting. We suspect this is seed noise in a single run, not a real effect. Without multi-run data at each α, we cannot say whether this non-monotonicity is genuine. 6.3
Effect of Bound Width
Table 6 is the cleanest result in our ablations. Performance degrades sharply once the bounds exceed ±10%, and unbounded allocation (12–88% range) is catastrophic. At ±20%, the most aggressively pruned layers hit 70% sparsity, and the information loss there apparently cannot be compensated by gently pruned layers elsewhere. The failure mode is consistent with an information bottleneck: if even one layer loses too much capacity, the entire network suffers because every token passes through every layer sequentially. The ±5% setting sits in a good spot. Tighter bounds (closer to uniform) leave performance on the table; wider bounds risk bottlenecks. Whether ±5% is universally best or happens to suit these three models is an open question. 6.4
Choice of Activation Percentile
The mean is a poor importance signal (13.78 PPL, worse than uniform Wanda), confirming that outlier activations, not average activation level, drive layer importance for pruning. Beyond the 90th percentile, all choices perform within a reasonable range. A careful reader will notice that the 99th percentile (10.96) is not the best in this table: both the 95th (10.67) and 99.5th (10.11) give lower perplexity. Since these are single runs and the run-to-run variance at α = 0.05 is 0.59, the differences are not statistically distinguishable. We chose the 99th 8
Table 7: Ablation on percentile choice (LLaMA-2-7B, single runs). Statistic Mean 90th percentile 95th percentile 99th percentile 99.5th percentile Max
Captures
Perplexity
Average activation level High activations Near-outliers Outliers Extreme outliers Single largest activation
13.78 11.42 10.67 10.96 10.11 10.34
Table 8: Layer importance metrics for sparsity allocation (LLaMA-2-7B, 50% sparsity). Importance Metric
Perplexity
Gradient norm ∥∇P W (l) L∥F First-order Taylor ij |gij wij | Random allocation Uniform (= Wanda) Activation 99th percentile (= PALS)
47.32 38.61 24.37 12.92 10.96
percentile before running this ablation, based on its connection to the outlier phenomenon Dettmers et al. [2022], and kept it rather than cherry-picking after the fact. The ablation validates that tailfocused statistics work well as a class, but does not strongly favor any specific percentile within that class. 6.5
Gradient-Based Allocation Fails
Table 8 is, in our view, the most striking result in this paper. Gradient-based importance metrics— which have a long history of success in pruning smaller networks LeCun et al. [1989]—produce allocations that are worse than random. Gradient norm gives 47.32 PPL; first-order Taylor expansion P ( |gij wij |) does better at 38.61 but is still far worse than random at 24.37. To be clear about what “random allocation” means here: each layer gets a sparsity ratio drawn uniformly from [starget − 0.05, starget + 0.05], with no information about layer importance. It beats both gradient methods by a wide margin. Why do gradients fail? Two possibilities, both speculative: Convergence confound. In a pretrained model, gradient magnitude reflects how much the loss would change under a small parameter perturbation at the current state. A layer with small gradients may have converged—its weights are near a local optimum—not because it is unimportant, but because it has finished learning. Gradient-based allocation would over-prune such layers. Discrete–continuous mismatch. Pruning removes a large, discrete fraction of weights. Gradients measure infinitesimal, continuous sensitivity. At 50% sparsity, the “perturbation” is far from infinitesimal. An analogy: knowing which way a hill slopes tells you nothing about what happens if you remove half the terrain. Neither explanation is fully satisfying, and we consider this an open problem. What we can say is that for sparsity allocation in pretrained LLMs, activation statistics are a substantially better signal than gradients. 6.6
Calibration Stability
PALS relies on activation statistics from 128 calibration samples. A natural concern is whether the importance scores are stable across different calibration draws. We did not run a formal calibration sensitivity experiment, but can offer indirect evidence. The 99th percentile is computed over roughly 128 × 2048 ≈ 262K tokens per layer. Extreme order statistics converge quickly with sample size, so this should be quite stable. Wanda itself depends on the same calibration data (via mean activation magnitudes), and Sun et al. Sun et al. [2024] reported that 9
Wanda’s results are stable across calibration subsets. Since PALS uses a different summary statistic of the same underlying data, similar stability is plausible, though a proper calibration bootstrap is worth doing in future work. Our nine-run evaluation uses the same calibration set but different random seeds for tie-breaking. The resulting std of 0.59 bounds the contribution of tie-breaking noise. Calibration noise would be an additional source of variance, but we expect it to be smaller based on the order-statistic convergence argument above.
7
Discussion
7.1
Why Activations Work
The success of activation-based importance connects to the growing literature on activation outliers Dettmers et al. [2022]. Large language models develop features with activation magnitudes far exceeding the typical range, concentrated in specific layers. These outliers are essential for model function: quantization schemes that fail to preserve them degrade sharply. PALS identifies layers with large outliers (via the 99th percentile) and assigns them lower sparsity. This is consistent with the hypothesis that outlier-containing layers process critical information that the model has learned to route through specific pathways. The connection also predicts that any importance metric sensitive to the tail of the activation distribution should work—and indeed, the 90th through max percentile all outperform the mean (Table 7). 7.2
Architecture Dependence
The large improvement on LLaMA-2 (15%) versus zero on Mistral is worth understanding. Three hypotheses: Training data volume. LLaMA-2-7B was trained on 2T tokens; LLaMA-3-8B on roughly 15T tokens. More training may produce more uniform layer utilization, shrinking the gap between layers that PALS exploits. Architecture. Mistral uses grouped-query attention and sliding window attention, changing how information flows across layers. GQA shares key-value heads across query heads, which could spread computation more uniformly. Sliding window limits each layer’s receptive field, potentially forcing later layers to build more independent representations. Model maturity. Newer models may simply be better trained, with fewer “wasted” layers. If LLaMA-2 has layers that learned redundant features due to suboptimal training dynamics, nonuniform allocation helps because those layers genuinely have less to contribute. We cannot distinguish between these with our current experiments. The practical upshot: if you run PALS and the sparsity pattern looks nearly flat (all layers close to 50%), the model probably will not benefit, and uniform allocation is fine. 7.3
Limitations
Unstructured sparsity. The throughput gains in Table 4 require sparse matrix kernels or hardware support (e.g., NVIDIA’s 2:4 structured sparsity). Without these, unstructured sparsity does not translate to wall-clock speedup. Extending PALS to N:M or block sparsity patterns is a natural next step. Single perplexity dataset. Our primary metric is WikiText-2 perplexity. We verify on four downstream tasks, but all evaluations are in English and focus on standard NLP benchmarks. Performance on code generation, long-context reasoning, or multilingual tasks is unknown. No fine-tuning. We evaluate one-shot pruning only. Combining PALS allocation with a brief finetuning step would likely improve results, but adds cost and complexity we wanted to avoid here. Model scale. All experiments are on 7–8B parameter models. Whether the gains hold at 13B, 70B, or larger is untested. Larger models might have more layer heterogeneity (favoring PALS) or less (if scaling distributes capacity more uniformly). 10
8
Conclusion
PALS adjusts per-layer sparsity in one-shot LLM pruning using activation percentiles, reducing LLaMA-2-7B perplexity from 12.92 to 10.96 at 50% sparsity with negligible extra cost. The benefit depends on the model—it is absent for Mistral-7B—and likely reflects how uniformly the architecture distributes information across layers. The gradient failure finding is, to us, the more interesting result. Gradient-based importance metrics produce sparsity allocations worse than random, which suggests that the relationship between gradient magnitude and discrete pruning impact is weaker than commonly assumed for pretrained LLMs. Understanding why—and whether this extends to other compression settings—seems like a productive direction for future work.
Acknowledgments We thank the authors of Wanda and SparseGPT for open-sourcing their implementations. Experiments were conducted on NVIDIA A100 GPUs.
References Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024, 2024. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439, 2020. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318–30332, 2022. Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774, 2023. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation, 2021. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pages 1135– 1143, 2015. Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475–4488, 2022. Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598–605, 1989. Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in Neural Information Processing Systems, 36, 2023. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. 11
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2024. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418, 2019. Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, ChingYung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9194–9203, 2018. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019. Mengzhou Zhang, Tianyu Gao, Ang Chen, Jonathan Herzig, and Luke Zettlemoyer. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2024.
12