MedCore: Boundary-Preserving Medical Core Pruning for MedSAM
arXiv:2605.13688v1 [cs.CV] 13 May 2026
Cenwei Zhang Shanghai Jiao Tong University Shanghai, China [email protected]
Suncheng Xiang∗ Shanghai Jiao Tong University Shanghai, China [email protected]
Lei You∗ Technical University of Denmark Ballerup, Denmark [email protected]
Abstract Medical segmentation foundation models such as SAM and MedSAM provide strong prompt-driven segmentation, but their image encoders are still too large for many clinical settings. Compression is also risky in medicine because a model can keep high Dice while losing boundary fidelity. We propose MedCore, a structured pruning framework for MedSAM. The main idea is to preserve two kinds of structures: structures that became important during SAM-to-MedSAM adaptation, and structures that have high boundary leverage. We identify the first type by a dual-intervention score that compares zeroing a group with resetting it to its original SAM weight. We identify the second type by boundary-aware Fisher estimation. We also introduce a boundary leverage principle, which shows that compression-induced boundary displacement is controlled by logit perturbation on the boundary divided by the logit spatial gradient. This principle explains why boundary metrics can degrade even when Dice remains high. On polyp segmentation benchmarks, MedCore reduces parameters by 60.0% and FLOPs by 58.4% while achieving Dice 0.9549, Boundary F1 0.6388, and HD95 5.14 after recovery fine-tuning. It also reaches 86.6% parameter reduction and 90.4G FLOPs with strong boundary quality. Our analysis further shows that MedSAM lies in a head-fragile boundary regime: head-pruning steps have 2.887 times larger 95th-percentile boundary leverage than MLP-pruning steps, and this logit-level effect is consistent with BF1 and HD95 degradation. Our code is available at https://github.com/cenweizhang/MedCore.
1
Introduction
Segment Anything Model (SAM) changed image segmentation by turning segmentation into a prompt-driven task [1]. MedSAM adapts this idea to medical images and gives strong results across many modalities [2]. These models are useful because they can accept prompts such as bounding boxes and produce high-quality masks. Their cost is still high. The MedSAM image encoder is a ViT-based model [3], and its inference cost can be difficult to support in real-time endoscopy, point-of-care ultrasound, and other resource-limited settings. A natural solution is model compression. Yet medical segmentation has a failure mode that ordinary compression metrics can hide. A compressed model may still identify the coarse lesion region ∗ Corresponding author.
Preprint.
and keep a high Dice score, while its predicted boundary becomes thick, broken, or shifted. This is a serious issue because boundary quality affects lesion measurement, treatment planning, and clinical interpretation. This paper studies structured pruning for MedSAM under a boundary-sensitive setting, directly pruning the existing adapted model rather than training a lightweight surrogate. Since MedSAM is adapted from SAM instead of trained from scratch, we exploit the transition from the SAM checkpoint θ S to the MedSAM checkpoint θ M to identify which structures remain general-purpose and which become medical-specific. We propose MedCore, a boundary-preserving medical core pruning framework that keeps a compact yet medically adapted MedSAM subnetwork. For each structure group g, MedCore uses a dualintervention score that combines its current prediction contribution under zeroing and its medicaladaptation contribution under reset to the original SAM weights, which are two counterfactual questions. A group is safe to prune only when both signals suggest low importance. We further introduce a boundary leverage principle by viewing the predicted mask boundary as a level set of the logit map. If pruning changes the logit map by δG (x), the first-order boundary displacement is proportional to δG (x) and inversely proportional to the spatial gradient norm ∥∇sθ (x)∥2 . This gives a direct geometric explanation of boundary collapse. Compression is dangerous when it creates a large logit perturbation on a boundary where the logit slope is small. We use this principle to motivate boundary-aware Fisher scoring and to analyze the different roles of attention heads and MLP connections. In summary, Our three contributions are as follows: First, we formulate MedSAM pruning as medicalcore extraction and introduce a dual-intervention score based on zeroing and reset-to-SAM operations. Second, we propose boundary-aware Fisher and Cross-Fisher approximations to make the intervention score practical and boundary-sensitive. Third, we introduce boundary leverage as a simple geometric principle for explaining Dice-boundary mismatch and head-MLP pruning trade-offs. Our results further show that this distinction matters. Experiments on polyp endoscopy, breast ultrasound, and dermoscopy demonstrate that MedCore can remove a large fraction of MedSAM parameters while preserving strong region and boundary performance. In a 7 × 7 head-MLP pruning sweep, head-pruning steps exhibit 2.887× larger 95th-percentile boundary leverage than MLPpruning steps, with all 15 valid paired comparisons favoring MLP pruning. At the metric level, the BF1 and HD95 damage densities of head pruning are respectively 2.607× and 2.432× those of MLP pruning, indicating that MedSAM lies in a head-fragile boundary regime: attention heads are not merely low-parameter components, but can have high boundary leverage.
2
Related Work
Promptable segmentation and medical SAM. SAM formulates segmentation as a promptable task using a ViT image encoder, a prompt encoder, and a lightweight mask decoder [1, 4]. MedSAM adapts this design to medical images via large-scale image-mask pairs [2]. Subsequent work improves medical adaptation along several axes: parameter-efficient tuning (SAMed, SAM-Med2D, MedicalSAM-Adapter [5, 6, 7]; Polyp-SAM for transfer to polyp segmentation [8]), boundary-aware feature design (I-MedSAM with implicit representations [9]; LDFSAM with distilled feature prompting [10]), 3D/video extension (MedSAM2 [11]), and feature fusion with task-specific networks [12, 13]. These methods focus on prompting, fusion, or boundary localization. We study a complementary question: how to compress an already adapted MedSAM checkpoint while preserving the structures that became important during SAM-to-MedSAM adaptation. Efficient SAM and medical segmentation models. The heavy ViT image encoder of SAM has stimulated extensive research on efficient segmentation, including lightweight encoder replacement, knowledge distillation, quantization, and structured pruning [14, 15, 16, 17, 18]. Medical settings also have efficient segmentation backbones, including U-Net, nnU-Net, Swin-Unet, EMCAD, MK-UNet, and quantized medical SAM variants [19, 20, 21, 22, 23, 18].Recent lightweight polyp models, such as HSSAM-Net, further show that careful multi-scale aggregation and boundary-aware enhancement can give strong accuracy with small parameter budgets [24]. These works confirm the need for efficient segmentation. Most of them redesign the architecture, train a compact surrogate, or quantize a model. MedCore instead directly prunes MedSAM and uses both the original SAM weights θ S and the adapted MedSAM weights θ M . 2
Figure 1: MedCore pruning overview. The Cross-Fisher signal F cross combines SAM and MedSAM (r) gradients to score attention heads and MLP channels via Qg , driving block-sensitive cascade pruning of the ViT encoder (blocks B10–B11 protected) with boundary- and frequency-aware recovery. Yellow gears mark where pruning operations are applied: Cross-Fisher estimation, priority computation, per-block head pruning, and recovery-loss-driven fine-tuning.
Pruning and Transformer compression. Classic pruning methods estimate the loss increase caused by removing weights or groups. Optimal Brain Damage and Optimal Brain Surgeon use second-order information to approximate this increase [25, 26]. Deep Compression combines pruning, quantization, and coding [27]. Later methods use movement, Fisher information, Hessian-aware saliency, or grouplevel sensitivity to prune pretrained and Transformer models [28, 29, 30]. Attention-head pruning studies show that many heads can be redundant, but some specialized heads are important and taskdependent [31, 32]. This literature motivates structured pruning, but it usually estimates importance from one endpoint checkpoint. Boundary preservation in medical segmentation. Medical segmentation models often optimize region losses such as Dice and cross-entropy. These losses can keep coarse foreground overlap but still miss fine contours. To address this limitation, prior work has explored boundary losses, surfaceor Hausdorff-inspired objectives, topology-aware regularization, and boundary-aware architectures for more accurate contour modeling [33, 34, 35, 36, 37, 38, 39]. Different from these task-specific segmentation methods, our work brings boundary preservation into model compression by asking not only whether pruning preserves Dice, but also whether it perturbs boundary-critical structures and the boundary logit field.
3
MedCore: Boundary-Preserving Medical Core Pruning
MedCore has two design goals. The first goal is to preserve structures that became important during medical adaptation. The second goal is to preserve structures whose removal causes large boundary displacement. 3.1
Problem Formulation
A training or calibration sample is written asz = (I, b, M ), where I ∈ R3×H×W is a medical image, b ∈ R4 is a bounding-box prompt, and M ∈ {0, 1}H×W is a binary mask. MedSAM maps (I, b) to a c = σ(sθ,ϕ (I, b)). The parameter vector θ = (θE , θD ) logit map sθ,ϕ (I, b) and a probability mask M contains the image encoder and mask decoder. The prompt encoder parameter ϕ is frozen. 3
Given a calibration set Dcal = {zn }N n=1 , we define the empirical medical risk as N 1 X b ℓseg (zn ; θ), R(θ) = N n=1
ℓseg = ℓDice + ℓBCE .
(1)
We partition pruneable parameters into structured groups G = {g1 , . . . , gK }. In our implementation, groups are attention heads and MLP connection groups in the ViT image encoder. A binary group mask s ∈ {0, 1}K gives the masked parameter s ⊙G θ. If cg is the cost of group g, the ideal pruning problem is X b ⊙G θ) s.t. min R(s sg cg ≤ B. (2) θ,s
g∈G
This problem is combinatorial. MedCore approximates it by scoring groups and removing low-score groups under a block-aware budget. We keep two checkpoints: θ S is the original SAM checkpoint, and θ M is the MedSAM checkpoint. Pruning starts from θ M , while θ S serves as the adaptation reference. 3.2
Boundary Leverage Principle
We now state the geometric principle that motivates boundary-aware pruning. For a fixed image and prompt, we write the logit map as sθ (x), where x indexes a spatial location. The predicted boundary is the zero level set of this logit map. A compression operation G changes the model to θ −G and induces the logit perturbation δG (x) = sθ−G (x) − sθ (x). The following assumption is standard in level-set perturbation analysis. It only says that the boundary is not locally flat in the logit field. Assumption 3.1 (Regular boundary). For a given image and prompt, the logit map sθ is continuously differentiable in a neighborhood of its zero level set Γθ = {x : sθ (x) = 0}. There exists κ > 0 such that ∥∇sθ (x)∥2 ≥ κ for every x ∈ Γθ . Definition 3.2 (Boundary leverage). Let G be a compression operation with compression gain ∆CG > 0. Let B be a narrow boundary band around a ground-truth or predicted boundary. We define the boundary leverage of G as 1 |δG (x)| λbd = . (3) E x∈B G ∆CG ∥∇sθ (x)∥2 + ε Here ε > 0 is a small constant. This quantity estimates the boundary displacement caused by one unit of compression. Theorem 3.3 (First-order boundary perturbation). Under Assumption 3.1, suppose that δG is sufficiently small in a neighborhood of Γθ . For each x ∈ Γθ , let uG (x) be the signed normal displacement from the old boundary to the new boundary induced by G. Then uG (x) = −
δG (x) + O(∥δG ∥2C 1 ). ∥∇sθ (x)∥2
(4)
Theorem 3.3 explains why boundary metrics can fail before Dice fails. Boundary displacement depends on the logit perturbation at the boundary and on the local logit slope. A group with small parameter count can still be dangerous if it causes a large boundary logit perturbation. We prove the theorem in Appendix A. Proposition 3.4 (Boundary-aware budget rule). Let cH and cM be compression budgets assigned to heads and MLPs, and let Ebd (cH , cM ) be a locally differentiable boundary error. If ∂Ebd ∂Ebd > ∂cH ∂cM at a feasible interior allocation, then moving a small amount of compression budget from head pruning to MLP pruning decreases boundary error to first order while keeping the total compression fixed. Proposition 3.4 turns boundary leverage into a budget principle. It does not say that heads are always more important than MLPs. It says that the safer compression direction is the one with lower marginal boundary damage. Section 5.2 verifies that, for MedSAM, head-pruning steps have much higher boundary leverage than MLP-pruning steps. 4
3.3
Medical-Core Scoring via Dual Intervention
The boundary leverage principle says which structures are dangerous for boundaries. We also need to preserve structures that became important during medical adaptation. MedSAM has a useful reference checkpoint, namely the original SAM checkpoint θ S . We therefore score each group g by two counterfactual interventions. The zero intervention sets group g to zero while keeping all other groups unchanged. If Tg0 denotes this operation, the exact zero cost is b ⊙G T 0 (θ M )) − R(s b ⊙G θ M ). ∆zero = R(s (5) g
g
A small value means that g contributes little to current MedSAM predictions. The reset intervention replaces group g in MedSAM by its SAM value. If TgS denotes this operation, the exact reset cost is b ⊙G TgS (θ M )) − R(s b ⊙G θ M ). ∆reset = R(s (6) g A large reset cost means that the change from θgS to θgM is functionally important for medical segmentation. This signal is different from the zero cost. It measures adaptation-specific importance rather than only endpoint importance. We combine both costs as b zero b reset Qg = αb(g) ∆ + (1 − αb(g) )∆ , (7) g g where b(g) is the Transformer block containing g. We use a block-level mixture weight because different blocks respond differently to boundary and adaptation signals. The pruning priority is Qg Pg = , (8) (cg + ε)τ where τ ≥ 0 controls cost normalization. Groups with smaller Pg are pruned first. 3.4
Boundary-Aware Fisher and Cross-Fisher Approximation
Computing Eq. (5) and Eq. (6) exactly for every group is expensive. We use a local second-order approximation. Around θ M , zeroing group g corresponds to the perturbation −θgM , and resetting group g corresponds to the perturbation θgS − θgM . We approximate the Hessian by diagonal Fisher information. The loss used for Fisher estimation should reflect the metric we want to preserve. We therefore use a boundary-weighted loss. Let B(M ) be a morphological boundary map computed by dilation minus erosion. With pixel index u, we define X c, M ). ℓbd (z; θ) = (1 + λbd B(M )u ) ℓBCE (sθ,ϕ (I, b)u , Mu ) + ℓDice (M (9) u
For a sub-distribution r, such as one dataset or modality, the MedSAM Fisher is X ∂ℓbd (zn ; θ M ) 2 1 M . Fr,i = (r) ∂θi |Dcal | (r)
(10)
zn ∈Dcal
This gives the zero approximation 1X M M 2 b zero,(r) ∆ = F (θ ) . g 2 i∈g r,i i For the reset cost, the perturbation is the SAM-to-MedSAM parameter change. We also compute the SAM Fisher FiS on the same calibration data and use a geometric mean, q cross MFS + ε . Fr,i = Fr,i (11) F i The reset approximation becomes X b reset,(r) = 1 ∆ F cross (θiM − θiS )2 . g 2 i∈g r,i
(12)
The geometric mean down-weights parameters that are insensitive in either SAM or MedSAM. This prevents the reset score from being dominated by large but functionally irrelevant weight shifts. 5
3.5
Distribution-Aware Aggregation and Budget Allocation
Medical data are heterogeneous. A structure that appears redundant on one dataset may be important b (r) on another dataset. We compute per-distribution scores Q g using Eq. (7) and aggregate them as b dist Q = g
R X
b (r) b (r) πr Q g + β Varr [Qg ],
(13)
r=1
P where πr ≥ 0 and r πr = 1. The variance term makes pruning more conservative for groups that are important on only some sub-distributions. We also allocate pruning budgets non-uniformly across Transformer blocks. Let Bℓ be the parameter set of block ℓ. We estimate block sensitivity by X X M Sℓ = Fi , Fi = πr Fr,i . r
i∈Bℓ
Blocks with high Sℓ receive smaller pruning quotas. In the main one-time pruning experiments, the deepest blocks are protected because they are closest to the mask decoder and have the highest Fisher sensitivity. In the extreme sequential setting, we apply a second conservative pruning pass to previously protected blocks after the first checkpoint has been selected. 3.6
Head-to-MLP Cascade and Recovery Fine-Tuning
MedCore prunes in a cascade. We first prune attention heads. Then we run a short recovery phase. We then prune MLP connection groups from the head-pruned model. This schedule avoids removing two different functional families at the same time. It also lets us analyze the marginal boundary effect of head and MLP pruning separately. After pruning, we recover the remaining weights with a lightweight objective: Lrec = Lseg + λ1 Lbd + λ2 Lfeat + λ3 Llogit + λ4 Lfreq .
(14)
Here Lbd is boundary-weighted BCE, Lfeat distills encoder features from the unpruned MedSAM teacher, Llogit distills boundary-region logits, and Lfreq penalizes high-frequency mask discrepancies. Recovery is not used to define the pruning score. It lets the remaining structures re-coordinate after structural removal.
4
Experiments
4.1
Experimental Setup
We evaluate MedCore on MedSAM with a ViT-B image encoder. The base model is the official MedSAM checkpoint, and the reference checkpoint is SAM ViT-B. We freeze the prompt encoder in all experiments. We use five medical image segmentation datasets across three modalities: CVCClinicDB [40], CVC-ColonDB [41], and Kvasir-SEG [42] for polyp endoscopy, BUSI [43] for breast ultrasound, and ISIC2018 [44, 45] for dermoscopy. Each dataset is split into training and test sets with a fixed random seed. For Fisher estimation, we sample 128 calibration images per dataset. We use batch size 1 for Fisher estimation to preserve per-sample gradients. We report Dice, Intersection over Union (IoU), Boundary F1 (BF1), and 95th-percentile Hausdorff distance (HD95). We also report parameters and FLOPs. BF1 and HD95 are the key metrics for boundary fidelity. Unless stated otherwise, the main MedCore configurations include post-pruning recovery fine-tuning. The component ablation in Section 5.1 is reported without post-pruning fine-tuning for the one-time settings, so that it isolates the direct effect of each pruning component. 4.2
Main Results on Polyp Segmentation
Table 1 reports macro-averaged results on the three polyp benchmarks. MedCore reaches strong compression while preserving boundary quality after recovery. The h50_m70 configuration removes 60.0% of parameters and reduces FLOPs by 58.4%, while achieving Dice 0.9549, BF1 0.6388, and 6
Table 1: Macro-averaged results on CVC-ClinicDB [40], CVC-ColonDB [41], and Kvasir-SEG [42]. MedCore configurations include post-pruning fine-tuning. One-time uses a single head-to-MLP cascade. Sequential starts from h70_m95 and applies a second conservative pruning pass. Category
Method
Params↓ FLOPs↓
Dice↑
Baseline
MedSAM [2]
89.7M
926.5G 0.9191 0.8648 0.5321
21.29
Efficient SAM
EfficientSAM-s [15] EfficientSAM-t [15] SlimSAM [17]
26.4M 10.2M 28.0M
188.0G 0.8765 0.8131 0.4340 55.9G 0.8723 0.8103 0.4218 189.9G 0.8435 0.7758 0.3922
29.73 25.98 34.11
Medical Seg.
Swin-Unet [21] nnU-Net [19, 20] EMCAD [22] MK-UNet [23]
27.2M 46.3M 26.8M 0.32M
5.92G 75.1G 10.6G 0.61G
0.4347 0.5382 0.5305 0.4635
23.88 32.15 30.31 46.21
Medical SAM
SAMed [5] QMedSAM [18]
91.2M 9.79M
206.7G 0.7419 0.6640 0.3283 75.7G 0.8089 0.7108 0.4118
97.57 14.55
MedCore One-time
h40_m30 h50_m70 h70_m70 h70_m95
61.3M 35.8M 30.1M 24.9M
620.7G 385.2G 287.5G 245.0G
0.6508 0.6388 0.6342 0.6259
7.23 5.14 6.23 6.78
MedCore Sequential h60_m70 h72_m84 h84_m95
15.6M 13.7M 12.1M
132.6G 0.9547 0.9176 0.6506 112.4G 0.9536 0.9153 0.6337 90.4G 0.9550 0.9174 0.6462
5.91 5.47 5.12
0.7845 0.9093 0.9141 0.8658
0.9547 0.9549 0.9524 0.9522
IoU↑
0.6933 0.8574 0.8608 0.7986
0.9173 0.9169 0.9130 0.9124
BF1↑ HD95↓
Figure 2: Qualitative segmentation comparison between our two high-pruning configurations and other models. Incorrect segmentation regions and boundary artifacts are highlighted with red boxes. HD95 5.14. The h70_m95 configuration removes 72.2% of parameters and still obtains BF1 0.6259 and HD95 6.78. The sequential configuration h84_m95 reaches 12.1M parameters and 90.4G FLOPs, while keeping strong region and boundary metrics. We present these results as compressed models after the same recovery pipeline. The improvement over the original MedSAM checkpoint should be read together with this recovery setting. The main conclusion is that MedCore keeps the benefit of boundary-aware recovery under heavy compression, rather than merely keeping region-level overlap. 4.3
Cross-Modality Generalization
Table 2 evaluates the same pruning pipeline on CVC-ClinicDB, BUSI, and ISIC2018. On ClinicDB and ISIC2018, MedCore improves boundary metrics after recovery. On BUSI, moderate compression preserves performance closely, while aggressive compression begins to degrade BF1 and HD95. This pattern is useful because it shows both the strength and the limit of direct cross-modality transfer. MedCore is not a replacement for target-domain validation. It is a compression framework that can preserve strong performance when the calibration and recovery data are representative.
5
Analysis and Ablation
5.1
Component Ablation: What Drives Boundary Preservation?
Table 3 ablates the main components. The one-time columns do not use post-pruning fine-tuning, so they isolate the structural effect of pruning decisions. The sequential column uses the same recovery protocol for all variants. 7
Table 2: Cross-modality results with post-pruning fine-tuning. Dataset
Dice↑
BF1↑ HD95↓
CVC-ClinicDB [40] MedSAM h50_m70 h70_m70 h70_m95 h80_m95
Method
Params↓ FLOPs↓ 89.7M 35.8M 30.1M 24.9M 23.2M
926.5G 377.0G 283.4G 240.9G 224.6G
0.9264 0.9601 0.9564 0.9523 0.9507
0.5477 0.6603 0.6409 0.6194 0.5951
13.77 2.64 3.85 6.30 4.46
BUSI [43]
MedSAM h50_m70 h70_m70 h70_m95 h80_m95
89.7M 35.8M 30.1M 24.9M 23.2M
926.5G 377.0G 283.4G 240.9G 224.6G
0.9334 0.9357 0.9336 0.9298 0.9316
0.4989 0.5048 0.4913 0.4703 0.4833
7.30 7.35 7.82 9.43 8.58
ISIC2018 [44, 45]
MedSAM h50_m70 h70_m70 h70_m95 h80_m95
89.7M 35.8M 30.1M 24.9M 23.2M
926.5G 377.0G 283.4G 240.9G 224.6G
0.9314 0.9525 0.9525 0.9530 0.9526
0.2401 0.4806 0.4732 0.4753 0.4708
7.47 6.12 5.57 5.86 5.94
Table 3: Component ablation on macro-average BF1 across the three polyp benchmarks. One-time columns are reported without post-pruning fine-tuning. The sequential column uses the same postpruning fine-tuning for all variants. ∆ is the difference from the full method at the same compression level. Method
h40_m30 BF1 ∆
h50_m70 BF1 ∆
h70_m70 BF1 ∆
h70_m95 Seq. h84_m95 BF1 ∆ BF1 ∆
Full MedCore 0.5162 – 0.4078 – 0.3144 – 0.4050 – 0.6462 – w/o Boundary Fisher 0.5035 -0.013 0.3724 -0.035 0.3096 -0.005 0.2781 -0.127 0.6304 -0.016 w/o reset 0.5149 -0.001 0.3967 -0.011 0.3628 +0.048 0.3227 -0.082 0.6279 -0.018 w/o variance 0.5221 +0.006 0.3997 -0.008 0.3812 +0.067 0.3249 -0.080 0.6462 0.000
Boundary-aware Fisher is the most direct boundary-preserving component. Removing it hurts BF1, and the effect is strongest at the most aggressive one-time setting h70_m95. The reset-to-SAM score has a different pattern. It has a small effect at moderate compression, but it becomes important when the pruning budget is severe. This supports our interpretation of reset scoring as an aggressivecompression safeguard. The distribution variance term follows a similar pattern. It protects structures that are unstable across datasets, and this becomes more useful when few structures remain. The ablation suggests a hierarchy. Boundary Fisher is the strongest component for direct contour protection. Reset-to-SAM and variance aggregation mainly act as safeguards under aggressive compression. This hierarchy is consistent with the design of MedCore. We use boundary Fisher to protect boundary-core structures, and we use reset-to-SAM to protect medical-core structures that were created by adaptation. 5.2
Boundary Leverage Reveals a Head-Fragile Regime
We test the empirical prediction in Definition 3.2 using a 7×7 sweep over h ∈ {0.3, . . . , 0.9} and m ∈ {0.3, 0.5, 0.7, 0.8, 0.85, 0.9, 0.95}, with 384 calibration images (128 each from Kvasir, ClinicDB, ColonDB). For each adjacent head step GH (i, j):(hi , mj )→(hi+1 , mj ) and MLP step GM (i, j):(hi , mj )→(hi , mj+1 ), we compute the 95th-percentile version of Eq. (3) on the boundary band, skipping steps with zero parameter reduction. Table 4 shows that head-pruning steps have substantially larger boundary leverage: median λbd 95 is 3.961 for heads versus 1.372 for MLPs (ratio 2.887), and head wins in all 15 valid paired comparisons. bd This supports the empirical law Median(λbd G | G ∈ H) > Median(λG | G ∈ M) for MedSAM. The BF1 and HD95 rows confirm at the final-mask level that logit-level boundary leverage translates to actual metric damage: head pruning incurs 2.4–2.6× greater BF1 and HD95 degradation per one percent parameter reduction. We do not claim heads affect only boundary pixels—their full-image 8
Table 4: Step-level boundary leverage analysis on the head-MLP pruning sweep. The logit-level boundary leverage result agrees with metric-level BF1 and HD95 damage densities. Quantity
Head steps
MLP steps
Head/MLP ratio
Median λbd 95
3.961 0.007692 0.025406
1.372 0.002951 0.010445
2.887 2.607 2.432
Valid steps Paired win rate
35
21 15/15 = 1.000
–
BF1 non-monotonicity at fixed head sparsity h = 0.30 Boundary F1 (no fine-tuning)
0.550 0.525
0.525
Boundary F1 Param. reduction
0.515
59.5%59.5%
0.500 0.475
53.7%
0.450
0.432
0.425
41.0%
0.400 0.375 0.350
trough m=0.80
28.4%
0.30
0.445
59.5%
0.456
+0.069 recovery
60 50 40
0.376
30
Param-reduction plateau (min-retention active)
0.50 0.70 0.80 0.85 MLP sparsity m (head sparsity h = 0.30)
70
0.95
Parameter reduction (%)
BF1 damage density HD95 damage density
20
Figure 3: BF1 non-monotonicity at fixed head sparsity h = 0.30 (no fine-tuning). BF1 drops to a trough of 0.376 at m = 0.80, then recovers to 0.456 at m = 0.95 along the 59.5% parameterreduction plateau where the min-retention constraint is active.
perturbation ratio is also high. Our narrower claim is that MedSAM heads are high-boundary-leverage components, making aggressive head pruning risky for boundary-sensitive segmentation. 5.3
Head-MLP Compression Landscape
The boundary leverage result above explains a non-trivial compression landscape: equal parameter reductions can produce markedly different BF1 depending on the head/MLP allocation. Near 60% reduction, h30_m80 yields BF1 0.376 while h50_m70 yields 0.408 (no fine-tuning)—parameter count alone does not determine boundary quality. Figure 3 shows one slice at fixed h = 0.30: BF1 first drops as m increases, then recovers once the effective parameter count saturates at the min-retention plateau. This non-monotonic behavior confirms that compression is not a scalar sparsity problem—both block allocation and structure type matter. This result does not mean that extreme sparsity is always better. It means that boundary-aware allocation can matter as much as the nominal sparsity level. Together with Table 4, it supports a simple practical rule: head pruning should be conservative, and most of the compression should come from lower-leverage MLP structures unless a calibration analysis shows otherwise.
6
Conclusion
We introduced MedCore, a structured pruning framework for MedSAM. MedCore treats pruning as medical-core extraction. It preserves structures that are important after SAM-to-MedSAM adaptation through a reset-to-SAM intervention, and it preserves contour-critical structures through boundary9
aware Fisher estimation. We also introduced boundary leverage as a geometric view of compressioninduced boundary damage. This view explains why a compressed MedSAM can keep high Dice while losing boundary fidelity, and it explains why attention heads require conservative pruning in our setting. Experiments show that MedCore can greatly reduce parameter count and FLOPs while keeping strong region and boundary performance across medical segmentation datasets. We note that MedCore inherits MedSAM’s underlying capabilities and biases; clinical deployment therefore requires task-specific validation and prospective safety evaluation.
References [1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv preprint arXiv:2304.02643, 2023. [2] Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15:654, 2024. [3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929. [4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. [5] Kaidong Zhang and Dong Liu. Customized segment anything model for medical image segmentation, 2023. [6] Jun Cheng, Jieneng Ye, Zhuoyang Deng, Jian Chen, Tianbin Li, Haoyu Wang, Yan Su, Ziyan Huang, Jiao Chen, Lei Jiang, Hui Sun, Jingwei He, Shaoting Zhang, and Min Zhu. Sam-med2d, 2023. [7] Junde Wu, Wei Ji, Yueming Liu, Huazhu Fu, Min Xu, Yanwu Xu, and Yanwu Jin. Medical sam adapter: Adapting segment anything model for medical image segmentation, 2023. [8] Yuheng Li, Mingzhe Hu, and Xiaofeng Yang. Polyp-SAM: Transfer SAM for polyp segmentation. In Medical Imaging 2024: Image Processing, volume 12927, page 1292735. SPIE, 2024. doi: 10.1117/12.3006809. URL https://arxiv.org/abs/2305.00293. [9] Xiaobao Wei, Jiajun Cao, Yizhu Jin, Ming Lu, Guangyu Wang, and Shanghang Zhang. IMedSAM: Implicit medical image segmentation with segment anything. In European Conference on Computer Vision, pages 90–107. Springer, 2024. doi: 10.1007/978-3-031-72684-2_6. URL https://arxiv.org/abs/2311.17081. [10] Xuanbo Zhao, Cheng Wang, Huaxing Xu, Hong Zhou, Zekuan Yu, Tao Chen, Xiaoling Wei, and Rongjun Zhang. LDFSAM: Localization distillation-enhanced feature prompting SAM for medical image segmentation. Journal of Imaging, 12(2):74, 2026. doi: 10.3390/jimaging12020074. URL https://www.mdpi.com/2313-433X/12/2/74. [11] Jun Ma, Zongxin Yang, Sumin Kim, Bihui Chen, Mohammed Baharoon, Adibvafa Fallahpour, Reza Asakereh, Hongwei Lyu, and Bo Wang. MedSAM2: Segment anything in 3d medical images and videos, 2025. URL https://arxiv.org/abs/2504.03600. [12] Han Zhong, Jiatian Zhang, and Lingxiao Zhao. MedSAM/MedSAM2 feature fusion: Enhancing nnUNet for 2d TOF-MRA brain vessel segmentation. Journal of Imaging, 11(6):202, 2025. doi: 10.3390/jimaging11060202. URL https://www.mdpi.com/2313-433X/11/6/202. [13] Zhaoting Mu, Yu Ning, Yueqi Zhao, Rong Zhang, Fuad Md Mubtasim, and Hailong Ning. SAM2-PolypNet: SAM2 with adaptive context enhancement model for polyp segmentation. Biomedical Signal Processing and Control, 2026. URL https://www.sciencedirect.com/ science/article/pii/S1746809426009353. In press. 10
[14] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything, 2023. [15] Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, Raghuraman Krishnamoorthi, and Vikas Chandra. Efficientsam: Leveraged masked image pretraining for efficient segment anything, 2023. [16] Chong Zhou, Xiangtai Li, Chen Change Loy, and Bo Dai. Edgesam: Prompt-in-the-loop distillation for sam. International Journal of Computer Vision, 133(12):8452–8468, 2025. [17] Zigeng Chen, Gongfan Fang, Xinyin Ma, and Xinchao Wang. Slimsam: 0.1% data makes segment anything slim, 2024. [18] Haisheng Lu, Yujie Fu, Fan Zhang, and Le Zhang. Efficient quantization-aware training on segment anything model in medical images and its deployment. In Medical Image Segmentation Foundation Models. CVPR 2024 Challenge: Segment Anything in Medical Images on Laptop, pages 137–150. Springer Nature Switzerland, 2025. [19] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, pages 234–241, 2015. [20] Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. nnu-net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18:203–211, 2021. [21] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. In Proceedings of the European Conference on Computer Vision Workshops, 2022. [22] Md Mostafijur Rahman, Mustafa Munir, and Radu Marculescu. Emcad: Efficient multi-scale convolutional attention decoding for medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11769–11779, 2024. [23] Md Mostafijur Rahman and Radu Marculescu. Mk-unet: Multi-kernel lightweight cnn for medical image segmentation, 2025. [24] Qing Feng, Shahzad Ahmed, Yueming Zhang, Lan He, and Muhammad Yaqub. HSSAMNet: Hyper-scale shifted aggregation network for precise colorectal polyp segmentation in endoscopic images. Scientific Reports, 15:38146, 2025. doi: 10.1038/s41598-025-21954-y. URL https://www.nature.com/articles/s41598-025-21954-y. [25] Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, volume 2, 1989. [26] Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, volume 5, 1993. [27] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016. [28] Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems, volume 33, pages 20378–20389, 2020. [29] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pages 7021–7032, 2021. 11
[30] Huanrui Yang, Hongxu Yin, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian-aware saliency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18547–18557, 2023. [31] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, volume 32, 2019. [32] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, 2019. [33] Hoel Kervadec, Jalal Bouchtiba, Christian Desrosiers, Eric Granger, Jose Dolz, and Ismail Ben Ayed. Boundary loss for highly unbalanced segmentation. In Medical Imaging with Deep Learning, pages 285–296, 2019. [34] Davood Karimi and Septimiu E. Salcudean. Reducing the hausdorff distance in medical image segmentation with convolutional neural networks. IEEE Transactions on Medical Imaging, 39 (2):499–513, 2020. [35] Suprosanna Shit, Johannes C. Paetzold, Anjany Sekuboyina, Ivan Ezhov, Andreas Unger, Andrey Zhylka, Josien P. W. Pluim, Ulrich Bauer, and Bjoern H. Menze. cldice: A novel topology-preserving loss function for tubular structure segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16560–16569, 2021. [36] Hao Shao, Yang Zhang, and Qibin Hou. Polyper: Boundary sensitive polyp segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4731–4739, 2024. doi: 10.1609/aaai.v38i5.28274. URL https://ojs.aaai.org/index.php/AAAI/ article/view/28274. [37] Qin Li, Tianchi Zhang, Parvej Md Mosharaf, and Jing Zhang. BMFA-Net: Boundary constraint multi-level feature aggregation framework for precise polyp segmentation. Applied Sciences, 14(10):4063, 2024. doi: 10.3390/app14104063. URL https://www.mdpi.com/2076-3417/ 14/10/4063. [38] Dingzhou Liu, Hongmin Deng, Zhengwei Huang, and Jinghao Fu. FCA-Net: Fully contextaware feature aggregation network for medical segmentation. Biomedical Signal Processing and Control, 91:106004, 2024. doi: 10.1016/j.bspc.2024.106004. URL https://www. sciencedirect.com/science/article/pii/S1746809424000624. [39] Tao Tong, Wen Zhang, and Wanni Zu. BEGA-UNet: Boundary-explicit guided attention U-Net with multi-scale feature aggregation for colonoscopic polyp segmentation. medRxiv preprint, 2026. URL https://www.medrxiv.org/content/10.64898/2026.03.04.26347608v2. [40] Jorge Bernal, F. Javier Sánchez, Gloria Fernández-Esparrach, Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics, 43: 99–111, 2015. ISSN 0895-6111. doi: https://doi.org/10.1016/j.compmedimag.2015.02.007. URL https://www.sciencedirect.com/science/article/pii/S0895611115000567. [41] David Vázquez, Jorge Bernal, F. Javier Sánchez, Gloria Fernández-Esparrach, Antonio M. López, Adriana Romero, Michal Drozdzal, and Aaron Courville. A benchmark for endoluminal scene segmentation of colonoscopy images. Journal of Healthcare Engineering, 2017(1): 4037190, 2017. doi: https://doi.org/10.1155/2017/4037190. URL https://onlinelibrary. wiley.com/doi/abs/10.1155/2017/4037190. [42] Debesh Jha, Pia H. Smedsrud, Michael A. Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D. Johansen. Kvasir-seg: A segmented polyp dataset. In Yong Man Ro, Wen-Huang Cheng, Junmo Kim, Wei-Ta Chu, Peng Cui, Jung-Woo Choi, Min-Chun Hu, and Wesley De Neve, editors, MultiMedia Modeling, pages 451–462, Cham, 2020. Springer International Publishing. ISBN 978-3-030-37734-2. 12
[43] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in Brief, 28:104863, 2020. ISSN 2352-3409. doi: https://doi.org/ 10.1016/j.dib.2019.104863. URL https://www.sciencedirect.com/science/article/ pii/S2352340919312181. [44] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1), August 2018. ISSN 2052-4463. doi: 10.1038/sdata.2018.161. URL http://dx.doi.org/10. 1038/sdata.2018.161. [45] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic), 2019. URL https://arxiv.org/abs/ 1902.03368. [46] Yawei Li, Kamil Adamczewski, Wen Li, Shuhang Gu, Radu Timofte, and Luc Van Gool. Revisiting random channel pruning for neural network compression, 2022. URL https: //arxiv.org/abs/2205.05676. [47] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks, 2015. URL https://arxiv.org/abs/1506.02626. [48] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one?, 2019. URL https://arxiv.org/abs/1905.10650. [49] Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning, 2018. URL https://arxiv.org/abs/1801.05787.
Appendix The appendix contains proofs, implementation details, and additional information about the boundary leverage analysis. We keep the main text focused on the method and key empirical findings.
A
Proofs
A.1
Proof of Theorem 3.3
Proof. Fix a point x ∈ Γθ . By definition of the old boundary, sθ (x) = 0. Since Assumption 3.1 gives ∥∇sθ (x)∥2 > 0, the unit normal direction of the level set at x is n(x) =
∇sθ (x) . ∥∇sθ (x)∥2
The new logit map after compression is se(x) = sθ (x) + δG (x). We seek the new boundary point on the normal line through x. We write it as e = x + uG (x)n(x), x e lies on the new boundary, it satisfies where uG (x) is the signed normal displacement. Because x 0 = se(e x) = sθ (e x) + δG (e x). We expand both terms around x. The first-order Taylor expansion gives sθ (e x) = sθ (x) + uG (x)∇sθ (x)⊤ n(x) + O(uG (x)2 ). Since sθ (x) = 0 and ∇sθ (x)⊤ n(x) = ∥∇sθ (x)∥2 , this becomes sθ (e x) = uG (x)∥∇sθ (x)∥2 + O(uG (x)2 ). The perturbation term satisfies δG (e x) = δG (x) + O(|uG (x)|∥δG ∥C 1 ). 13
Substituting the two expansions into the boundary equation gives 0 = uG (x)∥∇sθ (x)∥2 + δG (x) + O(uG (x)2 ) + O(|uG (x)|∥δG ∥C 1 ). For small perturbations, the implicit function theorem guarantees that uG (x) is also small and has the same order as δG (x). Therefore the two remainder terms are second order in ∥δG ∥C 1 . Solving the first-order part gives δG (x) uG (x) = − + O(∥δG ∥2C 1 ). ∥∇sθ (x)∥2 This is Eq. (4). A.2
Proof of Proposition 3.4
Proof. Let the current allocation be (cH , cM ) and keep the total compression fixed at C = cH + cM . We move a small amount η > 0 of compression budget from heads to MLPs. The new allocation is (cH − η, cM + η). The first-order Taylor expansion of Ebd at (cH , cM ) gives Ebd (cH − η, cM + η) = Ebd (cH , cM ) − η
∂Ebd ∂Ebd +η + o(η). ∂cH ∂cM
The first-order change in boundary error is therefore ∂Ebd ∂Ebd ∆Ebd = η − + o(η). ∂cM ∂cH ∂Ebd bd If ∂E ∂cH is larger than ∂cM , then the coefficient of η is negative. For sufficiently small positive η, the first-order term dominates the o(η) remainder, so ∆Ebd < 0. The boundary error decreases while the total compression remains C.
B
Algorithmic Summary
Algorithm 1 summarizes MedCore. The algorithm separates the scoring stage from the recovery stage. The reset-to-SAM operation is used only to compute importance. The final compressed model removes groups or keeps them active. It does not reset kept groups to SAM weights.
C
Additional Boundary Leverage Details
We compute boundary leverage on 384 calibration images, with 128 images from each polyp dataset. The sweep has seven head sparsity levels and seven MLP sparsity levels. We compute adjacent head steps and adjacent MLP steps. Steps with zero additional parameter reduction are undefined and are omitted. For a head step, we fix the MLP sparsity and increase head sparsity. For an MLP step, we fix head sparsity and increase MLP sparsity. We use the same parameter-reduction unit for both families. This normalization is important because a 10% increase in head sparsity and a 10% increase in MLP sparsity do not remove the same number of parameters. The boundary band width is tested at τ = 3 and τ = 5 pixels. Both choices give the same qualitative conclusion in our experiments. The main text reports the 95th-percentile boundary leverage because HD95 is also a high-percentile boundary metric. The mean boundary leverage gives the same direction but is less aligned with HD95. The summary values are: BLR95 = 2.887, RLR = 3.064, BSR = 0.942, paired win rate = 1.000, and median paired difference = 2.603. BSR is the ratio between boundary leverage ratio and region leverage ratio. Since BSR is slightly below one, we do not claim that head pruning is boundaryspecific. We claim that head pruning is high-boundary-leverage. This is the statement needed to explain boundary fragility. 14
Algorithm 1 MedCore pruning Require: MedSAM checkpoint θ M , SAM checkpoint θ S , frozen prompt encoder ϕ, calibration sets (r) {Dcal }R r=1 , structured groups G, target budget B Ensure: Pruned and recovered MedSAM checkpoint 1: Initialize all group masks as active. 2: for each sub-distribution r do M 3: Compute boundary-aware MedSAM Fisher Fr,i using Eq. (10). 4: end for 5: Compute SAM Fisher FiS on the same calibration data. 6: for each group g ∈ G do M b zero,(r) 7: Estimate ∆ using Fr,i . g reset,(r) b 8: Estimate ∆g using Eq. (12). b (r) b dist 9: Compute the fused score Q g and the distribution-aware score Qg . 10: Compute the cost-normalized priority Pg using Eq. (8). 11: end for 12: Allocate head-pruning budgets across blocks with block sensitivity. 13: Prune attention heads with the lowest priorities under the block budgets. 14: Run a short recovery fine-tuning stage. 15: Recompute or reuse priorities for MLP connection groups in the head-pruned model. 16: Prune MLP connection groups until the target budget is reached. 17: Run post-pruning recovery fine-tuning with Eq. (14). 18: Physically remove pruned structures and return the compact checkpoint.
D
Implementation Details
We use the official MedSAM ViT-B checkpoint as θ M and the SAM ViT-B checkpoint as θ S . The prompt encoder remains frozen. Fisher estimation uses calibration images and the same bounding-box prompt protocol as MedSAM. We compute boundary maps by morphological dilation minus erosion of the binary mask. We normalize losses so that the boundary-weighted BCE term does not dominate the Dice term at the beginning of recovery. For attention-head pruning, each group contains the corresponding Q, K, V, and output-projection parameters associated with one head. For MLP pruning, each group contains a connection group in the feed-forward block. We use physical removal after mask selection so that parameter count and FLOPs reflect the actual compact model. The main one-time configurations protect the deepest encoder blocks because they have high Fisher sensitivity and connect directly to the mask decoder. The sequential extreme setting is different. It starts from the h70_m95 checkpoint and then performs a conservative second pruning pass on the previously protected deepest blocks. This setting is used only to test the limit of compression.
E
Comparison with Structured Pruning Baselines
The main paper compares MedCore against external efficient SAM variants and medical segmentation backbones (Table 1), where each method has a different architecture and training pipeline. This appendix provides a controlled comparison that isolates the contribution of the scoring criterion itself. All baselines below operate on the same MedSAM ViT-B encoder, share identical structured groups (attention heads and MLP connection groups), and use the same target sparsity levels as MedCore. To remove the confound of recovery quality, post-pruning fine-tuning is disabled for all methods in this comparison, including MedCore. The reported numbers therefore reflect the direct effect of each importance score on the compressed model, before any recovery is applied. Baselines. We compare against five structured pruning criteria: (i) Random Pruning [46], which selects groups uniformly at random within each block; (ii) Magnitude Pruning [47, 27], which ranks groups by the ℓ2 norm of their parameters; (iii) Zero-only Fisher [48], which uses only the zerointervention score ∆zero from Eq. (5) and discards the reset-to-SAM signal; (iv) Vanilla Fisher [49], 15
which estimates Fisher information using the standard Dice+BCE loss without boundary weighting, Cross-Fisher, or distribution-aware aggregation; and (v) Original Medical Core, an internal earlier version of our method that retains the dual zero/reset intervention but drops the boundary-aware Fisher, Cross-Fisher, multi-dataset aggregation, and block-sensitivity allocation. Results. Table 5 reports macro-averaged metrics on the three polyp benchmarks across four compression levels. Table 5: Comparison with structured pruning baselines on macro-averaged polyp segmentation metrics. All methods operate on MedSAM ViT-B without post-pruning fine-tuning. Parameter and FLOPs reductions are relative to the MedSAM ViT-B baseline (89.7M parameters, 926.5G FLOPs). Best results in each compression level are in bold. Compression
Method
Param↓ FLOPs↓
Dice↑
IoU↑
BF1↑
HD95↓
Low: h40_m30
Random Pruning [46] Magnitude Pruning [47, 27] Zero-only Fisher [48] Vanilla Fisher [49] Original Medical Core MedCore (ours)
31.7% 31.7% 31.7% 31.7% 31.7% 31.7%
34.3% 34.3% 34.3% 34.3% 34.3% 33.0%
0.9076 0.8278 0.9137 0.9113 0.9123 0.9191
0.8436 0.7228 0.8541 0.8512 0.8519 0.8615
0.4598 0.2081 0.5035 0.4972 0.4988 0.5162
23.12 41.63 22.13 23.55 23.50 20.30
Mid: h50_m70
Random Pruning [46] Magnitude Pruning [47, 27] Zero-only Fisher [48] Vanilla Fisher [49] Original Medical Core MedCore (ours)
60.0% 60.0% 60.0% 60.0% 60.0% 60.0%
58.4% 58.4% 58.4% 58.4% 58.4% 58.4%
0.8394 0.8237 0.8881 0.8854 0.8870 0.8908
0.7391 0.7135 0.8124 0.8080 0.8106 0.8179
0.2272 0.1665 0.3911 0.3859 0.3879 0.4078
40.16 43.60 29.10 29.44 28.69 27.07
High: h70_m70
Random Pruning [46] Magnitude Pruning [47, 27] Zero-only Fisher [48] Vanilla Fisher [49] Original Medical Core MedCore (ours)
66.4% 66.4% 66.4% 66.4% 66.4% 66.4%
68.1% 68.1% 68.1% 68.1% 68.1% 69.0%
0.8371 0.8200 0.8555 0.8555 0.8558 0.8738
0.7344 0.7101 0.7630 0.7634 0.7629 0.7899
0.2121 0.1761 0.2784 0.2803 0.2799 0.3144
41.59 44.74 39.26 38.95 36.25 31.35
Random Pruning [46] Magnitude Pruning [47, 27] Zero-only Fisher [48] Extreme: h84_m95 Vanilla Fisher [49] Original Medical Core MedCore (ours)
74.2% 74.2% 74.2% 74.2% 74.2% 74.2%
75.8% 75.8% 75.8% 75.8% 75.8% 75.8%
0.8180 0.8161 0.8312 0.8338 0.8346 0.8492
0.7046 0.7033 0.7258 0.7298 0.7312 0.7542
0.1468 0.1643 0.2003 0.2048 0.2046 0.2539
43.95 45.93 42.18 42.39 42.46 41.92
Discussion. Three observations follow from Table 5, and each is consistent with the conceptual claims made in the main paper. First, Magnitude Pruning collapses on boundary metrics across all compression levels (BF1 drops to 0.2081 even at the low compression level h40_m30, and HD95 exceeds 41). This is the empirical signature predicted by Theorem 3.3: parameter magnitude is uninformative about boundary risk because boundary displacement is determined by the logit perturbation at the boundary, not by the size of the removed weights. A small-magnitude attention head can carry substantial boundary leverage, and Magnitude Pruning is structurally blind to this. Second, both Zero-only Fisher and Vanilla Fisher noticeably outperform Random and Magnitude Pruning at every level, confirming that a proper second-order importance score is necessary. However, their boundary metrics fall behind MedCore by a margin that widens as compression intensifies—for instance, at the Extreme level, MedCore improves BF1 by +24.1% over the strongest baseline (Original Medical Core). This trend matches the component ablation in Section 5.1: the boundaryaware Fisher provides direct contour protection, while the reset-to-SAM and distribution-aware aggregation act as safeguards under aggressive compression. When all three components are absent, even Fisher-based methods cannot localize medically critical structures under heavy sparsity. 16
Third, the monotone widening of MedCore’s margin from +3.5% BF1 at h40_m30 to +24.1% at h84_m95 (relative to the strongest baseline at each level) provides indirect support for the boundary leverage principle as a design lens. As more groups are removed, the remaining structures sit closer to the boundary-critical core, and the cost of misidentifying these structures grows superlinearly. Scoring criteria that lack a boundary-geometry signal incur this cost; MedCore avoids it by construction. This comparison is not a substitute for the main results in Table 1 of the main paper, which compares against architecturally distinct efficient and medical segmentation models. Rather, it isolates a single design question: among structured pruning criteria operating on the same MedSAM encoder, does boundary-aware adaptation-aware scoring improve compression-time decisions? The evidence in Table 5 suggests that it does, and the improvement is most pronounced precisely where compression is hardest.
F
Limitations
MedCore uses calibration data to estimate Fisher information and boundary leverage. If this calibration set is small or biased, the score may under-protect rare structures. The reset-to-SAM score also assumes that the SAM and MedSAM checkpoints are aligned parameter by parameter. This is true for the standard MedSAM adaptation path, but it may not hold for methods that change the architecture. The boundary leverage theorem is local. It explains first-order boundary displacement under small logit perturbations. Very aggressive pruning can create non-local effects, such as disconnected masks or missing components, that are not fully captured by the first-order formula. We therefore view boundary leverage as a useful diagnostic and design principle, not as a complete replacement for final BF1 and HD95 evaluation. Finally, our main experiments focus on 2D prompt-driven segmentation with bounding boxes. Extension to 3D medical segmentation and other prompt types requires additional validation. The same concepts can be applied, but the boundary band, distance metric, and computational cost must be adjusted for 3D volumes.
17