arXiv:2606.28516v1 [cs.CV] 26 Jun 2026
CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection Md Irtiza Hossain
Humaira Ayesha
Junaid Ahmed Sifat
Brac University [email protected]
Brac University [email protected]
Brac University [email protected]
Abstract—We present CLEAR-MoE, a four-phase posttraining pipeline that converts a frozen pretrained vision transformer (ViT) into a sparse mixture-of-experts (MoE) model without updating backbone weights. The pipeline (i) scores FFN layers by sparsity, clusterability, and output sensitivity; (ii) decomposes selected layers into a shared low-rank SVD basis plus per-cluster residual experts (k-means); (iii) fits lightweight routers supervised by cluster labels; and (iv) dispatches tokens via pluggable CUDA backends. On Imagenette with DeiT-Small, CLEAR-MoE retains 99.9% of dense accuracy (86.70 ± 0.02% vs. 86.73%). Our ablations isolate a consistent empirical finding: the shared SVD basis is the dominant accuracy-preserving component. Random routing, learned routing, and three router architectures yield numerically similar results spanning at most 0.06 pp (86.62– 86.68%), and accuracy remains stable across SVD rank, expert count E ∈ {2, . . . , 8}, calibration size N ∈ {50, . . . , 500}, and random seed. This finding generalizes across five ViT backbones (DeiT-T/S/B, ViT-S/B, 5.7M–86.6M parameters), with |∆|≤0.10 pp across all configurations. On a GTX 960, routing and scatter-gather overhead make CLEAR-MoE FFN 1.3–1.7× slower than dense. A dispatch microbenchmark indicates that routing (AI = 1.9 FLOPs/B) is an order of magnitude more memory-bound than expert GEMMs (AI = 22.7), identifying fused dispatch kernels as a plausible optimization target. Index Terms—mixture of experts, vision transformer, posttraining extraction, shared basis decomposition, calibrationdriven scoring, dispatch strategies, roofline analysis
I. I NTRODUCTION Vision transformers (ViTs) [1] devote roughly two-thirds of inference FLOPs to feed-forward network (FFN) sublayers that execute the same MLP computation for every token, regardless of semantic content [2]. Sparse mixtureof-experts (MoE) architectures [3] address this by routing each token to a subset of specialized sub-networks, reducing active compute proportional to k/E. However, existing vision MoE models [2], [4] require training from scratch, which is a resource barrier for practitioners who already hold a pretrained checkpoint. Post-training expert extraction converts a dense FFN into experts without retraining [5]–[7]. Despite recent progress, three gaps remain: (1) most methods convert all FFN layers, ignoring sensitivity variation across depth; (2) disjoint expert assignment discards shared low-level structure, destabilizing downstream transfer; (3) speedup claims are reported in MACs rather than wall-clock latency on real hardware.
CLEAR-MoE addresses all three with the following contributions: Calibration-driven layer scoring: composite score S(l) = 0.4 sparsity + 0.4 clusterability − 0.2 sensitivity selects FFN layers that cluster well and tolerate perturbation, automatically excluding high-sensitivity layers (e.g., block 0, sensitivity = 0.946). • Shared-basis decomposition: fc2 is decomposed into a shared truncated-SVD basis plus per-cluster residuals; fc1 is shared identically across all experts. This preserves common visual structure while allowing per-cluster specialization. • Hardware-transparent latency study: all timing on a single GTX 960 (4 GB, 112 GB/s) with cuda.synchronize()-fenced p50 reporting (100 forward passes), indicating that routing overhead (not expert arithmetic) is a likely bottleneck. • Comprehensive ablation: decomposition (D0–D8), layer selection (L0–L10), dispatch backends, expert count, SVD rank, router architecture, calibration size, and random seed, establishing which design choices actually matter. •
Scope. All measured results use a single GTX 960. No claims are made about data-center GPUs, distributed setups, or larger datasets. Multi-device throughput projections are modelled analytically using PCIe Gen3×16 bandwidth; no NCCL runs were performed. II. R ELATED W ORK Post-training expert extraction. MoEfication [5] first showed that clustering neuron activation patterns partitions a pretrained FFN into experts without retraining. D2DMoE [6] extended this to dynamic-k routing, achieving 30% latency reduction on an A100 for ViT-B/ImageNet-1K. Berisha et al. [7] used variance-based neuron grouping to recover 98% of dense performance with 36.3% MAC reduction for DeiTB. CLEAR-MoE differs by (a) selecting layers via a composite sensitivity-aware score, (b) preserving shared structure via SVD decomposition, and (c) providing a full wall-clock dispatch study on consumer hardware. Table I positions these methods side-by-side. Sparse Upcycling [8] converts dense language-model checkpoints to MoE by cloning FFN weights and training a load-balancing router via continued pre-training;
unlike CLEAR-MoE, backbone weights are updated, requiring gradient access and sufficient unlabelled data. Vision MoE training. V-MoE [2] demonstrated that sparse patch routing matches dense quality at ∼50% active compute for 15B-parameter ViTs. AdaMV-MoE [4] showed late transformer layers benefit most from expertization, a heuristic our composite score can override when sensitivity signals contradict it. DynamicViT [9] and A-ViT [10] achieve tokenlevel sparsification via halting gates, complementary to our FFN-level approach. MoE runtime systems. TUTEL [11] achieves 3.11× speedup via 2D all-to-all on 128 GPUs. Brainstorm [12] shows profile-guided dispatch yields 5× speedup over DeepSpeed for SwinV2-MoE. Lancet [13] reduces non-overlapping communication by 77% via whole-graph overlap. These systems target high-bandwidth multi-GPU clusters; CLEAR-MoE studies the single-consumer-GPU regime where bandwidth is 18× lower.
TABLE I P OST- TRAINING M O E EXTRACTION : METHOD COMPARISON . † = MAC REDUCTION ( NOT WALL - CLOCK ). ‡ = MINIMAL FINE - TUNING REPORTED BY ORIGINAL PAPER . CLEAR-M O E: ZERO BACKBONE WEIGHT UPDATES . Method
Backbone
MoEfication [5] ViT
HW
Acc.
N/A
∼99%
Latency Structure
N/A Disjoint k-means D2DMoE [6]‡ ViT-B/16 A100 ∼99% −30% Dynamic-k disj. Berisha [7]‡ DeiT-B N/A 98% −36.3%† Var.-based disj. CLEAR-MoE DeiT/ViT GTX 960 99.9% +1.3–1.7× Shared + (5) res. D2DMoE latency on A100 (2 TB/s); CLEAR-MoE on GTX 960 (112 GB/s). Bandwidth 18× lower.
III. M ETHOD CLEAR-MoE converts a frozen pretrained ViT into a selectively expertised model through four sequential phases (Fig. 1). No backbone weights are updated. A. Phase 1: Calibration Pass Forward Ncal representative images through the frozen model, recording pre-FFN activation tensors at each of the L selected layers via PyTorch hooks. For DeiT-Small (d=384, Ncal =200, 197 tokens/image), this produces a [39,400, 384] activation matrix per layer. B. Phase 2: Layer Scoring and Selection Each FFN layer l receives a composite score: S(l) = 0.4 sparsity(l) + 0.4 clusterability(l) − 0.2 sensitivity(l) (1) where sparsity is the fraction of activation magnitudes below 0.01; clusterability is the Silhouette score of k-means with k=E on the calibration activations; and sensitivity is the normalized logit change when the layer’s FFN output is zeroed. Sensitivity is subtracted because high sensitivity implies high degradation risk if expertized. The top-k=L/2 layers by S are selected; the remaining layers retain their original dense FFN. Table II reports per-layer scores for DeiT-Small (200-image calibration); block 0 ranks last due to sensitivity = 0.946. C. Phase 3: Expert Extraction For each selected layer, fc2 (W2 ∈ Rd×dffn ) is decomposed via truncated SVD; fc1 is shared identically across all experts: y=
·z + W {z } | shared
SVD-r approx.
Weres ∗ |{z}
·z, z = GELU(fc1(x)) (2)
Fig. 1. CLEAR-MoE pipeline. A frozen pretrained ViT yields calibration activations; scored FFN layers are decomposed into a shared SVD basis plus per-cluster residual experts; lightweight routers are fitted on k-means labels; inference uses pluggable CUDA dispatch backends.
cluster residual
Wshared = Ur Σr Vr⊤ retains the top-r singular values (default r = 50% of matrix rank, r=d/2=192 for DeiT-Small where W2 ∈R384×1536 has maximum rank 384; materialised as a dense matrix before inference, with no factorised compute reduction). k-means with k=E partitions calibration tokens into E clusters; the per-cluster residual is Weres = (W2 − Wshared ) ·
se , where se scales by the ratio of cluster-mean to global-mean activation norm. Because se is a scalar, all residual experts share the same directional component (W2 −Wshared ), differing only in magnitude; a misrouted token therefore receives a different scaling but the same directional correction. During inference, z is computed once and reused for both the shared
TABLE II P ER - LAYER COMPOSITE SCORES , D EI T-S MALL (200- IMAGE CALIBRATION ). C OMPOSITE = 0.4· SP + 0.4· CL − 0.2· SE . B OLD = SELECTED ( TOP -k=6).
Block blocks.1 blocks.4 blocks.5 blocks.3 blocks.6 blocks.2 blocks.7 blocks.11 blocks.9 blocks.8 blocks.10 blocks.0
Sparsity
Clusterab.
Sensitivity
Composite
0.180 0.152 0.152 0.149 0.147 0.141 0.147 0.134 0.135 0.142 0.171 0.210
0.526 0.518 0.520 0.519 0.522 0.518 0.519 0.516 0.518 0.512 0.514 0.516
0.171 0.109 0.113 0.113 0.123 0.147 0.162 0.149 0.167 0.175 0.252 0.946
0.248 0.246 0.246 0.245 0.243 0.234 0.234 0.230 0.228 0.227 0.224 0.101
and residual paths. D. Phase 4: Router Fitting A router gθ predicts the cluster assignment of each token. The k-means labels from Phase 3 provide supervised targets: router training minimizes cross-entropy between gθ (x) and cluster label c over the calibration tokens, using AdamW (lr=10−3 , 5 epochs, cosine decay). Three router architectures are evaluated: Linear (d × E parameters, 9.2K total), MLP (hidden layer d/6, 149K), and Adaptive (confidence-threshold top-1/top-2 escalation, same parameter count as Linear). E. Dispatch Backends Three CUDA backends are implemented: (1) Naive: boolean-mask loop over experts (reference). (2) Grouped: tokens are sorted by expert index; one GEMM per contiguous sub-batch. No padding; throughput stable under imbalance. (3) cuBLAS: expert sub-batches are padded to the largest and stacked into a 3D tensor dispatched to a batched GEMM call. Fastest at balanced load; degrades at high imbalance from padding waste.
TABLE III D ECOMPOSITION ABLATION (D EI T-S MALL , I MAGENETTE , E=4, LAST-k LAYERS , BATCH =8, GTX 960). ∆T OP -1 RELATIVE TO D0. D4/D5 DIFFER ONLY IN RANDOM SEED . D6 AND D8 ARE THE SAME ARCHITECTURE : D6 IS THE COLD - KERNEL FIRST DISPATCH ; D8 IS MEASURED AFTER 3 WARM - UP PASSES ( STEADY- STATE ). T HE 21 MS GAP REFLECTS CUDA JIT OVERHEAD ON FIRST RUN . ID
Sh. fc1 Sh. fc2
Router
Top-1
∆pp
ms (p50)
D0
–
Dense
86.73%
+0.00
59.6
–
D1 ✓ SVD rank-r None 86.55% −0.18 59.7 D2 ✗ Disjoint Random 66.42% −20.31 49.4 D3 ✓ Full res. None 86.73% +0.00 59.9 D4/D5 ✓ k-means res. Random 86.62–65% −0.08–11 92.2 D6/D8 ✓ k-means res. Linear 86.65% −0.08 78.8–100.3 D7 ✗ Disjoint k-means 65.22% −21.51 48.0 ✗ = disjoint (no shared fc1); ✓ = shared across all tokens.
The shared SVD basis is the dominant accuracypreserving component. D3 (shared basis + full global residual, no routing) achieves exact reconstruction: algebraically, Wshared ·z + (W2 −Wshared )·z = W2 ·z, matching D0 at 86.73%. D2 and D7 (disjoint experts, no shared fc1) collapse to 65– 66% regardless of routing strategy: disjoint expert weights estimated from 200 calibration images fail to generalize. In contrast, D4/D5 (shared basis, random routing) retain 86.62–65%, only 0.08–0.11 pp below D0; D6/D8 (shared basis, learned routing at 95% accuracy) achieve an identical −0.08 pp. Routing quality has limited measured impact once the shared basis is present. No latency reduction on GTX 960. CLEAR-MoE FFN (D4–D8) executes shared fc1+fc2 on all tokens plus residual paths that collectively process all T tokens (T /E per expert × E experts), adding approximately one full fc2-equivalent computation; total arithmetic is ≈ 1.5× dense FFN. On bandwidth-limited memory (112 GB/s), this incurs 1.3–1.7× latency overhead (78–100 ms vs. 59.6 ms). Only disjoint experts (D2/D7) are faster (48–49 ms) by eliminating the shared path entirely, at the cost of 21 pp accuracy. B. Layer-Selection Ablation (L0–L10)
IV. E XPERIMENTS Setup. DeiT-Small (22M parameters, d=384, dffn =1536, 12 blocks) evaluated on Imagenette (3,925 validation images, 10class ImageNet subset) with 200-image calibration set, E=4 experts, AdamW router training (5 epochs, lr 10−3 , cosine decay). All timing: p50 of 100 forward passes, batch size 8, GTX 960 (4 GB, 112 GB/s), cuda.synchronize()fenced. A. Decomposition Ablation (D0–D8) Table III ablates what each architectural choice contributes to accuracy and latency. Configurations D0–D8 span the design space from dense baseline to shared-basis CLEARMoE FFN, isolating the effect of the shared basis, the routing mechanism, and the expert assignment strategy.
Table IV compares all 11 layer-selection strategies on the same MoE configuration (E=4, k=6 of 12 FFN layers, composite scoring). All 11 policies span only 0.21 pp (86.62–86.83%), confirming that accuracy is policy-insensitive. The shared basis preserves quality regardless of which 6 blocks are expertized. The composite score’s practical value is principled block 0 exclusion (sensitivity = 0.946): L2 (first-k) includes block 0 and incurs the worst −0.13 pp drop. L10 avoids it, achieving 82.4 ms, comparable to L3 (last-k, 80.3 ms) while additionally excluding high-sensitivity blocks. L7 (high-sensitivity, deliberately selecting the most sensitive blocks as a stresstest baseline) achieves only −0.08 pp accuracy loss but is the slowest policy at 102.5 ms, demonstrating that layer choice affects latency more than accuracy.
TABLE IV C OMPLETE LAYER - SELECTION ABLATION (L0–L10). A LL : D EI T-S MALL , E=4, k=6 LAYERS , LINEAR ROUTER , 200- IMAGE CALIBRATION . L1 AVG OF 3 SEEDS . ∆ RELATIVE TO L0.
ID
Strategy
Layers
Top-1
∆pp
TABLE V D ISPATCH MICRO - BENCHMARK : 0% VS . 80% IMBALANCE (B=8, T =1568, E=4, GTX 960).
0% imb.
80% imb.
Backend
ms
K tok/s
ms
K tok/s
CPU Serial Naive Grouped cuBLAS
6.01 3.67 2.36 2.15
261 428 665 728
5.96 3.59 2.14 2.81
263 437 733 558
Roofline analysis
GTX 960
p50 ms
L0 Dense (no MoE) – 86.73% +0.00 57.0 L1 Random (n=3) varies 86.72±0.01% −0.01 78.5±0.8 L2 First k 0–5 86.60% −0.13 79.3 L3 Last k 6–11 86.68% −0.05 80.3 L4 Alternating (odd) 1,3,5,7,9,11 86.83% +0.10 87.2 L5 Sparsity only 0,1,3,4,5,10 86.73% +0.00 99.1 L6 Clusterability 1,2,3,5,6,7 86.73% +0.00 88.8 L7 High-sensitivity 0,1,7,8,9,10 86.65% −0.08 102.5 L8 Sp.+Cl. 0,1,4,5,6,10 86.62% −0.10 88.8 L9 Cl.−Se. 2,3,4,5,6,11 86.68% −0.05 94.0 L10 Composite (ours) 1–6 86.68% −0.05 82.4
We benchmark three CUDA backends (B=8, T =1568, E=4, GTX 960) under four imbalance levels (Table V). At balanced load, cuBLAS peaks at 728 K tok/s (2.79× CPU serial). At 80% imbalance, padding waste collapses cuBLAS to 558 K tok/s (−23%). Grouped dispatch remains stable at 665– 747 K tok/s across all imbalance levels. Recommendation: cuBLAS for predictably balanced load; Grouped otherwise.
Performance (GFLOP/s)
C. Dispatch Strategy Benchmark
Dense FFN Memorybound
103
Router gate
102
Token sort
E. Hyperparameter Sensitivity We ablate SVD rank, expert count, router architecture, calibration size, and random seed (Table VI); findings are: SVD rank (r ∈ {16, 32, 64, 96, 128, 192, 256}): accuracy range is 86.62–86.75% (0.13 pp). Reconstruction error decreases monotonically from 0.91 to 0.29, but this improvement does not translate to accuracy. Even r=16 captures the shared basis sufficiently. Expert count (E ∈ {2, 4, 8, 16}): accuracy is 86.65– 86.80% for E ∈ {2, 4, 8}; E=16 drops 0.26 pp as 200 calibration images provide insufficient statistics to estimate 16 fine-grained clusters. Latency grows ∼9 ms per doubling of E (73 ms at E=2 to 98 ms at E=16). No empty experts appear at any count. Router architecture: all three routers (Linear, MLP, Adaptive) achieve identical 86.68% Top-1. MLP routing accuracy reaches 0.980 vs. 0.925 for Linear, a 5.5 pp improvement in routing precision with zero downstream benefit. The gap between random routing (D4/D5, Table III) and all learned
Compute-bound Ridge 21.4 F/B
101 Compute-bound (blue) Memory-bound (red/orange) Roofline
100 10 1
D. Roofline Analysis Fig. 2 places each CLEAR-MoE operation on the GTX 960 roofline (peak FP32: 2.4 TFLOP/s, BW: 112 GB/s, ridge: 21.4 FLOPs/B). Dense FFN (AI = 74.3) and expert GEMMs (AI = 22.7) are compute-bound. The router gate (AI = 1.9) and token sort (AI = 1.0) are both deep in the memory-bound regime: 11× and 21× below the ridge point. This indicates that routing and token sort are strongly memory-bound (11– 21× below the ridge point), making fused dispatch kernels a plausible high-leverage optimization. Expert GEMMs are already near the compute ceiling and offer diminishing returns.
Expert GEMM
100
101
Arithmetic Intensity (FLOPs/Byte)
102
Fig. 2. Roofline for GTX 960. Blue = compute-bound; orange/red = memorybound. Router gate (AI = 1.9) is 11× below the ridge point (21.4 FLOPs/B); routing, not expert arithmetic, is the primary latency bottleneck. TABLE VI H YPERPARAMETER SENSITIVITY SUMMARY (D EI T-S MALL , I MAGENETTE ). A LL RANGES ≤0.41 PP. Study
Range tested
Top-1 span
∆pp
SVD rank r 16–256 86.62–86.75% 0.13 Expert count E 2–16 86.39–86.80%∗ 0.41 Router arch. Lin/MLP/Adaptive 86.68% (all) 0.00 Calibration N 50–500 86.55–86.70% 0.15 Random seed 42/123/456 86.70±0.02% 0.05 ∗ E∈{2, 4, 8} spans 0.15 pp; E=16 drops 0.26 pp (calibration instability: 200 images insufficient for 16 clusters).
routers is at most 0.06 pp (roughly two changed predictions out of 3,925). While this numerically exceeds the 3-seed standard deviation of ±0.02 pp (Table VI, row “Random seed”), a paired significance test would be needed to confirm a reliable effect; we treat 0.06 pp as practically negligible. Fig. 3 provides a mechanistic view: despite D6 (learned router) reducing mean per-token routing entropy 3.5× versus D5 (random, ln 4≈1.386 nats), the accuracy impact is negligible. Fig. 4 plots accuracy gap against router training across 13 configurations (D5 at x=0, no trained router); no strong monotonic relationship is visible, consistent with the hypothesis that shared-basis quality governs accuracy regardless of routing quality.
Per-layer routing entropy: D5 (random) vs D6 (learned) D5 D6
Mean routing entropy (nats)
1.6
Random router (max entropy, ln 4 1.39) Learned linear router
1.4 1.2
Backbone Params
1.0
DeiT-T/16 DeiT-S/16 ViT-S/16 DeiT-B/16 ViT-B/16
0.8 0.6 0.4
blk.1
blk.4
blk.5
blk.3
Expertized layer
blk.6
+0.04 +0.02
Top-1 vs. Dense (pp)
−0.05 −0.10 −0.05 −0.03 −0.08
+0.05† −0.05 −0.03 ±0.00 −0.10‡
0.913 0.925 0.956 0.927 0.967
0.090 0.099 0.139 0.094 0.150
blk.2
Routing accuracy vs. downstream accuracy (r=0.31)
+0.00 -0.02 -0.04
Linear Adaptive MLP
-0.06
-0.10
5.7 M 75.92% 22.1 M 86.73% 22.1 M 76.23% 86.6 M 91.77% 86.6 M 85.38%
‡ Highest load skew (0.150); see Discussion.
Fig. 3. Mean per-token routing entropy (nats) per expertized layer. Dashed red = D5 (random routing, max entropy ln 4≈1.386). Blue bars = D6 (learned linear router, mean ≈0.40 nats). Despite 3.5× entropy reduction, the D5-vsD6 accuracy gap is only 0.06 pp, confirming that routing confidence does not drive accuracy; the shared SVD basis does.
-0.08
Dense D5 ∆ pp D6 ∆ pp D6 Rtr. Acc D6 Skew
† Single-seed result for this backbone; per-backbone seed variance not characterised.
0.2 0.0
TABLE VII C ROSS - BACKBONE GENERALIZATION (I MAGENETTE , E=4, N =200, SEED 42). D5 = RANDOM ROUTING ; D6 = LEARNED LINEAR ROUTER . |∆|≤0.10 PP ACROSS ALL 10 CONFIGURATIONS .
D5 (random) D6 (learned linear) DeiT-S arch variant DeiT-T/16
0.0
0.2
DeiT-S/16 ViT-S/16 DeiT-B/16 ViT-B/16
0.4
0.6
0.8
Router Accuracy (vs. k-means labels)
1.0
Fig. 4. Accuracy gap (∆pp relative to dense) vs. router training across 13 configurations. D5 (random, no trained router) at x=0; D6 and cross-backbone routers at measured routing accuracy. No strong monotonic relationship is visible, consistent with the hypothesis that shared-basis quality governs accuracy regardless of routing precision.
Calibration size (N ∈ {50, 100, 200, 500}, multiple subsets for N <500): accuracy spans 86.55–86.70% across all sizes; N =50 yields 86.65–86.70% (within 0.15 pp of N =500). Router accuracy climbs from 0.83 to 0.95 as N grows, without accuracy benefit. Reproducibility: 86.70 ± 0.02% Top-1 across seeds 42, 123, 456 (full pipeline, composite scoring, E=4, N =200). F. Cross-Backbone Generalization Table VII tests whether the shared-basis finding extends across architectures. We apply the full CLEAR-MoE pipeline to five backbones spanning 5.7–86.6 M parameters and two pretraining lineages (DeiT, ViT); all other settings are identical to the DeiT-S ablation (E=4, N =200, seed 42).
Routing differences are numerically small. Learned routing (D6) provides a marginal numerical advantage over random routing (D5) in 4 of 5 backbones; however, the perbackbone improvements range from 0.02 to 0.10 pp, too small to confirm without paired significance testing. The one exception is ViT-B/16, where D5 (−0.08 pp) outperforms D6 (−0.10 pp): The ViT-B result coincides with the highest observed load skew (0.150, vs. 0.094 for DeiT-B), which may contribute to the difference; no ablation was performed to confirm a causal link. Taken together, these results are consistent with the hypothesis that routing quality has limited impact once the shared SVD basis is present; a definitive claim would require paired prediction-level testing across additional seeds. Fig. 5 visualises all 10 configurations. V. D ISCUSSION Why routing differences are small. All residual experts are scalar-conditioned variants of the same matrix (W2 − Wshared ), so mis-routing changes only the scaling factor applied to a token’s correction, not its direction. The 0.06 pp gap between random and learned routing is the direct consequence. Crossbackbone results (Table VII) are consistent across 4 of 5 architectures; a linear router (<10K parameters) appears adequate. Expert Choice routing [14], where experts select their tokens rather than the reverse, achieves better load balance in language models; on DeiT-S, however, random and learned routing differ by at most 0.06 pp, and all cross-backbone random-routing losses remain within 0.10 pp of dense, supporting the hypothesis that routing strategy is secondary to decomposition quality. Why CLEAR-MoE is slower on the GTX 960. CLEARMoE FFN computes shared fc1+fc2 on all tokens, then adds residual paths that each process T /E tokens but collectively span all T tokens across E experts, adding approximately one full fc2-equivalent computation. Total arithmetic is ≈ 1.5× dense FFN. On a bandwidth-limited GPU (112 GB/s vs. 2 TB/s for an A100), loading shared weights and expert-residual weights across memory dominates. The dispatch micro-benchmark isolates this: routing and token sort (AI = 1.0–1.9) are 11–21× below the compute ridge. Disjoint experts (D2/D7) are faster (48 ms) by eliminating shared paths, but sacrifice 21 pp accuracy. Analytical modelling suggests ≥800 GB/s bandwidth may be needed for latency parity; A100
Cross-backbone accuracy retention (Imagenette, E = 4, N = 200, seed 42)
VI. C ONCLUSION
Top-1 (pp vs. dense)
0.2 0.1
dense=75.9%
0.0
dense=86.7%
dense=76.2%
dense=91.8% dense=85.4%
0.1 D5 (random routing) D6 (learned linear router)
0.2 DeiT-T/16 (5.7M)
DeiT-S/16 (22.1M)
ViT-S/16 (22.1M)
DeiT-B/16 (86.6M)
ViT-B/16 (86.6M)
Fig. 5. Accuracy ∆ vs. dense for five ViT backbones under D5 (random) and D6 (learned linear) routing (Imagenette, E=4, N =200, seed 42). All deltas within |∆|≤0.10 pp. D6 provides marginal numerical advantage in 4 of 5 backbones; D5 outperforms D6 only on ViT-B (highest load skew, 0.150).
(2 TB/s) or H100 (3.35 TB/s) class hardware are the plausible candidates. Practical design guidelines. When to use CLEAR-MoE. CLEAR-MoE may be well suited when accuracy preservation is non-negotiable and backbone fine-tuning is infeasible: the evaluated DeiT and ViT backbones can be converted with 200 calibration images and no GPU cluster. The method is not competitive with disjoint-expert approaches if latency reduction is the primary goal on consumer hardware. Choosing E and r. Expert count E=4 provides a reasonable empirical trade-off: accuracy remains stable from E=2 to E=8, and E=4 balances router overhead with residual expressiveness. For E=16, our analytical model suggests ≥800 calibration images may be needed for stable cluster estimates; this was not directly validated. SVD rank r can be set as low as 16 without consistent measured accuracy reduction on Imagenette; the default r=d/2≈192 for DeiT-Small is conservative and may waste compute on deep layers. Choosing the dispatch backend. cuBLAS is optimal only for perfectly balanced expert load, which is common in classification but rare in detection/segmentation. For workloads with variable token-to-expert ratios, Grouped dispatch is approximately 31% faster than cuBLAS at 80% imbalance and remains comparatively stable across imbalance levels. Naive dispatch is a reference implementation only and is not recommended for latency-sensitive deployment. Limitations. Backbone and dataset scope. DeiT-T/S/B and ViT-S/B generalization is confirmed (Table VII); ViT-L, ImageNet-1K, and hierarchical backbones (Swin, ConvNeXt) are not evaluated. Parameter count, memory footprint, and FLOPs versus a same-hardware baseline are not reported. Hardware and compute. CLEAR-MoE is 1.3–1.7× slower than dense on GTX 960 (112 GB/s). Latency parity on highbandwidth hardware is modelled analytically only; multidevice projections are simulated (PCIe Gen3×16), not measured NCCL runs.
CLEAR-MoE demonstrates that post-training expert extraction preserves ≥99.9% of dense ViT accuracy via a shared SVD basis plus per-cluster residuals, requiring only 200 calibration images and a single consumer GPU. This finding is consistent across five ViT backbones spanning 5.7– 86.6 M parameters and two pretraining lineages (Table VII), showing that the shared SVD basis is the dominant accuracypreserving factor while routing quality, SVD rank, expert count (E ∈ {2, 4, 8}), and calibration size (N ∈ {50, . . . , 500}) are secondary. Random routing achieves 86.62% while learned routing at 98% accuracy achieves 86.68% (a 0.06 pp difference), simplifying the design space to a lightweight <10Kparameter linear router and suggesting future work should invest in decomposition quality over router expressiveness. On a bandwidth-constrained GTX 960 (112 GB/s), CLEAR-MoE’s FFN is 1.3–1.7× slower than dense because loading shared weights for all tokens dominates; roofline analysis confirms routing/scatter-gather (AI = 1.0–1.9) are 11–21× below the ridge point while expert GEMMs (AI = 22.7) approach the compute ceiling, identifying fused dispatch kernels as the primary engineering target. Future research directions include evaluating scaling on ViT-L and ImageNet-1K, implementing Triton-fused router-dispatch kernels, extending the composite scoring to hierarchical backbones (Swin, ConvNeXt) with nonuniform FFN widths, and empirically testing CLEAR-MoE on high-bandwidth hardware (A100/H100 class) to validate projected latency parity. R EFERENCES [1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” CoRR, vol. abs/2010.11929, 2020. [Online]. Available: https://arxiv.org/abs/2010.11929 [2] C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” 06 2021. [3] G. Hinton, N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” 01 2017. [4] T. Chen, X. Chen, X. Du, A. Rashwan, F. Yang, H. Chen, Z. Wang, and Y. Li, “Adamv-moe: Adaptive multi-task vision mixture-of-experts,” 10 2023, pp. 17 300–17 311. [5] Z. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, “Moefication: Transformer feed-forward layers are mixtures of experts,” 01 2022, pp. 877–890. [6] F. Szatkowski, B. W’ojcik, M. Pi’orczy’nski, and S. Scardapane, “Exploiting activation sparsity with dense to dynamic-k mixture-ofexperts conversion,” Advances in Neural Information Processing Systems 37, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID: 263831552 [7] U. Berisha, J. Mehnert, and A. P. Condurache, “Efficient data driven mixture-of-expert extraction from trained networks,” 2025. [Online]. Available: https://arxiv.org/abs/2505.15414 [8] A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. Ruiz, B. Mustafa, J. Ainslie, Y. Tay, M. dehghani, and N. Houlsby, “Sparse upcycling: Training mixture-of-experts from dense checkpoints,” 12 2022. [9] Y. Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, “Dynamicvit: Efficient vision transformers with dynamic token sparsification,” 06 2021.
[10] H. Yin, A. Vahdat, J. M. Alvarez, A. Mallya, J. Kautz, and P. Molchanov, “A-vit: Adaptive tokens for efficient vision transformer,” 06 2022, pp. 10 799–10 808. [11] C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, J. Chau, P. Cheng, F. Yang, M. Yang, and Y. Xiong, “Tutel: Adaptive mixture-of-experts at scale,” 06 2022. [12] W. Cui, S. Jiao, T. University, Z. Han, M. Research, Y. Wang, N. Zheng, L. Ma, Y. Yang, F. Yang, J. Xue, L. Qiu, L. Zhou, A. Quan, H. Tan, M. Guo, L. Ouyang, and Q. Chen, “Optimizing dynamic neural networks with brainstorm optimizing dynamic neural networks with brainstorm,”
07 2023. [13] C. Jiang, Y. Tian, Z. Jia, S. Zheng, C. Wu, and Y. Wang, “Lancet: Accelerating mixture-of-experts training via whole graph computation-communication overlapping,” ArXiv, vol. abs/2404.19429, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID: 269456940 [14] Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. Dai, Z. Chen, Q. Le, and J. Laudon, “Mixture-of-experts with expert choice routing,” 02 2022.
A PPENDIX A M ETHODOLOGY D ETAILS A. Glossary of Key Terms Table A1 defines domain-specific terminology used throughout the paper. TABLE A1 D OMAIN - SPECIFIC TERMINOLOGY.
Term
Definition
Token
Image patch embedded as d-dimensional vector; 197 tokens per 224×224 image Feed-forward network: two-layer MLP applied per token in each transformer block Routed residual branch processing a token subset; in CLEAR-MoE, all residual experts share one direction (W2 −Wshared ) and differ only by a cluster-conditioned scalar se Lightweight network mapping each token to a probability distribution over experts FFN replaced by E experts plus a router; only top-k<E experts activate per token Singular value decomposition; truncated SVD retains top-r singular components Partitions N points into k clusters by minimising within-cluster variance Small representative set used to measure model statistics (no weight updates) Arithmetic intensity: FLOPs per byte of memory accessed AI where compute ceiling equals memory bandwidth ceiling (21.4 FLOPs/B on GTX 960) Median over 100 repeated forward passes; robust to OS-scheduling outliers NVIDIA’s optimised linear-algebra library; batched GEMM calls dispatch through it
FFN Expert
Router/Gate MoE layer SVD k-means Calibration set AI Ridge point p50 Latency cuBLAS
B. Dataset Preparation and EDA Imagenette. 13,394 raw images were scanned using SHA256 deduplication, robust z-score filtering (τ =3.5), and Isolation Forest (contamination = 0.01). 118 images (0.88%) were removed, yielding a clean set of 9,391 train / 3,885 val (used for EDA analysis only). The standard Imagenette validation split (3,925 images, including the 40 removed as anomalies by our filter) was used for all reported Top-1 accuracy evaluation, ensuring comparability with prior work. ImageNet normalisation and Lanczos-4 resize to 224×224 are applied. Distribution shift analysis. DeiT-Small penultimate activations for all 3,885 clean val images (shape 3885×384): PSI = 0.043 (<0.1, negligible shift), JSD = 0.018, KS pvalues >0.05 for all 10 classes. Clustering quality. k=10 on penultimate image-level activations: Silhouette = 0.542,
Davies-Bouldin = 1.24, confirming that the feature space supports class-level cluster separation. Note that CLEAR-MoE clusters token-level intermediate activations with k=4, which is a complementary but distinct analysis; the image-level EDA establishes feature quality rather than directly validating token cluster assignments. SVD energy. Rank-192 truncation retains 99.2% of penultimate activation matrix variance; top10 singular values capture 67.8%. The effect of W2 truncation rank on accuracy and reconstruction error is evaluated in Table C3. Table A2 summarises these preprocessing and analysis statistics. TABLE A2 DATASET PREPROCESSING AND EDA STATISTICS .
Statistic
Value
Raw Imagenette images Removed (anomalous) Train / val split
13,394 118 (0.88%) 9,391 / 3,885 (clean, EDA only); 3,925 (standard, used for all Top-1) 0.043 (<0.1 = negligible shift) 0.018 0.542 1.24 99.2% 200 images
Train-test PSI Train-test JSD k=10 Silhouette score k=10 Davies-Bouldin Rank-192 SVD variance Calibration size
C. Algorithm Pseudocode Algorithms 1–3 give pseudocode for the three core CLEARMoE phases. Algorithm 1 Phase 2: Layer Scoring and Selection (i)
Require: Calibration activations {hl }, L = 12 layers, select k=6 1: for each layer P l do 1(|hl,j |<0.01) 2: sp(l) ← j numel(hl ) 3: cl(l) ← Silhouette(k-means(hl , E)) 4: se(l) ← ∥∆logits∥/∥logits∥ (zero FFN output) 5: S(l) ← 0.4 sp + 0.4 cl − 0.2 se 6: end for 7: Select top-k layers by S
Algorithm 2 Phase 3: Expert Extraction (fc1 shared; fc2 decomposed) Require: W2 ∈ Rd×dffn , activations z, E, rank r 1: [U, Σ, V ] ← SVD(W2 ); Wshared ← Ur Σr Vr⊤ 2: {C1 , . . . , CE } ← k-means(z, E) 3: for e ∈ {1, . .P . , E} do 1
∥zi ∥2
e se ← |Ce1| PNi∈C∥z i ∥2 i=1 N res 5: We ← (W2 − Wshared ) · se 6: end for
4:
Algorithm 3 CLEAR-MoE FFN Inference (Grouped backend) Require: Tokens x, shared fc1, Wshared , {Weres }, Router g 1: z ← GELU(fc1(x)) (shared, all tokens) ⊤ 2: hsh ← zWshared (shared SVD basis) 3: a ← argmax(g(x)) (expert assignment) 4: Sort z by a; get sub-batch boundaries via index search 5: hres [e] ← ze Weres⊤ for each expert e 6: Unshuffle hres ; y ← hsh + hres A PPENDIX B E XPERIMENTAL C ONFIGURATION
TABLE C1 3-S EED ROBUSTNESS . F ULL COMPOSITE - SCORE PIPELINE , E=4, D EI T-S MALL . Seed
Top-1
p50 ms
Router Acc
Skew
42 123 456
86.68% 86.70% 86.73%
85.05 83.90 78.89
0.925 0.928 0.928
0.099 0.116 0.101
86.70±0.02%
82.6±2.7
0.927±0.002
0.105±0.008
Mean±Std
TABLE C2 C ALIBRATION - SET SENSITIVITY. M ULTIPLE RANDOM SUBSETS PER SIZE .
A. Hardware and Software Setup
TABLE B1 H ARDWARE AND SOFTWARE CONFIGURATION FOR ALL EXPERIMENTS .
Item
Specification
OS GPU GPU memory Memory bandwidth Peak FP32 throughput Ridge point CUDA PyTorch Classification backbone
Windows 11 Pro NVIDIA GeForce GTX 960 4.0 GB GDDR5 112 GB/s 2.4 TFLOP/s 21.4 FLOPs/Byte 11.8 2.6.0+cu118 DeiT-Small; 22M parameters; patch size 16×16; d=384; dffn =1536; 12 transformer blocks Not evaluated (planned future work) 4 experts r=d/2=192 for DeiT-Small (50% of matrix rank) 200 images 5 epochs, AdamW, lr 10−3 , cosine decay p50 of 100 passes, batch = 8, cuda.synchronize()
Segmentation backbone Default E Default rank r Calibration size Router training Latency
Ncal
Sub.
Top-1
Router Acc
Skew
50 50 50 100 100 100 200 200 500
1 2 3 1 2 3 1 2 1
86.70% 86.68% 86.65% 86.55% 86.60% 86.70% 86.68% 86.60% 86.70%
0.867 0.833 0.856 0.896 0.908 0.899 0.926 0.928 0.953
0.139 0.073 0.139 0.094 0.150 0.153 0.099 0.128 0.098
TABLE C3 SVD RANK SWEEP. S EED 42, E=4, COMPOSITE SCORING .
B. Hyperparameter Sensitivity Studies Tables C1–C5 report the full numerical results for the five hyperparameter sensitivity studies summarised in Section IV.
p50 ms
Recon Err
16 32 64 96 128 192 256
86.75% 86.65% 86.65% 86.75% 86.68% 86.62% 86.70%
78.13 76.79 76.60 78.07 77.82 76.01 77.36
0.907 0.841 0.734 0.643 0.562 0.420 0.294
1.0
86.9
Top-1 accuracy (%)
Per-layer composite scores. Full composite scores for all 12 DeiT-Small FFN blocks appear in Table II (Section III). Block 0 ranks last (composite = 0.101) due to sensitivity = 0.946. Full layer-selection policy ablation. The complete L0–L10 ablation now appears in Table IV (Section IV-B).
Top-1
SVD rank vs. accuracy and reconstruction error (DeiT-Small, Imagenette, E = 4, seed 42)
A PPENDIX C E XTENDED E XPERIMENTAL R ESULTS A. Full Ablation Tables
Rank r
0.8
86.8 Top-1 (%) Recon. error
86.7 86.6
Reconstruction error
Table B1 lists the complete hardware and software configuration used in all experiments.
0.6 0.4 0.2
86.5 16 32
64
96
128
SVD rank r
192
256
0.0
Fig. C1. SVD rank vs. Top-1 accuracy (left axis, blue) and reconstruction error (right axis, red). Accuracy spans 0.13 pp across r ∈ {16, . . . , 256} while reconstruction error drops from 0.907 to 0.294, confirming rank does not gate accuracy once the shared basis captures the dominant singular subspace.
Dispatch throughput vs. load imbalance (B = 8, T = 1568, E = 4, GTX 960)
TABLE C4 E XPERT- COUNT STUDY. S EED 42, COMPOSITE SCORING , k=6.
Top-1
p50 ms
Rtr Acc
Skew
Empty
2 4 8 16
86.65% 86.65% 86.80% 86.39%
73.4 77.5 84.3 97.8
0.967 0.923 0.893 0.865
0.247 0.074 0.042 0.024
0 0 0 0
Throughput (K tok/s)
700
E
TABLE C5 ROUTER ARCHITECTURE COMPARISON . S EED 42, E=4, COMPOSITE SCORING . E NTROPY: AGGREGATE EXPERT- USAGE ENTROPY ( DISTRIBUTION OF TOKENS ACROSS EXPERTS , PER LAYER ; MAX = ln 4 ≈ 1.386). N OTE : THIS IS THE load-balance ENTROPY, DISTINCT FROM THE PER - TOKEN PREDICTIVE ENTROPY IN F IG . 3 (≈0.40 NATS ), WHICH MEASURES ROUTER CONFIDENCE . Router
Top-1
p50 ms
Rtr Acc
Skew
Entropy
Params
Linear MLP Adaptive
86.68% 86.68% 86.68%
78.0 75.6 77.6
0.925 0.980 0.936
0.099 0.095 0.099
1.296 1.300 1.296
9,240 149,400 9,240
C. Dispatch Benchmark and Parallel Scaling Table C6 benchmarks all three CUDA backends under four token-load imbalance levels (B=8, T =1568 tokens, E=4, GTX 960); CPU Serial is included as a baseline, allowing direct GPU-vs-CPU comparison at each imbalance level. The roofline plot appears in Fig. 2 (Section IV). Table C7 projects multi-device throughput analytically. TABLE C6 D ISPATCH MICRO - BENCHMARK : P 50 LATENCY ( MS ) AND THROUGHPUT (K TOK / S ) ACROSS IMBALANCE LEVELS . T =1568 = 8×196 TOKENS ( CLASS TOKEN EXCLUDED FROM EXPERT DISPATCH ; PROCESSED ON THE DENSE PATH ). C ALIBRATION USES 197 TOKENS / IMAGE INCLUDING THE CLASS TOKEN .
0% imb. 40% imb. 60% imb. 80% imb. Backend
ms K/s
ms
K/s
ms
K/s
ms
K/s
CPU Serial Naive Grouped cuBLAS
6.01 3.67 2.36 2.15
7.22 3.55 2.10 2.29
217 442 747 684
5.96 3.50 2.30 2.45
263 448 681 639
5.96 3.59 2.14 2.81
263 437 733 558
261 428 665 728
600 500 400 300 200
Naive Grouped cuBLAS
100 0
0%
40%
Token-load imbalance (%)
60%
80%
Fig. C2. Dispatch throughput vs. token-load imbalance for three CUDA backends (B=8, T =1568, E=4, GTX 960). Grouped is stable across all imbalance levels (665–747 K tok/s); cuBLAS degrades 23% from 0% to 80% imbalance due to padding waste. For variable-load workloads, Grouped is the recommended backend.
GPU vs. CPU and Parallel Scaling. GPU vs. CPU speedup at balanced load is readable from the CPU Serial row in Table C6 above: at 0% imbalance, cuBLAS achieves 728 K tok/s vs. 261 K tok/s for CPU Serial (2.79× speedup). Multi-device projection (simulated; no second GPU available). All-to-all and AllReduce costs are modelled analytically using PCIe Gen3×16 bandwidth (16 GB/s). Results are not measured NCCL runs. TABLE C7 PARALLEL SCALING PROJECTION . E XPERT /DATA /P IPELINE PARALLELISM ARE SIMULATED ; SINGLE -GPU IS MEASURED . Mode
tok/s
Devices
Speedup vs GPU-1
Efficiency
Full-model single-GPU Expert Parallel EP-2 (sim.) Pipeline Parallel PP-2 (sim.)
225K 175K 8K
1 2 2
1.00× 0.78× 0.03×
1.00 0.39 0.02
All multi-device entries are simulated via PCIe Gen3×16 bandwidth; no NCCL runs performed. EP-2 overhead: all-to-all token redistribution (AI = 0.1 F/B). PP-2: 50% pipeline bubble with 1 micro-batch, 2 stages. Single-GPU throughput is full-model (distinct from dispatch-only throughput in Table C6).