PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
Yuchen Yang1
Yifan Zhao1 Anisha Dasgupta1 Sasa Misailovic1 University of Illinois Urbana-Champaign {yucheny8, yifanz16, adasg, misailo}@illinois.edu 1
arXiv:2607.16184v1 [cs.LG] 17 Jul 2026
Abstract Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements of the model weights and the growing KV cache. We propose PagedWeight, a novel management method for MoE LLM serving that dynamically quantizes MoE model’s weights at runtime and balances expert-weight precision with the KV cache sizes. PagedWeight exposes and effectively navigates the complex tradeoff between the model’s task accuracy, memory consumption, and throughput/latency. Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over several existing quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94× throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.
1
Introduction
Mixture-of-Experts (MoE) models are gaining traction for efficiently solving complex language and reasoning tasks. An MoE layer consists of a bank of expert networks and a router that selects a subset of experts that are likely to perform well. Since only a subset of experts is active, MoE significantly reduces the amount of computation without sacrificing accuracy [18, 6, 13], making it particularly appealing for long-context tasks, such as repository-level coding and long document analysis. However, MoE scales up the total parameter count and leaves a large memory footprint on the GPU. GPU memory management is a key challenge in modern LLM serving systems, which need to store both the model weights and the KV cache in the GPU memory during inference. The size of KV caches grows with the context length and is a main source of memory pressure, for which system developers have developed various techniques to mitigate this. For example, vLLM PagedAttention manages the KV cache in paged blocks to reduce fragmentation and increase batching efficiency [17], and KV cache compression reduces memory usage with a minimal impact on accuracy [14]. Yet, all of these methods are limited by the other significant memory bottleneck, which is MoE weights in the GPU memory. With MoE, the loaded model size can occupy more than 60% of the GPU memory [17]. A convenient and automatic approach to reduce the memory footprint of MoE models can create more space for the KV cache and allow for longer contexts. Moreover, it opens up an interesting and unexplored tradeoff space between the task accuracy, computation time, and the allocation and balancing of GPU memory for model weights and the KV cache. Quantization is the most common way to reduce model size, as demonstrated by many post-training techniques [7, 30, 21, 2]. However, most quantization techniques can only be applied statically, i.e., they only modify weights before the model runs. Existing runtime precision-adaptation methods show that precision does not need to be fixed for an entire request [16]. However, they do not use precision changes to reallocate GPU memory at runtime. In MoE serving, a key challenge is to Preprint.
Static Quantization
PagedWeight Offload
Resident MoE Weights
Resident MoE Weights
Reload
Exceeds Budget
KV Cache Other
CPU RAM
KV Cache Other
Figure 1: Static quantization fixes MoE weights; PagedWeight offloads weight pages to free KV cache headroom. free memory from less critical expert weights as the KV cache grows, without hurting quality or interrupting inference. Our Work: We propose PagedWeight, a novel management method for MoE LLM serving that dynamically quantizes MoE model’s weights at runtime and balances the choice of quantization with the KV cache and context sizes. PagedWeight builds on our insight that experts’ weights can be treated in a similar way as paged blocks in KV caches, thus avoiding inflexibility of static quantization (see Figure 1). We designed PagedWeight to satisfy the following three objectives: • Dynamic KV/weight tradeoff. PagedWeight should release GPU memory that model weights consume when the KV cache grows and needs more GPU memory. • Quality-aware expert selection. PagedWeight should choose MoE linear-blocks for which it predicts that the bitwidth changes will have the smallest impact on the request’s output quality. • Asynchronous execution. Page movement should overlap with the model serving and should not add extra cost. To effectively represent multiple quantized experts, PagedWeight leverages Any-Precision LLM (APL), which can represent multiple bitwidths in an overlaid bit-plane format [24]. PagedWeight treats the Any-Precision (AP) bit-plane and lookup table (LUT) buffers of routed expert linear-blocks as GPU-resident pages whose bitwidth can be lowered or restored at runtime. Unlike PagedAttention, which pages KV cache blocks, PagedWeight manages quantized weight state. Unlike static MoE quantization, it adapts the expert precision to current KV cache size. PagedWeight’s structured planner uses both offline and runtime analyses to determine the quantization level that is likely not to impact task accuracy. Those include offline sensitivity (calibrated Hessianweighted quality-damage scores), routing-aware grouping (protection for frequently routed experts), and prompt residuals (prompt-specific corrections to quantization-impact estimates). To optimize execution time and hide memory transfer latencies, PagedWeight offloads/reloads quantized weight pages asynchronously and commits only at safe boundaries. We also implement a fused mixedprecision MoE kernel that reads Any-Precision bit-planes and LUTs directly. We evaluate PagedWeight on three common open MoE models, ranging in size between 14.3B and 46.7B and against three state-of-the-art quantization methods - APL [24], DP-LLM [16], and MxMoE [3]. PagedWeight consistently achieves a better task quality-memory tradeoff than all evaluated quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94× throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss. Contributions. This paper makes the following contributions: • We propose PagedWeight, a paged-weight system for online Any-Precision MoE weights that manages the committed bitwidths, desired bitwidths, page states, and memory consumptions. • We design a quality-aware runtime planner that combines offline sensitivity, online routing statistics, and prompt residuals to choose a low-damage movement strategy for the weight page under an increased memory demand from KV cache. • We use asynchronous page movement to hide offload/reload latency, with minimal throughput loss. 2
Usage
(a) C4 vs. Wikitext2 0.020 0.010
C4 Wikitext2
(b) Within C4
Usage
0.024 0.016 C4 (chunk 1) C4 (chunk 2)
0.008 0
10
20
30
Expert Index
40
50
60
Figure 2: Expert activation patterns on Qwen1.5-MoE-A2.7B. The y-axis shows the normalized routing weight assigned to each expert. • We evaluate PagedWeight on three popular MoE models and demonstrate that it outperforms all evaluated quantization baselines.
2
Background
2.1
Mixture-of-Experts Architecture and Routing Imbalance
In an MoE layer, the router assigns routing weights to E experts and selects the top K experts for each token [13]. During inference, experts in a layer are selected with highly unequal frequency [6]. This routing imbalance indicates that experts are not all equally important, and locating the less important experts allows us to quantize them more aggressively with minimal accuracy loss. The routing mass of an expert captures its importance, which is intuitively its routing weight normalized by the total weight of all experts in a layer. It is formally defined, for an expert e in layer ℓ, as: P P rℓ,t,k 1{zℓ,t,k = e} mℓ,e = t kP P (1) t k rℓ,t,k t indexes observed tokens, k indexes the top-k router choices, zℓ,t,k is the selected expert, and rℓ,t,k is its routing weight. As shown in Figure 2, routing mass varies substantially across experts and inputs. 2.2
Mixed-Precision Quantization and Runtime Quantization
To achieve higher accuracy, mixed-precision quantization (MPQ) assigns a different bitwidth to each component of the network (such as a weight matrix or an expert), and often uses the component’s sensitivity to estimate the accuracy impact. Hessian-based sensitivity is a common approach used in previous work [7, 16], which defines the sensitivity of a weight matrix Wi,j at bitwidth b as: X b 2 sbi = hi,j Wi,j − Wi,j (2) j
where i indexes a quantized component, j indexes elements in weight matrix i, hi,j is a Hessianb derived importance score of Wi,j , and Wi,j is the b-bit quantized version of Wi,j . Lower sbi indicates that the weight can tolerate more aggressive quantization. In the context of MoE models, each expert typically consists of weight linear-blocks [3]. Within the same expert, linear-blocks can have different sensitivity to quantization, and quantizing them to different bitwidths can yield better accuracy. Most existing quantization methods produce a fixed quantized model offline, which does not change at serving time [21, 7, 30, 2], thus missing out on the opportunity to adapt to dynamic inference conditions. There are a few works that explore runtime quantization of LLMs, such as APL [24], which allows the quantization level of the model to change at inference time.
3
Workload stats
Planner
Control flow Data movement
set desired bits dᵢ
Weight Page Table
GPU Memory Resident weight pages …
GPU residency
Expert/module i
qᵢ
dᵢ
GPU-resident bit-planes CPU-resident bit-planes
E₁ gate_up
5
6
b7 b6 b5 b4 b3 b2 b1 b0 b7 b6 b5 b4 b3 b2 b1 b0
4
6
b7 b6 b5 b4 b3 b2 b1 b0 b7 b6 b5 b4 b3 b2 b1 b0
E₁ down KV cache blocks
…
…
Eₙ gate_up
… … 7
8
…
CPU residency
CPU RAM Offloaded weight pages …
…
b7 b6 b5 b4 b3 b2 b1 b0 b7 b6 b5 b4 b3 b2 b1 b0
Reload weight pages
Commit current bits qᵢ
Inference Engine
Offload weight pages
Figure 3: PagedWeight system overview. Dashed arrows show control flow, and solid arrows show weight data movement. APL stores weights in a shared bit-plane format, where different subsets of bit-planes correspond to different effective bitwidths. Each quantized tensor is also associated with a lookup table (LUT), which stores the bitwidth-specific centroid values used by the AP representation.
3
PagedWeight System
We illustrate the workflow of PagedWeight in Figure 3. The weight page table records for each linear-block (of each expert in each MoE layer) the desired bitwidth di and the committed bitwidth qi . qi is the precision currently used in inference. The planner is PagedWeight’s runtime controller, in charge of quantization decisions and data movements. The planner decides desired bitwidths di and updates them in the weight page table, based on the current KV-cache pressure, routing statistics, and prompt-specific features. PagedWeight executes planned quantization decisions (making actual qi match planned di ) by moving weight pages between CPU and GPU, when it is safe to update them. 3.1
Paged Weight Representation
PagedWeight quantizes a network at the granularity of per expert linear-block (a weight in an expert). Each linear-block is identified by a triple index: i := (ℓ, e, u) for layer ℓ, expert e, and one of two linear-block types u ∈ {gate_up, down}. PagedWeight organizes linear-blocks and their quantization decision in a weight page table, where each linear-block i occupies one row. The page table row for i stores: a set of supported bitwidths Bi for the planner to choose from, the committed bitwidth qi that is currently used in inference, and the desired bitwidth di selected by the planner. Since linear-block weights are stored in a bit-plane representation, each row also carries a lookup table for the state of each bit-plane, where the state can be GPU-resident, CPU-resident, or in transfer. One bit-plane and its lookup table state together is one weight page. Each page table row i also contains a GPU-resident memory size table Mib for each b ∈ Bi , which records memory size for weight of each precision. If a linear-block weight goes from bitwidth b to a ′ lower bitwidth b′ , the amount of GPU memory released, ∆b→b , is i ′
′
∆b→b = Mib − Mib i
b′ < b
(3)
Mib includes both the AP bit-plane and LUT for b-bit of linear-block i. PagedWeight moves data between CPU and GPU to carry out planned quantization decisions. During an offload, PagedWeight first reduces the committed bitwidth qi in the table, and then moves unneeded pages from GPU memory to CPU memory. During a reload, PagedWeight first moves pages from CPU memory back to GPU memory. It only raises qi when the page state becomes GPU-resident. 3.2
Quality-Aware Runtime Planner
The runtime planner of PagedWeight makes quantization decisions based on the estimated impact of bitwidth reductions on model accuracy. A candidate action considered by the planner is (i, b → b′ ), where i = (ℓ, e, u) and b′ < b. For each action, the planner estimates accuracy impact by combining three factors: offline global sensitivity, online routing statistics, and runtime prompt residual. 4
As discussed in Section 2.2, offline sensitivity can guide bitwidth assignment with budget. PagedWeight uses this assignment once before serving to initialize committed bitwidths qi . During serving, the planner refreshes plans using runtime observations. Online routing statistics identify frequently selected experts and increase the estimated impact of quantizing their linear-blocks, protecting hot experts. Runtime prompt residual provides a prompt-conditioned correction to the offline sensitivity estimate, capturing that the same quantization action can have different impact on different prompts. Algorithm 1 shows one step of PagedWeight’s runtime planner. The input Tt is the page-table state at time step t, which includes the committed bitwidth qi , desired bitwidth di , page states, and weight size table Mib for linear-block i at bitwidth b. Ot is the runtime observation at t, including KV-cache pressure, routing statistics, and prompt features. Ωoff stores offline calibration results, including sensitivity scores, prompt-residual heads, and bitwidth floor. The planner also takes the current plan queue Qt as input and returns an updated Qt+1 . If memory pressure is triggered, it also returns a page-movement plan Πt .
Algorithm 1 PagedWeight Runtime Planner Inputs: Tt : page table state, Ot : runtime observation, Ωoff : offline calibration results, Qt : current plan queue. Return: Qt+1 : updated plan queue, Πt : page movement plan {(i, qi → di , ∆i )}. 1: function P LAN S TEP(Tt , Ot , Ωoff , Qt ) 2: Qt+1 ← Qt 3: Πt ← ∅ 4: Dt ← TARGET B YTES F ROM KVP RESSURE(Ot ) 5: for all s ∈ B ITWIDTH F LOOR S TAGES(Ωoff ) do 6: Rs ← S CORE ACTIONS(Tt , Ot , Ωoff , s) 7: Πs ← G REEDY S ELECT(Rs , Dt ) 8: Qt+1 ← R EFRESH P LAN Q UEUE(Qt+1 , Πs ) 9: end for 10: if P RESSURE T RIGGERED(Ot ) then 11: Πt ← S ELECT B EST P LAN(Qt+1 , Dt ) 12: end if 13: return (Qt+1 , Πt ) 14: end function
The planner step first copies Qt to Qt+1 and initializes the plan Πt (lines 2–3). It then converts KV-cache pressure into the byte target Dt (line 4). For each bitwidth-floor stage s from Ωoff , S CORE ACTIONS builds the legal transition set Rs , G REEDY S ELECT selects a candidate plan Πs to meet Dt , and R EFRESH P LAN Q UEUE updates Qt+1 (lines 5–9). When pressure exceeds the threshold, S ELECT B EST P LAN selects the best plan from Qt+1 as Πt (lines 10–13). The runtime invokes this planner step repeatedly as the request runs, so later steps can refresh the queue with newer observations. Offline global sensitivity. As discussed in Section 2.2, offline calibration gives a sensitivity score sbi for each linear-block i and supported bitwidth b ∈ Bi . PagedWeight reuses these scores and defines the global damage of reducing the bitwidth of linear-block i from b to b′ as ′
′
gib→b = max{sbi − sbi , 0}
b′ < b
(4)
Routing statistics. As discussed in Section 2.1, MoE routing is highly imbalanced, and the routing mass mℓ,e measures how much traffic an expert receives within a layer. During serving, PagedWeight collects the frequency of expert selection and routing weights, and updates routing mass online. PagedWeight sorts experts in each layer by their routing mass and assigns them to different buckets. βℓ,e is the bucket of expert e in layer ℓ. Each bucket has a damage multiplier µβℓ,e and a bitwidth floor. Hotter buckets contain experts with larger routing mass. They use larger damage multipliers and a higher bitwidth floor to protect them. Linear-blocks of an expert share the same routing bucket. Prompt residual. PagedWeight also uses prompt residual, which is a prompt-sensitive correction to the global damage metric. Prompt residual is useful because quantization sensitivity depends on the input. We keep the global damage as a prompt-independent prior, and use the prompt residual to adjust this estimate for the current input. The residual is predicted from prompt-specific features using linear regression heads. Each head is specific to a linear-block type and bitwidth transition (b → b′ ), and is shared across linear-blocks of that type. ′
The global damage gib→b estimates the average damage of reducing the bitwidth of linear-block i from b to b′ , but an action can have different damage for different inputs. PagedWeight therefore adds a prompt residual on top of the global damage. In offline calibration, we measure the per-sequence damage for each candidate action (i, b → b′ ) and ′ compare it with the global estimate. The training target is their log residual ρb→b , where a positive i value means the prompt is more sensitive than the global estimate and a negative value means it is less sensitive. We train a linear regression head for each linear-block type u and bitwidth transition
5
offload threshold triggered Planner
plan
plan
plan
plan
plan
plan
plan
offload (GPU CPU)
Page I/O Committed bitwidth
plan
reload threshold triggered
plan
plan
plan
plan
reload (CPU GPU)
6-bit committed
4-bit committed
5-bit committed
Time
Figure 4: Asynchronous page-movement pipeline. PagedWeight planner keeps preparing precisiontransition plans during runtime. At the offload threshold, the latest plan lowers the committed bitwidth and releases weight pages from GPU memory. At the reload threshold, the weight pages are restored before the higher bitwidth is committed. b → b′ :
′
′
′
ρbb→b = (wub→b )⊤ ϕi + ab→b . (5) i u Here, ϕi is a three-dimensional routed-input feature vector: routing-weighted mean input norm, routing-weighted RMS input norm, and maximum input norm. At runtime, PagedWeight computes ϕi for the current sequence and uses the trained linear head to ′ ′ predict the log residual ρbb→b . The prediction is clipped and scaled by a confidence score cb→b and a u i strength hyperparameter α: ′ ′ ′ ηib→b = exp αcb→b clip ρbb→b , ρmin , ρmax . (6) u i ′
′
The planner uses gib→b ηib→b as the prompt-adjusted damage. Planning objective. The hyperparameters in the planner are decided in calibration; more details are in Section 4. The planner quantifies KV-cache pressure as a target number of bytes D: D = max{0, Tblk + 1 − Fblk } BKV
(7)
where Tblk is the free-block threshold, Fblk is the current number of free blocks, and BKV is the byte ′ size of one KV block. For each legal candidate action, the predicted damage dbb→b is i ′ ′ ′ dbb→b = max{ϵ, µβi gib→b ηib→b } i
(8)
where ϵ is a small damage floor, and µβi is the damage multiplier of the routing bucket of expert ′ ′ (ℓ, e). gib→b ηib→b is the prompt-adjusted damage. For batched serving, the planner first merges request-level signals into one batch-level table. The routing masses, prompt residuals, confidence scores, and clipping bounds are averaged based on the request’s KV block usage weights. The planner considers each legal bitwidth reduction as one step. It sorts the steps by predicted damage b per released byte, d/∆, and greedily takes the cheapest steps until the target D is met. Depth caps are relaxed only when the current caps cannot provide enough bytes. If no legal plan fully satisfies the target, the planner returns the largest safe reduction under the current constraints. 3.3
Asynchronous Page Movement and Kernel Optimization
Asynchronous Page Movement. Figure 4 illustrates the asynchronous page movement pipeline. During serving, the planner monitors free KV blocks, allocator headroom, and pending KV growth, and keeps generating new plans with the updated information. When pressure exceeds the threshold, the planner chooses the newest plan destination bitwidth for selected linear-blocks. For offload, the system copies the weight page to CPU RAM and commits the lower qi . Once the transition is safe, the system retires the GPU pages above the committed precision, and the released allocator headroom can be used by the KV blocks. For reload, the runtime restores the offloaded weight pages first and commits the higher qi only after the page state is GPU-resident. Fused Mixed-Precision MoE Kernel. PagedWeight provides a fused mixed-precision MoE AP kernel to enable per-expert bitwidths at runtime. The kernel reads weight pages directly and enables 6
different bitwidths for each linear-block, and does the routing, expert activation, down projection, and output accumulation in one fused CUDA kernel. The kernel fusion reduces the overhead of executing the selected experts separately. Since the AP state is maintained at linear-block granularity, the kernel can follow the bitwidth chosen by the planner for each gate_up and down module.
4
Experimental Methodology
Evaluated Models. We evaluate PagedWeight Table 1: Architectural specifications of evaluated on three state-of-the-art base MoE language MoE models. models whose architectures cover different ex- Model Variant Params (B/GB) Experts TopK pert granularities and routing patterns. Qwen1.5- Qwen1.5-MoE-A2.7B 14.3 / 26.7 60+4 4 MoE-A2.7B [25] combines 60 routed experts Mixtral-8×7B-v0.1 46.7 / 92.9 8 2 with 4 shared experts and uses top-4 routing. Gemma-4-26B-A4B 25.2 / 53.1 128+1 8 Mixtral-8×7B-v0.1 [13] uses 8 routed experts with top-2 routing. Gemma-4-26B-A4B [9] represents a higher-expert-count design with 128 routed experts, one shared expert, and top-8 routing. Table 1 summarizes these architectural differences. Evaluation Metrics. For perplexity evaluation, we report results on Wikitext2 and C4. For downstream reasoning, we use GSM8K and MATH-500. For long-context evaluation, we report the LongBench scores for Passage Retrieval, NarrativeQA, and QMSum. For serving efficiency, we report generation throughput. For memory-sensitive comparisons, we report peak process-reserved GPU memory. For quality-memory comparisons in Section 5.1, we use batch size 16 to better reflect a realistic batched serving scenario. Policy Construction. The quantization and runtime hyperparameters are built offline on the C4 calibration set. We first form a prompt-independent sensitivity table for each routed expert linear-block and supported bitwidth transition. Then we train the prompt residual from routing weighted input norm features, and use a small calibration search to choose the routing buckets, bucket multipliers, depth caps, and residual weight used by the online planner. Hardware and Software Setup. Qwen1.5-MoE-A2.7B experiments are conducted on an NVIDIA RTX 6000 Ada workstation. The other evaluated models are run on an NVIDIA GH200 Grace Hopper system. We use vLLM v0.20.1 as the serving backend. Baselines. We compare PagedWeight against FP16, Any-Precision LLM [24] for uniform quantization, MxMoE [3] for static mixed-precision quantization, and DP-LLM [16] for dynamic mixedprecision quantization. Since the original APL implementation does not support MoE models, we compare against our MoE fused CUDA implementation of the uniform APL-style baseline.
5
Evaluation
RQ1: Can PagedWeight improve the quality-memory tradeoff of MoE LLMs across different tasks? RQ2: Can PagedWeight preserve long-context quality under large per-request KV-cache demand? RQ3: Can PagedWeight improve model quality while maintaining the throughput? RQ4: Ablation studies: How much do components of PagedWeight contribute to the performance? 5.1
Quality-Memory Tradeoff
Figure 5 reports model quality under different GPU memory consumption, the x-axis shows memory consumption in GB, and the y-axis shows task quality. We report perplexity on Wikitext2 and C4, where lower is better, and reasoning accuracy on GSM8K and MATH-500, where higher is better. Baseline memory comparison. For PagedWeight and APL, we report the measured real GPU memory consumption of the runtime. For DP-LLM and MxMoE, theoretical memory is the storage estimated directly from the assigned bitwidths, while real memory is the memory consumption required by the existing implementation; we report both. DP-LLM needs to keep the higher-bit tensors in memory due to its dynamic precision framework, and during inference, it dequantizes both the high and low-bit precision paths. MxMoE uses “fake” quantization for its weight-only quantization implementation, i.e., weights are simulated at low precision but stored and executed with higher-precision tensors. It also loads multiple sets of weights, so its real memory consumption is
7
much larger than the theoretical memory consumption. We report MxMoE’s 3.25-bit mixed-precision configuration, which gives the best accuracy among its reported configurations. Overall Trends. Across all three models and tasks, PagedWeight always reaches high quality with the lowest memory consumption (Figure 5). These results show that selecting routed expert pages to reduce is more effective than using one static quantization precision policy.
Ours
APL
DPLLM real
DPLLM theoretical
7.5 0 15
GSM8K
20
25
30
35
0
20
25
30
7.5 35 0 0.6
0.5
30
40
50
20
25
30
35
0 0.20
30
40
50
15
20
25
30
0
0 20 0.7
30
40
0.5 0 20 0.275 0.250 0.225
50
0.15
Memory consumption (GB)
7 0 20 14
25
30
35
40
25
30
35
40
25
30
35
40
25
30
35
40
0.6
0.2 0.1 0
Gemma-4-26B-A4B
12
0.5
0.4 0 15
FP16
8
8.0
11
MxMoE theoretical 9
4.0
10 0 15 0.6
MATH
Mixtral-8x7B 4.5
8.0
C4 PPL
Wikitext PPL
Qwen1.5-MoE-A2.7B
MxMoE real
30
40
50
Memory consumption (GB)
0 20
Memory consumption (GB)
Figure 5: Quality-memory tradeoff across three MoE models and four evaluation tasks. • Perplexity Benchmarks: On Wikitext2 and C4, PagedWeight reaches the near-FP16 region at 16 GB on Qwen1.5-MoE-A2.7B, 35 GB on Mixtral-8×7B, and 22 GB on Gemma-4-26B-A4B. APL loses more quality at the smallest memory consumption. DP-LLM reaches similar quality only with higher theoretical and real memory consumption. MxMoE shows better quality than the uniform quantization with theoretical memory consumption, but performs still worse than PagedWeight. • Reasoning Accuracy Benchmarks: GSM8K and MATH-500 show the same trend as perplexity. Under the same memory consumption, PagedWeight consistently achieves higher accuracy than the baselines. The improvement is more significant under tighter memory budgets. 5.2
Long-Context Quality under Large KV-Cache Demand
Long-context serving keeps a large KV cache on GPU. To answer RQ2, we evaluate whether PagedWeight preserves long-context quality under large KV-cache demand. We use Qwen1.5-MoE-A2.7B and three LongBench tasks: Passage Retrieval, NarrativeQA, and QMSum. Higher scores are better. Overall Trends. Table 2 shows that PagedWeight preserves long-context quality with much lower memory consumption than FP16. At the 10 GB budget, PagedWeight reaches the same 17.0% average score as FP16 while using only 9.86 GB. In contrast, APL reaches only 12.2% at a similar memory consumption. At medium memory consumption, PagedWeight continues to outperform APL. Under the 13 GB budget, PagedWeight reaches 16.5% average score, while APL only reaches 15.4%. The
8
Table 2: Long-context quality under different memory consumption on Qwen1.5-MoE-A2.7B. Each row of APL uses approximately the same memory as the corresponding row of PagedWeight. Method
Memory (GB)
Passage Retrieval
NarrativeQA
QMSum
Average
FP16
35.25
15.5%
11.9%
23.5%
17.0%
APL-3bit APL-4bit APL-5bit APL-6bit
9.40 11.11 12.92 14.97
10.5% 14.0% 13.0% 15.5%
4.9% 7.2% 9.7% 12.0%
21.1% 22.9% 23.4% 23.5%
12.2% 14.7% 15.4% 17.0%
PagedWeight-10GB PagedWeight-12GB PagedWeight-13GB PagedWeight-15GB
9.86 11.82 12.79 14.83
17.5% 14.5% 15.0% 15.5%
10.0% 9.4% 11.2% 11.4%
23.6% 23.3% 23.4% 23.6%
17.0% 15.7% 16.5% 16.8%
largest gains appear under tight memory, where PagedWeight improves Passage Retrieval from 10.5% to 17.5% and NarrativeQA from 4.9% to 10.0%. 5.3
Serving Throughput
This section reports generation throughput in tokens per second (TPS). To answer RQ3, we evaluate whether PagedWeight maintains throughput while using lower memory consumption.
Table 3: Qwen1.5-MoE-A2.7B throughput. Method
B=1 Mem. B=1 TPS B=4 Mem. B=4 TPS
FP16
27.04
67.1
28.17
258.0
Uniform Uniform Uniform Uniform
7.63 9.29 11.15 13.44
134.5 123.5 126.1 123.1
8.75 10.41 12.28 14.56
429.9 413.8 404.1 401.9
Table 3 reports long-decode throughput at sequence length 2048 under batch sizes 1 and 4. The memory columns report measured GPU 7.63 130.1 8.73 419.4 memory consumption, and the TPS columns PagedWeight 9.22 124.8 10.34 404.9 report generation throughput. Overall, Paged- PagedWeight PagedWeight 11.10 122.1 12.28 394.8 Weight matches the throughput of the uniform PagedWeight 13.34 120.1 14.55 385.4 APL-style baseline with similar or lower memory consumption. The largest throughput drop is 3.3% at batch size 1 and 4.1% at batch size 4. 5.4
Ablation Studies
We isolate the effect of each component on Qwen1.5- Table 4: Ablation study of PagedWeight. MoE-A2.7B perplexity. Lower perplexity is better. Ta- Configuration Wikitext2 PPL C4 PPL ble 4 shows that the full PagedWeight system achieves PagedWeight 7.22 10.06 the lowest perplexity. Removing routing statistics or w/o routing statistics 7.26 10.13 prompt residuals degrades perplexity to 7.26/10.13 and w/o prompt residual 7.31 10.19 7.31/10.19, showing that both online measurements help. w/o page movement 7.43 10.33 Removing page movement turns PagedWeight into a w/o global sensitivity 7.46 10.40 static mixed-precision plan and further increases perplexity to 7.43/10.33. Removing global sensitivity falls back to uniform quantization and performs worst.
6
Related Work
LLM KV Cache Management. A key goal of LLM serving systems is to efficiently store and manage the KV cache created during autoregressive generation. Many systems feature some optimizations to better organize and de-fragment the KV cache, such as vLLM’s PagedAttention [17], TTKV [4], LayerKV [31], and DiffKV [34], or offload it to CPU or disk when GPU memory is insufficient [26, 28, 22]. Some systems also compress the KV cache directly using quantization, such as MoQAE and KVTuner [27, 20], or with lossy compression algorithms [34, 14]. These methods optimize the memory consumption of the KV cache. PagedWeight is complementary as it acts on MoE weights, which creates additional available space for a growing KV cache. Post-Training and Runtime Model Quantization. Post-training quantization reduces model memory consumption and bandwidth demand. Many existing works, such as GPTQ [7], SmoothQuant [30], AWQ [21], ARQ [32], and SpQR [2] explore various quantization strategies to improve the amount
9
of compression achieved while controlling accuracy loss. These quantization methods produce a fixed quantized model before serving. To meet runtime objectives, such as dynamic latency targets at serving time, some works dynamically adjust bitwidths globally or layer-wise, such as Any-Precision LLM [24] and DP-LLM [16]. PagedWeight is a higher-level system that focuses on LLM memory management at serving time, adapting dynamic, mixed-precision quantization of MoE weights as one of its key techniques to counter KV cache memory pressure. MoE Serving Systems and Expert-Aware MoE quantization. Mixture-of-Experts (MoE) architectures route input to a mix of multiple experts [18, 6, 13]. Some recent systems optimize MoE serving cost by changing expert placement/residency and prefetching [10, 5, 29, 33, 23, 35], but are not capable of quantization. Quantization is profitable on MoE, since the weights of experts can differ significantly in quantization sensitivity, which motivates expert-aware quantization strategies. Recent systems, including QuantMoE-Bench [19], MiLo [11], MxMoE [3], EAQuant [8], and MoQE [15], explore various such quantization strategies. In parallel to our work, researchers proposed DynaExq and DyMoE [1, 12], which are capable of dynamic MoE quantization. However, PagedWeight allows for much finer-grained control over submodules of experts compared to per-expert control in DynaExq and DyMoE. PagedWeight quantization is guided by real-time memory pressure from KV cache instead of a constant HBM usage cap assumed in DynaExq, or the memory limit of the edge device in DyMoE.
7
Conclusion
We introduce PagedWeight, a runtime memory-management system for MoE LLM serving under KV-cache pressure. PagedWeight treats Any-Precision expert bit-planes and LUTs as weight pages, dynamically adjusts GPU-resident expert precision at safe boundaries, and uses a quality-aware planner to select low-damage reductions. Across three MoE models, PagedWeight improves the quality–memory tradeoff over baselines on language modeling, reasoning, and long-context tasks while maintaining throughput close to the uniform baseline. We demonstrated PagedWeight’s performance on specific quantization formats and sensitivity metrics. Future work includes extending PagedWeight to other quantized weight formats with compatible layouts and kernels, and exploring complementary methods for estimating prompt-wise expert sensitivity to further improve the quality–memory tradeoff under KV-cache pressure. Acknowledgments This research was supported in part by the NSF grants No. CCF-2217144 and CCF-2313028, and the IBM-Illinois Discovery Accelerator Institute. This research used DeltaAI advanced computing and data resource, supported by the NSF (award OAC 2320345) and the State of Illinois.
References [1] Kexin Chu, Dawei Xiang, Zixu Shen, Yiwei Yang, Zecheng Liu, and Wei Zhang. Dynamic expert quantization for scalable mixture-of-experts inference, 2025. URL https://arxiv. org/abs/2511.15015. [2] Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse-quantized representation for near-lossless LLM weight compression, 2023. URL https://arxiv.org/ abs/2306.03078. [3] Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. MxMoE: Mixed-precision quantization for MoE with accuracy and performance co-design. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 14793–14806. PMLR, 2025. URL https://proceedings.mlr.press/v267/duanmu25a.html. [4] Gradwell Dzikanyanga, Weihao Yang, Hao Huang, Donglei Wu, Shihao Wang, Wen Xia, and Sanjeeb K C. TTKV: Temporal-tiered KV cache for long-context LLM inference, 2026. URL https://arxiv.org/abs/2604.19769.
10
[5] Zhiyuan Fang, Yuegui Huang, Zicong Hong, Yufeng Lyu, Wuhui Chen, Yue Yu, Fan Yu, and Zibin Zheng. Klotski: Efficient mixture-of-expert inference via expert-aware multi-batch pipeline, 2025. URL https://arxiv.org/abs/2502.06888. [6] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), January 2022. ISSN 1532-4435. [7] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers, 2023. URL https://arxiv.org/abs/ 2210.17323. ICLR 2023. [8] Zhongqian Fu, Tianyi Zhao, Ning Ding, Xianzhi Yu, Xiaosong Li, Yehui Tang, and Yunhe Wang. EAQuant: Enhancing post-training quantization for MoE models via expert-aware optimization, 2025. URL https://arxiv.org/abs/2506.13329. Gemma-4-26B-A4B. https://huggingface.co/google/ [9] Google DeepMind. gemma-4-26B-A4B, 2026. Hugging Face model card. [10] Xin He, Shunkang Zhang, Kaijie Tang, Shaohuai Shi, Yuxin Wang, Zihao Zeng, Zhenheng Tang, Xiaowen Chu, Haiyan Yin, Ivor W. Tsang, and Yew Soon Ong. ExpertFlow: Efficient mixture-of-experts inference via predictive expert caching and token scheduling, 2026. URL https://arxiv.org/abs/2410.17954. DAC 2026. [11] Beichen Huang, Yueming Yuan, Zelei Shao, and Minjia Zhang. MiLo: Efficient quantized MoE inference with mixture of low-rank compensators, 2025. URL https://arxiv.org/abs/ 2504.02658. [12] Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, and Zibin Zheng. DyMoE: Dynamic expert orchestration with mixed-precision quantization for efficient MoE inference on edge, 2026. URL https://arxiv.org/abs/2603.19172. [13] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https://arxiv.org/abs/2401.04088. [14] Bo Jiang, Taolue Yang, Youyuan Liu, Chengming Zhang, Xubin He, and Sian Jin. KVComp: A high-performance, LLM-aware, lossy compression framework for KV cache, 2025. URL https://arxiv.org/abs/2509.00579. [15] Young Jin Kim, Raffy Fahim, and Hany Hassan Awadalla. Mixture of quantized experts (MoQE): Complementary effect of low-bit quantization and robustness, 2023. URL https: //arxiv.org/abs/2310.02410. [16] Sangwoo Kwon, Seong Hoon Seo, Jae W. Lee, and Yeonhong Park. DP-LLM: Runtime model adaptation with dynamic layer-wise precision assignment, 2025. URL https://arxiv.org/ abs/2508.06041. NeurIPS 2025. [17] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles, SOSP ’23, pages 611–626, 2023. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006.3613165. [18] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb.
11
[19] Pingzhi Li, Xiaolong Jin, Zhen Tan, Yu Cheng, and Tianlong Chen. QuantMoE-Bench: Examining post-training quantization for mixture-of-experts, 2024. URL https://arxiv.org/ abs/2406.08155. [20] Xing Li, Zeyu Xing, Yiming Li, Linping Qu, Hui-Ling Zhen, Wulong Liu, Yiwu Yao, Sinno Jialin Pan, and Mingxuan Yuan. KVTuner: Sensitivity-aware layer-wise mixedprecision KV cache quantization for efficient and nearly lossless LLM inference, 2025. URL https://arxiv.org/abs/2502.04420. ICML 2025. [21] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, 2024. URL https://proceedings.mlsys.org/paper_files/paper/ 2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html. [22] Mao Lin, Xi Wang, Guilherme Cox, Dong Li, and Hyeran Jeon. HybridGen: Efficient LLM generative inference via CPU-GPU hybrid computing, 2026. URL https://arxiv.org/abs/ 2604.18529. [23] Qingxiu Liu, Cyril Y. He, Hanser Jiang, Zion Wang, Alan Zhao, and Patrick P. C. Lee. FluxMoE: Decoupling expert residency for high-performance MoE serving, 2026. URL https://arxiv. org/abs/2604.02715. [24] Yeonhong Park, Jake Hyun, Sanglyul Cho, Bonggeun Sim, and Jae W. Lee. Any-precision LLM: Low-cost deployment of multiple, different-sized LLMs. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 39682–39701. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/park24e.html. [25] Qwen Team. Qwen1.5-MoE-A2.7B. https://huggingface.co/Qwen/Qwen1.5-MoE-A2. 7B, 2024. Hugging Face model card. [26] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with a single GPU, 2023. URL https://arxiv.org/abs/2303.06865. [27] Wei Tao, Haocheng Lu, Xiaoyang Qu, Bin Zhang, Kai Lu, Jiguang Wan, and Jianzong Wang. MoQAE: Mixed-precision quantization for long-context LLM inference via mixture of quantization-aware experts. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10810–10820, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.531. URL https://aclanthology.org/2025.acl-long.531/. [28] Abhishek Vijaya Kumar, Gianni Antichi, and Rachee Singh. Aqa: Network-accelerated memory offloading for LLMs in scale-up GPU domains. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’25, New York, NY, USA, 2025. Association for Computing Machinery. doi: 10.1145/3676641.3715983. URL https://doi.org/10.1145/3676641.3715983. [29] Wenfeng Wang, Jiacheng Liu, Xiaofeng Hou, Xinfeng Xia, Peng Tang, Mingxuan Zhang, Chao Li, and Minyi Guo. MoE-SpeQ: Speculative quantized decoding with proactive expert prefetching and offloading for mixture-of-experts, 2025. URL https://arxiv.org/abs/ 2511.14102. [30] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 38087–38099. PMLR, 2023. URL https://proceedings.mlr.press/v202/xiao23c.html.
12
[31] Yi Xiong, Hao Wu, Changxu Shao, Ziqing Wang, Rui Zhang, Yuhong Guo, Junping Zhao, Ke Zhang, and Zhenxuan Pan. LayerKV: Optimizing large language model serving with layer-wise KV cache management, 2024. URL https://arxiv.org/abs/2410.00428. [32] Yuchen Yang, Yifan Zhao, Shubham Ugare, Gagandeep Singh, and Sasa Misailovic. ARQ: A mixed-precision quantization framework for accurate and certifiably robust DNNs. In Proceedings of the 35th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2026. doi: 10.48550/arXiv.2410.24214. URL https://arxiv.org/abs/2410. 24214. [33] Yichao Yuan, Lin Ma, and Nishil Talati. MoE-Lens: Towards the hardware limit of highthroughput MoE LLM serving under resource constraints, 2025. URL https://arxiv.org/ abs/2504.09345. [34] Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C. S. Lui, and Haibo Chen. DiffKV: Differentiated memory management for large language models with parallel KV compaction, 2025. URL https://arxiv.org/abs/2412.03131. SOSP 2025. [35] Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, and Xin Liu. MegaScaleInfer: Serving mixture-of-experts at scale with disaggregated expert parallelism, 2025. URL https://arxiv.org/abs/2504.02263.
13