COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies

COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies Hanming Zhang and Erhu Feng Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University

arXiv:2610.33404v1 [cs.LG] 24 Oct 2026

Abstract

caching, and prefetching mechanisms for deploying MoE models whose weights exceed GPU memory capacity. However, deploying MoE models through offloading is considerably more complicated in practice. This complexity arises from three sources: variations in model sparsity, the diversity of offloading mechanisms, and the heterogeneity of hardware configurations. First, although MoE architectures have been widely adopted by recent LLMs, different models exhibit substantially different routing sparsity and expert-popularity distributions. Consequently, the number of experts activated per token and the distinction between frequently and infrequently accessed experts vary across models. Second, existing offloading systems adopt fundamentally different mechanisms. For example, llama.cpp [32] performs sequential offloading at layer granularity, vLLM [50] uses a unified CPU–GPU address space to access host-resident weights, and KTransformers [3] assigns different categories of model weights to different processors. Finally, hardware heterogeneity further complicates policy selection. MoE offloading often targets edge systems, including mobile phones, personal computers, and AI cubes. These platforms combine diverse CPU, GPU, and NPU architectures, providing no uniform hardware profile. As a result of this complexity, no single offloading strategy performs best across all model–platform configurations. A strategy that performs well in one setting can exhibit substantially different behavior in another. Implementing every candidate offloading strategy on each target platform would incur substantial engineering cost. Therefore, efficiently selecting the best-performing strategy for a given model–platform pair remains a key challenge for edge MoE deployment. We present COPE (Composable Offloading Policy Emulator), which structures the policy space and selects the MoE offloading policy expected to perform best for a target model– platform configuration before runtime integration. COPE answers two questions: (1) How can offloading designs be organized into reusable choices, and (2) which candidate is expected to deliver the best performance? To answer the first question, COPE introduces the Policy Matrix, which decomposes an MoE offloading design into two composable dimensions: expert residency and MoE execution strategy. Expert residency determines where each expert is stored in the memory hierarchy, including static schemes such as layer-wise and hot/cold placement, as well as dynamic schemes that update expert residency at runtime. Expert execution strategy

As LLMs scale, mixture-of-experts (MoE) architectures increase model capacity while keeping per-token computation manageable through sparse expert activation. This sparsity also enables memory-efficient deployment by retaining most expert weights in CPU memory and transferring or executing only the routed experts on demand. Existing MoE offloading systems typically target specific model–platform configurations, yet differences in model sparsity, CPU and GPU throughput, VRAM capacity, and host–device bandwidth can change the policy ranking. Because implementing and exhaustively evaluating all candidate policies incurs substantial engineering cost, efficiently selecting the bestperforming policy for each model–platform configuration remains a key challenge for practical MoE offloading. We present COPE (Composable Offloading Policy Emulator), which selects the best-performing MoE offloading policy for a target model–platform configuration before runtime integration. COPE comprises two components. Its Policy Matrix factors the design space into two independent axes, expert residency and MoE execution strategy, enabling mechanism reuse across candidate policies and exposing four new candidates. Its Virtual Heterogeneous Machine profiles combined-operator schedules on the target platform and combines these measurements with offline routing traces to estimate candidate performance. Compared with the evaluated baselines, COPE improves fastest-policy selection accuracy by 24%–100% and reduces performance-prediction error by 1.7×–6.8×. By identifying the preferred policy before integration, COPE further reduces policy-specific implementation effort by up to 74%.

1

Introduction

Sparse mixture-of-experts (MoE) architectures have become a widely adopted approach to scaling LLM capacity without proportionally increasing per-token computation. A router activates only a small subset of experts for each token, allowing Switch Transformer [14], Mixtral [25], DeepSeek-V2 [9], and Qwen3 [55] to combine a large parameter space with a much smaller active working set. This sparsity also creates a natural opportunity to tier expert weights across the memory hierarchy: CPU RAM can retain the complete expert set, while GPU VRAM holds only a selected subset. Academic systems and production runtimes [13, 27, 47, 54, 57, 64, 67] have consequently developed a range of expert offloading, 1

determines where and how each routed expert is computed, including CPU execution, weight transfer followed by GPU execution, and concurrent CPU–GPU execution, etc. The mechanisms in these two dimensions can be implemented independently and reused across policies. Combining one choice from each dimension yields a concrete offloading policy. This factorization exposes four candidate policies not covered by existing MoE offloading systems. The performance ranking of candidate offloading policies is highly sensitive to both model characteristics and the target hardware platform. Differences in CPU and GPU performance, interconnect bandwidth, synchronization overhead, and model sparsity can change which policy performs best. Selecting the best policy for each platform by implementing and benchmarking every candidate on real hardware would incur prohibitive engineering costs. Therefore, to answer the second question, COPE provides the Virtual Heterogeneous Machine (VHM). VHM profiles atomic compute, datatransfer, synchronization, and asynchronous-dispatch operators on the target hardware, as well as combined-operator schedules that capture their interactions. These measured costs absorb differences in accelerator speed, bandwidth, and overlap. Given the profile and timing-free routing traces containing routed and predicted expert IDs, VHM replays each candidate’s cache states and computes its predicted TPOT without consulting hardware specifications again. Across hardware configurations, COPE reuses the same offloadingpolicy mechanism in VHM, thereby reducing the engineering effort for new target platforms. We evaluate COPE’s policy-selection accuracy and its potential to reduce engineering effort through two complementary implementations. First, we implement nine Policy Matrix candidates, including those newly exposed by COPE, in llama.cpp [32]. We benchmark these implementations on three CPU–GPU platforms (A100, RTX 4090, and RTX 5090) and across 11 additional controlled A100 resource configurations. The end-to-end measurements serve as ground truth for policy selection. Second, we implement the same nine policies in the Virtual Heterogeneous Machine using existing computation-graph and operator interfaces, and compare its TPOT predictions with the measured ground truth. COPE selects the measured minimum-TPOT candidate in 134/150 A100, 20/20 RTX 4090, and 42/45 RTX 5090 cases. Compared with the Hierarchical Roofline Model [2], COPE reduces TPOT-prediction MAPE across the evaluated models from 61.53–68.52% to 9.39–17.43% on the RTX 4090, and from 47.21–53.92% to 7.95–13.34% on the RTX 5090. By identifying the best-performing candidate before runtime integration, COPE avoids unnecessary policy-specific engineering effort. For Qwen-3 on the RTX 4090, implementing only the COPEselected Fixed Parallel policy reduces policy-specific code by 74% compared with implementing all candidates using the Policy Matrix’s shared mechanisms.

2

Background

2.1

MoE Inference with Offloading

An MoE layer routes each token to a top-𝑘 subset of its experts. Sparse activation reduces per-token arithmetic, but any expert may be selected by a later token, so the complete parameter set must remain available. In a partial-fit deployment, CPU RAM holds all model weights while GPU VRAM holds attention, activations, the KV cache, workspace, and only some expert weights. A router selection without a GPU copy becomes a cache miss, which incurs host-side work or weight movement. A cache miss exposes several exact-computation paths. Keeping a whole MoE layer on the CPU avoids repeated weight transfers but runs even popular experts there [32]. With a partial GPU cache, a resident expert runs on the GPU while a miss runs on the CPU and returns its result [27]; alternatively, the selected weights are staged to the GPU, trading host computation for weight movement [13]. Mapped host weights permit GPU reads without explicit staging, but those reads still cross the interconnect [35, 50]. All four paths retain the router choice and stored precision. Overlap changes, but does not eliminate, these costs. GPU hits may run while CPU misses compute or weights transfer; asynchronous copies need pinned host buffers, and actual overlap depends on hardware and synchronization [34]. Every selected result must join before the layer finishes. Expert size, CPU/GPU throughput, link bandwidth, and cache hits therefore jointly determine the fastest path. This trade-off is acute in local batch-1 decode. Sparse activation permits models whose total weights exceed VRAM, while local execution keeps prompts and outputs away from remote services. General-purpose serving runtimes such as vLLM, SGLang, and TensorRT-LLM optimize broader serving efficiency [29, 37, 62], not this particular cache-miss decision. With one token in flight, CPU work, weight movement, GPU work, and coordination remain visible in time per output token (TPOT). We therefore study batch-1 decode when the model fits in host memory but not entirely in VRAM. We say a model– platform fixes the model, device, and runtime; a configuration is a resource slice of one platform.

2.2

Existing Offloading Policies

Existing MoE offloaders accelerate expert computation by changing where experts run or by modifying what the checkpoint computes. In the first case, a routed expert can run on the CPU or have its weights staged for GPU execution [13, 27]. Other approaches skip selected experts [33], substitute a cached expert for the router’s choice [66], or quantize expert weights [48]; these changes may alter the checkpoint’s outputs. We study exact-routing offloading: the checkpoint’s 2

router decides which experts run, every selected expert executes, and its stored precision is retained. Figure 1 contrasts these choices. Expert placement and prefetching can further reduce cache misses without changing the model’s computation. Fiddler uses expert popularity measured on a calibration workload to keep frequently selected experts in VRAM [27]. Lookahead offers another opportunity: DejaVu observes that representations change slowly across neighboring Transformer layers [31], while ProMoE predicts future MoE expert use from intermediate states and prefetches their weights [47]. Such predictions change which weights are cached, not which experts the checkpoint router eventually selects. We therefore also study the effects of placement and prefetching on batch-1 decode.

These examples show breadth; every target must pass its model–device gates. Policy rankings also cannot be extrapolated between models or configurations. The real question is which feasible template to integrate for a given model– platform without first implementing them all. 2.4

Existing simulators target broader deployment questions. Vidur explores inference scheduling, batching, and tensorparallel configurations [1]; LLMServingSim 2.0 and Frontier model heterogeneous or disaggregated serving, including multi-GPU parallelism and prefill–decode or attention–FFN separation [5, 15]. SimAI and ASTRA-sim2.0 instead emphasize large-scale training, collective communication, and GPU-interconnect modeling [51, 52]. Although Frontier and LLMServingSim 2.0 support MoE routing or expert parallelism, their evaluations do not address our central question: how different MoE offloading policies affect inference performance on memory-constrained edge devices. In particular, neither simulator integrates and compares the multiple MoE offloading policies in Figure 4 within a common model. COPE targets this narrower, policy-level prediction problem rather than general multi-GPU serving or training simulation.

GPU quantized

miss scheduler

CPU

+

router

GPU experts

CPU experts

skip

compute

copy

Figure 1. Expert offloading choices. RAM retains all experts; VRAM caches some copies. Solid paths preserve the checkpoint: a hit runs on the GPU, while a miss runs on the CPU or stages to GPU. Dashed rerouting, a crossed-out skip arrow, and a smaller quantized CPU weight illustrate alternatives outside our scope.

2.3

LLM simulators.

3

Observations

3.1

Observation I: Model–Platform Dependent Policy Performance

The Winning Policy Changes with the Model and Platform. The best MoE offloading policy depends on which operations lie on the decode critical path. CPU and GPU throughput, interconnect bandwidth, and VRAM capacity determine the costs of computation, staging, and expert residency. Expert size, the number of routed experts per token and routing locality determine how frequently these costs are incurred. Changes in either the resource configuration or the routing workload can therefore shift the bottleneck and reverse the ordering of candidate policies. Figure 2 isolates resource sensitivity for DeepSeek-CoderV2-Lite on a single A100. With the model and physical GPU unchanged, the measured winner shifts as the CPU resource configuration changes. Mutable cost-selected wins with one CPU worker, fixed Parallel with 16 CPU workers, and mutable Parallel with two workers and a larger 𝐾 = 32 expert cache. This comparison shows that resource availability alone can reorder the candidate policies. We further validate this configuration-dependent behavior across multiple models and hardware platforms. Sweeping CPU workers, requested MPS share, and VRAM capacity across three models produces multiple winners (Figure 7 and Table 2). Changing the physical platform produces a further shift. Across the evaluated A100 configurations, candidates using per-expert GPU residency achieve the lowest measured TPOT. At the evaluated RTX 4090 configuration, with a fixed

A Growing Space of Models and Configurations

Target platforms span consumer GPUs such as RTX 3090, RTX 4090, and RTX 5090, workstation accelerators such as L40S, and other vendors’ devices. GPU throughput, VRAM, and host–device bandwidth do not scale together; neither do CPU core count and memory bandwidth. Even one platform admits different CPU-worker, GPU-share, and VRAM-budget configurations. Models vary independently along four policy-critical quantities. Mixtral uses 8 experts/top-2 and DeepSeek-V2-Lite 64/top-6. Qwen3-30B-A3B uses 128/top-8; Qwen-3.5-35BA3B uses 256/top-8 [9, 25, 41, 55]. The latter three have 26, 48, and 40 MoE layers. F16 split gate/up/down experts occupy 16.5 MiB in our DeepSeek-V2-Lite target and 6 MiB in Qwen-3.5, while Mixtral’s geometry implies about 0.18B parameters (336 MiB F16) per expert FFN. Total/active parameters range from about 16B/2.4B for DeepSeek-V2-Lite to 46.7B/12.9B for Mixtral [9, 25]. These four quantities determine cache capacity, miss work, transfer bytes, and how many layers repeat them. 3

Relative throughput (%)

Layer partition Fixed + Parallel Fixed + Wait-then-GPU

100

Fixed + Ready-first Fixed + Cost-selected Mutable + Parallel

✓ fastest ✓ fastest

selected by the current layer are computed and how computation, weight transfers, and synchronization are coordinated. Combining one choice from each dimension defines a concrete candidate within the declared space. The two decisions are separable as policy choices because they govern different points in the decode sequence. A layer begins expert execution with its residency state already determined; prediction and cache updates launched alongside that computation prepare later layers. Execution can therefore use the current resident set without constraining how it was chosen. Thus 𝑟 residency choices and 𝑒 execution choices yield 𝑟𝑒 combinations from 𝑟 + 𝑒 reusable modules, although shared resource use still requires evaluating each combination’s timing. Scope of the Policy Space. The Policy Matrix is further restricted to policies designed to preserve model output quality. Each candidate follows the model’s original routing decisions, executes every selected expert, and retains the original expert precision. Methods that prune or dynamically skip experts [33], substitute a router-selected expert with a cached alternative [66], or execute cache-miss experts at a lower precision [48] alter either the set of executed experts or their numerical representation. Such methods therefore fall outside the scope of the current Policy Matrix.

Mutable + Wait-then-GPU Mutable + Ready-first Mutable + Cost-selected

✓ fastest

75

50

25

0

K=16 MPS=100% t=1

K=16 MPS=100% t=16

K=32 MPS=100% t=2

Figure 2. DeepSeek-Coder-V2-Lite policy landscape for three resource slices on one A100. Throughput is normalized to the fastest candidate in each slice; color identifies execution and hatching identifies residency. CPU allocation and full GPU compute share, layer partition is fastest for three of the four models, while Fixed Parallel wins for Qwen3. Simple Policies Can Beat Complex Ones. The controlled A100 study evaluates whether more adaptive offloading mechanisms consistently reduce TPOT. The results show that additional adaptivity does not always translate into better performance. Fixed Parallel achieves the lowest measured TPOT in both 16-worker configurations, outperforming candidates that use mutable residency or cost-selected execution. Even layer partition, the simplest candidate, is only 12.2–25.0% slower than the winner (Section 6.2). These results show that additional policy mechanisms do not necessarily justify their runtime overheads. Although COPE optimizes TPOT rather than engineering complexity, simple designs must remain explicit candidates because they can be competitive or even optimal under specific resource configurations. 3.2

4

Design

Observation I shows that the fastest offloading policy depends on both the model and the platform. COPE therefore separates policy specification from configuration-specific performance prediction (Figure 3). The Policy Matrix factors monolithic offloading designs into two independently specifiable choices: expert residency, which determines which experts occupy GPU memory and how the cache is updated for future layers; and the MoE execution strategy, which determines where and how the current layer’s routed experts execute. The Virtual Heterogeneous Machine (VHM) then estimates the TPOT of the resulting candidates without running each candidate end to end. It profiles atomic operators and short combined-operator schedules on the target platform, while an offline, timing-free routing trace records the model’s routed and predicted expert IDs. VHM replays each candidate’s cache states using the trace and composes the measured costs into a whole-model TPOT estimate.

Observation II: Composable and Executable Policy Space Modeling

Policies Decouple into Two Finite Dimensions. Prior systems have explored a range of MoE offloading mechanisms, including demand-driven expert loading, CPU–GPU co-execution, proactive caching, and joint scheduling and cache management [13, 27, 47, 57, 64, 67]. Although these systems adopt different mechanisms, they are typically presented as individual end-to-end designs. To our knowledge, prior work does not provide a shared abstraction for reasoning about which combinations have been explored and which remain uncovered within a clearly stated policy space. We find that the surveyed designs can be factored into two independently specifiable dimensions: expert residency and MoE execution strategy. Expert residency determines which experts occupy GPU memory when a layer is reached and how the resident set is updated for subsequent layers. The MoE execution strategy determines where the experts

4.1 Modularizing Offloading Policies with the Policy Matrix Existing MoE offloaders are typically presented as monolithic designs, making the space of alternatives difficult to navigate. On a new model–hardware configuration, finding the fastest design would require implementing and testing each existing system. Even then, an unordered list of systems offers little indication of which promising combinations have yet to be explored. 4

1

a) Shared Modules: Attention, KV Cache, Dense FFN, LM Head, same execution strategy for every candidate b) MoE Execution Strategies: Parallel CPU+GPU, Wait-then-GPU, Ready-first-GPU, Cost-selected

FFN Offloading Policy Core

3

2 Attention

Attention

{

MoE Execution Prediction & Prefetch

Attention MoE Execution

CPU

{

Shared Modules

GPU

PCIe

Trace-driven Routing

}

c) Expert Residency: Fixed, Mutable per-layer, Mutable cross-layer

5

Atomic Operators Profile

Profile

Summarize

}

Atomic Operators: Compute (GEMM/GEMV), Host–Device Copy (HtoD/DtoH), Asynchronous Dispatch, Backend Synchronization

{

}

4 Offline, Timing-free Routing Trace Layer 𝑖 MoE Router

0 1 2 3

···

Layer 𝑗 MoE Router

0 1 2 3

···

Layer 𝑖 MoE Router

0 1 2 3

decode token 𝑘

···

Layer 𝑗 MoE Router

decode token 𝑙

0 1 2 3

···

Routing Trace

VHM Cache Replay Trace-driven Prefetch

Profile-driven Execution Cost

. . . Shared Modules Trace-driven Routing

Trace-driven Prefetch

Profile-driven Execution Cost

. . .

Figure 3. Overview of COPE. The routing trace contains routed and predicted expert IDs; the panel illustrates routed IDs across decode steps. The atomic-operator profile and routing trace are the only inputs to VHM cache replay. The Policy Matrix exposes two decisions within these designs: expert residency and MoE execution strategy. Residency determines which experts occupy GPU memory when a layer is reached, and whether to update that cache in anticipation of later layers. Execution determines where the experts selected by the current layer’s router run, and how their computation and weight movement are coordinated. Within the Policy Matrix, four execution strategies and three residency families cover the mechanisms in our scope (Figure 4). Resident experts run on the GPU; a nonresident expert either runs on the CPU and returns its result, or moves its complete weights to the GPU before execution. Parallel sends all misses to the CPU. Wait-then-GPU stages misses before GPU computation; ready-first overlaps resident computation with staging. Cost-selected uses estimated costs to divide misses between CPU and staged GPU execution. On the residency axis, Fixed retains its initially loaded experts; Mutable per-layer permits replacement within each layer’s budget; and Mutable cross-layer permits replacement across layers. A Fixed cache may be filled in expert-ID order or in popularity order, ranked by historical routing frequency [27]; neither choice changes its no-replacement rule during decode. Figure 5 contrasts the execution orders: equal cache-hit and miss counts need not imply equal TPOT when computation and transfers overlap differently. The resulting policy space is the product of residency and execution choices: Poffload = Presidency × Pexecution . Mapping prior work onto this product reveals combinations not represented by the surveyed systems. COPE instantiates the Fixed and Mutable per-layer rows, yielding eight cache policies; the cross-layer row remains a classified design family. Both instantiated cache families begin with a popularity-ordered fill of each layer’s expert slots. We evaluate whole-layer partition separately as a ninth candidate. Because these combinations share expert computation and weight movement, VHM can profile their common operations and compose policy

estimates before each candidate is integrated as a production offloader. The matrix describes policies that preserve the model’s computation during batch-1 decode, even when only part of its expert set fits in VRAM. Each candidate follows the checkpoint’s router, executes every selected expert, and retains its stored precision. Methods that change routing, expert representation, or the set of executed experts fall outside this space [16, 23, 48, 66]. COPE selects the supported candidate with the lowest predicted TPOT, using a fixed order only to resolve an exact tie. 4.2

Predicting Offloading Performance with the Virtual Heterogeneous Machine

The Policy Matrix defines the candidates; VHM estimates their TPOT to select the predicted winner for each model– hardware configuration. The two axes organize the search, but building and testing every choice would still be costly. VHM instead works with reusable operations beneath those choices. Their measured costs alone are not enough, however: a policy’s latency also depends on which experts the model routes to, which of them hit in the cache, and which prefetches update later layers. VHM reduces profiling to atomic operators shared across policy choices and short combined-operator schedules that preserve their interactions. The operators capture computation, data movement, and synchronization on the target platform. Separating component costs from execution modeling also underlies inference simulators such as Vidur, LLMServingSim 2.0, and Frontier [1, 5, 15], as well as training simulators such as SimAI and ASTRA-sim 2.0 [51, 52]. VHM applies this approach to expert offloading, where cache residency and CPU/GPU execution determine which operations run and how they interact. Our analytic baseline adapts the Hierarchical Roofline Model (HRM) from MoE-Lightning [2]; direct measurements capture overheads its estimates omit 5

Policy Set (Naive, Unstructured)

{ Offloading Policy 1, Offloading Policy 2, Offloading Policy 3, ... } Parallel CPU+GPU

Execution Strategy

GPU

Residency

Policy Matrix (Composable, Structured) Ready-first GPU

Wait-then-GPU GPU

+

CPU

+

GPU

GPU

HtoD

+

Cost-selected GPU + HtoD CPU

GPU

HtoD

Fixed fill by index or popularity-order

𝐿𝑗 𝐿𝑖

KTransformers [3] llama.cpp𝑎 [32]

MoE-Lightning (batch-𝑛 ) [2]

Discovered by COPE

Fiddler [27]

Mutable per-layer

𝐿𝑗 𝐿𝑖

Discovered by COPE

Mixtral-Offloading [13] DuoServe-MoE [60]

ProMoE [47]

HybriMoE [64] DALI [67]

Mutable cross-layer

𝐿𝑗 𝐿𝑖

Discovered by COPE

MoE-Infinity𝑏 [54]

Discovered by COPE

FreeToken [57]

GPU

+

𝑎 MoE-only layer partition: 𝑅 = ∅ or 𝐸 ; one expert branch per layer, separately evaluated. 𝑙 𝑙 𝑏 Paper-level demand loading; resident-hit launch order is not established for the full wait-then-GPU execution sequence.

Outside this matrix: GPU remote reads (UVA) [50]; module prefetch [45, 49]; approximate routing.

Figure 4. Policy Matrix: three expert-residency families crossed with four canonical MoE execution schedules. Under Fixed residency, the initial set may follow expert-ID or offline popularity order [27]. Prior systems’ workload and residency granularity may differ from COPE’s batch-1, per-expert implementations. dependency order Earlier MoE layers: experts on CPU Layerwise offloading

Withinlayer expert offloading

CPU/GPU layer boundary

(a) Layer partition

GPU CPU D2H H2D

attention

(b) Parallel CPU + GPU

GPU CPU D2H H2D

attention

(c) Waitthen-GPU

GPU CPU D2H H2D

attention

(d) Readyfirst GPU

GPU CPU D2H H2D

attention

resident experts

(e) Costselected

GPU CPU D2H H2D

attention

staged resident experts selected CPU misses

attention

routed experts

act.

Later MoE layers: experts on GPU all routed experts

result resident experts

act.

merge

miss experts result

staged misses

resident experts

merge

miss-expert weights merge

staged misses

miss-expert weights

act. selected GPU weights

GPU expert computation

CPU expert computation

Activation/result copy

merge

merge result

Expert-weight H2D copy Dependency / synchronization

Figure 5. The layer-partition baseline and four cache execution schedules. Parallel sends misses to CPU; wait-then-GPU stages every miss; ready-first overlaps resident GPU work with staging; cost-selected assigns miss groups to CPU or staged-GPU execution. Prefetch is omitted. (Figure 6). The combined-operator schedules then capture what isolated measurements cannot: whether operations overlap, serialize, or contend when a policy executes. This gives VHM policy-relevant costs without timing a production offloader. VHM then uses an offline routing trace to drive a replay of each layer’s expert cache, turning those measured costs into a whole-model TPOT estimate. For each layer, VHM compares the routed experts with the current cache to identify hits and misses, then looks up the corresponding execution cost. For mutable residency, traced predictions and the next layer’s cache state likewise determine whether a prefetch is needed and its associated cost. VHM combines these costs with the work shared by all policies and aggregates them across layers and traced tokens. Once the target’s profile

and the trace are given, this composition needs no further hardware parameters; a new configuration changes the measurements, not the replay. 4.2.1 Abstracting Hardware with Atomic-Operator Profiles. VHM’s first phase turns the Policy Matrix’s structured choices into a reusable profile of the target model and hardware. Timing every policy end to end would bring back the cost of building every candidate. Instead, VHM decomposes residency and execution into atomic operators shared across choices (Figure 3, parts 2–3). Unlike broader serving and training simulators [1, 5, 15, 51, 52], VHM selects profiling units around the expert-offloading decision. CPU execution of a cache miss, for instance, serves both Parallel and cost-selected; staged GPU execution and expert-weight transfer serve wait-then-GPU, ready-first, and cost-selected. 6

Ground-truth

(a) GPU expert compute

Algorithm 1 VHM replay: predict one policy’s TPOT from the profile and an offline routing trace.

HRM

(b) CPU expert compute

(c) Expert copy

Time (µs)

10000

Input: trace: routed IDs Trace[𝑡, 𝑙] predicted next-layer IDs Pred[𝑡, 𝑙] profile: Shared, MoE[𝑒, ℎ], Prefetch[𝑟, 𝑞] policy 𝑝 = (𝑟, 𝑒): residency 𝑟 , execution 𝑒 Output: predicted TPOT of 𝑝 1: 𝑀𝐶𝑙 ← cached experts of layer 𝑙 2: acc ← 0 3: for 𝑡 = 1 to 𝑇 do 4: acc ← acc + Shared(𝑛) ⊲ shared work 5: for 𝑙 = 1 to 𝐿 do 6: ℎ𝑙 ← |Trace[𝑡, 𝑙] ∩ 𝑀𝐶𝑙 | ⊲ cache hits 7: acc ← acc + MoE[𝑒, ℎ𝑙 ] ⊲ expert execution 8: if 𝑟 = Mutable per-layer and 𝑙 < 𝐿 then 9: 𝑄 ← Select𝑟 (Pred[𝑡, 𝑙] \ 𝑀𝐶𝑙+1 ) ⊲ prefetch 10: 𝑀𝐶𝑙+1 ← Update(𝑀𝐶𝑙+1, 𝑄) 11: acc ← acc + Prefetch[𝑟, |𝑄 |] 12: end if 13: end for 14: end for 15: return acc/𝑇 ⊲ TPOT

1.5×

100

7500

400

2.4×

4.2× 5000 50

200 2500

0

0

1

2

3

4

Nexpert

5

6

0

0

1

2

3

4

Nexpert

5

6

0

0

1

2

3

4

5

6

Nexpert

Figure 6. Measured atomic-operator latency and HRM estimates for DeepSeek-V2-Lite on RTX 4090 with 32 CPU threads. The profile therefore measures common ingredients rather than each complete offloader. VHM measures atomic-operator costs on the target instead of inferring them solely from idealized compute and bandwidth limits. Our analytic baseline adapts MoE-Lightning’s Hierarchical Roofline Model (HRM) [2] to these operators. Figure 6 shows why direct measurement matters: CPU and GPU expert profiles incur nonzero costs even with no routed experts, where the analytic estimate is zero. Expert staging also includes preparation and readiness beyond weight transfer; at six experts, its measured cost is 2.4× the analytic estimate. Such probes capture costs that a roofline bound omits while remaining reusable across policies. Yet an accurate isolated cost is not necessarily an accurate combined cost. To further capture interactions that isolated measurements miss, VHM also profiles short combined-operator schedules. In ready-first, resident GPU work overlaps expert staging; Parallel and cost-selected may overlap CPU and GPU expert work. Wait-then-GPU instead holds GPU computation until staging finishes. These dependencies, as well as contention and dispatch, can make the observed duration differ from a sum or maximum of isolated costs. VHM measures the relevant operations together, under their intended ordering and concurrency. The resulting schedule costs, rather than an arithmetic assembly of isolated timings, inform the subsequent TPOT estimate. VHM profiles policy-invariant decode work once and reuses it for every candidate. Attention, KV-cache work, dense FFNs, and output projection follow the same execution path regardless of residency or MoE execution strategy. Their shared profile varies with the target and context length, but not with the offloading choice. Together with execution and prefetch schedule costs, it provides the hardware-dependent inputs to trace-driven TPOT composition.

profiled execution cost applies. Predicted IDs identify possible additions to the next layer’s cache and the associated prefetch cost; they need not match the experts later routed to that layer. Both kinds of IDs are collected without timing a production offloader. Once this trace and a target-measured profile are supplied, replay requires no further hardware parameters. VHM replays each candidate as a sequence of per-layer cache states (Algorithm 1). The cache policies start with each layer’s top-𝐾 experts in popularity order; layer partition instead places every expert in its later GPU-resident layers. For each token and layer, it counts routed cache hits and retrieves the execution cost for the candidate’s strategy. Mutable residency also uses the traced next-layer predictions to select at most one novel expert for prefetch, updates that layer’s cache, and accounts for prefetch cost. The compact cost lookups in the pseudocode stand for measured schedules, including any CPU/GPU assignment within cost-selected; they do not add isolated operator timings. Replay assumes a selected prefetch completes before that layer is used, while prediction errors remain visible when routed IDs differ from predicted IDs. Summing execution and prefetch costs with shared decode work across tokens and layers gives the predicted TPOT. A profile can be reused only while its target operating point remains valid; trace changes and platform changes have different remedies (Table 1).

4.2.2 Composing TPOT from Atomic Operator Profiles and Offline Traces. The offline routing trace contains both routed IDs (the experts actually routed to) and predicted IDs (the experts predicted for the next layer), allowing VHM to estimate cache-dependent TPOT. Routed IDs reveal which experts hit the current layer’s cache and thus which

5

Implementation

For external validation, we implemented nine Policy Matrix candidates in llama.cpp and evaluated their end-to-end paths 7

Table 1. Action required after an input changes. Action

and two physical RTX devices. We report absolute TPOT error, rank correlation, and top-1 selection separately. Top-1 means that predicted and measured minimum TPOT name the same candidate; complexity and prediction intervals do not enter it. We also assess prediction and calibration cost.

Change

Replay

Another candidate or a workload with an existing valid trace Retrace Changed workload or predictor without a valid trace Recalibrate Changed model or target operating point, including cache budget and compute/transfer resources Unsupported Missing capabilities required by the candidate

6.1

Platforms. We evaluate COPE on three GPU platforms: an A100-SXM4-80GB, an RTX 4090, and an RTX 5090. The A100-SXM4-80GB serves as the platform for our controlled resource study. On the same machine, we vary the number of CPU workers, the verified CUDA-MPS compute share [36], and the total-VRAM cap. These settings provide sensitivity controls rather than emulating other GPUs. The 24-GiB RTX 4090 is connected to two Xeon Gold 6430 CPUs over a Gen4-x16-capable link. We evaluate one full-compute-share operating point using the largest common feasible expert cache for each model, under an admitted host-memory envelope of 128 GiB. The RTX 5090 platform provides 32 GiB of VRAM and 62.2 GiB of physical host memory. We evaluate 16-, 12-, and 8-GiB VRAM reservations, which leave capacity for another application while retaining the full compute share. These reservations do not emulate contention for SMs, host-memory bandwidth, or PCIe bandwidth. We freeze CPU placement, memory limits, and the negotiated interconnect for each physical-device cell. Models. We evaluate four MoE checkpoints: Qwen3.535B-A3B [41, 42], Qwen3-30B-A3B [40, 55], DeepSeek-CoderV2-Lite-Instruct [7, 8], and DeepSeek-V2-Lite-Chat [9, 10]. The A100 experiments use Coder and the two Qwen models, while the RTX 4090 experiments include all four models. The RTX 5090 experiments use Coder, Chat, and Qwen3 Q8_0; Qwen-3.5 F16 is excluded because its combined model and runtime footprint exceeds the available host-memory budget. Baseline. We compare COPE with two estimators that replay identical routing traces and cache states but estimate per-layer latency differently. Our atomic-operator simulation, inspired by profile-based simulators such as LLMServingSim 2.0 [5], composes independently measured operator costs using max for nominal overlap and addition for serial execution. The Hierarchical Roofline Model (HRM) from MoE-Lightning [2] instead derives latency bounds from peak compute, memory, and interconnect throughput. COPE uses its Virtual Heterogeneous Machine to profile both atomic operators and combined-operator schedules. End-to-end measurements from nine llama.cpp implementations provide ground truth and are excluded from all estimators. Workloads. We synthesize long-form prompts spanning dialogue, mathematical reasoning, and software-engineering tasks, drawing on the domains of ShareGPT [43], MATH [19], and SWE-bench [26]. The evaluation prompts are disjoint from predictor training. We measure each policy–prompt

on RTX 4090, RTX 5090, and A100 platforms. This validation target uses the ggml/CUDA backend [18, 32]. Separately, we implemented VHM’s target profiler, routing-trace collector, and profile-driven replay engine, encoding the same nine candidates as execution schedules and cache-state transitions. We evaluate VHM’s prediction accuracy by comparing its candidate TPOT estimates with end-to-end measurements from the llama.cpp paths. The nine llama.cpp production paths serve only as validation and do not contribute to VHM construction. In both replay and production, fixed and mutable residency use the same per-layer popularity order and cache capacity, and therefore start with identical resident experts. Fixed residency preserves this set, whereas mutable residency updates it through prediction and prefetch. The four execution strategies differ in their handling of cache misses. Parallel executes hits on the GPU and all misses on the CPU. Waitthen-GPU transfers all misses before GPU execution, while ready-first overlaps these transfers with resident GPU work. Cost-selected partitions misses between the CPU and GPU, overlapping CPU execution, resident GPU work, and transfers before executing the transferred experts. The eight cache candidates share a bounded pinned buffer sized to concurrent routed work. Layer partition instead executes each complete MoE block on either the CPU or the GPU. 5.1

Expert Routing Trace Collection

We collect expert-routing traces by replaying a set of representative inputs through each MoE model. During policyneutral decoding, a hook records the experts selected by each layer’s router. The trained next-layer expert predictor also reads that layer’s input hidden state and produces IDs for the following layer; the same hook records these predictions. We pair the routed and predicted IDs for each token and layer to form the routing trace used by VHM. Routed IDs determine cache hits during replay, while predicted IDs drive the possible updates to the next layer’s cache.

6

Methodology

Evaluation

We first test whether offloading winners change with the model and resources, then assess winner prediction on A100 8

100

Table 2. Measured TPOT winner at representative A100 model–configuration points. Parentheses give speedup over layer partition, TPOTlayer partition /TPOTwinner . Each cell is tinted with its winner’s policy color from Figures 2 and 7.

50

Configuration

Throughput (tokens/s)

DS-Coder

Layer partition Fixed + Parallel Fixed + Ready-first

Mutable + Parallel Fixed + Cost-selected

DS-Coder

Qwen-3.5

Qwen-3

Mutable + Mutable + Mutable + CPU threads = 1 Ready-first (2.62 × ) Cost-selected (2.78 × ) Cost-selected (6.05 × ) Mutable + Mutable + Mutable + Parallel (1.90 × ) Cost-selected (1.74 × ) Parallel (4.13 × ) CPU threads = 2 Fixed + Fixed + Fixed + CPU threads = 16 Parallel (1.23 × ) Parallel (1.02 × ) Parallel (1.63 × ) Fixed + Mutable + Mutable + MPS = 50% Parallel (1.31 × ) Parallel (1.24 × ) Parallel (1.99 × ) Fixed + Mutable + Mutable + Parallel (1.29 × ) Parallel (1.14 × ) Parallel (1.53 × ) VRAM = 16 GiB

Qwen-3.5 40 20

Qwen-3 50

all nine candidates for DeepSeek-Coder-V2-Lite under four A100 configurations, comprising 540 runs. Figure 2 presents the 405 runs from three of these configurations. The winning candidate changes with CPU resources and cache capacity: mutable candidates win the two low-worker configurations, whereas Fixed Parallel wins both 16-worker configurations. In the latter two cases, layer partition is only 12.2–25.0% slower, while wait-then-GPU and ready-first are 163–364% slower. These results show that resource changes alone can reorder the candidates even for a fixed model and physical GPU, and additional staging or prediction mechanisms may not improve the end-to-end performance.

25

1

2

4 8 CPU threads

16

20

50 80 CUDA MPS (%)

100

16

24 28 32 VRAM (GiB)

Figure 7. Measured throughput of five selected candidates under 10 A100 resource configurations. pair in five fresh processes to reduce run-to-run variation in ground-truth TPOT. Predictor training. We collect model-generated traces on GSM8K [6], Magicoder-Evol-Instruct-110K [24], and UltraChat 200k [22]. For each MoE layer, we record its input hidden state and routed expert IDs. An MLP trained with PyTorch [39] uses one layer’s hidden state to predict the next layer’s routed IDs; training and validation are split by prompt. We also derive each layer’s cache popularity order from the training datasets, without using evaluation prompts. 6.2

6.3

Optimal Policy Prediction on A100

We next examine whether accurate policy selection requires measurements of combined-operator schedules, rather than atomic-operator costs alone. We compare COPE with an atomic-operator baseline across ten A100 resource configurations for each of DS-Coder, Qwen-3.5, and Qwen-3. Each configuration evaluates nine candidates using five prompts and five fresh-process trials. All admitted runs remain within their VRAM caps and avoid swapping. The atomic-operator baseline predicts the latency of each cache layer as

Optimal Policy Diversity on A100

Before evaluating prediction, we ask whether one fixed policy could suffice. Figure 7 crosses three MoE models with one-factor A100 resource sweeps. The CPU sweep uses 1, 2, 4, 8, and 16 workers, while the requested MPS sweep uses 20%, 50%, 80%, and 100%. The VRAM sweep uses caps of 16, 24, and 28 GiB for DS-Coder and 16, 24, and 32 GiB for the two Qwen models. Each sweep holds the other two controls at 8 workers, 100% requested MPS share, and 24 GiB of VRAM, as applicable. The figure plots the same five-policy slice in every panel; Table 2 reports the full nine-candidate argmin at five representative resource configurations. The results expose a mismatch among models, configurations, and policies: four candidates win across the 15 cells. With one CPU thread, DeepSeek picks Mutable ready-first, while Qwen models pick Mutable cost-selected. DeepSeek itself moves to Mutable Parallel at two threads and Fixed Parallel at 16. Yet Qwen-3.5’s 16-thread winner improves on layer partition by only 1.02×. Thus, neither the model nor the configuration alone determines the winner, and policy selection must account for their interaction. Controlled single-model matrix. To isolate the effect of resource configuration from model differences, we evaluate

max(𝑅 [ℎ], 𝐶 [𝑐]) + max(𝑆 [𝑢], 𝐺 [𝑔]) + 𝐴 + 𝐹 + 𝑀 + 1𝑝>0 𝑃 . For layer partition, it estimates CPU layers as 𝐶 [𝑘] + 𝐴 + 𝐹 + 𝑀 and GPU layers as max(𝑅 [𝑘], 𝐺 [𝑘]) + 𝑀. This baseline represents concurrency through the two maximum terms, without measuring the corresponding combined-operator execution schedule. COPE and the baseline otherwise use the same common decode cost, routing traces, replay states, and cost-selected inner assignments. The comparison therefore isolates how the two methods compose the same underlying costs into candidate-level TPOT predictions. We evaluate the resulting predictions at three levels. MAPE measures absolute TPOT error, Figure 8 evaluates the predicted ordering of all nine candidates, and Table 3 reports whether each method selects the candidate with the lowest measured TPOT. Together, these metrics show whether 9

Atomic-op sim.

1.0 0.8 0.6 0.2 -0.2 1.0 0.6 0.2 -0.2

Qwen-3

1.0 0.8 0.6 0.2 -0.2 2

4

8

16

20

CPU threads

50

80 100

CUDA MPS (%)

16

60 40 40 20

20 20

1 COPE DS-Coder Atomic-op sim. HRM COPE Qwen-3.5 Atomic-op sim. HRM COPE Qwen-3 Atomic-op sim. HRM

5/5 0/5 0/5 5/5 5/5 5/5 5/5 5/5 5/5

2

32 – – – 5/5 5/5 5/5 5/5 5/5 5/5

20

Qwen-3.5

40

60

Qwen-3

150

100 75

100

50

50

25 50

100

TPOT (ms)

DS-Coder

MPS VRAM (8, MPS, 24) (8, 100, VRAM)

4 16 20 50 80 16 28

60

150

40

60

80

100

120

Measured TPOT (ms)

Figure 9. RTX 4090 predicted versus measured TPOT for four models, five prompts, and nine candidates per model. Each measured point averages five fresh processes. The dashed identity line and shaded ±10% band provide an absoluteerror reference.

VRAM cap (GiB)

1/5 5/5 4/5 5/5 5/5 5/5 5/5 5/5 5/5 0/5 5/5 0/5 0/5 0/5 0/5 0/5 0/5 0/5 0/5 5/5 0/5 0/5 0/5 0/5 0/5 0/5 0/5 5/5 4/5 5/5 4/5 5/5 5/5 5/5 5/5 – 0/5 1/5 5/5 0/5 5/5 5/5 5/5 5/5 – 0/5 1/5 5/5 0/5 5/5 5/5 5/5 5/5 – 4/5 2/5 5/5 2/5 4/5 5/5 5/5 4/5 – 0/5 5/5 5/5 2/5 5/5 5/5 5/5 5/5 – 0/5 5/5 5/5 2/5 5/5 5/5 5/5 5/5 –

40

Measured TPOT (ms)

Table 3. Top-1 policy-selection accuracy across A100 resource sweeps for DS-Coder, Qwen-3.5, and Qwen-3. The measured lowest-TPOT policy among nine candidates is the ground truth for each prompt. Each 𝑥/5 reports the number of prompts for which the predicted and measured winners agree; dashes indicate infeasible configurations. threads (𝑡, 100, 24)

DS-Chat

60

24 28 32

Figure 8. Mean Spearman rank correlation over the nine candidates on the A100 resource sweeps for DS-Coder, Qwen3.5, and Qwen-3. Higher is better.

(threads, MPS, VRAM) (8, 100, 24)

HRM

DS-Coder

0.8

1

Atomic-op sim.

COPE

(c) Memory

Estimated TPOT (ms)

DS-Coder Qwen-3.5

HRM

(b) GPU

Estimated TPOT (ms)

COPE

(a) CPU

DS-Chat

Qwen-3.5 30

15

15

10

10

5

5

25 20

20 15

top-1 top-2 top-3

Measured

Qwen-3 30

10

10

top-1 top-2 top-3

COPE

top-1 top-2 top-3

Atomic-op sim.

top-1 top-2 top-3

HRM

Figure 10. Measured and predicted TPOT of the three fastest measured candidates on the RTX 4090. 6.4 Optimal Policy Prediction on RTX 4090 and RTX 5090 To determine whether the A100 results extend to a different physical platform, we evaluate COPE on RTX 4090 and RTX 5090 under a fixed CPU allocation and full GPU compute share. The study covers DeepSeek Coder, DeepSeek Chat, Qwen-3.5, and Qwen3. For each model, we evaluate nine candidates using five prompts and five fresh-process trials per candidate. The atomic-operator baseline uses the same calibration primitives and routing-trace replays as COPE, isolating the effect of schedule-level latency modeling. On the RTX 4090, Figure 9 compares predicted and measured TPOT across all candidates, while Figure 10 focuses on the three fastest measured candidates. Table 4 summarizes winner selection, candidate ordering, and absolute prediction error. Across the 20 model–prompt cases, COPE selects every measured winner and recovers 59 of the 60 top-three slots. In comparison, the atomic-operator baseline selects none of the measured winners and recovers 24 top-three slots. COPE also reduces MAPE to 9.39–17.43%, compared with 31.73–40.62% for the baseline. These results show that

schedule-level measurements improve not only latency estimation, but also the policy-selection decision. Across the A100 resource sweeps, schedule-level profiling generally predicts the policy-performance landscape more accurately than composing atomic-operator costs alone. For Qwen-3.5, COPE reduces MAPE from 15.12% to 8.71%, increases mean Spearman correlation from 0.884 to 0.944, and selects the measured winner in 48 of the 50 cases, compared with 36 for the atomic-operator baseline. Table 3 extends the selection comparison to the other two models. On DS-Coder, COPE selects 45 of the 50 measured winners, whereas the baseline selects only 5. On Qwen-3, COPE selects 41 of the 50 measured winners, with a mean Spearman correlation of 0.949 and a MAPE of 29.52%. Across the three models, these results show that combined-operator schedule profiles can recover both measured winners and the broader candidate ordering under changes in A100 resources. 10

Table 4. RTX 4090 selection counts, rank correlation, and candidate error for all estimators. Top-1 compares predicted and measured TPOT argmins; Top-3 is three-policy set overlap; MAPE averages over all prompt–candidate pairs. Estimator

Top-1 Top-3

COPE DS-Coder Atomic-op sim. HRM COPE DS-Chat Atomic-op sim. HRM COPE Qwen-3.5 Atomic-op sim. HRM COPE Qwen-3 Atomic-op sim. HRM

Reserve 16 GiB

DS-Coder

𝜌 MAPE

(a) Residency

(b) Execution Strategy

5000 4000

Qwen-3 (Q8)

3000

-74%

-47%

2000 1000

30

40

20

20

0

20

top-1

Reserve 12 GiB

Top-1 Top-3

COPE 15/15 44/45 0.991 9.10% Atomic-op sim. 0/15 17/45 0.745 23.03% HRM 0/15 17/45 0.745 53.81% COPE 12/15 45/45 0.979 7.95% DS-Chat Atomic-op sim. 0/15 15/45 0.719 24.13% HRM 0/15 15/45 0.736 53.92% COPE 15/15 30/45 0.964 13.34% Qwen-3 (Q8) Atomic-op sim. 15/15 45/45 0.986 19.60% HRM 0/15 45/45 0.962 47.21%

Fixed

10 top-2

top-3

25 20 15 10

top-1

top-2

top-3

top-2

50

25

40

20

30

15

20

top-3

20

top-2

top-2

top-3

top-1

top-2

top-3

top-1

top-2

top-3

30

10

10

5 top-1

top-2

top-3

top-1

top-2

top-3

TPOT at the measured ranks (ms)

Measured

COPE

Atomic-op sim.

All execution strategies

wait-then-GPU+ready-first-GPU cost-selected

Figure 11 compares measured and predicted TPOT for the three fastest candidates, while Table 5 summarizes selection, ranking, and absolute prediction error. COPE selects every measured winner for Coder and Qwen3, and 12 of the 15 winners for Chat. It outperforms the atomic-operator baseline on all three metrics for both DeepSeek models. For Qwen3, both methods select every winner, but their other metrics diverge: the atomic-operator baseline better preserves the candidate ordering, whereas COPE provides lower absolute TPOT error. The RTX 5090 results complement the fixed-configuration RTX 4090 study. COPE shows consistently stronger selection and ranking on the RTX 4090, while the RTX 5090 VRAM sweeps demonstrate that winner selection, candidate ordering, and absolute TPOT error do not always improve together. Taken with the A100 results, these experiments show that prediction quality depends on both the model and the available platform resources, motivating the separate reporting of all three metrics.

20

10

Parallel

Figure 12. Policy-specific lines added to llama.cpp for the fixed-Parallel policy selected for Qwen3 on the RTX 4090, versus implementing every option along each Policy Matrix axis. Shared plumbing and diagnostics are excluded.

40

15

All residency options

recommended policy mutable residency

top-3

10 top-1

20

15

top-1

30

10 top-1

Reserve 8 GiB

Estimator

DS-Coder

5/5 15/15 1.000 17.43% 0/5 5/15 0.678 40.11% 0/5 5/15 0.678 67.38% 5/5 15/15 1.000 12.38% 0/5 5/15 0.678 40.62% 0/5 5/15 0.678 67.77% 5/5 15/15 0.987 9.39% 0/5 5/15 0.729 31.73% 0/5 5/15 0.678 61.53% 5/5 14/15 0.993 13.62% 0/5 9/15 0.776 38.29% 0/5 9/15 0.776 68.52%

DS-Chat

30

Model

𝜌 MAPE

LoC

Model

Table 5. RTX 5090 prediction over three VRAM reservations per model. Denominators are 15 prompts and 45 top-three slots per model.

HRM

Figure 11. Measured and predicted TPOT of the three fastest measured candidates in each of nine model–reservation cells on the RTX 5090. schedule-level profiling preserves policy selection and ranking accuracy on a consumer GPU, rather than only within the controlled A100 resource sweeps. To complement the fixed-configuration RTX 4090 experiment, we evaluate COPE across three VRAM budgets on the RTX 5090. We test DeepSeek Coder, DeepSeek Chat, and Qwen3 with VRAM reservations of 16, 12, and 8 GiB, a fixed CPU allocation, and full GPU compute share. Each configuration includes five prompts and nine candidates. Qwen3 uses a Q8_0 deployment with a separate calibration.

6.5

Engineering Effort and Calibration Cost

Accurate TPOT prediction can avoid implementing policies that will not be deployed. For Qwen3 on the RTX 4090, COPE 11

Table 6. RTX 4090 profiling time in minutes. Atomicoperator times are measured; combined-operator times are projected as 25% of measured exhaustive-schedule time, assuming only replay-reachable states are profiled. Profile

DS-Coder

DS-Chat

Qwen-3.5

Qwen-3

2.8 25.9

2.8 25.9

3.4 43.5

3.5 33.9

Atomic-operator profile Combined ops

Other representation and routing choices. FineMoE moves weights at finer granularity, while ZipMoE applies lossless compression [58, 59]. Both depart from our wholeexpert representation assumption without necessarily reducing accuracy. Approximate routing, by contrast, changes the selected computation. COPE fixes both whole-expert representation and exact routing to compare scheduling and residency within one declared policy space.

selects fixed Parallel. Figure 12 compares the policy-specific lines added to llama.cpp for this choice with those needed to cover every option in each Policy Matrix dimension. The choice would avoid 47% of residency code and 74% of execution code. Across both dimensions, the selected policy requires 3,107 policy-specific lines of code, whereas implementing all options requires 8,276, even when shared mechanisms are counted only once. Shared plumbing and prediction/diagnostic code are excluded from both totals. Extending COPE’s prediction path to a new backend is simpler than implementing its offloading policies. Once the backend supports the atomic operations needed for inference, it only needs to expose those operations through profiling interfaces. VHM can then reuse its policy compositions and trace replay to estimate TPOT without integrating each candidate as a production offloader. Profiling once on the target amortizes calibration across all candidate policies. On the RTX 4090, atomic-operator profiles take 2.8–3.5 minutes across four models (Table 6). Profiling combined-operator schedules expands the measurement space beyond that of individual operators. Exhaustively covering this space takes 25.9–43.5 minutes.

7

8

Discussion

TPOT rather than TTFT. COPE optimizes decode TPOT, not time to first token. Standard MoE execution groups routed tokens by expert for batched or block-sparse matrix operations [17]. Fiddler observes that longer prefills can make CPU expert computation costlier than staging weights for GPU execution [27]. This shifts the CPU/GPU crossover; our batch-1 results do not predict prefill performance. Batch 𝑛 versus batch 1. Batch-𝑛 serving has a different CPU/GPU crossover. Multiple tokens can share an expert [27]; MoE-Lightning studies this regime with layerpaged weights [2]. It requires batch-size-specific probes and may rank policies differently from single-user batch-1 decode. Unmodified computation versus accuracy-for-speed offloading. COPE’s candidates preserve checkpoint routing, execute every selected expert, and retain stored precision. Other systems predict sparsity or change routing [23, 31, 38], change expert precision through quantization [4, 11, 12, 16, 20, 28, 48, 53], or skip, prune, substitute, or merge experts or neurons [21, 30, 33, 56, 61, 63, 65, 66]. Their accuracy– performance trade-offs are not directly comparable with TPOT for an unchanged checkpoint; the three invariants define our exactness and quality requirements. Disk versus host-memory offloading. COPE assumes exact routing, whole experts, and a model fitting in host RAM. FlexGen instead spans GPU, CPU, and disk, staging disk-resident tensors through DRAM [46]. Disk requires another set of templates and probes; on a shared-memory AI SoC, it becomes a capacity tier analogous to host RAM below discrete-GPU HBM.

Related Work

Expert caches. MoE-Infinity, Mixtral-Offloading, ProMoE, DuoServe-MoE, and FreeToken study demand loading, prediction, and overlap for sparse experts [13, 47, 54, 57, 60]. KTransformers uses fixed CPU-routed/GPU-shared placement in its full-accuracy path [3], and SGLang KT integrates that execution approach [44]. Fiddler selects CPU or GPU execution for misses using per-expert input size, while FreeToken splits them by measured host and PCIe bandwidth [27, 57]; HybriMoE and DALI also use workload or cost information [64, 67]. COPE reuses these mechanism families. Its contribution is to organize them as residency/execution candidates and predict which complete template to integrate without using a production candidate’s latency. Adjacent mechanisms. MoE-Lightning pages weights in batches, so its matrix placement captures a mechanism rather than a batch-1 expert cache [2]. vLLM UVA uses GPU remote reads from host memory [50]. vLLM Prefetch and SGLang OffloaderV2 instead stage modules in execution order [45, 49]. These latter mechanisms remain deployment alternatives but do not share the candidate semantics measured by the Policy Matrix.

9

Conclusion

MoE offloading performance depends jointly on expert residency and execution strategy, making policy selection specific to each model–platform configuration. COPE structures this design space with a Policy Matrix and evaluates candidates before integration using a Virtual Heterogeneous Machine. By replacing implementation-intensive trial and error with systematic pre-integration analysis, COPE lowers the engineering barrier to deploying MoE models on heterogeneous, resource-constrained hardware. 12

References

Challenges of ML for Systems (PACMI). Association for Computing Machinery, New York, NY, USA, 25–30. doi:10.1145/3766882.3767173 [16] Elias Frantar and Dan Alistarh. 2024. QMoE: Sub-1-Bit Compression of Trillion Parameter Models. Proceedings of Machine Learning and Systems 6 (2024). https://proceedings.mlsys.org/paper_files/ paper/2024/hash/c74b624843218d9b6713fcf299d6d5e4-AbstractConference.html [17] Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2023. MegaBlocks: Efficient Sparse Training with Mixture-ofExperts. Proceedings of Machine Learning and Systems 5 (2023), 288–304. https://proceedings.mlsys.org/paper_files/paper/2023/hash/ 5a54f79333768effe7e8927bcccffe40-Abstract-mlsys2023.html [18] ggml Contributors. 2026. ggml: Tensor Library for Machine Learning. Source repository. Accessed 25 September 2026. https://github.com/ ggml-org/ggml [19] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track. [20] Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and Xiaojuan Qi. 2025. Mixture Compressor for Mixture-of-Experts LLMs Gains More. In The Thirteenth International Conference on Learning Representations. https: //openreview.net/forum?id=hheFYjOsWO [21] Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Fei Chao, Rongrong Ji, and Liujuan Cao. 2025. Discovering Important Experts for Mixtureof-Experts Models Pruning Through a Theoretical Perspective. In Advances in Neural Information Processing Systems, Vol. 38. doi:10. 52202/085713-4538 [22] Hugging Face H4. 2023. UltraChat 200k. https://huggingface.co/ datasets/HuggingFaceH4/ultrachat_200k. [23] Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. 2024. Pre-gated MoE: An AlgorithmSystem Co-Design for Fast and Scalable Mixture-of-Expert Inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 1018–1031. doi:10.1109/ISCA59077.2024. 00078 [24] Intelligent Software Engineering Lab, UIUC. 2023. Magicoder-EvolInstruct-110K. https://huggingface.co/datasets/ise-uiuc/MagicoderEvol-Instruct-110K. [25] Albert Q. Jiang et al. 2024. Mixtral of Experts. arXiv:2401.04088 [cs.LG] [26] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In International Conference on Learning Representations. [27] Keisuke Kamahori, Tian Tang, Yile Gu, Kan Zhu, and Baris Kasikci. 2025. Fiddler: CPU-GPU Orchestration for Fast Inference of Mixtureof-Experts Models. In The Thirteenth International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/ paper/2025/hash/8cd1ce03ea58b3d7dfd809e4d42f08ea-AbstractConference.html [28] Young Jin Kim, Raffy Fahim, and Hany Hassan Awadalla. 2023. Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness. In Efficient Natural Language and Speech Processing Workshop at NeurIPS. https://neurips2023-enlsp.github.io/ papers/paper_81.pdf [29] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165

[1] Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, and Alexey Tumanov. 2024. VIDUR: A Large-Scale Simulation Framework for LLM Inference. Proceedings of Machine Learning and Systems 6 (2024), 16 pages. https://proceedings.mlsys.org/paper_files/paper/2024/hash/ b74a8de47d2b3c928360e0a011f48351-Abstract-Conference.html [2] Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E. Gonzalez, Matei Zaharia, and Ion Stoica. 2024. MoELightning: High-Throughput MoE Inference on Memory-constrained GPUs. arXiv:2411.11217 [cs.DC] [3] Hongtao Chen, Weiyu Xie, Boxin Zhang, Jingqi Tang, Jiahao Wang, Jianwei Dong, Shaoyuan Chen, Ziwei Yuan, Chen Lin, Chengyu Qiu, Yuening Zhu, Qingliang Ou, Jiaqi Liao, Xianglin Chen, Zhiyuan Ai, Yongwei Wu, and Mingxing Zhang. 2025. KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. Association for Computing Machinery, New York, NY, USA, 1014–1029. doi:10.1145/3731569.3764843 [4] Yuanteng Chen, Yuantian Shao, Peisong Wang, and Jian Cheng. 2025. EAC-MoE: Expert-Selection Aware Compressor for Mixtureof-Experts Large Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 12942–12963. doi:10.18653/v1/2025.acllong.633 [5] Jaehong Cho, Hyunmin Choi, Guseul Heo, and Jongse Park. 2026. LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure. In 2026 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, Piscataway, NJ, USA, 1–14. doi:10.1109/ISPASS69572.2026.00012 [6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 [cs.LG] [7] DeepSeek-AI. 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence. arXiv:2406.11931 [cs.SE] [8] DeepSeek-AI. 2024. DeepSeek-Coder-V2-Lite-Instruct Model Card. Official checkpoint documentation, accessed 25 September 2026. https: //huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct [9] DeepSeek-AI. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434 [cs.CL] [10] DeepSeek-AI. 2024. DeepSeek-V2-Lite-Chat Model Card. Official checkpoint documentation, accessed 25 September 2026. https:// huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat [11] Jianing Deng, Song Wang, Dongwei Wang, Zijie Liu, Tianlong Chen, Huanrui Yang, and Jingtong Hu. 2026. GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs. In Proceedings of the 43rd International Conference on Machine Learning. https://arxiv.org/ abs/2605.23078 [12] Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. 2025. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, 14793– 14806. https://proceedings.mlr.press/v267/duanmu25a.html [13] Artyom Eliseev and Denis Mazur. 2023. Fast Inference of Mixture-ofExperts Language Models with Offloading. arXiv:2312.17238 [cs.LG] [14] William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39. https://jmlr.org/papers/v23/21-0998.html [15] Yicheng Feng, Xin Tan, Kin Hang Sew, Yimin Jiang, Yibo Zhu, and Hong Xu. 2025. Frontier: Simulating the Next Generation of LLM Inference Systems. In Proceedings of the 4th Workshop on Practical Adoption 13

[30] Lujun Li, Qiyuan Zhu, Jiacheng Wang, Xiaoyu Qin, Wei Li, Hao Gu, Sirui Han, and Yike Guo. 2026. Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging. Proceedings of the AAAI Conference on Artificial Intelligence 40, 27 (2026), 22994–23002. doi:10.1609/aaai.v40i27.39464 [31] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. 2023. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 22137–22176. https://proceedings.mlr.press/v202/ liu23am.html [32] llama.cpp Contributors. 2026. llama.cpp. Source repository. Accessed 2 September 2026. https://github.com/ggml-org/llama.cpp [33] Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. 2024. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 6159–6172. doi:10.18653/v1/2024.acl-long.334 [34] NVIDIA. 2026. CUDA Programming Guide: Asynchronous Execution. Accessed 25 September 2026. https://docs.nvidia.com/cuda/cudaprogramming-guide/02-basics/asynchronous-execution.html [35] NVIDIA. 2026. CUDA Programming Guide: Unified and System Memory. Accessed 25 September 2026. https://docs.nvidia.com/cuda/cudaprogramming-guide/02-basics/understanding-memory.html [36] NVIDIA. 2026. Multi-Process Service: Environment Variables. Accessed 25 September 2026. https://docs.nvidia.com/deploy/mps/ appendix-environment-variables.html [37] NVIDIA. 2026. TensorRT-LLM Documentation. Accessed 25 September 2026. https://docs.nvidia.com/tensorrt-llm/ [38] Jie Ou, Yueming Chen, Buyao Xiong, Zhaokun Wang, and Wenhong Tian. 2025. Accelerating Mixture-of-Experts Language Model Inference via Plug-and-Play Lookahead Gate on a Single GPU. Computer Standards & Interfaces 94 (2025), 103996. doi:10.1016/j.csi.2025.103996 [39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An Imperative Style, HighPerformance Deep Learning Library. In Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc., Red Hook, NY, USA, 8024–8035. https://proceedings.neurips.cc/paper/2019/hash/ bdbca288fee7f92f2bfa9f7012727740-Abstract.html [40] Qwen Team. 2025. Qwen3-30B-A3B Model Card. Official checkpoint documentation, accessed 25 September 2026. https://huggingface.co/ Qwen/Qwen3-30B-A3B [41] Qwen Team. 2026. Qwen3.5-35B-A3B Model Card. Official checkpoint documentation, accessed 25 September 2026. https://huggingface.co/ Qwen/Qwen3.5-35B-A3B [42] Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5 [43] Ryoko AI. 2023. ShareGPT52K: Realistic Human-ChatGPT Conversations. https://huggingface.co/datasets/RyokoAI/ShareGPT52K. [44] SGLang Contributors. 2026. KT Expert Parallelism Wrapper for MoE Layers. SGLang source repository. Revision d97b796c, accessed 2026-08-17. https://github.com/sglproject/sglang/blob/d97b796c168bdc764a3c70be388bd199469ec9ba/ python/sglang/srt/layers/moe/kt_ep_wrapper.py [45] SGLang Contributors. 2026. SGLang OffloaderV2. SGLang source repository. Revision d97b796c, accessed 202608-17. https://github.com/sgl-project/sglang/blob/ d97b796c168bdc764a3c70be388bd199469ec9ba/python/sglang/ srt/utils/offloader.py [46] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang.

2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, Honolulu, HI, USA, 31094–31116. https://proceedings.mlr.press/v202/sheng23a.html [47] Xiaoniu Song, Zihang Zhong, Rong Chen, and Haibo Chen. 2024. ProMoE: Fast MoE-based LLM Serving using Proactive Caching. arXiv:2410.22134 [cs.DC] [48] Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, PhengAnn Heng, Chao Li, and Minyi Guo. 2026. MoE-APEX: An Efficient MoE Inference System with Adaptive Precision Expert Offloading. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vol. 2. Association for Computing Machinery, 1185–1200. doi:10.1145/3779212.3790187 vLLM Prefetch Offloader. [49] vLLM Contributors. 2026. vLLM source repository. Revision 1d3a8b9e, accessed 2026-08-17. https://github.com/vllm-project/vllm/blob/ 1d3a8b9e220f1edc77c190c3370cf0f76dfdd2fd/vllm/model_executor/ offloader/prefetch.py [50] vLLM Contributors. 2026. vLLM UVA Offloader. vLLM documentation and source repository. Accessed 2026-08-17. https://docs.vllm.ai/en/ latest/api/vllm/model_executor/offloader/uva/ [51] Xizheng Wang, Qingxu Li, Yichi Xu, Gang Lu, Dan Li, Li Chen, Heyang Zhou, Linkang Zheng, Sen Zhang, Yikai Zhu, Yang Liu, Pengcheng Zhang, Kun Qian, Kunling He, Jiaqi Gao, Ennan Zhai, Dennis Cai, and Binzhang Fu. 2025. SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, Philadelphia, PA, USA, 541–558. https://www.usenix.org/conference/ nsdi25/presentation/wang-xizheng-simai [52] William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. 2023. ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale. In 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, Piscataway, NJ, USA, 283–294. doi:10.1109/ISPASS57527.2023.00035 [53] Zhanhao Xie, Yuexiao Ma, Xiawu Zheng, Fei Chao, Wanchen Sui, Yong Li, Shen Li, and Rongrong Ji. 2025. Automated Fine-Grained Mixture-of-Experts Quantization. In Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, 27024–27037. doi:10.18653/v1/2025.findings-acl.1386 [54] Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. 2024. MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache. arXiv:2401.14361 [cs.LG] [55] An Yang et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] [56] Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. 2024. MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition. In Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics, 10456–10466. doi:10.18653/v1/2024.findingsemnlp.612 [57] Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, and Ion Stoica. 2026. FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution. arXiv:2608.16157 [cs.DC] [58] Yuchen Yang, Yaru Zhao, Pu Yang, Shaowei Wang, and Zhi-Hua Zhou. 2026. ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling. In Proceedings of the 43rd International Conference on Machine Learning (ICML). https: //arxiv.org/abs/2601.21198

14

[59] Hanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang, and Hao Wang. 2026. Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading. In Proceedings of the 21st European Conference on Computer Systems. Association for Computing Machinery, New York, NY, USA, 176–191. doi:10.1145/3767295.3769319 [60] Yuning Zhang, Grant Pinkert, Nan Yang, Yanli Li, and Dong Yuan. 2025. DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance. arXiv:2509.07379v2, revised 9 April 2026. arXiv:2509.07379 [cs.DC] [61] Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. 2025. Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts. In Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, 86–102. doi:10.18653/v1/2025.findings-acl.4 [62] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., Red Hook, NY, USA. doi:10.52202/079017-2000 [63] Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, and Meng Li. 2024. AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference. In Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided

Design. Association for Computing Machinery, Article 51, 9 pages. doi:10.1145/3676536.3676741 [64] Shuzhang Zhong, Yanfan Sun, Ling Liang, Runsheng Wang, Ru Huang, and Meng Li. 2025. HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference. In Proceedings of the 62nd ACM/IEEE Design Automation Conference (DAC). IEEE, San Francisco, CA, USA, 1–7. doi:10.1109/DAC63849.2025.11133274 [65] Yixiao Zhou, Ziyu Zhao, Dongzhou Cheng, Zhiliang Wu, Jie Gui, Yi Yang, Fei Wu, Yu Cheng, and Hehe Fan. 2025. Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixtureof-Experts LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, 15169–15186. doi:10.18653/v1/2025.findings-emnlp.820 [66] Guoying Zhu, Meng Li, Haipeng Dai, Xuechen Liu, Weijun Wang, Keran Li, Jun Xiao, Ligeng Chen, and Wei Wang. 2026. SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution. In 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 1302–1318. doi:10.1109/ ISCA66397.2026.00099 [67] Zeyu Zhu, Gang Li, Peisong Wang, Zitao Mo, Minnan Pei, Zhuoran Song, Xiaoyao Liang, and Jian Cheng. 2026. DALI: A WorkloadAware Offloading Framework for Efficient MoE Inference on Local PCs. arXiv:2602.03495 [cs.DC]

15

Related documents

• COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies#1455885
• Who Should Own the Expert Cache? Kernel-Managed Tiering for Trillion-Parameter MoE Inference#1043174
• Who Should Own the Expert Cache? Kernel-Managed Tiering for Trillion-Parameter MoE Inference#793848
• BaGuaLu-2: Taming Dynamic Memory Fluctuation for Efficient MoE Model Training over 10,000 GPUs#1387352
• BaGuaLu-2: Taming Dynamic Memory Fluctuation for Efficient MoE Model Training over 10,000 GPUs#1374998