arXiv:2609.12978v1 [cs.OS] 11 Sep 2026
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading Zihan Wang
Yuqi Wang
Lei Gong
University of Science and Technology of China Hefei, China [email protected]
University of Science and Technology of China Hefei, China [email protected]
University of Science and Technology of China Hefei, China [email protected]
Cheng Tang
Wenqi Lou
Teng Wang
University of Science and Technology of China Hefei, China [email protected]
Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China [email protected]
Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China [email protected]
Chao Wang
Xuehai Zhou
University of Science and Technology of China Hefei, China [email protected]
University of Science and Technology of China Hefei, China [email protected]
Abstract
1
Mixture-of-Experts (MoE) creates a structural advantage for offloading: only a small fraction of activated experts need to reside in device memory, and if they can be loaded in time for computation, offloading can in principle approach full-load performance, where all model weights reside in device memory. Yet translating MoE’s structural advantage into practical offloading gains remains challenging. We propose SeqMoE to bridge this gap. To maximize expert hits, we build predictive memory management: (i) Sequence-to-sequence prediction. We are the first to recast expert activation prediction as sequence modeling, enabling accurate multi-step, multi-layer forecasts that provide a long and reliable window for downstream decisions. (ii) Joint prefetch scheduling. We formulate prefetch scheduling as Job Sequencing with Deadlines to maximize expected expert hits and improve bandwidth efficiency. (iii) Forecast-driven caching. Leveraging the recursive nature of sequence modeling, we introduce a probabilistic Belady policy for future-aware eviction. To eliminate execution bottleneck, we develop (iv) Graph-compatible offloading runtime. We derive general runtime principles encompassing compute-transparent expert placement and synchronizationfree orchestration disciplines for end-to-end graph capture. With 45% expert residency, SeqMoE averages a 96.97% hit rate and 80.22% of full-load performance, advancing the state of the art in MoE offloading.
Growing demands for personalization, privacy, embodied intelligence, and autonomous driving are pushing large language models (LLMs) toward the edge [4]. However, limited edge-device memory hinders large-model deployment. Offloading inference addresses this mismatch and has emerged as a research focus [26, 28]. Its core principle is to treat device memory (e.g., GPU HBM) as a cache that holds only a subset of model weights, while relegating the rest to larger next-level memory (e.g., CPU DRAM). By extending usable capacity beyond device memory, offloading brings the intelligence of larger models to edge applications. Mixture-of-Experts (MoE), a standard architecture for modern LLMs, is particularly amenable to offloading. As shown in Fig. 1 (a), its structural advantage stems from sparse scaling: MoE layers dominate the model weights (93%), yet only a small subset of experts (6%) are activated at each inference step. This allows non-MoE layers to remain resident in device memory, while all experts reside in next-level memory and only activated ones need to be dynamically loaded into the device cache. With such dynamic sparsity, if all activated experts can be loaded in time for computation, offloading can in principle approach FullLoad performance, where all model weights reside in device memory. This defines the ideal target for offloading inference. However, translating MoE’s structural advantage into practical offloading gains remains challenging. The key bottleneck is load-on-demand stalls caused by expert misses, which expose expensive memory I/O on the critical path. As shown in Fig. 1 (b), eager offloading achieves only about 2% of
Keywords: MoE Offloading Inference, Sequence Modeling
Introduction
Zihan Wang, Yuqi Wang, Lei Gong, Cheng Tang, Wenqi Lou, Teng Wang, Chao Wang, and Xuehai Zhou (a) Structural Advantages for Offloading Comp
Qwen3-30B-FP8 on RTX 4090
Active Experts 8 per layer 1.8 GB
Device Memory
Non-Expert 2.2 GB
MoE-Infinity
ALL Experts 128 per layer 29.0 GB
(Static Offload)
Next-Level Memory
(CPU-GPU)
Attention and others
Qwen3-30B-FP8 (Total 31.2 GB)
E1
E2
Gating / Router
E0
(b) SeqMoE Narrows the Gap to FullLoad
E1
E2
E3
E4
45% Loaded for Single-Batch Decoding
Eager (Naive)
E5
(Trace Match)
2.3 tok/s 11.9 tok/s
Llama.cpp
30.9 tok/s
KTrans
34.2 tok/s
SeqMoE
104.1 tok/s
(Ours)
FullLoad
122.7 tok/s
(Ideal)
(c) Sequence-Level Dependency in Expert Activations: New Opportunities and Challenges
A1
A2
A𝑖𝑖: Expert Activation State of All Layers when Generating the 𝑖𝑖-th Token A3
A4
A5
A6
Seq2Seq Predictor (Transformer, Mamba) Converged (Recall ~92%) A0
A1
A2
A3
A4
A5
A6
Shuffled
A4
A2
A0
A1
A5
Seq2Seq Predictor (Transformer, Mamba) Can Not Converged (Recall ~23%) A3
A6
A4
A2
A0
A1
Figure 1. Realizing MoE’s Offloading Advantage. FullLoad performance. Despite extensive efforts to mitigate this bottleneck, existing systems remain far from FullLoad. In contrast, SeqMoE not only substantially outperforms existing MoE offloading systems, but also can approach FullLoad performance, unleashing MoE’s structural advantage for high-performance, low-memory inference. To achieve this, we take maximizing expert hits as the primary objective and build predictive memory management that jointly optimizes early prediction, prefetch scheduling, and cache residency. As load-on-demand stalls recede, we further develop a graph-compatible offloading runtime to eliminate the emerging execution bottleneck. Activation sequence modeling achieves long overlap windows and high accuracy simultaneously. Expert activation is late-bound, so early prediction is essential for downstream decisions. Mainstream methods stay within a single autoregressive step, predicting deeper-layer expert choices from shallow-layer states. Their accuracy decays rapidly with prediction distance, leaving a prefetch window of at most three layers [32, 34, 38, 42]. A few recent works attempt cross-step prediction, but rely on per-layer conditional activation statistics between adjacent tokens. Such a memoryless, first-order estimate is too inaccurate for prefetch decisions [43, 45]. In contrast, as shown in Fig. 1 (c), we encode the activation state of the full layer stack per generation step and train a Transformer [35] probe to perform next-step prediction. Convergence holds on the ordered sequence and disappears under step shuffling, indicating that expert activation carries learnable sequence-level dependencies [33] (Section 3.1). Therefore, we recast expert activation prediction as sequence modeling, extending the overlap window across autoregressive steps at high accuracy. Activation sequence prediction extends overlap windows, but complicates prefetch scheduling. A longer overlap window relaxes the horizon constraint but not bandwidth: a slow link such as PCIe sustains only a fraction of prefetch tasks, so the
prefetcher must decide which experts deserve bandwidth. Candidates differ along two axes. Prediction confidence is an objective: it quantifies the transfer’s expected contribution to expert hits. Deadline is a constraint: it temporally couples tasks into bandwidth contention. Existing prefetchers overlook the joint scheduling structure and conflate both into independent per-task scores [32, 38], leading to suboptimal selection (Section 3.2). This waste grows as longer overlap windows deepen task queue and enlarge scheduling space. Instead, we formulate prefetching as Job Sequencing with Deadlines [10] to improve bandwidth efficiency. Activation sequence forecasting transforms history-statistics caching into future-aware eviction. Effective caching retains reusable experts, avoiding redundant transfers and reducing both prefetch bandwidth pressure and expert misses. Existing policies typically adapt classical eviction algorithms such as LRU and LFU, collapsing historical activation states into coarse statistics to extrapolate reuse [34, 41]. We compare them against variants of the optimal Belady’s MIN [5] and show that exploiting future activations yields consistently better eviction decisions (Section 3.3). Fortunately, the recursive nature of sequence modeling makes future-aware eviction practical: rather than relying on lossy historical summaries, we roll the historical sequence forward into multistep forecasts to guide cache eviction. As expert misses recede, offloading runtime emerges as the new execution bottleneck. An offloading runtime is essential to coordinate the offloading pipeline with LLM inference. However, existing runtimes introduce substantial coordination overhead that prevents execution from achieving full-load efficiency. The fundamental limitation is their incompatibility with graph-based execution mechanisms [15], such as CUDA Graphs [29], that reduce dispatch overhead by replaying pre-captured operations (Section 3.4). First, dynamic expert placement in the cache hinders fused MoE kernels [36, 44], a key enabler of graph-based MoE execution. Second, prefetching and caching rely heavily on host-device synchronization, which fragment graph execution and diminish its benefits. FreeToken [41]1 explores graph-based offloading but is misaligned with predictive management, which is critical to high expert-hit rates. In contrast, we derive general runtime principles that coordinate predictive memory management with LLM inference for end-to-end graph capture. Building on these insights, we present SeqMoE, an MoE offloading system that delivers low-memory inference approaching full-load performance. To maximize expert hits, we build predictive memory management: (i) Sequence-toSequence (seq2seq) expert activation predictor. We are the first to recast expert activation prediction as sequence modeling, capturing routing dependencies across the full layer stack and generation trajectory. We adopt Mamba2 [11], a lightweight and efficient sequence model, as the predictor 1 A recent concurrent work relying solely on LRU for expert hits.
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
backbone. During inference, the predictor continuously forecasts multi-step, multi-layer activations, providing long and reliable windows for downstream decisions. (ii) Joint prefetch scheduling. We formulate prefetch scheduling as a matroid optimization problem [13] to maximize expected expert hits under deadline constraints, thereby improving bandwidth efficiency. (iii) Forecast-driven caching. Leveraging the predicted sequence of future activations, we introduce probabilistic Belady eviction, which prioritizes experts by their likely near-term reuse, effectively retaining reusable experts. To eliminate execution bottleneck, we develop (iv) Graphcompatible offloading runtime. We derive general principles encompassing compute-transparent expert placement for fused MoE kernels and synchronization-free offloading orchestration disciplines. They coordinate predictive memory management with LLM inference for graph-based execution. We evaluate SeqMoE across diverse datasets on Qwen330B-A3B, Qwen3.6-35B-A3B, GPT-OSS-120B, and DeepSeekV4-Flash against llama.cpp [27], MoE-Infinity [38], KTransformers [7], and FreeToken [41]. At 25% and 45% expert residency, SeqMoE averages 91.72% and 96.97% hit rates, versus 74.24% and 88.50% for the strongest baseline. At 45% residency, it averages 80.22% of full-load performance, versus 51.50% for FreeToken and at most 28.03% for other baselines. • We are the first to formulate expert activation prediction as sequence modeling, enabling long overlap windows with high accuracy. • We propose predictive memory management for MoE offloading, jointly optimizing early prediction, prefetching, and cache residency to maximize expert hit rates. • We propose a graph-compatible runtime that enables graph-based execution of MoE offloading. • Results show that SeqMoE approaches full-load performance and advances the state of the art in MoE offloading inference.
2
Background
2.1
MoE’s Structural Advantages for Offloading
Mixture-of-Experts (MoE) has become a standard architecture for modern LLMs, employing a router to dynamically select active experts [22]. Consider an MoE model with 𝐿 layers, hidden dimension 𝑑, and 𝑒 experts per layer, of which 𝑘 experts are activated per token. A request comprises 𝐼 token steps spanning both prefill and decoding, with batch size 𝑏. At step 𝑖 ∈ [0, 𝐼 ) and layer 𝑙 ∈ [0, 𝐿), given the input X𝑖,𝑙 ∈ R𝑏 ×𝑠 ×𝑑 , where sequence length 𝑠 = 1, the router and MoE computation can be formulated as A𝑟𝑖,𝑙 = Linear(𝑋𝑖,𝑙 ), J𝑖,𝑙 , W𝑖,𝑙 = Φ(A𝑟𝑖,𝑙 ), Y𝑖,𝑙 = MoE(X𝑖,𝑙 , W𝑖,𝑙 , 𝐸𝑙 [Ji,l ]),
Table 1. Configurations of evaluated MoE models.
LLM Model
Parameters (MoE / Total)
Experts per Layer (Active / Total)
Qwen3-30B-A3B-FP8 Qwen3.6-35B-A3B-BF16 GPT-OSS-120B-MXFP4 DeepSeek-V4-Flash
29B / 31B 32B / 35B 115B / 117B 277B / 284B
8 / 128 8 / 256 4 / 128 6 / 256
−1 denotes the expert collection of layer where E𝑙 = {𝐸𝑙,𝑗 }𝑒𝑗=0 𝑙. A𝑟𝑖,𝑙 ∈ R𝑏 ×𝑠 ×𝑒 denotes the raw expert activation state, and Φ(·) typically consists of softmax followed by a Top-𝑘 operation. J𝑖,𝑙 ∈ [𝑒]𝑏 ×𝑠 ×𝑘 , W𝑖,𝑙 ∈ R𝑏 ×𝑠 ×𝑘 denote selected expert indices and their routing weights. The output Y𝑖,𝑙 ∈ R𝑏 ×𝑠 ×𝑑 is computed by dispatching each token to its selected experts and aggregating their weighted outputs. As shown in Table 1, sparse scaling makes MoE layers dominate model parameters while activating only a small fraction of experts per inference step. The dynamic sparsity creates a structural advantage for offloading: only the activated experts need to reside in device memory at each inference step. If they can be loaded in time for computation, offloading can in principle approach FullLoad performance with a much smaller memory footprint.
2.2
Expert Activation Prediction
Expert activation is late-bound, making early prediction essential for downstream scheduling. Existing approaches mainly fall into three categories. Similarity-based cross-layer prediction. ProMoE [32] predicts deeper-layer activations from shallow-layer hidden states with MLPs. Pre-gated [20] advances routing decisions through fine-tuning. MoE-APEX [34] directly feeds shallowlayer hidden states into future routers. These methods rely on residual-induced cross-layer similarity, causing accuracy to degrade rapidly with prediction distance and limiting lookahead to only one to three layers. For example, on Qwen3-30BFP8 with an RTX 4090 over PCIe 4.0, one layer’s computation overlaps the transfer of only 1.39 experts on average, leaving little room for prefetching. Trace-based cross-layer prediction. MoE-Infinity [38] and Taming-MoE [42] match shallow-layer activation states against request-level and iteration-level histories, respectively. As activation patterns evolve, matching must be updated layer by layer, again limiting lookahead to at most three layers. First-order cross-step prediction. STMoE [45] and Patterns-MoE [43] explore cross-step prediction based on per-layer conditional activation statistics between adjacent tokens. This memoryless, first-order Markov model is too coarse for precise scheduling: ST-MoE uses it only to assist cross-layer prediction, while Patterns-MoE confines it to lightweight cache guidance.
Zihan Wang, Yuqi Wang, Lei Gong, Cheng Tang, Wenqi Lou, Teng Wang, Chao Wang, and Xuehai Zhou
Loss (S)
Recall-Topk+3 (O)
5
6
0.9 0.8
4
0.4 2
0.3 0.2
1
0.8
0.1
0.7 0.6
4
Loss
0.5
Recall
3
0.9
5
0.7 0.6
Loss
Recall-Topk+3 (S)
GPT-OSS-120B-MXFP4
0.5 3
0.4 0.3
2
0.2 0.1
1
0.0 0
10
20
30
Epoch
Recall
Loss (O)
Qwen3-30B-A3B-FP8
0.0 0
10
20
30
Epoch
Figure 2. Sequence Dependence in Expert Activation. Overall, existing methods cannot simultaneously achieve high prediction accuracy and sufficient lookahead to effectively overlap computation with memory transfers, leaving downstream scheduling with a narrow and unreliable optimization window. 2.3
Sequence Modeling
Sequence modeling captures dependencies in ordered data to represent or predict states from historical context [33]. Given a sequence 𝑥 1:𝑇 = (𝑥 1, 𝑥 2, . . . , 𝑥𝑇 ), it learns a mapping 𝑔 : (𝑥 1, . . . , 𝑥𝑡 ) ↦→ 𝑦𝑡 . MoE expert activations naturally form sequences across generation steps and thus fit this framework. Representative models include Transformers [35], Recurrent Neural Networks (RNNs) [14], Long ShortTerm Memory (LSTMs) [18], and recent State Space Models such as Mamba [11]. Transformers offer strong long-range modeling but incur growing Key-Value (KV) cache memory overhead. We therefore use a Transformer only to verify sequence-level dependencies in expert activations and provide an accuracy upper bound. In contrast, RNNs, LSTMs, and Mamba compress history into a fixed-size state: 𝐻𝑡 = 𝑓 (𝐻𝑡 −1, 𝑥𝑡 ),
𝑦𝑡 = 𝑔(𝐻𝑡 , 𝑥𝑡 ),
where 𝑓 (·) and 𝑔(·) are the state-update and output functions. The fixed-size state 𝐻𝑡 enables lightweight prediction with low memory overhead. However, RNNs and LSTMs struggle with long-range dependencies and require sequential training. Mamba enhances long-range modeling through selective, content-aware state updates, while its state-space formulation supports efficient parallel training. Therefore, Mamba combines a fixed-size historical state, strong long-range modeling capability, and efficient parallel training. We adopt Mamba as the backbone of our sequenceto-sequence (seq2seq) expert activation predictor. 2.4
Graph-based Execution Mechanisms
Graph-based execution reduces host-side dispatch overhead by capturing device operations into a graph and replaying them with a single dispatch [15]. Mechanisms such as NVIDIA CUDA Graphs [29], AMD HIP Graphs [2], Ascend ACL Graphs [19], and Intel SYCL Graphs [21] are particularly
effective for small-batch LLM decoding, where short kernels make dispatch overhead significant. Edge inference typically operates at small batch sizes and thus benefits substantially. For MoE offloading, as on-demand loading stalls recede, reducing dispatch overhead becomes critical to approaching full-load performance. Efficient graph replay favors fixed execution structures and memory addresses while minimizing host intervention and synchronization. These requirements conflict with MoE offloading. First, dynamic expert placement hinders the use of fused MoE kernels, a key enabler of graph-based MoE execution. Although kernels such as Triton Fused MoE [36] and DeepGEMM [44] encapsulate dynamic activation within fixed execution structures, offloading dynamically maps experts to cache slots, making their addresses unstable. Second, prefetching and caching traditionally rely on CPU-GPU synchronization, which breaks graph execution and re-exposes dispatch overhead. FreeToken [41], a recent concurrent work, moves LRU cache management to the GPU to adapt offloading to CUDA Graph execution. However, it does not support prefetching, a basic offloading operation, nor does it derive general principles for graph-compatible offloading. The offloading runtime must therefore be redesigned to coordinate prediction, prefetching, and caching with LLM inference under graph execution constraints.
3
Motivation and Key Idea
3.1
Expert Activation Sequence Modeling
Expert activation exhibits strong sequence-level dependence that can be effectively exploited through sequence modeling. We encode expert activations across the full layer stack at each generation step into a vector and arrange these vectors in generation order to form an activation sequence. We train decoder-only Transformers for next-step prediction on the original (O) and shuffled (S) sequences, respectively. Fig. 2 reveals two observations. First, training fails to converge after shuffling, indicating that predictability stems from sequence-level dependence in expert activations rather than merely model capacity, demonstrating the sufficiency of sequence modeling. Second, top-11 recall on Qwen3-30B-A3B (𝑒 = 128, 𝑘 = 8) and top-7 recall
Table 2. Prefetch Scheduling Example Candidates Deadline Confidence Capacity Optimal MoE-Infinity ProMoE
A 1 0.55 1 A B A
B 2 0.90
C 2 0.40 3
B D B
D E C
D 3 0.75
E 3 0.70 4 E F D
F 3 0.30
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
Belady
Belady-8
Belady-F8
Qwen3-30B-A3B-FP8
LRU
Qwen3-30B-A3B-FP8
LFU
GPT-OSS-120B-MXFP4
200
175.7
GPT-OSS-120B-MXFP4
Decode (tok/s)
100
Hit Rate
90 80 70
150
122.7 91.7
100
74.9
68.6
Sync
Eager
52.1
50
60
0
50
Eager 15%
25%
35%
45%
15%
Loaded
25%
35%
CG
CG
Sync
45%
Loaded
Figure 4. Impact of End-to-End Graph Capture. Figure 3. Benefits of Future-Aware Eviction. on GPT-OSS-120B (𝑒 = 128, 𝑘 = 4) both exceed 90%, substantially outperforming existing cross-step prediction methods (Section 9.4), demonstrating the necessity of explicitly modeling such sequential dependence. 3.2
Joint Structure in Prefetch Scheduling
Bandwidth scarcity turns prefetch scheduling into a constrained task-selection problem. A slow interconnect can transfer only a subset of prefetch candidates, which differ along two dimensions: prediction confidence, the likelihood that an expert will be activated, and deadline, the time until its owning layer executes. Let ddl𝑡 , pos ∈ [0, 𝐼 × 𝐿) denote the deadline of task 𝑡 and the current position, respectively. conf𝑡 denotes its expected hit contribution, and 𝐶 (ddl) the transfer capacity available before ddl. Maximizing expert hits can then be formulated as: ∑︁ max conf𝑡 𝑠.𝑡 . |{𝑡 ∈ 𝑇 : ddl𝑡 ≤ ddl}| ≤ 𝐶 (ddl), ∀ddl. 𝑇
𝑡 ∈𝑇
MoE-Infinity prioritizes tasks by conf𝑡 (1− (ddl𝑡 −pos)/(𝐿 + 1)), with pos advancing dynamically, while ProMoE sorts tasks lexicographically by (−ddl𝑡 , conf𝑡 ). As Table 2 shows, neither yields an optimal selection. Both rank tasks independently, overlooking how their selections jointly consume transfer capacity under deadline constraints. The resulting bandwidth waste can grow as longer overlap windows expand the scheduling space. In contrast, the feasible task sets form a matroid, allowing weighted matroid greedy to optimally solve the formulated problem [13]. We use this formulation to guide online prefetch scheduling and improve bandwidth efficiency. 3.3
Future-Aware Cache Eviction
Future-aware eviction consistently outperforms historybased policies. Existing approaches typically adapt classical eviction policies such as LRU and LFU. Fig. 3 compares them with variants of the optimal Belady’s MIN [5], where the x-axis denotes the fraction of experts that can reside in the cache. Belady uses the complete future activation sequence and evicts the expert whose next activation is farthest in the future. Belady-8 limits this oracle knowledge to the next eight steps, while Belady-F8 replaces oracle activations with
our eight-step forecasts (Section 7.1). We make three observations. First, LRU and LFU perform similarly, suggesting that both capture only part of the expert reuse patterns. Second, future-aware policies consistently outperform history-based ones, highlighting the value of future activation information for eviction. Third, Belady-F8 closely approaches Belady-8, showing that predicted activations provide a reliable basis for cache eviction. 3.4
End-to-End Graph Execution
As on-demand loading stalls recede, maintaining low dispatch overhead becomes critical to approaching fullload performance. We place all model weights in device memory to simulate a 100% expert hit rate and isolate the performance impact of graph-based execution. Fig. 4 compares three execution modes. Eager represents existing runtimes that cannot accommodate dynamic expert placement in fused MoE kernels and instead dispatch individual GEMMs according to expert activation. CG combines fused MoE kernels with CUDA Graphs, representing ideal full-load execution. Sync breaks the graph between attention and MoE to reflect CPU–GPU synchronization introduced by prefetching and caching operations in existing runtimes. Both eager execution and synchronization substantially degrade overall inference performance. These results highlight the necessity of end-to-end graph capture for achieving high-performance MoE offloading inference.
4
Overview
Fig. 5 presents an overview of SeqMoE. During inference, each Transformer block enters the offloading pipeline after attention and routing. The system first lands previously issued prefetch tasks, which may target experts from any layer, and updates cache slot states (Section 7.2). It then loads any missing experts required by the current layer from host to device memory. At a predictor’s trigger layer 𝑙 ′ , the predictor forecasts expert activation probabilities over multiple steps for its assigned layers 𝑙 to 𝑙 ′ (Section 5). The first-step probabilities generate prefetch tasks (Section 6.1), which the prefetcher schedules as a job sequencing with deadlines problem using matroid greedy (Section 6.2). Subsequent-step
Predictive Memory Management for MoE Offloading Device
𝑨𝑨𝑖𝑖−1,𝑙𝑙′
Cache
𝑨𝑨′′ 𝑖𝑖+𝑐𝑐,𝑙𝑙:𝑙𝑙 ′
Slot State Transition
𝑨𝑨′′ 𝑖𝑖+1,𝑙𝑙:𝑙𝑙 ′
𝑨𝑨′𝑖𝑖,𝑙𝑙:𝑙𝑙′
′ 𝑨𝑨1,𝑙𝑙:𝑙𝑙 ′
...
Prefetch
𝑨𝑨′𝑖𝑖−1,𝑙𝑙:𝑙𝑙′
History
Offloading
Forecast-Driven Cache (Future-Aware Eviction)
...
Fetch
𝑨𝑨′′ 𝑖𝑖+𝑐𝑐−1,𝑙𝑙:𝑙𝑙 ′
D1: Graph-Capturable Prediction
Attention
1
Prefetch Landing
2
Expert Miss
Empty Comp
Probabilistic Belady Policy
𝑨𝑨′𝑖𝑖,𝑙𝑙:𝑙𝑙′
...
5
3
step 𝑖𝑖 − 1 layer 𝑙𝑙𝑙
𝑨𝑨𝑖𝑖−1,𝑙𝑙:𝑙𝑙′
...
Router
𝑨𝑨𝑖𝑖−2,𝑙𝑙:𝑙𝑙′
Mamba2-Based Model
Used
D2: Device-Resident Cache 2
D4: Device-Initiated Miss Load
3
MoE Offloading Pipeline Prediction
4
Gen Cache
Host
Joint Scheduling Prefetcher (Job Sequences with Deadline) 4
Matroid Greedy Optimization Confidence
Activation Predictor (Sequence Modeling)
D3: Async Host-Device Communication
Fused MoE
𝑨𝑨0,𝑙𝑙:𝑙𝑙′
Graph-Compatible Offloading Runtime
Zihan Wang, Yuqi Wang, Lei Gong, Cheng Tang, Wenqi Lou, Teng Wang, Chao Wang, and Xuehai Zhou
T8
T3 T1
T4 T2
T5
T7 T6
Task Deadline 1
5
D3: Host-Initiated DMA Prefetch
Update Prefetch
6
Transparent
Figure 5. SeqMoE System Overview. probabilities update the probabilistic Belady policy for futureaware eviction (Section 7.1). Finally, the runtime’s computetransparent expert placement allows execution to leave the offloading pipeline and resume fused MoE computation (Section 8.1). Beyond this pipeline, we establish four runtime orchestration disciplines that govern how these memorymanagement components coordinate to enable graph-based execution (Section 8.2).
5
Seq2Seq Expert Activation Prediction
5.1
Expert Activation Collection
We collect expert activation traces across diverse workloads, including mathematics (GSM8K [8], MATH [16]), code (CodeForces [9]), text comprehension (OpenOrca [25]), and multiturn conversations (ShareGPT [3]). To improve collection efficiency, we adopt a two-pass process. The first pass collects token ID traces through continuous batching. The second packs multiple token ID traces into a single prefill pass to dump expert activations, fully exploiting the inference engine’s parallelism. We denote the expert activations spanning multiple steps and layers as ′
′
A𝑟𝑖:𝑖 ′ ,𝑙:𝑙 ′ = Stack𝑙𝑘=𝑙−1 (Concat𝑖𝑗=𝑖−1 A𝑟𝑗,𝑘 ) ∈ R𝑏 ×𝑠 ×𝑛×𝑒 , where 𝑠 = 𝑖 ′ − 𝑖 is the sequence length and 𝑛 = 𝑙 ′ − 𝑙 is the number of layers. Each collected expert activation trace is represented as A𝑟0:𝐼,0:𝐿 ∈ R𝑏 ×𝐼 ×𝐿×𝑒 . 5.2
Model Architecture
We next introduce the preprocessing and model architecture of the predictor. The preprocessing is defined as A𝑖:𝑖 ′ ,𝑙:𝑙 ′ = Ψ(A𝑟𝑖:𝑖 ′ ,𝑙:𝑙 ′ ) ∈ R𝑏 ×𝑠 × (𝑛×𝑒 ) ,
where Ψ(·) consists of two steps. First, activation-masked normalization sets inactive experts to -inf and applies softmax along the expert dimension. This preserves the relative preference among activated experts as prediction confidence for prefetch scheduling, while suppressing noise from irrelevant experts to improve recursive prediction and provide more reliable signals for cache eviction. Second, reshaping merges the layer and expert dimensions into a single feature dimension, treating activations across multiple layers as a unified representation at each generation step. The number of included layers introduces a trade-off between prediction accuracy and prefetch overlap: including more layers improves accuracy but shortens the overlap window. Based on the preprocessed input, the predictor can be formulated as Z1 = Linear(A𝑖:𝑖 ′ ,𝑙:𝑙 ′ ) ∈ R𝑏 ×𝑠 ×𝑚 , Z2, H𝑖 ′ = Mamba2(Z1, H𝑖 ), ′ 𝑏 ×𝑠 × (𝑛×𝑒 ) A𝑖+1:𝑖 , ′ +1,𝑙:𝑙 ′ = Linear(Z2 ) ∈ R
where 𝑚 denotes hidden dimension of Mamba2. Its learned recurrent state accumulates information from the 𝑖 ′ − 𝑖 input activation states, updating from H𝑖 to H𝑖 ′ . Each output in ′ A𝑖+1:𝑖 ′ +1,𝑙:𝑙 ′ predicts the activation probabilities for the step immediately following its corresponding input. 5.3
Prediction Execution
The predictor supports four execution modes. Consider a predictor responsible for layers [𝑙, 𝑙 ′ ): Training. The predictor takes A0:𝐼 −1,𝑙:𝑙 ′ ∈ R𝑏 × (𝐼 −1) × (𝑛×𝑒 ) ′ as input and produces A1:𝐼,𝑙:𝑙 ′ , supervised by the corresponding ground truth A1:𝐼,𝑙:𝑙 ′ . No recurrent state needs to be retained across training sequences.
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
Prefill. For a prompt length 𝑝, the predictor consumes ′ A0:𝑝,𝑙:𝑙 ′ ∈ R𝑏 ×𝑝 × (𝑛×𝑒 ) and outputs A1:𝑝+1,𝑙:𝑙 ′ , while retaining the recurrent state H𝑝 . Predictor prefill proceeds along with ′ LLM prefill, and only the final prediction A𝑝,𝑙:𝑙 ′ is used for downstream operations. One-step prediction. Suppose LLM inference reaches step 𝑖 − 1 at layer 𝑙 ′ . Given H𝑖 −1 , the predictor takes A𝑖 −1,𝑙:𝑙 ′ ′ ∈ R𝑏 ×1× (𝑛×𝑒 ) and produces A𝑖,𝑙:𝑙 ′ , updating the state to H𝑖 . This mode runs at every generation step to issue prefetch tasks for the next step. Recursive prediction. Starting from the one-step prediction above, the predictor repeatedly feeds its latest output ′ back as input. The first recursive input is Ψ(A𝑖,𝑙:𝑙 ′ ). After 𝑐 ′′ 𝑏 ×𝑐 × (𝑛×𝑒 ) , which foreiterations, it produces A𝑖+1:𝑖+𝑐+1,𝑙:𝑙 ∈ R ′ casts expert activations over the next 𝑐 steps. The recurrent state is then restored to H𝑖 so that recursive rollout does not affect the next one-step prediction. Recursive prediction also runs at every generation step to update the cache policy. 5.4
Algorithm 1 Joint Prefetch Scheduling Inputs: 𝑄 task : current prefetch task queue, pos: current inference position, 𝐵 avg : average transfer budget per layer, 𝐵 cur : remaining transfer budget of current layer. Output: selected: selected task for prefetching. 1: if pos is updated then 2: 𝐵 avg ← 𝐵 avg + 𝛼 × 𝐵 cur 3: 𝐵 cur ← 𝐵 avg 4: 𝑄 task ← {𝑡 ∈ 𝑄 task | ddl𝑡 > pos + 1} 5: Sort 𝑄 task by ddl𝑡 in ascending order 6: Initialize an empty min-heap Heap 7: for 𝑡 in 𝑄 task do 8: Heap.Push(𝑡; conf𝑡 , −ddl𝑡 ) 9: if |Heap| > 𝐶 (ddl𝑡 ) then 10: Heap.Pop() 11: selected ← arg min𝑡 ∈Heap ddl𝑡 12: B𝑐𝑢𝑟 ← B𝑐𝑢𝑟 − 1 13: return selected
Training when the expert is needed for computation, are defined as
We combine a ranking loss [17] with KL divergence loss [23]. Since inference selects predicted experts by Top-𝑘, the ranking loss directly optimizes this decision: it enforces groundtruth Top-𝑘 experts to score higher than hard negatives by a margin, improving the ordering around the Top-𝑘 decision boundary. We further apply KL divergence between the ′ prediction A1:𝐼,𝑙:𝑙 ′ and ground truth A1:𝐼,𝑙:𝑙 ′ . Since the groundtruth distribution follows the same space as the predictor input A0:𝐼 −1,𝑙:𝑙 ′ , this objective aligns the input and output distributions, facilitating recursive prediction. To further improve recursive prediction, we first train the predictor with teacher forcing, where ground-truth activations are always used as inputs for subsequent predictions. After convergence, we adopt Scheduled Sampling [6] by probabilistically replacing a subset of A0:𝐼 −1,𝑙:𝑙 ′ with the cor′ responding predictions from A1:𝐼,𝑙:𝑙 ′ and continue training. This gradually exposes the predictor to its own outputs during training, improving its robustness to error accumulation in recursive inference.
6
Joint Prefetch Scheduling
6.1
Prefetch Task Generation
Suppose inference proceeds to step 𝑖 − 1, layer 𝑙 ′ , where the predictor performs one-step prediction and outputs the ′ 𝑏 ×1×𝑛×𝑒 . Based on expert activation probabilities A𝑖,𝑙:𝑙 ′ ∈ R this prediction, the runtime generates 𝑘 ≤ 𝑘 ′ < 𝑒 prefetch tasks for experts with high activation probabilities. Consider a generated task 𝑡 corresponding to expert 𝐸𝑝,𝑞 , where 𝑙 ≤ 𝑝 < 𝑙 ′ denotes its layer and 0 ≤ 𝑞 < 𝑒 denotes its expert index. Its prediction confidence, which quantifies its expected contribution to expert hits, and deadline, which indicates
′ conf𝑡 = Mean(A𝑖,𝑙:𝑙 ′ [:, 0, 𝑝, 𝑞]),
ddl𝑡 = (𝑖 − 1) · 𝐿 + 𝑝 Accordingly, task 𝑡 has an overlap window of 𝐿 − (𝑙 ′ − 1 − 𝑝), during which the expert transfer can overlap with ongoing inference. Within this window, the task remains a candidate for scheduling and, once selected, performs the actual expert transfer from the next-level memory to device memory. 6.2
Prefetch Task Selection
Algorithm 1 shows how the current prefetch task is selected. The average transfer budget 𝐵 avg is profiled during warmup. When inference advances to a new layer (Line 1), we update 𝐵 avg with an Exponential Moving Average (EMA) to adapt to runtime bandwidth variation and reset the current-layer budget 𝐵 cur accordingly (Lines 2–3). We then remove tasks whose deadlines are no later than the next layer, as they are either already expired or may not complete in time (Line 4). The remaining tasks are processed in ascending deadline order to construct the maximum-confidence feasible set under the deadline constraints (Lines 5–10). Specifically, for a task with deadline ddl, the available transfer capacity before its deadline is 𝐶 (ddl) = max(1, 𝐵 cur ) + 𝐵 avg × (ddl − pos − 1), which accounts for the remaining budget of the current layer and the transfer capacity of subsequent layers. Following matroid greedy, tasks are inserted into a min-heap keyed by confidence, and the lowest-confidence task is removed whenever the retained set exceeds 𝐶 (ddl), ensuring feasibility while prioritizing tasks with higher expected contributions to expert hits. This preserves the highest-confidence feasible
Zihan Wang, Yuqi Wang, Lei Gong, Cheng Tang, Wenqi Lou, Teng Wang, Chao Wang, and Xuehai Zhou
task set under all deadline constraints. Finally, Earliest Deadline First (EDF) selects the task with the earliest deadline for transfer (Line 11). The prefetcher waits for the transfer to complete and then reruns the algorithm to select the next prefetch task.
7
Forecast-driven Caching
7.1
Probabilistic Belady Policy
Suppose inference reaches step 𝑖 − 1 at layer 𝑙 ′ . After completing the single-step prediction, the predictor recursively ′′ generates A𝑖+1:𝑖+𝑐+1,𝑙:𝑙 ′ , which estimates expert activation probabilities over the next 𝑐 steps. The optimal Belady’s MIN retains objects that will be reused sooner and evicts those whose next reuse lies farthest in the future. Inspired by this principle, we propose a probabilistic Belady policy that extends future reuse to predicted activation probabilities. For expert 𝐸𝑝,𝑞 , where 𝑙 ≤ 𝑝 < 𝑙 ′ denotes its layer and 0 ≤ 𝑞 < 𝑒 its expert index, we define its retention priority as Í𝑐 −1 𝑗 ′′ 𝑗=0 (𝜆 · Mean(A [:, 𝑖, 𝑝, 𝑞])) , CacheScore𝑝,𝑞 = Í𝑐 −1 𝑗 𝑗=0 𝜆 where 0 < 𝜆 < 1. This score prioritizes experts with higher activation probabilities in the nearer future, approximating Belady’s preference for earlier reuse. 7.2
each layer’s expert weights to follow a fixed layout: [num_experts, dim0, dim1 ], where experts are stored contiguously at a fixed base address and selectively computed according to expert_id. Offloading breaks this structure by dynamically placing experts at arbitrary locations. Our key idea is to share cache slots across all MoE layers, whose expert-weight pointers reference the same fixed base address with the layout: [num_slots, dim0, dim1 ]. This design offers four benefits. First, it restores the fixed base address required by fused MoE kernels. Once all activated experts are cached, a lookup table maps expert_id to physical slot_id, preserving computation semantics. Second, although num_slots can far exceed num_experts, sparse activation keeps kernel latency unchanged in our measurements, preserving computation performance. Third, global sharing improves the flexibility of caching and prefetch scheduling. Finally, it accommodates the many experts activated during prefill without the per-layer capacity limits of statically partitioned slots. This design fully decouples the offloading pipeline from model computation, restoring fused-kernel efficiency and a fixed execution structure amenable to graph capture.
Slot State Transition
Each cache slot has one of four states. Empty denotes a free slot that can be directly allocated without eviction. Used holds a resident expert that is eligible for eviction. Fetch denotes a slot reserved for an ongoing prefetch and is protected from eviction. Compute holds an expert awaiting computation and is also protected from eviction. Four events in the offloading pipeline trigger slot-state transitions. First, when creating a prefetch task, an Empty or Used slot is reserved and transitioned to Fetch. Second, when the prefetch lands, the slot transitions from Fetch to Compute if the transfer completes successfully. Otherwise, it falls back to Empty. Third, an on-demand load directly transitions an allocated Empty or Used slot to Compute. Finally, immediately before executing each MoE layer, all Compute slots belonging to that layer are transitioned to Used, including both activated experts and prefetched-but-unactivated experts, making them eligible for subsequent eviction. Following these transitions, slot allocation always prioritizes Empty slots. If none is available, we evict the expert with the lowest CacheScore among Used slots.
8
Graph-Compatible Offloading Runtime
8.1
Compute-Transparent Expert Placement
Fused MoE kernels are a key enabler of graph-based MoE execution, as they encapsulate dynamic expert activation within a fixed execution structure. They typically require
8.2
Offloading Orchestration Discipline
We further eliminate host-device synchronization in the offloading pipeline to incorporate the pipeline itself into graphbased execution. As shown in Fig. 5, we introduce four orchestration disciplines for its core components: the predictor, cache, prefetcher, and on-demand loading. • Graph-capturable prediction. The predictor must run entirely on the device, and its prediction process can be directly captured into the graph. • Device-resident cache. Cache management must run entirely on the device, avoiding frequent hostdevice synchronization for expert access information and cache decisions. • Device-initiated miss loading. On a cache miss, the device must initiate the required data transfer and track its progress, avoiding host-device synchronization to determine transfer completion. • Asynchronous host-device communication and host-initiated DMA. The prefetcher runs on the CPU for scheduling flexibility, while both control communication and host-initiated data transfers must remain fully asynchronous. Together, these disciplines coordinate prediction, cache management, prefetch scheduling, and on-demand loading without introducing host-device synchronization into the execution path. This coordination enables end-to-end graph
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
capture of the offloading pipeline while preserving flexible host-side scheduling and asynchronous data movement. 8.3
Implementation
Following these disciplines, our implementation uses two threads. The main thread issues only CUDA operations for model inference, prediction, cache management, and ondemand loading, while a CPU worker thread runs prefetcher. CUDA Graph captures the main thread’s CUDA operations, with prefetcher activity fully overlapped. For prediction, our seq2seq predictor has no dynamic control flow, allowing direct capture of both one-step and recursive prediction. For cache management, all metadata, including lookup and state tables, reside on the GPU, with operations such as lookup and eviction implemented as CUDA kernels. For on-demand loading, we allocate the host expert pool with cudaHostAllocMapped. A custom kernel on the main thread’s CUDA stream copies missing experts from host to GPU memory, avoiding host synchronization while remaining graph-capturable. For prefetching, we similarly use cudaHostAllocMapped to allocate pinned, mapped host memory as a shared control region between CPU and GPU, supporting three forms of asynchronous communication. First, the GPU updates the inference position pos at each layer, which the CPU prefetcher reads after completing each prefetch task. Second, when a cache miss triggers on-demand loading, the GPU sets a shared flag to pause new prefetch submissions, reducing bandwidth contention. Third, monotonically increasing tags coordinate task dispatch and prefetch landing. The GPU assigns each new task a slot and binds its tag to that slot, and the CPU processes only unseen tags. The GPU updates the cache state only when the returned tag matches the slot’s expected tag. Prefetched weights bypass the control region: the CPU prefetcher initiates host-todevice DMA using cudaMemcpyAsync on a dedicated CUDA copy stream. Consequently, prefetch configuration, task generation, data transfer, and completion notification proceed asynchronously with the main computation stream, and GPU execution never waits for the CPU prefetcher. We implement SeqMoE in Hugging Face Transformers [37] using hooks. A single offloading function, inserted after routing and before MoE computation, incorporates these operations without changing external interfaces. 8.4
Prefill Offloading
Prefill rarely achieves full-load performance under offloading because many tokens activate a broad set of experts, substantially weakening MoE’s structural advantage from sparsity. Chunked prefill, a common optimization in cloud serving [1], is poorly suited to offloading because it incurs repeated expert transfers across chunks. With many tokens routed simultaneously, expert activation increasingly reflects statistical popularity. We therefore
profile expert activation frequencies offline over the dataset. Before prefill and after each layer, we prioritize prefetch tasks for high-frequency activated experts in subsequent layers. We keep num_slots − 𝑒 slots in the Fetch state, reserving 𝑒 slots for evicting and loading missed experts. We omit graph capture during prefill because of its variable sequence lengths and substantial computation, consistent with common inference practice [24].
9
Experimental Evaluation
9.1
Experimental Methodology
Hardware. We evaluate on three GPU-based platforms. The NVIDIA RTX 4090 platform has 24 GB GPU memory, an Intel Xeon Gold 6430 CPU, and 120 GB host memory, connected via PCIe 4.0. The NVIDIA RTX 5090 platform has 32 GB GPU memory, an Intel Xeon Platinum 8470Q CPU, and 120 GB host memory, connected via PCIe 5.0. The NVIDIA RTX PRO 6000 Blackwell platform has 96 GB GPU memory, an Intel Xeon Platinum 8470Q CPU, and 256 GB host memory, connected via PCIe 5.0. Models. We evaluate four advanced, widely used MoE-based LLM models with diverse architectures and precisions: Qwen330B-A3B-FP8 (QW3) [40], GPT-OSS-120B-MXFP4 (GPT) [30], Qwen3.6-35B-A3B-BF16 (QW36) [31], and DeepSeek-V4-Flash (DSV4) [12]. Table 1 summarizes their model weight configurations and expert activation characteristics. Datasets. For each model, we sample 5K, 5K, 5K, 30K, and 5K examples from MATH [16], GSM8K [8], CodeForces [9], OpenOrca [25], and ShareGPT [3], respectively, to construct a pool of 50K traces covering mathematics, coding, general text comprehension, and multi-turn dialogue, with sequence lengths of 0.1K–10K tokens. Configuration. We use 90% of each subset, totaling 45K traces, for predictor training and the remaining 5K for predictor testing. We further select 600 test traces for inference evaluation. We evaluate QW3 on the NVIDIA RTX 4090, QW36 on the NVIDIA RTX 5090, and both GPT and DSV4 on the NVIDIA RTX PRO 6000 Blackwell. During inference, we generate 𝑘 ′ = 𝑘 + 3 prefetch tasks per prediction and use 𝑐 = 8 recursive prediction steps with 𝜆 = 0.5 for the cache policy. We deploy two predictors, triggered at 𝑙 ′ = 𝐿/2 and 𝑙 ′ = 𝐿, respectively. Baselines. We compare SeqMoE’s inference performance against four frameworks. Llama.cpp [27] supports concurrent CPU–GPU computation in C/C++. MoE-Infinity [38] uses request-level expert activations for GPU prefetching and caching. KTransformers combines CPU–GPU hybrid inference with optimized CPU expert kernels. FreeToken [41], a concurrent work, supports graph-based offloading without prefetching and uses LRU without explicit expert-hit optimization. FullLoad keeps all weights on GPUs, using pipeline parallelism (PP) when multiple GPUs are needed for fairness with offloading. It runs on vLLM [24] 0.28.0 for
Zihan Wang, Yuqi Wang, Lei Gong, Cheng Tang, Wenqi Lou, Teng Wang, Chao Wang, and Xuehai Zhou
34.0 57.4
29.7 51.7
×
×
×
25%
35%
45%
21.8 15.7 33.0
21.0 14.8
26.1 44.3
18.5 13.9
16.2 12.8
× 15%
19.5 13.9 29.7 44.1
0
45%
21.2 31.7
30
16.7 12.8 26.1 39.9
35%
Loaded
Full-load
60 15.0 11.7
101.2 139.2 33.1
×
16.9
85.7 123.4 29.1
14.6
25.5
21.1
12.9 61.9 99.9
25%
15%
25%
35%
45%
Loaded
45%
4
× 1
2
60 30
×
Batch Size
56.9 82.4
64.9
90
Loaded
36.8 20.6 40.0
×
Full-load
0
32.9
96.2
85.7 123.4
31.7
80
98.2 135.6
35%
240 160
32.1
83.2 120.5
×
Loaded
60
18.5 13.9 29.7 51.7
25%
90 74.4
30
×
106.0 173.5
15%
28.2
63.5 97.2
×
29.1
225.8
2
Batch Size
24.7
×
163.2
40%
41.9 80.2
20.4
90.3 116.6 38.5 32.7
Full-load
0 1
2
4
0 4
1
Batch Size
× 45%
Loaded
Loaded
×
×
×
25%
35%
45%
Loaded
15%
35%
Loaded
2.8 2.0
2.8 2.2
2.9 2.4
25%
8.6 15.8
10.5 16.3
12.4 18.5
14.1 21.5
5.0
5.6
6.5
× 15%
10 2.8 2.7
40%
1.1 0.7
35%
1.2 0.8
25%
1
1.1 0.9
15%
1
1.1 1.0
4.1
4.3
4.9
6.8
1.8 1.2 0.8
× 35%
1.8 1.2 0.8
× 25%
2.0 1.2 1.0
0.6
0.7
× 15%
10
2.2 1.2 1.1
10
7.5
100 1.6 2.1
0.9
×
74.8
15.3 11.6 21.2 31.3
35.9 30.5 80.6 111.2
35%
Loaded
0
4
1.9 2.2
2.1 2.2
2.4 2.3
11.1 43.6 83.3
37.9 35.0 12.6 86.2 118.3
35.4 31.7 9.2 77.4 110.7
31.9 29.8 8.5 60.2 92.1 31.9 32.2 62.4 91.1
29.0 26.5 45.3 71.3
171.1
Lower is Better (↓)
1
×
Deepseek-V4-Flash 90
32.7 17.1 38.2
2
×
15%
80
46.2 49.3 103.5
34.7 49.0
×
Batch Size
25%
300
100
×
Full-load
0 15%
200
31.0 41.7
×
80
40%
0
154.7
105.8
Higher is Better (↑)
Full-load
40.8 40.6 91.5 154.0
45%
35%
Loaded
35.4 31.7 77.4 110.7
35%
Loaded
28.2 26.9 7.3 46.3 70.2
104.1 102.3
79.3 30.8 36.2
66.7
25%
1
TTFT (s)
27.2 30.7
48.4
15%
0
10
24.5 26.4
21.5 23.9
×
25%
SeqMoE
240 180
156.2
60
×
175.7
0 15%
120
×
Full-load
0
45%
×
120 60
35%
Loaded
81.4
180
60
×
Higher is Better (↑)
0
180
156.3
120
Full-load
100 50
×
25%
27.0 32.2
Decode (tok/s)
Decode length: 4096–8192 Loaded: 35%
Decode (tok/s)
150 119.5
81.4
67.4
×
FreeToken
GPT-OSS-120B-MXFP4 240
30.9 34.2 11.9
×
0
27.0 32.2 9.7
47.9
50
MoE-Infinity
Qwen3.6-35B-A3B-BF16
Full-load
100
0.9
122.7
15%
Prefill length: 1024
KTransformers
Higher is Better (↑)
24.1 27.6 7.4
150
21.1 25.9 6.8
Decode (tok/s)
Decode length: 128–1024
llama.cpp Qwen3-30B-A3B-FP8
45%
Figure 6. End-to-End Inference Performance Comparison. DSV4 and 0.26.0 for other models. For expert hit rates, we compare the caching policies of these frameworks and ablate our prefetch scheduling. For prediction accuracy, we compare against Patterns-MoE [43], the first work to explore cross-step expert prediction using conditional probabilities. Metrics. Tokens per second (tokens/s) measures decoding throughput, while time to first token (TTFT, s) measures prefill latency. Expert hit rate (%) is the fraction of required experts already resident in device memory before on-demand loading for MoE computation, indicating memory management efficiency and the potential to approach FullLoad performance. Recall (%) measures prediction accuracy as the fraction of actually activated experts correctly predicted. 9.2
End-to-End Performance
Fig. 6 compares end-to-end performance. Crosses indicate unsupported configurations. We evaluate MoE-Infinity only in the first experiment group due to its low inference speed. The x-axis shows the number of available device cache slots as a fraction of all experts, ranging from 15% to 40% for QW36 due to RTX 5090 memory limits, and from 15% to 45% for the other three models. Continuous serving at the edge often focuses on inference with a batch size of one [34, 39, 41]. The first row reports performance on shorter sequences (128–1,024 tokens). At 45% cache capacity (40% for QW36), SeqMoE achieves 84.82%, 75.69%, 79.20%, and 76.65% of FullLoad performance on QW3,
QW36, GPT, and DSV4, respectively. FreeToken reaches only 55.15%, 57.60%, and 45.45% on QW36, GPT, and DSV4, while the remaining baselines achieve at most 28.03%. Moreover, SeqMoE’s high expert hit rate enables it to match FreeToken at 45% or 40% capacity using only 25%. The second row reports results on longer sequences (4,096–8,192 tokens). Increased computation and hit-rate fluctuations cause minor performance changes, but the overall ranking persists as SeqMoE maintains prediction accuracy over long sequences. The third row evaluates small-batch inference with varying numbers of concurrent requests. SeqMoE achieves the largest performance gains as concurrency increases, since competition among sequences for limited cache capacity reduces expert reuse and makes timely prefetching increasingly important. The fourth row compares prefill latency across cache capacities. FreeToken uses ping-pong buffers that alternate between layers, resulting in largely stable latency as cache capacity increases. In contrast, SeqMoE achieves lower prefill latency through flexible prefetching and cache state transitions. 9.3
Expert Hit Rates
Fig. 7 compares expert hit rates, a key indicator of memorymanagement efficiency and the potential to approach FullLoad performance. LRU, LFU, and PrefillHot are adopted by FreeToken, MoE-APEX, and KTransformers. PrefillHot selects the most frequently activated experts during prefill and
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
SeqMoE
SeqMoE(Infinity)
Qwen3-30B-A3B-FP8
SeqMoE(CacheOnly)
Qwen3.6-35B-A3B-BF16
LRU
LFU
GPT-OSS-120B-MXFP4
PrefillHot Deepseek-V4-Flash
100 90
Hit Rate
80 70 60 50 40 30 15%
25%
35%
45%
15%
25%
Loaded
35%
40%
15%
Loaded
25%
35%
45%
Loaded
15%
25%
35%
45%
Loaded
Figure 7. Expert Hit Rate Comparison. keeps this selection fixed throughout decoding. At cache capacities of 15%, 25%, 35%, 40%, and 45%, SeqMoE achieves average hit rates of 84.56%, 91.72%, 95.09%, 96.11%, and 96.97%, respectively, compared with 60.02%, 74.24%, 84.49%, 86.37%, and 88.50% for LRU, the strongest baseline. We further evaluate two ablation variants: SeqMoE (Infinity), which uses score-based prefetch scheduling from MoE-Infinity, and SeqMoE (CacheOnly), which relies solely on forecast-driven caching. Future-aware eviction alone outperforms existing history-based policies, demonstrating the value of predicted activations in guiding cache residency. Prefetching further improves hit rates by loading experts before they are needed. However, independent score-based prioritization struggles to select the most beneficial combination of prefetch tasks under bandwidth and deadline constraints. This limitation is especially pronounced at smaller cache capacities, where higher miss rates generate more prefetch tasks and intensify bandwidth contention, highlighting the importance of joint deadline-aware scheduling.
Seq-1
Seq-8
Seq-L1
Qwen3-30B-A3B-FP8 90
90
85
Recall
Recall
85 80 75 50 45 40 8
9
10
11
80 75 50 45 40
12
8
10
11
Top-K
GPT-OSS-120B-MXFP4
Deepseek-V4-Flash
90
85
85
80
80 75 50 45 40 4
9
Top-K
Recall
Recall
Patterns-MoE
Qwen3.6-35B-A3B-BF16
5
6
Top-K
7
8
12
75 70 50 45 40 6
7
8
9
Top-K
Figure 8. Expert Prediction Recall Comparison.
10
9.4
Prediction Accuracy
Fig. 8 compares prediction recall. Patterns-MoE lacks sufficient accuracy for reliable prefetching, as fetching incorrect experts not only wastes transfer bandwidth but can also evict cached experts that would otherwise be reused. Seq-1 denotes one-step prediction over full sequences. Selecting the top 𝑘 + 3 experts yields recalls of 90.90%, 88.39%, 90.76%, and 85.81% on QW3, QW36, GPT, and DSV4, respectively. Seq-8 recursively predicts eight steps, with an average recall drop of only 3.58 percentage points relative to Seq-1. This limited degradation demonstrates robustness to recursive error accumulation and supports the effectiveness of activation-state preprocessing and scheduled sampling in maintaining accuracy over longer prediction horizons. Seq-L1 reports recall on sequences longer than 4,096 tokens and remains comparable to Seq-1, suggesting that prediction accuracy remains stable even for longer sequences within the evaluated range of 0.1K–10K tokens.
9.5
Offloading Overhead
Fig. 9 breaks down offloading inference time into four components at 45% cache capacity (40% for QW36). Offloading overhead comprises three components: on-demand stalls, predictor inference, and management operations. Despite an average miss rate of only 3.25%, on-demand loading remains the largest component, highlighting the substantial cost of data transfers on the critical path and the priority of eliminating expert cache misses. Predictor inference accounts for 3.46% of execution time on average. Increasing the number of predictors extends the overlap window but introduces additional prediction overhead and slightly reduces accuracy, creating a trade-off between lookahead, prediction cost, and reliability. The total predictor memory footprints are 120 MB, 228 MB, 112 MB, and 223 MB for QW3, QW36, GPT, and DSV4, respectively. Increasing the number of predictors does not increase their total memory footprint, since each predictor covers fewer layers and can therefore use a smaller hidden dimension. CUDA management kernels, including
Zihan Wang, Yuqi Wang, Lei Gong, Cheng Tang, Wenqi Lou, Teng Wang, Chao Wang, and Xuehai Zhou
Computation
Predictor
Management
On-demand stall
Qwen3-30B-A3B-FP8
83.50%
4.02% 2.10%
10.38%
Qwen3.6-35B-A3B-BF16
73.98%
3.91% 2.39%
19.71%
GPT-OSS-120B-MXFP4
80.56%
3.73% 1.97%
13.73%
Deepseek-V4-Flash
77.98%
2.16%1.73%
18.12%
Time Breakdown
Figure 9. Inference Time Breakdown. cache management, prefetch dispatch, and prefetch landing, account for 2.05% of execution time on average.
10
Conclusion
We presented SeqMoE, an MoE offloading system that approaches full-load performance through predictive memory management and a graph-compatible runtime. By formulating expert activation prediction as sequence modeling, SeqMoE provides accurate, long-horizon forecasts that jointly guide prefetch scheduling and cache eviction. This coordinated design improves expert hit rates across the memory hierarchy to reduce on-demand stalls. Moreover, computetransparent expert placement and synchronization-free orchestration disciplines reconcile dynamic offloading with graph-based execution mechanisms, enabling end-to-end graph capture and translating improved memory management into inference performance. Evaluations demonstrate that SeqMoE substantially outperforms existing offloading systems, highlighting the importance of jointly designing prediction, memory management, and execution to realize MoE’s structural advantages in high-performance, lowmemory inference. By combining timely expert availability with efficient execution, SeqMoE advances the state of the art in MoE offloading inference.
References [1] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, 117–134. https://www.usenix.org/conference/osdi24/presentation/agrawal [2] AMD. [n. d.]. HIP Graphs. HIP documentation. Accessed: 2026-0909. https://rocm.docs.amd.com/projects/HIP/en/latest/how-to/hip_ runtime_api/hipgraph.html [3] anon8231489123. 2023. ShareGPT_Vicuna_unfiltered. Hugging Face dataset. Accessed: 2026-09-09. https://huggingface.co/datasets/ anon8231489123/ShareGPT_Vicuna_unfiltered [4] Apple. 2024. Introducing Apple’s On-Device and Server Foundation Models. Apple Machine Learning Research. https://machinelearning. apple.com/research/introducing-apple-foundation-models [5] L. A. Bélády. 1966. A Study of Replacement Algorithms for a VirtualStorage Computer. IBM Systems Journal 5, 2 (1966), 78–101. doi:10. 1147/sj.52.0078 [6] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. In Advances in Neural Information Processing Systems, Vol. 28. https://arxiv.org/abs/1506.03099 [7] Hongtao Chen, Weiyu Xie, Boxin Zhang, Jingqi Tang, Jiahao Wang, Jianwei Dong, Shaoyuan Chen, Ziwei Yuan, Chen Lin, Chengyu Qiu, Yuening Zhu, Qingliang Ou, Jiaqi Liao, Xianglin Chen, Zhiyuan Ai, Yongwei Wu, and Mingxing Zhang. 2025. KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. doi:10.1145/3731569.3764843 [8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168 (2021). https://arxiv.org/abs/2110.14168 [9] Codeforces. [n. d.]. Codeforces Problemset. Online resource. Accessed: 2026-09-09. https://codeforces.com/problemset [10] Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2009. Introduction to Algorithms (3 ed.). MIT Press. [11] Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 10041–10071. https://proceedings.mlr.press/v235/dao24a.html [12] DeepSeek-AI, Anyi Xu, Bangcai Lin, et al. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348 (2026). https://arxiv.org/abs/2606.19348 [13] Jack Edmonds. 1971. Matroids and the Greedy Algorithm. Mathematical Programming 1 (1971), 127–136. doi:10.1007/BF01584082 [14] Jeffrey L. Elman. 1990. Finding Structure in Time. Cognitive Science 14, 2 (1990), 179–211. doi:10.1207/s15516709cog1402_1 [15] Abhishek Ghosh, Ajay Nayak, Ashish Panwar, and Arkaprava Basu. 2026. GraCE: Unlocking CUDA Graphs with Compiler Support for ML Workloads. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association, 1927–1947. https://www.usenix.org/conference/osdi26/presentation/ghosh [16] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Vol. 1. https://arxiv.org/abs/2103.03874 [17] Ralf Herbrich, Thore Graepel, and Klaus Obermayer. 2000. Large Margin Rank Boundaries for Ordinal Regression. In Advances in Large-Margin Classifiers. MIT Press, 115–132.
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
[18] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735–1780. doi:10.1162/neco. 1997.9.8.1735 [19] Huawei. [n. d.]. ACL Graph: Cross-Stream Capture. CANN documentation. In Chinese. Accessed: 2026-09-09. https://www.hiascend.com/ document/detail/zh/CANNCommunityEdition/920beta2/others/ acldevg/runtime_doc_dev_0031.html [20] Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. 2024. Pre-Gated MoE: An AlgorithmSystem Co-Design for Fast and Scalable Mixture-of-Expert Inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). 1018–1031. doi:10.1109/ISCA59077.2024.00078 [21] Intel. [n. d.]. SYCL Graph Usage Guide and Examples. Intel LLVM documentation. Accessed: 2026-09-09. https://github.com/intel/llvm/ blob/sycl/sycl/doc/syclgraph/SYCLGraphUsageGuide.md [22] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of Experts. arXiv:2401.04088 [cs.LG] https://arxiv.org/abs/ 2401.04088 [23] Solomon Kullback and Richard A. Leibler. 1951. On Information and Sufficiency. The Annals of Mathematical Statistics 22, 1 (1951), 79–86. doi:10.1214/aoms/1177729694 [24] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. Association for Computing Machinery, 611–626. doi:10.1145/3600006.3613165 [25] Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and Teknium. 2023. OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces. Hugging Face dataset. https://huggingface.co/datasets/Open-Orca/OpenOrca [26] Jiacheng Liu, Peng Tang, Wenfeng Wang, Yuhang Ren, Xiaofeng Hou, Pheng-Ann Heng, Minyi Guo, and Chao Li. 2026. A Survey on Inference Optimization Techniques for Mixture of Experts Models. Comput. Surveys 58, 10, Article 247 (2026), 37 pages. doi:10.1145/3794845 [27] llama.cpp Contributors. [n. d.]. llama.cpp: LLM Inference in C/C++. GitHub repository. Accessed: 2026-09-09. https://github.com/ggmlorg/llama.cpp [28] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia. 2025. Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems. Comput. Surveys 58, 1 (2025). doi:10.1145/3754448 [29] NVIDIA. [n. d.]. CUDA Programming Guide: CUDA Graphs. Online documentation. Accessed: 2026-09-09. https://docs.nvidia.com/cuda/ cuda-programming-guide/04-special-topics/cuda-graphs.html [30] OpenAI. 2025. gpt-oss-120b & gpt-oss-20b Model Card. Technical Report. OpenAI. https://openai.com/index/gpt-oss-model-card/ [31] Qwen Team. 2026. Qwen3.6-35B-A3B. Hugging Face Model Card. Accessed: 2026-09-09. https://huggingface.co/Qwen/Qwen3.6-35BA3B [32] Xiaoniu Song, Zihang Zhong, Rong Chen, and Haibo Chen. 2025. ProMoE: Fast MoE-based LLM Serving using Proactive Caching. arXiv:2410.22134 [cs.DC] https://arxiv.org/abs/2410.22134 [33] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, Vol. 27. https://arxiv.org/abs/1409.3215 [34] Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, PhengAnn Heng, Chao Li, and Minyi Guo. 2026. MoE-APEX: An Efficient
MoE Inference System with Adaptive Precision Expert Offloading. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 1185–1200. doi:10.1145/3779212.3790187 [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems, Vol. 30. https://arxiv.org/abs/1706.03762 [36] vLLM Contributors. [n. d.]. vLLM: Fused MoE Triton Kernels. GitHub repository. Accessed: 2026-09-09. https://github.com/vllmproject/vllm/blob/main/vllm/model_executor/layers/fused_moe/ fused_moe.py [37] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, 38–45. doi:10.18653/v1/2020.emnlp-demos.6 [38] Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. 2025. MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache. arXiv:2401.14361 [cs.LG] https://arxiv. org/abs/2401.14361 [39] Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. 2024. PowerInfer-2: Fast Large Language Model Inference on a Smartphone. arXiv preprint arXiv:2406.06282 (2024). https: //arxiv.org/abs/2406.06282 [40] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 (2025). https://arxiv.org/abs/2505.09388 [41] Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, and Ion Stoica. 2026. FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution. arXiv preprint arXiv:2608.16157 (2026). https://arxiv.org/abs/2608.16157 [42] Hanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang, and Hao Wang. 2026. Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading. In Proceedings of the 2026 European Conference on Computer Systems. 176–191. doi:10.1145/3767295. 3769319 [43] Zhongkai Yu, Yue Guan, Zihao Yu, Chenyang Zhou, Zhengding Hu, Shuyi Pei, Yangwook Kang, Yufei Ding, and Po-An Tsai. 2026. Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference. In 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). 68–84. doi:10.1109/ISCA66397.2026.00021 [44] Chenggang Zhao, Zhean Xu, Liang Zhao, Jiashi Li, Chenhao Xu, Anyi Xu, Shengyu Liu, Kexing Zhou, and Kuai Yu. 2025. DeepGEMM: Clean and Efficient BLAS Kernel Library on GPU. GitHub repository. https://github.com/deepseek-ai/DeepGEMM [45] Yingnan Zhao, Razvan Bunescu, Ahmed Louri, Avinash Karanth, and Ke Wang. 2026. A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference. arXiv:2606.15453 [cs.AR] https://arxiv.org/abs/2606.15453