GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cloud distributedcomputing parallelcomputing

distributed computing, parallel computing, cloud

GEM: GPU-Variability-Aware Expert-to-GPU Mapping For Mixture-of-Experts Models Sourish Wawdhane

Avinash Kumar

Poulami Das

[email protected] The University of Texas at Austin

arXiv:2605.19945v1 [cs.DC] 19 May 2026

Abstract Mixture-of-Expert (MoE) models enable efficient inference by employing smaller experts and activating only a subset of them per token. MoE serving engines distribute experts across multiple GPUs and route tokens to appropriate GPUs at inference time based on experts activated. They process tokens in lock-step fashion, where tokens within a batch must finish processing before proceeding to the next layer. This synchronization barrier acts as a critical bottleneck because the performance of MoE models is limited by the straggler GPU that finishes last. Stragglers emerge when too many heavily used experts are placed on the same GPU or the slowest GPU. While prior works place experts that balance token loads across GPUs, they all overlook GPU variability and often place highly used experts on the slowest GPUs. We propose GEM, GPU-variability-aware Expert Mapping, a framework for GPU variability-aware expert to GPU mapping for MoE models. GEM exploits two insights. First, we must place experts such that each GPU receives non-uniform token loads based on their variability and they all finish processing a layer at about the same time. Our studies show that there are two types of experts: consistent that are used most of the time and temporal that are often used together for the remaining time. Our second insight is that we must place simultaneously used consistent and temporal experts on different GPUs and avoid placing them on slower GPUs to reduce slowdown. GEM gathers the variability profile of GPUs for each model and task and uses the token load distributions per task to map experts to GPUs. Our experiments show that GEM improves end-to-end latency by 7.9% on average and by up to 16.5% compared to the baseline.

Introduction

Feed-forward-networks (FFNs) account for up to two-thirds of per token compute costs in Large Language Models (LLMs) [1, 2]. Mixture of Experts (MoE) models are variants of LLMs that lower these costs by replacing each FFN with smaller sub-networks, called experts, and use only a subset of them to process each token. For example, Mixtral 8x7B [3] activates only 2 out of its 8 experts for each token and yet, achieves accuracy comparable to the dense Llama-2 70B model while using ∼5× fewer floating point operations per token [3]. This makes MoE models very attractive for high-throughput and low-latency inference, leading to their increased adoption in real-world serving environments in recent years [4–10]. 1

The performance of MoE models is severely limited by stragglers. Figure 1(a) shows how MoE models use expert parallelism, where the experts are distributed across multiple GPUs; and a router determines the experts to be activated for each token and sends the token to the GPUs hosting those experts. However, due to lockstep execution, a batch can proceed to the next layer only after the tokens corresponding to all its requests have been processed in the current layer. This synchronization barrier creates a major bottleneck because a batch must wait for the slowest GPU or the straggler to finish execution before proceeding. The problem worsens because experts are not used uniformly; some are used more often than others and the most frequently used experts vary across layers [3, 11]. For instance, our experiments using the Qwen3-235B model on the ShareGPT dataset show that the most frequently used expert is used 4.2× more often than the least used expert. Reducing the impact of straggler GPUs is critical to improve the performance of MoE models. A GPU becomes a straggler if it hosts too many highly utilized experts and/or if it is inherently slower than the others due to hardware variability. While prior works balance the mapping of highly used experts by introducing artificial uniformity in expert utilization which degrades accuracy [11, 12] or by dynamically re-distributing experts across GPUs incurring substantial memory overheads [4, 13], they all overlook the inherent variability in the GPU hardware. In reality, modern GPUs exhibit substantial performance variability due to process variation, environmental factors, and runtime power management [14, 15]. Our experiments across 128 NVIDIA L40 GPUs show that inference throughput differs by up to 27.7% between the fastest and the slowest GPUs and Figure 1(b) shows that even within a single 8-GPU node, the spread persists at 7.7% over a week. Even if we were to perfectly balance the load across the GPUs, the problem of stragglers would still persist due to fundamental limitations in the hardware. The severity of the problem increases in realworld deployment scenarios using dozens of GPU [4, 6, 7] because the likelihood of encountering non-uniformity in the hardware and the probability of using slower GPUs increase. Straggler GPUs become the steady-state bottleneck because even a single slow GPU stalls every layer of every forward pass throughout the deployment. Ideally, MoE model serving frameworks must map experts on GPUs such that the impact of stragglers stemming from both non-uniform expert utilization and GPU variability is minimized.

Sourish Wawdhane, Avinash Kumar, and Poulami Das

E2 E3

MoE-Router Tokens (a)

E0 E1 (b)

Proposed Slow GPU Fast GPU High Utilization

E0 E1

Prior Work Slow GPU Fast GPU Time

GPU-1 Time

GPU-0

E2 E3

E2 E1

E0 E3

Variability Aware

Low

(c)

Figure 1. (a) MoE models are deployed such that experts are distributed across multiple GPUs and a router selects expert(s) to process each token. (b) GPUs exhibit performance variability. (c) GEM places experts on GPUs such that a fast GPU processes proportionally more tokens in the same time as a slow GPU and avoids placing heavily used experts on slower GPUs. In this paper, we propose GEM, GPU-variability-aware Expert Mapping, a framework for high-throughput low-latency MoE model serving via GPU variability-aware expert mapping. A natural strategy to enable GPU-variability-aware expert mapping would be to simply place the most heavily used experts on the fastest GPUs. But this approach does not work for two reasons. First, the fastest GPU in a node is often only modestly faster than the slowest one. This limited gap constrains the extra token load that the fastest GPU can process. Naively placing too many frequently-used experts on the fastest GPU may exceed this limit, causing it to become a straggler itself. Second, the most heavily-used experts fall into two categories: consistent experts that are active most of the time, and temporal experts that are highly utilized in the remaining time (small fraction of the total time). Unfortunately, while metrics like average token load over a span of time, used in prior works [10, 11, 16], suffice for detecting consistent experts, they cannot detect temporal experts. GEM addresses these challenges by exploiting two key insights. First, rather than trying to equalize token loads across GPUs, GEM places experts such that GPUs receive non-uniform token loads based on their performance variability and the faster GPUs only process proportionally more tokens than the slower ones, allowing all GPUs to complete processing for each layer at about the same time. For example, on an NVIDIA 8xL40, the fastest GPU processes 14% more tokens than the slowest at the same latency. This enables GEM to transform hardware variability from a liability to a mapping constraint that the framework exploits to improve performance. Second, GEM optimizes for both consistent and temporal experts. By observing token loads at a much finer granularity rather than relying on average utilization, GEM detects both types of experts. To optimally place the temporal experts, GEM exploits our observation that many of these experts are often used together. In other words, their utilization is correlated. Figure 1(c) illustrates how GEM ensures simultaneously used consistent and temporal experts are placed on different GPUs and that they are not placed on slower GPUs (which is GPU-0 in the illustration).

GEM employs a four-step process. First, GEM captures the number of tokens that each expert receives at each step during online inference to balance the mapping of both temporal and consistent experts. Second, GEM profiles performance variability across GPUs by measuring per-GPU MoE-layer latency as a function of token load. However, building these profiles at a fine granularity is non-trivial because it can take hours during which the GPUs are unavailable for inference, whereas coarse sampling under-samples the staircase pattern of MoE latency, where latency jumps sharply at certain token counts. To address this, GEM samples token counts only at tile boundaries, reducing profile time from hours to minutes. Third, GEM must identify an expert-to-GPU mapping that minimizes straggler latencies caused by both expert imbalance and performance variability. However, the expert mapping space grows combinatorial in the number of experts and GPUs. So, GEM runs a heuristic-based iterative search to refine mappings. Finally, GEM loads each expert’s weights onto its assigned GPU at model load time and continues to use it throughout the deployment. Our evaluations across five MoE models and representative tasks show that GEM improves end-to-end latency by 7.9% on average and by up to 16.5% compared to the baseline. Overall, this paper makes the following contributions: 1. We observe that performance of MoE models is limited by stragglers resulting from too many heavily used experts placed on the same GPU and inherent hardware variability. 2. We propose GEM, GPU-variability-aware Expert Mapping, a framework for high-throughput low-latency MoE model serving via GPU-variability-aware expert to GPU mapping. 3. We place experts such that the faster GPUs only process proportionally more tokens than the slower ones, allowing all GPUs to complete processing at about the same time. 4. We identify two types of experts: consistent (used more often and easy to detect) and temporal (used infrequently and hard to detect). We design GEM such that both of them can be detected and the experts that are simultaneously used are placed on different GPUs and avoid the slower GPUs. 2

GEM : GPU-Variability-Aware Expert-to-GPU Mapping For Mixture-of-Experts Models

Background and Motivation

2.3

2.1

Mixture-of-Expert Models

Currently, there exists various prior works that address the straggler problem caused by expert imbalance. Training Strategies: These approaches use an auxiliary load balancing loss [11, 12] to penalize expert imbalance. However, this introduces a trade-off between model quality (primary training objective) and load balancing of experts. Setting the loss co-efficient too high forces artificial uniformity and degrades accuracy, while setting it too low causes the router to converge on a small subset of experts, starving the rest (called “routing-collapse") [18, 29]. DeepSeekV3 [4, 16] mitigates this by introducing a bias term that adjusts routing without affecting training gradients, but even when trained with such techniques, models exhibit significant load imbalance at inference time [28]. This is because token distributions during inference differ from that during training. This causes the router to produce skewed expert selections, despite how balanced the pre-training data was. Inference Strategies: Dynamic approaches, such as vLLM’s Expert Parallel Load Balancer (EPLB) [4, 20], periodically re-distribute experts across GPUs based on observed load. However, this requires copying expert weights across GPUs, which introduces memory overheads. On the other hand, static approaches like MOETuner [23] and ExFlow [30] compute expert mapping offline using profiled routing traces. We illustrate this by using Figure 3. Initially, assume experts 𝐸 0 , 𝐸 1 are placed on GPU-0 and 𝐸 2 , 𝐸 3 are on GPU-1, as shown in Figure 3(a). But it turns out that the most frequently used experts for Layer-0 are 𝐸 0 and 𝐸 1 , which causes GPU0 to process a significantly larger number of tokens while GPU-1 remains idle, eventually making GPU-0 a straggler. Similarly, experts 𝐸 2 and 𝐸 3 are used more often for Layer-1, making GPU-1 the straggler for this layer. Prior methods redistribute the experts, moving 𝐸 1 to GPU-1 and 𝐸 2 to GPU0 as shown in Figure 3(b), where the most frequently used experts are not co-located on the same GPU.

2.2

Straggler Problem With Expert Parallelism

Utilization (%)

The problem of stragglers worsens because MoE models suffer from expert imbalance, where a few experts receive a disproportionately large number of tokens as compared to the remaining experts. For example, for the Qwen3-235B model with 128 experts, if each expert was used uniformly, then the expert utilization rate would be 0.78%. However, as shown in Figure 2, the most frequently used expert is used 4.2× and 2.4× more often than the idealized uniform utilization rate. Moreover, the utilization of each expert depends on the specific layer of the model and the distribution of the expert utilization for each layer is different from another.

4.2x

2 1

Layer 92 Layer 0

2.4x

GPU 0

Uniform 32

64 96 Sorted Expert Rank

GPU 1 (a)

128

L0 L1

High

Latency

Mixture-of-Expert (MoE) models are an emerging class of LLMs that replace dense feed forward networks with multiple smaller networks, called “experts" [17, 18]. Each expert specializes in different linguistic representations [9, 10] that differ across layers. For example, one expert may specialize in coding-related tasks, while another specializes in punctuation. At inference time, a router selects a subset of experts for each token [11, 18]. For instance, the Qwen3-235B MoE model uses only 8 out of 128 experts available per token while attaining high accuracy across most tasks [5]. Thus, MoE models drastically reduce computational costs of token generation, improving throughput and latency. Consequently, they are becoming increasingly prevalent [3–5, 8, 11, 12, 19]. Modern inference engines, like vLLM [20], distribute experts across GPUs, a deployment strategy called expert parallelism. Each MoE layer’s router selects experts for each token and dispatches it to the GPU(s) hosting those experts. As tokens in a batch are processed in a lockstep manner, all experts must finish processing before the batch can proceed to the next layer. This creates a synchronization barrier at each layer, where the last GPU to finish computation, called the straggler, determines the effective latency. Minimizing stragglers is critical for performance [4, 13, 21–28].

Prior Works On Addressing Stragglers

GPU 0 (fast)

(b)

GPU 1 (slow)

Low

Figure 3. (a) Trivially allocating experts causes some GPUs to process significantly more tokens and become stragglers because some experts are used more often than others. (b) Prior works attempts to overcome this problem by redistributing experts to balance token loads across GPUs.

Figure 2. Expert utilization for Layers 0 and 92 of Qwen3235B on ShareGPT. Some experts are used more often than others and the most used experts differ across layers.

Sourish Wawdhane, Avinash Kumar, and Poulami Das

Limitations of Prior Works

Throughput

While more effective than trivial expert allocation methods, prior works still suffer from a critical drawback. They are agnostic of the variability in the underlying hardware. Our experiments show that even state-of-the-art Nvidia 8xL40 GPUs exhibit significant performance variability. Figure 4 shows the variation in throughput of the ShareGPT dataset for the DeepSeek-R1-Distill-Qwen-7B model served at batch size of 128 on a cluster with 8 GPUs over a week. We observe up to 7.7% difference in the throughput between the best and worst GPUs. This aligns with observations from other prior work that modern GPUs exhibit significant performance variability [14, 15]. Such heterogeneity directly impacts the performance of MoE models because even if we re-distributed the experts perfectly, some GPUs would continue to remain stragglers due to the inherent non-uniformity, such as GPU-1 in the illustration in Figure 3(b).

We propose GEM, a framework for high-throughput inference using MoE models. GEM enables GPU-variability-aware expert mapping to minimize the impact of stragglers. In this section, we discuss the challenges in enabling GEM before describing how our design addresses them. 3.1

Challenge-1: GPUs exhibit variability, but only limited While GPU performance variability indeed exists, the fastest GPU in a compute node is only modestly faster than the slowest one. This limited gap constrains the extra token load that the fastest GPU can process. Naively placing too many frequently used experts on the fastest GPU exceeds this limit, eventually causing this GPU to become a straggler itself.

7.7%

Challenge-2: Temporal nature of expert activations Most frequently used experts can often be grouped into two categories depending on their activations: (1) consistent experts that are used in almost every time step and (2) temporal experts that are often used together in a small fraction of time steps. In state-of-the-art LLM serving frameworks such as vLLM, a time step refers to the step where a token is generated for each request in a batch. If we were to simply monitor the average token load across 𝑁 steps to identify the most frequently used experts, we would only be able to detect the consistent experts while the temporal experts would go undetected because their average individual token loads across 𝑁 steps is typically low. For example, Figure 6 shows the activations for the 16 most frequently used experts in Layer 43 of the Llama-4 Scout model over many time steps.

90 1

3 4 5 GPU index

Figure 4. Throughput (tokens/s) for DeepSeek-R1-DistillQwen-7B on NVIDIA 8xL40s over a week shows that modern GPUs exhibit persistent performance variability.

Temporal

Time Steps

Mean

Fastest

12 Slowest

Frequency (%)

Figure 5 shows the distribution of the variation in throughput across 16 different Nvidia 8xL40 GPUs for a total of 128 GPUs. The best GPU offers 10.8% higher throughput compared to the average. In contrast, the worst GPU reduces the throughput by 13.23% compared to the average. We present more characterization results in Appendix A.

8 4 0

5 0 5 10 Throughput Deviation (%)

Challenges in Designing GEM

A natural strategy to enable GPU-variability-aware expert mapping would be to simply place the most heavily used experts on to the fastest GPUs. However, this approach is non-trivial due to two reasons.

100

Our Proposal: GEM

Figure 5. Distribution of throughput variation for DeepSeekR1-Distill-Qwen-7B on 128 NVIDIA L40s. The fastest GPU profiled offers 27.7% higher throughput than the slowest.

56 49 42 35 28 21 14

Consistent

20 15 10 5 0 1 2 3 4 5 6 7 8 9 101112131415 Expert ID

Utilization (%)

2.4

Figure 6. Per-step utilization for the 16 experts in Layer-43 of Llama-4 Scout. Consistent experts (2, 5, 15) are used in 85% of timesteps, whereas temporal experts (0, 3, 10) are used in only 17% of time steps but process 3x more tokens.

Our goal: GPU-variability-aware expert mapping for MoE models to minimize stragglers and improve throughput. 4

GEM : GPU-Variability-Aware Expert-to-GPU Mapping For Mixture-of-Experts Models

Experts 2, 5, and 15 are the consistent experts used across 85% of the time steps, whereas experts 0 and 3 are temporal experts that are heavily used together only in certain phases. As temporal experts are often used simultaneously, they cause slowdown when placed on the same GPU. For instance, if experts 0 and 3 were placed on the same GPU, it would receive more than 50% of the tokens in 10% of time steps which is 2× the load (25%) of a fully balanced 4-GPU system. 3.2

GPUs. In contrast, placing temporal experts is non-trivial because the time steps in which they are utilized are not known in advance. However, we observe that the utilization of temporal experts is often non random; a subset of these experts are frequently used together. We refer to them as correlated temporal experts. For instance, Figure 8 shows that Experts 0 and 3 in Layer 43 of Llama-4 Scout have a Pearson’s correlation coefficient of 0.88 across time steps. The Pearson’s correlation coefficient quantifies the similarity between two variables with values ranging between -1 to 1 and values closer to 1 denoting higher correlation [31]. By placing correlated temporal experts on different GPUs, their impact on overall slowdown can be minimized.

Key Insights

MoE Latency (ms)

3.50 3.25 3.00

Utilization (%)

Insight-1: Balance end latencies, not token loads Placing too many of the most frequently used experts on the fastest GPUs risks making those GPUs the new stragglers, whereas placing experts such that each GPU processes the same number of tokens does not mean they will all complete processing at the same time for each time-step. Our key insight is that rather than uniformly allocating token loads across GPUs (as done in prior works), we must place experts such that faster GPUs process only proportionally more tokens than the slower ones based on the performance gap. For instance, Figure 7 shows that GPU-1 (fastest) on an NVIDIA 8xL40 can process 14% more tokens in the same amount of time as GPU-0 (slowest). Thus, experts must be placed by accounting for this relationship such that GPU-1 only hosts a subset of the most utilized experts such that it processes 14% more tokens compared to GPU-0.

High Joint Utilization

Expert 0 Expert 3

40 20

Forward step Figure 8. Experts 0 and 3 in Layer 43 of Llama-4-Scout are strongly correlated across engine steps (𝑟 =0.88).

GPU 0 GPU 1

Insight-2: Place both consistent and correlated temporal experts on different GPUs to minimize slowdown.

Same Latency 3.3

2.75 2.50

GEM places experts by accounting for both expert utilization patterns of a task and the GPU performance variability. However, expert utilization depends on the task and is not known prior to inference. Moreover, the number of tokens processed by each GPU depends on the task and model. Thus, the impact of GPU variability on model performance for a given task is also unknown. To address this challenge, GEM employs four steps, outlined in Figure 9. 1 GEM assesses the expert utilization patterns of a task by counting the number of tokens routed to each expert during online inference. Initially, it places experts using vLLM’s [20] default linear mapping to capture this information. 2 Next, GEM profiles variability across GPUs offline by using a micro-benchmark to measure MoE-layer latency for different token loads. 3 Then, GEM uses the expert utilization patterns and variability information to identify a better expert to GPU mapping or an assignment of expert to GPUs to minimize stragglers. 4 Finally, GEM replaces the initial mapping (from Step-1) with this task-specific variability-aware mapping for every MoE layer during deployment. Such in-deployment expert swapping is similar to prior works like vLLM EPLB and has minimal overheads. Next, we discuss these steps in detail.

14% More Tokens

3.8

4.1

4.4

Tokens per GPU (×103 )

Design Overview and Implementation

4.7

Figure 7. Latencies for MoE computation for Llama-4 Scout on an NVIDIA 8xL40 shows that latencies are identical even as GPU 1 receives 14% more tokens than GPU 0. Insight-1: We must place experts so that each GPU receives non-uniform token loads based on their performance and all GPUs finish processing simultaneously per step. Insight-2: Optimize for both types of experts While most time steps are bottle-necked by consistent experts, a considerable fraction of them are impacted by the temporal ones. An ideal expert-to-GPU mapping policy must therefore reduce the slowdown caused by both types of experts. Placing consistent experts is relatively easier because the mapping policy can simply put these experts on separate 5

Sourish Wawdhane, Avinash Kumar, and Poulami Das

Variability Profiling Latency

High

GPUs

Low

Expert Utilization Trace Requests

High

Layer0

…

Layer1

…

Iterative Expert Mapping

Final Mapping

Next Mapping

Current Mapping

Deploy

E0 E3?

Swaps

M i+1

E1 E2?

Experts Low

Converge?

GEM Scorer

Figure 9. Overview of GEM. First variability is profiled across GPUs. Next, expert utilization patterns are captured to form a trace. Then, an iterative search refines expert mapping. Finally, the expert mapping is deployed on the system.

Latency Reduction (%)

3.3.1 Step-1: Expert Utilization Trace. To optimally place experts on the GPUs, GEM must know how experts are typically used over time steps. To assess this, GEM counts the number of tokens each expert receives during online inference. Accessing this information is trivial because it is already created during inference by the MoE router when it computes the top-𝑘 expert assignment for each token. GEM simply collects this information while serving requests to form an expert utilization trace and then uses this trace to enable GPU-variability-aware expert mapping. What is the right trace length? There exists a tradeoff between the quality of the expert utilization trace and the time to deployment. A short trace collected over a small number of time steps successfully captures the utilization patterns of the consistent experts but often fails to capture the utilization of the temporal experts because they are used in a small subset of time steps. In contrast, a longer trace captures both experts but delays deployment and reduces the efficacy of GEM. Ideally, we want to find the shortest trace length that captures the utilization patterns of both consistent and temporal experts. We conduct an experiment to confirm our hypothesis and identify the sweet operating point by sweeping the trace length from 1 to 256, placing experts based on the trace, and measuring the end-to-end latency reduction on 500 unseen requests from the ShareGPT [32] dataset. Figure 10 shows the relationship between the trace length and end-to-end latency for three architecturally distinct MoE models: Qwen3-30B-A3B, Hunyuan-A13B, and Llama-4-Scout. From this study, we make two observations. First, a single time step trace does not fully capture the expert utilization patterns of temporal experts and therefore has high end-to-end latency, as expected. For Llama-4-Scout, this short trace results in an expert mapping which has a 2.2 % worse end-to-end latency than vLLM’s default linear expert mapping. This data highlights the importance of temporal experts to reducing latency. Second, the performance saturates at 16 time steps for all three models. This indicates that expert utilization patterns are stable over time. If utilization had drifted, longer traces would continue to improve the mapping. Based on this study, GEM selects 16 as the default trace window length across models.

7.5

Saturation

5.0 2.5

Qwen3-30B-A3B Hunyuan-A13B Llama-4-Scout

0.0 2.5

Expert Trace Length (Time Steps)

256

Figure 10. Latency reduction for Qwen3-30B-A3B, HunyuanA13B, and Llama-4-Scout on 500 unseen ShareGPT requests across trace lengths. Even a short trace of 16 time steps suffice to build a robust variability-aware expert mapping. 3.3.2 Step-2: Performance Variability Profiling. GEM profiles the latency of the MoE-layers to capture the throughput variation across GPUs. For this, GEM uses a microbenchmark that launches an isolated MoE expert kernel with synthetic input batches at each target token count and records the average processing latency on each GPU. This produces a per-GPU token-vs-latency curve which is used to score expert mappings in subsequent steps. Fast and accurate profiling: Profiling at an extremely finegranularity, such as each token, offers highly accurate profiles but requires more time during which the GPUs are unusable for actual inference tasks. For example, on a 4×H200 NVL system, sweeping every token count from 1 to 16K with 500 kernel launches per count takes 12 hours for Llama-4Scout [6]. Also, these profiles often become obsolete as environmental factors influencing dynamic voltage frequency scaling, such as cooling and power setup, change [14, 15, 33]. Performance variability also depends on the model, those with larger experts exhibit more variability than models with smaller experts [3, 6]. In contrast, too coarse fixed intervals of hundreds of tokens fails to capture the latency distribution accurately because latency does not increase smoothly with token count, but follows a staircase pattern [34, 35]. GEM exploits the insight that MoE networks group tokens 6

GEM : GPU-Variability-Aware Expert-to-GPU Mapping For Mixture-of-Experts Models

into fixed-size batches called tiles, typically multiples of 32 or 64 tokens depending on hardware and software configurations [20, 34] and latency only jumps upon crossing tile boundaries. Figure 7 illustrates this on Llama-4-Scout [6], where latency increases sharply at a granularity of 512 tokens per GPU. Thus, the tile size offers the right granularity for both accurate and fast profiling.

weights remains the same across GPUs and sufficient memory capacity is available to hold the KV caches (necessary to hold context information for fast inference) on each GPU. GEM must identify the optimal mapping or a mapping of the experts on to the GPUs that minimizes the impact of stragglers by accounting for both expert utilization as well as GPU performance variability by exploiting the insights described earlier. However, identifying the optimal mapping is infeasible because the number of possible mappings grows combinatorial in the number of experts and GPUs. To address this limitation, GEM employs a heuristic based iterative approach, where it selects an initial mapping, identifies its impact on end-to-end latency, uses this information to find an alternate mapping, and updates the current mapping to this alternate candidate for the subsequent iteration. The process is repeated until the mapping does not improve any further. However, the performance of each iteration depends on the initial mapping that GEM started with. To avoid convergence on sub-optimal mappings, GEM repeats this iterative approach starting with multiple initial mappings, each called a restart. Finally, GEM selects the best expert mapping across all restarts. Our experiments show that about 30 restarts are sufficient for achieving good performance and we use this as the default in our design. Nonetheless, it may be adjusted in the future as models and GPUs evolve. Figure 12 illustrates this selection process, where 𝑀𝑖𝑗 refers to the expert to GPU mapping at the beginning of the 𝑗 𝑡ℎ iteration of the 𝑖 𝑡ℎ restart. GEM generates the initial expert mapping 𝑀𝑖0 for the 𝑖 𝑡ℎ restart. From each 𝑀𝑖0 , GEM runs an iterative refinement procedure yielding a sequence of subsequent mappings (𝑀𝑖1, 𝑀𝑖2, . . . , 𝑀𝑖𝑚 ) which terminates when the latency does not improve further. GEM then selects the mapping 𝑀 ★ with the lowest latency among the 𝐾 final mappings (𝑀0𝑚 , . . . , 𝑀𝑘𝑛 ) for deployment.

Probability

Profiling Strategy: GEM must account for variable token counts processed by each GPU depending on the model during profiling. This variation arises from differences in model architectures, like number of experts per layer and maximum batch size [3, 10, 11]. For example, Figure 11 shows that token counts vary across the Llama-4-Scout [6], Hunyuan-A13B [7], and Qwen3-30B-A3B [5] MoE models when deployed on a 4xH200. GEM therefore adapts the profiled token counts range for each model. For instance, GEM only profiles token counts between 1 and 10K tokens for Llama-4-Scout. Some models, however, have per-GPU token distributions that range up to tens of thousands of tokens, where sampling at even the granularity of tile size requires a large number of samples. We observe that at high token counts, the latency increase per tile shrinks to a small fraction of the total latency. In this range, GEM samples sparsely at intervals of thousands of tokens and reconstructs the full curve by linearly interpolating between the sampled points. Together, these policies make profiling fast: on a 4×H200 NVL system, GEM generates a profile for Llama-4-Scout [6] within 3 minutes, sampling 62 token counts with 500 measurements each.

0.50

Qwen3-30B-A3B Hunyuan-A13B Llama-4-Scout

0.25

Unique Distributions Restart 0

10 K

20 K

30 K

Per-GPU Token Counts

𝑀!!

𝑀!"

𝑀!#

𝑀$!

𝑀$"

𝑀$%

…

0.00

Final

Initial

40 K

Restart k

Figure 11. Per-GPU token counts for Qwen3-30B-A3B, Hunyuan-A13B, and Llama-4-Scout shows that each model receives a different number of tokens per GPU.

Selected

Pick Best

𝑀⋆

Iterative Process

Figure 12. GEM’s expert mapping starts with 𝐾 initial mappings, each iteratively refined until latency saturates, and then selects the final mapping with lowest latency.

3.3.3 Step-3: Variability-Aware Expert mapping. GEM then performs the expert mapping step during which it decides on which GPU to place each expert. For this assignment, the number of experts per GPU is determined by dividing the total number of experts by the number of GPUs, such that each GPU hosts the same number of experts. This ensures that the memory consumed by expert

Scoring Expert Mappings: To assess a mapping, GEM replays the trace from Step-1 in software and predicts the latency of each step (to account for expert utilization) by using the variability profiles from Step-2 (to account for non-uniformity in hardware). Recollect that faster GPUs should process proportionally more tokens than slower ones 7

Sourish Wawdhane, Avinash Kumar, and Poulami Das

so that all GPUs finish at the same time (Insight-1). So, the scoring function, 𝑆 (), captures latency rather than actual token counts. GEM also ensures that consistent and correlated temporal experts are placed on different GPUs to avoid slowdown (Insight-2) by scoring each step separately to appropriately place both types of experts. Given a mapping 𝑀, trace 𝑇 , with per-GPU latency cost 𝐶𝑔 (·), GEM scores the mapping as shown in Equation 1, where 𝑛𝑔 (𝑀, 𝑡) is the number of tokens routed to GPU 𝑔 at step 𝑡 for a given mapping 𝑀. The max() captures the straggler GPU at each step and the outer summation accumulates the impact of stragglers across the entire trace. ∑︁ 𝑆 (𝑀) = max ( 𝐶𝑔 𝑛𝑔 (𝑀, 𝑡) ) (1)

mapping. GEM therefore tries to find this pair of experts and swaps them to find 𝑀𝑖𝑗+1 such that 𝑆 (𝑀𝑖𝑗+1 ) < 𝑆 (𝑀𝑖𝑗 ). Of all such expert pairs, GEM selects the pair with minimal 𝑆 (𝑀𝑖𝑗+1 ). This procedure is repeated until no better mapping is found, terminating at 𝑀𝑖𝑚 . A key challenge of running this type of search is that the quality of the final mapping depends on the initial mapping. Thus, to improve the quality of the final mappings and optimize the search process, GEM therefore starts with carefully chosen values of 𝑀𝑖0 . This initial mapping starts with all GPUs empty. Then, GEM sorts all experts from the most to the least utilized and places each expert on the GPU that minimizes 𝑆 (𝑀𝑖0 ). This heuristic operates based on the intuition that the most used experts have the maximum impact on latency and therefore, they must be placed first. But this starting point may result in a sub-optimal final mapping 𝑀𝑖𝑚 due to the heuristic nature of the search. So, GEM operates with 𝑘 initial mappings 𝑀00, . . . , 𝑀𝑘0 . To ensure that these mappings are unique, GEM adds a 20% noise component to the utilization metric used to sort experts. Our experiments show that this search algorithm converges in under 18 expert swaps for all models evaluated, meaning that 𝑀𝑖0 is already a good initial mapping, and the search algorithm just refines it. More detailed description of our algorithms are presented in Appendix B.

𝑔 𝑡 ∈𝑇

We explain this scoring process using the illustration in Figure 13. Consider the time step 0 of the expert trace depicted in Figure 13(a), where experts 𝐸 0, 𝐸 1, 𝐸 2, 𝐸 3 each receive 1, 2, 3, 3 tokens respectively. We compute per-GPU token counts for this step as 1 + 2 = 3 and 3 + 3 = 6 for GPU-0 and GPU-1 respectively. Then, the variability profiles of GPU-0 and GPU-1 are consulted to determine latencies to process these many tokens in each GPU, as shown in Figure 13(b). So, for step 0, the latency cost of processing the tokens is computed as 𝐶 0 (3) = 2 for GPU-0 and 𝐶 1 (6) = 5 for GPU-1. Finally, the score for the mapping is computed as the sum of straggler latencies over all time steps. Figure 13(c) shows that for time steps 0, 1, and 2, GPU-1, GPU-0, and GPU0 are the straggler GPUs respectively with latencies 5,4, and 4 respectively. These latencies are summed as 5 + 4 + 4 = 13 to compute the score for this expert mapping. GPU Variability Profiles

1 2 E0 E1

3 3 E2 E3

GPU-1 GPU-0

2 3

Tokens

Require: utilizations 𝑢 Ensure: expert-to-GPU mapping 𝑀𝑚 1: 𝑀0 ← ∅ // Initial mapping: place heaviest experts first 2: for 𝑒 in experts sorted by 𝑢+0.2·noise do 3: place 𝑒 on GPU minimizing increase in 𝑆 (𝑀0 ) 4: end for 5: repeat // Refine via best cross-GPU swap 6: 𝑠 ← 𝑆 (𝑀𝑖 ) 7: (𝑒𝑎 , 𝑒𝑏 ) ← swap expert pair that most reduces 𝑆 (𝑀𝑖 ) 8: 𝑀𝑖+1 ← Swap(𝑀𝑖 , 𝑒𝑎 , 𝑒𝑏 ) 9: until 1 − 𝑆 (𝑀𝑖 )/𝑠 < 0.001 10: return 𝑀𝑚

Mapping Score Time Steps

GPU-1 2 0

Latency

Time Steps

Expert Trace GPU-0 4 1

Algorithm 1 GEM’s Search Algorithm

GPU-0 4

GPU-1 4

2 C0

5 C1

Token Counts GPU-1 GPU-0 1+2= 3 3+3= 6

Step-0 Latency Cost GPU-1 GPU-0 C0( 3 )=2 C1( 6 )=5

Total Mapping Score

(a)

(b)

(c)

S= 5+4+4 = 13

Figure 13. (a) Example of an expert trace used to determine the token count of each time step. (b) which is then used to compute the latency cost per GPU by consulting the variability profiles. (c) The mapping score is computed by summing the cost of the straggler GPUs over all time steps in the trace.

3.3.4 Step-4: Deployment and Serving. Once Step-3 produces an expert mapping for each MoE layer, GEM loads each expert’s weights onto its assigned GPU at model load time, thereby replacing the initial linear mapping with the taskspecific GPU variability-aware mapping. The MoE router continues to dispatch each token to its top-k experts as before, but the experts now reside on GPUs chosen by GEM’s variability-aware search.

Finding Alternate Mappings For Iterative Search: The objective of GEM’s iterative search algorithm, outlined in Algorithm 1, is to take a mapping as an input and find an alternative mapping that outperforms it. The search process is based on a simple intuition: if 𝑀𝑖𝑗 is sub-optimal, then there exists two experts that are poorly placed in the current

Time to deployment: GEM is very fast. For instance, variability profiling for Llama-4-Scout takes 2.13 minutes and mapping takes just 8.8 seconds. These costs are negligible and the benefits of GEM far outweigh this latency in longrunning deployment scenarios that serve requests for hours or days [36, 37]. 8

GEM : GPU-Variability-Aware Expert-to-GPU Mapping For Mixture-of-Experts Models

Evaluation Methodology

Norm. Throughput

We discuss the methodology used to evaluate GEM. 4.1

Models

We evaluate GEM on five state-of-the-art MoE models spanning the Mixtral [3], Hunyuan [7], Qwen3 [5], and Llama4 [6] families comprising 8 to 128 routed experts per layer and 30B to 141B total parameters, as shown in Table 1. Such a comprehensive evaluation across diverse MoE architectures shows GEM’s widespread applicability.

Layers

Experts/Layer

Params

Mixtral-8x7B

47B

Mixtral-8x22B

141B

Llama-4-Scout

109B

Hunyuan-A13B

80B

Qwen3-30B-A3B

128

30B

0.85 444

480 500

550

600

Power Cap (W) Figure 14. We profile power caps vs throughput to select per-GPU caps that emulate a desired variability setup. Table 2. Per-GPU power caps (W) for each variability profile. Variability

4.3 4.2

4.0% 0.95 7.7% 0.90 12.0% 400 418

Table 1. MoE models used to evaluate GEM. Model

1.00

Experimental Setup

GPU 0

GPU 1

GPU 2

GPU 3

High

418

600

Moderate

418

444

480

600

Low

600

Baselines

We compare GEM against two expert-mapping strategies natively supported in vLLM. 1. Linear (vLLM default) splits experts into equally sized contiguous groups by index, such that each expert 𝑖 on an 𝑁 GPU system is placed on GPU ⌊𝑖/𝑁 ⌋ 2. EPLB (vLLM’s expert-parallel load balancer) sums perGPU token counts across time steps and periodically redistributes experts to balance load.

We conduct our experiments on a compute node with 4 NVIDIA H200 GPUs (141 GB HBM3 each) and a dual-socket AMD EPYC 9354 32-core CPU. All GPUs are connected through a full-mesh NVLink fabric (900 GB/s per GPU). We implement GEM in vLLM [20], a state-of-the-art LLM inference framework. We deploy MoE models with expert and tensor parallelism ranks of 4 to reduce memory requirements and enable large batches. Emulating Performance Variability: As we only have access to a limited number of GPUs (only 4), we scale their performance (throughput) to emulate the performance variability observed at large server scale. This approach is based on prior works [33, 38, 39]. We emulate three different GPU setups corresponding to three unique performance variability characteristics. We derive these emulated setups from the throughput variation observed across 128 NVIDIA L40 GPUs during our characterization experiments. The studies show that the slowest GPU is 12% slower than the average GPUs, whereas the fastest GPU is 11% faster. The low-variability setup assumes all 4 GPUs in the cluster have similar performance, corresponding to the mean GPU performance from the characterization. The moderate-variability setup represents the average variability differences observed in our characterization studies. To obtain this, we average variation across 1000 Monte Carlo samples [40] of size four from the throughput distribution. The high variability setup assumes a single straggler GPU which is 12% slower than others, consistent with the slowest GPU from our characterization study. We calibrate per-GPU power caps to match the requirements of each variability setup. Figure 14 shows this calibration for for Llama-4-Scout. We invert this curve to pick power caps that produce each normalized throughput, listed in Table 2.

4.4

Datasets

We evaluate GEM on 4K requests from two datasets representative of LLM workloads. 1. ShareGPT [32] is a collection of real-world conversations sourced from a live inference server. 2. CodeContests [41], sourced from Codeforces, is a competitive programming dataset representing technical workloads. 4.5

Figure-of-Merit

We use two metrics. First, end-to-end latency, defined in Equation (2), which is the elapsed time from the arrival of the request 𝑡 arrival to the emission of the final token 𝑡 finish : 𝑡 e2e = 𝑡 finish − 𝑡 arrival,

(2)

Second, we report the 90th-percentile time-per-output-token (p90 TPOT), defined in Equation (3): p90 TPOT = 𝑄 0.90 (Δ𝑡) ,

(3)

where Δ𝑡 denotes inter-token latencies (time between consecutive output tokens) across all decoded tokens in the evaluation set, and 𝑄 0.90 (·) is the empirical 90th percentile. This metric quantifies the latency of the slowest decode steps, which affect users’ perceived streaming smoothness. 9

Sourish Wawdhane, Avinash Kumar, and Poulami Das

E2E Latency Reduction (%)

Linear

EPLB

GEM

High Variability

Moderate Variability

Low Variability

Mixtral Hunyuan Llama-4 Mixtral Qwen38x22B A13B Scout 8x7B 30B-A3B

High Variability

Moderate Variability

Low Variability

Mixtral Hunyuan Llama-4 Mixtral Qwen38x22B A13B Scout 8x7B 30B-A3B

10 0 10

(a) ShareGPT

10 0 10

(b) CodeContests Figure 15. End-to-end latency reduction relative to linear mapping across all five MoE models (higher is better) on (a) ShareGPT and (b) CodeContests for three different GPU variability setups (high, medium, low).

Results

16.9% in the best case, exceeding the mean-latency improvements. This is because per-step latency is bounded by the straggler GPU, and the p90 tail is dominated by steps where heavily used or correlated experts are placed on a slow GPU. GEM reduces tail latency by placing experts to balance perGPU latencies (Insight-1) and separating correlated temporal and consistent experts across GPUs (Insight-2). We present tail latency results for all variability setups in Appendix C.

In this section, we evaluate GEM against baselines for two datasets, compare expert mappings against baselines, and quantify the cost of GEM’s variability-profiling. 5.1

End-to-End Latency

GEM outperforms both baselines across all configurations, as shown in Figure 15. As expected, the efficacy of GEM depends on the GPU variability. The end-to-end latency for the high-variability setup is reduced by 7.9% on average and by up to 16.5% in the best-case. For the moderate-variability setup, it reduces by 5.9% on average and by up to 10.4%. Lastly, for the low-variability setup, it reduces by 1.5% on average and by up to 4.5%. We observe two key trends. First, models with a few large experts concentrate utilization on a small number of GPUs and benefit the most from variabilityaware mapping. For example, Mixtral-8x22B benefits the most on the high and moderate variability setups (by up to 15.0% and 9.9%). In contrast, Qwen3-30B-A3B’s near-uniform routing across 128 small experts benefits the least (around 1.5%). Second, even when GPU variability is low, GEM still offers latency reduction by appropriately placing correlated temporal experts. For instance, Llama-4-Scout has the most improvement (3.5%) because it is rich in temporal experts while Qwen3-30B-A3B has the least (0.4%). 5.2

p90 TPOT Reduction (%)

Linear

Tail-Latency (p90 TPOT)

EPLB

GEM

20 10 0 Mixtral 8x22B

Hunyuan A13B

Mixtral 8x7B

Llama-4 Scout

Qwen330B-A3B

Mixtral 8x22B

Mixtral 8x7B

Llama-4 Scout

Hunyuan A13B

Qwen330B-A3B

(a) ShareGPT

20 10 0

(b) CodeContests

Figure 16. p90 TPOT latency reduction over linear mapping on the high-variability profile, for (a) ShareGPT and (b) CodeContests. Bars are normalized against linear mapping.

GEM’s gains are most pronounced on tail latency which is critical to perceived responsiveness. Figure 16 shows that GEM reduces p90 TPOT by 9.1% on average and by up to 10

GEM : GPU-Variability-Aware Expert-to-GPU Mapping For Mixture-of-Experts Models

Consistent

GPU-0

GPU-1

GPU-2

Time Steps

10 5

5 0

0 0 1 2 3

4 5 6 7

8 9 10 11 12 13 14 15

(a) Linear mapping (vLLM default) GPU-0

GPU-1

GPU-2

GPU-3

Time Steps

10 5

Utilization (%)

5 0

0 0 1 3 11

5 9 12 13

2 8 10 14

5.4

4 6 7 15

(b) vLLM EPLB GPU-0

GPU-1

GPU-2

GPU-3

Time Steps

10 5

5 1 4 9 11

0 2 6 13

5 8 10 14

Profile Time (min)

Variability Profiling Latency

Figure 18 shows that GEM completes variability profiling for all five MoE models within 0.5 to 3.6 minutes by sampling only at token counts where latency actually changes for each model. This approach significantly reduces the latency compared to a fine-grained sampling method which sweeps token counts from 1 to 16K and takes 3.4 to 20.5 hours. This results in a 265 to 515× reduction in the offline profiling cost. The savings are largest for models with wider FFNs, such as Mixtral-8x22B [3], Mixtral-8x7B [3], and Llama-4-Scout [6], where each kernel launch is expensive. These latency savings result in lower system downtimes and reduce time to deploy GEM in real inference serving frameworks.

Utilization (%)

Comparison of Expert Mapping Policies

We compare the expert mapping of GEM against the two baselines using a four GPU setup, where GPU-0 is the slowest. For illustration, we only show the utilization of 16 experts of Layer 43 of Llama-4-Scout over 25 time steps. The vLLM default, linear expert mapping, allocates four experts to each GPU in a linear fashion based on expert indices, such that experts 0 to 3 are placed on GPU-0, experts 4 to 7 are placed on GPU-1, and so on, as shown as Figure 17(a). This is sub-optimal because this expert-to-GPU allocation results in placing consistent expert 2 and temporal experts 0 and 3 on the slowest GPU-0. vLLM’s expert-parallel load balancer, EPLB, adjusts this allocation and places the consistent expert 2 on GPU-1, as shown in Figure 17(b). As a result, EPLB reduces end-to-end latency by 1.0% compared to the default linear mapping. However, temporal experts 0 and 3 still remain on GPU-0. In contrast, GEM optimizes the GPU allocation for both consistent and temporal experts. As shown in Figure 17(c), GEM ensures that the consistent experts 2, 5, and 15 as well as temporal experts 0 and 3 are allocated different GPUs and are not allocated GPU-0. It tries to minimize the overall utilization of GPU-0 compared to the other GPUs. As expected, such a GPU-variability-aware expert mapping enables GEM to reduce end-to-end latency by 6.2% and 7.1% compared to the EPLB and linear mapping policies respectively.

GPU-3

Utilization (%)

5.3

Temporal

3 7 12 15

(c) GEM Figure 17. Comparison of expert mapping strategies for Layer 43 of Llama-4-Scout on the high-variability setup. (a) vLLM default linear policy places experts based on indices, resulting in consistent and temporal experts being placed on GPU-0 (slowest). (b) The EPLB policy improves the allocation of the consistent experts but the temporal experts still remain on GPU-0. (c) GEM’s expert mapping places both temporal and consistent experts on separate GPUs and avoids placing highly utilized experts on GPU-0.

Full sweep

GEM

10 hr 1 hr 10 min 1 min Mixtral 8x22B

Mixtral 8x7B

Llama-4 Hunyuan Qwen3Scout A13B 30B-A3B

Figure 18. GEM reduces the latency to build per-GPU tokenvs-latency curves as compared to a full sweep which profiles every token count from 1 to 16 K. 11

Sourish Wawdhane, Avinash Kumar, and Poulami Das

Performance Variability at Scale

Dynamic expert provisioning: Several prior works handle expert imbalance by replicating high utility experts. These methods primarily differ in how they decide which experts to replicate. EPLB [4] uses recent expert utilization data to predict which experts are likely to be used. Lina [25] uses prior layers’ expert utilization to predict future layers’ experts. Libra [24] directly runs hidden states through future layers’ MoE routers. HarMoEny [22] waits for exact routing results to replicate experts. In all cases, the parameter weights of replicated experts compete with KV-caches for GPU memory, constraining context lengths [13]. These works are agnostic of performance variability across GPUs.

The variability quantified in Section 2.4 is a problem that is expected to worsen for large-scale GPU deployments as both models and systems scale. This is because the performance gap between the slowest GPU and the fastest scales as more GPUs are aggregated into a single deployment. To model the effects, we use the throughput distributions of the 128 NVIDIA L40 GPUs we characterized. For each system size 𝑁 we sample from the distribution using 10,000 Monte Carlo samples [40] and compute the throughput gap between the slowest sampled GPU and the fastest.

Norm. Throughput

Fastest GPU

Mean

Slowest GPU

Training-Time MoE optimization: A separate line of work introduces mechanisms during training to encourage balanced expert activation. For instance Switch Transformer [11] and GShard [12] use expert-imbalance losses during training. However, this interferes with the model’s training objective and degrades accuracy [16, 42]. By contrast, Loss-Free Balancing [16] adds routing biases to steer tokens toward underused experts, but this weakens expert specialization [4]. Expert choice [42] balances loads by training experts to select top-k tokens (instead of tokens selecting top-k experts). However, this makes generation non-deterministic since the token composition of a batch changes which experts each token uses. Moreover, these methods do not address inference-time hardware heterogeneity, where identical expert loads produce different latencies across GPUs.

1.0

Gap Widens

0.9 0.8 2

System size N (GPUs)

Figure 19. Expected per-GPU throughput across 𝑁 GPUs, normalized to the fastest GPU, shows that the effect of variability grows with system size 𝑁 . Figure 19 shows that the expected slowest-to-fastest throughput gap grows monotonically with 𝑁 , increasing from 11.9% at 𝑁 =4 (the scale of our evaluation setup) to 23.4% at 𝑁 =64. This trend shows the importance of variability-aware expert mapping at scale, where straggler effects dominate.

Conclusion

Mixture-of-Experts (MoE) models reduce the per-token computational cost of LLM inference. However, their end-toend latency is increased by straggler GPUs at every layersynchronization barrier which are caused by differences in expert utilization across GPUs and hardware performance variability. Prior works place experts to balance token counts processed by each GPU but remain unaware of the underlying hardware variability. Therefore, this mapping causes the slowest GPU to remain a persistent straggler even if each GPU receives an equal number of tokens. In this paper, we propose GEM which is to our knowledge the first hardware variability aware expert mapping framework for MoE inference. GEM exploits two key insights: (1) that latencies, not token counts should be balanced across GPUs, and (2) that heavily used experts (both consistent and correlatedtemporal) must be separated across GPUs. GEM applies these insights to construct a four-step pipeline that captures an expert utilization trace, profiles MoE-layers at different token sizes, runs an iterative search to find a task-specific, GPUvariability-aware expert mapping, and deploys the resulting mapping during inference. Across five state-of-the-art MoE models and two representative workloads GEM improves end-to-end latency by 7.9% on average and up to 16.5% over baseline expert mapping strategies.

Related Work

In this section, we discuss related work and compare or contrast as appropriate. Reducing communication overheads: Various prior works focus on reducing the impact of communication between GPUs in expert parallel settings. MoETuner [23] places experts correlated across adjacent layers on the same GPU. However, this yields limited latency benefits, as each token must return to its origin GPU (where its KV-cache resides) before being re-dispatched to the next layer’s experts. ExFlow overcomes this by replicating the KV-cache of each request across all GPUs, but introduces memory overheads that limits batch sizes. Occult [27] places experts which are likely to be used together on the same GPU, but this approach concentrates heavy experts, exacerbating expert imbalance. Speculative-MOE [26] reduces latency by dispatching tokens earlier to GPUs which are predicted to be used in advance of routing decisions. These approaches are orthogonal to GEM because they all overlook the non-uniformity of the underlying GPU hardware. The methods can be combined with GEM for even higher benefits. 12

GEM : GPU-Variability-Aware Expert-to-GPU Mapping For Mixture-of-Experts Models

Acknowledgments

report, 2025. [5] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. [6] Meta AI. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. https://ai.meta.com/blog/llama-4-multimodalintelligence/, April 2025. Accessed: 2026-05-06. [7] Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, Jiahao Bu, Zhongzhi Chen, Xuemeng Huang, Fengzong Lian, Saiyong Yang, Jianfeng Yan, Yuyuan Zeng, Xiaoqin Ren, Chao Yu, Lulu Wu, Yue Mao, Tao Yang, Suncong Zheng, Kan Wu, Dian Jiao, Jinbao Xue, Xipeng Zhang, Decheng Wu, Kai Liu, Dengpeng Wu, Guanghui Xu, Yanggan Chen, Shengyi Chen, Shuang Chen, Xiao Feng, Yigeng Hong, Junqiang Zheng, Chengcheng Xu, Zongwei Li, Xiong Kuang, Jianglu Hu, Yiqi Chen, Yuchi Deng, Guiyang Li, Ao Liu, Chenchen Zhang, Shihui Hu, Zilong Zhao, Zifan Wu, Yao Ding, Weichao Wang, Han Liu, Roberts Wang, Hao Fei, Peijie She, Ze Zhao, Xun Cao, Hai Wang, Fusheng Xiang, Mengyuan Huang, Zhiyuan Xiong, Bin Hu, Xuebin Hou, Lei Jiang, Jiajia Wu, Yaping Deng, Yi Shen, Qian Wang, Weijie Liu, Jie Liu, Meng Chen, Liang Dong, Weiwen Jia, Hu Chen, Feifei Liu, Rui Yuan, Huilin Xu, Zhenxiang Yan, Tengfei Cao, Zhichao Hu, Xinhua Feng, Dong Du, Tinghao She, Yangyu Tao, Feng Zhang, Jianchen Zhu, Chengzhong Xu, Xirui Li, Chong Zha, Wen Ouyang, Yinben Xia, Xiang Li, Zekun He, Rongpeng Chen, Jiawei Song, Ruibin Chen, Fan Jiang, Chongqing Zhao, Bo Wang, Hao Gong, Rong Gan, Winston Hu, Zhanhui Kang, Yong Yang, Yuhong Liu, Di Wang, and Jie Jiang. Hunyuan-large: An open-source MoE model with 52 billion activated parameters by tencent, 2024. [8] DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [9] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024. [10] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 1280–1297, 2024. [11] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. [12] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint:2006.16668, 2020. [13] Adrian Zhao, Zhenkun Cai, Zhenyu Song, Lingfan Yu, Haozheng Fan, Jun Wu, Yida Wang, and Nandita Vijaykumar. Craft: Cost-aware expert replica allocation with fine-grained layerwise estimations, 2026. [14] Prasoon Sinha, Akhil Guliani, Rutwik Jain, Brandon Tran, Matthew D Sinclair, and Shivaram Venkataraman. Not all gpus are created equal: characterizing variability in large-scale, accelerator-rich systems. In SC22, pages 01–15. IEEE, 2022.

We thank Sujay Sanghavi for some of the technical discussions during the course of this project and acknowledge the Texas Advanced Computing Center (TACC) and the Center for Generative AI (CGAI) at UT Austin for providing computational resources that helped develop the research results reported in this paper. We thank the generous support from the Cockrell School of Engineering and the Amazon AI PhD Fellowship Program through the Amazon Science Hub at UT Austin, including the AWS cloud credits provided through the fellowship, which enabled the large-scale experiments conducted in this work. Poulami Das acknowledges the generous support through the AMD endowment at UT Austin.

References [1] Zehua Pei, Lancheng Zou, Hui-Ling Zhen, Xianzhi Yu, Wulong Liu, Sinno Jialin Pan, Mingxuan Yuan, and Bei Yu. Cmoe: Converting mixture-of-experts from dense to accelerate llm inference. arXiv preprint arXiv:2502.04416, 2025. [2] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories, 2021. [3] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. [4] DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, et al. Deepseek-v3 technical 13

Sourish Wawdhane, Avinash Kumar, and Poulami Das

[15] Rutwik Jain, Brandon Tran, Keting Chen, Matthew D. Sinclair, and Shivaram Venkataraman. Pal: A variability-aware policy for scheduling ml workloads in gpu clusters, 2024. [16] Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408.15664, 2024. [17] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 1991. [18] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [19] Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, 2025. [20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023. [21] Shwai He, Weilin Cai, Jiayi Huang, and Ang Li. Capacity-aware inference: Mitigating the straggler effect in mixture of experts. arXiv preprint arXiv:2503.05066, 2025. [22] Zachary Doucet, Rishi Sharma, Martijn de Vos, Rafael Pires, AnneMarie Kermarrec, and Oana Balmau. HarMoEny: Efficient multi-gpu inference of MoE models. arXiv preprint arXiv:2506.12417, 2025. [23] Seokjin Go and Divya Mahajan. Moetuner: Optimized mixture of expert serving with balanced expert placement and token routing. arXiv preprint arXiv:2502.06643, 2025. [24] Jaehoon Yang, Yushin Kim, Seokwon Moon, Yeonhong Park, and Jae W. Lee. Libra: Effective yet efficient load balancing for large-scale moe inference. In The 14th ICLR, 2026. [25] Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. Accelerating distributed MoE training and inference with lina. In 2023 USENIX ATC, pages 945–959. USENIX Association, 2023. [26] Yan Li, Pengfei Zheng, Shuang Chen, Zewei Xu, Yuanhao Lai, Yunfei Du, and Zhengang Wang. Speculative moe: Communication efficient parallel moe inference with speculative token and expert prescheduling. arXiv preprint arXiv:2503.04398, 2025. [27] Shuqing Luo, Pingzhi Li, Jie Peng, Hanrui Wang, Yang, Zhao, Yu, Cao, Yu Cheng, and Tianlong Chen. Occult: Optimizing collaborative communication across experts for accelerated parallel moe training and inference. 2025. [28] Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S Lee, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee. Toward efficient inference for mixture of experts. Advances in Neural Information Processing Systems, 37:84033–84059, 2024. [29] Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Demons in the detail: On implementing load balancing loss for training specialized mixture-of-expert models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 5005–5018, 2025. [30] Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, and Dhabaleswar K DK Panda. Exploiting inter-layer expert affinity for accelerating mixture-of-experts model inference. In IPDPS. IEEE, 2024. [31] Karl Pearson. Mathematical contributions to the theory of evolution. iii. regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London. Series A, 187:253–318, 1896. [32] Sharegpt, 2023. https://sharegpt.com. [33] Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, Brijesh Warrier, Nithish Mahalingam, and Ricardo Bianchini. Characterizing power management opportunities for LLMs in the cloud. In Proceedings of the 29th ACM International Conference on Architectural Support

for Programming Languages and Operating Systems (ASPLOS), 2024. [34] Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. MegaBlocks: Efficient sparse training with mixture-of-experts. In Proceedings of Machine Learning and Systems (MLSys), 2023. [35] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in NeurIPS, 2022. [36] Yuxin Wang, Yuhan Chen, Zeyu Li, Zhenheng Tang, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. Burstgpt: A real-world workload dataset to optimize llm serving systems. arXiv preprint arXiv:2401.17644, 2024. [37] Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St. Amant, Chetan Bansal, Victor Rühle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan. Sageserve: Optimizing llm serving on cloud data centers with forecast aware auto-scaling. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 9(3), 2025. [38] Adam Krzywaniak, Paweł Czarnul, and Jerzy Proficz. GPU power capping for energy-performance trade-offs in training of deep convolutional neural networks for image recognition. In Proceedings of ICCS, pages 123–133. Springer, 2022. [39] Jae-Won Chung, Yile Gu, Insu Jang, Luoxi Meng, Nikhil Bansal, and Mosharaf Chowdhury. Reducing energy bloat in large model training. In Proceedings of the 30th SOSP, 2024. [40] Nicholas Metropolis and Stanislaw Ulam. The monte carlo method. Journal of the American Statistical Association, 44(247):335–341, 1949. [41] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with AlphaCode. Science, 378(6624):1092–1097, 2022. [42] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. Mixture-of-experts with expert choice routing, 2022. [43] Kohei Yoshida, Rio Sageyama, Shinobu Miwa, Hayato Yamaki, and Hiroki Honda. Analyzing performance and power-efficiency variations among nvidia gpus. In Proceedings of the 51st ICPP, pages 1–12, 2022. [44] Abhishek Tiwari, Smruti R Sarangi, and Josep Torrellas. Recycle: Pipeline adaptation to tolerate process variation. ACM SIGARCH Computer Architecture News, 35(2):323–334, 2007. [45] Marco Kurzynski, Shaizeen Aga, and Di Wu. Lit silicon: A case where thermal imbalance couples concurrent execution in multiple gpus. arXiv preprint arXiv:2511.09861, 2025. [46] Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. Analysis of large-scale multitenant gpu clusters for dnn training workloads. In USENIX ATC, 2019. [47] Quan Chen, Hailong Yang, Jason Mars, and Lingjia Tang. Baymax: Qos awareness and increased utilization for non-preemptive accelerators in warehouse scale computers. ACM SIGPLAN Notices, 51(4), 2016. [48] Quan Chen, Hailong Yang, Minyi Guo, Ram Srivatsa Kannan, Jason Mars, and Lingjia Tang. Prophet: Precise qos prediction on nonpreemptive accelerators to improve utilization in warehouse-scale computers. In ASPLOS, pages 17–32, 2017. [49] Xin Xu, Na Zhang, Michael Cui, Michael He, and Ridhi Surana. Characterization and prediction of performance interference on mediated passthrough gpus for interference-aware scheduler. In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19), 2019. [50] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 51st ISCA. IEEE, 2024.

GEM : GPU-Variability-Aware Expert-to-GPU Mapping For Mixture-of-Experts Models

1.44%

4.68%

15.86%

(b)

(c)

(a)

Figure 20. Aggregated TPOT variability across nodes for (a) DeepSeek-R1-Qwen-7B on Amazon Trainium (BS=32), (b) Qwen2.5-72B on AMD MI300X (BS=128), and (c) Mistral-7B on NVIDIA L40 (BS=128). The vertical lines mark the P5 and P95 values, bounding the central 90% of the distribution. 7.3% 7.1%

(a)

(b)

(c)

Figure 21. Intra-node TPOT variability across a NVIDIA 8xL40 node for DeepSeek-R1-Qwen-7B at batch sizes (a) 32, (b) 64, (c) 128. The median deviation between the best (GPU 7) and the worst (GPU 5) increases with batch size, showing that larger batches widen performance variability across GPUs within the same node.

Variability in GPU Systems

for the majority of end-to-end latency in generative workloads [50]. Figure 20 shows the aggregated TPOT across several nodes of a given platform. We observe that Amazon Tranium shows the tightest spread with samples within a range of only 1.44%, while NVIDIA L40 has the highest spread and shows 15.86% variability, with AMD MI300X falling in between. Moreover, even for GPUs within a single node show increasing variability with batch size. For example, Figure 21 shows that the difference between TPOT for GPU device 5 (slowest) and GPU device 7 (fastest) increases with batch size. This is because larger batches increase perGPU power consumption and push each GPU closer to its thermal limits, this widens the variation in operating frequency on each GPU device. Furthermore, we note that the identity of the slowest and fastest GPUs remain identical across batch sizes, confirming that variability observed is persistent. This observed variability has direct implications for MoE inference. In expert-parallel deployments, experts are distributed across GPUs and each inference step requires allto-all communication to route tokens to the selected expert. In these scenarios, the slowest GPU becomes the bottleneck or “straggler" at every step and forces faster GPUs to idle at layer synchronization barriers. Thus, expert mapping strategies must account for variability in GPU performance to maximize MoE inference throughput.

Modern GPU-accelerated systems exhibit significant performance variability, so devices running identical workloads can produce measurably different execution times. This variability stems from several hardware and system level factors. At the hardware level, process variation during manufacturing produces differences in achievable clock frequencies and leakage characteristics across dies [14, 43, 44]. During runtime, dynamic voltage and frequency scaling (DVFS) continuously adjusts GPU clock frequency to stay within power and thermal limits. However, these adjustments depend on the thermal state of each GPU, which varies with physical placement and cooling mechanisms, causing GPUs in the same node to operate at different frequencies for the same workload. At the system level, OS scheduling, PCIe contention and virtualization layers further contribute to performance differences across GPUs [45–49]. To quantify the effects of variability on LLM workloads, we conducted a long-term measurement study across three GPU platforms: Amazon Tranium, AMD MI300X, and NVIDIA L40. We run state-of-the-art LLMs using vLLM with prompts from the ShareGPT [32] dataset. We report the variation in time per output token (TPOT), as decode iterations account 15

Sourish Wawdhane, Avinash Kumar, and Poulami Das

GEM’s Expert Mapping Algorithm

each iteration. The loop terminates when no swap reduces the score by more than 0.1%, indicating the mapping is stable.

This appendix describes the full expert mapping algorithm that is summarized in Section 3.3.3. The expert mapping algorithm consists of two components: (i) an algorithm that constructs an initial expert mapping, and (ii) an iterative refinement algorithm that improves the initial mapping by evaluating alternate mappings. These two routines are combined in an outer loop that performs 𝐾 restarts and returns the lowest-scoring mapping across all restarts.

Algorithm 3 Refine (𝑀𝑖0,𝑇 , 𝐶) Require: initial mapping 𝑀𝑖0 , trace 𝑇 , curves 𝐶 Ensure: refined mapping 𝑀𝑖𝑚 1: 𝑀 ← 𝑀𝑖0 2: repeat 3: 𝑠𝑝𝑟𝑒𝑣 ← 𝑆 (𝑀); 𝑏𝑒𝑠𝑡 ← None; 𝑏𝑒𝑠𝑡𝐷𝑟𝑜𝑝 ← 0 4: for each pair (𝑒𝑎 , 𝑒𝑏 ) on different GPUs in 𝑀 do 5: // try all cross-GPU swaps 6: 𝑀 ′ ← Swap(𝑀, 𝑒𝑎 , 𝑒𝑏 ) 7: 𝑑𝑟𝑜𝑝 ← 𝑠𝑝𝑟𝑒𝑣 − 𝑆 (𝑀 ′ ) 8: if 𝑑𝑟𝑜𝑝 > 𝑏𝑒𝑠𝑡𝐷𝑟𝑜𝑝 then 9: 𝑏𝑒𝑠𝑡𝐷𝑟𝑜𝑝 ← 𝑑𝑟𝑜𝑝 10: 𝑏𝑒𝑠𝑡 ← (𝑒𝑎 , 𝑒𝑏 ) 11: end if 12: end for 13: if 𝑏𝑒𝑠𝑡 = None then break 14: // no swap improves score 15: end if 16: 𝑀 ← Swap(𝑀, 𝑏𝑒𝑠𝑡) // commit best swap 17: until 𝑏𝑒𝑠𝑡𝐷𝑟𝑜𝑝 / 𝑠𝑝𝑟𝑒𝑣 < 0.001 // converged 18: 𝑀𝑖𝑚 ← 𝑀 19: return 𝑀𝑖𝑚

Notation. We use the following notation: • 𝑀𝑖𝑗 : the expert-to-GPU mapping at the beginning of the 𝑗 th iteration of the 𝑖 th restart. 𝑀𝑖0 is the initial mapping for restart 𝑖, and 𝑀𝑖𝑚 is the final mapping. • 𝑇 : the expert utilization trace collected in Step-1.𝑇 [𝑡] [𝑒] is the number of tokens routed to 𝑒 at time step 𝑡. • 𝐶𝑔 (·): the per-GPU latency cost function from the variability profile in Step-2. • 𝑢: per-expert mean utilization derived from 𝑇 . • 𝑆 (𝑀): the score defined in Section 3.3.3, Equation 1 that is used to evaluate mappings. • Swap(𝑀, 𝑒𝑎 , 𝑒𝑏 ): the mapping obtained after the positions of experts 𝑒𝑎 and 𝑒𝑏 are exchanged in 𝑀. • 𝑀 ∪ {𝑒 → 𝑔}: the mapping obtained by adding the assignment of expert 𝑒 to GPU 𝑔 in 𝑀. B.1

Initial Mapping Construction

Algorithm 2 builds an initial mapping 𝑀𝑖0 greedily for restart 𝑖. Experts are sorted from most to least mean utilization and inserted one at a time onto whichever GPU yields the lowest score on the partial mapping. This ordering reflects the intuition that the heaviest experts have the largest impact on latency and should be placed first. To ensure that subsequent restarts refine a different initial mapping, the utilizations are perturbed by 20% noise on each restart except the first.

B.3

Because the greedy initialization and pairwise refinement are both heuristic, the final mapping depends on the starting point. To mitigate this, Algorithm 4 runs the full pipeline from 𝐾 different initial mappings and returns the lowestscoring result among them. The first restart uses the unperturbed utilizations; subsequent restarts add 20% noise to diversify their starting points (via Algorithm 2).

Algorithm 2 Initialmapping (𝑢, 𝑖,𝑇 , 𝐶) Require: utilizations 𝑢, restart index 𝑖, trace 𝑇 , curves 𝐶 Ensure: initial mapping 𝑀𝑖0 1: 𝑀𝑖0 ← ∅ // start with all GPUs empty 2: if 𝑖 > 1 then // diversify across restarts 3: Add 20% noise to each utilization in 𝑢 4: end if 5: Sort experts by 𝑢, highest first 6: for each expert 𝑒 in sorted order do 7: 𝑔★ ← arg min𝑔 𝑆 (𝑀𝑖0 ∪ {𝑒 →𝑔}) 8: 𝑀𝑖0 ← 𝑀𝑖0 ∪ {𝑒 →𝑔★ } // heaviest first 9: end for 10: return 𝑀𝑖0 B.2

Outer Loop: 𝐾 Restarts

Algorithm 4 GEM-Place (𝐾, 𝑢,𝑇 , 𝐶) Require: restart count 𝐾, utilizations 𝑢, trace 𝑇 , curves 𝐶 Ensure: best expert-to-GPU mapping 𝑀 ★ 1: 𝑀 ★ ← None; 𝑠 ★ ← ∞ 2: for 𝑖 = 1 to 𝐾 do // independent restart 3: 𝑀𝑖0 ← Initialmapping(𝑢, 𝑖,𝑇 , 𝐶) 4: 𝑀𝑖𝑚 ← Refine(𝑀𝑖0,𝑇 , 𝐶) 5: 𝑠 ← 𝑆 (𝑀𝑖𝑚 ) 6: if 𝑠 < 𝑠 ★ then // keep best so far 7: 𝑠 ★ ← 𝑠; 𝑀 ★ ← 𝑀𝑖𝑚 8: end if 9: end for 10: return 𝑀 ★

Iterative Refinement

GEM deploys the mapping 𝑀 ★ returned by Algorithm 4 in Step-4 (Section 3.3). All four algorithms run on the CPU.

Algorithm 3 improves an initial mapping 𝑀𝑖0 by swapping the pair of experts which creates the largest reduction in 𝑆 at 16

GEM : GPU-Variability-Aware Expert-to-GPU Mapping For Mixture-of-Experts Models

Linear

Mean TPOT Reduction (%)

EPLB

GEM

High Variability

Moderate Variability

Low Variability

Mixtral Hunyuan Llama-4 Mixtral Qwen38x22B A13B Scout 8x7B 30B-A3B

High Variability

Moderate Variability

Low Variability

Mixtral Hunyuan Llama-4 Mixtral Qwen38x22B A13B Scout 8x7B 30B-A3B

High Variability

Moderate Variability

Low Variability

Mixtral Hunyuan Llama-4 Mixtral Qwen38x22B A13B Scout 8x7B 30B-A3B

High Variability

Moderate Variability

Low Variability

Mixtral Hunyuan Llama-4 Mixtral Qwen38x22B A13B Scout 8x7B 30B-A3B

10 0 10

p90 TPOT Reduction (%)

(a) Mean TPOT, ShareGPT

10 0 10

p95 TPOT Reduction (%)

(b) p90 TPOT, ShareGPT

10 0 10

p99 TPOT Reduction (%)

10 0 10

(d) p99 TPOT, ShareGPT

Figure 22. TPOT reduction relative to linear mapping on ShareGPT across all five MoE models (higher is better). Rows report the TPOT statistic ((a) mean, (b) p90, (c) p95, (d) p99); columns report the variability setup.

Tail-Latency Characterization

C.1

MoE models are frequently deployed in streaming setups, where responses are sent to users one token at a time. When a few decode steps take longer to execute than most, the user notices visible stalls in token generation. These stalls hurt the user experience of the workload. To understand this, we report GEM’s effect on tail latency, reporting the 90th, 95th, and 99th percentiles of time-per-output-token (TPOT) across both datasets and all three GPU variability setups.

Metrics

Time-per-output-token (TPOT), is the elapsed time between two consecutive output tokens. We evaluate the TPOT distribution across 90th (p90), 95th (p95), and 99th (p99) percentiles as defined in Equation (4): p𝑘 TPOT = 𝑄𝑘/100 (Δ𝑡) ,

(4)

where Δ𝑡 denotes the inter-token latency across all tokens in the evaluation set, and 𝑄𝑞 (·) is the empirical 𝑞-quantile. 17

Sourish Wawdhane, Avinash Kumar, and Poulami Das

Linear

Mean TPOT Reduction (%)

p90 TPOT Reduction (%)

High Variability

Moderate Variability

Low Variability

Mixtral Hunyuan Llama-4 Mixtral Qwen38x22B A13B Scout 8x7B 30B-A3B

High Variability

Moderate Variability

Low Variability

Mixtral Hunyuan Llama-4 Mixtral Qwen38x22B A13B Scout 8x7B 30B-A3B

High Variability

Moderate Variability

Low Variability

Mixtral Hunyuan Llama-4 Mixtral Qwen38x22B A13B Scout 8x7B 30B-A3B

High Variability

Moderate Variability

Low Variability

Mixtral Hunyuan Llama-4 Mixtral Qwen38x22B A13B Scout 8x7B 30B-A3B

0 10

(a) Mean TPOT, CodeContests

10 0 10

p95 TPOT Reduction (%)

GEM

(b) p90 TPOT, CodeContests

10 0 10

p99 TPOT Reduction (%)

EPLB

10 0 10

(d) p99 TPOT, CodeContests

Figure 23. TPOT reduction relative to linear mapping on CodeContests across all five MoE models (higher is better). Rows report the TPOT statistic ((a) mean, (b) p90, (c) p95, (d) p99); columns report the variability setup. C.2

Tail-Latency Results

8.9% on average and by up to 16.8% in the best case. On the moderate-variability setup, it reduces mean TPOT by roughly 6% on average. On the low-variability setup, it reduces mean TPOT by 1% to 2%. Second, TPOT reductions are consistent across the distribution. At a given variability level, the mean, p90, p95, and p99 reductions align to within half a percentage point. This indicates that GEM shifts the entire per-token latency distribution, and improves both the typical streaming experience and the worst-case stalls.

Figures 22 and 23 report TPOT reduction over linear mapping for ShareGPT and CodeContests. In each figure, the rows correspond to a TPOT statistic (mean, p90, p95, p99), and the three columns correspond to the high-variability, moderate-variability, and low-variability setups. We make two observations. First, similar to end-to-end latency, GEM’s benefit grows with the underlying hardware variability. On the high-variability setup, GEM reduces mean TPOT by 18

Record · ID 204749 · SHA-256 b4e7a38abe165084
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.