MoE Pretraining:Parallelism Layout Changes the Model, Not Quality

MoE Pretraining: Parallelism Layout Changes the Model, Not Quality Esther Xin Independent Researcher yixin@microsoft.com Code and data: https://github.com/ethxin0011/moe_parallelism

arXiv:2611.15271v1 [cs.DC] 11 Nov 2026

Abstract Large Mixture-of-Experts (MoE) models train across many machines, and the way work is split—the parallelism layout—changes from run to run. You might resume a job on a different GPU count, resize a cluster, or run an ablation on a new setup.In exact arithmetic the layout should not matter. In real arithmetic it changes the order in which numbers are added, and some MoE rules (load balancing, expert capacity) are often computed per machine. We ask a simple question: does the layout change the trained MoE model? We train the same small MoE language models (25M–117M parameters, FineWeb-Edu, 20M tokens) with the same seed, data and hyperparameters under 9 data-parallel/expert-parallel layouts, in fp32 on CPUs and fp16 on a GPU, and compare every run against how much a new random seed changes the model. Across 80 training runs we find: (1) the layout does not change model quality, but it does change the model—by the end of training the predictions differ about 80% as much as a new seed would make them, and over 90% of routing decisions differ; any layout, a one-bit change to one weight, or a different CPU thread count has the same effect. (2) The cause is the router: routing decisions that flip between runs are almost exactly those where two experts were nearly tied (AUC 0.97–1.00 in the first 16 training steps). (3) Splitting experts more finely makes near-ties more common (2.7% of decisions with 8 experts, 23% with 64) and routing less reproducible (68% vs. 100% of decisions differ), but does not make the outputs drift further. (4) A dense model of the same active size is far more stable: in fp32 it ends 0.12× as far apart as a new seed, and in fp16 0.58×, while the MoE reaches about 0.8× at both precisions and gets there about twice as fast. (5) When a pretraining run is resumed on a new layout, the timing matters: switching at 84% of pretraining (step 1,024 of 1,221) barely changes the final model, while switching at 10% (step 128, just after learning-rate warm-up) changes it substantially. We will release the code, all run logs, and the measured data.

1

Introduction

Mixture-of-Experts (MoE) models replace one large feed-forward block with many smaller “experts” and a small router that sends each token to a few of them (Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2022). Most frontier open models now use this design, often with many finegrained experts (Dai et al., 2024; Yang et al., 2025). Training them needs many machines, and the work is split in two main ways: data parallelism (DP), where each machine processes part of the batch, and expert parallelism (EP), where each machine holds part of the experts and tokens are exchanged between machines (Lepikhin et al., 2021; Yan et al., 2026). We call the choice of DP and EP sizes the layout. In practice the layout is not fixed. A job is preempted and resumed on fewer GPUs, a cluster grows, a team re-runs an ablation on a different machine count. The usual assumption is that this is harmless: with the same seed, data and global batch, the mathematics is identical. But computers add numbers in a fixed number of digits, so (a + b) + c and a + (b + c) can differ in the last digit, and changing the layout changes the order in which gradients and expert outputs are added up. 1

On top of that, several MoE rules are commonly computed per machine rather than over the whole batch—the load-balancing statistics (Qiu et al., 2025), the capacity limit that decides which tokens are dropped, and the counts used by loss-free balancing (Wang et al., 2024). For these, a new layout changes what is computed, not just its rounding. Our question. Does the parallelism layout change the trained MoE model—and if so, why, and does it get worse as experts get more fine-grained? We answer it with a controlled study. We train the same models with the same seed, data order and global batch under different layouts, using real multi-process training with real communication between processes, and compare every pair of runs against three yardsticks: a new random seed, a new data order, and a change of a single weight by the smallest possible amount (one unit in the last place, “1 ULP”). We separate two kinds of layout effects: numerical (only the order of additions changes) and definitional (a per-machine rule switched on). We then look inside the router to find what turns tiny rounding differences into different models, and we repeat the study with 8 to 64 experts, at three model sizes, at two precisions, and with a dense model of the same active size. Main findings. 1. Same quality, different model. Every layout ends with the same validation loss as the reference, within seed-to-seed variation. But the predictions end about 80% as different as a new seed would make them, and over 90% of routing decisions differ. All eight layouts, a 1-ULP change to one weight, and a different CPU thread count are indistinguishable from one another (§5.1). 2. Per-machine rules start large but end the same. Computing load-balancing or capacity per machine creates differences about 400× larger at the start of training, yet the final difference is the same as for a pure layout change (§5.2). 3. The cause is near-ties in the router. The routing decisions that flip between two runs are almost exactly the ones where the router’s last chosen expert and the best unchosen expert had nearly equal scores. Their score gap is several hundred times smaller than usual, and they flip 65–130× more often than other decisions (§5.3). 4. More experts: less reproducible routing, same output drift. With total and active size fixed, going from 8 to 64 experts raises the share of near-tie decisions from 2.7% to 23.3% and the share of routing decisions that differ at the end from 68% to 100%. The difference in the model’s outputs, however, stays at about 0.8× the seed level for every expert count (§5.4). 5. MoE vs. dense, and the role of precision. A dense model of the same active size barely moves in fp32 (0.12× the seed level, and a straight line between the two models has no loss bump). In fp16 it moves more (0.58×). The MoE reaches about 0.8× at both precisions, and reaches the 0.01 level of difference at step 64 instead of 128 (§5.5). 6. Layout sensitivity is pronounced in early pretraining but muted in late pretraining. In the early phase, resuming on a new layout at 10% of pretraining (step 128, shortly after learning-rate warm-up, while the loss is still dropping fast) changes the final model substantially. In the late phase, resuming at 84% of pretraining (step 1,024 of 1,221, after the loss has fallen from 9.8 to about 4.7) leaves the final model almost unchanged. In both phases, a layout switch behaves exactly like a 1-ULP nudge (§5.6). All experiments concern pretraining from scratch: “early” and “late” denote phases within pretraining, not separate mid-training or post-training stages, which we did not study. 2

Why it matters. If you only care about loss, layouts are interchangeable. If you care about getting the same model back—to debug a loss spike, to compare two methods fairly, to reuse routing decisions in reinforcement learning (Ma et al., 2025), or to prune or interpret specific experts—then for an MoE the layout is not a detail. Higher precision alone does not fix this, and finer-grained experts make the routing part worse.

2

Background in Plain Terms

Router and near-ties. For every token, the router gives each expert a score and keeps the top k. If the k-th best and the (k+1)-th best scores are almost equal—a near-tie—a tiny rounding difference can swap which expert gets the token. That expert then receives a different gradient, and the two runs are no longer on the same path. We measure the margin of a decision as the gap between the k-th and (k+1)-th score, divided by the spread of that token’s scores, and call a decision a near-tie when the margin is below 0.01. Layouts. We write a layout as (DP, EP). With EP = 2, for example, each group of two processes shares the experts half-and-half and exchanges tokens inside the group. We keep the global batch the same in every layout, so each step sees exactly the same 32 sequences in the same order. Two kinds of layout effects. A numerical effect changes only the order of additions. A definitional effect changes what is computed: for example, the load-balancing loss uses expert usage counted on one machine instead of the whole batch (Qiu et al., 2025), or the capacity limit is applied per machine. In our code every such rule can be set to “global” (the same in every layout) or “per-rank” (the common implementation), so we can switch each one on separately. How we measure “different”. We compare two runs at the same step on the same 16,384 held-out tokens. Our main measure is the KL divergence between their predicted next-token distributions: 0 means identical predictions. We also report how often their top prediction agrees, how many routing decisions differ, and the loss barrier: the largest rise in loss along the straight line between the two sets of weights (Frankle et al., 2020). A barrier near zero means the two models sit in the same valley of the loss landscape. To make numbers easy to read we divide each KL by the KL between two runs that differ only in their random seed (“vs. own seed”). A value of 1.0 means “as different as a new seed”.

3

Related Work

Training is sensitive to tiny changes. Summers and Dinneen (2021) showed that all sources of training randomness have a similar effect on how different the resulting models are, and that changing one bit of the initial weights is enough to produce a very different model. Kwok et al. (2025) showed that tiny perturbations make runs drift apart mainly during an initial phase of training that they describe as chaotic, and that this sensitivity fades as training proceeds. Zhuang et al. (2021) measured how tooling and hardware choices add randomness. These works study dense models and generic noise. We study the parallelism layout specifically, separate its numerical and definitional parts, compare MoE against dense directly, and explain the MoE case through the router. Layout-invariant computation. He and Thinking Machines Lab (2025) showed that run-to-run differences in LLM inference come from batch-dependent addition order and built kernels that remove them. Zhang et al. (2026) made inference and training produce bit-identical results across 3

tensor-parallel sizes. NVIDIA’s Megatron Core supports bitwise-deterministic pretraining, which guarantees identical results only when, among other things, the parallel configuration is kept fixed (Aithal et al., 2026). EasyScale keeps accuracy consistent when the number of GPUs changes by preserving data-parallel behaviour exactly (Li et al., 2023). These works remove the difference; we measure how large it is when it is not removed, and why MoE models are hit harder. MoE routing instability. In MoE reinforcement learning, small numerical differences between the inference and training engines change which experts are chosen, which can make training collapse; replaying the inference engine’s routing fixes this (Ma et al., 2025). We show the same mechanism during pretraining, caused by the layout, and we identify which decisions flip. Per-machine load balancing. Qiu et al. (2025) showed that computing the load-balancing statistics over the global batch instead of each micro-batch improves quality and expert specialization in large models. Loss-free balancing adjusts a per-expert bias from expert counts (Wang et al., 2024; Jin et al., 2026). These works ask which rule gives a better model. We ask how much switching the rule between per-machine and global changes the model, and how that compares with the pure numerical effect.

4

Experimental Setup

Models and data. We train decoder-only language models from scratch in the style of Qwen3MoE (Yang et al., 2025): QK-Norm, no bias in the attention projections, fine-grained experts, top-k routing and no shared expert (Table 1). The base MoE has about 25M parameters with about 10M active per token. The dense control has one feed-forward block with the same active size. We train on FineWeb-Edu (Penedo et al., 2024) with a 16k-token vocabulary, a global batch of 32 sequences of 512 tokens, and 1,221 steps (20M tokens). The learning rate is constant after warm-up, so every run sees the same schedule. Real parallel training. Every multi-process run uses real processes that communicate through PyTorch’s gloo backend: gradients are summed across processes, and with EP > 1 tokens and expert outputs are exchanged between processes. CPU runs use fp32 on Azure E16s_v3 nodes, with the math libraries pinned to one instruction set so that the CPU generation is not a hidden factor. GPU runs use fp16 with loss scaling on a single NVIDIA T4, with up to four processes sharing the GPU. Before any training, a correctness check confirmed that every layout computes the same gradients as a single process (relative error below 10−4 in fp32), that two runs of the same layout are bit-identical, and that checkpoints move between layouts without change (Appendix A). So any difference we see between layouts is caused by the layout, not by a bug or by random scheduling. Runs. Table 2 lists the 80 training runs. The reference run uses one process. Each other run changes exactly one thing.

5

Findings

4

Item

Setting

Model (decoder-only, trained from scratch) Base MoE 6 layers, width 256, 4 heads, RoPE, RMSNorm, QK-Norm, no QKV bias, tied embeddings Experts 16 SwiGLU experts (width 256), top-4 routing, no shared expert; ∼25M total / ∼10M active parameters Dense control same model, one SwiGLU block of width 1024 (same active size as the MoE) Larger sizes (GPU) m: width 384, 8 layers (∼68M); l: width 512, 8 layers (∼117M) Expert-count sweep (GPU) 8 / 16 / 32 / 64 experts with top-2 / 4 / 8 / 16 and expert width 512 / 256 / 128 / 64, so total and active size stay fixed Load balancing auxiliary loss (coefficient 0.01, softmax router) or loss-free bias (sigmoid router, bias step 10−3 ) Capacity factor 1.25 during training (top-1 choices kept first); no dropping at evaluation Data and training Data Optimizer Batch and length Same in every run Hardware and numerics CPU runs (fp32) GPU runs (fp16) Parallelism

FineWeb-Edu (sample-10BT); 16,384-token byte-level BPE; 120M training tokens, 2M validation tokens AdamW, peak LR 10−3 , 100 warm-up steps then constant, weight decay 0.1, β = (0.9, 0.95), clip 1.0 32 sequences × 512 tokens per step (fixed for every layout); 1,221 steps = 20M tokens initial weights (unless the seed is the variable), data order, global batch, hyperparameters Azure Standard_E16s_v3 nodes, one run per node; real multi-process PyTorch + gloo; up to 8 ranks; math libraries pinned to one AVX2 code path one NVIDIA T4; up to 4 ranks share the GPU; fp16 autocast with loss scaling; collectives through gloo data parallel (DP) ∈ {1, 2, 4, 8}, expert parallel (EP) ∈ {1, 2, 4}, real expert exchange between ranks

Table 1: Model, data, training and hardware settings.

Group

What changes (everything else held fixed)

Layout

9 DP/EP layouts: (1,1) reference, (2,1), (2,2), (4,1), (4,2), (4,4), (8,1), (8,2), (8,4) 4 new seeds, 3 new data orders, 3 one-ULP weight nudges same layout, different CPU thread count load-balancing statistics, capacity unit, or both, switched from global to per-rank at three layouts bias controller fed by global or per-rank expert counts, two layouts, plus 2 seeds 2 seeds, 2 one-ULP nudges, 2 layouts reference pretraining run continued from step 128 (10%) or step 1,024 (84%) on a new layout or with a 1-ULP nudge 3 layouts, 2 seeds, 1 one-ULP nudge sizes m and l: reference, 2 seeds, 1-ULP, 1–2 layouts; base: 1 extra layout 8, 32, 64 experts: reference, seed, layout; dense: reference, seed, layout

Noise yardsticks Thread count Per-rank definitions Loss-free balancing Dense control Resume on new layout Low precision Model size Expert count Total

Runs

Where

9

CPU fp32

10 1 9

CPU fp32 CPU fp32 CPU fp32

7

CPU fp32

7 6

CPU fp32 CPU fp32

7 12

GPU fp16 GPU fp16

12

GPU fp16

80

Table 2: The 80 training runs. Each run differs from its reference in one thing only.

5

5.1

The layout changes the MoE model, but not its quality

Figure 1 follows the difference from the reference over training, and Table 3 gives the final numbers. Three things stand out. It starts tiny and grows fast. At step 8, a layout change has made the predictions differ by a KL of only about 4 × 10−9 , roughly 25 million times less than a new seed. By step 64 the difference has grown to about 2 × 10−2 , and by the end of training it is 0.280—0.80× the seed level. The top prediction agrees on only 69% of tokens (66% for a new seed), and 92% of routing decisions differ. Every small change looks the same. The eight layouts, the three 1-ULP nudges and the different CPU thread count follow one curve from step 32 onward and end between 0.277 and 0.283. Training in fp16 on the GPU instead of fp32 on the CPU also ends at 0.279. The particular layout does not matter: any change to the arithmetic is enough. Quality does not change. The final validation loss of the reference and the eight layouts ranges from 4.576 to 4.610, inside the range of the four new-seed runs (4.582–4.626; Appendix Table 6). The layout runs are also much closer to the reference than a new seed in weight space: the loss barrier between them is about 0.42 nats, against about 3.5 for a new seed. They are different models that share most of their structure. For comparison, a new data order makes the model more different than a new seed does at this budget (0.51 vs. 0.35), so the order of the data is a larger source of variation than the layout.

KL divergence from reference

MoE trained in fp32 on CPU

10−1

10−3

10−5 different random seed different data order

10−7

per-rank definition switched on (9 runs) different parallel layout (8 layouts) one weight nudged by 1 ULP (3 runs)

10−9

different CPU thread count

23

24

25

26

27

28

29

210

training step

Figure 1: Difference from the reference run during training (MoE, fp32 on CPU). Lines are medians; shaded areas cover all runs in a group. Layout changes, 1-ULP nudges and a thread-count change start about 10−9 and end at about 80% of the seed level. Switching on a per-machine rule starts about 400× higher but ends in the same place.

6

KL at step 8

KL at end

vs. own seed

Top-1 agree

Routing differs

Loss barrier

Loss change

MoE: different random seed MoE: different data order MoE: different parallel layout (8) MoE: one weight nudged by 1 ULP (3) MoE: different CPU thread count MoE: per-rank definition on (9) MoE, loss-free: new layout (2) MoE, loss-free: per-rank bias (2) MoE: fp16 GPU vs. fp32 CPU

1.0 × 10−1 1.0 × 10−3 4.1 × 10−9 3.6 × 10−9 6.6 × 10−9 1.7 × 10−6 2.2 × 10−8 8.8 × 10−6 2.2 × 10−7

0.350 0.512 0.280 0.283 0.277 0.282 0.285 0.285 0.279

1.00 1.46 0.80 0.81 0.79 0.80 0.84 0.83 0.80

0.664 0.590 0.689 0.685 0.689 0.687 0.687 0.683 0.691

– 96% 92% 93% 92% 95% 97% 96% 93%

3.538 – 0.417 0.432 – 0.486 – 0.580 –

+0.023 +0.025 +0.005 +0.016 +0.006 +0.011 +0.002 -0.001 -0.000

Dense: different random seed Dense: different parallel layout (2) Dense: one weight nudged by 1 ULP (2)

1.0 × 10−1 0 0

0.365 0.043 0.048

1.00 0.12 0.13

0.659 0.862 0.859

– – –

3.552 – 0.000

+0.011 -0.002 -0.002

Comparison with the reference run

Table 3: Difference from the reference at step 8 and at the end of training. “vs. own seed” divides the final KL by the KL between two seeds of the same model type. “Routing differs” is the share of (token, layer) decisions whose chosen expert set differs; it is not shown for seed runs, where expert numbers are not comparable. “Loss change” is the final validation loss minus the reference’s. Counts in brackets are the number of runs averaged.

7

5.2

Per-machine rules start large but end the same

Switching the load-balancing statistics or the capacity unit from global to per-machine is a real change to the training rule, not just to rounding. At step 8 it makes the model differ about 400× more than a pure layout change (1.7 × 10−6 vs. 4 × 10−9 ; red band in Figure 1). But by step 64 the two are indistinguishable, and the final difference is the same (0.282 vs. 0.280, compared against the same layout without the per-machine rule). The same holds for the loss-free bias controller fed by per-machine counts (0.285). None of these rules changed the final loss beyond seed variation at our scale. The one visible side effect is that per-machine capacity at 8 processes drops more tokens (1.8% vs. 1.2%). This does not mean the per-machine rules are harmless at large scale, where global load balancing is known to improve quality (Qiu et al., 2025); it means that, for the question “do I get the same model back”, a definitional change and a rounding change end up in the same place.

5.3

Why: near-ties in the router

To find what turns rounding differences into different models, we look at every routing decision at every saved step and ask which ones flipped between the reference and the changed run (Figure 2, Table 4). Flips happen at near-ties. In the first 16 training steps, the decisions that flip are almost exactly the ones with the smallest margin. If we pick a random flipped decision and a random unchanged one, the flipped one has the smaller margin 97–100% of the time (AUC 0.97–1.00 at step 8, for every layout, the 1-ULP nudge and the thread change). At step 16, the median margin of a flipped decision is 380–700× smaller than that of an unchanged one, and the 10% of decisions with the smallest margins flip 65–130× more often than the rest. After that, everything moves. Once enough decisions have flipped, the weights themselves start to differ, and flips spread to decisions with large margins too. From step 128 on, the AUC falls to about 0.6 and most decisions differ. Near-ties explain where the divergence starts; once started, it spreads on its own. Per-machine rules flip decisions directly. With a per-machine capacity or balancing rule, 2.5–3.4% of decisions already differ at step 16 (vs. 0.06–0.10% for a pure layout change), and the margin explains them less well (AUC about 0.95), because the rule itself chooses differently, not only at near-ties.

8

AUC (flipped had smaller margin)

(a) flips start at near-ties

(b) near-ties flip far more often

1.0

layout DP2 layout DP4

0.9

layout DP8/EP4

0.8

layout DP2 layout DP4 layout DP8/EP4 1-ULP nudge thread count fp16, 16 experts fp16, 8 experts fp16, 32 experts fp16, 64 experts

0.7 0.6 0.5 23

24

25

1-ULP nudge thread count fp16, 16 experts fp16, 8 experts fp16, 32 experts fp16, 64 experts

26

27

28

10−2

training step

10−1

100

choices flipped at step 16 (%, log scale) 10% of choices with smallest margin

all other choices

Figure 2: (a) Probability that a flipped routing decision had a smaller margin than an unchanged one (0.5 = no relation). Up to step 32, flips happen almost only at near-ties. (b) At step 16, the 10% of decisions with the smallest margins flip far more often than the rest (log scale).

AUC

at step 16

at

Choices

Median margin

Flip rate

Pair (reference vs. changed run)

step 8

flipped

AUC

flipped

kept

lowest 10%

other 90%

layout DP2 (fp32) layout DP4 (fp32) layout DP8/EP4 (fp32) 1-ULP nudge (fp32) thread count (fp32) per-rank capacity (fp32) per-rank balancing loss (fp32) loss-free, layout DP8/EP4 (fp32) 16 experts, DP2/EP2 (fp16) 8 experts, DP2/EP2 (fp16) 32 experts, DP2/EP2 (fp16) 64 experts, DP2/EP2 (fp16)

1.000 1.000 1.000 1.000 1.000 0.972 0.970 0.997 0.988 0.969 0.990 0.987

0.08% 0.08% 0.06% 0.06% 0.10% 3.36% 2.52% 0.22% 0.23% 0.44% 0.25% 0.32%

0.970 0.985 0.985 0.968 0.972 0.946 0.950 0.993 0.965 0.897 0.979 0.985

0.0002 0.0002 0.0001 0.0001 0.0002 0.0046 0.0037 0.0006 0.0003 0.0034 0.0003 0.0002

0.073 0.073 0.073 0.073 0.073 0.077 0.076 0.123 0.073 0.234 0.049 0.026

0.66% 0.73% 0.60% 0.56% 0.86% 26.70% 21.35% 2.18% 2.00% 3.31% 2.36% 3.02%

0.010% 0.006% 0.005% 0.008% 0.012% 0.764% 0.427% 0.005% 0.029% 0.123% 0.018% 0.018%

Table 4: Routing flips and margins at training steps 8 and 16 (0.7% and 1.3% of pretraining). “AUC” is the probability that a flipped decision had a smaller margin than an unchanged one. Margins are divided by the spread of each token’s router scores.

9

5.4

More experts: routing gets less reproducible, outputs do not drift further

Modern MoEs split the feed-forward capacity into many small experts (Dai et al., 2024; Yang et al., 2025). If near-ties drive the effect, finer experts should make it stronger. We test this on the GPU with 8, 16, 32 and 64 experts, choosing top-k and expert width so that total and active size stay exactly the same (Figure 3, Table 5). Near-ties roughly double each time the number of experts doubles: 2.7%, 6.6%, 11.9% and 23.3% of routing decisions at steps 8–32 for 8, 16, 32 and 64 experts. With more experts, more of them compete for each slot, and their scores sit closer together. Routing becomes less reproducible. At the end of training, a layout change leaves 68% of routing decisions different with 8 experts, 94% with 16, and essentially all of them with 32 or 64. But the outputs do not drift further. The final difference in predictions is 0.80, 0.81, 0.85 and 0.81 of the seed level for 8 to 64 experts, and every expert count reaches the 0.01 level at the same step (64). Many small experts appear to be largely interchangeable: a token sent to a different expert still gets a similar result. For practitioners this cuts both ways: fine-grained MoEs are no worse in output stability, but their expert assignments are almost impossible to reproduce across layouts, which matters for routing replay, expert pruning and interpretability.

23.3%

25 20 15

11.9%

10 5

6.6% 2.7%

(b) routing diverges more, outputs do not

% at the end of training

near-tie routing choices (%)

(a) more experts, more near-ties

0

100 80 60 40 routing choices that differ

20

output difference vs. new seed dense model, output difference

0 8

16

32

64

8

number of experts (total and active size fixed)

16

32

64

number of experts

Figure 3: Expert count with total and active size fixed (fp16, GPU). (a) Share of routing decisions that are near-ties, averaged over steps 8, 16 and 32. (b) After a layout change: share of routing decisions that differ at the end, and the output difference relative to a new seed. The dashed line is the dense model.

10

Near-tie

KL after a layout change (DP2/EP2)

vs. own

Routing

Experts

choices

step 32

step 64

end

seed

differs

8 (top-2, width 512) 16 (top-4, width 256) 32 (top-8, width 128) 64 (top-16, width 64) dense (width 1024)

2.7% 6.6% 11.9% 23.3% –

2.9 × 10−4 4.5 × 10−5 9.9 × 10−5 8.8 × 10−8 5.8 × 10−8

3.5 × 10−2 4.7 × 10−2 2.6 × 10−2 2.5 × 10−2 5.8 × 10−4

0.274 0.284 0.292 0.272 0.206

0.80 0.81 0.85 0.81 0.58

68% 94% 100% 100% –

Table 5: Effect of the number of experts (total and active size fixed; fp16 on one T4). The layout change is DP2/EP2 against the single-process reference. “Near-tie choices” is the mean over the first three saved steps.

11

5.5

MoE versus dense, and the role of precision

Figure 4 compares the MoE with a dense model of the same active size. In fp32, the dense model barely moves. Its predictions match the reference to within what we can measure (KL below 10−10 ) for the first 16 steps after a layout change, are still only 7 × 10−7 at step 128 (the MoE is at 7 × 10−2 ), and end at 0.043, or 0.12× its seed level. The straight line between the dense reference and its 1-ULP run has no loss bump at all: they are the same solution. The MoE, in contrast, ends at 0.80× with a loss barrier of 0.42. In fp16, the dense model moves too, but more slowly and less. With coarser arithmetic, the dense model’s layout difference ends at 0.58× its seed level, against 0.81× for the 16-expert MoE. The MoE still gets there much faster: every MoE crosses the 0.01 level at step 64, the dense model at step 128, and at step 64 the MoEs differ 40–80× more than the dense model. What this says. For an MoE, the size of the rounding noise hardly matters: a difference in the last bit is enough to flip a near-tie, and once one decision flips, the run diverges. The MoE ends at about 80% of the seed level whether we use fp32 or fp16. For a dense model, the outcome depends on how large the rounding noise is: small noise in fp32 stays small, larger noise in fp16 grows. In other words, the router makes MoE divergence close to all-or-nothing.

KL divergence from reference

(a) fp32 (CPU)

(b) fp16 (T4 GPU)

10−1 10−3 10−5 MoE: layout change

10−7

MoE: new seed dense: layout change

10−9

dense: new seed

23

24

25

26

27

28

29

210

training step

23

24

25

26

27

28

29

210

training step

Figure 4: Difference after a layout change, MoE vs. a dense model of the same active size. (a) fp32 on CPU: the dense model stays almost identical; the MoE diverges. (b) fp16 on GPU: both diverge, but the MoE does so earlier and further.

5.6

Resuming on a new layout is safe in the late phase, not the early phase

Elastic training resumes a job on whatever machines are available. To test how timing affects this, we resumed the reference pretraining run from two checkpoints on two other layouts, as well as with a 1-ULP nudge (Figure 5): • Early phase (10% of pretraining; step 128 of 1,221; 2.1M of 20M tokens). This occurs just 28 steps after the 100-step learning-rate warm-up ends, while validation loss is still dropping sharply (6.8 at step 100 to 6.1 at step 200).

12

• Late phase (84% of pretraining; step 1,024; 16.8M tokens). By this point, the learning rate has stayed constant for over 900 steps and validation loss has nearly flattened at 4.70 (step 1,000), close to the final 4.58. Switching at 84% leaves the final model almost unchanged (KL 0.007, about 2% of the seed level; 15% of routing decisions differ). Switching at 10% changes it much more (KL 0.12–0.13, about 35% of the seed level; 58–61% of routing decisions differ). In both cases the loss barrier is zero, so the resumed model stays in the same valley as the original, unlike a run whose layout differs from step 0 (barrier 0.42). And in both cases a layout switch behaves exactly like a 1-ULP nudge. This matches the observation that training is most sensitive in its initial phase (Kwok et al., 2025). For MoE pretraining, our two checkpoints show that this sensitive period still has a large effect 10% into the run and a small one at 84%. With only two switch points we cannot say exactly where in between the effect becomes small, and all our runs use a constant learning rate after warm-up, so we do not know how a switch during learning-rate decay, mid-training or post-training would behave.

KL divergence from reference

Resuming pretraining on a new layout

layout different from step 0 (median of 8) different random seed step 128 (10%): switch to DP8/EP4 step 128 (10%): switch to DP4 step 128 (10%): 1-ULP nudge step 1024 (84%): switch to DP8/EP4 step 1024 (84%): switch to DP4 step 1024 (84%): 1-ULP nudge

10−1

10−2

0

200

400

600

800

1000

1200

pretraining step (of 1,221; vertical lines = switch points)

Figure 5: Continuing the reference pretraining run on a different layout at step 128 (10% of pretraining) or step 1,024 (84%), compared with a 1-ULP nudge at the same point and with runs whose layout differs from step 0.

5.7

Model size

At three sizes (25M, 68M, 117M parameters, 16 experts, fp16), the final layout difference is 0.81, 0.88 and 0.86 of the seed level, and the share of near-tie decisions grows slightly with size (6.6%, 6.8%, 7.8%). The larger models diverge earlier: at step 32 their layout difference is about 100× that of the base model (Appendix Table 7 and Figure 6). Within this range, size does not change the overall picture.

13

6

What This Means in Practice

Do not expect the same MoE back from a different layout. Even with the same seed, data and batch, a new layout gives an MoE that is about 80% as different as a new seed. If you need the same model—to debug a loss spike, audit a result, or reproduce a release—keep the layout fixed and use bitwise-deterministic training (Aithal et al., 2026). Higher precision alone does not help an MoE much; it helps a dense model a lot. Compare methods on the same layout, with several seeds. A change of layout alone moves an MoE almost as much as a change of seed. A difference between two methods trained on different layouts that is smaller than the seed-to-seed spread can simply be the layout. During pretraining, avoid layout switches in the first part of the run. For elastic or preemptible pretraining jobs, a layout switch at 84% of pretraining changed the model little, while a switch at 10% (just after warm-up) changed it a lot. If the cluster size must change, doing it after the loss has flattened costs much less reproducibility than doing it right after warm-up. We did not test layout switches during learning-rate decay, mid-training or post-training. Watch the router, not only the loss. Loss did not reveal any of these differences. The share of near-tie decisions and the share of routing decisions that change are cheap to log and show how reproducible a run is. Expect them to get worse with more, smaller experts. Plan for routing to be the fragile part. For work that depends on specific experts—routing replay in reinforcement learning (Ma et al., 2025), expert pruning, or interpretability—fine-grained MoEs need either deterministic, fixed-layout training or a recorded routing table, because their expert assignments do not survive a layout change.

7

Limitations

Scale. Our models have 25M–117M parameters and are trained on 20M tokens, far below production budgets; the base model sees only a small fraction of a compute-optimal token budget. The exact ratios (for example, 0.8× the seed level) will change with longer training. Hardware. CPU runs use one node with up to 8 processes; GPU runs share a single T4. Communication uses gloo, not NCCL, and we do not test tensor or pipeline parallelism. Production training in bf16 or FP8 adds more rounding noise per step; based on our fp16 results we expect the MoE effect to be at least as strong, but we have not measured it. Repeats. The expert-count sweep and model-size study use one seed and one changed layout per setting. fp16 1-ULP control. On the GPU, a 1-ULP change to an fp32 master weight was often lost when the weight was cast to fp16, giving identical runs; we therefore use only seed and layout comparisons for the GPU runs. Quality. We find no quality cost from per-machine rules at our scale, but large-scale results show that per-micro-batch load balancing can hurt quality (Qiu et al., 2025); our result should not be read as saying these rules are harmless in general. Training phase. Every run is pretraining from scratch with a constant learning rate after warm-up, and the resume experiment uses only two switch points (10% and 84% of pretraining). We make no claim about switching layouts during learning-rate decay or annealing, mid-training, or post-training such as fine-tuning or reinforcement learning. Capacity. We use a capacity factor of 1.25 with token dropping during training; many production MoEs are dropless.

14

8

Conclusion

We asked whether the parallelism layout changes a trained Mixture-of-Experts model. It does not change how good the model is, but it does change which model you get: about 80% as much as a new random seed, with most routing decisions different. The cause is the router: decisions where two experts are nearly tied flip under the smallest rounding difference, and the runs then drift apart on their own. Finer experts create more near-ties and make routing much less reproducible, yet the outputs drift no further. When a pretraining run is resumed on a new layout, the point at which it switches matters: a switch 10% into pretraining changes the final model far more than a switch at 84%. A dense model of the same active size is far more stable and depends on how much numerical noise there is, while the MoE diverges at any precision. For anyone who needs the same MoE twice, the layout belongs in the experiment record, and fixed-layout deterministic training is the only reliable way to get it. Code and data. The training code, the correctness checks, all run logs and the measured data behind every table and figure will be released with this paper.

References Ashwath Aithal, Eric Harper, Santosh Bhavani, Yan Xu, Zhiyu Li, et al. Scale bitwise-deterministic pretraining with NVIDIA Megatron Core. NVIDIA Technical Blog, https://developer. nvidia.com/blog/scale- bitwise- deterministic- pretraining- with- nvidia- megatroncore/, 2026. Damai Dai, Chengqi Deng, Chenggang Zhao, et al. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (ICML), 2020. arXiv:1912.05671. Horace He and Thinking Machines Lab. Defeating nondeterminism in LLM inference. Thinking Machines Lab blog, https://thinkingmachines.ai/blog/defeating-nondeterminism-inllm-inference/, 2025. Peng Jin, Zihan Qiu, Zekun Wang, Bo Zheng, Yang Xu, Tian Xie, Xiao Li, Huaqing Zhang, Haoran Lian, Rui Men, and Dayiheng Liu. ID balancing: Stable training of extremely sparse MoE via PID-based load control. arXiv preprint arXiv:2609.39137, 2026. Devin Kwok, Gül Sena Altıntaş, Colin Raffel, and David Rolnick. The butterfly effect: Neural network training trajectories are highly sensitive to initial conditions. In International Conference on Machine Learning (ICML), 2025. arXiv:2506.13234. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR), 2021. arXiv:2006.16668. 15

Mingzhen Li, Wencong Xiao, Hailong Yang, Biao Sun, Hanyu Zhao, Shiru Ren, Zhongzhi Luan, Xianyan Jia, Yi Liu, Yong Li, Wei Lin, and Depei Qian. EasyScale: Elastic training with consistent accuracy and improved utilization on GPUs. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2023. arXiv:2208.14228. Wenhan Ma, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Zhifang Sui, and Fuli Luo. Stabilizing MoE reinforcement learning by aligning training and inference routers. arXiv preprint arXiv:2510.11370, 2025. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. arXiv:2406.17557. Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Demons in the detail: On implementing load balancing loss for training specialized mixture-of-expert models. arXiv preprint arXiv:2501.11873, 2025. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2017. arXiv:1701.06538. Cecilia Summers and Michael J. Dinneen. Nondeterminism and instability in neural network optimization. In International Conference on Machine Learning (ICML), 2021. arXiv:2103.04514. Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408.15664, 2024. Zijie Yan, Hongxiao Bai, Xin Yao, et al. Scalable training of mixture-of-experts models with Megatron Core. arXiv preprint arXiv:2603.07685, 2026. An Yang et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Ziyang Zhang, Xinheng Ding, Jiayi Yuan, Rixin Liu, Huizi Mao, Jiarong Xing, and Zirui Liu. Deterministic inference across tensor parallel sizes that eliminates training-inference mismatch. In International Conference on Machine Learning (ICML), 2026. arXiv:2511.17826. Donglin Zhuang, Xingyao Zhang, Shuaiwen Leon Song, and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. arXiv preprint arXiv:2106.11872, 2021.

A

Correctness Checks and Implementation

Before training, each job ran a self-test with four processes on a tiny model. (1) For EP sizes 1, 2 and 4, with both balancing methods, the loss and the combined gradients of the multi-process run matched a single-process run (relative error below 10−4 in fp32). (2) Three optimizer steps run twice from the same start gave bit-identical weights. (3) A checkpoint saved under one layout and loaded under another reproduced every weight and optimizer state exactly. (4) Every per-machine variant ran without errors. Both the CPU and the GPU self-tests passed. Expert-parallel communication is built only from sum-reductions. To gather rows from other processes, each process writes its rows into a zero-filled buffer and the buffers are summed; since 16

adding zero is exact, gathering itself adds no rounding. All per-machine rules have a “global” version that gives the same result in every layout, so the numerical-only runs change nothing but the order of additions. Checkpoints are written every 15 minutes in a layout-independent format, which is what makes the resume experiments possible. Data order is fixed per step: step s always uses the same 32 sequences, and each process takes a contiguous slice of them.

B

Additional Results Final validation loss

Run group MoE reference + 8 layouts MoE new seeds MoE new data orders MoE 1-ULP nudges MoE per-rank definitions MoE loss-free balancing Dense control Resume on new layout fp16 GPU, base size fp16 GPU, size m fp16 GPU, size l

Expert

Tokens

Runs

mean

range

imbalance

dropped

9 4 3 3 9 7 7 6 8 6 5

4.587 4.599 4.582 4.595 4.593 4.582 4.654 4.582 4.590 4.508 4.454

4.576–4.610 4.582–4.626 4.571–4.604 4.591–4.602 4.585–4.610 4.571–4.596 4.648–4.674 4.577–4.587 4.576–4.610 4.486–4.527 4.444–4.458

0.28 0.29 0.28 0.28 0.28 0.52 – 0.20 0.28 0.30 0.32

1.21% 1.28% 1.29% 1.20% 1.36% 5.37% – 0.14% 1.21% 1.49% 1.72%

Table 6: Final validation loss and routing health by run group. “Expert imbalance” is the mean over layers of the most-used expert’s share divided by the average share, minus one. “Tokens dropped” is the share of routing assignments dropped by the capacity limit. Resumed runs report statistics only after the switch point, where imbalance is lower.

Near-tie Model size (16 experts, fp16) base (∼25M) m (∼68M) l (∼117M)

KL after layout change

vs. own

Routing

choices

step 32

end

seed

differs

6.6% 6.8% 7.8%

4.5 × 10−5 5.5 × 10−3 5.7 × 10−3

0.284 0.285 0.281

0.81 0.88 0.86

94% 98% 97%

Table 7: Effect of model size (16 experts, fp16). The layout change is DP2/EP2.

17

10−1 10−3 10−5 10−7

base (~25M) m (~68M)

10−9

l (~117M)

24

26

28

(b) near-ties by model size near-tie routing choices (%)

KL divergence from reference

(a) layout difference by model size (fp16)

210

base

10

m l

8

6

4 24

training step

26

28

210

training step

Figure 6: (a) Difference after a layout change for three model sizes. (b) Share of near-tie routing decisions during training.

18

C

Reproducibility

All runs used the settings in Table 1. CPU runs took about 2.2 hours each on one 16-vCPU node; a single-process base-size GPU run took about 13 minutes on one T4. Every number in this paper is read from the CSV files in data/, and the scripts figures/generate_figures.py and figures/make_tables.py regenerate every figure and table from them.

19

Related documents

• Joint Effects of GPU Server Topology, Parallelism, and Congestion Control on MoE Inference: A Controlled Simulation Study#1142136
• Joint Effects of GPU Server Topology, Parallelism, and Congestion Control on MoE Inference: A Controlled Simulation Study#1122251
• Rethinking MoE Load Balancing as Feedback Control#1625403
• A Per-Step Load-Balancing Engine for Expert-Parallel MoE Serving#1635501
• A Per-Step Load-Balancing Engine for Expert-Parallel MoE Serving#1605210