FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs Lorenzo Sani1,2 , Zeyu Cao1 , Meghdad Kurmanji1 , Alex Iacob1,2 , Andrej Jovanovic1 , Yan Gao1,2 , Wanru Zhao1 , Nicholas D. Lane1,2 2
Department of Computer Science and Technology, University of Cambridge, Cambridge, UK
Flower Labs, UK
Abstract. Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators. While increasing model and dataset scale remains the dominant driver of performance, Mixture-of-Experts (MoEs) architectures have recently achieved state-of-the-art results by decoupling parameter count from computational cost. This efficiency enables training massive models on constrained compute budgets, yet it typically requires the high-speed interconnects of a single datacenter. To overcome these physical limits, recent approaches such as DiLoCo and Photon use low-communication data-parallel methods to enable scaling across geographically distributed, weakly connected data centers. However, these methods suffer from a fundamental inefficiency: they require full model replicas at every site, which imposes prohibitive memory constraints and communication overheads. In this work, we introduce FoMoE, a system that breaks the full-replica paradigm by partitioning expert layers across workers. We demonstrate that FoMoE: (I) reduces communication costs by up to 1.42× over efficient baselines and 45.44× over DDP via partial expert replication in the studied regimes; (II) achieves empirical throughput speedups of up to 1.4× through a novel “skip-token” mechanism; and (III) shows stable routing in the trained proxy regimes and projects the communication/memory benefits to 100B-scale configurations through system modelling.
To eliminate the physical constraints of single datacenters – and thus aggregate more compute – frontier labs have shown interest in training across geographically distributed, Preprint.
ng
S
ge
id
C am b r
La b
State-of-the-art Large Language Models (LLMs) continue to grow in scale, driven by the relationship between parameter count and model quality (Hoffmann et al., 2022). However, training frontier models that are fully-connected (dense) neural networks has become prohibitively expensive, straining the compute, memory, and bandwidth limits of even the largest specialized hyperclusters. To address this computational bottleneck, the field has increasingly adopted the Mixture-of-Experts (MoEs) architecture (Shazeer et al., 2017; Fedus et al., 2022). By activating only a sparse subset of parameters per token, models like DeepSeek-V2/V3 (DeepSeek-AI et al., 2024; DeepSeekAI, 2024) decouple computational cost from model capacity. This modularity enables trillion-parameter models to be trained with the FLOP budget allocated to training smaller dense networks. However, while MoE architectures can reduce compute requirements, they still typically require the active compute and routing fabric to be colocated in large datacenters with high-bandwidth interconnects between compute nodes.
Learni
ms
I NTRODUCTION
in e
te
1
ch
ys
Ma
ls
k
Correspondence: Lorenzo Sani <[email protected]> Keywords: machine learning systems, mixture of experts, federated learning, large language models
m
arXiv:2606.19025v1 [cs.LG] 17 Jun 2026
1
ys
.cs t
.c am.
ac
.u
weakly connected sites, effectively simulating a single supercomputer (Comanici et al., 2025; Guthrie, 2025; Sevilla & Troynikov, 2025). For example, Google’s Gemini LLM was trained using connected data centers (Comanici et al., 2025). However, current understanding of this training paradigm remains in its infancy. In search of more efficiency in such cross-datacenter training, algorithms like DiLoCo or Photon have emerged that reduce bandwidth overhead by limiting the synchronization frequency during training (Douillard et al., 2023; Sani et al., 2025). However, a critical inefficiency remains: these methods typically treat the model as a dense, monolithic replica and do not reduce payload size. This approach forces every participating site to store the entire model state and exchange full parameter payloads, which often amount to hundreds of gigabytes, at every synchronization step. Consequently, cross-datacenter training faces two prohibitive barriers: (1) Bandwidth: Full-payload transfers create massive latency spikes on WAN links, rendering training extremely challenging for truly large models, and (2) Memory: The maximum model size is capped by the storage capacity of the most memory-constrained site. This creates a mismatch: while modern MoE models possess sparse data paths, their distributed training infrastructure remains communicationdense and memory-inefficient. We argue that leveraging communication sparsity complements the computational
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs Data Center Transformer (DC) Block
Expert Selection
Expert Available
Ghost Experts
❷
Our current evaluation deliberately studies a controlled setting with homogeneous workers, i.i.d. data partitions, and shaped WAN bandwidth and latency. This isolates the effect of expert partitioning and payload reduction from orthogonal production concerns such as asymmetric links, jitter, stragglers, and stale synchronization. We return to these concerns in Section 7 rather than claiming production-wide robustness from a static-link emulator.
❸
FoMoE enables the training of massive models across multiple data centers without sacrificing joint-training performance. Our contributions are as follows:
Expert 1 ❶
Router MHA
Expert 2
Router MHA
❶
Expert 3 ❷
Router MHA
Expert 4
Router MHA
Expert 5 ❸
Router MHA
DiLoCo/Photon (Full Replica)
Expert 6 FFN Layer Expert View
Router MHA
FoMoE (Partial Replica)
Figure 1: FoMoE breaking the “full-replica” barrier for WAN-connected MoE training. Methods such as DiLoCo and Photon maintain a full MoE replica at each site and communicate this object at round boundaries. FoMoE instead leverages MoE sparsity so each worker trains and communicates only the experts assigned to it. This design targets cross-site WAN settings where activation dispatch is too expensive, rather than replacing expert parallelism inside a single high-bandwidth cluster.
efficiency of architectures like DeepSeek, enabling the pooling of large-scale distributed resources and further closing the gap between open and proprietary models. In this work, we present FoMoE, a cross-site training system that breaks the “full-replica” barrier by co-designing the training strategy with the model’s structural sparsity. Rather than maintaining identical replicas at every site, FoMoE partitions each expert layer and assigns subsets of expert modules to different datacenters. As a result, FoMoE recovers the bandwidth-efficiency properties of previous methods (Douillard et al., 2023; Sani et al., 2025) by reducing synchronization frequency, but also reduces the payload size sent during each synchronization. Relative to data-center approaches such as expert parallelism (EP), FoMoE changes the communication boundary. EP shards experts within a high-bandwidth cluster and sends token activations to remote experts during the forward and backward passes (Rajbhandari et al., 2022; Hwang et al., 2023; Gale et al., 2023; DeepSeek-AI et al., 2024; DeepSeek-AI, 2024). FoMoE instead assumes that the cross-site link is the scarce resource: workers never exchange token activations over the WAN, and only selected model state is synchronized at round boundaries. Thus, FoMoE complements EP inside each site while addressing a different bottleneck from full-replica low-communication training.
1. The FoMoE System. We introduce a framework for cross-site MoE training that supports partial expert replication. We define the design space of partitioning (balancing memory vs. bandwidth) and placement (mapping experts to workers), exploring both fixed and random strategies to trade off expert specialization against generalization. 2. Partitioned FFN Optimization. We demonstrate that partial expert replication linearly reduces communication volume, outperforming communication-efficient baselines by up to 1.42× and DDP by up to 45.44× in communication costs while maintaining perplexity in our controlled static-link evaluation. This establishes a new Pareto frontier, enabling ML performance at lower system costs in the studied regimes. 3. “Skip-Token” Throughput Gains. We introduce a “skip-token” mechanism that bypasses computation for non-local experts (“ghost experts”). We show that this improves throughput linearly in theory and achieves empirical speedups up to 1.4×. 4. Scaling and Stability. We separate the evidence by scale: trained proxy configurations provide empirical routing and convergence evidence, while the larger configurations in Table 1 are analysed through communication, memory, FLOP, and wall-clock system models. Our results show that FoMoE maintains high routing entropy and avoids expert collapse in the trained regimes, and that the same partial-replication mechanism projects favourable payload and memory scaling under the modelled large-scale configurations. FoMoE bridges the gap between high-bandwidth, tightlycoupled MoE training and low-bandwidth, loosely-coupled distributed paradigms. By reducing the payload size per synchronization, it complements existing low-frequency methods (Douillard et al., 2023; Sani et al., 2025), opening the path to training next-generation LLMs on globally sourced compute infrastructure.
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
2
BACKGROUND
Large-scale model training. Distributing computation is essential for large-scale model training. Standard data parallelism replicates the full model and synchronizes gradients per step (Li et al., 2020a), often saturating interconnects (Narayanan et al., 2019). For memory constraints, model parallelism partitions the network: pipeline parallelism divides layers into sequential stages (Huang et al., 2019; Narayanan et al., 2019), while tensor parallelism splits layers across devices (Shoeybi et al., 2019). The Zero Redundancy Optimizer (ZeRO) eliminates memory redundancy by sharding parameters and optimizer states (Rajbhandari et al., 2020). While these methods assume high-bandwidth fabrics, Varuna adapts pipeline parallelism to commodity cloud VMs with limited WAN bandwidth (Athlur et al., 2022). MoE for scalable LLMs. Mixture-of-Experts (MoE) architectures decouple parameter count from compute cost by activating sparse expert MLPs per token (Shazeer et al., 2017). Implementations like GShard (Top-2) and Switch Transformers (Top-1) achieve trillion-parameter capacities efficiently (Lepikhin et al., 2021; Fedus et al., 2022). Recent advances reinforce this efficiency (GLaM (Du et al., 2022)) and optimize architectural design (DeepSeek-V2/V3 (DeepSeek-AI et al., 2024; DeepSeek-AI, 2024)). Theoretical works unify MoE perspectives via σ-MoE (Csordás et al., 2023) and establish fine-grained scaling laws (Ludziejewski et al., 2024). Alternative routing strategies, including BASE layers (Lewis et al., 2021), Expert-Choice (Zhou et al., 2022), and Hash Layers (Roller et al., 2021), address load balancing, while V-MoE addresses vision (Riquelme et al., 2021). Expert pruning and skipping methods remove or bypass low-utility experts for deployment efficiency (Lu et al., 2024); FoMoE uses a different skip-token semantics during training, where missing remote experts are deliberately not executed across the WAN. Systems support for MoE and expert parallelism. Commodity MoE training leverages specialized runtimes (e.g., DeepSpeed-MoE, Tutel, MegaBlocks, FasterMoE, SmartMoE) to optimize All-to-All communication, memory management, and kernel execution (Rajbhandari et al., 2022; Hwang et al., 2023; Gale et al., 2023; He et al., 2022; Zhai et al., 2023). These systems typically rely on expert parallelism (EP): experts are sharded across devices, while token activations are dispatched to the owning experts over a fast intra-DC fabric. EP can reduce per-device memory and enable large MoEs within a cluster, but it does not remove the need for high-bandwidth activation exchange during each forward and backward pass. DeepSeek-style node-limited routing narrows this dispatch scope within a cluster (DeepSeek-AI, 2024); FoMoE instead treats the WAN as a hard boundary, avoids cross-site activation dispatch entirely, and communicates only selected
model state at synchronization boundaries. Thus, our design is orthogonal to intra-DC EP and targets the payload and memory overheads that remain in full-replica cross-site training. Low-bandwidth distributed training. Training across weakly connected sites builds upon distributed optimization and local-update foundations (McMahan et al., 2017; Reddi et al., 2021; Li et al., 2020b; Stich, 2019). Standard data parallelism synchronizes a full model every step; DiLoCo (Douillard et al., 2023) and Photon (Sani et al., 2025) instead perform multiple local steps before each synchronization, reducing the frequency-dependent bandwidth cost by roughly the local-step interval. Theoretically, (Cheng & Glasgow, 2025) proves convergence for adaptive optimizers in this regime, provided optimizer states are synchronized. Further communication reductions are achieved by decoupling model components (DEPT (Iacob et al., 2025a)), desynchronizing optimizer states (DES-LOC (Iacob et al., 2025b)), co-designing modular paths (DiPaCo (Douillard et al., 2024)), or combining local updates with compression and sparse peer communication (SparseLoCo, DiLoCoX, GASLoC) (Sarfi et al., 2025; Qi et al., 2025; Cagnasso et al., 2026). Related systems work studies geodistributed and weakly connected training through workloadaware WAN scheduling, pipeline parallelism, and volunteerdevice MoEs, including Atlas/BubbleTea, SWARM Parallelism, and Learning@home/DMoE (Palak et al., 2024; Ryabinin et al., 2023; Ryabinin & Gusev, 2020). However, a direct mapping of these methods to MoE pre-training still materializes the full dense backbone and all experts at each site, or targets different axes such as pipeline scheduling, peer topology, or compression. FoMoE addresses the complementary dimension of which MoE experts must be present and synchronized at each site, reducing per-round payload size and steady-state memory rather than only synchronization frequency.
3
F O M O E: T HE F EDERATED M O E S YSTEM
Although current low-communication pre-training methods reduce the synchronization frequency, and thus the cumulative communication cost, they still require the transmission of full parameter payloads during updates (Douillard et al., 2023; Sani et al., 2025). The straightforward adoption of MoE architectures in this setting achieves a FLOP reduction for each worker by leveraging the structural sparsity of the model – only k ≪ E of the total of experts (E) are activated per token (Fedus et al., 2022; DeepSeekAI et al., 2024; Ludziejewski et al., 2024; Shazeer et al., 2017). However, when applied directly to methods such as Photon or DiLoCo, the communication bottleneck remains unresolved. By design, these methods require that the full payload of the parameter is still communicated at every
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Architecture and design
3.2
Model partitioning design
Although our system can flexibly accommodate arbitrary MoE architectures (DeepSeek-AI et al., 2024; Team et al.,
80
Sparse Dense Expert params
104
Expert Parameters (Millions)
100
103
60
40
102
20
101
Model Configuration (d_model × n_layers)
8 ×1 2 40 96
×6 4 20 48
×3 2
0
4
Architecture and distributed setting. We target crossdatacenter training environments characterized by a distinct bandwidth hierarchy typical of real-world deployments: high intra-datacenter bandwidth Bintra and severely constrained cross-datacenter bandwidth Bcross (Palak et al., 2024). We explicitly focus on the regime where Bintra ≫ Bcross ; the utility of local training strategies decreases when bandwidths are comparable (Bintra ≈ Bcross ). Formally, we consider a set W of homogeneous data-parallel workers M = |W |, where each worker functions as a datacenterclass node with accelerators interconnected at Bintra (e.g. ≈ 200.00 Gbps). Each worker has an i.i.d. data partition and connects to peers through WAN links with effective bandwidth Bcross (e.g., ≈ 1.00 Gbps). In this setting, standard computation-communication overlap offers minimal benefit as transfer times dominate the latency. To mitigate this, we perform K local optimization steps between fully synchronous rounds (Douillard et al., 2023; Sani et al., 2025). We employ a central coordinator to manage initialization, broadcasting, and aggregation, restricting our study to synchronous training. We leave asynchronous extensions to future work, due to the complexity of managing stale updates (Douillard et al., 2025), and note these methods are fully compatible and orthogonal to our work.
To illustrate this breakdown, we consider a typical Transformer configuration. The token embedding matrix comprises dmodel × |V| parameters where V is the vocabulary. Each attention layer contains four weight matrices totaling 4d2model parameters (assuming standard head dimensions). In contrast, an MoE layer adds a routing network with dmodel × Ne parameters (where Ne is the total expert count) and the experts’ FFNs. For a dense expansion ratio e (e.g., e = 4), we allocate a total FFN hidden dimension of e · dmodel per layer, divided among Ne experts (each with hidden size dff,expert = e dNmodel ). Consequently, the combined e expert parameter count is ≈ e × d2model , comparable to a dense FFN.
25 6×
Our system architecture is optimized for communicationconstrained federated environments. By breaking the “fullreplica” barrier, we decouple the model size from synchronization costs. We detail the setting below and in Section 3.2.
Ratio (%)
3.1
1. Dense components (θdense ): Fully replicated parameters including attention weights, layer norms, shared experts, and input/output embeddings (tied by default). 2. Sparse components (θsparse ): Potentially partitioned parameters including expert weights and routing networks. We detail the worker partitioning and assignment of sparse components in Sections 3.3.1 and 3.3.2.
10 24
We now present the FoMoE system and its design space, covering: (a) model partitioning, which determines the optimal degree of expert replication; (b) expert placement, which orchestrates the distribution of experts across workers; and (c) machine learning optimization, which ensures stability and performance parity with high-communication baselines. Figure 1 shows a simplified system view of FoMoE.
16
To alleviate this issue, we propose a federated framework MoE tailored to bandwidth-constrained wide-area network environments (WAN) that leverage the sparsity present in MoE architectures to reduce both computational and communication costs by only training, and thus communicating, a subset of experts in each worker.
2025), we fix the architecture design in this work. Specifically, our system trains decoder-only Transformer models (Vaswani et al., 2017) in which we replace the standard feedforward network (FFN) with a σ-MoE-FFN modules (Csordás et al., 2023; Ludziejewski et al., 2024) whose expert and router parameters are, potentially, only partially replicated across subsets of workers (we provide full details of the architecture design in Section B). Our architecture choice also supports the capture of shared experts’ common knowledge (DeepSeek-AI et al., 2024). Conceptually, FoMoE categorizes the model parameters θ into the following.
51 2×
synchronization step, despite the inherent sparsity observed in these MoE architectures.
Figure 2: Parameter distribution trends. Scaling model width (dmodel ) dramatically increases the proportion of sparse parameters (θsparse ), motivating our partitioning strategy. As model scale increases, the ratio θsparse /θ shifts significantly. While scaling depth adds dense parameters, scaling
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
width (increasing dmodel ) adds experts. Adopting a standard width-to-depth ratio of dmodel : L ≈ 32 (Dey et al., 2025), θsparse grows from ≈ 20% of total parameters at the 150M scale (dmodel = 512) to > 60% at the 26B scale (dmodel = 4096), as shown in Figure 2. This relationship shows that the efficient synchronization of θsparse is the primary lever for enhancing communication efficiency as model size increases. This scaling design is not only well-established and justified in the literature (Csordás et al., 2023; Malasnicki et al., 2025; Ludziejewski et al., 2024), but also enables our work to study the broader design space of expert placement options given the number of expert modules. 3.3
Partitioning and placement strategies
FoMoE enables the distribution of sparse parameters controlled via two coupled design decisions: partitioning, which determines expert replication to balance memory and bandwidth; and placement, which maps experts to workers. These mechanisms jointly control communication costs and effective model capacity. We define participation in Section 3.3.1 and the placement algorithms in Section 3.3.2. 3.3.1
Experts partitioning and scalability
To analyze the impact of partitioning at scale, consider a layer ℓ with Ne,ℓ experts distributed across M workers. We introduce the overlapping factor Oe ∈ {1, . . . , M }, defined as the number of distinct workers hosting any specific expert. This parameter controls the design spectrum from completely disjoint partitions (Oe = 1) to full replication (Oe = M ). Consequently, each worker hosts a reduced e local subset of Nle,ℓ = O M Ne,ℓ experts. The design of the overlapping factor ensures that each expert is trained by at least one worker, where multiple experts are permitted to share the same experts. During training, local routing is bounded by min(k, Nle,ℓ ) experts (where k = 2 by default). Partitioning (Oe < M ) yields advantages over full-replicas: 1. Communication Reduction: Synchronization volume scales with the fraction of globally shared params e Oe /M , saving the remaining 1 − O M from cross-site transfer. 2. Compute Efficiency: Computation is reduced in two ways. First, the router scores only Nle,ℓ experts, reducing router FLOPs by ≈ Oe /M . Second, if Nle,ℓ < k, the system activates fewer experts per token (e.g., if Oe = 1, which implies that Nle,ℓ = 1; this activates one expert instead of two), reducing FFN FLOPs. Otherwise, the per-token compute matches standard MoE (Nle,ℓ ≥ k). 3. Memory Efficiency: The per-worker memory foot-
Algorithm 1 Expert Placement & Migration Strategy Require: M (workers), Ne (total experts), Oe (overlap) (t−1) Require: T (shuffle period), t (round), Sm (prev. set) Require: Mode (Fixed or Random) 1: (t) (t−1) 2: Sm ← Sm // Default: retain set 3: Phase 1: Placement Logic 4: if t = 0 or (Mode = Rand and t (mod T ) = 0) then 5: P ← GenPlacement(Ne , Oe , M ) (t) 6: Sm ← GetLocalSubset(P, m) e 7: ▷ Assigns Nle = O M Ne experts to worker m 8: end if 9: Phase 2: Migration & State Mgmt. (t) (t−1) 10: ∆in ← Sm \ Sm // Identify new experts 11: if ∆in ̸= ∅ then 12: Broadcast(∆in ) e 13: ▷ Fetch missing weights (prob. pmiss ≈ 1 − O M ) 14: else 15: ▷ Maintain local momentum (no migration) 16: end if (t) 17: return Sm
print for expert weights decreases by a factor of M/Oe . This reduction is essential for scaling to model sizes (Ne ) and cluster sizes (M ) where naive full replication would be infeasible due to bandwidth and memory constraints. 3.3.2
Expert placement modes
Once the expert partitioning configuration (defined by Oe and Ne,ℓ ) is set, FoMoE enables flexible expert-to-worker assignment. We define two modes, detailed in Algorithm 1: Fixed placement. Upon initialization of the M workers, the central server deterministically maps each expert in layer ℓ to Oe specific workers. Consequently, each worker is assigned a static set of Nle,ℓ experts. These assignments remain invariant throughout training, with each worker responsible solely for updating its local experts. In the limiting case where Oe = 1, this scheme reduces to disjoint, fully local experts; workers collaborate exclusively on the dense parameters θdense . This effectively specializes each site to a specific model subset, analogous to decoupled training methods like DEPT (Iacob et al., 2025a). While DeepSeek’s Expert Parallelism treats the cluster as a unified memory pool by dynamically sharding experts to maximize capacity (DeepSeek-AI et al., 2024), FoMoE’s fixed placement treats the WAN as a hard constraint, strictly pinning experts to specific workers to enforce data locality and eliminate weight synchronization during local steps. Random placement. To improve expert generalization, this mode reassigns experts to workers at a cadence of T synchronization rounds. The server randomly redistributes experts subject to the constraint that every expert must reside
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs Architecture Connection
Residual Stream
LN
LN
LN
Top-K Selection
σ
LN
LN
LN
LN
σ
Router
Token Dataflow σ
Top-K Selection Router
σ
FFN Experts Multi-Head Attention
LN
σ
σ
σ
MoE Expansion Factor
FFN Layer
Missing Expert Skipped Token Shared Replica
Local Replica
Transformer Block (Partial Replica)
Transformer Block (Full Replica)
Figure 3: Comparison between a transformer MoE block as a full versus partial replica. In the partial-replica form, we observe that the same experts are activated per token as in the full-replica case. However, in this case, we incur FLOPs, communication cost, and memory savings as the expert is missing and thus skipped in the computational graph. on exactly Oe workers, and every worker must host Nle,ℓ distinct experts. Under this scheme, the specific expert set resident on a worker evolves over time. A small reshuffling period ensures experts are trained across worker data partitions, potentially enhancing generalization. However, this introduces trade-offs: it incurs network overhead for migrating model states and may affect local optimizer momentum (which we discuss in Section 3.4). In practice, the reshuffling period is tuned to balance data mixing against stability and costs. Equation (8) demonstrates that while reshuffling can, in the worst case, necessitate migrating every expert, this overhead decreases linearly as Oe → M and as the reshuffling interval increases relative to the communication period. System context and routing logic. We emphasize that in our federated setting, token routing is strictly local: tokens processed at a site are dispatched exclusively to experts hosted at that site, with no inter-site exchange of activations. This design maximizes locality, avoiding high-latency WAN communication during the forward and backward passes. This distinguishes FoMoE from systems like DeepSpeed-MoE, Tutel, MegaBlocks, FasterMoE, SmartMoE, which optimize execution within high-bandwidth clusters via all-to-all communication (Rajbhandari et al., 2022; Hwang et al., 2023; He et al., 2022; Zhai et al., 2023). Our placement strategies thus operate at a distinct granularity from typical intra-layer load balancing. While we assume standard mechanisms (e.g., BASE layers (Lewis et al., 2021), Expert-Choice (Zhou et al., 2022), or capacity scaling (Fedus et al., 2022)) are employed to balance loads among available local experts, our work addresses the WANconstrained challenge of determining which experts are available at each site. Placement trade-off. Random expert placement may outperform fixed placement when data or hardware heterogeneity makes additional expert mixing valuable. First, while
a fixed topology restricts a worker’s router to a static support set of Nle experts, limiting the routing combinations to Nkle , random placement reshuffles assignments every synchronization interval. Over R rounds, a worker explores Ne up to min(R, N ) distinct support sets. Second, randomle ization can act as a regularizer when the i.i.d. assumption fails, since it decouples experts from specific data shards and exposes them to a broader range of data distributions. In the homogeneous i.i.d. setting evaluated in Section 6.2, however, the migration and optimizer-state costs dominate this hypothesized mixing benefit. 3.3.3
“Ghost Experts” and “Skip-Token”
For most of our work, we partition each router along with the experts so as not to route individual tokens to experts that are not available to the local worker. However, we investigate the opportunity to go beyond such biased routing and further leverage experts’ sparsity and partitioning to save computation. We propose an alternative partitioning, as seen in Figure 3, that does not partition the routing matrices, allowing each router to assign individual tokens to experts who are not locally available to the workers. The cost of not partitioning the routing matrices and sending them in full, so as if they are part of the dense components θdense , is negligible compared to that of the expert modules. We refer to those experts that can receive token assignments but are not available as ghost experts. The token assignments to the “ghost experts” offer an opportunity to reduce FLOPs further, since we cannot take the following computational path due to missing expert weights. When the router’s assignments are perfectly balanced, we expect to save FLOPs in proportion to the ratio of local experts to total experts. Additionally, this does not drastically affect machine learning performance: the router aims to make the best routing decision given the current state of the training and would prefer not to assign a particular token to an expert that is not best suited for it. In contrast to DeepSeek’s dispatch mecha-
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
nism, which relies on high-bandwidth interconnects to route tokens to remote experts for full computation (DeepSeek-AI et al., 2024), FoMoE uses its “ghost expert” strategy that dynamically skips computation for remote experts, trading a marginal reduction in active parameter count for the complete elimination of cross-node activation transfer. ❶ Compute Ring ❶❷❸ ❹
❶
❶
Router
Router
Router
MHA
MHA
MHA
❷
Available Experts Ghost Experts
❷
Router
Router
Router
MHA
MHA
MHA
❸ ❹ Router
Router
Router
Router
MHA
MHA
MHA
MHA
Transformer Block Holding Full Replica
Transformer Block Holding Partial Replica
(a)
(b)
❸
❹
Router
Router
MHA
MHA
Transformer Block Holding Individual Replica
(c)
Figure 4: Expert Placement and Ring: In a full-replica regime, each available expert incurs its own full ring across workers. As the Oe → 1, the communication cost R(Oe ) decreases with fewer workers considered for each expert’s ring, where it reaches Oe , where each ring is localized to its expert. 3.4
Synchronization and model optimization
FoMoE training (as we show in Algorithm 2) proceeds in rounds (McMahan et al., 2017): the server broadcasts parameters, workers perform K local steps (this implies a synchronization frequency reduction of 1/K), and updates are aggregated. We make several choices regarding the training process. Round structure and collectives. We utilize bandwidthoptimal Ring-AllReduce for aggregation. Averaging a tensor of size S across M peers entails transmitting 2(M −1) S bytes per peer (Rabenseifner, 2004; Patarasuk & M Yuan, 2009). While we employ a flat reduction, the framework supports hierarchical strategies for heterogeneous networks (Kairouz et al., 2021). Furthermore, asynchronous techniques like Streaming DiLoCo (Douillard et al., 2025) can overlap communication with computation; while we have not implemented such overlapping in FoMoE, it can be readily incorporated into our performance modelling (e.g., as a compression factor). Outer optimization. We combine infrequent synchronization (large K) with adaptive optimization. Workers execute AdamW (Loshchilov & Hutter, 2019) locally on their parameter subsets, while the server applies momentum (FedOPT) to the aggregated model to maintain convergence (Reddi et al., 2021; Khaled et al., 2025). Crucially, the local optimizer states are preserved rather than reset. For the global θdense , states are tracked by all workers; for θsparse , workers
maintain history only for their own experts. A challenge arises in the random placement regime, where expert migration separates weights from their optimizer history. To avoid the overhead of transferring states when experts are re-assigned, we re-initialize optimizers for migrated experts with a learning-rate warmup applied only to the reassigned weights. In contrast, fixed placement retains the state continuously. Empirically, we observe that preserving local momentum prevents the divergence observed in stateless approaches. To ensure scalability without retuning, we transfer the following hyperparameters: global learning rate schedules (scaled via µP/CompleteP (Yang & Hu, 2021; Dey et al., 2025; Malasnicki et al., 2025)), AdamW settings (β1 , β2 , ϵ), warmup duration, and router temperature. See Section C for more details. Bandwidth regimes and trends. Unlike high-speed intradatacenter settings (100.00 Gbps – 400.00 Gbps) where communication overlaps computation (Rajbhandari et al., 2022), WAN constraints (1.00 Gbps – 20.00 Gbps) necessitate minimizing volume (small Oe ) and frequency (large K). Although increasing M or K exacerbates model drift, the server momentum effectively stabilizes training (Stich, 2019). Recent works confirm that with tuned hyperparameters, infrequent synchronization maintains perplexity parity (Douillard et al., 2023; Sani et al., 2025) while reducing communication overhead (Iacob et al., 2025b). Gradient compression (Alistarh et al., 2017) remains orthogonal.
4
F O M O E S CALABILITY & R ESOURCE C ONSUMPTION
Building on the system design in Section 3, we define a quantitative cost model to optimize FoMoE configurations and hyperparameters under resource constraints. Given M weakly connected sites with effective WAN bandwidth Beff and fixed hardware, we determine the expert overlap Oe , synchronization interval K, and expert count Ne that maximize performance per wall-clock time. These models serve to determine how to setup FoMoE optimally with reference to specific requirements related to the underlying training infrastructure. The model comprises three components: (1) computational cost based on FLOPs (Section 4.1); (2) communication cost based on data volume (Section 4.2); and (3) a wall-clock time model synthesizing these factors within the distributed environment (Section 4.3). This framework encapsulates the partitioning methodologies of Section 3, the isolating synchronization frequency K, the expert overlapping factor Oe , and the worker count M as the primary levers. While prior work (Douillard et al., 2023; Sani et al., 2025) examined general communication-computation trade-offs, they neglected MoE-specific partitioning. Our model explicitly captures
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
the trade-offs of partial expert replication in bandwidthconstrained multi-datacenter settings, demonstrating how tuning Oe and K overcomes the “full-replica” barrier to efficient distributed training. 4.1
Compute and Memory Scaling
Training a dense Transformer with N parameters on D tokens requires approximately 6N D FLOPs (Kaplan et al., 2020), obeying power-law scaling between compute and model quality (Hoffmann et al., 2022). Sparse models MoE decouple the parameter count from the compute cost by activating only k ≪ Ne experts per layer. The effective parameter count for compute becomes Ndense + ϕNexp , where ϕ = k/Ne is the active fraction (Du et al., 2022) of the total expert parameters. Since ϕ ≪ 1, MoEs are capable of achieving a trillion-parameter scale with the per-token cost of significantly smaller dense models (Riquelme et al., 2021; Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2022). In our federated topology, local compute depends on the available experts. If a worker holds Nle,ℓ ≥ k experts, the costs match that of centralized training. However, if Nle,ℓ < k (e.g., each worker has a single expert when k = 2), the worker activates only min(k, Nle,ℓ ) experts, reducing FFN FLOPs (Fedus et al., 2022). Furthermore, the gating network projects only to local experts, scaling router operations by Nle,ℓ /Ne,ℓ = Oe /M . Consequently, reducing expert overlap Oe lowers per-token compute, especially when forcing sparse activation (e.g., top-1) (Gale et al., 2023). We estimate the per-token FLOPs Ftoken for a worker with e overlap Oe and local expert count Nle,ℓ = O M Ne,ℓ as: ! Ftoken ≈ Fdense +
PL
ℓ=1
min(k, Nle,ℓ ) FFFN,ℓ k
+
Nle,ℓ Ne,ℓ Fgate,ℓ
(1)
where Fdense , FFFN,ℓ , and Fgate,ℓ denote the per-token FLOPs of the dense components, unpartitioned expert FFNs, and full gating networks, respectively. In the presence of “ghost experts” adopting the “skip-token” procedure, the formula requires applying a scaling factor that accounts for the possibility of skipping the FFN computation for a token. With a perfectly balanced routing network, such a scaling factor can be written as: σst =
Nle,ℓ ≤1 Ne,ℓ
(2)
and, consequently, the total flops per token can be expressed, more generally, as: ! Ftoken ≈ Fdense +
PL
ℓ=1
σst
min(k, Nle,ℓ ) FFFN,ℓ k
N
+ Nle,ℓ Fgate,ℓ e,ℓ
(3)
We note that the routing decisions are not always perfectly balanced; the introduction of a load-balancing loss ensures they converge to a balanced state. In the presence of a severe imbalance in the assignment, the scaling factor above should be replaced with the actual frequency of the token assignments to “ghost experts” against the total number of assignments. In our experiments, we observe an overall balance, with the empirical scaling factor always being very close to its ideal counterpart. We recommend running a few warm-up iterations in full-replica mode to detect excessive imbalance before it becomes too problematic to maintain machine learning performance. Memory Footprint. Large-scale training memory comprises steady-state allocations for parameters and optimizer states, and dynamic allocations for activations and temporary buffers (Rajbhandari et al., 2020; Micikevicius et al., 2018). A standard mixed-precision AdamW accounting requires ≈ 18.00 B per parameter (2 for FP16 weights, 4 for FP32 master weights, 8 for moments, and 4 for gradients), while inference-style accounting removes gradients and optimizer states. Activation memory scales with batch size and sequence length, and activation checkpointing can trade additional compute for lower activation storage (Chen et al., 2016). Our FoMoE architecture reduces both memory components. Partitioning experts across M sites with overlap Oe ree duces the stored expert parameters per site to a fraction O M . TheP expected per-site steady-state memory is approximately Oe ℓ,e sℓ,e + Sdense , where sℓ,e and Sdense denote the size M of expert and dense parameters, respectively (Rajbhandari et al., 2022). Reducing Oe toward 1 yields nearly linearly proportional memory savings, particularly for models where sparse experts dominate the parameter count (e.g., > 60% in DeepSeek-V2 (DeepSeek-AI et al., 2024)). Furthermore, fewer local experts reduce the dimensionality of the gating network, providing additional steady-state savings. Dynamic memory also decreases with lower Oe . When a worker holds fewer experts than the global top-k (i.e., Nle,ℓ < k), it computes activations for only min(k, Nle,ℓ ) experts per token. For example, a local top-1 execution in a global top-2 regime halves the stored FFN activation tensors. Consequently, fewer local experts reduce activation memory and gradient buffer usage. These savings are additive to standard memory-optimization techniques (Fedus et al., 2022; Rajbhandari et al., 2020). 4.2
Communication Scaling
We quantify the communication overhead by deriving the bytes-per-token (BPT) exchanged between sites. This metric links the synchronization interval K and expert overlap Oe directly to WAN bandwidth requirements.
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Setting and Aggregation. As detailed in Algorithm 2, training proceeds in synchronous rounds where M sites perform K local steps before aggregating parameters (McMahan et al., 2017; Reddi et al., 2021; Li et al., 2020b). We partition the model into fully replicated dense parameters θdense and sparse expert weights {θsparse } distributed according to the overlap factor Oe (where Oe denotes the number of workers hosting a specific expert). Various strategies modify the synchronization payload and frequency: LocalSGD (Stich, 2019) and DiLoCo (Douillard et al., 2023) sync only model parameters, keeping optimizer states local; LocalAdamW (Cheng & Glasgow, 2025) syncs both, roughly tripling traffic; and DES-LOC (Iacob et al., 2025b) decouples parameter and state synchronization schedules, enabling flexible trade-offs in parameter vs optimizer state synchronization. Outer-optimizer analyses further explain the roles of learning rates, momentum, and acceleration in local-update regimes (Khaled et al., 2025). We consider a subset of these strategies for our parameter-partitioned MoE regime. Modeling Communication Costs. Synchronizing fully replicated components (θdense of size Sdense in bytes) via Ring-AllReduce (Rabenseifner, 2004; Patarasuk & Yuan, 2009) incurs a per-site cost of: per-site Bdense (M ) =
2(M − 1) Sdense M
2(Oe − 1) Oe
(5)
Note that when Oe = 1, this cost disappears since there’s no synchronization and aggregation to be made. During step (2), the communication cost is determined by the set difference between the experts assigned at round t + 1 and those already present at round t. We model this as a probabilistic process. Assuming a uniform random assignment, the probability that a specific expert e assigned to a worker in the new placement was already present in its local partition St is equal to the fraction of global experts held locally: phit =
( Oe Ne,ℓ ) Nle,ℓ Oe = M = Ne,ℓ Ne,ℓ M
We define the weighting factor P (Oe , M ) as this miss probability, scaled by the reshuffling frequency γ (where γ = 1 implies reshuffling every synchronization round, and γ < 1 implies amortized periodic reshuffling). As discussed in Section 3.3.2, we investigate two placement modes, fixed and random, which correspond to the setting γ = 0 and γ = 1, respectively. Substituting into Equation (7): Oe P (Oe , M, γ) = γ 1 − M
(7)
This derivation aligns with the boundary conditions discussed in Section 4.2: • Full Replica (Oe = M ): The term becomes 1 − 1 = 0. Since workers hold all the experts, no migration is ever required, and the synchronization term also accounts for the cost of weight updates. • Disjoint Partition (Oe = 1): The term becomes 1 ≈ 1. A worker holds a negligible fraction 1− M of the global experts, so nearly every reshuffling event requires replacing the local expert set in its entirety.
(4)
For sparse components, communication depends on the overlap Oe . Assuming a uniform assignment where each e worker holds Nle,ℓ = O M Ne,ℓ experts, the per-site communication cost depend on two steps: (1) synchronization and aggregation of experts’ weights which can be performed with Ring-AllReduce; and (2) the broadcast due to potential re-shuffling of the experts across workers, which only occurs for the random placement strategy. During step (1), a worker will need to exchange data for each local expert with a cost weighted by a ring factor: R(Oe ) =
Conversely, the probability that an assigned expert is missing and requires migration (a “broadcast” event) is pmiss = e 1− O M .
(6)
Adding these components together, the communication cost due to sparse components of the model is: per-site Bsparse (Oe , M, γ) = (R(Oe ) + P (Oe , M, γ)) per-site e Bsparse (Oe , M, γ) = (R(Oe ) + P (Oe , M, γ)) O M
P
(8)
P ℓ Ne,ℓ sexpert,ℓ
(9)
ℓ
Nle,ℓ sexpert,ℓ
where Nle,ℓ is the number of local experts, sexpert,ℓ is the single expert size in bytes, which includes up and down projection, and its router projector if not using the “skiptoken” procedure. This formulation recovers the full-replica cost when Oe = M (matching Equation (4)). Notably, with Oe = 1, the system requires one worker to act as an orchestrator to handle the reassignment of distinct sparse components that are not shared among workers. In such cases, the ring factor 2(OOee−1) disappears, and the cost uniquely depends on the summation. Strategies synchronizing opper-site timizer states (e.g., LocalAdamW) multiply BAR by a constant factor (typically 3×) (Cheng & Glasgow, 2025). Bytes-Per-Token (BPT). For a site processing Tlocal = KBl tokens per round, where K is the number of local optimizaB tion steps and Bl = Mg is the local batch size (global batch size Bg split across the M parallel workers), the total BPT is derived from the sum of dense and sparse costs per-site per-site (B per-site = Bdense + Bsparse ): BPT =
B per-site M B per-site = KBl KBg
(10)
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
This relation highlights three key levers: increasing K or batch size B amortizes communication; minimizing Oe caps the sparse term. 4.3
Model Dynamics. This framework highlights three critical dynamics captured by the model: 1. The Federated Compute-Bound Regime: Ideally, training satisfies K · tcomp ≫ (1 − α)tcomm . If intersite bandwidth is comparable to intra-site (e.g., dedicated dark fiber), partitioning offers minimal benefit. However, on commodity WAN, full-replica approaches (Oe = M ) may require destabilizingly large K to satisfy this inequality. FoMoE achieves this regime with moderate K by aggressively reducing payload size via partial replication (Oe → 1). This improvement is multiplicative with other optimizations; for example, maximal overlap (α → 1) and compression (ρ ≪ 1) can make low-bandwidth training more attainable even for 100B+ modelled configurations on 1.00 Gbps links (Qi et al., 2025).
Wall-Clock Time Model
We synthesize the computational (Section 4.1) and communication (Section 4.2) models to estimate the total training time Ttotal . This unified framework functions as a “Roofline” model for federated MoE training, identifying bottlenecks to guide the selection of the synchronization interval K, expert overlap Oe , and reshuffling frequency γ. Computational Time (tcomp ). The local optimization step time, tcomp , depends on the dynamic FLOP count and hardware efficiency. Unlike dense baselines, tcomp scales with the partitioning strategy. Let Φpeak be the theoretical peak FLOPs and MFU(·) be the Model FLOPs UtilizaB tion, which is sensitive to local batch size Bl = Mg and arithmetic intensity of the reduced expert kernels. Using Ftoken (Oe ) from Equation (3):
2. The Memory-Time Coupling: Equation (13) reveals a quadratic efficiency gain from reducing Oe . Lowering overlap reduces memory pressure (Section 4.1), enabling a larger local batch size Bl . Since the commu1 nication overhead ratio is proportional to K·B , increasl ing Bl effectively dilutes the fixed communication cost over more tokens. Thus, reducing Oe linearly reduces the payload (B per-site ↓) increases throughput per sync (Bl ↑).
B
tcomp (Oe , M ) =
Ftoken (Oe ) · Mg B
MFU( Mg , Nle,ℓ ) · Φpeak
(11)
Reducing Oe lowers Ftoken due to skipping tokens, reducing tcomp . However, this is bounded by the fragmentation limit: if the local expert count Nle,ℓ becomes too small relative to Bl , MFU degrades due to insufficient tensor core saturation. Communication Time (tcomm ). Communication time is governed by the effective WAN bandwidth Beff . While intradatacenter interconnects (e.g., NVLink) offer 100–400 Gbps (Rajbhandari et al., 2022), cross-site WAN links typically operate at 1–20 Gbps (Jaghouar et al., 2024), making transfer time bandwidth-dominated. We extend the payload formulation to account for gradient compression (e.g., QSGD (Alistarh et al., 2017)), which is an orthogonal addition to our work, via a factor ρ ∈ (0, 1], which is ρ = 1.0 by default: tcomm (M, Oe , γ) ≈
per-site per-site ρ·[Bdense (M )+Bsparse (Oe ,M,γ)] Beff
(12)
This term captures the convex trade-off of our strategy: lowering Oe minimizes gradient synchronization (the aggregation cost), but utilizing random placement (γ > 0) introduces a state migration penalty (the broadcast factor P → γ). Total Training Time. The total time Ttotal for a budget of Ntokens sums the costs of K compute steps and one communication step per round, subject to an overlap factor α. With global batch size Bg = M · Bl , the total rounds are Ntokens K·Bg : Ntokens Ttotal ≈ M ·Bl ·K [K · tcomp (M, Oe ) + (1 − α)tcomm (M, Oe , γ)]
(13)
3. The Placement Convexity: Under random placement, the interaction between aggregation and migration costs dictates an optimal overlap Oe∗ . If reshuffling is frequent (γ ≈ 1), the migration penalty makes disjoint partitioning (Oe = 1) suboptimal. FoMoE targets a “Semi-Static” regime (γ ≪ 1) to amortize migration costs, allowing the system to reduce bandwidth via disjoint experts. Solving Equation (13) for a specific hardware profile (Beff , Φpeak ) yields the theoretical optimal configuration (Oe∗ , K ∗ ) that minimizes wall-clock time. In practice, we generate a set of hardware-efficient configurations ranked by increasing training time and empirically evaluate their task performance (e.g., language modeling loss). This sweep optimizes the trade-off between system efficiency and task performance, aiming to maximize performance per time unit.
5
E XPERIMENTAL S ETTING
We evaluate FoMoE across a range of model scales to quantify trade-offs among communication efficiency, computational throughput, and machine learning quality. The experiments focus on the modeling and cost effects of partial expert replication: convergence, routing stability, communication volume, and throughput. They deliberately do not
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
attempt to characterize every production systems concern, such as time-varying jitter, asymmetric links, stragglers, or heterogeneous accelerators.
set of the en C4 corpus (Raffel et al., 2020), where we report both the mean and the standard deviation of the evaluation perplexity achieved across the entire dataset.
Model Configurations. We define five model configurations: Small Proxy (54M), Medium (150M), Large (2B), XL (13B), and XXL (100B). These span five operational regimes: rapid prototyping, entry-level modeling, compact open-weight scaling, standard open-weight deployment, and memory-bound scaling. All models utilize a decoder-only Transformer backbone as described in Section B. All comparisons are made on an iso-parameter basis: each sparse MoE configuration matches the parameter count of its dense counterpart while reducing inference FLOPs by a factor 0.25. To configure the larger model sizes, internal components are scaled from the Small Proxy (54M) model according to (Dey et al., 2025; Malasnicki et al., 2025; Yang & Hu, 2021). The specifics of the architectures are detailed in Table 1.
Communication Efficiency. We quantify network overhead using Bytes Per Token (BPT) and total synchronization volume per round. We compare FoMoE against full-replica baselines (e.g., DiLoCo/Photon), modeling the communication cost as a function of expert overlap. Theoretically, setting Oe = 1 (disjoint experts) restricts synchronization to the dense backbone (θdense ), so the payload reduction scales with the share of parameters in sparse experts. In the XL and XXL configurations of Table 1, where sparse experts dominate the model state, this removes most expert payload from each cross-site round.
Table 1: Overview of the model configurations used in the “ghost experts” studies. Configurations range from a small proxy model to a massive 100B parameter model, maintaining the same head dimension (dhead = 64) and expansion factor (4). Model Architecture
MoE Configuration
Scale
Configuration
dmodel
nlayers
nheads
Nexperts
dexpert
Nactive
Total Params
Small Proxy Medium (150M) Large (2B) XL (13B) XXL (100B)
256 512 1024 2048 4096
4 16 128 256 512
4 8 16 32 64
8 16 32 64 128
128 128 128 128 128
2 4 8 16 32
∼54M ∼153M ∼1.8B ∼13.3B ∼104B
Machine Learning Performance. We evaluate performance stability and convergence speed. To circumvent the prohibitive cost of hyperparameter tuning at the 100B scale, we adopt the CompleteP parameterization framework (Dey et al., 2025; Malasnicki et al., 2025; Yang & Hu, 2021). We first validate the optimal learning rates across cross-site configurations derived at the Small Proxy (54M) scale and then transfer them to large model settings. All models are trained on a large pre-training corpus composed of a mixture of general web-scraped text and code inspired by the SmolLM Corpus (Ben Allal et al., 2024). We adopted the GPTOss tokenizer for pre-processing the raw text (OpenAI, 2025). The training duration has been chosen to reach at least the Chinchilla compute optimality, i.e., 20 tokens per parameter, and slightly adjusted to allow cross-site experiments to terminate at a round boundary (Hoffmann et al., 2022). Subsequently, we conduct ablation studies on the overlapping factor Oe to quantify the perplexity degradation resulting from reduced expert visibility. Finally, we analyze the routing dynamics of the “skip-token” mechanism to ensure that dropping tokens for “ghost experts” does not destabilize convergence relative to forced local routing. Each trained model was evaluated on the entire validation
Computational Efficiency. Efficiency is measured via Model FLOPs Utilization (MFU) and wall-clock throughput (tokens/sec). While standard partial replication maintains an iso-FLOP count per token, the “skip-token” mechanism dynamically reduces the effective computational graph. We model the theoretical throughput gains of this mechanism and compare them against empirical results to identify overheads from router divergence and memory fragmentation. Model Size Scaling and Placement. We explore the highdimensional design space defined by the interactions among model size (N ), expert count (Ne ), and cluster size (M ). By varying the number of experts and workers, we analyze the fragmentation limit: the threshold where the reduction in local batch size per expert (due to high Ne /M ratios) can degrade GPU tensor core saturation. Training Environment. Experiments are executed on GPU cluster nodes with NVIDIA accelerators. Unless a result is explicitly identified as an empirical throughput measurement, WAN bandwidth and latency values are treated as controlled cost-model parameters rather than a claim of packet-level network emulation. In particular, our modeled cross-site setting uses 1.00 Gbps and 10.00 Gbps links with 50.00 ms latency, without time-varying jitter, asymmetric bandwidth, or persistent stragglers. Since the present implementation is synchronous, a slow site would delay the corresponding synchronization round. Stale-synchronous or asynchronous variants, such as Streaming DiLoCo, are compatible with the cost model but left to future work. Implementation. We implement FoMoE using PyTorch collective communication primitives and a lightweight coordination layer. The coordinator initializes dense parameters globally, assigns sparse expert subsets according to the placement policy, triggers local training for K steps, and aggregates only the parameters owned by each expert replica group. Standard DistributedDataParallel
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Reproducibility. Section D reports the full reproducibility summary for the experiments. The appendix records the training settings, sweep grids, dataset mixture, evaluation protocol, artifact status, and known gaps.
6
102 109
E VALUATION AND R ESULTS
We systematically evaluate FoMoE to quantify the tradeoffs between communication, throughput, and quality. The evidence combines three layers: trained small-proxy quality runs for convergence and routing stability, measured skip-token throughput on accelerator hardware, and analytical/modelled communication, FLOP, memory, and wallclock scaling for larger configurations. Section 6.1 benchmarks full-replica baselines and confirms that stateful outer optimizers are necessary to maximize Pareto efficiency compared to methods such as DiLoCo. Section 6.2 and Section 6.3 evaluate expert overlap, placement, and token skipping. The main takeaway is that FoMoE shifts the Pareto frontier relative to DDP and full-replica low-communication baselines, providing improved performance at a given communication or compute cost in the studied regimes. Section 6.4 evaluates modelled scalability, demonstrating that reducing expert overlap yields linear reductions in communication and wall-clock components under the cost model, while “skip-token” improves measured throughput without specialized kernels. To ensure that our baselines are well-tuned across all trained model scales, we adopt the CompleteP framework (Dey et al., 2025; Yang & Hu, 2021; Malasnicki et al., 2025). We provide further context and details of the sweeping procedure in Section C. 6.1
2 × 102
Perplexity
and FSDP assume static full-model replication or highbandwidth intra-cluster sharding, so they cannot directly express the variable expert subset synchronization used by FoMoE without an additional placement and aggregation layer.
Baseline Evaluation for Full Replica
We benchmark FoMoE against two “full replica” baselines – dense and MoE implementations – on the perplexityversus-communication Pareto front. Specifically, we evaluate FoMoE across varying synchronization intervals K ∈ {32, 64, 128} and worker counts M ∈ {2, 4, 8}, utilizing both DiLoCo and LocalAdamW outer optimizers. Experiments employ our small proxy configuration (see Table 1) and an iso-token budget, training all models to computeoptimal parameter counts. In these plots, centralized denotes the single-site high-bandwidth reference training run with no WAN throttling or low-frequency cross-site synchronization; it is a quality reference rather than a deployable low-bandwidth baseline. Full replica denotes a crosssite configuration that keeps all dense and sparse param-
1011
1010
Total Communication (Bytes) Centralized
LocalAdamW
DiLoCo
Figure 5: Pareto frontier of communication costs versus perplexity for full-replica baselines across outer optimizers. “Centralized” is the high-bandwidth single-site quality reference. LocalAdamW offers a better trade-off than DiLoCo at a given communication budget, motivating the stateful optimizer used in subsequent FoMoE configurations.
eters at every site and reduces only synchronization frequency. Fig. 5 demonstrates that FoMoE establishes a superior Pareto frontier for low-bandwidth training. Despite higher communication costs from transmitting optimizer states, LocalAdamW significantly outperforms DiLoCo in perplexity, both absolutely and at isocommunication points. Crucially, LocalAdamW enables FoMoE to match centralized baseline performance with significantly reduced communication, whereas DiLoCo struggles. We attribute this to DiLoCo’s stateless nature; synchronizing optimizer states is known to enhance stability and performance (Iacob et al., 2025b; Cheng & Glasgow, 2025). Furthermore, consistent with prior work (Sani et al., 2025; Douillard et al., 2023), Table 4 confirms that increasing K or M degrades perplexity due to increased local drift and gradient noise, respectively. These findings indicate a minimum communication floor required to maintain the Pareto optimality observed in Fig. 5. 6.2
Oe and Expert Placement Strategies
Following our baseline evaluation in Section 6.1, we ablate the impact of Oe and the placement strategy on the Pareto frontier of FoMoE, as shown in Figure 6. Consistent with the findings in Figure 5, LocalAdamW consistently outperforms DiLoCo due to optimizer state synchronization (Cheng & Glasgow, 2025; Iacob et al., 2025b). Across both DiLoCo and LocalAdamW, fixed placement yields more optimal settings on the Pareto frontier for both computation and communication cost in our homogeneous, i.i.d. setting. We interpret random placement as a trade-off rather than
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs 2.4 × 102 2.2 × 102 2 × 102
Perplexity
Perplexity
2 × 102
1.8 × 102 1.6 × 102 1.4 × 102 1.2 × 102
102 109
1010
Total Communication (Bytes) DiLoCo LocalAdamW
102
4.55 × 1017 4.56 × 1017 4.57 × 1017 4.58 × 1017 4.59 × 1017
Total FLOPs
fixed random
LocalAdamW, fixed LocalAdamW, random
DiLoCo, fixed DiLoCo, random
Skip Token Non Skip Token
2 × 102
Perplexity
Figure 7: Impact of “skip-token” and “ghost experts” on the perplexity versus FLOPs Pareto frontier. Across outer optimizer configurations, these mechanisms move FoMoE toward more efficient regions of the Pareto frontier, improving perplexity at a given FLOP cost relative to full-replica baselines.
102 4.56
4.57
4.58
Total FLOPs (1017)
DiLoCo LocalAdamW
4.58
fixed random
Figure 6: Pareto frontier comparing communication cost (top) and computation cost (bottom) against perplexity for FoMoE across different outer optimizers and placement strategies. FoMoE with LocalAdamW provides a better Pareto-optimal trade-off under fixed expert placement. Momenta averaging and expert reassignment affect the trade-off between regularization, data mixing, and additional migration overhead.
a resolved improvement: it may help under heterogeneous or non-i.i.d. data, but in the current setting its migration and optimizer-state costs dominate the additional mixing benefit. Addressing the impact of Oe : across all combinations of synchronization intervals K, placement strategies, and workers M we find that as Oe → M , performance converges towards full-replica performance as seen in Table 7. Smaller values of Oe achieve higher perplexity under an iso-token training budget, as the models have less active capacity per site to capture the underlying data distribution. However, larger Oe incur higher communication and computation costs; choosing Oe ≈ 4 ensures a Pareto-optimal trade-off in our studied setting. 6.3
“Skip-Token” and “Ghost Experts”
We now determine the influence of the “skip-token” and “ghost expert” mechanisms on the Pareto frontier, as seen in Figure 7. Across both outer optimizers, introducing these
mechanisms consistently leads to more favorable settings on the Pareto frontier. We can achieve lower perplexity at a lower compute budget by skipping missing local tokens in each worker’s partial replica. Additionally, these results show that adding the “skip-token” mechanism, where these tokens are not concatenated back into the residual stream, does not destabilize training convergence in the studied runs. Quality interpretation. Skip-token is not output-equivalent to full-replica training or to EP with remote activation dispatch: it changes the effective computation graph by removing expert paths that are unavailable at the local site. We therefore interpret Figure 7 as evidence about convergence and compute-quality trade-offs, not as proof of exact functional equivalence. Our current quality evidence consists of validation perplexity and routing stability during pretraining. Downstream task evaluations and longer horizon comparisons against full-replica/EP trajectories are important follow-up work, especially for reasoning-heavy tasks where small pre-training differences may be amplified. 6.4
FoMoE at Scale
We assess FoMoE’s scalability using the theoretical framework from Section 4. We analyze FLOPs, communication costs, and wall-clock time as analytical/modelled quantities, not as full large-scale training runs. Results show that minimizing expert overlap (Oe ) significantly reduces compute, communication, and time in the modelled configurations, enabling FoMoE to outperform DDP and full-replica baselines under those assumptions. FLOPs Scalability. Figure 8 instantiates predictions from
Communication Scalability. Figure 9 shows modelled cumulative communication in which FoMoE reduces bytes versus DDP and full-replica baselines. Unlike DDP’s continuous overhead, FoMoE employs periodic, smaller bursts. Wall-clock Time Scalability. Figure 10 demonstrates that minimizing communication via lower Oe reduces modelled training time across the plotted bandwidths (1–1000 Gbps). Speedups are most critical in low-bandwidth regimes (e.g., 1 Gbps), where minimizing overlap is essential. In these modelled bandwidth regimes, FoMoE provides a benefit over the full-replica variant. Since the gains provided by FoMoE are multiplicative with comms-efficient methods, their relative magnitude holds across bandwidths under the cost-model assumptions.
2.4
1e17 14%
14%
14%
2.3 2.2
8%
FFN (with token skipping) FFN (extra w/o skipping)
8% 4%
2.1
4% 1%
16
8
4
2
Overlapping Factor
1%
1
Total Training FLOPs
Total Training FLOPs
Measured Skip-Token Throughput. Figure 11 confirms that the “skip-token” mechanism enables the linear throughput gains predicted in Section 4 by bypassing computation for ghost experts. While reducing expert overlap generally improves throughput, skipping-token scales nearly linearly and outperforms the baseline. The current implementation of the mechanism is limited by a lack of a dedicated kernel; however, we expect that the theoretical gains will be at least partially transferable to any sufficiently optimized implementation. 4.5
1e20
31%
4.0
31%
31%
19%
3.5
FFN (with token skipping) FFN (extra w/o skipping)
18% 11%
10% 3%
32
16
8
4
1%
2
Overlapping Factor
5% 1%
1
Figure 8: Analytical/modelled total training FLOPs versus expert overlap (Oe ). Left: Small model (dmodel = 512, 16 layers, 16 workers). Right: Large model (dmodel = 8192, 512 layers, 32 workers). Green bars show modelled FLOPs with token skipping; red bars indicate avoided computation. Lowering overlap reduces compute, with greater relative savings for larger models.
7
L IMITATIONS
FoMoE establishes a new Pareto frontier for crossdatacenter training, but it should be read as a systems result with a bounded empirical envelope. The training evidence comes from proxy-scale quality runs, routing diagnostics, and measured skip-token throughput. The XL/XXL and
DDP Full Replica Oe=16 Oe=8 Oe=4
1014 1013 1012 0
1
2
3 4 Global tokens processed
5
6
7
1e8
Figure 9: Analytical/modelled cumulative communication overhead versus global tokens. Communication cost (log scale) for a large model (dmodel = 8192, 512 layers) on 32 workers using DiLoCo with random placement. FoMoE achieves increasing bandwidth savings over DDP and Full Replica as expert overlap decreases from Oe = 16 (green) to Oe = 4 under the model assumptions. 105 Total Training Time (Hours, log)
Section 4: (a) partitioning reduces router FLOPs, with linear FFN savings when Oe < k; (b) the “skip-token” mechanism further linearly reduces FLOPs by bypassing nonlocal experts; and (c) these savings grow with model scale as sparse parameters increasingly dominate.
Cumulative bytes / worker (log)
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
104 103 102 5
10
15 20 Expert Overlap (Oe)
BW = 1.0 Gbps BW = 10.0 Gbps BW = 100.0 Gbps BW = 1000.0 Gbps 25 30
Figure 10: Analytical/modelled total training time versus expert overlap (Oe ). Modelled time for a model (dmodel = 2048, 64 layers) trained for 60B tokens on 32 H100s using DiLoCo (random). Reduced overlap accelerates training, especially under bandwidth constraints.
100B-scale results are communication, memory, FLOP, and wall-clock projections from the system model rather than full empirical pre-training runs at those scales. Likewise, skip-token is a throughput-quality trade-off rather than an equivalence claim: it is not output-equivalent to full-replica training or to WAN expert parallelism with remote activation dispatch, because it deliberately omits expert paths that are unavailable at the local site. The systems setting is also deliberately controlled. Our evaluation assumes homogeneous workers, static traffic shaping, synchronous rounds, and representative WAN bandwidth/latency. It does not yet cover time-varying jitter, asymmetric links, persistent stragglers, hierarchical WAN topologies, or heterogeneous GPU clusters. Since the present implementation is synchronous, a slow site can still delay the corresponding synchronization round. Finally, the baseline and optimization scope are narrower than the full design space. We focus on DDP and full-replica low-communication methods such as DiLoCo/Photonstyle training; a direct WAN-deployed EP baseline would be a useful stress test, although standard EP assumes highbandwidth activation exchange and is therefore not the
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs FoMoE (amp_bf16) [Skip] FoMoE (amp_bf16)
Relative Speedup (Factor)
1.4 1.3 1.2 1.1 1.0 0.9 0
8
Number of Ghost Experts
16
Figure 11: Measured speedup from token skipping. Relative speedup versus the number of ghost experts for a 150M MoE model with dmodel = 512 and expert size 128 on 8 NVIDIA B200s. Skipping tokens achieves near-linear scaling, validating the theoretical savings despite lacking a custom kernel.
bottleneck targeted by FoMoE. We also use standard lowfrequency distributed optimization, adapted to partial expert replication, in order to isolate the payload-size effect. Future work should co-design optimizers, placement schedules, and optimizer-state migration rules specifically for partial expert replication.
8
C ONCLUSION
This paper presents FoMoE, a framework for exploiting MoE sparsity at the synchronization boundary of crossdatacenter training. Existing low-communication methods primarily reduce how often sites synchronize; FoMoE targets the complementary bottleneck of how much model state crosses the WAN at each round. By partitioning expert layers, FoMoE reduces communication overhead by up to 1.42× over efficient baselines and 45.44× over DDP in the studied regimes, while the “skip-token” mechanism delivers measured throughput speedups of up to 1.4×. The trained proxy regimes show stable routing and convergence behaviour, and the system model projects that the same payload and memory reductions become more important as sparse experts dominate larger configurations. The durable lesson is that MoE sparsity should not stop at the datacenter boundary: when WAN bandwidth is the scarce resource, reducing the synchronization payload is a first-class systems lever alongside reducing synchronization frequency.
ACKNOWLEDGMENTS This research was supported by the following entities: The Royal Academy of Engineering via DANTE (a RAEng Chair); the European Research Council, specifically the REDIAL project; SPRIND under the composite learning challenge; Google through a Google Academic Research Award; in addition to both IMEC and the Ministry of Education of Romania (through the Credit and Scholarship Agency).
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
A PPENDIX
A.2
A
E XTENDED R ELATED W ORKS
A.1
MoE architectures and scaling
Load balancing is critical for maximizing MoE throughput and model quality. Approaches include BASE, which formulates assignment as a balanced linear optimization (Lewis et al., 2021); Expert-Choice, which inverts routing to allow experts to select tokens, ensuring perfect load balance (Zhou et al., 2022); and Hash Layers, which replace learned gating with fixed hashing functions (Roller et al., 2021). In practice, training stacks employ temperature schedules, capacity factors, and auxiliary losses to prevent expert collapse and minimize token dropping (Fedus et al., 2022; Lepikhin et al., 2021). Standard diagnostics—including routing entropy, load coefficients of variation, and overflow rates—are essential for monitoring these stability dynamics.
The sparsely-gated MoE layer (Shazeer et al., 2017) established the paradigm of conditional computation, utilizing trainable gating to activate a sparse subset of expert FFNs per token. In this work, we adopt the σ-MoE formulation (Csordás et al., 2023), which replaces the competitive softmax gating with independent sigmoid activations to better approximate dense feed-forward networks. Formally, given an input token vector x ∈ Rd , the layer output y is the weighted sum of a sparse subset of expert networks {Ei }N i=1 : y=
X
si (x)Ei (x) ,
(14)
i∈T (x) ⊤ where si (x) = σ(wg,i x) is the gating score for expert i, computed using the logistic sigmoid function σ(·) and a trainable gating vector wg,i . The set of active indices T (x) is determined by the Top-K operator applied to the sigmoid scores:
⊤ T (x) = TopK {σ(wg,j x)}N j=1 , k .
(15)
Unlike standard Softmax gating, which introduces competition by normalizing scores to sum to one, the σ-MoE allows independent expert activations, mitigating instability. This computation is conditional because the experts Ej (x) for indices j ∈ / T (x) are never executed. This mechanism decouples model capacity from compute cost: while the total parameter count scales linearly with the total number of experts N , the computational cost (FLOPs) per token scales only with the active set size k (where k ≪ N ), outside of the router network and sigmoid operations. This enables massive parameter scaling with near-constant training/inference costs. Subsequent systems scaled this approach: GShard introduced Top-2 gating with automatic sharding (Lepikhin et al., 2021), while Switch Transformers demonstrated trillionparameter training via Top-1 routing (Fedus et al., 2022). GLaM further validated the compute efficiency of MoE architectures against dense baselines (Du et al., 2022). Modern iterations, such as DeepSeek-V2, integrate MoE with optimized attention and KV-cache designs (DeepSeek-AI et al., 2024). Theoretical advancements include fine-grained scaling laws (Ludziejewski et al., 2024), contextualized by foundational dense scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022). Beyond NLP, scalability has been confirmed in vision domains (Riquelme et al., 2021).
A.3
Routing and load balancing
Systems support for MoE execution
Specialized frameworks optimize MoE execution through all-to-all communication, kernel fusion, and parallelization strategies. DeepSpeed-MoE and Tutel enable efficient expert parallelism and hierarchical communication (Rajbhandari et al., 2022; Hwang et al., 2023), while MegaBlocks introduces dropless block-sparse kernels to mitigate padding inefficiencies (Gale et al., 2023). FasterMoE and SmartMoE further optimize end-to-end training via dynamic hybrid parallelization schemes (He et al., 2022; Zhai et al., 2023). Crucially, these systems are designed for high-bandwidth intra-DC fabrics; our work adapts these principles to the bandwidth-constrained interDC regime. A.4
Low-bandwidth distributed and federated pre-training
To accommodate low-bandwidth environments, federated and LocalSGD methods reduce synchronization frequency (McMahan et al., 2017; Stich, 2019), with extensions like FedProx and FedOPT addressing data heterogeneity (Li et al., 2020b; Reddi et al., 2021). DiLoCo applies these principles to LM pre-training, utilizing large local steps and outer momentum to train across weakly connected device islands (Douillard et al., 2023), a capability operationalized by OpenDiLoCo (Jaghouar et al., 2024). Photon further demonstrates federated pretraining up to 7B parameters with perplexity parity and significantly reduced communication (Sani et al., 2025). Additional optimization strategies include decoupling embeddings via DEPT (Iacob et al., 2025a), desynchronizing optimizer states via DES-LOC (Iacob et al., 2025b), and co-designing modular paths via DiPaCo (Douillard et al., 2024). While these systems motivate our design, they rely on full replicas.
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
A.5
Expert placement and topology awareness
While intra-datacenter routing is well-studied, expert placement across heterogeneous wide-area networks (WAN) remains underexplored. Existing systems model placement primarily for intra-cluster efficiency (Hwang et al., 2023; He et al., 2022; Gale et al., 2023). Our work investigates whether selective expert replication can minimize WAN traffic without degrading quality. We evaluate fixed, random, and affinity-based placement policies, analyzing their impact on stability (routing entropy), specialization, and load balance (Lewis et al., 2021; Zhou et al., 2022; Fedus et al., 2022). A.6
Federated MoE
Early research in Federated MoE focused on personalization, combining global generalists with local specialists (Zec et al., 2020; Guo et al., 2021), or employing client-specific gating (FedMix) to enhance generalization (Reisser et al., 2021). Subsequent systems extended this to Transformers: FedMoE assigns personalized sub-MoEs to clients (Mei et al., 2024), while A3SMoE optimizes expert utilization across heterogeneous devices (Tran et al., 2025). Decentralized approaches like dFLMoE, Learning@home/DMoE, and SWARM explore expert fusion, volunteer computing, and heterogeneous weak-link model parallelism (Xie et al., 2025; Ryabinin & Gusev, 2020; Ryabinin et al., 2023). However, these studies primarily address personalization, volunteerdevice execution, inference constraints, or model-parallel scheduling; they do not address the payload size limitations inherent to communication-efficient federated pre-training. A.7
Optimization, privacy, and governance
Our approach relies on established analyses of adaptive federated optimizers (FedAdam, FedYogi) and LocalSGD (Reddi et al., 2021; Stich, 2019). While privacy mechanisms like secure aggregation (Bonawitz et al., 2017; Abadi et al., 2016; Kairouz et al., 2021) can be integrated, they introduce trade-offs in quality and complexity. Additionally, routing and placement decisions can be logged to ensure auditability and fairness in non-IID settings.
expert placement, thereby improving Pareto trade-offs in perplexity, communication, and wall-clock time.
B
To address the stability challenges of training large-scale models over bandwidth-constrained federated networks, FoMoE adopts a specialized Transformer architecture. We depart from monolithic baselines by integrating SigmaMoE (σ-MoE) routing with a Peripheral LayerNorm (PeriLN) strategy. These choices maintain gradient flow and prevent router collapse in the Partial Replica regime (Oe < M ), where expert availability fluctuates due to random placement. The full architecture is shown in Fig. 12. B.1
Synthesis
Prior work establishes that: (i) MoE architectures achieve dense-model quality with reduced compute (Fedus et al., 2022; Du et al., 2022; DeepSeek-AI et al., 2024); and (ii) low-communication training can approach centralized baselines under WAN constraints (Douillard et al., 2023; Sani et al., 2025). However, current cross-silo systems remain limited by the requirement to transmit full dense models. Our work bridges this gap, introducing a federated MoE framework that targets bytes per sync through optimized
Peripheral LayerNorm (Peri-LN)
Standard Pre-LayerNorm (Pre-LN) allows unbounded growth in the residual stream, while Post-LayerNorm (PostLN) impedes gradient propagation, causing instability in low-frequency synchronization settings. We employ Peripheral LayerNorm (Kim et al., 2025), which applies normalization exclusively to the residual branches—both immediately preceding (Pre-LN) and following (Post-LN) the computational blocks—while preserving a linear, un-normalized main data path. Formally, for a layer input xl and computational block F (e.g., MHA or MoE): xl+1 = xl + LN(F (LN(xl )))
(16)
This configuration ensures that inputs to the routing mechanism maintain unit variance, preventing dot-product logits from saturating the gating function, while maintaining a clear gradient highway from loss to input. We additionally apply a Post-Embedding LayerNorm to the initial token embeddings. As demonstrated in (Kim et al., 2025), this modification helps control the variance entering the first Transformer block, which has been shown to improve training stability. B.2
A.8
M ODEL A RCHITECTURE AND C ONFIGURATION
Attention Mechanism
We utilize Multi-Head Attention (MHA) enhanced with Rotary Positional Embeddings (RoPE) (Su et al., 2024). Positional information is injected into the Query (Q) and Key (K) vectors within the normalized branch (post-PeriLN), ensuring relative positional information is preserved without dampening by residual stream statistics. B.3
Federated Mixture-of-Experts (σ-MoE)
We replace the Feed-Forward Network (FFN) with a sparse Mixture-of-Experts layer based on the σ-MoE formulation.
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
B.3.1
Expert Configuration and Fine-Grained Scaling
To balance routing flexibility with memory fragmentation, we adopt a Fine-Grained MoE strategy (Ludziejewski et al., 2024). We decompose the standard dense FFN (df f = 4dmodel ) into Ne experts with granularity G = 8 and expansion E = 1, leading to Ne = G · E experts. To maintain an iso-parameter count with the dense baseline, the expert dimension scales as: dexpert =
4dmodel Ne
(17)
After each expert projection from dmodel to dexpert , the SiLU nonlinearity is applied. For a dmodel = 512 configuration, we employ Ne = 16 experts of dimension dexpert = 128. This granular partitioning allows the router to approximate larger functional manifolds by combining specialized sub-experts, mitigating the impact of missing experts in partial replication. B.3.2
Sigmoid Routing
Unlike competitive Softmax routers (e.g., Switch Transformer), FoMoE employs Sigmoid Routing to enforce independent expert selection. This independence is critical for the Partial Replica setting (Oe < M ): the unavailability of a specific expert on a worker must not skew the probability distribution of the remaining available experts. The gating probability for expert i is given by: pi (x) = σ(x⊤ wr,i )
(18)
After projecting scores from dmodel to Ne , the router selects the top-k experts (typically k = 2) based on these independent scores, with inputs normalized via the FFN’s Peri-LN to ensure stable routing entropy. Architecture Connection Embedding LN
Residual Stream
Depth Multiplier
Depth Multiplier
Token Dataflow Unembedding
LN LN
LN
LN
Top-K Selection
σ
LN σ
Router σ
FFN Experts Multi-Head Attention
XN
σ
FFN Layer
Transformer Block
Figure 12: FoMoE Layer Architecture. The diagram illustrates the Peripheral LayerNorm (Peri-LN) strategy, where normalization is applied exclusively to the branches feeding the Attention and MoE blocks (Pre-LN) and immediately following them (Post-LN). This design maintains a linear residual stream to preserve gradient magnitude. Additionally, the architecture includes a depth multiplier applied to the output of each computational block, as well as layer normalization on the output of the embedding layer and the input of the unembedding layer.
C
H YPERPARAMETER T RANSFERABILITY AND I MPLEMENTATION
To avoid prohibitive hyperparameter retuning at scale, we leverage established transfer techniques. Maximal Update Parametrization (µP) enables the transfer of learning rates and schedules across model widths (Yang & Hu, 2021), while CompleteP extends this invariance to model depth via residual rescaling, preventing instability in deep Transformers (Dey et al., 2025). We note, however, that scaling the expert count E requires specific adjustments to preserve feature learning efficiency (Malasnicki et al., 2025). Transfer protocols. We transfer the following hyperparameters from baseline configurations: global learning rate schedules (scaled via µP/CompleteP), AdamW parameters (β1 , β2 , ϵ), warmup duration, and router temperature. Conversely, regularization parameters (dropout, weight decay) and system-specific controls (overlap factor Oe , reshuffling cadence T , local steps K) are treated as nontransferable and tuned individually at each scale. Consistent with (Malasnicki et al., 2025), we find that while µP effectively handles width scaling, significantly increasing E necessitates extending the learning rate warmup. Tuning was performed over a compute-optimal token budget (Hoffmann et al., 2022) using a linear warmup of 25% without a subsequent cooldown phase. For FoMoE, we evaluated learning rates η ∈ {0.001, 0.002, 0.004, 0.01, 0.02, 0.05}. Centralized baselines (dense and MoE) were tuned over η ∈ {0.01, 0.02, 0.03, 0.04, 0.07}. Across all architectures and aggregation methods, we observe that η ≈ 0.01 yields the optimal learning rate, as shown in Fig. 13. We use CompleteP to transfer the global learning rate schedule, AdamW settings (β1 , β2 , ϵ), warmup duration, and router temperature across model scales for both our federated and centralized baseline experiments. This was repeated for different regularization factors and systemspecific knobs, such as the overlapping factor Oe , the shuffling cadence T , and the local steps K, as these do not scale with the CompleteP framework. In the sweeps, we found that the optimal learning rate across all our experiment types was η ≈ 0.01. Implementation. We implemented FoMoE using PyTorch’s collective communication primitives. Standard frameworks such as DistributedDataParallel or FSDP are unsuitable for our setting, as they assume static, full-model replication and high-bandwidth intra-cluster interconnects. To support dynamic expert placement in WAN environments, we developed a custom coordination layer where the server explicitly manages the broadcast and aggregation of variable expert subsets at each synchronization round.
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
100 80 60
120 100 80 60
10 4 10 3.5 10 3 10 2.5 10 2 10 1.5
Type: localadamw Batch Size: 512
200
10 2.5
10 2
Learning Rate
120 100 80 60 10 2.5
10 2
Learning Rate
10 1.5
10 2.5
10 2
140 120 100 80
Learning Rate
10 2.5
10 2
Learning Rate
Local Steps: 1
10 1.5
Local Steps: 32
120 100 80 10 3
Type: nesterov Batch Size: 256
10 2.5
10 2
Learning Rate
10 1.5
160 140 120 100 80 60 5.62341 × 10 3
10 2
1.77828 × 10 2
Learning Rate
Type: nesterov Batch Size: 512
200 180
160 140 120 100 80 60
10 3
140
10 1.5
Type: localadamw Batch Size: 256
180
160
60
180
160
60 10 3
80
200
LanguagePerplexity
LanguagePerplexity
140
100
10 3
180
160
120
10 1.5
Type: nesterov Batch Size: 128
200
180
140
60 10 3
Learning Rate
160
LanguagePerplexity
120
140
200
180
LanguagePerplexity
140
160
Type: fedavg Batch Size: 512
200
180
LanguagePerplexity
160
Type: fedavg Batch Size: 256
200
180
LanguagePerplexity
LanguagePerplexity
180
LanguagePerplexity
Type: fedavg Batch Size: 128
200
LanguagePerplexity
Type: centralized Batch Size: 1024
200
160 140 120 100 80 60
10 3
10 2.5
10 1.5
10 3
Local Steps: 128
FFN: mptmlp
10 2
Learning Rate
Local Steps: 64
10 2.5
10 2
Learning Rate
10 1.5
FFN: sigma_moe
Figure 13: Learning rate tuning results for centralized (local step size of one) and federated baselines across varying local steps, architectures (dense vs. MoE), and aggregation methods. Across all configurations, a base learning rate of η ≈ 0.01 consistently minimizes perplexity. We adopt this optimal rate for subsequent model scaling experiments.
D
R EPRODUCIBILITY D ETAILS
This appendix expands the compact experimental description in Section 5. The entries below report settings that are needed to reproduce the experiments.
Regime 1: Distributed Data Parallelism (DDP). Defined by Oe = M (Full Replication) and K = 1. The communication payload is maximal (P = 0, R ≈ 2). Training is strictly bandwidth-bound on WANs: tcomm (DDP) ≈
E
T HEORETICAL F OUNDATIONS AND C OST M ODELING
This appendix serves as the formal theoretical underpinning for the FoMoE system. We provide rigorous derivations for the cost models presented in Section 4, analyze the system’s operational regimes, and formalize the trade-offs that govern the "Sweet Spot" for distributed MoE training. E.1
(19)
Utilization approaches zero as M increases across lowbandwidth links. Regime 2: Federated / Low-Frequency (e.g., DiLoCo). Defined by Oe = M and K ≫ 1. Communication is amortized by K, but the payload remains the full model size Smodel . The memory constraint is identical to DDP, limiting the maximum model size to the capacity of a single worker.
System Definitions and Parameters
We model a synchronous training setup with M homogeneous workers connected by a heterogeneous network topology. The system is defined by the global parameters listed in Table 3. E.2
2Smodel ≫ tcomp Beff
Operational Regimes and The Distributed Paradigm Shift
To contextualize the FoMoE architecture, we mathematically distinguish it from prior paradigms based on the constraints imposed on the synchronization interval K and the expert overlap Oe .
Regime 3: The FoMoE Regime (Partial Replication). Defined by Oe < M (typically Oe ≪ M ) and K > 1. This regime introduces a "Communication-Sparsity" coupling. By partitioning experts, we reduce the steady-state memory and the synchronization payload linearly with Oe . This breaks the "Full-Replica" barrier, allowing model size to scale linearly with M while tcomm decreases. E.3
Extended Computational Cost Modeling
We refine the FLOPs derivation to explicitly account for both Router and FFN efficiency gains. Router Efficiency. Standard gating networks project token embeddings of dimension dmodel to Ne logits. In
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Table 2: Reproducibility summary for the experimental setting. Values are grouped by the setting they reproduce. Item
Reproducibility detail
Model configurations
Five scales are used: Small Proxy (54M), Medium (150M), Large (2B), XL (13B), and XXL (100B). The larger configurations scale width and depth from the 54M proxy while keeping dhead = 64, expansion factor 4, expert hidden size 128, and active expert ratio 0.25. Text is tokenized with openai/gpt-oss-120b; the tokenizer vocabulary size is 200,019. All reported training configurations use sequence length 2048. The pre-training mixture uses HuggingFaceTB/smollm-corpus subsets fineweb-edu-dedup and python-edu in a 90/10 stream proportion. Evaluation uses the C4 en validation split. The reported standard deviations are computed across runs with different seeds. Global batch size is 1024 sequences. For M ∈ {2, 4, 8} workers, local batch size is 1024/M . The proxy training budget targets 20 tokens per parameter; after synchronization-boundary rounding this corresponds to 640 optimizer batches, 160 warmup batches, and approximately 1.34B realized tokens for the 54M proxy. Local training uses AdamW with weight decay 0, betas (0.9, 0.95), and ϵ = 10−8 . The proxy learning-rate sweep covers {0.01, 0.01144714, 0.01310371, 0.015} before selecting transferred rates for the main sweeps. DiLoCo/Nesterov uses server learning rate 0.7 and momentum 0.9; LocalAdamW synchronizes first and second optimizer moments. The main grid uses K ∈ {32, 64, 128} local steps, M ∈ {2, 4, 8} workers, seeds {42, 123, 456}, and full participation. Full-replica MoE baselines use Oe = M ; fixed and random placement sweeps use the valid lower powers of two for each M ; skip-token sweeps additionally include full overlap and set skip_tokens=True. Experiments were run on NVIDIA GPU cluster nodes. The system analysis uses controlled WAN parameters of 1.00 Gbps and 10.00 Gbps with 50.00 ms latency, no time-varying jitter, no asymmetric bandwidth, and no persistent stragglers.
Tokenization and context Training corpus Evaluation data Batch and budget
Optimization
Sweep grid
Hardware reporting WAN parameters
Table 3: Global System and Model Parameters
transfers of size Sexpert /Oe . Total data volume per expert: 2(Oe −1) Sexpert . This confirms the ring factor R(Oe ) = Oe
Symbol
Definition
Context
M K Beff Oe Nle γ
Total number of workers (sites) Synchronization interval (local steps) Effective WAN bandwidth Expert Overlap Factor (∈ [1, M ]) Number of Local Experts per worker Reshuffling frequency (1/Tshuffle )
Infrastructure Optimization Infrastructure FoMoE Configuration e Derived: O M Ne Random Placement
FoMoE, tokens are only routed to locally resident experts. The router projection size is reduced from Ne to Nle . Fgate (Oe ) =
Nle full Oe full F F = Ne gate M gate
(20)
This yields a linear reduction in router FLOPs proportional to the partition fraction. FFN Efficiency and Skip-Token Scaling. As derived in the Main Text, the FFN cost is scaled by σst , accounting for the "Skip-Token" mechanism when Nle < k. Nle Ne σst = min 1, · (21) Ne k Combining these, the total FLOPs per token is: Ftoken ≈ Fdense +
Note that for disjoint ex-
The Broadcast Factor (P ). Under random placement with reshuffling frequency γ, we model the set difference between expert assignments St and St+1 . Assuming uniform random assignment from Ne experts: • Probability of "Hit" (Expert e ∈ St+1 is in St ): Oe t| phit = |S Ne = M . • Probability of "Miss" (Broadcast required): e pmiss = 1 − O M . The expected broadcast volume is proportional to the miss probability scaled by the reshuffling frequency γ: Oe P (Oe , M ) = γ 1 − M
(23)
This derivation highlights the convexity of the cost function: P maximizes when Oe is small (disjoint), which is exactly where R is minimized. This necessitates γ ≪ 1 (infrequent reshuffling) to make the disjoint regime viable.
Oe min(k, Nle ) FFFN + Fgate ℓ=1 σst k{z | } |M {z }
PL
FFN Reduction
E.4
2(Oe −1) used in Equation (5). Oe perts (Oe = 1), R = 0.
(22)
Router Reduction
Formal Derivation of Communication Factors
We formally derive the probabilistic broadcast factor P and the aggregation factor R used in the Main Text. The Aggregation Factor (R). For a specific expert replicated on Oe workers, Ring-AllReduce requires 2(Oe − 1)
E.5
Optimization Surface and "Sweet Spot" Analysis
We analyze the optimization surface of the Master Equation (13) to identify the efficient "Sweet Spot" for training. We classify the operational space into four regions based on the variables Oe and K. 1. Region 1: Comm-Bound (High Oe , Low K). Approximating DDP. tcomm dominates due to the full ring factor R ≈ 2. Throughput is throttled by Beff .
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
2. Region 2: Memory-Constrained (High Oe , High K). While communication is amortized by K, the high Oe prevents memory savings. Local batch size Bl is constrained, preventing the "dilution" of communication costs (see Memory-Time Coupling in Main Text). 3. Region 3: The FoMoE Sweet Spot (Low Oe , High K). Minimizing Oe drives R → 0 and maximizes permissible Bl . This maximizes the Communicationto-Compute Ratio (CCR), enabling high throughput on low-bandwidth WANs. This is the optimal operating point. 4. Region 4: Placement Collapse (High γ, Low Oe ). If reshuffling is too frequent (γ ≈ 1), the Broadcast factor P dominates tcomm , negating the benefits of disjoint aggregation. The system must operate with γ ≪ 1/tcomm . E.6
Model Falsifiability
The theoretical models proposed here make specific, falsifiable predictions that can be validated empirically. Prediction 1: Linearity of Broadcast Cost. We predict that for a fixed M , the broadcast volume during reshuffling e will follow the curve y = C · (1 − O M ). Prediction 2: Memory-Throughput Inverse Scaling. We predict that the maximum throughput (Tokens/Sec) scales super-linearly with 1/Oe due to the compounding effects of reduced payload size and increased local batch size Bl .
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
F
A DDITIONAL R ESULTS
Table 4: Evaluation for FoMoE with LocalAdamW and DiLoCo across different configurations of workers M and synchronization frequencies K. Table 5: LocalAdamW
Table 6: DiLoCo
K
k
M
32
64
128
M
32
64
128
2 4 8
105.03 ± 14.67 103.91 ± 14.43 106.30 ± 14.92
104.70 ± 14.80 104.47 ± 14.74 112.99 ± 15.71
106.91 ± 14.99 111.89 ± 15.57 159.71 ± 22.41
2 4 8
118.01 ± 16.06 137.27 ± 18.35 145.64 ± 19.69
117.94 ± 15.85 124.17 ± 16.43 133.37 ± 17.77
507.21 ± 74.84 420.49 ± 65.81 647.11 ± 94.47
Table 7: Evaluation for FoMoE with LocalAdamW and DiLoCo across different configurations of workers M , synchronization frequencies K, overlapping factors Oe and placement strategies. Oe Strategy
K
Placement Strategy
M
1
2
4
8
LocalAdamW
32
Fixed
2 4 8 2 4 8 2 4 8 2 4 8 2 4 8 2 4 8 2 4 8 2 4 8 2 4 8 2 4 8 2 4 8 2 4 8
111.45 ± 15.52 121.42 ± 16.55 135.74 ± 18.57 116.43 ± 16.06 124.47 ± 17.03 140.37 ± 19.07 112.38 ± 15.53 130.79 ± 17.67 141.48 ± 19.31 119.73 ± 16.79 129.74 ± 17.72 145.92 ± 19.68 120.71 ± 16.53 143.72 ± 19.07 181.07 ± 23.70 128.38 ± 17.88 148.62 ± 19.81 195.09 ± 26.19 131.69 ± 17.99 142.94 ± 18.82 168.39 ± 21.84 142.38 ± 18.67 157.49 ± 20.78 170.65 ± 22.94 144.15 ± 18.66 154.73 ± 20.29 187.66 ± 23.94 152.42 ± 19.27 155.17 ± 19.95 182.99 ± 23.34 289.86 ± 41.29 342.25 ± 47.39 531.75 ± 75.05 403.32 ± 59.70 282.21 ± 40.20 533.30 ± 71.11
105.03 ± 14.67 112.00 ± 15.45 122.49 ± 17.16 105.03 ± 14.67 113.65 ± 15.81 124.56 ± 17.32 104.70 ± 14.80 119.53 ± 16.34 132.39 ± 18.30 104.70 ± 14.80 114.88 ± 15.88 131.02 ± 18.31 106.91 ± 14.99 123.50 ± 16.91 156.84 ± 21.39 106.91 ± 14.99 125.36 ± 17.15 146.16 ± 19.65 118.01 ± 16.06 130.58 ± 17.58 149.95 ± 20.03 118.01 ± 16.06 136.01 ± 18.03 180.43 ± 24.33 117.94 ± 15.85 142.25 ± 18.41 158.91 ± 20.59 117.94 ± 15.85 136.46 ± 17.82 158.05 ± 20.29 507.21 ± 74.84 383.65 ± 57.42 384.71 ± 54.90 507.21 ± 74.84 430.88 ± 63.02 314.18 ± 44.39
103.91 ± 14.43 114.69 ± 16.08 103.91 ± 14.43 115.48 ± 15.91 104.47 ± 14.74 121.36 ± 17.02 104.47 ± 14.74 121.04 ± 16.54 111.89 ± 15.57 138.88 ± 19.34 111.89 ± 15.57 131.86 ± 18.16 137.27 ± 18.35 145.17 ± 19.42 137.27 ± 18.35 160.62 ± 21.63 124.17 ± 16.43 149.30 ± 19.85 124.17 ± 16.43 144.43 ± 19.23 420.49 ± 65.81 748.52 ± 105.02 420.49 ± 65.81 424.59 ± 65.96
106.91 ± 14.99 106.30 ± 14.92 112.99 ± 15.71 112.99 ± 15.71 159.71 ± 22.41 159.71 ± 22.41 145.64 ± 19.69 145.64 ± 19.69 133.37 ± 17.77 133.37 ± 17.77 647.11 ± 94.47 647.11 ± 94.47
Random
64
Fixed
Random
128
Fixed
Random
DiLoCo
32
Fixed
Random
64
Fixed
Random
128
Fixed
Random
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
G
T RAINING A LGORITHM Algorithm 2: FoMoE- Simplified View
Require: Model Partition and Groups 1: θ = {θdense , θsparse } — params partitioned into dense and sparse components 2: Wall = {1, . . . , M } — set of all workers Require: Hyper-parameters 3: θ0 — initial params 4: β1 , β2 ∈ [0, 1) — decay rates AdamW momenta 5: LocalOptimization — update model parameters locally with AdamW and skip tokens 6: OuterOpt — update aggregated params using an outer optimizer (e.g., Adam) Ensure: θT , uT −1 , v T −1 7: −1 M 8: for each worker m ∈ Wall : initialize θ0 , {(u−1 initialise random initial model and all local momenta. m , vm )}m=1 9: for t = 0, . . . , T − 1 do 10: 11: 12: 13: 14: 15: 16: 17: 18:
T synchronization rounds 1. Global Calculation (happens once) t S ← PlacementStrategy(θsparse , Oe , Placement)
generate allocation set S = {s0 , . . . , sM −1 }
2. Vectorized / Simultaneous Assignment t θm,sparse ← sm , ∀sm ∈ S t t θm,dense ← θdense , ∀m
simultaneous distribution of sparse subsets broadcast shared dense parameters
3. Model Compilation t t t θm ← {θm,dense , θm,sparse } for all workers m = 0, . . . , M − 1 in parallel do for k = 0, . . . K − 1 do
broadcast specialized models to workers
K local training steps t 19: LocalOptimization(θm ) 20: end for t M 21: ∆t+1 ←Ring All-Reduce({θm }m=1 ) 22: θt+1 ←OuterOpt(∆t+1 ) 23: end for 24: end for
The execution protocol proceeds in three phases per synchronization round (T ). First, the PlacementStrategy generates a global allocation set S, mapping expert subsets to workers according to the overlap factor Oe ; this step supports both static partitioning and dynamic reshuffling. Second, the system executes a vectorized assignment where dense parameters are broadcast globally, while sparse experts are routed exclusively to their assigned hosts. Finally, workers perform K steps of LocalOptimization in parallel. Crucially, this phase utilizes a ‘skip-token’ mechanism to bypass computations for non-resident experts, ensuring the computational savings of partial replication are realized. The round concludes with sparse aggregation, where gradients are synchronized only among the Oe replicas via Ring All-Reduce before the global state is updated.
R EFERENCES Abadi, M., Chu, A., Goodfellow, I. J., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In CCS, pp. 308–318. ACM, 2016. Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. QSGD: communication-efficient SGD via gradient
AdamW optimization with skip tokens implementation aggregate dense parameters globally and sparse parameters within expert replica groups perform outer optimization procedure
quantization and encoding. In NIPS, pp. 1709–1720, 2017. Athlur, S., Saran, N., Sivathanu, M., Ramjee, R., and Kwatra, N. Varuna: scalable, low-cost training of massive deep learning models. In EuroSys, pp. 472–487. ACM, 2022. Ben Allal, L., Lozhkov, A., Penedo, G., Wolf, T., and von Werra, L. SmolLM-Corpus, July 2024. URL https: //huggingface.co/datasets/HuggingFac eTB/smollm-corpus. Bonawitz, K. A., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., and Seth, K. Practical secure aggregation for privacypreserving machine learning. In CCS, pp. 1175–1191. ACM, 2017. Cagnasso, P., Belilovsky, E., and Oyallon, E. Unifying local communications and local updates for LLM pretraining. CoRR, abs/2606.11081, 2026. doi: 10.48550/arXiv.2606. 11081. URL https://arxiv.org/abs/2606.1 1081. Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
nets with sublinear memory cost, 2016. URL https: //arxiv.org/abs/1604.06174.
Szlam, A. Dipaco: Distributed path composition. CoRR, abs/2403.10616, 2024.
Cheng, Z. and Glasgow, M. Convergence of distributed adaptive optimization with local updates. In ICLR. OpenReview.net, 2025.
Douillard, A., Donchev, Y., Rush, K., Kale, S., Charles, Z., Garrett, Z., Teston, G., Lacey, D., McIlroy, R., Shen, J., Ramé, A., Szlam, A., Ranzato, M., and Barham, P. Streaming diloco with overlapping communication: Towards a distributed free lunch. CoRR, abs/2501.18512, 2025.
Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. CoRR, abs/2507.06261, 2025. doi: 10.48550/arXiv.2507.06261. URL https://arxiv.org/abs/2507.06261. Csordás, R., Irie, K., and Schmidhuber, J. Approximating two-layer feedforward networks for efficient transformers. In EMNLP (Findings), pp. 674–692. Association for Computational Linguistics, 2023. DeepSeek-AI. Deepseek-v3 technical report. CoRR, abs/2412.19437, 2024. URL https://arxiv.or g/abs/2412.19437. DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Deng, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Yang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Chen, J., Yuan, J., Qiu, J., Song, J., Dong, K., Gao, K., Guan, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Zhu, Q., Chen, Q., Du, Q., Chen, R. J., Jin, R. L., Ge, R., Pan, R., Xu, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Zheng, S., Wang, T., Pei, T., Yuan, T., Sun, T., Xiao, W. L., Zeng, W., An, W., Liu, W., Liang, W., Gao, W., Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Chen, X., Nie, X., Sun, X., Wang, Z., and et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, abs/2405.04434, 2024. Dey, N., Zhang, B. C., Noci, L., Li, M. B., Bordelon, B., Bergsma, S., Pehlevan, C., Hanin, B., and Hestness, J. Don’t be lazy: Completep enables compute-efficient deep transformers. CoRR, abs/2505.01618, 2025. Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., and Shen, J. Diloco: Distributed low-communication training of language models. CoRR, abs/2311.08105, 2023. doi: 10.48550/arXiv.2311.08105. URL https://arxiv.org/abs/2311.08105. Douillard, A., Feng, Q., Rusu, A. A., Kuncoro, A., Donchev, Y., Chhaparia, R., Gog, I., Ranzato, M., Shen, J., and
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M. P., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K. S., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., and Cui, C. Glam: Efficient scaling of language models with mixture-of-experts. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 5547– 5569. PMLR, 2022. Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1–120:39, 2022. Gale, T., Narayanan, D., Young, C., and Zaharia, M. Megablocks: Efficient sparse training with mixture-ofexperts. In MLSys. mlsys.org, 2023. Guo, B., Mei, Y., Xiao, D., and Wu, W. PFL-MoE: Personalized federated learning based on mixture of experts. In Web and Big Data, volume 12858 of Lecture Notes in Computer Science, pp. 480–486. Springer, 2021. doi: 10.1007/978-3-030-85896-4_37. URL https://doi. org/10.1007/978-3-030-85896-4_37. Guthrie, S. Inside the world’s most powerful ai datacenter, Sept 2025. URL https://blogs.microsoft.co m/blog/2025/09/18/inside-the-worlds-m ost-powerful-ai-datacenter/. He, J., Zhai, J., Antunes, T., Wang, H., Luo, F., Shi, S., and Li, Q. Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models. In PPoPP, pp. 120–134. ACM, 2022. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optimal large language models. CoRR, abs/2203.15556, 2022. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M. X., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., and Chen, Z. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In NeurIPS, pp. 103–112, 2019.
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Hwang, C., Cui, W., Xiong, Y., Yang, Z., Liu, Z., Hu, H., Wang, Z., Salas, R., Jose, J., Ram, P., Chau, J., Cheng, P., Yang, F., Yang, M., and Xiong, Y. Tutel: Adaptive mixture-of-experts at scale. In MLSys. mlsys.org, 2023. Iacob, A., Sani, L., Kurmanji, M., Shen, W. F., Qiu, X., Cai, D., Gao, Y., and Lane, N. D. DEPT: decoupled embeddings for pre-training language models. In ICLR. OpenReview.net, 2025a. Iacob, A., Sani, L., Safaryan, M., Giampouras, P., Horváth, S., Jovanovic, A., Kurmanji, M., Aleksandrov, P., Shen, W. F., Qiu, X., and Lane, N. D. DES-LOC: desynced low communication adaptive optimizers for training foundation models. CoRR, abs/2505.22549, 2025b. Jaghouar, S., Ong, J. M., and Hagemann, J. Opendiloco: An open-source framework for globally distributed lowcommunication training. CoRR, abs/2407.07852, 2024. Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K. A., Charles, Z., Cormode, G., Cummings, R., D’Oliveira, R. G. L., Eichner, H., Rouayheb, S. E., Evans, D., Gardner, J., Garrett, Z., Gascón, A., Ghazi, B., Gibbons, P. B., Gruteser, M., Harchaoui, Z., He, C., He, L., Huo, Z., Hutchinson, B., Hsu, J., Jaggi, M., Javidi, T., Joshi, G., Khodak, M., Konečný, J., Korolova, A., Koushanfar, F., Koyejo, S., Lepoint, T., Liu, Y., Mittal, P., Mohri, M., Nock, R., Özgür, A., Pagh, R., Qi, H., Ramage, D., Raskar, R., Raykova, M., Song, D., Song, W., Stich, S. U., Sun, Z., Suresh, A. T., Tramèr, F., Vepakomma, P., Wang, J., Xiong, L., Xu, Z., Yang, Q., Yu, F. X., Yu, H., and Zhao, S. Advances and open problems in federated learning. Found. Trends Mach. Learn., 14(1-2):1–210, 2021. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020.
Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., and Zettlemoyer, L. BASE layers: Simplifying training of large, sparse models. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 6265–6274. PMLR, 2021. Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020a. Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. In MLSys. mlsys.org, 2020b. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In ICLR (Poster). OpenReview.net, 2019. Lu, X., Liu, Q., Xu, Y., Zhou, A., Huang, S., Zhang, B., Yan, J., and Li, H. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6159–6172. Association for Computational Linguistics, 2024. doi: 10.18653/v1/ 2024.acl-long.334. URL https://aclanthology .org/2024.acl-long.334/. Ludziejewski, J., Krajewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygózdz, T., Sankowski, P., Cygan, M., and Jaszczur, S. Scaling laws for fine-grained mixture of experts. In ICML. OpenReview.net, 2024. Malasnicki, J., Ciebiera, K., Borun, M., Pióro, M., Ludziejewski, J., Stefaniak, M., Krutul, M., Jaszczur, S., Cygan, M., Adamczewski, K., and Krajewski, J. µ-parametrization for mixture of experts. CoRR, abs/2508.09752, 2025.
Khaled, A., Kale, S., Douillard, A., Jin, C., Fergus, R., and Zaheer, M. Understanding outer optimizers in local SGD: learning rates, momentum, and acceleration. CoRR, abs/2509.10439, 2025.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In AISTATS, volume 54 of Proceedings of Machine Learning Research, pp. 1273– 1282. PMLR, 2017.
Kim, J., Lee, B., Park, C., Oh, Y., Kim, B., Yoo, T., Shin, S., Han, D., Shin, J., and Yoo, K. M. Peri-ln: Revisiting normalization layer in the transformer architecture. In ICML. OpenReview.net, 2025.
Mei, H., Cai, D., Zhou, A., Wang, S., and Xu, M. Fedmoe: Personalized federated learning via heterogeneous mixture of experts. arXiv preprint arXiv:2408.11304, 2024.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. In ICLR. OpenReview.net, 2021.
Micikevicius, P., Narang, S., Alben, J., Diamos, G. F., Elsen, E., García, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., and Wu, H. Mixed precision training. In ICLR (Poster). OpenReview.net, 2018.
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., and Zaharia, M. Pipedream: generalized pipeline parallelism for DNN training. In SOSP, pp. 1–15. ACM, 2019. OpenAI. gpt-oss-120b & gpt-oss-20b model card. CoRR, abs/2508.10925, 2025. Palak, Reddy, T. R., Kataria, B., Gandhi, R., Tandon, K., Bhattacherjee, D., and Padmanabhan, V. N. Improving training time and GPU utilization in geo-distributed language model training. CoRR, abs/2411.14458, 2024. doi: 10.48550/arXiv.2411.14458. URL https: //arxiv.org/abs/2411.14458. Patarasuk, P. and Yuan, X. Bandwidth optimal all-reduce algorithms for clusters of workstations. J. Parallel Distributed Comput., 69(2):117–124, 2009. Qi, J., Zhu, W., Li, L., Wu, M., Wu, Y., He, W., Gao, X., Zeng, J., and Heinrich, M. Dilocox: A lowcommunication large-scale training framework for decentralized cluster. CoRR, abs/2506.21263, 2025. Rabenseifner, R. Optimization of collective reduction operations. In International Conference on Computational Science, volume 3036 of Lecture Notes in Computer Science, pp. 1–9. Springer, 2004. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (140):1–67, 2020. URL https://jmlr.org/pap ers/v21/20-074.html. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: memory optimizations toward training trillion parameter models. In SC, pp. 20. IEEE/ACM, 2020. Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 18332– 18346. PMLR, 2022. Reddi, S. J., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., and McMahan, H. B. Adaptive federated optimization. In ICLR. OpenReview.net, 2021. Reisser, M., Louizos, C., Gavves, E., and Welling, M. Federated mixture of experts. CoRR, abs/2107.06724, 2021. Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Pinto, A. S., Keysers, D., and Houlsby, N. Scaling vision with sparse mixture of experts. In NeurIPS, pp. 8583–8595, 2021.
Roller, S., Sukhbaatar, S., Szlam, A., and Weston, J. Hash layers for large sparse models. In NeurIPS, pp. 17555– 17566, 2021. Ryabinin, M. and Gusev, A. Towards crowdsourced training of large neural networks using decentralized mixtureof-experts. Advances in Neural Information Processing Systems, 33:3659–3672, 2020. doi: 10.48550/arXiv.2002. 04013. URL https://arxiv.org/abs/2002.0 4013. Ryabinin, M., Dettmers, T., Diskin, M., and Borzunov, A. SWARM parallelism: Training large models can be surprisingly communication-efficient. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 29416–29440. PMLR, 2023. URL https://proceedings.mlr.press/v202/r yabinin23a.html. Sani, L., Iacob, A., Cao, Z., Lee, R., Marino, B., Gao, Y., Zhao, W., Cai, D., Li, Z., Qiu, X., and Lane, N. D. Photon: Federated LLM pre-training. In Eighth Conference on Machine Learning and Systems, 2025. URL https: //openreview.net/forum?id=AQgYcfg5EI. Sarfi, A., Thérien, B., Lidin, J., and Belilovsky, E. Communication efficient LLM pre-training with SparseLoCo. CoRR, abs/2508.15706, 2025. doi: 10.48550/arXiv.2508. 15706. URL https://arxiv.org/abs/2508.1 5706. Sevilla, J. and Troynikov, A. Could decentralized training solve ai’s power problem?, 2025. URL https://ep och.ai/blog/could-decentralized-tra ining-solve-ais-power-problem. Accessed: 2025-12-11. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR (Poster). OpenReview.net, 2017. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multibillion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019. Stich, S. U. Local SGD converges fast and communicates little. In ICLR (Poster). OpenReview.net, 2019. Su, J., Ahmed, M. H. M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. Team, K., Bai, Y., Bao, Y., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y., Chen, Y., Chen, Y., Chen, Z., Cui, J., Ding, H., Dong, M., Du, A., Du, C., Du, D., Du, Y., Fan,
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Y., Feng, Y., Fu, K., Gao, B., Gao, H., Gao, P., Gao, T., Gu, X., Guan, L., Guo, H., Guo, J., Hu, H., Hao, X., He, T., He, W., He, W., Hong, C., Hu, Y., Hu, Z., Huang, W., Huang, Z., Huang, Z., Jiang, T., Jiang, Z., Jin, X., Kang, Y., Lai, G., Li, C., Li, F., Li, H., Li, M., Li, W., Li, Y., Li, Y., Li, Z., Li, Z., Lin, H., Lin, X., Lin, Z., Liu, C., Liu, C., Liu, H., Liu, J., Liu, J., Liu, L., Liu, S., Liu, T. Y., Liu, T., Liu, W., Liu, Y., Liu, Y., Liu, Y., Liu, Y., Liu, Z., Lu, E., Lu, L., Ma, S., Ma, X., Ma, Y., Mao, S., Mei, J., Men, X., Miao, Y., Pan, S., Peng, Y., Qin, R., Qu, B., Shang, Z., Shi, L., Shi, S., Song, F., Su, J., Su, Z., Sun, X., Sung, F., Tang, H., Tao, J., Teng, Q., Wang, C., Wang, D., Wang, F., Wang, H., Wang, J., Wang, J., Wang, J., Wang, S., Wang, S., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Z., Wang, Z., Wang, Z., Wei, C., Wei, Q., Wu, W., Wu, X., Wu, Y., Xiao, C., Xie, X., Xiong, W., Xu, B., Xu, J., Xu, J., Xu, L. H., Xu, L., Xu, S., Xu, W., Xu, X., Xu, Y., Xu, Z., Yan, J., Yan, Y., Yang, X., Yang, Y., Yang, Z., Yang, Z., Yang, Z., Yao, H., Yao, X., Ye, W., Ye, Z., Yin, B., Yu, L., Yuan, E., Yuan, H., Yuan, M., Zhan, H., Zhang, D., Zhang, H., Zhang, W., Zhang, X., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Z., Zhao, H., Zhao, Y., Zheng, H., Zheng, S., Zhou, J., Zhou, X., Zhou, Z., Zhu, Z., Zhuang, W., and Zu, X. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. Tran, V.-T., Khiem, L. H., and Pham, Q.-V. Revisiting sparse mixture of experts for resource-adaptive federated fine-tuning foundation models. In ICLR 2025 Workshop on Modularity for Collaborative, Decentralized, and Continual Deep Learning, 2025. URL https: //openreview.net/forum?id=IwNOUYgtuz. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS, pp. 5998–6008, 2017. Xie, L., Luan, T., Cai, W., Yan, G., Chen, Z., Xi, N., Fang, Y., Shen, Q., Wu, Z., and Yuan, J. dflmoe: Decentralized federated learning via mixture of experts for medical data analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10203–10213, 2025. Yang, G. and Hu, E. J. Tensor programs IV: feature learning in infinite-width neural networks. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 11727–11737. PMLR, 2021. Zec, E. L., Mogren, O., Martinsson, J., Sütfeld, L. R., and Gillblad, D. Specialized federated learning using a mixture of experts. CoRR, abs/2010.02056, 2020. doi: 10.48550/arXiv.2010.02056. URL https: //arxiv.org/abs/2010.02056. Zhai, M., He, J., Ma, Z., Zong, Z., Zhang, R., and Zhai, J. Smartmoe: Efficiently training sparsely-activated models
through combining offline and online parallelization. In USENIX ATC, pp. 961–975. USENIX Association, 2023. Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V. Y., Dai, A. M., Chen, Z., Le, Q. V., and Laudon, J. Mixtureof-experts with expert choice routing. In NeurIPS, 2022.