HeLoCo: Efficient asynchronous low-communication training under data and device heterogeneity

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

HeLoCo: Efficient asynchronous low-communication training under data and device heterogeneity

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cloud distributedcomputing parallelcomputing

distributed computing, parallel computing, cloud

HeLoCo: Efficient asynchronous low-communication training under data and device heterogeneity Abdullah Al Asif1 Patrick Diem2 Juan Pablo Muñoz3 Felix Wolf2 Ali Jannesari1 Arya Mazaheri2,4 1

arXiv:2606.00271v1 [cs.DC] 29 May 2026

Iowa State University Technical University of Darmstadt 3 Maro Systems 4 PanocularAI

Abstract Distributed Low-Communication (DiLoCo) training reduces communication overhead by allowing workers to perform multiple local optimization steps before sending pseudo-gradients to a global outer update. Its asynchronous variant further improves hardware utilization by removing synchronization barriers, but at the cost of stale pseudo-gradients computed from outdated model states. As a result, these updates can become misaligned with the current global optimization direction, particularly in heterogeneous systems. This issue becomes even more pronounced when data are non-IID, a setting that has not been well studied in asynchronous low-communication training. To address this limitation, we propose HeLoCo, a direction-aware correction method for asynchronous low-communication training that uses outer momentum as a reference for the current optimization trajectory and selectively adjusts incoming pseudo-gradients before the outer update. Updates that remain aligned are preserved, while directionally conflicting components are corrected. On multilingual language-model training with heterogeneous workers and non-IID data, HeLoCo consistently improves validation loss. It outperforms existing asynchronous DiLoCo-based baselines by up to 7.5% at a fixed token budget, exceeds asynchronous momentum look-ahead by up to 3.3% at a fixed wall-clock budget, and surpasses the synchronous baseline by up to 22.1% under severe system heterogeneity. Our analysis further shows how staleness, worker speed, and data heterogeneity shape update quality and convergence in highly decentralized and heterogeneous training setups.

Introduction

Training large language models (LLMs) typically relies on tightly coupled, high-performance clusters with high-bandwidth, low-latency interconnects. While effective within a single data center, this setup becomes challenging when compute resources are geographically distributed or heterogeneous. In such cases, communication slows down and synchronization can take up a lot of time [Douillard et al., 2023, 2025]. When multiple workers train together, some may end up waiting for others due to slow connections or imbalances in workload. To solve this issue, the Distributed Low Communication (DiLoCo) algorithm lets worker nodes perform several updates locally before sending their updates, called pseudo-gradients [Douillard et al., 2023]. By decoupling local and global updates, DiLoCo reduces communication requirements, while preserving training efficiency, making it particularly well suited to loosely connected clusters.[Douillard et al., 2023, Jaghouar et al., 2024, Douillard et al., 2025]. Preprint.

Synchronous DiLoCo requires all workers to send their updates before starting the next global step. When workers operate at different speeds or are distributed across regions, these differences can introduce stragglers, leading to delays and idle time. Switching to asynchronous variants allows each worker’s updates to be applied immediately. This improves both hardware efficiency and overall performance. However, a key challenge in this setting is gradient staleness, meaning that some gradient estimates are based on outdated model versions, which introduces inconsistency when they are consumed by the outer optimizer. Prior work [Liu et al., 2024] shows that this inconsistency can degrade convergence. As a result, although the asynchronous variant of DiLoCo permits more frequent updates than its synchronous counterpart, it can potentially yield worse optimization performance. Figure 1 illustrates the gap in synchronization between fast and slow workers. Fast workers contribute updates based on the most recent global model, ensuring high relevance. In contrast, slow workers (often called "stragglers") submit gradients derived from outdated model parameters. Because the global optimizer has already progressed, these "stale" updates can be counterproductive or even lead to training instability. Existing correction methods only partially address the problem. Traditional delayed-update methods apply an exponential discount to updates based on their age or delay compensation [Zinkevich et al., 2009, Zheng et al., 2017]. Recent asynchronous DiLoCo schemes introduce corrections using delayed Nesterov-like and momentum lookahead techniques [Liu et al., 2024, Ajanthan et al., 2025]. However, these methods only apply corrections to the entire pseudogradient, which proves to be too coarse for heterogeneous systems and non-IID (Independent and Identically Distributed) dataset distributions. Figure 1: Staleness asymmetry in asynchronous An update from a slower worker is not only more DiLoCo with heterogeneous workers. delayed but may also be less compatible with the current distribution utilized by the global trajectory [Li et al., 2020]. These two effects do not necessarily impact all parameter blocks of the pseudo-gradient equally; some may align with the current direction while others may contradict it, potentially hindering the convergence of the outer method. To address this issue, we propose HeLoCo (Heterogeneity-aware Low-Communication training), a direction-aware correction algorithm designed for asynchronous low-communication training. The core idea is to use outer momentum as a reference for determining the trustworthiness of each parameter block, where each block is an individual model tensor, such as a weight matrix. This approach allows processing of various stale gradient directions in alignment with the current outer update process. We further include a lightweight momentum-based [Zhang et al., 2019] initialization step to improve directional consistency. Our contributions are as follows: • We identify the directional mismatch as a key failure mode in asynchronous DiLoCo: under system heterogeneity and non-IID data, individual tensor blocks within a stale pseudogradient can conflict with the outer update direction. • We propose HeLoCo, a momentum-guided correction method that initializes workers with a look-ahead model and selectively keeps or adjusts the pseudo-gradient before the outer update. • Empirically, our method reduces validation loss by up to 7.5% over asynchronous Nesterov and 3.3% over asynchronous MLA at a fixed token budget, and by 22.1% over synchronous Nesterov under both system and data heterogeneity.

Background and Related Work

Stale pseudo-gradients in asynchronous distributed training have been studied from the perspective of delay compensation and momentum correction, but the question of whether different parts of the 2

same update deserve different treatment has received little attention. We review the three lines of work most relevant to this gap. Low-communication training for language models. DiLoCo showed that separating local SGD steps from a global outer optimizer dramatically cuts synchronization frequency without sacrificing model quality [Douillard et al., 2023]. OpenDiLoCo confirmed this recipe holds even across continents [Jaghouar et al., 2024]. Follow-up work pushed further: Streaming DiLoCo overlaps communication with computation to reduce peak bandwidth [Douillard et al., 2025], DiLoCoX scales to larger models via parallelism and compression [Qi et al., 2025], and NoLoCo [Kolehmainen et al., 2025] explores dropping global synchronization altogether. These works share a common focus on when and how often workers communicate. We ask a different question: given that some updates will inevitably arrive late, can we do better than applying them as-is? Asynchrony and stale-update correction. Stale gradients are as old as asynchronous optimization itself. Early work showed they can be tolerated in sparse settings [Recht et al., 2011], and parameterserver systems exposed their broader effects [Zinkevich et al., 2009, Dean et al., 2012]. A subtler insight came later: asynchronous execution does not just add noise, it implicitly introduces a momentum-like bias [Mitliagkas et al., 2016]. This makes the outer momentum a natural reference for diagnosing staleness. Taylor-style compensation was one early attempt to exploit this [Zheng et al., 2017]. More recently, MLA [Ajanthan et al., 2025] corrects staleness by extrapolating the negative momentum direction uniformly across the entire pseudo-gradient. We argue, however, that tensor blocks within the same stale update can have fundamentally different alignment with the outer trajectory, and a single uniform correction is too coarse to handle this. Heterogeneity, non-IID data, and directional conflict. The staleness problem is further compounded when workers operate at different speeds and train on non-IID data — conditions that are unavoidable in globally distributed training. In such heterogeneity, stale updates do not just arrive late; they carry gradients shaped by a different data distribution and a different point in the loss landscape [McMahan et al., 2017]. This directional distortion has been well documented in federated learning, where proximal regularization [Li et al., 2020], control variates [Karimireddy et al., 2020], and normalized averaging [Wang et al., 2020] were each designed to counter it. In multi-task optimization, PCGrad made a related observation: gradient conflicts are better resolved geometrically, block by block, than through a single global correction [Yu et al., 2020]. Our method brings this geometric intuition into asynchronous DiLoCo, using outer momentum as a reference to selectively preserve, dampen, or reorient each tensor block of a stale update — depending on how well it agrees with the direction the outer optimizer is already moving.

Methodology

Section 1 highlights the shortcomings of asynchronous training with low communication. The local update might be based on an old version of the global model, leading to issues in scenarios where there are heterogeneous worker update speeds and non-IID data. Despite this, delayed pseudo-gradients preserve some of the local gradient information; however, some directions in the tensor may not correspond with the latest global model update directions. In order to solve this issue, HeLoCo employs two strategies: one for initialization, allowing workers to start with a look-ahead model state, and another for alignment, ensuring returned gradients properly align with the current update direction. Problem setup. We consider distributed training over K workers. Worker i samples data from a local distribution Di , and the target objective is: min F (θ) = θ

K X

pi Fi (θ),

Fi (θ) = Ez∼Di [ℓ(θ; z)],

(1)

i=1

P where θ denotes the model parameters, pi ≥ 0, i pi = 1, and ℓ(θ; z) is the training loss. The weights pi define the intended contribution of each local objective to the global objective. The local distributions Di may be non-IID, and workers may have different computation or communication speeds. By contrast, in asynchronous training, update arrivals are driven by worker speed, so faster workers can influence the outer model more frequently than slower workers. 3

At outer step t, the global model maintains parameters θt and an outer momentum buffer mt . Here, outer refers to the global DiLoCo-level optimization step performed when a worker returns a pseudogradient, while inner refers to the local optimization steps performed independently on each worker. When worker i becomes available at outer step si , the synchronizer initializes it with an outer model state θ̄si . The worker initializes its local model from this state and performs H inner optimization steps: (s ) (si ) (s ) (s ) θi,0i = θ̄si , θi,h+1 = θi,hi − αh gi (θi,hi ), h = 0, . . . , H − 1, (2) where αh is the inner learning rate and gi (·) is a stochastic gradient on worker i’s local data. After local training, the worker returns the pseudo-gradient: (s )

(s )

∆i i = θ̄si − θi,Hi .

(3)

(s ) This sign convention makes ∆i i the descent displacement accumulated by local training, so the

synchronizer applies it through a negative outer step. If the update arrives at outer step t, its staleness is defined as: τi (t) = t − si .

(4)

(s ) Thus, ∆i i was computed from the initial model θ̄si , while the outer model now holds θt . This delay

is the source of the stale-update error that HeLoCo is designed to reduce. Momentum-guided worker initialization. The first use of outer momentum occurs before local training begins. Instead of initializing a worker with the current outer model θr at outer step r, HeLoCo initializes it with a look-ahead model: θ̄r = θr − ηr µmr , (5) where ηr is the outer learning rate at the time of initialization and µ is the outer momentum coefficient. This model is a one-step prediction of where the outer model is moving under its current momentum. Since the worker will return after some delay, starting local training from θ̄r reduces the positional gap between the worker’s initial model and the outer model state at arrival. For a worker initialized at step si , the returned pseudo-gradient is therefore: (s )

(s )

∆i i = (θsi − ηsi µmsi ) − θi,Hi .

(6)

This look-ahead initialization reduces the mismatch at the beginning of local training. However, it cannot guarantee that the returned pseudo-gradient will remain directionally useful after several outer updates. This motivates a second decision when the pseudo-gradient returns: which tensor blocks should be kept unchanged, and which should be corrected before the outer update? (s )

Measuring update alignment. When ∆i i arrives at outer step t, HeLoCo compares each tensor block of the update with the corresponding tensor block of the current outer models’ momentum mt . For readability, we write ∆b for a block of the arriving pseudo-gradient and (mt )b for the corresponding momentum block. All norms and inner products are computed after vectorizing each tensor block; ∥ · ∥ denotes the Euclidean norm, equivalently the Frobenius norm for tensor-shaped parameters. If both blocks have non-negligible norm, define: ∆b (mt )b ûb = , v̂b = . ∥∆b ∥ ∥(mt )b ∥ The directional compatibility score is defined as:

(7)

cb = û⊤ (8) b v̂b . A positive value indicates that the stale tensor still agrees with the current global model trajectory. A negative value indicates that the tensor contains a component opposing the momentum direction. A value near zero indicates that the tensor is not directly conflicting but is only weakly supported by the current trajectory. This tensor-wise view is the key design choice in HeLoCo. A stale pseudo-gradient is not treated as uniformly good or uniformly harmful. Within the same worker update, some tensors may remain aligned with the current global models’ trajectory, while others may conflict with it. A single scalar penalty on the entire pseudo-gradient cannot make this distinction. 4

Correcting stale directions. HeLoCo applies a geometric correction to each tensor block. If either ∥∆b ∥ or ∥(mt )b ∥ is below the numerical threshold ε, the momentum direction is unreliable for that block and the block is passed through unchanged. Let cok denote the threshold for sufficient alignment. If the block is sufficiently aligned, cb ≥ cok , it is also preserved: b b = ∆b . ∆

(9)

This leaves unchanged the tensor blocks that already agree with the current outer update direction. If cb < 0, the block contains an anti-momentum component. HeLoCo reduces only that conflicting component rather than discarding the entire tensor: b b = ∆b − βb cb ∥∆b ∥v̂b , ∆

(10)

where βb = min {ks (−cb )conf b , βmax } . (11) Here ks > 0 controls the shrinkage strength and βmax ∈ (0, 1] prevents overcorrection. Since cb < 0, the correction reduces the component that points against the current outer update direction. Geometrically, this can be interpreted as a partial projection-based correction along the normalized momentum direction v̂b . The incoming block ∆b can be decomposed into a parallel component ⊤ (∆⊤ b v̂b )v̂b and an orthogonal residual ∆b − (∆b v̂b )v̂b . HeLoCo attenuates only the anti-momentum part of the parallel component, while leaving the orthogonal residual unchanged so that local information not directly conflicting with the current trajectory is retained. For weakly aligned blocks, 0 ≤ cb < cok , HeLoCo does not remove a component. Instead, it gently reorients the block toward the current momentum direction while preserving its magnitude: ũb = (1 − λb )ûb + λb v̂b , ũb b b = ∥∆b ∥ , ∆ max{∥ũb ∥, ε}

(12) (13)

with λb = min {kd (1 − cb )conf b , 1} . (14) The coefficient kd > 0 controls the reorientation strength. The less aligned the block is, the more strongly it is pulled toward the momentum direction. In the non-degenerate case where ∥ũb ∥ ≥ ε, b b ∥ = ∥∆b ∥ and changes only the direction of the tensor. this operation preserves ∥∆ The confidence factor used in both correction cases is defined as: conf b =

∥∆b ∥ , ∥∆b ∥ + κ∥(mt )b ∥ + ε

(15)

where κ > 0 controls the relative scale of the momentum norm. This factor makes the correction conservative when the arriving block is small relative to the accumulated momentum and stronger when the arriving block is large. After all blocks are processed, the corrected pseudo-gradient is reassembled as: b = {∆ b b }b . ∆

(16)

Appendix A.2 provides a theoretical justification for the tensor-wise correction used by HeLoCo. Outer-model update. The outer update applies the corrected pseudo-gradient immediately. If the asynchronous training rule uses a worker or delay weight, which we denote by ρt ; otherwise, ρt = 1. The weighted corrected update is: b Gt = ρt ∆. (17) The global model then updates its momentum and parameters as mt+1 = µmt + (1 − µ)Gt , θt+1 = θt − ηt (Gt + µmt+1 ) .

(18) (19)

The updated momentum mt+1 is used in the parameter step, allowing the arriving pseudo-gradient to first update the outer direction estimate and then contribute to the model update. 5

The HeLoCo communication method works the same way as DiLoCo. Each worker processes a single model, updates it locally, and creates a single pseudo-gradient vector. However, the way they prepare for data exchange is different. Before local training starts, the synchronizer sets up the look-ahead model. After this, to prevent incorrect updates, the synchronizer modifies the gradients using momentum before applying the outer update procedure. This adjustment only needs to access the existing momentum buffer and requires an extra pass through the gradient vector, adding a cost of O(d) per arrival. Appendix A.1 summarizes the notation used in this section, and Appendix A.3 provides the HeLoCo pseudocode.

Experiments and Analyses

Setup. We evaluate HeLoCo using TinyGPT [Radford et al., 2019] on the multilingual mC4 [Raffel et al., 2020] dataset. Across experiments, we compare Nesterov, MLA, and HeLoCo under different worker-speed and data heterogeneity settings. Appendix A.4 provides implementation, hardware, and runtime details. 4.1

Convergence Under Heterogeneity

For the convergence experiments, we compare methods based on two key factors: how fast the workers are (either all the same speed or different speeds) and how the data is organized (either similar or different types). In the setting with different speeds, five workers take between 0.74 and 7.50 seconds for each step. This leads to an uneven update process: the fastest worker provides 58.2% of all updates and has an average delay of 0.72. Meanwhile, the two slowest workers together account for only 12% of the updates, with delays of 14.38 and 16.50, respectively. We also look at Dynamic Local Updates (DyLU) [Wang and Joshi, 2019], which helps balance how often workers provide updates by changing the number of local steps they take. With DyLU, contributions from workers become almost equal (19-22%), and the average staleness is approximately τ̄ ≈ 4. Crucially, DyLU is orthogonal to HeLoCo, as DyLU balances worker participation by changing the local-update schedule, while HeLoCo corrects returned pseudo-gradients before the outer update. Figure 2 compares training loss across IID and non-IID data settings under both heterogeneous and homogeneous worker configurations. When we use IID data on heterogeneous device 2(a), HeLoCo outperforms the other methods after about 6000 steps, achieving a final loss of 7.35. In comparison, MLA has a final loss of 7.41, and Nesterov has 7.89. Since the data are IID, this improvement mainly comes from the time delays introduced by workers with different speeds. In the homogeneous IID setting 2(b), the difference between HeLoCo and MLA decreases, as expected. When workers move at similar speeds, and delays are minimal, most parts of the data align well with the recent global model updates and remain unchanged during corrections. The non-IID results highlight the importance of correcting finer-grained components when both statistical heterogeneity and system heterogeneity are present. In the heterogeneous non-IID setting, HeLoCo shows a clear advantage, with a final loss of 7.91, compared to 7.96 for MLA and 8.29 for Nesterov. In this scenario, outdated gradients often mix different levels of alignment, so correcting at the global update level is not precise enough. In the homogeneous, non-IID setting, our method again outperforms MLA and Nesterov. This shows that having non-IID data alone can cause enough differences to make per-block corrections beneficial, even with low delays. The DyLU variants also provide a helpful comparison. DyLU lowers staleness variance by equalizing how often updates occur, but it also reduces local training time for some workers. In both IID and non-IID settings, DyLU variants do not consistently perform better than the corresponding non-DyLU HeLoCo runs. This suggests that, in our case, keeping local optimization strong and fixing stale updates at the outer optimizer is more effective than cutting local training time to reduce staleness. Non-IID data domain analysis. Figure 3 shows the results for different languages, where each language represents a distinct data domain with its own distribution, and how HeLoCo improves with staleness. Each language has a fixed-speed worker, keeping staleness the same during training. English dataset, trained by the most active worker (58.2% of training updates, average staleness 0.72), shows only a slight gain over MLA (7.91 vs. 7.93). This happens because most parameter blocks are aligned with the global model and need little correction. German and French, handled by slower workers (average staleness of 14.38 and 5.12, respectively), show larger improvements of 0.13 and 0.24, respectively. In these cases, stale gradients cause misalignment, and HeLoCo’s correction 6

(a) Heterogeneous (IID)

8.8 8.4 8.2

Loss

(b) Homogeneous (IID)

Nesterov-DyLU MLA-DyLU HeLoCo(Ours)-DyLU Nesterov MLA HeLoCo(Ours)

8.6

(d) Homogeneous (Non-IID)

Nesterov MLA HeLoCo(Ours)

8.0 7.8

Nesterov-DyLU MLA-DyLU HeLoCo(Ours)-DyLU Nesterov MLA HeLoCo(Ours)

7.6 7.4 7.2

2000

4000

6000

Step

8000

10000 2000

4000

6000

Step

8000

10000 2000

4000

6000

Step

8000

10000 2000

4000

6000

Step

8000

10000

Figure 2: Validation loss over training steps on C4 under IID and non-IID data distributions with heterogeneous and homogeneous worker speeds. DyLU variants are shown only for heterogeneous settings.

Figure 3: Evaluation loss comparison across five languages (de, en, es, fr, it) on multilingual C4 under non-IID training with five workers. focuses only on the misaligned parts, preserving useful information rather than discarding everything. The Async-DyLU row confirms this: when DyLU forces all workers to similar staleness (τ̄ ≈ 4), our approach still outperforms MLA, although by a smaller margin. This shows that per-block correction helps even when the staleness is the same. The synchronous row serves as a staleness-free baseline, where HeLoCo still outperforms MLA due to differences in the data. The slowest workers (contributing only 6.4% and 5.6% of total training updates) show fewer benefits from correction. This is expected, since infrequent updates mean the real issue shifts from misalignment to insufficient updates. These findings suggest that this method works best under moderate staleness: enough misalignment to benefit from correction, but enough updates for those corrections to matter. 4.2

Effect of worker staleness in non-IID setting

Building on Section 4.1, which evaluates a single fixed worker pace in the heterogeneous setting, this section examines how varying worker pace configurations affects optimization and loss dynamics under non-IID data distributions. Specifically, we analyze cases c) and d) of Figure 2 in greater depth to demonstrate HeLoCo’s robustness across a spectrum of heterogeneous worker configurations. We compare HeLoCo against three baselines: (i) synchronous DiLoCo with Nesterov, (ii) asynchronous DiLoCo with Nesterov, and (iii) asynchronous DiLoCo with MLA. Following the findings of Section 4.1, we exclude DyLu variants as they yield no performance gain in any configuration, and use synchronous Nesterov as the primary reference to assess HeLoCo and MLA in terms of loss progression over both steps and wall-clock time. Each worker is assigned a speed drawn from {1, 2, 6, 15} (seconds per step). We construct scenarios with progressively increasing staleness, culminating in the extreme configuration (1, 15, 15, 15, 15), 7

Loss

9 Worker paces

async-mla

async-nesterov

Worker paces

async-HeLoCo 1x

1x 15x

15 10 Steps

sync-nesterov

Worker paces

8 7 0

Loss

9 Worker paces

15 10 Steps 1x

20 2x

25 ×103

Worker paces

25 ×103

15 10 Steps

Worker paces

25 ×103

1x 15x 15x 15x

8 7 0

Loss

9 Worker paces

15 10 Steps 1x

20 1x

25 ×103

15 10 Steps

Worker paces

30 40 Time (s)

25 ×103

1x 15x

15 10 Steps

Worker paces

4 6 Time (s)

25 ×103

8 7 0

Loss

9 Worker paces

2 3 Time (s) 1x

5 ×103

Worker paces

50 6x

60 6x

70 ×103

Worker paces

30 40 Time (s)

10 ×103

1x 15x 15x 15x

8 7 0

4 6 Time (s)

10 ×103

15 Time (s)

30 ×103

70 ×103

Figure 4: Validation loss over steps (left) and wall-clock time (right) for different worker pace configurations in the non-IID setting. Worker-pace colors indicate staleness level: green for low staleness, yellow for moderate staleness, and red for high staleness.

in which one fast worker and four highly stale workers operate concurrently. Training runs for 300 outer steps of 80 inner steps each (24k total steps). Hyperparameters are listed in Appendix A.5 and match those from [Liu et al., 2024]. Figure 4 and Table 1 (left) show training loss over a fixed token budget across staleness configurations. HeLoCo consistently achieves lower loss than MLA in most settings. In the homogeneous case (1, 1, 1, 1, 1) and several other low-staleness configurations, HeLoCo also outperforms synchronous Nesterov, indicating that tensor-wise correction is beneficial even in the absence of staleness. As staleness grows, the performance gap between methods widens. Asynchronous Nesterov exhibits unstable behavior and diverges across all configurations. In high-staleness regimes, synchronous Nesterov surpasses both MLA and asynchronous Nesterov in token-budget loss; however, HeLoCo remains consistently more robust, achieving lower loss than MLA across nearly all configurations. The maximum token-budget improvement of HeLoCo over MLA is 3.3%. In extreme cases such as (1, 15, 15, 15, 15), synchronous Nesterov achieves substantially lower loss than both asynchronous methods at a fixed token budget. This picture changes when runtime is taken into account. Due to synchronization barriers, the wallclock time of synchronous training is bottlenecked by the slowest worker. As a result, asynchronous variants exhibit an advantage in loss-vs.-time performance across most configurations, as shown in Figure 4 and Table 1 (right). HeLoCo achieves the best trade-off between convergence speed and loss in all but two configurations; in the extreme staleness setting (1, 15, 15, 15, 15), it falls short of the synchronous baseline. At its best, HeLoCo’s time-budget loss is 22.1% lower than that of synchronous Nesterov. While HeLoCo typically improves over MLA, the margin is configuration-dependent: in moderate-staleness regimes the gains are consistent, whereas in the most extreme staleness cases 8

a slight degradation is observed, suggesting that adaptive mechanisms for managing highly stale updates remain an open direction. Table 1: Validation loss for a fixed token budget (left) and a fixed time budget (right) across worker pace configurations. Left: cell color indicates the best (green) and worst (red) method per row. Right: color indicates HeLoCo’s improvement (green) or degradation (red) relative to each baseline at matched wall-clock time. Legend: L-HeLoCo: HeLoCo loss; L-AMLA: async MLA loss; L-AN: async Nesterov loss; L-SN: sync Nesterov loss; ∆X: relative improvement of HeLoCo over X at 24k steps; T∆X: relative improvement of HeLoCo over X at time T . Worker pace

(1,6,6,6,6) (1,2,2,2,2) (1,1,6,6,6) (1,1,1,6,6) (1,1,2,2,2) (1,1,1,1,1) (1,15,15,15,15) (1,1,1,2,2) (1,1,1,1,6) (1,1,1,1,15) (1,1,1,1,2) (1,1,1,15,15) (1,1,15,15,15)

L-HeLoCo

6.98 6.54 6.69 6.66 6.64 6.50 7.64 6.64 6.60 6.66 6.61 6.84 7.36

L-AMLA

7.22 6.72 6.85 6.80 6.78 6.63 7.78 6.74 6.69 6.75 6.69 6.89 7.29

L-AN

7.20 7.04 7.29 7.15 7.07 7.03 7.81 7.06 7.10 7.13 7.05 7.21 7.47

L-SN

6.62 6.62 6.62 6.62 6.62 6.62 6.62 6.62 6.62 6.62 6.62 6.62 6.62

∆AMLA

∆AN

∆SN

T∆AMLA

T∆AN

(%)

(s)

(%)

3.33 2.62 2.45 2.03 1.99 1.84 1.75 1.48 1.36 1.30 1.23 0.75 -1.00

3.04 7.08 8.23 6.75 6.00 7.50 2.15 5.98 7.10 6.67 6.22 5.22 1.42

-5.43 1.22 -5.48 -0.62 -0.34 1.78 -15.39 -0.22 0.40 -0.54 0.16 -3.22 -11.17

14400 8000 9600 7200 6880 4800 19200 6080 5760 5920 5360 7680 10960

3.33 2.62 0.92 2.03 1.99 1.84 1.75 1.48 1.36 1.30 1.23 0.75 -1.00

3.04 7.08 3.05 6.75 6.00 7.50 2.15 5.98 7.10 6.67 6.22 5.22 1.42

5.80 3.04 4.68 13.04 3.13 1.78 -0.26 10.46 15.78 22.07 10.80 18.85 7.81

T∆SN

Conclusion

We presented HeLoCo, a correction method for asynchronous low-communication training that targets the core failure mode of gradient staleness: directional mismatch between stale pseudo-gradients and the current outer optimization trajectory. Rather than applying a single global correction, HeLoCo operates at the tensor-block level, using outer momentum as a trust reference to selectively preserve, attenuate, or interpolate each arriving update. The result is a method that remains aligned with the outer optimizer under both system and data heterogeneity, without discarding gradient information unnecessarily. Our experiments confirm that staleness is not a uniform problem: the benefit of correction scales with per-worker staleness and is most pronounced when system heterogeneity and non-IID data compound. In this regime, HeLoCo consistently outperforms asynchronous baselines at both fixed token and wall-clock time budgets, and often matches or exceeds synchronous training without its synchronization overhead. These results suggest a broader principle for asynchronous low-communicatoin distributed training, in which reducing communication frequency is not enough and the direction of each arriving update must also be actively managed. Limitations and future work. We conducted the evaluation using five workers and 15M-parameter models on subsets of C4 dataset. This controlled setting allows us to study HeLoCo under varying heterogeneous conditions, but it does not yet capture the scale of practical low-communication LLM training. Future work includes testing HeLoCo with more workers, larger models, longer training periods, and more diverse datasets. An interesting next step is to combine HeLoCo with decoupled DiLoCo-style systems [Douillard et al., 2026] to explore larger and more diverse deployments. Moreover, our results indicate that extreme staleness is still challenging asynchronous settings. As mentioned in Appendix A.6, highly stale pseudo-gradients may become unreliable, which suggests that discarding them may sometimes be preferable to correcting them. One promising idea is to merge HeLoCo with adaptive staleness-aware filtering, where we adjust, down-weight, or discard a returned pseudo-gradient based on how stale it is and its alignment statistics. Acknowledgements We gratefully acknowledge the support from Germany’s Federal Ministry of Breakthrough Innovation (SPRIN-D) through the Composite Learning Challenge under the SymphonyLearn project, and the hessian.AI Service Center (funded by the Federal Ministry of Research, Technology and Space, BMFTR, grant no. 16IS22091), the hessian.AI Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S-DIW04/0013/003). We would also like to thank the National Science Foundation (NSF) for supporting this project under grant 2211982 and 2426580. 9

This work utilized the Delta system at the National Center for Supercomputing Applications (NCSA) through allocation CIS240626. We also acknowledge support from the National Science Foundation under grants 2138259, 2138286, 2138307, 2137603, and 2138296.

References Thalaiyasingam Ajanthan, Sameera Ramasinghe, Gil Avraham, Yan Zuo, and Alexander Long. Momentum look-ahead for asynchronous distributed low-communication training. In ICLR 2025 Workshop on Modularity for Collaborative, Decentralized, and Continual Deep Learning, 2025. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, volume 25, 2012. Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed lowcommunication training of language models. arXiv, 2023. doi: 10.48550/arxiv.2311.08105. Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, et al. Streaming diloco with overlapping communication: Towards a distributed free lunch. arXiv preprint arXiv:2501.18512, 2025. Arthur Douillard, Keith Rush, Yani Donchev, Zachary Charles, Nova Fallen, Ayush Dubey, Ionel Gog, Josef Dean, Blake Woodworth, Zachary Garrett, et al. Decoupled diloco for resilient distributed pre-training. arXiv preprint arXiv:2604.21428, 2026. Sami Jaghouar, Jack Min Ong, and Johannes Hagemann. OpenDiLoCo: An open-source framework for globally distributed low-communication training. arXiv preprint arXiv:2407.07852, 2024. Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132–5143. PMLR, 2020. Jari Kolehmainen, Nikolay Blagoev, John Donaghy, Oğuzhan Ersoy, and Christopher Nies. NoLoCo: No-all-reduce low communication training method for large models. arXiv preprint arXiv:2506.10911, 2025. Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020. Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A Rusu, Jiajun Shen, Arthur Szlam, and Marc’Aurelio Ranzato. Asynchronous local-sgd training for language modeling. arXiv preprint arXiv:2401.09135, 2024. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1273–1282. PMLR, 20–22 Apr 2017. Ioannis Mitliagkas, Ce Zhang, Stefan Hadjis, and Christopher Ré. Asynchrony begets momentum, with an application to deep learning. In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 997–1004. IEEE, 2016. Ji Qi, WenPeng Zhu, Li Li, Ming Wu, YingJun Wu, Wu He, Xun Gao, Jason Zeng, and Michael Heinrich. DiLoCoX: A low-communication large-scale training framework for decentralized cluster. arXiv preprint arXiv:2506.21263, 2025. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 10

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URL http://jmlr.org/papers/v21/20-074.html. Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances in neural information processing systems, 24, 2011. Jianyu Wang and Gauri Joshi. Adaptive communication strategies to achieve the best error-runtime trade-off in local-update sgd. Proceedings of Machine Learning and Systems, 1:212–229, 2019. Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Advances in Neural Information Processing Systems, volume 33, pages 7611–7623, 2020. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, volume 33, pages 5824–5836, 2020. Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer: k steps forward, 1 step back. Advances in neural information processing systems, 32, 2019. Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie-Yan Liu. Asynchronous stochastic gradient descent with delay compensation. In International conference on machine learning, pages 4120–4129. PMLR, 2017. Martin Zinkevich, John Langford, and Alex J Smola. Slow learners are fast. In Advances in Neural Information Processing Systems, volume 22, pages 2331–2339, 2009.

Appendix

A.1

Notation

Table 2 summarizes the notation used in this paper Table 2: Symbols used in the methodology and appendix algorithm. Symbol

Description

Distributed training objective K i, j Di F (θ) Fi (θ) pi z ℓ(θ; z) θ d

Total number of workers participating in distributed training. Worker indices. Typically, i denotes the worker whose update arrives, while j denotes an available worker being initialized. Local data distribution of worker i. These distributions may be non-IID across workers. Global training objective. Local objective of worker i, defined over its local data distribution. P Nonnegative weight of worker i in the global objective, where K i=1 pi = 1. Training sample drawn from a local data distribution. Loss of model parameters θ on sample z. Generic model parameter vector. Total number of scalar model parameters, used in the overhead discussion.

Asynchronous training setup t r si τi (t) θt θr mt mr µ ηt ηr

Current server step at which a returned pseudo-gradient is processed. Server step at which an available worker is initialized. Initialization step of worker i, i.e., the server step when worker i receives its initial model. Staleness of worker i’s update at server step t, defined as τi (t) = t − si . Server model at server step t. Server model at initialization step r. Server momentum buffer at step t, representing the current server trajectory. Server momentum buffer at initialization step r. Outer momentum coefficient used by the server. Outer learning rate at server step t. Outer learning rate used when constructing the look-ahead model at initialization step r.

Local worker training θ̄si θ̄r H h (s ) θi,hi (si ) θi,0 (s ) θi,Hi αh gi (·) (s ) ∆i i

Initial model sent to worker i at step si . In HeLoCo, this is the look-ahead model. Look-ahead model used to initialize an available worker, defined as θ̄r = θr −ηr µmr . Number of local optimization steps performed by a worker before returning its pseudo-gradient. Index of a local optimization step, where h = 0, . . . , H − 1. Local model of worker i after h inner steps, initialized from server step si . Initial local model of worker i, equal to θ̄si . Final local model of worker i after H local steps. Inner learning rate used at local step h. Stochastic gradient computed on worker i’s local data. (s ) (s ) Worker pseudo-gradient, defined as ∆i i = θ̄si − θi,Hi .

Tensor-wise compatibility and correction b ∆b (mt )b

Index of a tensor block, such as a weight tensor or bias tensor. Tensor block b of the arriving pseudo-gradient. Tensor block b of the current server momentum. Continued on next page

Symbol

Description

∥·∥

Euclidean norm after vectorization; equivalent to the Frobenius norm for tensorshaped parameters. Unit direction of the pseudo-gradient block, defined as ûb = ∆b /∥∆b ∥. Unit direction of the momentum block, defined as v̂b = (mt )b /∥(mt )b ∥. Directional compatibility score between the pseudo-gradient block and the momentum block, defined as cb = û⊤ b v̂b . Threshold above which a tensor block is considered sufficiently aligned and is kept unchanged. Small numerical threshold used to avoid division by zero and to detect near-zero unreliable directions. Corrected version of tensor block ∆b . Full corrected pseudo-gradient after all tensor blocks are processed. Confidence factor controlling how strongly tensor block b should be corrected. Positive constant controlling how the momentum norm affects conf b . Positive coefficient controlling shrinkage strength when a tensor block conflicts with momentum. Tensor-wise shrinkage coefficient used when cb < 0. Upper bound on βb to prevent overcorrection. Positive coefficient controlling reorientation strength for weakly aligned tensor blocks. Tensor-wise interpolation coefficient used when 0 ≤ cb < cok . Interpolated direction between the pseudo-gradient direction and the momentum direction.

ûb v̂b cb cok ε bb ∆ b ∆ conf b κ ks βb βmax kd λb ũb Outer model update ρt Gt mt+1 θt+1

Optional arrival weight applied to the corrected pseudo-gradient. If unused, ρt = 1. b Weighted corrected update, defined as Gt = ρt ∆. Updated global momentum after incorporating the arriving corrected update. Updated global model after applying the outer update.

Local variables in the pseudocode (s )

Tensor block of the returned pseudo-gradient, u = (∆i i )b . Corresponding momentum block, v = (mt )b . Normalized update direction, û = u/∥u∥. Normalized momentum direction, v̂ = v/∥v∥. Direction obtained by blending û with v̂ for weakly aligned blocks.

u v û v̂ ũ

A.2

Theoretical Justification of the HeLoCo Correction

This appendix gives a local theoretical justification for the tensor-wise correction used by HeLoCo. The result is not a convergence proof for the full asynchronous training process. Instead, it shows that, for each non-degenerate tensor block, the correction does not reduce the signed component along the current outer momentum direction and does not increase the block norm. The correction is motivated by the same geometry used in gradient surgery methods such as PCGrad Yu et al. [2020]. PCGrad treats two gradients as conflicting when their cosine similarity is negative and removes the conflicting projection component. HeLoCo uses this idea in a different setting: it compares each stale pseudo-gradient block with the corresponding outer momentum block. Unlike PCGrad, HeLoCo applies a damped correction in the anti-aligned case and also adjusts weakly aligned blocks by blending their direction with the momentum direction. For the proof only, we use shorthand notation; these symbols correspond to the notation in Table 2. Lemma.

Fix a tensor block b. Let u = ∆b ,

v = (mt )b 13

denote the returned pseudo-gradient √ block and the corresponding outer momentum block. Assume ∥u∥, ∥v∥ > ε, βmax ≤ 1, and ε < 1/ 2. Define u v û = , v̂ = , c = û⊤ v̂. ∥u∥ ∥v∥ b b be the Here c is the cosine similarity between the returned block and the momentum block. Let ∆ corrected block produced by HeLoCo. Then b ⊤ v̂ ≥ u⊤ v̂, ∆ b

b b ∥ ≤ ∥u∥. ∥∆

Thus, HeLoCo does not reduce the signed component of the block along the outer momentum direction and does not amplify the returned block. Proof.

If c ≥ cok , HeLoCo leaves the block unchanged, so both claims hold with equality.

If c < 0, HeLoCo applies b b = u − βb c∥u∥v̂, ∆

0 ≤ βb ≤ 1.

⊤

Since u v̂ = c∥u∥, we have ⊤ ⊤ b⊤ ∆ b v̂ = (1 − βb )u v̂ ≥ u v̂,

because u⊤ v̂ < 0. Also, b b ∥2 = ∥u∥2 1 − βb (2 − βb )c2 ≤ ∥u∥2 . ∥∆ Therefore, counter-directed blocks become less opposed to the momentum direction without norm amplification. If 0 ≤ c < cok , HeLoCo forms ũ = (1 − λb )û + λb v̂,

0 ≤ λb ≤ 1.

Since 0 ≤ c ≤ 1, we have ∥ũ∥2 = 1 − 2λb (1 − λb )(1 − c) ≥

1 . 2

√ Because ε < 1/ 2, the numerical guard is inactive in this case, and b b = ∥u∥ ũ . ∆ ∥ũ∥ b b ∥ = ∥u∥. Let Thus ∥∆ c′ =

ũ⊤ v̂ ∥ũ∥

be the cosine similarity after correction. Then (1 − λb )c + λb c′ = p . (1 − λb )2 + λ2b + 2λb (1 − λb )c For 0 ≤ c ≤ 1 and 0 ≤ λb ≤ 1, the numerator is at least c and the denominator is at most 1, so c′ ≥ c. Therefore, ′ ⊤ b⊤ ∆ b v̂ = ∥u∥c ≥ ∥u∥c = u v̂. Hence weakly aligned blocks are moved toward the outer momentum direction without changing their norm. The lemma applies to non-degenerate blocks for which the cosine comparison is well defined. When ∥u∥ < ε or ∥v∥ < ε, HeLoCo skips the correction because the update or momentum direction is numerically unreliable. This completes the proof. Interpretation. The lemma shows that HeLoCo does not blindly suppress stale pseudo-gradients. Well-aligned blocks are preserved, anti-aligned blocks have their opposing component damped, and weakly aligned blocks are adjusted toward the momentum direction without increasing their magnitude. Since outer momentum summarizes recent outer updates, this local guarantee supports the use of momentum as the reference direction for correcting stale pseudo-gradient blocks. 14

A.3

HeLoCo Algorithm

Algorithm 1 Asynchronous DiLoCo with HeLoCo Input: Initial model θ(0) , outer momentum m0 =0, outer learning rate η, momentum coefficient µ, k workers, data shards {D1 , . . . , Dk } Output: Updated global model θt+1 and outer momentum mt+1 1: for outer step t = 1, 2, . . . do 2: if worker j is available at step r then 3: θ̄r ← θr − ηr µmr ▷ HeLoCo: look-ahead initialization 4: Send θ̄r to worker j; record sj ← r 5: end if 6: for inner step h = 1 . . . H do 7: x ∼ Dj (t) 8: L ← f (x, θj ) (t)

(t)

θj ← InnerOpt(θj , ∇L ) 10: end for (s ) 11: if worker i returns ∆i i at step t then 12: τi ← t − s i ▷ Measure staleness b t ← B LOCK C ORRECT(∆(si ) , mt , ρt ) ▷ HeLoCo: per-block correction (Algorithm 2) 13: G i bt 14: mt+1 ← µmt + (1 − µ)G b t + µmt+1 15: θt+1 ← θt − η G 16: end if 17: end for 9:

Algorithm 2 Tensor Directional Correction (s )

Input: Pseudo-gradient ∆i i , outer momentum mt , arrival weight ρt ; alignment threshold cok , correction constants ks , kd , κ, βmax , ε bt Output: Corrected update G (s ) i b ←∆ 1: ∆ i 2: for each tensor block b do (s ) 3: u ← (∆i i )b ; v ← (mt )b 4: if ∥u∥ < ε or ∥v∥ < ε then bb ← u 5: ∆ ▷ Skip: direction unreliable 6: else 7: û ← u/∥u∥; v̂ ← v/∥v∥ 8: cb ← û⊤ v̂ ▷ Cosine alignment with momentum 9: conf b ← ∥u∥/(∥u∥ + κ∥v∥ + ε) 10: if cb ≥ cok then bb ← u 11: ∆ ▷ Keep well-aligned blocks 12: else if cb < 0 then 13: βb ← min{ks (−cb ) conf b , βmax } b b ← u − βb cb ∥u∥v̂ 14: ∆ ▷ Shrink the anti-momentum component 15: else 16: λb ← min{kd (1 − cb ) conf b , 1} 17: ũ ← (1 − λb )û + λb v̂ b b ← ∥u∥ ũ/ max{∥ũ∥, ε} 18: ∆ ▷ Weakly aligned: rotate toward momentum 19: end if 20: end if 21: end for b t ← ρt ∆ b 22: G b 23: return Gt

This appendix gives the procedural form of HeLoCo. The pseudocode shows the two actions performed by the synchronizer during asynchronous training: initializing an available worker with the look-ahead outer model, and updating the outer model when a pseudo-gradient returns. Because workers run independently, their updates need not arrive in the same order in which they were initialized. The synchronizer therefore records the initialization step si for each worker and computes the staleness τi (t) = t − si when the update is processed. A.4

Implementation and Compute Details

All experiments were implemented in Python using PyTorch 2.10 and CUDA 12.6. We use a TinyGPT-style decoder-only Transformer with 15M parameters and the GPT-2 tokenizer. The main software dependencies are Transformers, NumPy, Pandas, and Matplotlib. Since different experiment groups were run on different GPU nodes, we report the hardware configuration separately for each group. For the experiments in Section 4.1, each run uses five workers. Each worker performs 20 local steps before returning a pseudo-gradient to the server, and training runs for 100 outer updates. These experiments were run on a node with 4 NVIDIA A100 GPUs, 128 GB memory, and 8 CPU cores. The five workers were run in parallel, with each complete run taking approximately 60 minutes. For the remaining results, we use a longer training setting with 80 local steps per worker update and 300 outer updates. To study different levels of system heterogeneity, we vary the worker paces, which directly changes the amount of staleness in the asynchronous update stream. These experiments were run on nodes with 8 NVIDIA H100 GPUs (80 GB memory each), 112 CPU cores and 2 TB RAM. A training run with 24k steps requires approximately 15 minutes of runtime for one experiment, while using 5 GPUs - one for each worker. Within each experimental group, all methods use the same model, data preprocessing, optimizer settings, and evaluation protocol. To save compute time, a full model evaluation on the evaluation dataset was omitted after each outer update during the middle of each training run. This was only performed at the beginning and at the end of the training runs, which is why gaps appear in the middle of the loss plots. A.5

Training Hyperparameters

The hyperparameters used for all experiments in Sections 4.2 and A.6 are summarized in table 3. Table 3: Run configuration for non-iid experiments. There√are different possibilities to manipulate weights. base means the weighting factor is calculated as k/k, where k is the number of workers. This calculation was taken from the original async DiLoCo paper and showed the best results in our case. average means a factor of 1/k is used. For the inner learning rate the identical cosine scheduler from [Liu et al., 2024] is used. Additional HeLoCo-specific parameters are: kd = 1.0, cok = 0.2, , κ = 3.0, βmax = 0.5, ks = 0.5 Method

Weight Factor

Outer learning rate

Momentum

async-HeLoCo async-mla async-nesterov sync-nesterov

base base base average

0.7 0.7 0.07 0.7

0.9 0.9 0.9 0.9

A.6

Deeper Insights into Non-IID Training

Fixed shard-to-worker setup In Section 4.2, we observed that HeLoCo achieves a lower overall loss than the other algorithms in the non-IID setting where shards are permanently assigned to workers. This conclusion was based on the average loss across all languages. In the following, we analyze in more detail how different levels of staleness affect the representation of individual shards (i.e., languages) within the global model. In Figure 5, we first analyze the behavior when introducing a single stale worker, starting from the homogeneous case and examining what happens when the staleness of one worker is increased. When a worker with pace 2 (seconds/iteration) is added, the loss of the fast worker (pace 1 second/iteration) changes only marginally, while the loss associated with worker 4 (the stale worker) increases immediately. As the staleness is increased further, this effect becomes more pronounced. The loss associated with the stale worker continues to deteriorate, and its corresponding language becomes increasingly underrepresented in the global model. This behavior remains stable for up to three stale workers with pace 2. Once the number of stale workers reaches four, or when three workers exhibit a staleness of 6 or higher, a different behavior emerges for both MLA and HeLoCo. The loss associated with the fast worker (worker 0) begins to increase again during training, while the losses of the slow workers continue to decrease. This suggests that beyond a certain level of staleness, the global model starts to forget information associated with the fast workers language in favor of the stale workers. In other words, the model appears to lose previously acquired knowledge for well-represented shards while adapting to underrepresented ones. Interestingly, this effect is not visible when considering only the mean validation loss across all languages, as shown in Figure 6. The degradation in performance for worker 0 is compensated by improvements for the stale workers, resulting in a relatively stable average loss. For future work, it may be interesting to investigate this behavior in more detail to better understand under which conditions this forgetting effect occurs and how it might be mitigated through appropriate training strategies. Flexible shard-to-worker setup In this section, we investigate a flexible shard-to-worker assignment strategy, in which each worker dynamically selects the shard that has been processed the least overall. This setup corresponds to the scenario used in the original async DiLoCo work of [Liu et al., 2024]. The key difference in our experiments is that the shards remain highly non-IID. Compared to the fixed shard-to-worker setup, Figure 7 shows that the discrepancy between fast and slow workers has a much smaller impact on the language-specific losses. The flexible assignment strategy leads to a more balanced representation of the different shards, because fast workers are no longer restricted to a single shard and can contribute to multiple languages over time. Finally, we investigate whether discarding stale gradients can improve the overall training performance. This means for example, that in the case (1, 1, 6, 6, 6) the pseudo-gradient of three stale workers is set to 0 in order to suppress their influence on the server model. Figure 8 shows that discarding gradients in scenarios with more than three stale workers and a staleness of 6 or greater consistently improves the performance of MLA-based algorithms, while it leads to degraded loss for asynchronous Nesterov. This suggests that neither MLA-based HeLoCo nor the pure MLA approach is able to effectively exploit highly stale updates in this setting, in contrast to asynchronous Nesterov. Rather than contributing useful information to training, these stale updates appear to impair the quality of the global model. Overall, this behavior indicates that there is still room for improvement, and that developing an adaptive mechanism to determine which workers provide beneficial updates and which do not could be a promising direction for future work.

async-mla_w0 async-mla_w4

Worker paces

async-nesterov_w0 async-nesterov_w4

Worker paces

1x 1x 1x 1x 1x

async-HeLoCo_w0

1x 1x 1x 1x 2x

async-HeLoCo_w4

sync-nesterov_w0

sync-nesterov_w4

Worker paces

1x 1x 1x 1x 6x

Worker paces

1x 1x 1x 1x 15x

Loss

6 0

15 10 Steps

25 ×103

15 10 Steps

25 ×103

2 3 Time (s)

5 ×103

4 6 Time (s)

10 ×103

15 10 Steps

25 ×103

30 ×103

15 10 Steps

25 ×103

70 ×103

Loss

8 7 6

Worker paces

1x 1x 1x 1x 1x

Worker paces

1x 1x 1x 2x 2x

15 Time (s)

Worker paces

1x 1x 1x 6x 6x

30 40 Time (s)

1x 1x 1x 15x 15x

Loss

6 0

15 10 Steps

25 ×103

15 10 Steps

25 ×103

2 3 Time (s)

5 ×103

4 6 Time (s)

10 ×103

15 10 Steps

25 ×103

30 ×103

15 10 Steps

25 ×103

70 ×103

Loss

8 7 6

Worker paces

1x 1x 1x 1x 1x

Worker paces

1x 1x 2x 2x 2x

15 Time (s)

Worker paces

1x 1x 6x 6x 6x

30 40 Time (s)

1x 1x 15x 15x 15x

Loss

6 0

15 10 Steps

25 ×103

15 10 Steps

25 ×103

2 3 Time (s)

5 ×103

4 6 Time (s)

10 ×103

15 10 Steps

25 ×103

30 ×103

15 10 Steps

25 ×103

70 ×103

Loss

8 7 6

Worker paces

1x 1x 1x 1x 1x

Worker paces

1x 2x 2x 2x 2x

15 Time (s)

Worker paces

1x 6x 6x 6x 6x

30 40 Time (s)

1x 15x 15x 15x 15x

Loss

6 0

15 10 Steps

25 ×103

15 10 Steps

25 ×103

2 3 Time (s)

5 ×103

4 6 Time (s)

10 ×103

15 10 Steps

25 ×103

30 ×103

15 10 Steps

25 ×103

70 ×103

Loss

8 7 6 5

15 Time (s)

30 40 Time (s)

Figure 5: Influence of the number and degree of stale workers on the validation loss for different languages in the server model (fixed setup). In each plot, worker 0 (lowest staleness) and worker 4 (highest staleness) are shown.

async-mla

9.0

Worker paces

async-mla: 6.63 async-nesterov: 7.03 async-HeLoCo: 6.50 sync-nesterov: 6.62

8.5 8.0

9.0

Worker paces

async-nesterov

9.0

async-mla: 6.69 async-nesterov: 7.05 async-HeLoCo: 6.61 sync-nesterov: 6.62

8.5 8.0 7.5

7.5

7.0

9.0

Worker paces

15 10 Steps 1x

20 1x

25 ×103

async-mla: 6.63 async-nesterov: 7.03 async-HeLoCo: 6.50 sync-nesterov: 6.62

8.5 8.0

6.5

9.0

Worker paces

15 10 Steps 1x

20 1x

25 ×103

8.0

6.5

9.0

async-mla: 6.69 async-nesterov: 7.05 async-HeLoCo: 6.61 sync-nesterov: 6.62

8.5

Worker paces

async-mla: 6.69 async-nesterov: 7.10 async-HeLoCo: 6.60 sync-nesterov: 6.62

8.0

7.0 0

sync-nesterov

8.5

7.5 6.5

async-HeLoCo

9.0

Worker paces

15 10 Steps 1x

20 1x

25 ×103

async-mla: 6.69 async-nesterov: 7.10 async-HeLoCo: 6.60 sync-nesterov: 6.62

8.5 8.0

6.5

9.0

7.5

7.0

6.5

9.0

Worker paces

async-mla: 6.63 async-nesterov: 7.03 async-HeLoCo: 6.50 sync-nesterov: 6.62

8.5 8.0

9.0

Worker paces

4 6 Time (s)

10 ×103

9.0

async-mla: 6.74 async-nesterov: 7.06 async-HeLoCo: 6.64 sync-nesterov: 6.62

8.5 8.0

Worker paces

15 Time (s)

30 ×103

async-mla: 6.80 async-nesterov: 7.15 async-HeLoCo: 6.66 sync-nesterov: 6.62

8.5 8.0

6.5

9.0

7.5

7.0

6.5

9.0

Worker paces

15 10 Steps 1x

20 1x

25 ×103

async-mla: 6.63 async-nesterov: 7.03 async-HeLoCo: 6.50 sync-nesterov: 6.62

8.5 8.0

9.0

Worker paces

15 10 Steps 1x

20 2x

25 ×103

9.0

async-mla: 6.74 async-nesterov: 7.06 async-HeLoCo: 6.64 sync-nesterov: 6.62

8.5 8.0 7.5

7.5

7.0

9.0

Worker paces

2 3 Time (s)

5 ×103

async-mla: 6.63 async-nesterov: 7.03 async-HeLoCo: 6.50 sync-nesterov: 6.62

8.5 8.0

6.5

9.0

Worker paces

4 6 Time (s)

10 ×103

8.0

6.5

9.0

async-mla: 6.78 async-nesterov: 7.07 async-HeLoCo: 6.64 sync-nesterov: 6.62

8.5

20 6x

25 ×103

async-mla: 6.80 async-nesterov: 7.15 async-HeLoCo: 6.66 sync-nesterov: 6.62

8.0

7.0 0

Worker paces

15 10 Steps

8.5

7.5 6.5

6.5

9.0

Worker paces

15 Time (s)

30 ×103

async-mla: 7.13 async-nesterov: 7.29 async-HeLoCo: 7.06 sync-nesterov: 6.62

8.5 8.0

6.5

9.0

7.5 7.0

6.5

9.0

Worker paces

async-mla: 6.63 async-nesterov: 7.03 async-HeLoCo: 6.50 sync-nesterov: 6.62

8.5 8.0

9.0

Worker paces

15 10 Steps 1x

20 2x

25 ×103

9.0

async-mla: 6.78 async-nesterov: 7.07 async-HeLoCo: 6.64 sync-nesterov: 6.62

8.5 8.0

Worker paces

15 10 Steps 1x

20 6x

25 ×103

async-mla: 7.13 async-nesterov: 7.29 async-HeLoCo: 7.06 sync-nesterov: 6.62

8.5 8.0

6.5

9.0

7.5

7.0

6.5

9.0

Worker paces

2 3 Time (s)

5 ×103

async-mla: 6.63 async-nesterov: 7.03 async-HeLoCo: 6.50 sync-nesterov: 6.62

8.5 8.0

9.0

Worker paces

4 6 Time (s)

10 ×103

9.0

async-mla: 6.72 async-nesterov: 7.04 async-HeLoCo: 6.54 sync-nesterov: 6.62

8.5 8.0 7.5

7.5

7.0

6.5 0

9.0

Worker paces

15 10 Steps 1x

20 1x

25 ×103

async-mla: 6.63 async-nesterov: 7.03 async-HeLoCo: 6.50 sync-nesterov: 6.62

8.5 8.0

0 9.0

Worker paces

15 10 Steps 1x

20 2x

25 ×103

8.0

6.5

9.0

async-mla: 6.72 async-nesterov: 7.04 async-HeLoCo: 6.54 sync-nesterov: 6.62

8.5

Worker paces

15 Time (s)

30 ×103

async-mla: 7.22 async-nesterov: 7.20 async-HeLoCo: 6.98 sync-nesterov: 6.62

8.0

7.0 6.5

8.5

7.5

6.5

9.0

Worker paces

15 10 Steps 1x

20 6x

25 ×103

async-mla: 7.22 async-nesterov: 7.20 async-HeLoCo: 6.98 sync-nesterov: 6.62

8.5 8.0

6.5

9.0

7.5 7.0

6.5

4 6 Time (s)

10 ×103

async-mla: 6.89 async-nesterov: 7.21 async-HeLoCo: 6.84 sync-nesterov: 6.62

15 10 Steps

Worker paces

25 ×103

1x 15x 15x async-mla: 6.89 async-nesterov: 7.21 async-HeLoCo: 6.84 sync-nesterov: 6.62

Worker paces

30 40 Time (s)

70 ×103

1x 15x 15x 15x async-mla: 7.29 async-nesterov: 7.47 async-HeLoCo: 7.36 sync-nesterov: 6.62

15 10 Steps

Worker paces

25 ×103

1x 15x 15x 15x async-mla: 7.29 async-nesterov: 7.47 async-HeLoCo: 7.36 sync-nesterov: 6.62

Worker paces

30 40 Time (s)

70 ×103

1x 15x 15x 15x 15x async-mla: 7.78 async-nesterov: 7.81 async-HeLoCo: 7.64 sync-nesterov: 6.62

15 Time (s)

30 ×103

15 10 Steps

Worker paces

6.5

25 ×103

1x 15x 15x 15x 15x async-mla: 7.78 async-nesterov: 7.81 async-HeLoCo: 7.64 sync-nesterov: 6.62

8.0

7.0 0

70 ×103

1x 15x 15x

8.5

7.0 5 ×103

7.0 5

7.0 4

7.5

2 3 Time (s)

8.0

7.5

Worker paces

30 40 Time (s)

8.5

7.5

8.0

7.5

8.5

7.5

8.0

7.0 0

1x 15x async-mla: 6.75 async-nesterov: 7.13 async-HeLoCo: 6.66 sync-nesterov: 6.62

8.5

7.0 25 ×103

25 ×103

7.0 0

7.0 20

7.5

15 10 Steps

8.0

7.5

8.5

7.5

Worker paces

8.0

7.5

15 10 Steps

8.5

7.5

8.0

7.0 5 ×103

1x 15x async-mla: 6.75 async-nesterov: 7.13 async-HeLoCo: 6.66 sync-nesterov: 6.62

8.5

7.0 4

7.0 5

7.5

2 3 Time (s)

7.5

8.0

7.5

Worker paces

8.5

30 40 Time (s)

70 ×103

Figure 6: Influence of the number and degree of stale workers on the mean validation loss of the server model.

async-mla_w0 async-mla_w4

Worker paces

async-nesterov_w0 async-nesterov_w4

Worker paces

1x 1x 1x 1x 1x

async-HeLoCo_w0

1x 1x 1x 1x 2x

async-HeLoCo_w4

sync-nesterov_w0

sync-nesterov_w4

Worker paces

1x 1x 1x 1x 6x

Worker paces

1x 1x 1x 1x 15x

Loss

6 0

15 10 Steps

25 ×103

15 10 Steps

25 ×103

2 3 Time (s)

5 ×103

4 6 Time (s)

10 ×103

15 10 Steps

25 ×103

30 ×103

15 10 Steps

25 ×103

70 ×103

Loss

8 7 6

Worker paces

1x 1x 1x 1x 1x

Worker paces

1x 1x 1x 2x 2x

15 Time (s)

Worker paces

1x 1x 1x 6x 6x

30 40 Time (s)

1x 1x 1x 15x 15x

Loss

6 0

15 10 Steps

25 ×103

15 10 Steps

25 ×103

2 3 Time (s)

5 ×103

4 6 Time (s)

10 ×103

15 10 Steps

25 ×103

30 ×103

15 10 Steps

25 ×103

70 ×103

Loss

8 7 6

Worker paces

1x 1x 1x 1x 1x

Worker paces

1x 1x 2x 2x 2x

15 Time (s)

Worker paces

1x 1x 6x 6x 6x

30 40 Time (s)

1x 1x 15x 15x 15x

Loss

6 0

15 10 Steps

25 ×103

15 10 Steps

25 ×103

2 3 Time (s)

5 ×103

4 6 Time (s)

10 ×103

15 10 Steps

25 ×103

30 ×103

15 10 Steps

25 ×103

70 ×103

Loss

8 7 6

Worker paces

1x 1x 1x 1x 1x

Worker paces

1x 2x 2x 2x 2x

15 Time (s)

Worker paces

1x 6x 6x 6x 6x

30 40 Time (s)

1x 15x 15x 15x 15x

Loss

6 0

15 10 Steps

25 ×103

15 10 Steps

25 ×103

2 3 Time (s)

5 ×103

4 6 Time (s)

10 ×103

15 10 Steps

25 ×103

30 ×103

15 10 Steps

25 ×103

70 ×103

Loss

8 7 6 5

15 Time (s)

30 40 Time (s)

Figure 7: Influence of the number and degree of stale workers on the validation loss for different languages in the server model (flexible setup). In each plot, worker 0 (lowest staleness) and worker 4 (highest staleness) are shown.

async-mla async-mla_drop

9.0

Worker paces

1x 1x 1x 1x 1x

8.0 7.5

async-nesterov_drop

1x 1x 1x 1x 2x

async-HeLoCo: 6.51 async-HeLoCo_drop: 6.51 sync-nesterov: 6.62 async-mla: 6.63 async-mla_drop: 6.63 async-nesterov: 7.03 async-nesterov_drop: 7.03

8.5 Loss

async-nesterov

async-HeLoCo

async-HeLoCo_drop

Worker paces

1x 1x 1x 1x 6x

sync-nesterov: 6.62 async-HeLoCo_drop: 6.66 async-mla: 6.71 async-HeLoCo: 6.75 async-mla_drop: 6.78 async-nesterov: 7.05 async-nesterov_drop: 7.12

sync-nesterov

Worker paces

1x 1x 1x 1x 15x

async-HeLoCo_drop: 6.51 async-HeLoCo: 6.56 sync-nesterov: 6.62 async-mla_drop: 6.66 async-mla: 6.71 async-nesterov_drop: 7.06 async-nesterov: 7.07

async-HeLoCo_drop: 6.58 async-HeLoCo: 6.60 sync-nesterov: 6.62 async-mla_drop: 6.75 async-mla: 6.75 async-nesterov: 7.04 async-nesterov_drop: 7.04

7.0 6.5 0 9.0

Worker paces

15 10 Steps

25 ×103

Worker paces

15 10 Steps

8.0 7.5

25 ×103

Worker paces

1x 1x 1x 1x 2x

async-HeLoCo: 6.51 async-HeLoCo_drop: 6.51 sync-nesterov: 6.62 async-mla: 6.63 async-mla_drop: 6.63 async-nesterov: 7.03 async-nesterov_drop: 7.03

8.5 Loss

1x 1x 1x 1x 1x

15 10 Steps

25 ×103

sync-nesterov: 6.62 async-HeLoCo_drop: 6.66 async-mla: 6.71 async-HeLoCo: 6.75 async-mla_drop: 6.78 async-nesterov: 7.05 async-nesterov_drop: 7.12

15 10 Steps

Worker paces

1x 1x 1x 1x 6x

25 ×103

1x 1x 1x 1x 15x

async-HeLoCo_drop: 6.51 async-HeLoCo: 6.56 sync-nesterov: 6.62 async-mla_drop: 6.66 async-mla: 6.71 async-nesterov_drop: 7.06 async-nesterov: 7.07

async-HeLoCo_drop: 6.58 async-HeLoCo: 6.60 sync-nesterov: 6.62 async-mla_drop: 6.75 async-mla: 6.75 async-nesterov: 7.04 async-nesterov_drop: 7.04

7.0 6.5 0 9.0

Worker paces

2 3 Time (s)

5 ×103

Worker paces

1x 1x 1x 1x 1x

4 6 Time (s)

8.0 7.5

10 ×103

Worker paces

1x 1x 1x 2x 2x

async-HeLoCo: 6.51 async-HeLoCo_drop: 6.51 sync-nesterov: 6.62 async-mla: 6.63 async-mla_drop: 6.63 async-nesterov: 7.03 async-nesterov_drop: 7.03

8.5 Loss

15 Time (s)

30 ×103

Worker paces

1x 1x 1x 6x 6x

sync-nesterov: 6.62 async-HeLoCo: 6.63 async-HeLoCo_drop: 6.69 async-mla: 6.69 async-mla_drop: 6.85 async-nesterov: 7.05 async-nesterov_drop: 7.17

30 40 Time (s)

70 ×103

1x 1x 1x 15x 15x

async-HeLoCo: 6.55 async-HeLoCo_drop: 6.56 sync-nesterov: 6.62 async-mla: 6.66 async-mla_drop: 6.70 async-nesterov: 7.06 async-nesterov_drop: 7.07

async-HeLoCo: 6.52 async-HeLoCo_drop: 6.52 sync-nesterov: 6.62 async-mla: 6.67 async-mla_drop: 6.68 async-nesterov: 7.04 async-nesterov_drop: 7.05

7.0 6.5 0 9.0

Worker paces

15 10 Steps

25 ×103

Worker paces

15 10 Steps

8.0 7.5

25 ×103

Worker paces

1x 1x 1x 2x 2x

async-HeLoCo: 6.51 async-HeLoCo_drop: 6.51 sync-nesterov: 6.62 async-mla: 6.63 async-mla_drop: 6.63 async-nesterov: 7.03 async-nesterov_drop: 7.03

8.5 Loss

1x 1x 1x 1x 1x

15 10 Steps

25 ×103

sync-nesterov: 6.62 async-HeLoCo: 6.63 async-HeLoCo_drop: 6.69 async-mla: 6.69 async-mla_drop: 6.85 async-nesterov: 7.05 async-nesterov_drop: 7.17

15 10 Steps

Worker paces

1x 1x 1x 6x 6x

25 ×103

1x 1x 1x 15x 15x

async-HeLoCo: 6.55 async-HeLoCo_drop: 6.56 sync-nesterov: 6.62 async-mla: 6.66 async-mla_drop: 6.70 async-nesterov: 7.06 async-nesterov_drop: 7.07

async-HeLoCo: 6.52 async-HeLoCo_drop: 6.52 sync-nesterov: 6.62 async-mla: 6.67 async-mla_drop: 6.68 async-nesterov: 7.04 async-nesterov_drop: 7.05

7.0 6.5 0 9.0

Worker paces

2 3 Time (s)

5 ×103

Worker paces

1x 1x 1x 1x 1x

4 6 Time (s)

8.0 7.5

10 ×103

Worker paces

1x 1x 2x 2x 2x

async-HeLoCo: 6.51 async-HeLoCo_drop: 6.51 sync-nesterov: 6.62 async-mla: 6.63 async-mla_drop: 6.63 async-nesterov: 7.03 async-nesterov_drop: 7.03

8.5 Loss

15 Time (s)

30 ×103

Worker paces

1x 1x 6x 6x 6x

async-HeLoCo: 6.54 sync-nesterov: 6.62 async-mla: 6.71 async-HeLoCo_drop: 6.83 async-mla_drop: 7.00 async-nesterov: 7.06 async-nesterov_drop: 7.35

30 40 Time (s)

70 ×103

1x 1x 15x 15x 15x

async-HeLoCo_drop: 6.57 sync-nesterov: 6.62 async-mla_drop: 6.74 async-HeLoCo: 6.86 async-nesterov: 7.06 async-mla: 7.06 async-nesterov_drop: 7.13

async-HeLoCo_drop: 6.47 async-HeLoCo: 6.55 sync-nesterov: 6.62 async-mla_drop: 6.62 async-mla: 6.76 async-nesterov_drop: 7.05 async-nesterov: 7.06

7.0 6.5 0 9.0

Worker paces

15 10 Steps

25 ×103

Worker paces

15 10 Steps

8.0 7.5

25 ×103

Worker paces

1x 1x 2x 2x 2x

async-HeLoCo: 6.51 async-HeLoCo_drop: 6.51 sync-nesterov: 6.62 async-mla: 6.63 async-mla_drop: 6.63 async-nesterov: 7.03 async-nesterov_drop: 7.03

8.5 Loss

1x 1x 1x 1x 1x

15 10 Steps

25 ×103

async-HeLoCo: 6.54 sync-nesterov: 6.62 async-mla: 6.71 async-HeLoCo_drop: 6.83 async-mla_drop: 7.00 async-nesterov: 7.06 async-nesterov_drop: 7.35

15 10 Steps

Worker paces

1x 1x 6x 6x 6x

25 ×103

1x 1x 15x 15x 15x

async-HeLoCo_drop: 6.57 sync-nesterov: 6.62 async-mla_drop: 6.74 async-HeLoCo: 6.86 async-nesterov: 7.06 async-mla: 7.06 async-nesterov_drop: 7.13

async-HeLoCo_drop: 6.47 async-HeLoCo: 6.55 sync-nesterov: 6.62 async-mla_drop: 6.62 async-mla: 6.76 async-nesterov_drop: 7.05 async-nesterov: 7.06

7.0 6.5 0 9.0

Worker paces

2 3 Time (s)

5 ×103

Worker paces

4 6 Time (s)

8.0 7.5

10 ×103

Worker paces

1x 2x 2x 2x 2x

async-HeLoCo: 6.51 async-HeLoCo_drop: 6.51 sync-nesterov: 6.62 async-mla: 6.63 async-mla_drop: 6.63 async-nesterov: 7.03 async-nesterov_drop: 7.03

8.5 Loss

1x 1x 1x 1x 1x

15 Time (s)

30 ×103

sync-nesterov: 6.62 async-HeLoCo: 6.74 async-mla: 6.86 async-nesterov: 7.04 async-HeLoCo_drop: 7.27 async-mla_drop: 7.35 async-nesterov_drop: 7.63

Worker paces

1x 6x 6x 6x 6x

30 40 Time (s)

70 ×103

1x 15x 15x 15x 15x

sync-nesterov: 6.62 async-HeLoCo_drop: 6.72 async-mla_drop: 6.90 async-nesterov: 7.06 async-nesterov_drop: 7.28 async-mla: 7.39 async-HeLoCo: 7.40

async-HeLoCo_drop: 6.51 async-HeLoCo: 6.56 sync-nesterov: 6.62 async-mla_drop: 6.70 async-mla: 6.71 async-nesterov: 7.08 async-nesterov_drop: 7.14

7.0 6.5 0 9.0

Worker paces

15 10 Steps

25 ×103

Worker paces

1x 1x 1x 1x 1x

15 10 Steps

8.0 7.5

25 ×103

Worker paces

1x 2x 2x 2x 2x

async-HeLoCo: 6.51 async-HeLoCo_drop: 6.51 sync-nesterov: 6.62 async-mla: 6.63 async-mla_drop: 6.63 async-nesterov: 7.03 async-nesterov_drop: 7.03

8.5 Loss

15 10 Steps

25 ×103

15 10 Steps

Worker paces

1x 6x 6x 6x 6x

sync-nesterov: 6.62 async-HeLoCo: 6.74 async-mla: 6.86 async-nesterov: 7.04 async-HeLoCo_drop: 7.27 async-mla_drop: 7.35 async-nesterov_drop: 7.63

25 ×103

1x 15x 15x 15x 15x

sync-nesterov: 6.62 async-HeLoCo_drop: 6.72 async-mla_drop: 6.90 async-nesterov: 7.06 async-nesterov_drop: 7.28 async-mla: 7.39 async-HeLoCo: 7.40

async-HeLoCo_drop: 6.51 async-HeLoCo: 6.56 sync-nesterov: 6.62 async-mla_drop: 6.70 async-mla: 6.71 async-nesterov: 7.08 async-nesterov_drop: 7.14

7.0 6.5 0

2 3 Time (s)

5 ×103

4 6 Time (s)

10 ×103

15 Time (s)

30 ×103

30 40 Time (s)

70 ×103

Figure 8: Influence of dropping pseudo-gradients from stale workers on the mean validation loss over all languages.

Record · ID 246534 · SHA-256 5e5864efe5d22112
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.