arXiv CS open access

FRESCO: A Novel Consistency Control for Asynchronous Pipeline Parallel Training

Dongyeop Lee et al. · arxiv_cs

arXiv CS · Papers

cloud distributed-computing parallel-computing

Asynchronous pipeline parallelism can accelerate distributed model training, but suffers from instability caused by inconsistent gradient computations. We identify a fundamental source of this inconsistency: error signals are coupled to forward-time features, not just parameter states. This means th

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

F RESCO: A Novel Consistency Control for Asynchronous Pipeline Parallel Training

arXiv:2611.11238v1 [cs.LG] 8 Nov 2026

Dongyeop Lee POSTECH dongyeop.lee2@postech.ac.kr

Namhoon Lee POSTECH namhoon.lee@postech.ac.kr

Abstract Asynchronous pipeline parallelism can accelerate distributed model training, but suffers from instability caused by inconsistent gradient computations. We identify a fundamental source of this inconsistency: error signals are coupled to forward-time features, not just parameter states. This means that parameters may evolve as long as they preserve forward-equivalence. Leveraging this insight, we introduce F RESCO, which frames asynchronous updates as a constrained optimization problem: it finds the minimal update modification that acts trivially on the activation subspace. We demonstrate that F RESCO consistently outperforms representative asynchronous baselines in LLM pre-training, with the gains most pronounced under deep pipelines and high inconsistency levels: at pipeline depth 32, it reaches the same loss as the strongest baseline with 43% fewer iterations and 30% less wall-clock time, while requiring 8–921× less memory overhead. Moving beyond the current limitations, F RESCO establishes a new foundation for high-utilization pipeline parallel training, providing a scalable path for large-model development on diverse, resource-constrained systems.

Introduction

Pipeline parallelism has played a central role in the scaling of modern machine learning by partitioning enormous neural networks across multiple processing units along their sequential data-processing structure, enabling large-scale distributed training across hundreds or even thousands of devices [10, 29, 43]. Pipeline scheduling has been integral to fully exploiting the computational capabilities of these systems, continuously streaming micro-batches through successive stages to keep different processing units concurrently occupied rather than idle [35, 27, 16, 22, 21, 32]. Asynchronous pipelining offers a straightforward way to achieve sustained, bubble- and halt-free pipeline execution by allowing parameter updates to proceed concurrently with the forward and backward computations of in-flight micro-batches (see Figure 1 for illustrations). However, lifting this parameter-equivalence condition can introduce two types of inconsistencies into these concurrent gradient computations: chain inconsistency, where partial gradients are chained across different parameter versions, and gradient staleness, where even a chain-consistent gradient no longer corresponds to the current parameter state, both of which can degrade training stability and convergence. To mitigate these adverse effects, various correction mechanisms have been developed to compute gradients with respect to either past parameter versions, obtained through stashing or reconstruction [32, 33, 47], or predicted future parameter states [5, 18, 19]. Yet, prior methods have largely overlooked the possibility of controlling parameter updates themselves to preserve concurrent gradient computations consistent, even though preventing interference with these computations was the very motivation for imposing the restriction in the first place. Consequently, asynchronous pipeline optimization has yet to establish what principle should replace the parameter-equivalence condition that safeguards gradient computations in synchronous execution. In40th Conference on Neural Information Processing Systems (NeurIPS 2026).

time

Stage 4 Stage 3 Stage 2

Stage 1

Parameter w state <latexit sha1_base64="WL8tLovw+ZvLUJryYm2mFtCpTbM=">AAAB6HicbVDLTgJBEOzFF+IL9ehlIjHxRHaNQY9ELx4hkUcCGzI79MLI7OxmZlZDCF/gxYPGePWTvPk3DrAHBSvppFLVne6uIBFcG9f9dnJr6xubW/ntws7u3v5B8fCoqeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR7cxvPaLSPJb3ZpygH9GB5CFn1Fip/tQrltyyOwdZJV5GSpCh1it+dfsxSyOUhgmqdcdzE+NPqDKcCZwWuqnGhLIRHWDHUkkj1P5kfuiUnFmlT8JY2ZKGzNXfExMaaT2OAtsZUTPUy95M/M/rpCa89idcJqlByRaLwlQQE5PZ16TPFTIjxpZQpri9lbAhVZQZm03BhuAtv7xKmhdlr1Ku1C9L1ZssjjycwCmcgwdXUIU7qEEDGCA8wyu8OQ/Oi/PufCxac042cwx/4Hz+AOhHjQY=</latexit>

Stage 1

<latexit sha1_base64="WL8tLovw+ZvLUJryYm2mFtCpTbM=">AAAB6HicbVDLTgJBEOzFF+IL9ehlIjHxRHaNQY9ELx4hkUcCGzI79MLI7OxmZlZDCF/gxYPGePWTvPk3DrAHBSvppFLVne6uIBFcG9f9dnJr6xubW/ntws7u3v5B8fCoqeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR7cxvPaLSPJb3ZpygH9GB5CFn1Fip/tQrltyyOwdZJV5GSpCh1it+dfsxSyOUhgmqdcdzE+NPqDKcCZwWuqnGhLIRHWDHUkkj1P5kfuiUnFmlT8JY2ZKGzNXfExMaaT2OAtsZUTPUy95M/M/rpCa89idcJqlByRaLwlQQE5PZ16TPFTIjxpZQpri9lbAhVZQZm03BhuAtv7xKmhdlr1Ku1C9L1ZssjjycwCmcgwdXUIU7qEEDGCA8wyu8OQ/Oi/PufCxac042cwx/4Hz+AOhHjQY=</latexit>

(a) Synchronous pipeline parallelism

<latexit sha1_base64="4mVzPtgSPqYAMvY0o1ZVAywrkEs=">AAAB6nicbVDLSgNBEOz1GeMr6tHLYBAEIeyKRI9BLx4jmgcka5idzCZDZmeXmV4lLPkELx4U8eoXefNvnDwOmljQUFR1090VJFIYdN1vZ2l5ZXVtPbeR39za3tkt7O3XTZxqxmsslrFuBtRwKRSvoUDJm4nmNAokbwSD67HfeOTaiFjd4zDhfkR7SoSCUbTS3dPDaadQdEvuBGSReDNShBmqncJXuxuzNOIKmaTGtDw3QT+jGgWTfJRvp4YnlA1oj7csVTTixs8mp47IsVW6JIy1LYVkov6eyGhkzDAKbGdEsW/mvbH4n9dKMbz0M6GSFLli00VhKgnGZPw36QrNGcqhJZRpYW8lrE81ZWjTydsQvPmXF0n9rOSVS+Xb82LlahZHDg7hCE7AgwuowA1UoQYMevAMr/DmSOfFeXc+pq1LzmzmAP7A+fwBAx+Now==</latexit>

Forward Backwards Idle Update

(b) Asynchronous pipeline parallelism

■

Figure 1: Gantt charts illustrating stage-processor states over time for micro-batch pipelining under synchronous and asynchronous parameter updates. Asynchronous updating allows the stage-1 threads for micro-batches 1 and 2 to overwrite the parameters while micro-batch 3 is still waiting on its downstream error signal • after reading the previous parameter state •, thus the updated parameters no longer form a proper derivative chain, degrading its utility for improving stage 1.

943× less memory overhead

6.5

Train loss

30% faster

7.0

6 5

LWP (P=32) Fresco (P=32)

Iteration (k)

6.0 5.5 LWP Fresco

5.0 15

(a) Robust under deep pipeline

0.0

0.2

0.4

Wall-clock time (h)

(b) Fast training

Memory overhead (GB)

43.2% fewer iterations

+2.67

100 10 1

+0.05

10 2 10 3

+3 MB Stash

Prediction

Figure 2: Summary of results. (a) Under a deep asynchronous pipeline, F RESCO improves optimization over best baseline, reaching the same training loss with fewer iterations, (b) yielding faster training speeds under wall-clock time. (c) F RESCO avoids the large memory overhead of maintaining multiple full parameter versions, using substantially less memory than stashing-based approaches. stead, existing methods have primarily focused on compensating post hoc for errors within individual micro-batches, rather than controlling the inter-micro-batch interference that gives rise to them. Simply lifting the restriction to permit unconstrained parameter updates provides no principled criterion for determining which changes can preserve gradient consistency without introducing staleness. In this work, we address this gap by showing that parameter changes degrade gradient computation not simply by altering the parameters, but by breaking the compatibility between a returned error signal and the forward behavior of the updated stage at the inter-stage boundary. Concretely, when the updated parameters would produce a different stage output from the one on which the error signal was evaluated, the error signal and the local Jacobian no longer form a valid gradient for the updated parameters. This reveals that the relevant consistency requirement is preserving the forward behavior seen by in-flight micro-batches even if the full parameter tensor changes, i.e., forward-equivalence over parameter-equivalence. Guided by this insight, we propose F RESCO, an asynchronous pipeline-parallel training method that controls parameter updates according to their effect on the forward activations of in-flight micro-batches. Concretely, we formulate this as a constrained optimization problem that finds the closest modification of each optimizer update subject to acting trivially on the activation subspace occupied by in-flight micro-batches. To implement this efficiently, F RESCO maintains a compact spectral basis for this activation subspace and projects each layer-wise update onto its orthogonal complement, thereby removing exactly the components that would violate forward equivalence. This allows parameters to evolve asynchronously while preserving the forward behavior required for the returned error signals to remain valid. Finally, we establish convergence of F RESCO, showing that it control the degree of perturbation introduced during forward and backwards pass to guarantee convergence. 2

time

Stage 4 Stage 3 Stage 2

time 1

Stage 1

time

time1

time

1 time

4 1 2 3 4 1 1 2 2 3 3 4 4 5 time time 2 3 4 1 2 2 3 3 4 4 w+5 53 6 3 31 w Parameter state Stage 4 Stage 3 1 2 3 4 1 2 3 1 4 2 1 3 24 3 4 1 2 3 1 14 12 2 23 35 34 Stage 3 1 2 3 4 1 2 3 4 Stage 44 Stage Stage 1 1 2 3 11Stage 4 223 33 44 111 222 333 4441 2 3 14 2 3 4 1 2 1 1321 1431 2 12 2 42 3 23 1313 24534 34254 15645 45 365 26 576 4 Stage 2 1 2 3 4 1 2 3 4 1 2 3 4 1 2 5 3 6 Stage33 2 1 112 223 334 44 1 1 21 2 32 3 43 1 1 4 14 2 2 2 3 53 5 35 4 64 6 46 5 75 Stage Stage 11 221 332 443 4 Stage 2 1 2 3 4 3 41 5 12 6 23 1 Asynchronous 2 3 4 1 2 pipeline 1 2 3 4 pipeline 1 2 13 24 3 4 (b) Stage 1 (a) Synchronous parallelism parallelism 1 1 21 2 32 3 43 4 4 1 112 223 334 44 Stage22 1 Stage Stage 11 221 332 443 4 1 1 1 2 52 5 25 3 63 6 36 4 74 7 47 1 2 3 4 1 5 1 2 3 4 1 2 3 4 Stage 1 3 3 w 3 1 2 3 4 1 5 2 6 w 1 2 3 4 1 2 3 4 Figure 2: Gantt charts illustrating stage-processor states over time for micro-batch Stage 1 1w 2 3 3 4 1 5 2 6 + 3 3pipelinin 7 7w4+ 4 3 1 Parameter 2state3 3 4 1 2 3 3 4 Stage 1 w 3 RESCO w time Stage 4 Stage 4 time w

Empirically, F RESCO improves asynchronous pipeline-parallel training across model scales and pipeline depths, achieving lower loss than representative asynchronous baselines in an established pipeline-parallel pre-training setting with a 95M-parameter language model, especially at moderate and deep pipeline depths (Figure 2a), allowing F to achieve the same loss with 30% less synchronous and parameter updates. updating allows the stage-1 1 2asynchronous 3 4 1 2 3 Asynchronous 4 1 2 3 4 1 5 2 6 3 wall-clock time (Figure 2b). Moreover, because F RESCO caches compact dominant feature 1 2 3 only 4 1 and 1 2 3 4 1 2 3 4 micro-batch 1 5 2 6 3 7 4 for micro-batches 1 2 2 3to 4overwrite the parameters 1 2 3 while 4 1 2 3 43 is still waitin 1 5 spectra rather than full parameter copies, its memory downstream overhead remains with signal marginal after pipeline reading parameter thus1 1no5 5longer 2the 6 3forming 3 7 7 4para 4 (a) (b) Asynchronous 11 22 33error 44 Synchronous 11 compared 2parallelism 33 previous 44 1 1 2 2 3 3 4 4state, 2 2 6 pipeline (a) Synchronous pipeline parallelism (b) Asynchronous pipeline parallelism chain with the error signal,pipeline degrading its utility for improving(b) stage 1. (a) Synchronous parallelism Asynchronous pipe existing baselines, especially for larger models (Figure derivative 2c). Figure 2: Gantt charts illustrating stage-processor states over time for micro-batch pip <latexit sha1_base64="4mVzPtgSPqYAMvY0o1ZVAywrkEs=">AAAB6nicbVDLSgNBEOz1GeMr6tHLYBAEIeyKRI9BLx4jmgcka5idzCZDZmeXmV4lLPkELx4U8eoXefNvnDwOmljQUFR1090VJFIYdN1vZ2l5ZXVtPbeR39za3tkt7O3XTZxqxmsslrFuBtRwKRSvoUDJm4nmNAokbwSD67HfeOTaiFjd4zDhfkR7SoSCUbTS3dPDaadQdEvuBGSReDNShBmqncJXuxuzNOIKmaTGtDw3QT+jGgWTfJRvp4YnlA1oj7csVTTixs8mp47IsVW6JIy1LYVkov6eyGhkzDAKbGdEsW/mvbH4n9dKMbz0M6GSFLli00VhKgnGZPw36QrNGcqhJZRpYW8lrE81ZWjTydsQvPmXF0n9rOSVS+Xb82LlahZHDg7hCE7AgwuowA1UoQYMevAMr/DmSOfFeXc+pq1LzmzmAP7A+fwBAx+Now==</latexit>

Parameter state

w Stage 1 w Parameter Parameter state state <latexit sha1_base64="WL8tLovw+ZvLUJryYm2mFtCpTbM=">AAAB6HicbVDLTgJBEOzFF+IL9ehlIjHxRHaNQY9ELx4hkUcCGzI79MLI7OxmZlZDCF/gxYPGePWTvPk3DrAHBSvppFLVne6uIBFcG9f9dnJr6xubW/ntws7u3v5B8fCoqeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR7cxvPaLSPJb3ZpygH9GB5CFn1Fip/tQrltyyOwdZJV5GSpCh1it+dfsxSyOUhgmqdcdzE+NPqDKcCZwWuqnGhLIRHWDHUkkj1P5kfuiUnFmlT8JY2ZKGzNXfExMaaT2OAtsZUTPUy95M/M/rpCa89idcJqlByRaLwlQQE5PZ16TPFTIjxpZQpri9lbAhVZQZm03BhuAtv7xKmhdlr1Ku1C9L1ZssjjycwCmcgwdXUIU7qEEDGCA8wyu8OQ/Oi/PufCxac042cwx/4Hz+AOhHjQY=</latexit>

Stage 1

ww <latexit sha1_base64="WL8tLovw+ZvLUJryYm2mFtCpTbM=">AAAB6HicbVDLTgJBEOzFF+IL9ehlIjHxRHaNQY9ELx4hkUcCGzI79MLI7OxmZlZDCF/gxYPGePWTvPk3DrAHBSvppFLVne6uIBFcG9f9dnJr6xubW/ntws7u3v5B8fCoqeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR7cxvPaLSPJb3ZpygH9GB5CFn1Fip/tQrltyyOwdZJV5GSpCh1it+dfsxSyOUhgmqdcdzE+NPqDKcCZwWuqnGhLIRHWDHUkkj1P5kfuiUnFmlT8JY2ZKGzNXfExMaaT2OAtsZUTPUy95M/M/rpCa89idcJqlByRaLwlQQE5PZ16TPFTIjxpZQpri9lbAhVZQZm03BhuAtv7xKmhdlr1Ku1C9L1ZssjjycwCmcgwdXUIU7qEEDGCA8wyu8OQ/Oi/PufCxac042cwx/4Hz+AOhHjQY=</latexit>

w <latexit sha1_base64="WL8tLovw+ZvLUJryYm2mFtCpTbM=">AAAB6HicbVDLTgJBEOzFF+IL9ehlIjHxRHaNQY9ELx4hkUcCGzI79MLI7OxmZlZDCF/gxYPGePWTvPk3DrAHBSvppFLVne6uIBFcG9f9dnJr6xubW/ntws7u3v5B8fCoqeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR7cxvPaLSPJb3ZpygH9GB5CFn1Fip/tQrltyyOwdZJV5GSpCh1it+dfsxSyOUhgmqdcdzE+NPqDKcCZwWuqnGhLIRHWDHUkkj1P5kfuiUnFmlT8JY2ZKGzNXfExMaaT2OAtsZUTPUy95M/M/rpCa89idcJqlByRaLwlQQE5PZ16TPFTIjxpZQpri9lbAhVZQZm03BhuAtv7xKmhdlr1Ku1C9L1ZssjjycwCmcgwdXUIU7qEEDGCA8wyu8OQ/Oi/PufCxac042cwx/4Hz+AOhHjQY=</latexit>

3 3

w+w <latexit sha1_base64="4mVzPtgSPqYAMvY0o1ZVAywrkEs=">AAAB6nicbVDLSgNBEOz1GeMr6tHLYBAEIeyKRI9BLx4jmgcka5idzCZDZmeXmV4lLPkELx4U8eoXefNvnDwOmljQUFR1090VJFIYdN1vZ2l5ZXVtPbeR39za3tkt7O3XTZxqxmsslrFuBtRwKRSvoUDJm4nmNAokbwSD67HfeOTaiFjd4zDhfkR7SoSCUbTS3dPDaadQdEvuBGSReDNShBmqncJXuxuzNOIKmaTGtDw3QT+jGgWTfJRvp4YnlA1oj7csVTTixs8mp47IsVW6JIy1LYVkov6eyGhkzDAKbGdEsW/mvbH4n9dKMbz0M6GSFLli00VhKgnGZPw36QrNGcqhJZRpYW8lrE81ZWjTydsQvPmXF0n9rOSVS+Xb82LlahZHDg7hCE7AgwuowA1UoQYMevAMr/DmSOfFeXc+pq1LzmzmAP7A+fwBAx+Now==</latexit>

Stage 11 Stage

Synchronous pipeline parallelism (b)Asynchronous Asynchronous pipelineparallelism parallelism parallelism (b) pipeline Figure (a) 2: Synchronous Gantt charts pipeline illustrating stage-processor states over time for micro-batch pipelining

synchronous parameter updates. Asynchronous updating the st Figureand 2: asynchronous Gantt charts illustrating stage-processor states over timeallows for micro-b synchronous and asynchronous parameter updates.states Asynchronous updating allows the stage-1 th These results indicate that F RESCO provides an effective foundation for reliable, Gantt charts illustrating stage-processor states overtime time for pipelining Figure 2: charts illustrating over for micro-batch pipelining u forGantt micro-batches 1 and andasynchronous 2stage-processor to high-utilization overwriteparameter the parameters while micro-batch 3model is still w updates. Asynchronous updating allo 78 Empirically, Fsynchronous RESCO improves asynchronous pipeline-parallel training across sca for micro-batches 1 and 2 to overwrite the parameters while micro-batch 3 is still waiting pipeline-parallel training, opening a new avenue beyond strictdownstream parameter consistency toward and asynchronous parameter updates. Asynchronous updating allows the stage-1 synchronous and updates. Asynchronous updating allows the stage-1 thr error signalparameter after reading thescalable previous state, thus no longer form forasynchronous micro-batches 1 and 2 to overwrite theparameter parameters while micro-batch 3t 79 pipeline depths. On a 95M-parameter language model, it achieves lower loss than repres downstream error reading the previous parameter state, thus no 3longer forming a op for micro-batches 1 and andwith 2 to to overwrite the parameters while micro-batch 3state, still waiting derivative chain the error signal, degrading its utility for improving stage downstream error signal after reading thewhile previous parameter thus no lon micro-batches 1signal 2after overwrite the parameters micro-batch isisstill waiting training on conventional and resource-constrained collaborative systems. 80 asynchronous baselines moderate deep pipeline Moreover, because F1. RESCO derivative chain with the at error signal, and degrading its utilitydepths. for improving stage 1.

downstream error signal after after reading theerror previous parameter state, thuscopies, longer forming derivative chain with the signal, degrading its thus utility for improving stage error signal reading the previous parameter state, nono longer a apr only compact dominant feature spectra rather than full parameter itsforming memory o chain with with the error errorwith signal, degrading itsutility utility forimproving improving stage derivative chain the signal, degrading its for stage 1.1. remains marginal compared existing baselines, especially for larger models. 78 Empirically, F RESCO improves asynchronous pipeline-parallel training across mod 78 Empirically, FEmpirically, RESCO improves asynchronous pipeline-parallel across model scale 2 Problem analysis 83 These provides an effective foundation for reliable, 79 results pipeline depths.that On Fa RESCO 95M-parameter model, ittraining achieves lower losshigh-uti than r 78 indicate improveslanguage asynchronous pipeline-parallel training acro 79 pipeline depths. On abaselines 95M-parameter language model, it achieves lower loss than represen 84 Empirically, pipeline-parallel training, opening aanew avenue beyond strict parameter consistency toward 80 asynchronous at moderate andpipeline-parallel deep pipeline depths. Moreover, because Flos R 78 FRESCO RESCO improves asynchronous pipeline-parallel training across model scale 79 F pipeline depths. On 95M-parameter language model, itacross achieves lower improves asynchronous training model scales 80 asynchronous baselines at moderate and deep pipeline depths. Moreover, because F RESCO c 85 pipeline training on and baselines resource-constrained collaborative systems. 81 only compact featurelanguage spectra rather full parameter copies, its memo 79 depths. On aa dominant 95M-parameter language model, achieves lower loss thanrepresent represen 80conventional asynchronous at moderate andthan deep pipeline depths. Moreover, bec depths. On 95M-parameter model, ititachieves lower loss than 81 only compact dominant feature spectra ratherpipeline than full parameter copies, its models. memory ove We begin by presenting a detailed formulation of 80the parameter inconsistency problem inspectra asyn82 remains marginal with existing baselines, especially for parameter larger 81 baselines only compact dominant rather than full copies, asynchronous baselines atcompared moderate andfeature deep depths. Moreover, because RESCOcaic asynchronous at moderate and deep pipeline depths. Moreover, because FF RESCO 82 remains marginal compared with existing baselines, especially for larger models. 82condition. remains marginal compared with existing baselines, especially for larger mode chronous pipeline-parallel training and the forward-equivalence 81 compact dominant feature spectra rather than fulleffective parameter copies,itsits memory ove only compact dominant feature spectra rather than full parameter copies, memory over These results indicate that F RESCO provides an foundation for reliable, hig 86 2 83Problem formulation 8283 remains marginal compared with existing especially for larger models. marginal compared with existing especially for larger models. These results indicate that F RESCO provides anavenue effective foundation for reliable, high-utiliz 84 pipeline-parallel training, opening abaselines, beyond strict parameter consistency tow 83 These results indicate thatbaselines, Fnew RESCO provides an effective foundation for reli 84 pipeline-parallel opening a new avenueanbeyond strict parameter consistency toward sca training ontraining, conventional and resource-constrained collaborative systems. 84 pipeline-parallel training, opening aeffective new beyond strict parameter consis 83 results indicate that◦ RESCO provides foundation for reliable, high-utili results that RESCO provides anmultiple effective foundation for reliable, high-utiliza 87 These We85 begin by indicate presenting aFF·detailed formulation of theavenue parameter inconsistency problem Preliminary A pipeline-parallel training setup partitions a model F = f · · ◦ f across 85 training on85conventional and resource-constrained collaborative systems. P 1 and training onopening conventional resource-constrained collaborative systems. 84 pipeline-parallel training, opening new avenue beyond strictparameter parameter consistency towardsca sc 88 pipeline-parallel chronous pipeline-parallel training and the forward-equivalence condition. training, aanew avenue beyond strict consistency toward processing units, each responsible for a stage fp . Each stageonprocesses aand stream of micro-batches, 85 conventional and resource-constrained collaborative systems. training conventional resource-constrained collaborative systems. 86 2 Problem formulation typically chosen as the largest units that fit in device simplicity, wesetup describe the GPU processor a partition of the 89 A86pipeline-parallel training assigns each 86 Setup 2memory. Problem formulation 2For Problem formulation F = fparameters → ·begin · · → formulation f1by . We this interaction at a generic stage, writinginconsistency its local module a PWe computation from the perspective of a generic stage8690f with wdescribe and a asingle micro-batch, 87Problem presenting detailed formulation of the parameter prob 2 formulation w.presenting Each processor, ortraining pipeline stage, continually processes a streamproblem of micro87 parameters We88begin by a detailed formulation offorward-equivalence the parameter in chronous pipeline-parallel and the 87 We begin by presenting a detailed formulation of inconsistency the condition. parameter inconsisten represented by its input and output features s and s+ ,91 omitting stage and micro-batch indices. We also 92 each micro-batch istraining typically chosen as the largest unitcondition. of data that can be process 88 where chronous pipeline-parallel and thetraining forward-equivalence 88 presenting chronous and forward-equivalence condition. by presenting detailed formulation thethe parameter inconsistency probleminina We begin by aa (i.e., detailed formulation ofof the parameter inconsistency problem denote the function that takes the input feature s and8793evaluate the loss valuepipeline-parallel the composition of device at once. 89 Setup A pipeline-parallel training setup assigns each GPU processor a partition 88 pipeline-parallel training trainingand andthe theforward-equivalence forward-equivalencecondition. condition. chronous pipeline-parallel downstream stages and the loss function) as simply L(s ) 90∈ R. At each stage repeatedly 89 +Setup setup assigns each processor a partition of the F A= fPaSetup →high · · · → level, fA1 . pipeline-parallel Wetraining describe this interaction at aGPU generic stage, writing its local mod 89pipeline-parallel training setup assigns each GPU processor a pm 94 gradient and updates Atawriting each stage, aamicro-batch F = → 90·pipeline-parallel · ·forward →Ff= 90 Pipelining We ats) astage, generic stage, itsastage, local module asm f performs three operations for each micro-batch: it computes the signal = fpipeline (w, andGPU 91 fP parameters Each processor, orparameter continually processes stream of 1 .w. fPcomputation →describe · · · training →training f1this .sWe describe this interaction at generic writing its lo +interaction 89 setup assigns each GPU processor partition of the Setup A pipeline-parallel setup assigns each processor a partition of the 95 two forms: aEach forward signal its corresponding error signal. stage receives am 91 in parameters w. processor, or pipeline stage, continually processes stream of micro-ba 92 where each micro-batch isand typically chosen as the largest unit The ofaitsits data that can be pr 91 parameters w. Each processor, or pipeline stage, continually processes a stre sends it downstream; it receives a downstream error e and computes the backward signal · → f 90 signal . We describe this interaction at a generic stage, writing local module as F = f → · · · → f . We describe this interaction at a generic stage, writing local module as f P + 11 upstream stage, applies its local module to produce the outgoing forwar 96 from the 92 signal where each micro-batch typically chosen as thecontinually largest of data thatstream canofbeof processed 93 s device at once. 92w. eachismicro-batch is typically chosenunit the largest unit data that c Each processor, orpipeline pipeline stage, processes micro-ba w. Each processor, or stage, continually a astream micro-bat e together with the local parameter gradient g; and,919793inparameters between, itwhere waits forthis downstream stages to asprocesses sdevice (w, s), and sends signal to downstream stages. Let L denote the of final loss co + = fat once. 93 device at once. 92 micro-batch typically chosen asthe thelargest largest unitsignal data thatcan can processe where each micro-batch isis typically chosen as unit ofofdata that bebeprocessed 98 after the downstream stages process s . The returned error is e = ωL/ωs , the g + and parameter updates At each + complete the dependent forward or backward computations. 94 gradient computation stage, a + micro-b 93 once. device atPipelining once. 99 of this loss with respect to the outgoing signal, which is sent back toeach the stage, stage 94 Pipelining computation and parameter updates At each stage, aAt micro-batch apfa 95 in two forms: a forward signal andforward its corresponding error signal. The stage receiv 94gradient Pipelining gradient computation and parameter updates 100 backpropagation. For a single such micro-batch computation, the local procedure can be wr 95 in two forms: a forward signal and its corresponding error signal. The stage receives a for 96 signal s from the upstream stage, applies its local module to produce the outgoing fo 95 in twocomputation forms: a forward signal andupdates its corresponding error The stag 94 gradient computation andparameter parameter updates Ateach eachstage, stage,asignal. amicro-batch micro-batch a Pipelining gradient and At app A close look at asynchronous pipelining Writing asynchronous computation for aitssignal 97 = fsignal (w,upstream s),sspipeline and this signal to stages. Let denote the finalout lo 96 the signal s +from the stage, applies its local module tolocal produce theLstage outgoing forward 96 from upstream stage, applies module to produce the =sends fthe (w, s) εdownstream Forward + signal 95 forms: forward signal and itscorresponding corresponding error signal. The receives fos in two forms: aa forward and its error signal. The stage receives a aforw 98 after the downstream stages process s . The returned error signal is e = ωL/ωs 97 s = f (w, s), and sends this signal to downstream stages. Let L denote the final loss com + +, generic stage and a single micro-batch reveals its characteristic structure: 97 s+upstream = f(wait (w, s), sends this signal to downstream stages. Let+ L forward denote the + 96 from the upstream stage, applies itslocal local module toproduce produce theoutgoing outgoing forward signal s from the stage, its module toerror the si forand eto = ωoutgoing L) ε forward Returned signal +applies 99 the of downstream this loss with respect the signal, which is = sent backis+to the stω 98 after stages process s+s.+The returned error signal is eerror ωL/ωs ,ethe gra 98 after the downstream stages process s . The returned signal = + + + 97 s), and and sends sends this signal to[edownstream downstream stages.Let LetLerror Ldenote denote thefinal finalloss losscomp com s+ = f (w, s), this signal to stages. the → e = ω f (w, s) ] ε Backpropagate signal such+forward micro-batch computation, theback local procedure can 99 of100 this backpropagation. loss99 with respect towith the outgoing signal, which issignal, sent to thesent stage for sa single of this lossFor respect thereturned outgoing forward which is back 98 downstream stages process . The The returned error signalisisee ωL/ωs , the gr after the downstream stages process ss+→+.to error signal , the grad ++==ωL/ωs ++ 100 backpropagation. For ga single such the local gradient procedure can be writt backpropagation. Forfmicro-batch a single micro-batch computation, the local proced = (w, s) [eforward εcomputation, Compute parameter w + ] such of (1) this loss100with respect to signal, which issignal stage for 99 with respect toωthe the outgoing forward signal, which issent sentback backto tothe the stage fol sf+outgoing = (w, s) ε Forward s+ = f (w, s) ▷ Forward signal + backpropagation. such micro-batch the local 100 backpropagation. For For aaswsingle single such micro-batch localprocedure procedure canbebewritte writ ==f(wait (w, Forward signal w ↑s) U εcomputation, Update parameters. s(g) s) εthe Forward signal can + for e+==f (w, ωsε+computation, L) ε Returned error signal + (w → w+ , wait for ∂s+ L) ▷ Asynchronous updates (2) → eε+Returned e+s) = ω L) error signal (wait for = ω L) ε Returned error signal s s ffor (w, ε Forward signal = ffrom s) ε Forward signal + e(w, = ωthe f (w, s) [e ] ε Backpropagate error signal + Strategy Method Wfwd Wfwd101→ Pipeline Wbwd concurrency Wbwd ss(wait Consist. Stale (↑) Mem. (↑) Thpt. (↓) + arises waiting interval in this thread-like lifecycle. While on += s + → to its outgoing forward signal, the sam → Backpropagate e+ = ∂s+ L(s+ ) ▷ Returned error signal (3)is waiting for the 102 batch eeωf→ω+s[e corresponding e error = ωfor fsignal (w, s) ]s) error signal gradient = ω s) [e ]error εsignal Backpropagate error signal (wait e = L) ε Returned (wait for e = L) ε Returned error signal = ω (w, [e ] ε Compute parameter sg + s f (w, + + + s w + + + t 103 process other micro-batches. The farther a stage is from the output, the Synchronous sync Wt W tprocessor can W ✁ 0→ε[eUpdate a + w 1 lon →g →→= + =ωωωssw f(w, (w, s) [e Compute gradient s) ε error Compute parameter gradient + ]parameter ff(w, s) [e ]ω] ]w fε(w, signal s) [e εεBackpropagate Backpropagate error signal = w ↑ parameters. +++ e = ∂s f (w+ , s)∗ [e+ ] ▷ Backprop error signal (4) interval tendseeg== 104 waiting to be,wbecause e U (g) must be computed and propagated back throug + →→ + + t t+ω t+ω w = w ↑ U (g) ε Update parameters. w = w ↑ U (g) Update parameters. g = ω f (w, s) [e ] ε Compute parameter gradient g = ω f (w, s) [e ] ε Compute parameter gradient w warises + 105 downstream stages. This view makes of parameter explicit. Wh 101 Pipeline concurrency from the source waiting↔ interval in thisinconsistency lifecycle. Inconsistent W gradient W W ✂ +the ωa +thread-like w ↗ωWhi g = ∂w f (w-+ , s)∗ [e+ ] ▷ Compute (5) + + the error signal e corresponding to its outgoing forward signal, th 102 batch waiting for w = εεUpdate w =w w↑ ↑Uthe U(g) (g) Update parameters. +interval 101 isPipeline concurrency arises from the waiting interval in this thread-like 101 Pipeline concurrency arises from waiting inparameters. this thread-like lifecycle. Whilelifecy one m t+ω (6) t process other micro-batches. The farther a stage is from the output, th 103 is processor can ∆ = −U (g)weight stashing ▷ Compute 102 W batch is waiting for ✁ the signal 3e+ corresponding to its outgoing 102 batch waiting for the error signal e+error corresponding its outgoing signal, forward the sames Figure 3: from Parameter-inconsistent W t update, W ωininto ωthread-like a + ωforward wlifecycle. ↗ω Pipeline concurrency arises interval this thread-like While one 101 concurrency arises from thewaiting waiting interval thiscomputed lifecycle. While onem Retrospective 104 waiting interval tends to the be, because e+ farther must be and propagated 103 processor can process other micro-batches. The is farther athe stage is from the ot 103t+ω processor can ! process other micro-batches. The a stage from output, theback longe t tthe gradient computation. batch is downstream waiting for error signal ee++corresponding totoits outgoing forward signal, the same 102 waiting for the error signal corresponding its outgoing forward signal, the sames weight recovery W W W ✁ ω ω a + w ↗ω 105 stages. This view makes the source of parameter inconsistency explici 104 waiting tends toe+be,must because e+ mustand be computed propagated 104 waiting interval tends interval to be, because be computed propagatedand back through processor can other micro-batches. The farther a stage from the the 103 can process process otherstages. micro-batches. The farther stageisisof from theoutput, output, thelonger long 105 stages. downstream Thistheview makes theasource parameter inconsistency 105 downstream This view makes source of parameter inconsistency explicit. Whil waiting interval tends 104 t+ω interval t+ω tends to to be, be, because because ee++ must mustbe becomputed computedand andpropagated propagatedback backthrough throughm + t+ω∗ ! where ∂s f (w+ ,spectrain s)∗ [e+ ] and ∂w f (wW , s) [e+ ] denote the reverse-mode chain-rule operations. W W ✁ 0 ω a + w ↗ω 3 downstream stages. inconsistency 105 stages. This This view view makes makesthe thesource sourceofofparameter parameter inconsistencyexplicit. explicit.While Whi t+ω t+ω ! t+ω xpipe by this formulation W 0 ω a3 + w ↗ω One property exposed is howWinconsistencyWbecomes more ✁ severe for 3earlier Prospective 81 82

Table 1: (fix) Consistency strategies in pipeline parallel training. Consist.: whether forward–backward consistency is preserved. Stale: gradient staleness, where lower is better. Mem.: memory usage, where a and w denote activation and weight memory per stage. Thpt.: throughput. N : pipeline stages, ω : pipeline depth. <latexit sha1_base64="oe/frwLLz9HrOaOJDgj18weP14Q=">AAAB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqseiF48V7Ae0oWy2k3bpZhN3N0Ip/QtePCji1T/kzX/jps1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4NvPbT6g0j+WDmSToR3QoecgZNZnUk/jYL1fcqjsHWSVeTiqQo9Evf/UGMUsjlIYJqnXXcxPjT6kynAmclXqpxoSyMR1i11JJI9T+dH7rjJxZZUDCWNmShszV3xNTGmk9iQLbGVEz0steJv7ndVMTXvtTLpPUoGSLRWEqiIlJ9jgZcIXMiIkllClubyVsRBVlxsZTsiF4yy+vktZF1atVa/eXlfpNHkcRTuAUzsGDK6jDHTSgCQxG8Ayv8OZEzovz7nwsWgtOPnMMf+B8/gAbn45N</latexit>

→=

! t+ω upstream: 33 0 pipeline stages. pipeoptim The wait in (2) accumulates before an upstream its returned W W t+ω W t+ω stage receives ✁ t+ω t+ω t+ω error signal, the pbsc neighboring downstream waiting interval, ! stage mustWitself complete W Wits ✁ together with 0 the additional forward and backward computations in (1–5). As shown in Figure 1, this interval is long t+ω Algebraic F RESCO stage to process W t other micro-batches W t+ω ✁ updates, so0the enough for the upstream andWapply intervening + parameter state may drift from w to w before the original micro-batch returns for backpropagation in (3), illustrated in Figure 3. 136 137 138 139 140 141 142 143 144 145

ωa + w ωa + w

↗ω ↗ω

ωa + w

↗ω

Prospective consistency and Attending forecasttoerror Another lineinof predicts Forward-equivalence condition the gradient computation (5),work we identify the the weights to be between the returned e+ and the parameter state used inreducing the local Jacobian usedinconsistency at backward time insteaderror of signal storing forward-time copies, memory overhead at the + , s), whicherror. generally prevents them from forming a proper derivative chain. This raises a cost∂woff (w prediction [5, 12] SpecTrain [5] performs lookahead-style weight prediction using natural question: can the same error signal e+ remain valid across different parameter states w+ momentum-smoothed futureis weights at forward time. XPipe [12] refines this that induce different localgradients Jacobians? to Theforecast key observation that e+ depends on the downstream + by incorporating optimizer-aware update directions (e.g., Adam moments) improve prediction feature s+ , not directly on the parameter tensor w itself. Therefore, if the updated parameter wto accuracy and robustness. DC-ASGD [28] instead compensates for delay at the gradient level by applying a first-order Taylor correction, approximating gradients at the current weights. Similarily to 3 Pipemare, reliance on local approximations to extrapolate non-local weight trajectories would depend on sufficiently smooth optimization trajectories (often induced by momentum [1]) and may degrade under noisy or rapidly changing gradients.

produces the same forward feature for the in-flight micro-batch, then the returned error signal can still be chained with the local Jacobian at w+ : ∀w+ : f (w+ , s) = f (w, s) = s+

=⇒

g = ∂w f (w+ , s)∗ [e+ ] is a proper gradient at w+

This forward-equivalence condition is the key idea behind our work: the full parameter tensor need not remain fixed, as long as the stage preserves the forward behavior seen by the in-flight micro-batch until its error signal returns. Under this condition, the returned error signal and the changed local Jacobian still form a valid gradient at the updated parameter state. The remaining question is therefore how to constrain asynchronous parameter updates so that they preserve these boundary signals while still allowing the parameters to evolve.

Method

Building on this insight, we design Forward-Equivalent Subspace-Constrained Optimization, or F RESCO, as the solution to the constrained optimization problem described below. The resulting update rule exploits a natural non-interference structure of linear operators: updates that annihilate the in-flight activation subspace do not change the corresponding forwarded features. 3.1

Derivation

Consider a linear layer with bias absorbed into the weight matrix: Y = W X̄, where W ∈ Rdout ×(din +1) . For each such layer, we solve the constrained optimization problem 1 ′ min ∥∆ − ∆∥2F s.t. (W + ∆′ )X̄ = W X̄, (7) ∆′ 2 where X̄ ∈ R(din +1)×τ n denotes the concatenated input features of all τ in-flight micro-batches, each of size n, whose forward computations are to be protected from asynchronous parameter updates ∆ ∈ Rdout ×(din +1) . Thus, the goal is to construct the feasible update ∆′ closest in Frobenius norm ∥ · ∥F to the original update ∆ such that the updated weight W + ∆′ preserves the same forward pass as W on the protected micro-batch. If every asynchronous update that intersects the lifetime of this micro-batch is subjected to the same constraint, then the final updated weight W + remains forward-equivalent to the weight state under which the micro-batch entered protection. Rewriting the preservation condition as

∆′ X̄ = 0, (8) shows that the constraint does not explicitly depend on the weight W ; it only depends on the action of the update on the protected in-flight activations. We solve the constrained problem through the KKT conditions. Introducing a matrix-valued Lagrange multiplier Λ ∈ Rdout ×τ n with the same shape as ∆′ X̄, the Lagrangian is 1 L(∆′ , Λ) = ∥∆′ − ∆∥2F + ⟨Λ, ∆′ X̄⟩F , 2 ⊤ where ⟨A, B⟩F = tr(A B) denotes the Frobenius inner product. Because the objective is convex quadratic and the constraint is affine, the KKT conditions are necessary and sufficient: ∆′ X̄ = 0, The stationarity condition in Equation (9) gives

∇∆′ L(∆′ , Λ) = 0.

0 = ∆′ − ∆ + ΛX̄ ⊤ ,

∆′ = ∆ − ΛX̄ ⊤ .

(9) (10)

Substituting Equation (10) into the feasibility condition gives 0 = (∆ − ΛX̄ )X̄ = ∆X̄ − ΛX̄ ⊤ X̄, and therefore ΛX̄ ⊤ X̄ = ∆X̄. Using the Moore–Penrose pseudoinverse1 , we write Λ = ∆X̄(X̄ ⊤ X̄)† . Finally, substituting this into Equation (10) yields ∆′⋆ = ∆ I − X̄(X̄ ⊤ X̄)† X̄ ⊤ .

⊤

(11)

1 Although Λ may not be unique when X̄ ⊤ X̄ is singular, all valid choices yield the same product ΛX̄ ⊤ , and hence the

same primal solution ∆′⋆ .

Here, the factor X̄(X̄ ⊤ X̄)† X̄ ⊤ in Equation (11) is the orthogonal projector onto col(X̄). Using the compact SVD X̄ = U ΣV ⊤ , we can write PX̄ := X̄(X̄ ⊤ X̄)† X̄ ⊤ = U U ⊤ . Thus, ∆′⋆ = ∆(I − PX̄ ) = ∆(I − U U ⊤ ). 3.2

(12)

Relaxation

For extremely deep pipelines, concurrency can reach a regime where each parameter update affects an increasing number of in-flight micro-batches, posing distinct challenges to both existing methods and F RESCO. While existing methods suffer from increasingly stale gradients or unreliable predictions of future parameter states, F RESCO may find that protecting all affected micro-batches leaves no permissible direction for parameter movement, potentially annihilating the update altogether, i.e., ∆′⋆ = 0. Existing correction-based methods are not inherently equipped to handle this regime and often resort to auxiliary mechanisms such as step-size damping or momentum to maintain stability [47, 1]. Unlike these methods, F RESCO offers an explicit trade-off between gradient fidelity and effective parameter movement by relaxing the protection condition to preserve the most influential directions while allowing greater freedom for asynchronous parameter updates. We do this by protecting only the spectrally dominant components of col(X̄). Specifically, we construct a rank-k protection matrix X̄k as the best low-rank approximation of X̄ in the Frobenius norm: X̄k = arg minrank(A)≤k ∥X̄ − A∥F . Substituting this approximation into F RESCO yields the projection matrix Pk = Uk Uk⊤ , where Uk contains the top-k left singular vectors of X̄. The resulting rank-limited update is ∆′k = ∆(I − Uk Uk⊤ ). (13) As the rank budget k increases, the relaxation approaches exact preservation; when k = rank(X̄), Equation (13) recovers the exact projection in Equation (12). Since weight matrices vary in dimensions dout and din , their protection capacities min(dout , din + 1) also differ. Using a fixed rank across layers may therefore lead to disproportionate protection or unnecessarily restrict parameter updates. Instead, we allocate a fraction α ∈ [0, 1] of each layer’s capacity, capped by the maximum number τ n of in-flight feature directions: k = min {τ n, ⌈α min(dout , din + 1)⌉} .

(14)

This provides a consistent relative protection budget across layers without allocating more directions than the in-flight features can span. In practice, Uk is assembled from per-micro-batch bases rather than computed directly from X̄, as described in Section 3.3. 3.3

Algorithm

Here, we describe the algorithmic operation at a high level; the full procedure is given in Algorithm 1. For each linear layer, F RESCO maintains a FIFO cache of low-rank activation bases for the in-flight micro-batches. Since the protected rank k is shared among the τ in-flight micro-batches, each micro-batch contributes k1 = ⌈k/τ ⌉ directions, so that the cache holds τ k1 ≥ k candidate directions. When a new micro-batch enters the layer, F RESCO calls E XTRACT BASIS(X̄, k1 ) to compute a rankk1 basis of its bias-augmented input activations and enqueues the result. We use an inexpensive approximation: a single step of block power iteration, orthogonalized by a single Newton–Schulz step (Section C.3). When the corresponding backward error signal returns, that basis is dequeued, since the micro-batch no longer needs protection. Before applying an optimizer update, F RESCO calls B UILD P ROTECTED BASIS(Q, k), which concatenates the queued bases and keeps their dominant k directions by block power iteration. This produces a single orthonormal basis Q ∈ R(din +1)×k , the rank-k protected subspace shared by the in-flight micro-batches. The base optimizer first proposes the usual update ∆; F RESCO then applies ∆′ = ∆(I − QQ⊤ ) instead, removing only the components that would change the protected forward features of the still in-flight micro-batches. Because each micro-batch stores only ⌈k/τ ⌉ directions, the cache holds fewer than k + τ basis vectors per layer regardless of the pipeline depth, and it stores only compact activation bases rather than delayed weight copies or predicted parameter buffers. Thus, the main overhead is computation for subspace extraction, basis construction, and projection, rather than model-scale memory. 5

Algorithm 1 F RESCO step Require: Weight W ∈ Rdout ×(din +1) (bias absorbed), input activation X, gradient G, protection budget α, number τ of micro-batches in flight at this stage, mode ∈ {F ORWARD, BACKWARD} State: FIFO queue Q storing protected activation bases 1: if Q is uninitialized then 2: Initialize Q ← ∅ 3: end if 4: if mode = F ORWARD then 5: X̄ ← [X; 1⊤ ] ∈ R(din +1)×n ▷ n = batch × sequence 6: k ← min(τ n, ⌈α min(dout , din + 1)⌉) ▷ protected rank 7: k1 ← ⌈k/τ ⌉ ▷ rank per micro-batch 8: V ← E XTRACT BASIS(X̄, k1 ) ▷ V ∈ R(din +1)×k1 9: Q.ENQUEUE(V ) 10: return 11: end if 12: if mode = BACKWARD then 13: ∆ ← O PTIMIZERU PDATE(W, G) ▷ base optimizer proposes update 14: Q.DEQUEUE() ▷ remove the completed micro-batch basis 15: Q ← B UILD P ROTECTED BASIS(Q, k) ▷ top-k subspace of the remaining bases 16: ∆′ ← ∆(I − QQ⊤ ) ▷ project away feature-changing directions 17: W ← W + ∆′ ▷ apply forward-equivalent update 18: end if

Theory

We provide a preliminary convergence analysis of F RESCO under asynchronous pipeline execution. Building on the perturbed forward–backward SGD framework of Kong et al. [25], we examine how the forward perturbations induced by concurrent parameter updates contribute to gradient errors and convergence. For a linear stage, the residual forward perturbation of a projected update is ′ Ri,t := ∥∆Wi,t S̄i,t ∥F ,

(15)

′ Ri,t ≤ σi,t,k+1 ∥∆Wi,t ∥F ,

(16)

where S̄i,t contains the protected in-flight features. For rank-k projection, this residual is bounded by which vanishes when the complete activation subspace is protected. Under bounded-delay execution, these residuals can be related to asynchronous gradient errors through the perturbation bounds of Kong et al. [25]. To examine the convergence implications, we consider a soft extension of F RESCO, ′ ∆Wi,t = ∆Wi,t Ai,t ,

⊤ Ai,t := (I + λi Ui,t,k Σ2i,t,k Ui,t,k )−1 ,

(17)

which approaches hard projection as λi → ∞. Theorem 4.1 (Convergence under soft projection). Under the assumptions in Section B, consider the update wt+1 = wt − ηAt get , where√At collects the soft projection operators and satisfies aI ⪯ At ⪯ I for some a > 0. For η = c/ T with 0 < c ≤ 1/(4L), T −1

1 X 2∆L + 4Lc2 σ 2 3c2 (Kf s2 + Kb ) √ E∥∇L(wt )∥2 ≤ + , T t=0 aT ac T

(18)

where ∆L := L(w0 ) − L⋆ , s bounds the residual forward sensitivity, and Kf , Kb depend on the computational graph and pipeline delays. This suggests that controlling forward perturbations is compatible with the standard O(T −1/2 ) stochastic convergence rate under the stated conditions. The guarantee is limited to the simplified soft-projection model and does not establish convergence acceleration or a full-gradient guarantee for hard F RESCO. Details are provided in Section B. 6

# of stages

Train loss

(e) PipeOptim

Iteration (k)

(f) SpecTrain

(g) LWP

Iteration (k)

Slowdown: 1.69x

P=1 P=2 P=4 P=8 P=16 P=32

6 5

Iteration (k)

(d) Parameter recovery 8

P=1 P=2 P=4 P=8 P=16 P=32

Slowdown: 2.40x

Iteration (k)

P=1 P=2 P=4 P=8 P=16 P=32

Iteration (k)

Slowdown: 2.97x

P=1 P=2 P=4 P=8 P=16 P=32

6 5

Slowdown: 2.40x

P=1 P=2 P=4 P=8 P=16 P=32

Inconsistent Stash Recovery Pipeoptim LWP Fresco

(b) P =32 training curve

Slowdown: 2.80x

(a) Depth degradation

4.5

Slowdown: 2.97x Train loss

5.0

Train loss

5.5

Train loss

6.0

Pipeoptim LWP Spectrain Fresco

Train loss

Sync Inconsistent Stash Recovery

Train loss

Final train loss

6.5

Iteration (k)

(h) F RESCO

Figure 4: Main results on 95M language-model pre-training. (a) F RESCO remains robust performance under deeper pipeline compared to other baselines, with (b) prominent optimization performance at P = 32. (c–h) training curves across pipeline depths similarly demonstrates robustness of F RESCO.

Evaluation

We compare F RESCO with a parameter-inconsistent baseline and five representative asynchronous pipeline-parallel strategies: parameter stashing [32], parameter recovery [47], and three parameterspeculation baselines, SpecTrain, PipeOptim, and LWP [5, 19, 27], which are described in detail in Section A.1. Following prior study[1], we evaluate all methods on next-token prediction by pre-training a randomly initialized 95M-parameter NanoGPT decoder-only Transformer [24]. The model is partitioned across P = 1 to 32 pipeline stages and trained on OpenWebText (OWT) [17] with a micro-batch size of 8 and a sequence length of 512 for 20k iterations. Additional experimental details are presented in Section C. The code to reproduce the results of the paper is provided in https://github.com/LOG-postech/fresco. 5.1

Robust optimization under deep pipelines

We evaluate the robustness of F RESCO under increasingly severe inconsistency induced by deep asynchronous pipelining. Specifically, we measure how well each method preserves optimization speed relative to the shallow-pipeline reference as the number of pipeline stages increases up to P = 32. Figure 4a reports the final training loss as the number of pipeline stages increases. Most asynchronous baselines degrade sharply with larger P , indicating that their consistency mechanisms become increasingly insufficient as more micro-batches are simultaneously in flight. By contrast, F RESCO maintains a substantially lower final loss under deep pipelines. As shown in Figure 4b, at P = 32, F RESCO converges substantially faster than the strongest baseline, LWP, reaching the same training loss with 43.2% fewer iterations. Additionally, the remaining panels of Figure 4 provide the full training curves across pipeline depths for each method. We report slowdown as the relative number of iterations required to reach a fixed target loss compared with the shallow-pipeline reference. Fully inconsistent asynchronous training, parameter stashing, parameter recovery, and PipeOptim all show slowdowns of approximately 2.80– 2.97×, while SpecTrain and LWP still slow down by 2.40× because they cannot perfectly predict the future trajectory. In contrast, F RESCO reduces the slowdown to 1.69×. Thus, even though F RESCO uses the current updated parameters rather than reconstructing or predicting a matched full parameter state, subspace-level preservation provides stronger robustness under deep asynchrony. These results suggest that F RESCO establishes a more reliable strategy than prior baselines, supporting forward-equivalent update projection as effective in preserving optimization progress, avoiding both the stale-gradient limitation of parameter stashing and recovery and the instability of prediction-based 7

Train loss

6.5

6.0 5.0 0.0

7.0 34% less memory-time

+0.003 GB

6.5

PBSC Recovery PipeOptim Stash

Inconsistent Stash Recovery Pipeoptim LWP Spectrain Fresco

5.5

8×~921× less memory

0.2

0.4

Wall-clock time (h)

(a) Loss vs. wall-clock time

Train loss

30% faster

7.0

+0.02 GB +0.05 GB +0.08 GB +2.67 GB

10 1

10 2

100

Memory overhead (GB) (b) Memory overhead

101

6.0

Inconsistent Stash Recovery Pipeoptim LWP Spectrain Fresco

5.5 5.0 0

Memory-time product (GB h) (c) Loss vs. memory-time

Figure 5: Compute efficiency of F RESCO. F RESCO improves both runtime and memory efficiency under deep pipelining: it reaches the target loss about (a) 30% faster in wall-clock time, uses (b) from 8× up to 921× less memory overhead, and achieves a (c) 34% lower memory-time product. These gains come from storing compact activation-subspace summaries rather than delayed full-parameter copies. methods under uncontrolled asynchronous updates to forward features that pending backward signals still depend on. 5.2

Compute efficiency

We next evaluate the compute efficiency of F RESCO in terms of wall-clock time, memory overhead, and their combined memory-time product on P = 32. Figure 5a shows that F RESCO reaches the same target loss about 30% faster than the strongest baseline in wall-clock time, despite the additional computation required for subspace estimation, basis merging, and update projection. Also, the total memory overhead added by each asynchronous pipeline methods in Figure 5b show a complementary advantage. This is due to all baselines rely on parameter-sized auxiliary buffers that is much larger than what F RESCO rely on, with parameter stashing additionally scaling with the number of pipeline stages by storing full parameter versions, reaching about 2.67 GB in our setting. In contrast, F RESCO stores only compact spectral summaries of in-flight activation subspaces, requiring about 3.0 MB of additional memory, corresponding to a 921× reduction over stashing. This also translates into the end-to-end memory-time view in Figure 5c, where the memory-time product measures the cumulative memory footprint required to reach a target loss. By this metric, F RESCO achieves the same loss with a 34% lower memory-time product, showing that its gains do not merely shift cost from memory to additional training time.

Discussion

6.1

Protection capacity

Any asynchronous pipeline-parallel method faces a finite consistency budget: as pipeline depth grows, more in-flight micro-batches must be made compatible with later parameter states, increasing either staleness, prediction error, memory, or update restriction. Unlike prior asynchronous pipeline-parallel strategies, F RESCO provides explicit insight into how this capacity scales with pipeline depth: the protected rank should remain well below the layer input dimension, otherwise the feasible update space becomes too small. As a rough estimate, if DeepSeek-V3, with its 61 Transformer blocks and smallest capacity being the expert intermediate width 2048, is considered under its 4K pre-training sequences [29], this would imply that even one delayed batch-size-one micro-batch contains 4096 token features, exceeding the 2048-dimensional input of restrictive expert projections. However, optimistically, recent practical measurements indicate that transformer activation spectra concentrate most of their mass in the top singular directions, fitting λj ∝ j −2ρ with 2ρ ≈ 1.2–1.3 [13], under which the top 64 directions in 7168 dimensions capture about 72%–80% of the spectral mass. Thus, while raw sequence length gives a pessimistic upper bound on the consistency burden, these empirical findings suggest that the effective burden may be much smaller in practice if spectral concentration generalizes, giving reason for optimism about scaling F RESCO and motivating the relaxed protection schemes proposed in Section 3.2. 8

6.2

Trade-off between protection and progress

Complete protection is likely not always beneficial, even when it is possible. Indeed, the underlying nature of asynchronous pipelining is not merely that each update must protect an increasing number of in-flight micro-batches, but that this protection is inherently in tension with the update’s own contribution to optimization progress. Thus, practical schemes that expose dials for this balance, such as relaxed versions of F RESCO, may be important for identifying configurations that maximize performance within a fixed training time. Another possible remedy is to improve the trade-off itself by strengthening the update or derivative oracle, as recent directions such as Nesterov-style updates or basis-rotation methods suggest that stronger optimization techniques may be naturally more robust to inconsistency, perhaps because their local approximation remains useful over a larger region [9, 1, 23]. Together, these observations suggest that future asynchronous pipeline training methods should not treat consistency preservation and optimization as separate components, but should instead co-design the projected update rule, the degree of protection, and the optimizer dynamics, especially in regimes where deeper pipelines make update interference increasingly severe. 6.3

Extensions

One potential extension concerns the update components discarded by F RESCO. Although these components interfere with in-flight forward computations, they may still contain useful optimization information. Rather than discarding them permanently, one could accumulate and reapply them once the corresponding protection constraints expire, resembling error-feedback mechanisms in efficient-state optimizers and distributed optimization [42, 38]. However, these delayed updates may themselves become stale, making their effective reuse a nontrivial problem. Another extension is to study how F RESCO interacts with hybrid parallelism in large-scale training, including data, tensor, sequence/context, and recent paradigms such as SWARM parallelism [41, 26, 50, 39]. Subspace Networks [36] are particularly relevant, as they argue that training can proceed with low-rank micro-batch forward and error signals, suggesting a naturally favorable setting for F RESCO. A larger-scale stress test across model sizes, pipeline depths, and hybrid-parallel configurations is therefore an important next step, though beyond our available compute. Finally, a more comprehensive convergence analysis would need to account for the coupled dynamics between parameter changes, the resulting asynchronous gradient errors, and the effects of different update-control mechanisms on subsequent optimization. Characterizing these feedback interactions through a dynamical-systems perspective remains an interesting direction for future work.

Conclusion

We proposed a novel and principled forward-equivalence condition that directly controls the asynchronous updates, leading us to designing F RESCO based on constrained optimization that demonstrates consistently strong performance and efficiency under deep pipelines compared to existing representative baselines. We believe this shift from parameter-level consistency provides a promising foundation for future high-utilization training systems for stable and efficient training of larger models and more resource-limited distributed environments.

Acknowledgment This work was partly supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (RS-2019-II191906, Artificial Intelligence Graduate School Program (POSTECH), RS-2026-25507427, Development of Efficient Architectures and Training Techniques for High-Performance Lightweight AI Models), the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (Grant No. RS-2026-25612350), Project Title: Development of an End-to-End Design and Validation Framework for Mathematically Principled Next-Generation AI Architectures and Learning Algorithms), the Korea Basic Science Institute(National research Facilities and Equipment Center) grant funded by the Korea government(MSIT) (RS-2026-25500419). 9

References [1] Thalaiyasingam Ajanthan, Sameera Ramasinghe, Yan Zuo, Gil Avraham, and Alexander Long. Nesterov method for asynchronous pipeline parallel optimization. ICML, 2025. [2] B. R. Badrinath and Krithi Ramamritham. Semantics-based concurrency control: Beyond commutativity. ACM Transactions on Database Systems, 17(1):163–199, 1992. [3] Gerard M Baudet. Asynchronous iterative methods for multiprocessors. Journal of the ACM (JACM), 25(2):226–244, 1978. [4] Daniel Chazan and Willard Miranker. Chaotic relaxation. Linear algebra and its applications, 2(2):199–222, 1969. [5] Chi-Chung Chen, Chia-Lin Yang, and Hsiang-Yun Cheng. Efficient and robust parallel dnn training through model parallelism on multi-gpu platform. arXiv preprint arXiv:1809.02839, 2018. [6] Huiqiang Chen, Tianqing Zhu, et al. Machine unlearning via null space calibration. In IJCAI, 2024. [7] Zihao Chen, Chen Xu, Weining Qian, and Aoying Zhou. Elastic averaging for efficient pipelined dnn training. In PPoPP, 2023. [8] De Cheng, Yue Lu, Lingfeng He, Shizhou Zhang, Xi Yang, Nannan Wang, and Xinbo Gao. Mamba-CL: Optimizing selective state space model in null space for continual learning. arXiv preprint arXiv:2411.15469, 2024. [9] Timothy Dozat. Incorporating nesterov momentum into adam. ICLR Workshop, 2016. [10] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [11] Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. In ICLR, 2025. [12] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In AISTATS, 2020. [13] Damien Ferbach, Courtney Paquette, Gauthier Gidel, Katie Everett, and Elliot Paquette. Logarithmic-time schedules for scaling language models with momentum. arXiv preprint arXiv:2602.05298, 2026. [14] Hamid Reza Feyzmahdavian and Mikael Johansson. Asynchronous iterations in optimization: New sequence results and sharper algorithmic guarantees. arXiv preprint arXiv:2109.04522, 2023. doi: 10.48550/arXiv.2109.04522. [15] Andreas Frommer and Daniel B. Szyld. On asynchronous iterations. Journal of Computational and Applied Mathematics, 123(1–2):201–216, 2000. doi: 10.1016/S0377-0427(00)00409-X. [16] Alexander L Gaunt, Matthew A Johnson, Maik Riechert, Daniel Tarlow, Ryota Tomioka, Dimitrios Vytiniotis, and Sam Webster. Ampnet: Asynchronous model-parallel training for dynamic neural networks. arXiv preprint arXiv:1705.09786, 2017. [17] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/ OpenWebTextCorpus, 2019. [18] Lei Guan, Wotao Yin, Dongsheng Li, and Xicheng Lu. Xpipe: Efficient pipeline model parallelism for multi-gpu dnn training. arXiv preprint arXiv:1911.04610, 2019. [19] Lei Guan, Dongsheng Li, Yongle Chen, Jiye Liang, Wenjian Wang, and Xicheng Lu. Pipeoptim: Ensuring effective 1f1b schedule with optimizer-dependent weight prediction. IEEE Transactions on Knowledge and Data Engineering, 2025. 10

[20] Tuan Hoang, Santu Rana, Sunil Gupta, and Svetha Venkatesh. Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection. In WACV, 2024. [21] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. NeurIPS, 2019. [22] Zhouyuan Huo, Bin Gu, Heng Huang, et al. Decoupled parallel backpropagation with convergence guarantee. In ICML, 2018. [23] Hyunji Jung, Sungbin Shin, and Namhoon Lee. Mitigating staleness in asynchronous pipeline parallelism via basis rotation. arXiv preprint arXiv:2602.03515, 2026. [24] Andrej Karpathy. NanoGPT. https://github.com/karpathy/nanoGPT, 2022. [25] Boao Kong, Hengrui Zhang, and Kun Yuan. On the convergence of stochastic gradient descent with perturbed forward-backward passes. arXiv, 2026. [26] Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. MLSys, 2023. [27] Atli Kosson, Vitaliy Chiley, Abhinav Venigalla, Joel Hestness, and Urs Koster. Pipelined backpropagation at scale: training large models without batches. MLSys, 2021. [28] Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. TRGP: Trust region gradient projection for continual learning. arXiv preprint arXiv:2202.02931, 2022. [29] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. [30] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In NeurIPS, 2021. [31] Yue Lu et al. Visual prompt tuning in null space for continual learning. NeurIPS, 2024. [32] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. SOSP, 2019. [33] Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memoryefficient pipeline-parallel dnn training. ICML, 2021. [34] Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memoryefficient pipeline-parallel dnn training. In ICML, 2021. [35] A Petrowski, G Dreyfus, and C Girault. Performance analysis of a pipelined backpropagation parallel algorithm. IEEE Transactions on Neural Networks, 1993. [36] Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham, Yan Zuo, and Alexander Long. Subspace networks: Scaling decentralized training with communication-efficient model parallelism. NeurIPS, 2025. [37] Rodolfo F. Resende, Divyakant Agrawal, and Amr El Abbadi. Semantic locking in objectoriented database systems. In Proceedings of the ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages, and Applications, pages 388–402, 1994. [38] Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu, and Dan Alistarh. Ldadam: Adaptive optimization from low-dimensional gradient statistics. ICLR, 2025. [39] Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can be surprisingly communication-efficient. ICML, 2023. 11

[40] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In ICLR, 2021. [41] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [42] Sebastian U Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for sgd with delayed gradients and compressed communication. JMLR, 2020. [43] Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. [44] Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. In CVPR, 2021. [45] William E. Weihl. Commutativity-based concurrency control for abstract data types. IEEE Transactions on Computers, 37(12):1488–1505, 1988. [46] William E. Weihl. Local atomicity properties: Modular concurrency control for abstract data types. ACM Transactions on Programming Languages and Systems, 11(2):249–283, 1989. [47] Bowen Yang, Jian Zhang, Jonathan Li, Christopher Ré, Christopher Aberger, and Christopher De Sa. Pipemare: Asynchronous pipeline parallel dnn training. MLSys, 2021. [48] PengCheng Yang, Xiaoming Zhang, Wenpeng Zhang, Ming Yang, and Hong Wei. Group-based interleaved pipeline parallelism for large-scale dnn training. In ICLR, 2022. [49] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In NeurIPS, 2020. [50] Fanlong Zeng, Wensheng Gan, Yongheng Wang, and Philip S Yu. Distributed training of large language models: A survey. Natural Language Processing Journal, 2025. [51] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 2019. [52] Jiayun Zhang, Shuheng Li, Haiyu Huang, Xiaofan Yu, Rajesh K. Gupta, and Jingbo Shang. Orthogonal calibration for asynchronous federated learning. arXiv preprint arXiv:2502.15940, 2025. [53] Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie-Yan Liu. Asynchronous stochastic gradient descent with delay compensation. ICML, 2017.

Related works

A.1

Asynchronous pipeline parallelism methods

Retrospective consistency and staleness One dominant strategy is to compute gradients using the forward-time parameters to ensure consistency between the forward and backward passes [32, 34, 48, 47]. PipeDream [32] introduced parameter stashing, which keeps a snapshot of the forwardtime parameters for each micro-batch and uses it during the backward pass, analogous to process forking. However, this approach incurs significant memory overhead; in the worst case, the first pipeline stage must store parameters equivalent to the entire model. PipeMare [47] addresses this limitation by maintaining an exponential moving average of gradients and applying a first-order Taylor approximation of the optimization trajectory to recover the parameters. Although these approaches produce gradients consistent with the forward-time parameters, they remain stale relative to the latest parameters. Prospective consistency and forecast error Another line of work predicts the parameters to be used at backward time instead of storing forward-time copies, reducing memory overhead at the cost of prediction error. [5, 18] SpecTrain [5] performs lookahead-style parameter prediction using momentum-smoothed gradients to forecast future parameters at forward time. XPipe [18] refines this by incorporating optimizer-aware update directions (e.g., Adam moments) to improve prediction accuracy and robustness. DC-ASGD [53] instead compensates for delay at the gradient level by applying a first-order Taylor correction, approximating gradients at the current parameters. Similarily to Pipemare, reliance on local approximations to extrapolate non-local parameter trajectories would depend on sufficiently smooth optimization trajectories (often induced by momentum [1]) and may degrade under noisy or rapidly changing gradients. Restricting optimization progress Another line of work regularizes updates to limit the deviation between forward- and backward-time parameters, reducing inconsistency without explicit prediction or stashing. AvgPipe [7] applies elastic averaging to keep local parameters close to a shared reference, effectively constraining drift across pipeline stages. PipeMare [47] incorporates delay-aware learning rate rescheduling to dampen updates under larger staleness. Momentum correction [1] can be viewed as implicitly smoothing the optimization trajectory, reducing abrupt deviations. However, these methods restrict update dynamics globally, which may limit optimization efficiency and do not directly resolve parameter inconsistency. A.2

Connection to asynchronous and semantic concurrency control

Our work is conceptually connected to asynchronous numerical computation, which studies alternatives to conventional iterative methods that proceed without global synchronization under delayed or partially inconsistent updates [4, 3, 15, 14]. It is also distantly related to semantic concurrency control, where conflicts are defined by the abstract effects of operations rather than by low-level read–write overlap [45, 46, 2, 37]. We draw only on the shared abstraction principle: synchronization need not always be imposed on the full memory representation when the semantics of the computation identify a smaller conflicting component. Our method can be seen as a domain-specific instantiation of this principle in pipeline-parallel continuous optimization: consistency is enforced on the semantically relevant effect of an update, rather than on its full parameter representation. A.3

Connection to gradient projection methods in continual and multi-task learning

Projecting parameter updates onto the orthogonal complement of a protected feature subspace recurs across several areas of literature. In continual learning, OWM [51], OGD [12], GPM [40], and Adam-NSCL [44] constrain new-task updates to the null space of features from previous tasks, with later work relaxing this constraint or extending it to transformers and state-space models [28, 31, 8]. The same construction appears in knowledge editing [11], multi-task gradient surgery [49, 30], and machine unlearning [6, 20]. The closest precedent is OrthoFL [52], which projects asynchronous federated-learning drift onto a subspace orthogonal to stale client updates. In all of the above, the protected set is essentially stationary—fixed before the update—so the projector can be precomputed once and reused. In our setting, the protected set is the in-flight micro-batches, which turns over 13

every step; the projector must be rebuilt on the critical path under the per-stage budget of pipeline parallelism.

Proof of Theorem 4.1

We extend the perturbed forward–backward SGD analysis of Kong et al. [25] to a simplified model of F RESCO. We first bound the perturbations induced by asynchronous parameter changes and then derive the convergence rate under soft projection. B.1

Model and Assumptions

Consider a pipeline of P stages with linear parameterized operators fi (w, s) = Wi s. For an inf b c flight micro-batch, let Wi,t , Wi,t , and Wi,t denote the parameters used during forward propagation, backward propagation, and gradient application, respectively. The corresponding forward and backward perturbations are f f c δi,t := (Wi,t − Wi,t )si,t , (19) b b c ⊤ δi,t := (Wi,t − Wi,t ) e+,i,t .

These quantities measure the local computational discrepancies relative to the current parameters, holding the input feature and incoming error signal fixed. At update event u, let S̄i,u collect the protected in-flight features, with compact SVD S̄i,u = Ui,u Σi,u Q⊤ i,u . For analysis, we consider the soft-projection update ′ ∆Wi,u = ∆Wi,u Ai,u ,

⊤ Ai,u := (I + λi,u Ui,u,k Σ2i,u,k Ui,u,k )−1 .

(20)

Here ∆Wi,u = −ηGi,u is the proposed update, with Gi,u denoting the matrix representation of the asynchronously computed stage gradient. The following assumptions are used. Assumption B.1 (Smoothness). The objective L is bounded below by L⋆ and has an L-Lipschitz gradient: ∥∇L(w) − ∇L(w′ )∥ ≤ L∥w − w′ ∥. (21)

Assumption B.2 (Stochastic gradients). Let Ft denote the information available before sampling the ideal stochastic gradient, including the current parameters and update operator At . The unperturbed gradient gt satisfies E[gt | Ft ] = ∇L(wt ), (22) and

E[∥gt − ∇L(wt )∥2 | Ft ] ≤ σ 2 .

(23)

Assumption B.3 (Operator smoothness). For each operator fi , let D1 fi and D2 fi denote its input and parameter Jacobians. There exist finite constants C1 , C2 , Lf , L1 , L2 > 0 such that, for all computational states z = (s, wi ) and z ′ = (s′ , wi′ ) under consideration, ∥Dj fi (z)∥op ≤ C1 ,

j ∈ {1, 2},

2 ∥D21 fi (z)∥op ≤ C2 , ′

and

(24) (25)

′

∥fi (wi , s) − fi (wi , s )∥ ≤ Lf ∥s − s ∥,

(26)

∥Dfi (z) − Dfi (z ′ )∥F ≤ L1 ∥z − z ′ ∥,

(27)

∥D2 fi (z) − D2 fi (z ′ )∥F ≤ L2 ∥z − z ′ ∥.

These conditions apply to the differentiable computational graph, including the terminal loss operator. Assumption B.4 (Bounded computational signals). There exist finite constants Gi , Ei , Mi > 0 such that, almost surely, ∥Gi,t ∥F ≤ Gi ,

∥e+,i,t ∥ ≤ Ei , 14

∥S̄i,t ∥2 ≤ Mi .

(28)

Assumptions B.1–B.3 follow the framework of Kong et al. [25]. The stochastic-gradient condition is strengthened to condition on the update operator, which is required for the projected descent argument and is not automatic when the projection depends on the gradient sample. Assumption B.4 supplies uniform bounds on the quantities involved in asynchronous updates. We consider a fixed pipeline schedule for which the number of intervening stage updates is bounded. Let Hi,t and Ji,t denote the updates separating the forward and backward parameter versions from the current version. The execution model determines finite constants τif , τib satisfying |Hi,t | ≤ τif ,

|Ji,t | ≤ τib .

(29)

For the convergence analysis, we use a logical-update model that collects the stage-wise soft transformations into a block-diagonal operator At : (30)

wt+1 = wt − ηAt get .

This model retains the gradient perturbations induced by the pipeline while abstracting away the exact ordering of physical stage updates. For 0 ≤ λi,t ≤ Λ < ∞, the eigenvalues of Ai,t lie in [(1 + ΛM 2 )−1 , 1], where M := maxi Mi . Hence 1 aI ⪯ At ⪯ I, a := > 0. (31) 1 + ΛM 2 This bound follows from the soft-projection definition. B.2

Perturbation Bound and Convergence

We first characterize the residual forward perturbation. By the SVD of S̄i,u , Equation (20) gives 2 ′ Ri,u := ∥∆Wi,u S̄i,u ∥2F

k 2 X σi,u,j ∥∆Wi,u ui,u,j ∥2 2 (1 + λi,u σi,u,j )2 j=1

(32)

ri,u

j=k+1

Define

2 σi,u,j ∥∆Wi,u ui,u,j ∥2 .

(

σi,u,j , σi,u,k+1 si,u := max max 2 j≤k 1 + λi,u σi,u,j

) ,

(33)

with σi,u,ri,u +1 = 0 and empty maxima equal to zero. Then Ri,u ≤ si,u ∥∆Wi,u ∥F ≤ ηGi si,u .

(34)

E∥rt ∥2 ≤ η 2 (Kf s2 + Kb ),

(35)

Let s := supi,u si,u ≤ M . For hard rank-k projection, obtained as λ → ∞, the first sum in Equation (32) vanishes. In particular, protecting the complete activation subspace gives Ri,u = 0. Lemma B.5 (Asynchronous gradient error). Under Theorems B.3 and B.4 and the bounded-delay execution model, the gradient discrepancy rt := get − gt satisfies where Kf and Kb depend on the computational graph, update delays, and bounds on computational signals.

Proof. The difference between the forward and current parameter states is the sum of intervening updates. Since each such update protects the feature of the affected in-flight micro-batch, Equation (34) gives X f ∥δi,t ∥≤ Ri,u u∈Hi,t (36) ≤ ητif Gi s. 15

Similarly, since ∥Ai,u ∥2 ≤ 1,

b ∥δi,t ∥ ≤ ∥e+,i,t ∥

X u∈Ji,t

′ ∥∆Wi,u ∥2

(37)

≤ ητib Gi Ei . By the perturbation-propagation result of Kong et al. [25, Theorem 1], there exist nonnegative coefficients Cif , Cib such that X f E∥rt ∥2 ≤ Cif E∥δi,t ∥2 i

X i

b 2 Cib E∥δi,t ∥ .

(38)

Substituting Equations (36) and (37) proves the result with X Kf := Cif (τif Gi )2 , i

Kb :=

Cib (τib Gi Ei )2 .

(39)

The lemma bounds the squared gradient error by O(η 2 ), with its forward component depending on the residual spectral sensitivity s. Proof of Theorem 4.1. Write get = ∇L(wt ) + ξt + rt , where ξt := gt − ∇L(wt ) is the unperturbed stochastic-gradient noise. By L-smoothness and Equation (30), L(wt+1 ) ≤ L(wt ) − η⟨∇L(wt ), At get ⟩ +

Lη 2 ∥At get ∥2 . 2

Let ht := ∇L(wt ). By conditional unbiasedness, E[⟨ht , At ξt ⟩ | Ft ] = 0. Furthermore, Young’s inequality gives 1 1/2 1/2 −⟨ht , At rt ⟩ ≤ ∥At ht ∥2 + ∥At rt ∥2 , 4 and ∥At get ∥2 ≤ 2∥At ht ∥2 + 4∥At ξt ∥2 + 4∥At rt ∥2 .

Using aI ⪯ At ⪯ I and η ≤ 1/(4L), taking expectations in Equation (41) yields aη E∥∇L(wt )∥2 ≤ E[L(wt ) − L(wt+1 )] 2 3η + 2Lη 2 σ 2 + E∥rt ∥2 . 2

(40)

(41)

(42) (43) (44)

(45)

Summing over t = 0, . . . , T − 1, using L(wT ) ≥ L⋆ , and applying Theorem B.5, we obtain T −1

1 X 2∆L 4Lησ 2 E∥∇L(wt )∥2 ≤ + T t=0 aηT a 3η 2 (Kf s2 + Kb ) , + a where ∆L := L(w0 ) − L⋆ . √ Choosing η = c/ T proves Equation (18). 16

(46)

Table 1: Common experimental configuration. Hyperparameters marked as swept are selected separately for each method and pipeline depth. Configuration

Value

Model Architecture Embedding dimension d Attention heads a Transformer blocks L Context length s Parameters Positional embeddings LM head

nanoGPT decoder-only Transformer 384 6 32 512 ∼ 95M Learnable Untied

Training Dataset Micro-batch size b Precision Optimizer β1 (swept) β2 , ϵ Weight decay Gradient clipping Peak LR (swept) LR warmup LR decay Training horizon Random seed

OpenWebText 8 FP32 NAdamW {0.9, 0.99} 0.999, 10−8 0.01 1.0 {10−4 , 3 × 10−4 , 10−3 } 3,000 steps, linear Cosine to 0.1× peak LR 20,000 steps 0

Distributed Setup Pipeline depths P GPUs per run Training GPUs Timing GPUs Backend Activation recomputation

{1, 2, 4, 8, 16, 32} 8 RTX 3090 / RTX A6000 8× RTX 3090 (24 GB) gloo Disabled

The bound preserves the standard O(T −1/2 ) nonconvex stochastic convergence rate, with an additional O(T −1 ) term depending on the residual forward sensitivity and backward perturbations. It does not imply that stronger projection necessarily improves convergence, since the minimum retained update gain a also decreases with the protection strength. The guarantee concerns the simplified soft-projection model; convergence of the implemented hard projection under the full asynchronous execution dynamics remains outside its scope.

Experimental Details

C.1

Model and Training Configuration

Our experimental setup generally follows Ajanthan et al. [1]. We use a nanoGPT-style decoder-only Transformer [24] with L = 32 Transformer blocks, embedding dimension d = 384, a = 6 attention heads, and context length s = 512, comprising approximately 95M parameters. The model uses learnable positional embeddings and an untied language modeling head. At pipeline depth P , each stage contains L/P Transformer blocks, with the embedding layer assigned to the first stage and the LM head to the last. We train on OpenWebText for 20,000 optimizer steps with micro-batch size b = 8 and FP32 precision. We use NAdamW with weight decay 0.01, gradient clipping at 1.0, and a 3,000-step linear warmup followed by cosine decay to 0.1 times the peak learning rate. We disable activation recomputation for all methods. The common configuration is summarized in Table 1. 17

C.2

Hyperparameter Selection

Learning rate and momentum. For each method and pipeline depth P ∈ {1, 2, 4, 8, 16, 32}, we sweep the peak learning rate and first-moment coefficient over ηmax ∈ {10−4 , 3 × 10−4 , 10−3 },

β1 ∈ {0.9, 0.99}.

(47)

Each configuration is evaluated with a single seed (0). For the main results, we select the configuration with the lowest final training loss, measured by an exponential moving average (EMA) with decay factor 0.97, evaluated at step 19,900. Ablations over the learning rate, β1 , learning-rate damping, and the protection budget are reported in Section D. Protection budget. We select the protection budget α at the deepest pipeline depth, P = 32, over the full 20,000-step training horizon. Specifically, we sweep k . (48) 384 For each linear layer, the protected rank is ⌈α min(dout , din )⌉, with bias augmentation adding an input coordinate, so k = α · 384 for the 384-dimensional layers (k + 1 for the 385-dimensional ones). Each budget is evaluated using its best learning rate and β1 . The results are shown in Figure 7a. k ∈ {12, 24, 48, 64, 96, 154, 269, 384},

α=

We select α = 96/384 = 0.25 based on the lowest evaluation loss. Smaller budgets provide insufficient protection against forward interference, whereas larger budgets increasingly restrict the available update directions. Full-rank protection can reach a lower training loss (4.98 at a peak LR of 10−3 ) but then generalizes poorly (evaluation loss 6.19), which is why we select budgets by evaluation loss. We use α = 0.25 for the pipeline-depth sweep. For wall-clock comparisons, we instead use α = 48/384 = 0.125, which provides the best loss at a fixed wall-clock budget. For Recovery, we similarly sweep the discrepancy-correction coefficient over and use D = 0.02 throughout. C.3

D ∈ {0.01, 0.02, 0.05, 0.1},

(49)

Subspace Extraction and Projection

F RESCO uses bias-augmented projection: each input activation matrix gains a constant row of ones, X̄ = [X; 1⊤ ] ∈ Rd×n with d = din + 1, so that the projection also protects the bias term. With the protected rank k and τ in-flight micro-batches, each micro-batch contributes k1 = ⌈k/τ ⌉ directions. Subspace extraction. E XTRACT BASIS(X̄, k1 ) approximates the leading k1 left singular vectors of X̄ by block power iteration. From an initial block V0 ∈ Rd×k1 , we iterate Vj+1 = NSTns (X̄ X̄ ⊤ Vj ),

j = 0, . . . , Tbp − 1,

(50)

where NST normalizes its argument, C ← C/∥C∥F , and applies T Newton–Schulz steps C ← 1 ⊤ 2 C(3I − C C), and return V = VTbp . We use Tbp = 1 and Tns = 1, so each micro-batch costs two multiplications by its activations and a few small matrix products. Protected-basis construction. B UILD P ROTECTED BASIS(Q, k) combines the bases Q = {V (1) , . . . , V (τ ) } of the in-flight micro-batches into a rank-k orthonormal basis. Their concatenation C = [V (1) , . . . , V (τ ) ] ∈ Rd×τ k1 , τ k1 ≥ k, (51) spans up to τ k1 directions. When C has at most k columns, we take Q = qr(C); otherwise, we keep the dominant k directions of C by block power iteration from an initial block Q0 ∈ Rd×k , Qj+1 = qr(CC ⊤ Qj ),

j = 0, . . . , Tc − 1,

(52)

with Tc = 2, where qr returns the orthonormal factor of a Householder QR decomposition. The resulting Q ∈ Rd×k is used to project the proposed parameter update onto the complement of the protected subspace, ∆′ = ∆(I − QQ⊤ ). 18

β1 = 0.9

7.0

LRD

6.0 5.5 5.0

7.0

LRD

3e-4

6.0 5.5

1e-3

1e-4

β1 = 0.99

β1 = 0.9

3e-4

5.0

β1 = 0.9

LRD

Eval loss

5.5

Peak LR

(e) LWP

β1 = 0.99

β1 = 0.9

3e-4

5.5

1e-4

3e-4

1e-3

6.0 5.5

Peak LR

1e-4

3e-4

Peak LR

(f) SpecTrain

(g) F RESCO

1e-3

(d) PipeOptim Inconsistent Stash

LRD

5.0 1e-4

LRD

Peak LR

6.5

6.0

β1 = 0.9

6.0

Peak LR

5.0 1e-3

1e-3

7.0

Eval loss

5.5

3e-4

7.0

β1 = 0.99

5.0 1e-4

LRD

6.0

3e-4

5.5

1e-3

6.5

1e-4

6.0

(b) Parameter stashing

6.5

7.0

LRD

6.5

Peak LR

(a) Inconsistent

β1 = 0.9

5.0

Peak LR

7.0

β1 = 0.99

6.5

5.0 1e-4

Eval loss

β1 = 0.9

Eval loss

β1 = 0.99

6.5

Eval loss

β1 = 0.99

6.5

Selected peak LR

7.0

1e-3

Recovery Pipeoptim

LWP Spectrain

Fresco

1e-3 3e-4 1e-4

# of stages

(h) Selected LR per depth

Figure 6: Learning rate, momentum, and learning-rate damping. (a–g) Evaluation loss over the peak learning rate at P = 32 with β1 = 0.99 (solid) and β1 = 0.9 (dotted), both without LRD, and with LRD at the best β1 (dashed). (h) The peak LR selected for each method and pipeline depth in the main results (Figure 4); hollow markers denote β1 = 0.9, and coinciding lines are offset slightly for visibility.

C.4

Hardware and Pipeline Scheduling

Experiments are performed on a GPU cluster comprising 3 nodes with 8 NVIDIA RTX 3090 (24 GB) GPUs per node, with RTX A6000 GPUs also used for training runs. Each run uses 8 GPUs, and wall-clock measurements are conducted on a single node with 8 RTX 3090 GPUs. Communication uses the gloo backend. The pipeline depth varies over P ∈ {1, 2, 4, 8, 16, 32}. When P > 8, multiple virtual pipeline stages are assigned to each physical GPU, giving P/8 stages per GPU. This preserves the logical pipeline depth, the number of in-flight micro-batches, and the order of forward, backward, and update operations. The resulting wall-clock measurements may include additional context-switching overhead, which is shared across the compared methods.

Hyperparameter Ablations

We vary the peak learning rate, β1 , and learning-rate damping at P = 32, and report the learning rate selected at each depth. Each point is the lowest evaluation loss over β1 (unless varied) and the method-specific setting (α for F RESCO, D for Recovery); all runs use a single seed. D.1

Learning rate

Every method except Recovery attains its lowest evaluation loss at a peak LR of 3 × 10−4 (Figure 6), and F RESCO has the lowest loss at every learning rate. F RESCO is also the least sensitive to a larger step size: raising the LR from 3 × 10−4 to 10−3 increases its loss by 0.17, compared with 0.23–1.00 for the baselines. The three prediction-based methods degrade the most (PipeOptim 0.68, LWP 0.95, SpecTrain 1.00), consistent with larger updates making their predicted weights less accurate. Across depths (Figure 6h), Inconsistent, Stash, and Recovery select the smallest LR (10−4 ) at P = 8 and 16 (Recovery also at P = 32), and PipeOptim at P = 16. F RESCO instead selects the largest LR (10−3 ) up to P = 16 and 3 × 10−4 at P = 32, a 10× larger step than these baselines at P = 8 and 16. LWP and SpecTrain select 3 × 10−4 at every depth. 19

D.2

Momentum

β1 = 0.99 gives a lower loss than β1 = 0.9 for every method except SpecTrain (5.66 versus 5.47; Figure 6). The coefficient matters most for F RESCO (5.51 → 4.86) and LWP (5.91 → 5.45); for the other baselines the difference is at most 0.09. Across depths, every method that is swept over both values selects β1 = 0.99 (Figure 6h). Because the preferred β1 differs across methods, we tune it per method rather than fixing it. D.3

Learning-rate damping

We evaluate the staleness-aware learning-rate damping (LRD) of PipeMare [47], which sets the learning rate of a stage with delay τ to ηt τ −pt with pt = 1 − min(t/K, 1), where ηt is the scheduled learning rate at step t; we use K = 15,000. LRD improves every baseline except SpecTrain, which is essentially unaffected (5.47 → 5.46; Figure 6); the largest gain is for Recovery (5.96 → 5.34), whose discrepancy correction PipeMare proposes together with LRD. In contrast, LRD increases the evaluation loss of F RESCO from 4.86 to 4.99 and shifts its best budget to the smallest one, α = 12/384 (Figure 7c). One possible explanation is that LRD and the projection of F RESCO both attenuate the updates that would interfere with in-flight micro-batches, so that combining them over-damps optimization. Even when LRD is enabled for every baseline, F RESCO without LRD remains the best configuration, 0.39 below the strongest damped baseline (LWP, 5.24), although LRD reorders the baselines (e.g., Recovery moves from fourth to second). We therefore disable LRD for all methods in the main comparisons, so that every method uses the same learning-rate schedule. Protection budget Eval loss

Train loss

5.2 5.0

1e-4

LRD off

5.6

Peak LR

5.75

Eval loss

Loss

5.4

3e-4

LRD on

1e-3

Eval loss

D.4

5.50 5.25

5.4 5.2 5.0

5.00

Protected rank k

(a) Budget at P = 32

(b) Budget and LR

4.75

Figure 7: Protection budget of F RESCO over the protected rank k (α = k/384). (a) Evaluation loss and training loss (EMA 0.97 at step 19,900) at P = 32, each budget at its best peak LR and β1 by evaluation loss (3 × 10−4 and 0.99 for every budget). (b) One line per peak LR at P = 32. (c) LRD off and on at P = 32. Stars in (a) and (c) mark the best budget of each line. At P = 32, the evaluation loss is lowest at α = 96/384 and stays within 0.08 of it over α ∈ [48/384, 269/384] (Figure 7a). A peak LR of 3 × 10−4 gives the lowest loss at every budget (Figure 7b), so the budget can be selected at a single learning rate. At 10−3 , however, large budgets degrade sharply, up to an evaluation loss of 6.19 at full rank, while at 10−4 the loss varies by less than 0.2 across budgets.

FRESCO: A Novel Consistency Control for Asynchronous Pipeline Parallel Training

Analyze & explore related research in Conceptio

Related documents