ML-for-ML

ML-for-ML Yutong Zhao∗

Noga H. Rotman∗

UCL and BUPT United Kingdom

UCL United Kingdom

Gianni Antichi

Ran Ben Basat

Politecnico di Milano Italy

UCL and Broadcom United Kingdom

arXiv:2608.06046v1 [cs.NI] 6 Aug 2026

Abstract ML-for-ML Controller

AI training workloads are growing rapidly, making their time, energy, and infrastructure costs increasingly important. In shared cloud clusters, training and fine-tuning jobs compete with co-running workloads for network resources, while network mechanisms and ML training choices are typically optimized separately: networking controls how bytes move, whereas ML systems control when and how much communication occurs. We argue that this separation leaves end-to-end performance on the table. We present ML-for-ML, a cross-layer perspective in which network-side and ML-side knobs are selected jointly under a shared time-to-target-loss objective. Our preliminary prototype shows that by co-optimizing the ML and network parameters, we reach the target loss up to 42% faster.

1

Observe Network stats

Observe ML stats

Jointly Optimizes ML + Networking A Unified Objective

Joint Decision

Apply Apply Network ML decisions decisions

ML Knobs

Network Knobs

Batch size

Sending rate

Learning rate

Routing

Local steps

Compression

…

…

hhhh(((( (Networking-for)-ML Figure 1: The envisioned ML-for-( (((hhhh loop: the controller observes training progress and network conditions, jointly selects ML-side and networkside knobs under a unified objective, and applies the resulting decisions to the training job and the network. Updated ML and network measurements then inform the next decision, closing the loop.

Introduction

Many distributed machine learning (ML) training jobs run in shared cloud environments [17], academic clusters [38], or multi-tenant datacenter environments [15], where the same network fabric carries traffic from co-running workloads and background services. In such environments, ML training traffic competes with other workloads for network bandwidth [5, 8], and network conditions can change throughout training [33]. Adapting to this variability by deciding when, how much, and how to communicate is, therefore, critical to achieving high training performance. Traditionally, this problem has been addressed primarily from the networking perspective. Congestion control [4, 39], network-aware scheduling [27], and routing/topology optimization [2] adapt the network to accommodate the communication demands of distributed training, mitigating contention and improving utilization and training performance. These mechanisms react to changing network conditions but treat the communication workload as an external input. However, the communication workload is itself programmable. In addition to statistical efficiency, ML-side choices such as batch size [9, 20], gradient accumulation [22], and local steps [21, 31] affect when or how much data is exchanged

across the network. Rather than relying solely on networkside mechanisms to react to changing load, we argue that they should be selected jointly with ML-side decisions. The objective is ultimately shared: reaching a target model quality while minimizing costs such as execution time or energy. Improvements on one side naturally create opportunities on the other. For example, a less contended network may make more frequent synchronization preferable, while lowering synchronization frequency or batching decisions can directly reduce network contention. The problem lies at the intersection of two active but largely separate communities. ML-for-networking uses ML to improve network mechanisms such as congestion control [14, 37] and traffic engineering [24, 32]. Networking-forML improves transport and rate control [6, 39], in-network aggregation [8, 16], and network-aware communication and job scheduling [5, 27] for training workloads. We envision a unified loop, as illustrated in Figure 1, in which the training system observes its progress and network conditions and

∗ Equal contribution.

1

Yutong Zhao, Noga H. Rotman, Gianni Antichi, and Ran Ben Basat

optimizes network-side and ML-side knobs jointly under the shared control objective. Realizing this loop introduces several challenges. First, the usefulness of a knob combination depends on both training progress and network load. Second, decisions must be made online, as the future background traffic load is unknown. Finally, estimating the training progress (e.g., validation loss) or network conditions (e.g., background load) requires resources such as running a validation set or injecting telemetry probes, which compete with the job’s own resources. As a first step toward this vision, this paper investigates whether optimizing network-side and ML-side knobs independently is sufficient. Our methodology evaluates candidate combinations by asking how each combination changes both foreground-visible communication time and training progress. Our case study instantiates this with two representative knobs: communication volume on the network side and batch sizing on the ML side. Across the evaluated background-load traces, we show that composing independently selected knob settings takes 1.13–1.42× the time required by joint optimization to reach the target quality. More broadly, as distributed training scales and diversifies, we envision broadening the scope of ML-for-ML to include richer joint action spaces, broader objectives and workloads, other deployment settings, and coordination among multiple adaptive jobs. As the networking-for-ML and MLfor-networking communities mature, we call for mechanisms and interfaces that unite the communities around shared end-to-end goals.

2

2.1

Control Objective

Although our framework can optimize for other objectives, such as minimizing energy consumption, for ease of exposition, we hereafter consider the time-to-target loss goal: minimizing the wall-clock time for the foreground model to reach a fixed evaluation loss. This end-to-end objective depends jointly on training progress, measured by evaluation loss, and the wall-clock cost of training, including both local computation and network communication. We therefore assess progress toward the target loss as a function of elapsed wall-clock time.

2.2

Why Independent Knob Optimization Can Fail

A natural baseline is to optimize one knob at a time, holding all other knobs at their baseline settings, and then compose the resulting choices. However, this combination need not be the best choice over the full joint action space because network-side and ML-side knobs are generally not separable. A knob choice can change communication volume, communication timing, update frequency, or training dynamics, thereby altering the effects of the other knobs. A value that appears preferable when evaluated at the baseline settings can therefore become suboptimal once combined with other independently selected values. Conversely, the best joint combination can include a value that does not appear best when evaluated alone. For example, communication compression can reduce communication cost but may also alter training progress. Batchsize choices can change how often the model is updated and how often workers communicate, as well as the resulting loss trajectory. These effects are coupled: compression can provide greater benefit when communication is frequent, but less additional benefit under a batch-size setting that already communicates less often. Conversely, once compression has reduced communication cost, a batch-size setting that further reduces communication frequency may no longer offer the best tradeoff between communication cost and training progress. Thus, the value of each setting depends on the other, and the settings preferred when the knobs are evaluated separately need not form the best combination. This motivates evaluating network-side and MLside knobs jointly, rather than composing independently selected single-knob settings.

ML-for-ML

This section defines the general joint optimization problem over network-side and ML-side knobs underlying the MLfor-ML loop. The controller’s action space consists of combinations of network-side and ML-side knobs. Table 1 lists its representative examples. The preferred knob combination depends on the current training phase, the network’s background load, and interactions among the knobs. Because these conditions change over time and differ across workloads, a combination selected before training need not remain preferable throughout the run. A learned controller provides a natural way to address this state-dependent decision problem. It can learn from prior observations and periodically update the joint choice during training, rather than optimize the network-side and ML-side knobs in isolation. The remainder of this section characterizes the joint decision structure that such a controller must capture. We state the control objective, analyze why independent knob tuning can fail, describe what the controller should observe, and explain how those observations lead to joint optimization.

2.3

Observing the Training Process and Network Conditions

Choosing a combination setting of knobs requires observations from both sides of the system. The controller may draw on both raw system signals and prior knowledge. Networkside inputs may include recent collective durations, active 2

ML-for-ML

Table 1: ML-side and network-side control knobs, with representative works highlighting their impact on distributed training performance Type ML

Possible knobs Batch size [9, 20]; gradient accumulation [22]; learning rate [18, 29]; local steps [21, 31]; worker synchronization / participation [7, 12] Network Sending rate [4, 39]; routing [2]; communication precision/compression [1, 10, 11, 28]; redundancy / reliability trade-off [34]; bandwidth allocation / prioritization [3] probes, NIC counters, switch telemetry, queueing depth, ECN marks, or packet-level delay. ML-side inputs may include loss trends, step times, sample efficiency, or known loss costs. Network conditions and training dynamics, including background traffic, training phase, and communication placement, change throughout a run, so a one-time profile may quickly become outdated. A deployed controller must therefore refresh its observations and make decisions periodically over short intervals. The duration of each interval is constrained by the time required to collect the relevant network and loss telemetry and adapt the knobs. It must also account for the delay between observation and actuation, as conditions may change before a selected combination takes effect. Rather than reacting only to the latest measurement, the controller can use current and historical observations to anticipate the conditions under which the decision will take effect.

2.4

Table 2: Representative policy choices during training under contention generated by co-running GPT-1B jobs. Policy Static Knob-Precision Knob-GA Decoupled Joint

Rule fixed precision only GA only compose projections joint optimization

Representative choice GA4-BF16 GA4-MXFP4 GA16-BF16 GA16-MXFP4 GA4-MXFP4

single-knob decisions. Sensing, prediction, and actuation are needed to implement this rule in a full online controller; the methodology here focuses on the joint comparison that such a controller relies on.

3

A Two-Knob Case Study

We instantiate the methodology with a two-knob case study: one network-side knob that changes communication volume, and one ML-side knob that changes batch sizing. This small combination space lets us test the key question directly: whether independently chosen knobs can compose into a suboptimal system-level outcome, and whether joint optimization avoids it.

From Observations to Joint Optimization

The controller does not merely need a scalar label such as “congested.” It needs to translate network and ML signals into estimates that distinguish among candidate combinations, because the same observed state may affect candidate combinations in different ways. As a practical design choice, we use a greedy per-interval decision rule as a proxy for the global time-to-target objective. At each decision point, the controller estimates the expected loss reduction over the upcoming interval for every combination of network-side and ML-side knob settings under the anticipated training and network conditions. These estimates account for both the foreground-visible communication time and the training progress achievable within the interval. The controller then selects the combination with the largest estimated loss reduction. Although this local rule is not an exact solution to the global objective, it provides a practical online rule for comparing combinations and updating choices as observations change. This comparison provides the basis for testing the necessity of joint optimization while accounting for both timing and training effects. It evaluates whether network-side and ML-side knob combinations lead to different choices from those obtained by composing independently optimized

3.1

Case-Study Setup

Control knobs. The network-side knob is communication precision 𝑝, the numeric format of each gradient exchange, which changes communication volume. Here, the action space is 𝑝 ∈ {BF16, MXFP4} [23]. The ML-side knob is the gradient accumulation factor 𝑔, the number of microbatches accumulated before a data-parallel (DP) synchronization and optimizer update. It changes the batch size and how often the training job synchronizes. Here, 𝑔 ∈ {4, 16}, corresponding to GA4 and GA16, respectively. Across GA settings, the learning rate follows the square-root scaling rule [13]. Foreground workload. The foreground task we set is GPT2 Large [26], a 774M-parameter model with 36 layers, a hidden size of 1280, 20 attention heads, and a sequence length of 1024. Policies and targets. Based on these two knobs, we compare five policies throughout. Static fixes the baseline combination at GA4-BF16. Knob-Precision only varies precision 3

60 40 20 0

GA4 BF16 GA4 MXFP4

4

8

12

16

2.62 2.61 2.60 2.59 2.58 2.57

GA4 BF16 GA16 BF16 1.02x

1.01x

2

4

6

8

10

Static Knob-GA Decoupled Knob-Precision/ Joint

200 150 100 50 0

1.40x

1.21x

1.16x

1.12x

4

8

12

16

Time to target (min)

80

Time to target (min)

100

Eval loss

Exposed DP share of iteration (%)

Yutong Zhao, Noga H. Rotman, Gianni Antichi, and Ran Ben Basat

500 400 300 200 100 0

Static Knob-GA Decoupled Knob-Precision/ Joint 1.42x

1.22x

1.18x

1.13x

4

8

12

16

Number of background jobs

Optimizer updates (thousands)

Number of background jobs

Number of background jobs

(a) DP exposure

(b) GA tradeoff

(c) 1.02 × 𝐿min

(d) 1.01 × 𝐿min

Figure 2: Policy behavior under contention generated by co-running GPT-1B jobs. Panel (a) shows that, under fixed GA4, compression reduces the exposed DP share of each foreground-job iteration from 73–91% with BF16 to 40–58% with MXFP4. Panel (b) shows the complementary GA tradeoff under fixed BF16: larger GA reduces communication frequency but changes the number of optimizer updates required to reach a target loss. Panels (c) and (d) report end-to-end first-crossing times to 1.02𝐿min and 1.01𝐿min , respectively, where 𝐿min = 2.55048 is the final evaluation loss measured for the Static baseline. Joint coincides with Knob-Precision and is shown as the blue bar, while Decoupled composes the two single-knob projections and is consistently slower. 𝑝 while holding the ML-side knob at its baseline setting; Knob-GA only varies 𝑔, which raises the batch size and lowers the synchronization frequency together, while holding the network-side knob at its baseline setting. Decoupled optimizes each knob independently and composes the two resulting values, while Joint evaluates over the full combination space. We use 𝐿min = 2.55048, the final evaluation loss measured for the Static baseline, as the common reference for the target-loss thresholds used to compare all policies. Timing model and measurement setup. We obtain wallclock evaluation-loss trajectories by combining separate measurements of computation, communication, and training progress. The foreground iteration time under network condition 𝑧 is modeled as

3.2

The Cost of Decoupled Optimization

This scenario compares single knob, decoupled optimization, and joint optimization to assess the end-to-end impact of coordinating the two knobs. The background traffic is recorded from concurrently running GPT-1B jobs that alternate between non-DP computation and DP communication, share the network, and interact through congestion control. The resulting traces preserve the phase-structured and bursty contention produced by co-running training workloads. We vary the number of background jobs, 𝑁 ∈ {4, 8, 12, 16}. To characterize the resulting network load, we report 95th-percentile background traffic volumes of 17.55, 22.02, 25.83, and 26.16 GB per 50 ms, respectively. For each value of N, we replay the foreground job against the corresponding traffic trace at four relative start offsets and average the exposed DP time across the replays, reducing sensitivity to phase alignment. Table 2 captures the five policies’ representative choice during training. Panels (a) and (b) of Figure 2 show the two one-dimensional projections underlying these choices. At this point, the precision projection prefers MXFP4 at fixed GA4, while the GA projection prefers GA16 at fixed BF16. The resulting choices are Knob-Precision → MXFP4 and Knob-GA → GA16, which Decoupled composes into GA16MXFP4. Joint, by contrast, evaluates all four combinations directly. Under MXFP4, the remaining exposed DP cost is low enough that the additional communication savings from GA16 do not offset its training-progress cost. Joint therefore chooses GA4-MXFP4, matching the precision-only projection but differing from Decoupled. Panels (c) and (d) report end-to-end first-crossing times for the five policies at 1.02𝐿min and 1.01𝐿min . At both targets, Decoupled is slower than Joint, while Knob-Precision matches Joint. At 1.01𝐿min , the first-crossing-time ratios

𝑇iter (𝑔, 𝑝, 𝑧) := 𝑔𝑇𝑐 + 𝑇dp (𝑝, 𝑧). We profile 𝑇𝑐 = 59.3 ms, the average per-microbatch compute time of our foreground job, from forward-backward passes on A100 GPUs. 𝑇dp (𝑝, 𝑧) denotes the DP time exposed to the foreground job after accounting for overlap with local computation. We estimate 𝑇dp (𝑝, 𝑧) using SimAI/NS-3 replays [36] driven by co-run traffic traces, with the simulator configured to match the A100-cluster setting. Furthermore, we obtain the corresponding evaluation-loss trajectories from PyTorch training runs under the knob choices of each policy. With a faster GPU, 𝑇𝑐 decreases and communication accounts for a larger share of iteration time. This may shift the controller’s preferred knob combination, while the formulation remains unchanged. We evaluate the policies in two complementary settings: one examines the insufficiency of composing the knobs independently, while the other examines how the preferred joint choice differs across contention conditions. 4

Eval loss

ML-for-ML

2.90 2.85 2.80 2.75 2.70 2.65 2.60 2.55

2.61 2.60 2.59 2.58 2.57 2.56

0

500

Static Joint Decoupled Knob-Precision Knob-GA

1.02x 1.015x 1.01x 1.005x

500

1000

1000

1500

1500

2000

2000

2500

Simulated wall-clock time (min)

2500

3000

3500

4000

Figure 3: Evaluation-loss trajectories under the contention schedule generated by co-running GPT-13B jobs. Adaptive policies update their knob settings over the course of the run. The inset highlights first-crossing times for the four common target losses, 1.005, 1.01, 1.015, and 1.02 times 𝐿min . relative to Joint across 𝑁 ∈ {4, 8, 12, 16} are 1.13–1.42× for Decoupled, 1.71–3.65× for Static, and 1.54–2.26× for KnobGA. As 𝑁 increases, the relative slowdowns of Static and Knob-GA increase with 𝑁 , because both retain BF16 as communication pressure grows. By comparison, Decoupled’s absolute first-crossing time increases, but its slowdown relative to Joint narrows from 1.42× to 1.13×. As traffic becomes heavier, the value of reducing synchronization frequency increases, narrowing this relative gap.

3.3

This changing tradeoff is reflected in the choices Joint makes. GA4-MXFP4 remains its most frequent selection at every contention level, but the share of GA16-MXFP4 rises monotonically with the number of background jobs, from 0% with no background traffic to 40% at the highest level. Under light contention, MXFP4 reduces the exposed DP cost enough that the additional communication savings from GA16 do not offset its training-progress cost; as contention grows, the residual DP cost after compression stays high enough that less frequent synchronization becomes worthwhile for a growing fraction of updates. Decoupled, by contrast, selects GA16-MXFP4 for 43–58% of its updates based on the additive combination of the single-knob choices. Figure 3 compares the evaluation-loss trajectories of the five policies. We report first-crossing times at four common target losses reached by all policies: 1.005, 1.01, 1.015, and 1.02 times 𝐿min = 2.55048. Across all four targets, Joint reaches each target first. Relative to Joint, the slowdown ranges from 5.4% to 20.8% for Decoupled and 22.7% to 36.3% for Knob-Precision, while Knob-GA and Static are slower by 4.4–4.9× and 5.3–6.3×, respectively. The two single-knob policies capture complementary benefits: Knob-Precision reduces communication volume via compression, while Knob-GA reduces synchronization frequency. Each is useful on its own: across the four targets, Knob-Precision is 5.1–5.6× faster than Static and KnobGA is 1.1–1.3× faster. Decoupled takes both and reaches 5.2– 6.8×, while Joint reaches 6.3–7.3× from the same two knobs. The remaining margin is therefore not a knob that Decoupled lacks: it comes from re-evaluating the pair as contention shifts the residual communication cost after compression, which composing two independent projections cannot do.

A Moving Optimum

We next examine whether the preferred joint combination differs across contention levels for the same training job. This scenario spans a wide range of foreground-visible communication pressure, so a combination that is preferred at one level need not be preferred at another. We construct contention levels from zero to five concurrently running GPT-13B training jobs. Each co-running job follows its measured computation and DP-communication cycle, preserving the temporal structure of the training workload. Jobs arrive at a Poisson rate of 0.002 s−1 and release the network when they finish, so the number of active jobs varies over the run, averaging 2.7. All policies experience the same arrival schedule for a fair comparison, and the adaptive policies update their choices as training progresses and foreground-visible network conditions change. The schedule produces substantial variation in the communication cost exposed to the foreground job. Compression reduces communication exposure at every contention level, but leaves a larger residual DP cost under stronger contention. As this residual cost grows, the communication savings from reducing synchronization frequency through a larger GA factor can become large enough for less frequent synchronization to be worthwhile. 5

Yutong Zhao, Noga H. Rotman, Gianni Antichi, and Ran Ben Basat

3.4

provide an evaluation-loss trajectory against which networkside and ML-side choices can be assessed, so time to a target evaluation loss remains a suitable objective. Serving is a related extension in which the objective shifts from loss reduction to outcomes such as latency, throughput, and quality. ML-side choices such as batching, precision, parallelism, and placement continue to interact with network-side choices such as routing, prioritization, and transport behavior. The corresponding research question is how to coordinate these choices under changing request workloads and service-level objectives, where loss is no longer the feedback signal.

Takeaways

Taken together, our initial results show that treating individual knobs independently does not reliably identify the best combination, and that the optimum changes with contention. By jointly optimizing the network-side and ML-side knob settings, Joint achieves the best observed first-crossing time across all reported target losses.

4

Discussion

Treating our case study as a starting point, we outline the research directions opened up by this control interface: extending the ML-for-ML perspective to richer control spaces, broader objectives and workloads, other deployment settings, and multiple adaptive jobs.

Interaction with other jobs. Our experiments study one foreground job adapting to background traffic. A shared cluster raises the question of how the system behaves when background jobs adapt too. For example, bandwidth-hungry jobs might increase their sending rates as a direct consequence of our job reducing it. While we expect ML-for-ML to allow multiple foreground jobs and be a fair network citizen, running alongside inconsiderate or adversarial flows may pose interesting challenges.

Toward a knob taxonomy for ML-for-ML. The two knobs studied in this paper cover only a small part of a broader design space. Network-facing controls include routing, collective algorithms, compression, and congestion response; training-facing controls include gradient accumulation, batch size, checkpointing, optimizer choice, and learningrate schedules; resource-facing controls include GPU count, placement, scheduling, and power caps. The community needs a set of knobs that should be controlled together, while taking interactions into account, to ensure overall utility.

5

Related Work

Network-aware optimizations for ML systems. Prior work focuses on network-side mechanisms to accommodate the heavy communication demands of distributed training. These include transport-layer rate control [4, 39], networkaware job scheduling [27], and routing or topology optimization [2]. Collectively, these approaches optimize network behavior while treating the training process and the communication demand it generates as given.

Beyond time: resources, energy, and cost. We use timeto-target loss because it is a clean objective shared by networking and ML decisions, but it is not the only one. In cloud and multi-tenant settings, users and operators may also care about GPU-hours, dollar cost, energy consumption, power limits, fairness, or meeting deadlines under a fixed budget. Once the objective expands, the resource-facing controls above become first-class: GPU allocation, placement, and power caps are direct actions for trading time against cost and energy. A broader agenda, therefore, asks not only which action minimizes time-to-target loss but also which action best balances user- and cluster-level objectives.

Communication-efficient training optimization. A complementary direction adapts the training process to reduce or hide communication overhead. These techniques include gradient quantization and sparsification [1, 28], local steps [21, 31], and gradient accumulation [22]. While some approaches jointly optimize internal ML knobs (e.g., batch size and learning rate [30]), they do not explicitly model, provide visibility into, or dynamically alter network behavior.

Beyond datacenter training. We focus on distributed training over a shared datacenter network, but the jointcontrol perspective extends to other settings. In federated learning, for example, participating workers differ in bandwidth, latency, compute, and availability, so the relevant system state is participant-specific rather than summarized by a single shared background-load signal. The same principle applies, with selection made per participant rather than once for the job, which raises the question of whether effective per-participant decisions can be made without a global view of the system.

Adaptive and joint optimization. Closest to our work are runtime optimization frameworks. For instance, systems like KungFu [19] dynamically tune training configurations and execution strategies, while Pollux [25] co-adapts training configuration and resource allocation to optimize training goodput. Adaptive distributed algorithms also adjust communication frequency based on communication cost and training progress [35]. However, these approaches optimize training-side or resource-allocation choices in response to observed system conditions, rather than jointly optimizing both control planes.

Across training stages and into serving. The current design applies to both pre-training and fine-tuning: both 6

ML-for-ML

[16] Minghao Li, Ran Ben Basat, Shay Vargaftik, ChonLam Lao, Kevin Xu, Michael Mitzenmacher, and Minlan Yu. 2024. THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, 1191–1211. https: //www.usenix.org/conference/nsdi24/presentation/li-minghao [17] Edo Liberty, Zohar Karnin, Bing Xiang, Laurence Rouesnel, Baris Coskun, Ramesh Nallapati, Julio Delgado, Amir Sadoughi, Yury Astashonok, Piali Das, et al. 2020. Elastic machine learning algorithms in amazon sagemaker. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 731–737. [18] Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations. https://openreview.net/forum?id=Skq89Scxx [19] Luo Mai, Guo Li, Marcel Wagenländer, Konstantinos Fertakis, AndreiOctavian Brabete, and Peter Pietzuch. 2020. KungFu: Making training in distributed machine learning adaptive. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). 937–954. [20] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. 2018. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162 (2018). [21] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 54). PMLR, 1273–1282. [22] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on GPU clusters using Megatron-LM. In Proceedings of the international conference for high performance computing, networking, storage and analysis. 1–15. [23] Open Compute Project. 2023. OCP Microscaling Formats (MX) Specification. Specification Version 1.0. Open Compute Project. https://www.opencompute.org/documents/ocp-microscalingformats-mx-v1-0-spec-final-pdf [24] Yarin Perry, Felipe Vieira Frujeri, Chaim Hoch, Srikanth Kandula, Ishai Menache, Michael Schapira, and Aviv Tamar. 2023. DOTE: Rethinking (predictive) WAN traffic engineering. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 1557–1581. [25] Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R Ganger, and Eric P Xing. 2021. Pollux: Co-adaptive cluster scheduling for goodputoptimized deep learning. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21). USENIX Association, 1–18. [26] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Technical Report. OpenAI. https://cdn.openai.com/better-language-models/language_models_ are_unsupervised_multitask_learners.pdf [27] Sudarsanan Rajasekaran, Manya Ghobadi, and Aditya Akella. 2024. CASSINI: Network-Aware job scheduling in machine learning clusters. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 1403–1420. [28] Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 2014. 1Bit Stochastic Gradient Descent and Its Application to Data-Parallel Distributed Training of Speech DNNs. In Proceedings of Interspeech. 1058–1062. [29] Leslie N Smith. 2017. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV). IEEE, 464–472.

References [1] Alham Fikri Aji and Kenneth Heafield. 2017. Sparse communication for distributed gradient descent. In Proceedings of the 2017 conference on empirical methods in natural language processing. 440–445. [2] Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat. 2008. A scalable, commodity data center network architecture. ACM SIGCOMM computer communication review 38, 4 (2008), 63–74. [3] Mohammad Alizadeh, Shuang Yang, Milad Sharif, Sachin Katti, Nick McKeown, Balaji Prabhakar, and Scott Shenker. 2013. pFabric: Minimal near-optimal datacenter transport. ACM SIGCOMM Computer Communication Review 43, 4 (2013), 435–446. [4] Joao Araujo, Alex Chow, Mark Handley, Ryder Lewis, Christoph Paasch, Jitendra Padhye, Michael Papamichael, Greg Steinbrecher, Amin Tootoonchian, Lihua Yuan, et al. 2026. Resilient AI Supercomputer Networking using MRC and SRv6. arXiv preprint arXiv:2605.04333 (2026). [5] Jiamin Cao, Yu Guan, Kun Qian, Jiaqi Gao, Wencong Xiao, Jianbo Dong, Binzhang Fu, Dennis Cai, and Ennan Zhai. 2024. Crux: GPU-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference. 1–15. [6] Xiaoqi Chen, Shay Vargaftik, and Ran Ben Basat. 2024. When ML Training Cuts Through Congestion: Just-in-Time Gradient Compression via Packet Trimming. In Proceedings of the 23rd ACM Workshop on Hot Topics in Networks (HotNets ’24). 177–185. doi:10.1145/3696348.3696880 [7] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. 2012. Large scale distributed deep networks. In Advances in neural information processing systems, Vol. 25. 1232–1240. [8] Nadeen Gebara, Manya Ghobadi, and Paolo Costa. 2021. In-network aggregation for shared machine learning clusters. In Proceedings of Machine Learning and Systems, Vol. 3. 829–844. [9] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677 (2017). [10] Wenchen Han, Shay Vargaftik, Michael Mitzenmacher, and Ran Ben Basat. 2026. DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce. In Proceedings of the 2026 ACM conference on SIGCOMM. [11] Wenchen Han, Shay Vargaftik, Michael Mitzenmacher, Brad Karp, and Ran Ben Basat. 2024. Beyond throughput and compression ratios: Towards high end-to-end utility of gradient compression. In Proceedings of the 23rd ACM Workshop on Hot Topics in Networks. 186–194. [12] Qirong Ho, James Cipar, Henggang Cui, Seunghak Lee, Jin Kyu Kim, Phillip B Gibbons, Garth A Gibson, Greg Ganger, and Eric P Xing. 2013. More effective distributed ML via a stale synchronous parallel parameter server. In Advances in neural information processing systems, Vol. 26. 1223–1231. [13] Elad Hoffer, Itay Hubara, and Daniel Soudry. 2017. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Advances in neural information processing systems 30 (2017). [14] Nathan Jay, Noga H. Rotman, P. Brighten Godfrey, Michael Schapira, and Aviv Tamar. 2019. A Deep Reinforcement Learning Perspective on Internet Congestion Control. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97). PMLR, 3050–3059. [15] Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. 2019. Analysis of LargeScale Multi-Tenant GPU clusters for DNN training workloads. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). 947–960. 7

Yutong Zhao, Noga H. Rotman, Gianni Antichi, and Ran Ben Basat

[30] Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. 2018. Don’t Decay the Learning Rate, Increase the Batch Size. In International Conference on Learning Representations. https://openreview. net/forum?id=B1Yy1BxCZ [31] Sebastian U Stich. 2019. Local SGD Converges Fast and Communicates Little. In International Conference on Learning Representations. https: //openreview.net/forum?id=S1g2JnRcFX [32] Asaf Valadarsky, Michael Schapira, Dafna Shahaf, and Aviv Tamar. 2017. Learning to route. In Proceedings of the 16th ACM workshop on hot topics in networks. 185–191. [33] Raajay Viswanathan, Arjun Balasubramanian, and Aditya Akella. 2020. Network-accelerated distributed machine learning for multi-tenant settings. In Proceedings of the 11th ACM Symposium on Cloud Computing. 447–461. [34] Ashish Vulimiri, Philip Brighten Godfrey, Radhika Mittal, Justine Sherry, Sylvia Ratnasamy, and Scott Shenker. 2013. Low latency via redundancy. In Proceedings of the ninth ACM conference on Emerging networking experiments and technologies. 283–294. [35] Jianyu Wang and Gauri Joshi. 2019. Adaptive communication strategies to achieve the best error-runtime trade-off in local-update SGD. In Proceedings of Machine Learning and Systems, Vol. 1. 212–229.

[36] Xizheng Wang, Qingxu Li, Yichi Xu, Gang Lu, Dan Li, Li Chen, Heyang Zhou, Linkang Zheng, Sen Zhang, Yikai Zhu, et al. 2025. SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and Precision. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). 541–558. [37] Keith Winstein and Hari Balakrishnan. 2013. TCP ex Machina: Computer-Generated Congestion Control. ACM SIGCOMM Computer Communication Review 43, 4 (2013), 123–134. [38] Kaiqiang Xu, Decang Sun, Hao Wang, Zhenghang Ren, Xinchen Wan, Xudong Liao, Zilong Wang, Junxue Zhang, and Kai Chen. 2025. Design and operation of shared machine learning clusters on campus. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 295–310. [39] Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. 2015. Congestion control for large-scale RDMA deployments. ACM SIGCOMM Computer Communication Review 45, 4 (2015), 523–536.

8

Related documents

• Maximum Likelihood Estimator#137705
• A generalization of the maximum likelihood expectation maximization (MLEM) method: Masked-MLEM#491857
• ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs#660867
• Poisson Image Denoising Using Minimax Concave and Reweighted $\ell_1$ Penalties: Nonblind and Blind Approaches#985326
• Equivariant Neural Belief Propagation#259480