Network Contention State Aware Dynamic Optimizer for Accelerator Collective Communication Libraries

Network Contention-Aware Dynamic Optimizer for Accelerator Collective Communication

arXiv:2611.10984v1 [cs.DC] 7 Nov 2026

Rajat Bisht Tennessee Technological University Anthony Skjellum Tennessee Technological University

Tevfik Kosar University at Buffalo, Buffalo

Gene Cooperman Northeastern University

Abstract

hybrid combinations thereof [42, 52, 54]. These paradigms require periodic synchronization using collective communication operations [19, 39, 41] (e.g., AllReduce, ReduceScatter, AllGather), invoked repeatedly to exchange gradients during training, and intermediate results during inference. The resulting communication overhead is substantial, with collective operations accounting for up to 80% of total execution time in large model training [30] and nearly two-thirds of a typical training cycle [21]. Prior efforts to address the communication bottleneck in distributed GPU training fall into the following three broad categories: (a) efficient communication algorithms (e.g., topology-aware [9, 10], Pareto-optimal [5], data compression [1, 2, 30], quantization [45]); (b) computationcommunication overlap techniques [20, 26, 31, 38]; and (c) low-level optimized communication library stacks [13, 18, 35, 39, 41]. We identify a fundamental limitation common to all the three categories: an implicit assumption that network conditions are stationary, leading to systematically suboptimal communication decisions under real-world network contention. This observation motivates us to explore a novel category of optimization, which we instantiate on NCCL [35] given its status as the de facto standard in distributed AI training and its growing adoption within the HPC community. The stationary network assumption manifests in two distinct limitations. First, heuristic-based statically tuned libraries (like NCCL and RCCL) select communication configurations optimized for peak hardware bandwidth and minimum latency [19, 41], without any mechanism to detect or respond to changes in available bandwidth and transmission delays due to network contention at runtime. As a result, when network contention arises, these libraries continue operating under configurations chosen for ideal conditions and systematically under-perform, unaware of the degraded network state. Second, run-time tuning approaches for online searching of optimal configuration parameter (such as AutoCCL [50]) address this static heuristics limitation by measuring runtime bandwidth and latency telemetry of initial few cycles. However, this approach carries significant tuning overhead and cap-

Collective Communication Libraries (CCL) are critical for large-scale distributed AI/ML training, yet their configurations are determined by static heuristics that do not account for real-time network conditions. Our empirical study shows that these heuristics perform near-optimally in contention-free environments, but degrade substantially under background traffic. To address this, we present a cooperative multi-agent run-time optimizer for accelerator collective communication. Our approach employs a two-agent architecture: (i) detector – a Long Short Term Memory (LSTM) recurring neural network fed classifier head with a sequential probability ratio test that distinguishes sustained contention from transient spikes; and (ii) selector – a configuration selector built on a Dueling Double DQN (D3QN) with Conservative Q-Learning that adapts configuration parameters in response. We implement this dynamic tuner over NCCL, a widely used CCL developed by NVIDIA. When evaluated on Chameleon Cloud’s isolated bare-metal cluster, this dynamic tuning approach reduces AllReduce latency by 36% over NCCL and 32% over a fully converged AutoCCL. We further evaluate it across six real AI workloads spanning convolutional networks, transformers, and recommendation architectures. Our multi-agent tuner approach delivers up to 65% speedups on Delta cluster with NVIDIA A100 GPUs and Slingshot-11 fabric internode connectivity.

1

MD S Q Zulkar Nine Arkansas Tech University

Introduction

The rapid proliferation of AI/ML workloads and LLM-based agentic systems have driven model sizes to scales that fundamentally exceed the memory and compute capacity of an individual accelerator, both during training as well as during inference [19, 21, 27, 39, 53]. To overcome this constraint, researchers and industry practitioners have turned to distributed GPU architectures, employing various training paradigms such as data parallelism (DP) [25, 37, 38], tensor parallelism (TP) [32, 40, 51], pipeline parallelism (PP) [20, 26, 31], and 1

tures only a static snapshot of network conditions at startup time. This also leads to collective communication performing extremely poorly during exploration cycles due to bad configurations as well as online tuning overhead. More critically, the chosen configuration grows increasingly suboptimal as network contention evolves. Together, these limitations reveal a fundamental measurement bias: existing solutions optimize for network conditions they observe at startup, rather than the conditions actually encountered at run-time. We propose a novel network contention aware dynamic tuning optimizer for accelerator collective communication libraries, that explicitly models stochastic network contention rather than assuming stationary conditions. This dynamic tuning optimizer consists of two cooperative intelligent agents trained offline on latency traces collected from production grade clusters, ensuring the learned models capture clean network behavior free from background traffic interference. The ’detector’ agent continuously monitors collective communication latency at runtime to infer the current contention state of the network and records current network state in LSTM embeddings. The ’selector’ agent consumes these network state embeddings and selects a NCCL configuration for the current conditions using learned policy, thus incurring no tuning overhead at runtime. To account for the uniqueness of each deployment, whether bare-metal or shared, and across varying network fabrics (e.g., standard Ethernet, RoCE, InfiniBand, Slingshot), the dynamic tuning optimizer supports a one-time, lightweight cluster calibration step that adapts the pre-trained models to cluster-specific characteristics. Once deployed, the dynamic tuning optimizer integrates seamlessly into existing training stacks using NCCL’s Tuner Plugin API. By design, both agents run inference on CPU rather than GPU, imposing no burden on training accelerators. We evaluate our tuner on two clusters: a) Chameleon Cloud [23], providing isolated bare-metal nodes for controlled microbenchmark and AI workload evaluation; and b) Delta Cluster with A100 GPUs and the Slingshot-11 interconnect for scaling validation. On Chameleon, the dynamic tuning optimizer achieves an average latency reduction of 36% over NCCL and 32% over a fully converged AutoCCL, with substantially lower latency variance than both baselines, yielding more stable and predictable training iterations. We further evaluate across six real AI training workloads: ResNet50 [16], BERT [11], NanoGPT [22], DLRM [33], GPT-2 Medium [36], and ViT-Large [12]. Scaling to over 24 nodes on Delta’s Slingshot-11 fabric under real-world network contention, the dynamic tuning optimizer achieves speedups up to 65%. A detailed evaluation is presented in Section 5. This paper makes the following contributions: 1. Network-State-Aware Collective Communication. We introduce a new class of collective communication optimizations that models for stochastic network contention state instead of considering it stationary. The dynamic tuning optimizer realizes this as a deployable system se-

lecting NCCL configurations at runtime with negligible overhead. 2. Decoupled Coordinator-Plugin Architecture. We propose a two-tier MPI-based skeleton-stub coordinator hierarchy that offloads configuration selection entirely from the NCCL critical path to a separate process, scaling with the number of nodes participating in NCCL workload rather than the number of NCCL ranks. 3. Sustained Contention Detection. We design contention detector, which observes collective communication latency and employs a novel approach of using long shortterm memory (LSTM) [17] recurring neural network to detect anomaly in network state. This network anomaly is then classified using a classifier head combined with a sequential probability ratio test (SPRT [7]) to strongly distinguish sustained contention, transient spikes, and contention-free states with low detection latency. 4. Congestion-Aware Configuration Selection. We design configuration selector agent, a conservative dueling double deep Q-learning agent that operates with network state awareness using LSTM raw embeddings and inferred contention states to identify new NCCL tuning configurations. 5. Constant-Time Critical Path. We cache the configuration decisions from configuration selector agent in a KV store to reduce duplication of efforts and amortize inference call benefits. 6. CPU-Only Agent Inference with Negligible Overhead. Both agents run inference exclusively on CPU, imposing no additional burden on training accelerators, with a combined inference latency of 3-4 ms.

2 2.1

Background and Motivation Collective Communication Primitives

Collective communication operations are synchronized primitives requiring all participating processes to coordinate jointly to complete a well-defined communication task [49]. These operations fall into two broad categories: (a) communicationonly operations such as Broadcast and AllGather, which redistribute data across processes without modification, and (b) reduction-communication operations such as AllReduce and ReduceScatter, which combine reduction computations with data exchange. Collective operations are realized through two primary algorithmic families: (a) ring-based algorithms offering bandwidth-optimal performance; and (b) tree-based algorithms for latency sensitive implementations. These collective algorithms operate on Simple and Low-Latency protocols across shared memory for intra-node and RDMA for inter-node transport. Together, these primitives form the synchronization backbone of distributed computing, ensuring all participating processes reach a consistent shared state before proceeding. 2

(a) 256MB

(b) 1GB

(a) 50% Network Contention

(b) 80% Network Contention

Figure 1: NCCL configuration sweep on 25 Gbps duplex network under various levels of contention.

Figure 2: Comparing NCCL-default configuration with empirically optimal solution under network contentions.

To efficiently execute these primitives in modern GPUaccelerated clusters, specialized communication libraries and interfaces have been developed. While AMD’s ROCm Communication Collectives Library (RCCL) [13] provides optimized primitives for AMD accelerators, Microsoft’s MSCCL [18] introduces programmable collective topologies, and CUDA-aware MPI implementations [46] facilitate direct GPU-to-GPU transfers in traditional HPC environments, the NVIDIA Collective Communications Library (NCCL) [35] has emerged as the de facto industry standard backend [15]. Because mainstream deep learning frameworks such as PyTorch [25] rely on NCCL as default backend for distributed training and inference across modern GPU clusters, low-level collective optimizations built on NCCL offer the most direct and impactful performance gains for large-scale AI workloads. Consequently, this work adopts NCCL as the foundational target for our optimizations. Distributed AI training and inference rely fundamentally on collective communication to synchronize state across accelerators. Data, tensor, and pipeline parallelism each invoke AllReduce, ReduceScatter, and AllGather repeatedly, either to aggregate gradients, synchronize activations, or pass results across pipeline stages [39, 41]. Modern LLM frameworks further decompose AllReduce into ReduceScatter and AllGather to enable memory efficiency and computation-communication overlap. The volume and frequency of these operations make collective communication the dominant bottleneck at scale. During LLM inference, communication accounts for 10–40% of end-to-end execution time, with GPT-3 [4] spending approximately 30% in AllReduce alone [39]. During LLM training, LLaMA-2-7B spends up to 80% of execution time in communication [30], and scaling LLaMA-2-13B [43] from 8 to 1024 GPUs collapses Model FLOPs Utilization (MFU) from 47% to 4% [6] due to delays in communication. Across models like MicroLlama, OPT, GPT-2, and DistilGPT-2, communication constitutes 23%–64% of training time [21], with collective operations accounting for nearly two-thirds of a typical training iteration [21].

2.2

NCCL: Internals and Limitations

State-of-the-art collective communication libraries such as NCCL and its various extensions [15, 39, 41] are oblivious to real-time network contention. NCCL bridges the CPU processes with the GPU kernels, selecting configurations across three dimensions: (a) algorithm (Ring, Tree, and hardwaredependent variants requiring NVIDIA SHARP or NVSwitch); (b) protocol (LL, Simple, and NVLink-dependent LL128); and (c) number of channels, controlling link parallelism [19]. NCCL runtime logs show that configuration is determined using heuristics with data collected at initialization by probing network interfaces with small messages (8B–4KB) and inferring network bandwidth from the static topology. For added flexibility, NCCL also allows a Tuner Plugin API enabling users to inject configurations at runtime. NCCL’s configuration selection rests on two narrow foundations: small-message probes that never induce queue buildup or link saturation, and static topology information that reflects nominal state rather than effective bandwidth under contention. To assess the optimality of NCCL’s heuristic choices, we conduct an exhaustive sweep over its configuration space under varying levels of injected network contention. The results of this sweep are summarized in Figures 1 and 2. While NCCL’s heuristics remain near-optimal under contention-free conditions, alternative configurations yield 3%–22% speedup at 50% contention and 3%–52% at 80% contention, spanning message sizes from 32 MB to 32 GB on a 25 Gbps duplex link. These findings establish that NCCL’s heuristics-based configuration choices are systematically suboptimal in the presence of network contention, motivating a runtime-aware approach to configuration selection.

2.3

Runtime Tuning and Its Limitations

Run time configuration tuning approaches, like the ones implemented by AutoCCL [50], challenges the prevailing view that NCCL is inherently well-tuned by showing that substan3

3

tial performance headroom remains accessible with runtime configuration tuning. This observation provides the foundational insight that motivates our work. By modifying the open-source NCCL library, AutoCCL introduces support for runtime measurements of bandwidth usage and latency of collective communication calls. AutoCCL then runs online search by exploring the configuration space using coordinate descent, while incurring a runtime tuning overhead across a predefined number of tuning calls. As shown in Figure 3, this online search imposes substantial overhead, with per-iteration latency spiking to as much as 61 seconds during the tuning phase. Beyond this direct cost, we further examine the sensitivity of online tuning to two qualitatively distinct contention regimes: short-lived transient spikes and sustained contention. When transient contention occurs during the tuning phase, the optimization trajectory is perturbed, causing convergence to a configuration tuned for a network snapshot that no longer reflects current conditions. More fundamentally, existing runtime tuners commit to a single configuration upon convergence and do not re-optimize thereafter. Therefore these online tuning approaches also inherit the same measurement bias as NCCL: the selected configuration reflects conditions observed during tuning rather than those encountered over the remainder of execution.

2.4

Network Contention Aware Dynamic Tuning

Network-state-aware configuration selection for collective communication is governed by three properties: (a) stochasticity, where contention is driven by external workloads that are neither predictable nor controllable; (b) temporal dependency, where contention anomalies are better detected with historical context than from instantaneous observations alone; and (c) sequentiality, where each configuration decision influences the performance of subsequent collective calls. Therefore, heuristic- and rule-based approaches cannot adapt to conditions outside of their design envelope. Reinforcement learning (RL) offers a more principled fit, as an RL agent learns policy from the environment (e.g., network condition) guided by reward accumulation over time, naturally accommodating the sequential and stochastic nature of the problem. Configuration selection maps directly onto a Markov Decision Process (MDP): the agent observes network state, selects a configuration, receives a reward reflecting collective performance, and transitions to a new state. However, a single monolithic agent faces a structural tension: a contention detection model requires a recurrent architecture for accumulating evidence across multiple iterations, whereas a configuration selection model requires a policy optimization based decision making. Therefore, the monolithic model must run inference on every collective call and generate configurations regardless of whether the network state has changed or not, imposing unnecessary overhead on the critical path. To solve this, we decompose the optimizer into a two-agent cooperative model. A contention detector agent observes collective call latency over time and classifies the network state into three categories: SUSTAINED-CONTENTION, TRANSIENT-SPIKE, or STEADY-STATE. A configuration selector agent is then triggered exclusively during SUSTAINED-CONTENTION, using raw embeddings from the detector to produce a network-state-aware configuration. This decomposition isolates network dynamic detection from configuration optimization, ensuring that inference cost for configuration selection is only incurred when an actionable state change is confirmed.

Network-State-Aware Optimization

Existing state-of-the-art models treat the network as static, optimizing for conditions observed once and assumed to persist. In practice, contention on shared multi-user clusters is stochastic and time-varying, driven by competing workloads, background traffic, and transient link saturation. Any solution that commits to a fixed configuration without tracking network state will inevitably suffer from a measurement bias. Addressing this gap requires a solution that continuously observes network state, distinguishes sustained contention from transient noise, and selects configurations adaptive to current conditions. The structural properties of this problem motivate a learning-based optimization framework, which we characterize formally in Section 3.

3.1

Temporal Contention Detection

At each collective communication operation, detector agent addresses a single well-defined question: upon detecting a performance perturbation, is detected anomaly a safely ignorable transient spike, or a reconfiguration warranting persistent state? To quantify the magnitude of degradation, one of the input to detector agent is a computed contention ratio: ρt = Lobs /Lbaseline

(1)

where Lbaseline is the median latency observed during the first few collective operations under zero contention for the participating ranks and nodes (nRanks, nNodes) executing a particular collective type (collType) for message size (nBytes).

Figure 3: AutoCCL run-time profiling overhead. 4

Table 1: Configuration action space A for Agent-2-OCS. The 96 discrete actions are the Cartesian product of the four dimensions below.

Values of ρt near 1.0 indicate zero network contention; values above 1.0 reflect degradation proportional to contention severity. Because a single ρt value cannot distinguish a transient spike from sustained contention without temporal context and rendering the problem partially observable, a temporal dependency must be incorporated into the state representation. This temporal dependency motivates the use of an Long Short-Term Memory (LSTM) [17] recurring neural network, which maintains two recurrent states across iterations: a hidden state ht encoding a short-term summary of recent observations, and a cell state ct carrying longer-range context through gated updates. Together, (ht , ct ) accumulate evidence over successive collective calls, enabling the model to distinguish sustained shifts from transient fluctuations. Although we initially evaluated a recurrent Asynchronous Advantage Actor-Critic (A3C) [29] agent, it suffered from premature action extinction, collapsing spike detection accuracy below 10%. Given the high variability of shared production clusters, this failure mode was unacceptable, prompting us to adopt a supervised LSTM formulation instead.

Symbol

Dimension

Values

A P nCh nT h

Algorithm Protocol Channels Threads

Ring, Tree LL, Simple 1, 2, . . . , 8 128, 256, 512

|A |

Total actions

2 × 2 × 8 × 3 = 96

3.2 Contention-Aware Configuration Selection Given the current detected contention state, the configuration selector agent dynamically selects an appropriate NCCL configuration to mitigate communication overhead. We formulate this as a Markov Decision Process (MDP) (S , A , P , R ): (a) State s ∈ S : a 153-dimensional input vector combining contention detector LSTM agent’s hidden state ht embeddings of accumulated temporal contention history, with instantaneous system telemetry using contention vector ct , collective parameters, encoded presently selected configuration and normalized GPU statistics (like memory, compute and power usage).

Rather than applying a fixed threshold on contention detector agent’s output, we employ the Sequential Probability Ratio Test (SPRT) [7], which accumulates likelihood ratios sequentially and commits only when sufficient statistical confidence is reached. With calibrated upper and lower thresholds τU = 0.95 and τL = 0.05 respectively, the decision rule is:

(b) Action a ∈ A : A configuration tuple of {A, P, nCh, nT h} drawn from the 96-element action space defined in Table 1.

  SUSTAINED-CONTENTION pt > τU at = TRANSIENT-SPIKE (2) pt < τL   STEADY-STATE τL ≤ pt ≤ τU

(c) Transition Function P : Governed jointly by the stochastic evolution of background network contention and the state modifications induced by action at . (d) Reward r ∈ R : The normalized latency improvement relative to the best zero-contention baseline for the given collective parameters (Equation 3) At each decision step t, the reward rt received by configuration selector agent is defined as:

where pt = P(SUSTAINED-CONTENTION | u1:t ) is the posterior probability of sustained contention given all observations up to iteration t. The SPRT boundaries jointly govern the false-positive rate corresponding to unnecessary reconfigurations, and the false-negative rate corresponding to missed sustained contention detection.

rt =

Network contention detector agent is trained with supervised binary cross-entropy against oracle labels from offline traces and requires no reward signal or online interaction with the environment. Following offline training, the detector achieves 97.5% accuracy for SUSTAINED-CONTENTION detection, and 84.9% accuracy for TRANSIENT-SPIKE isolation.

Lzero-best − Lobs Lzero-best

(3)

where Lobs is the observed collective latency and Lzero-best is the baseline optimal latency under zero contention for the given collective parameters. Because Lobs ≥ Lzero-best , it makes the reward rt ≤ 0, where rt = 0 represents hardwarelimit performance. This non-positive formulation grounds the agent’s objective in physically achievable baseline performance rather than an arbitrary target. We adopt a Dueling Double DQN [44, 48] architecture, which decouples value and advantage estimation while using a target network to mitigate Q-value overestimation:

Along with the SUSTAINED-CONTENTION signal, the detector agent also generates a continuous contention vector capturing direction, magnitude, and confidence of the degraded network state. Rather than passing a simple binary alert, this rich representation is forwarded to the configuration selector agent, providing the precise network state context needed to drive downstream reconfiguration.

yt = rt + γ(1 − dt ) Qθ− st+1 , arg max Qθ (st+1 , a′ ) a′

5

(4)

CollType(N) fwd pass → bwd pass coord replies with cached config C

coord reads metrics triggers agents Agent-1 + Agent-2 run here

CollType(N+1) (100ms – seconds) Time

ground. As illustrated in Figure 4, the agents execute during the AI application’s forward and backward computation passes which take anywhere from tens of milliseconds to seconds. This allows contention detector and configuration selector agents to complete and update the local cache with new configuration C′ , well before the next collective operation (CollType(N + 1)) arrives. The resulting one-step configuration lag is absorbed entirely within the application’s compute time, removing optimization overhead from the critical path. compute time.

coord replies with updated config C′

Figure 4: Agent inference runs during the forward and backward pass compute phase, hiding optimization latency behind work the application performs regardless. Config C′ is available in the KV cache before CollType(N+1) arrives. where θ and θ− are the online and target network parameters respectively (with θ− slowly updated using soft target updates, τ = 0.005), γ is the discount factor, and dt is a terminal state indicator. To keep inference lightweight, configuration selector agent inherits temporal context directly from contention detector agent’s LSTM hidden state ht without maintaining its own recurrence and executes in a single feedforward pass. Because training is performed offline on isolated baremetal cluster traces, the dataset contains an inherent bias: TRANSIENT-SPIKE events are underrepresented as compared to SUSTAINED-CONTENTION and STEADY-STATE conditions. Therefore, without regularization, the agent would overestimate Q-values for the underrepresented state-action pairs. We address this by augmenting the Bellman loss with Conservative Q-Learning (CQL) regularization:

L (θ) = LBell (θ) + αCQL LCQL (θ)

4

The network-contention-aware dynamic configuration tuner consists of three primary infrastructure components connected through a pluggable optimizer interface: a Python telemetry library, an NCCL tuner-plugin coordinator, and an online configuration optimizer (Figure 7). During an NCCL collective call, the tuner: (a) captures per-rank GPU statistics and collective call latency via the Python telemetry library (Section 4.1); (b) interposes on the collective call using the NCCL tuner plugin (Section 4.2); and (c) detects sustained network contention to replace NCCL’s default configuration with an optimized selection (Section 4.3). We propose DynamICCL (pronounced as "dynamical"), a network contention aware dynamic and intelligent tuner for collective communication library. Figure 5 illustrates the complete tuner architecture and its numbered control flow. During the NCCL initialization phase, each rank loads the tuner plugin and connects to its local coordinator 1 , while the GPU monitor initializes alongside the application process 2 , writing telemetry data to shared memory (SHM) via a zero-copy mechanism 3 . When the AI/ML training workload invokes a collective operation 4 , the plugin intercepts the call 5 . The leader rank plugin sends a configuration request to the skeleton coordinator 6 , which performs a constant-time KV cache lookup 7 8 . The fetched configuration is then propagated across nodes using an inter-node broadcast 9 and relayed locally to all worker plugins on the node 10 . Concurrently, after dispatching the configuration reply, an asynchronous background thread 11 reads the latest telemetry from SHM 12 and feeds it to contention detector agent (LSTM + SPRT) for contention inference. If contention detector agent detects SUSTAINED-CONTENTION 13 , configuration selector agent (D3QN) evaluates the updated network state to select an optimized configuration, writing it back to the KV cache 14 for subsequent collective operations.

(5)

where LCQL penalizes Q-values for out-of-distribution actions, ensuring the agent defaults to conservative, safe configurations when encountering unfamiliar network states.

3.3

Design and Implementation

Decoupled Hierarchical Coordinator

Each NCCL rank loads a tuner plugin that interposes on every collective operation call to inject configuration parameters. Embedding the optimizer directly within this plugin would place agent inference on the critical path of every collective call, adding optimization overhead directly to communication latency. Furthermore, running optimization per rank introduces redundant computation, as co-located ranks evaluate identical network states and reach identical decisions. We resolve both issues by relocating the optimizer to a dedicated coordinator process. The per-rank plugin is thus reduced to a lightweight relay that forwards collective parameters to the coordinator and receives configuration decisions, successfully decoupling optimization overhead from total rank count. Decoupling the coordinator also enables completely hiding optimization latency through compute-overlap execution. When a collective call occurs, the coordinator replies immediately with a constant-time lookup to a local configuration cache, returning the currently active configuration C. It then triggers agent inference asynchronously in the back-

4.1

Telemetry and Shared-Memory Pipeline

The telemetry pipeline captures per-collective latency and GPU metrics off the critical path using two concurrent threads (Figure 6). The main thread interposes on PyTorch collective 6

Figure 5: Overview of DynamICCL Architecture. calls with high-precision timers to measure per-call execution latency. Concurrently, a background thread samples GPU statistics using the NVML API [8] at 100 ms intervals (a polling frequency bounded by NVML’s hardware read latency) collecting Streaming Multiprocessor (SM) utilization, memory utilization, PCIe transmit/receive throughput, and power consumption. Both telemetry streams are written to a per-rank, shared-memory circular buffer using a zero-copy, lock-free design, allowing the coordinator process to asynchronously read the latest sample without lock contention. Because the telemetry library integrates directly at PyTorch’s collective communication layer, it remains fully compatible with alternative backend extensions such as RCCL, NCCLX and MSCCL++ without requiring manual application instrumentation.

4.2

ordinator executes an inter-node broadcast to the STUB coordinators on Nodes 1 through N − 1 3 , after which each coordinator performs an intra-node broadcast to deliver the configuration to its local rank plugins 4 . Accessing the thread-count parameter (nT h) requires a one-time lightweight patch to NCCL’s plugin interface, provided in the dynamic tuner’s source repository. Encapsulating the optimizer inside the coordinator eliminates all PyTorch/C++ (libtorch) dependencies from the plugin, keeping it a portable, zerodependency C/C++ library compatible across arbitrary NCCL deployments. The coordinator runs as a standalone process on each node, compatible with both SLURM-managed clusters and baremetal environments. Under its SKELETON/STUB topology, one designated node executes in SKELETON mode (hosting the primary optimizer), while all other nodes run as lightweight STUB processes. The SKELETON distributes updated configurations across node boundaries using MPI_Bcast; each STUB then relays the configuration locally to its per-rank plugins. This two-tier hierarchy restricts cross-node control traffic to a single broadcast per reconfiguration event, successfully decoupling coordination overhead from NCCL’s rank count to number of participating nodes.

Coordinator Architecture

NCCL’s Tuner Plugin API serves as our tuner’s entry point into NCCL, using three hooks from API v4 to intercept every collective scheduling decision. During initialization, each rank on a node connects to its local coordinator through TCP; the coordinator designates the first connecting local-rank as the LEADER and all remaining local-ranks as WORKERS. Figure 7 illustrates the control flow when a collective operation is intercepted by the tuner’s plugin-coordinator architecture. At every collective invocation, the local-rank LEADER transmits the operation metadata (collType, nBytes) to the local coordinator 1 , which fetches the active configuration (A, P, nCh, nT h) from the optimizer 2 . This configuration is then distributed hierarchically: the Node-0 SKELETON co-

4.3

Optimizer Design

The optimizer interacts with the coordinator using a nonblocking, two-function interface. On each collective call, process() performs a constant-time KV-cache lookup keyed by (collType, nBytes) and immediately returns the active configuration. Once the reply is dispatched, observe() snap7

Algorithm 1 Asynchronous Optimizer Thread Runs asynchronously during forward/backward pass Constants: COOLDOWN_TICKS = 3 1: while running do 2: Wait on condition variable ▷ woken by observe() 3: Snapshot (info, cfg, stats) under mutex 4: action ← contention_detector(info, cfg, stats) ▷ runs always 5: if global_tick = 0 then ▷ NCCL cold-start overhead 6: Discard contention_detector output, keep LSTM state 7: else if tick ≥ 8 then ▷ Normal operation after warmup 8: if cooldown > 0 then 9: cooldown ← cooldown − 1 ▷ New config settling 10: else if action = SUSTAINED then 11: Run configuration_selector, update KV_Cache 12: cooldown ← COOLDOWN_TICKS 13: end if 14: end if ▷ Ticks 1–8: contention_detector builds LSTM context only 15: tick ← tick + 1 16: end while

Figure 7: Two-tier coordinator control flow: (1) configuration request from LEADER rank, (2) optimizer lookup, (3) internode broadcast from SKELETON to STUB coordinators, and (4) intra-node dispatch to per-rank WORKER plugins. 4.3.1

Contention Detection

shots the latest telemetry and signals the asynchronous optimizer thread (Algorithm 1). The optimizer thread executes continuously in the background (Algorithm 1, line 1). Upon awakening, it captures a snapshot of current collective metadata, active configuration parameters, and GPU metrics (line 2) and unconditionally invokes contention detector (line 3). At iteration 0, the inference output is discarded to eliminate NCCL cold-start latency artifacts while preserving the LSTM hidden state h0 (lines 4–5). During iterations 1–8, contention detector agent accumulates temporal context across collective calls without emitting classification decisions (line 14). From iteration 8 onward, if a cool-down period remains active from a prior reconfiguration, it is decremented (lines 8–9); otherwise, an emission of SUSTAINED-CONTENTION triggers configuration selector (D3QN) agent. The configuration selected by the configuration selector agent is written back to the KV cache, and a cooldown counter of COOLDOWN_TICKS = 3 is set (lines 10–12), preventing premature re-triggering before the newly applied configuration has stabilized across sufficient iterations.

At each iteration, contention detector agent (detailed in Algorithm 2) constructs a 15-dimensional input vector by concatenating 11 scalar features, including contention ratio ’ρt ’ (Equation 1), log-scaled topology parameters, normalized GPU statistics, and a configuration-change flag—with a 4dimensional learned embedding of the collective operation type (lines 1–4). This vector is processed by an LSTM layer with a 128-dimensional hidden space, where the recurrent hidden and cell states (ht , ct ) persist across consecutive iterations (line 5). Passing the LSTM hidden state through a feedforward classification head yields the sustained contention probability psustained (line 6), which is subsequently evaluated against the Sequential Probability Ratio Test (SPRT) thresholds from Equation 2 (lines 7–12). With a compact memory footprint of approximately 358 KB, contention detector agent completes inference in sub-millisecond time on a CPU core. Contention detector agent is trained offline with classbalanced sampling at 17,400 episodes per class per epoch

Figure 6: GPU Telemetry and Shared Memory Pipeline

Figure 8: Contention detector agent training evaluation 8

Algorithm 2 Network Contention Detector Agent

Algorithm 3 Configuration Selector Agent

Input: collInfo (collType, nRanks, nNodes, nBytes), cfg, stats Output: action (WAIT / SPIKE / SUSTAINED) Persistent: LSTM hidden state (h, c) 1: r ← latency / baseline ▷ Congestion ratio 2: scalars ← [r, CollectiveInfo, GpuStats, cfg_changed] ▷ 11-dim 3: emb ← Embed(collType) ▷ 5 → 4-dim 4: x ← Concat(scalars, emb) ▷ 15-dim 5: out, (h, c) ← LSTM(x, h_prev, c_prev) ▷ hidden = 128 6: p ← FeedForwardNet(out) ▷ 128 → 256 → 128 → 1 7: if p > 0.95 then ▷ SPRT upper threshold 8: return SUSTAINED 9: else if p < 0.05 then ▷ SPRT lower threshold 10: return SPIKE 11: else 12: return WAIT 13: end if

Input: LSTM hidden state h, info, cfg, stats Output: Config (A, P, nCh, nTh) Action space: 2 Algo × 2 Proto × 8 Ch × 3 Th = 96 1: cong ← CongestionVector(action, r, p_sustained) ▷ direction ∈ [-1,1], magnitude ∈ [0,3], confidence ∈ [0,1] 2: state ← [Congestion, collInfo,last_config, GPU_stats] 3: x ← [state, LSTM_history] ▷ input vector size 153 4: z = DQN(x) ▷ 153 → 256 → 128 5: V ← FeedForwardNet(z) ▷ Value: 128 → 64 → 1 6: A ← FeedForwardNet(z) ▷ Advantage: 128 → 64 → 96 7: for a∈Action Space do 8: Q(a) ← V + A(a) − mean(A) ▷ Dueling aggregation 9: end for 10: idx ← argmax Q(a) ▷ Greedy selection 11: nTh ← {128, 256, 512}[idx mod 3] 12: nCh ← (idx / 3) mod 8 + 1 13: P ← {LL, Simple}[(idx / 24) mod 2] 14: A ← {Tree, Ring}[idx / 48] 15: Update KV_Cache[(collType, nBytes)] ← (A, P, nCh, nTh) 16: return (A, P, nCh, nTh)

to prevent the minority SPIKE class from being too underrepresented. Figure 8 shows BCE loss stabilizing by epoch 3 (a), AUC=0.976 (b), and 97.5% sustained recall with less than 5% false-positive rate on held-out episodes (c). Detection speed adapts to severity: approximately 4 iterations at 80% contention and 9 at 50%.

4.3.2

Configuration Selection

Configuration selector agent (detailed in Algorithm 3) is invoked exclusively upon a SUSTAINED-CONTENTION emission from contention detector. It first constructs a contention vector by encoding contention direction, magnitude, and confidence from contention detector’s output (line 1), assembles a 153dimensional state vector combining current system telemetry with the recurrent LSTM hidden state ht (lines 2–3), and forwards it through the trunk network (line 4). The dueling streams evaluate a state-value estimate V (s) and per-action advantages A(s, a) (lines 5–6), which are aggregated to compute Q-values (lines 7–9). The greedy action a∗ = arg maxa Q(s, a) is then decoded into a valid configuration tuple (lines 10–14) and written to the KV cache for the subsequent collective operation (line 15). With a compact memory footprint of approximately 168 KB, configuration selector agent executes CPU inference in under 2 ms.

Figure 9: Configuration selector agent training evaluation

4.4

Implementation

The network-contention-aware dynamic configuration tuner is implemented in approximately 4,250 lines of C++17 across four core modules: the coordinator, tuner plugin, online optimizer, and TCP communication utilities. An additional 610 lines of Python comprise the telemetry library. The system is built using GNU Make with -std=c++17 -O2 -fPIC, eliminating external build dependencies like CMake. Core runtime dependencies are restricted to POSIX shared memory, POSIX-threads, a CPU-only PyTorch build (libtorch 2.7.1), and MPI for multi-node deployments. To maintain isolation, the build vendors only the NCCL Tuner Plugin ABI headers without linking against NCCL at compile time. Telemetry library. The Python telemetry library runs as a process alongside each GPU rank, capturing hardware metrics from the NVML API and logging per-call collective latencies into a per-rank POSIX shared-memory region. Each

COnfigruation selector agent is trained offline using Dueling Double DQN with Conservative Q-Learning [24] regularization (αCQL = 0.001), optimizing the reward function and loss formulation defined in Equations 3 and 5. Figure 9 confirms empirical convergence within 10K training steps, demonstrating Q-value stability, clear dueling-head differentiation, and strong ensemble agreement across 20 independent random seeds compared to the baseline. 9

that operates without requiring an MPI runtime. Optimizer. Both inference models are serialized as TorchScript binaries and loaded in C++ using torch::jit::load(). The recurrent LSTM hidden and cell states (ht , ct ) of dimension 1 × 1 × 128 are maintained as C++ torch::Tensor instances across collective iterations. Inference executes in FP32 on CPU without quantization, completing in under 3.6 ms combined for both agents. Under the REPLY_BYPASS path—active during initial warmup as well as on SPIKE or WAIT emissions, configuration selector agent is bypassed entirely, incurring zero computational overhead. Source code for the network-contention-aware dynamic configuration tuner will be made publicly available on GitHub upon publication.

5

Figure 10: Contention Detection Timeline

Evaluation

We evaluate the dynamic tuner on two platforms selected for complementary evaluation objectives: Chameleon Cloud, which provides isolated bare-metal nodes across heterogeneous GPU generations (A100, V100, and P100) for controlled microbenchmark experiments, and the Delta cluster, which we use to validate scaling behavior. We compare three systems built against identical NCCL v2.29.2+cuda12.6 binaries: the default NCCL heuristic, AutoCCL’s coordinatedescent tuner [50], and our dynamic tuner. Speedups are computed using a paired per-run methodology: the N-th run of a baseline is matched with the N-th run of our dynamic tuner, per-pair speedups are computed, and summary statistics are reported over the resulting distribution. We report the mean for microbenchmark results, where isolated bare-metal nodes yield low run-to-run variance, and the median for scaling results, where stochastic background traffic on the shared cluster introduces heavy-tailed variance that would otherwise skew the mean.

shared-memory buffer contains 1,024 slots of 64 bytes each, prepended by a 64-byte header maintaining a monotonically incrementing write index (totaling 65,600 bytes per rank). The coordinator reads the latest sample in O (1) time by evaluating (write_index - 1) % 1024, enabling non-blocking, zero-copy reads without lock contention. Tuner plugin. The plugin is compiled as a shared library loaded by NCCL at runtime using the NCCL_TUNER_PLUGIN environment variable. It implements the NCCL Tuner Plugin ABI v4.1 interface across three hooks: init, getCollInfo, and destroy. By isolating machine learning dependencies to the coordinator binary, the plugin remains a compact C++ component of approximately 460 lines. When the coordinator emits a REPLY_BYPASS signal, the plugin returns ncclSuccess without modifying NCCL’s cost tables, delegating configuration selection directly to NCCL’s built-in heuristics. Furthermore, if the coordinator fails to initialize, NCCL transparently falls back to its default runtime behavior without interrupting the workload. Coordinator. The coordinator is an MPI application executed using mpirun, deploying exactly one MPI process (rank) per cluster node hosting the NCCL workload. The SKELETON process (MPI rank 0) hosts the online optimizer and handles configuration decisions. STUB processes (MPI ranks 1 through N − 1) receive decisions from the SKELETON through MPI_Bcast using a compact message structure comprising of five uint32_t fields, further relaying parameters to node-local per-rank plugins over TCP on localhost. This design scales internode coordination overhead with node count rather than rank count (Section 3.3). In production, coordinators are launched prior to the application using mpirun, followed by the training job via torchrun. The coordinator is validated across SLURM-managed clusters and bare-metal environments. To simplify local testing and single-node development, a TCP-only coordinator variant is also provided

5.1

Microbenchmark Results

The microbenchmark testbed comprises four nodes (two with dual P100-SXM2 GPUs and two with dual P100-PCIe GPUs), interconnected via a 10 Gbps full-duplex network. Collective operations on 2 GB float32 tensors serve as the baseline workload, supplemented by tensor-size sweeps ranging from 256 MB to 4 GB. Network contention is injected using symmetric BBR iperf3 background streams that consume 50%– 90% of link capacity across both inter-node pairs, with precise onset control. 5.1.1

Runtime Contention Detection

Contention detector agent’s LSTM recurrent network accumulates evidence proportional to contention severity, adjusting its detection rate dynamically. As illustrated in Figure 10, the 10

Figure 11: Transient spike rejection under short contention bursts. agent detects sustained contention in approximately 4 iterations under 80%–90% link congestion and 9 iterations at 50% congestion, with psustained crossing the SPRT threshold (0.95) proportionally faster under heavier degradation. This adaptive response is a direct benefit of pairing the LSTM classifier head with a Sequential Probability Ratio Test (SPRT) to evaluate decision confidence: unlike fixed-window heuristic detectors, it avoids imposing a rigid detection delay under severe congestion." Equally critical is the reliable rejection of transient network noise. Figure 11 demonstrates that the detector agent correctly classifies all 2-iteration contention spikes (at 30%, 40%, and 50% severity across six total bursts), resulting in zero false reconfigurations and keeping configuration selector agent uninvoked. To isolate onset timing from adaptation speed, we evaluate contention injected at iterations 10, 20, and 30 under 80% link congestion. The post-reconfiguration iteration speedup remains consistent at approximately 42% regardless of onset timing, confirming that the contention detector agent’s state progression does not depend on specific temporal iteration indices. Figure 12 illustrates the dynamic tuner handling both escalating (0% → 60% → 80%) and de-escalating (0% → 80% → 60%) congestion shifts. In both patterns, the system executes a single, stable configuration switch, holding average latency

Figure 13: Equal-opportunity and fair-opportunity comparison: NCCL vs. AutoCCL vs. our dynamic tuner. within 5,000–5,800 ms compared to NCCL’s 6,500–8,400 ms without triggering redundant re-optimizations. 5.1.2

Performance Analysis

We evaluate all systems under two complementary execution regimes. The equal-opportunity regime initializes all systems concurrently and measures performance from the first iteration under contention, reflecting realistic cold-start online deployment. The fair-opportunity regime grants AutoCCL 100 pre-contention iterations to converge prior to measurement, isolating its converged configuration quality from its online search overhead. Together, these regimes decouple runtime search overhead from asymptotic parameter quality. Figure 13 presents per-iteration latency across contention levels for both regimes. Under equal-opportunity evaluation, AutoCCL’s live coordinate descent active search increases iteration latency by 87%–195% relative to baseline NCCL. At 80% contention, baseline NCCL averages 8,673 ms, our dynamic tuner achieves 6,147 ms (+29.1% speedup), and AutoCCL averages 21,697 ms (−150% slowing), highlighting the efficiency gap between pre-trained CPU-bound inference and online coordinate search. Under fair-opportunity evaluation, AutoCCL achieves a modest 6.4% speedup over NCCL, whereas our dynamic tuner achieves a 36% latency reduction over NCCL and a 32% reduction over AutoCCL (p < 0.0001). The latency distributions in Figure 15 confirm these improvements with high

Figure 12: Multi-level contention transition response. 11

(a) 80% Network contention

Figure 14: Heatmap of speedup across varying tensor sizes and network contention levels.

Figure 16: Performance improvement of our dynamic tuner across real AI workloads under network contention.

statistical significance (p < 0.0001). Furthermore, the proposed tuner exhibits significantly lower variance, yielding highly predictable iteration times. Under clean (uncongested) network conditions, pre-contention latency differs by less than 0.2% between NCCL and our tuner (3,204 ms vs. 3,210 ms), confirming negligible runtime overhead. Across varying contention levels, the dynamic tuner reduces AllReduce latency by 8.2% (at 50% congestion), 19.0% (70%), 29.1% (80%), and 30.2% (90%). The dynamically selected new configuration maintains stable iteration times between 5,500–6,100 ms, while default NCCL degrades from 6,136 ms to 8,673 ms. Figure 14 shows speedup heatmaps across five tensor sizes (256 MB to 4 GB) and three contention levels. Under 70%–90% congestion, the tuner delivers consistent 19%–31% speedups across all tensor sizes, demonstrating that configuration selector agent’s policy captures generalizable network-configuration dynamics rather than overfitting to specific payload volumes.

5.2

(b) 100% Network contention

contention, the average speedup reaches 12.06%; at near 100% contention, it increases to 12.98%, with BERT and NanoGPT achieving median speedups of 18.9% and 28.3%, respectively. These higher gains stem from the large per-iteration AllReduce volumes in transformer architectures, which exacerbate default NCCL routing penalties under congestion. Conversely, lighter workloads like ResNet-50 and DLRM invoke smaller collectives with lower absolute communication overhead, leaving less room for optimization. The consistent performance gains across diverse architectures—spanning CNNs (ResNet-50), Transformers (BERT, GPT-2 Medium, ViT-Large), and Recommendation models (DLRM)—confirm that our contention-aware configuration strategy generalizes across varied collective communication patterns.

5.3

Scaling Analysis

We evaluate system scaling on the Delta cluster (A100 GPUs with Slingshot-11 interconnect) using data-parallel training of ’GPT-2 Medium’ model under real background contention. To isolate how contention sensitivity changes with cluster topology, we evaluate both density scaling (1 to 4 GPUs per node across a fixed 4-node cluster) and node scaling (1 GPU per node scaled out from 4 to 24 nodes). This dual-axis design captures the underlying topology dynamics: inter-node traffic exposed to network congestion scales directly with total node count, whereas higher per-node rank densities shift communication locally to intra-node interconnects, reducing exposure to external link contention. As shown in Figure 17(a), density scaling done by fixing number of nodes while increasing number of NCCL ranks yields speedups of 6.2%–65.0% (+65.0% at 4 ranks, +61.4% at 8 ranks, +10.2% at 12 ranks, and +6.2% at 16 ranks), tapering as per-node rank density increases. At higher densities, additional intra-node ranks contend for shared NIC bandwidth, reducing per-rank link headroom and compressing the headroom within which the dynamic tuner can identify a better configuration. Conversely, Figure 17(b) shows that node

AI Workload Analysis

We evaluate the dynamic tuner across six representative AI training workloads—ResNet-50, BERT, NanoGPT, DLRM, GPT-2 Medium, and ViT-Large—under 80% and 100% network contention. As shown in Figure 16, the proposed tuner yields positive median speedups across all 12 workloadcontention configurations, ranging from 7% to 28%. At 80%

Figure 15: Latency distribution and t-test results comparing default NCCL, AutoCCL, and the dynamic tuner. 12

(a) Density scaling

(b) Node scaling

(a) Intrinsic vs. total influence

Figure 17: Scaling behavior of the dynamic tuner for GPT-2 Medium model training on the Delta cluster (A100 GPUs with Slingshot-11).

Figure 18: Behavioral impact: share of decisions changed and Q-value displacement when each component is permuted. Crucially, Figure 18(a) resolves the dimensionality question: ht does not appear dominant merely due to vector size, as each constituent dimension carries substantial individual weight.

scaling maintains consistent speedups between 35.3% and 65.0% across 4 to 24 nodes (averaging over +45% speedup). This confirms that as inter-node communication dominates, the coordinator hierarchy scales efficiently to mitigate the primary network bottleneck regardless of cluster scale. Across all scaling configurations, pre-contention system overhead remains below 3%.

5.4

(b) Behavioral impact

• Behavioral Impact (Figure 18–b): High gradient sensitivity is only meaningful if it alters control outputs. By permuting individual input components while holding others fixed, we isolate their direct effect on parameter selection. Scrambling ht alters **41.2%** of configuration decisions and causes nearly 9% Q-value displacement. In contrast, permuting any non-network input (such as message size, active parameters, or GPU stats) changes at most 3.4% of decisions.

Agent Configuration Selection Analysis

A central question in learning-based system optimization models is whether the control policy actively leverages highdimensional runtime telemetry or merely relies on static workload features. To evaluate whether configuration selector agent’s network-awareness mechanism actively drives configuration selection, we conduct a feature sensitivity and gradient attribution analysis across 800 realistic execution states (spanning 5 workload contexts across 160 temporal ticks of a real-world contention trace, with congestion onset at tick 120). We evaluate feature importance across two√complementary axes: intrinsic gradient sensitivity (∥∇x Q∥/ d) to measure input attention, and decision permutation testing to measure behavioral impact (Q-value displacement and percentage of flipped configuration decisions). As shown in Figure 18, the results confirm that temporal network state is the primary driver of agent behavior:

Stripping away temporal network information changes agent behavior significantly. Together, Figures 18(a) and 18(b) demonstrate that configuration selector agent actively attends to dynamic network context, and this awareness directly governs configuration choices. This confirms that the networkadaptive design is the primary driver of performance gains under contention.

6

Related Work

Efficient Communication Algorithms. Topology-aware approaches such as Swing [10] and MSCCLang [9] construct collective communication algorithms tailored to specific physical topologies, while SCCL [5] synthesizes Pareto-optimal algorithms automatically. Compression-based methods reduce transferred data volumes using dynamic sparsity or quantization (e.g., HiPress [2], ZeRO++ [45]). The dynamic tuner presented in this work is orthogonal to these techniques: it optimizes runtime execution parameters for whichever collective algorithm is active, rather than altering the underlying algorithm or message payload itself. Overlapping Computation and Communication. Pioneered by frameworks like Horovod [38], overlapping AllReduce collectives with backward pass computation reduces

• Intrinsic vs. Total Influence (Figure 18–a): To determine if configuration selection depends on the recurrent network state embedding (ht ), we measure output gradient variance relative to input perturbations. Featurefor-feature, ht demonstrates intrinsic per-dimension sensitivity (0.522) comparable to the most dominant static input (topology at 0.596), outperforming all other state features by 3× to 5×. Across its 128 dimensions, ht accounts for approximately 97% of total policy sensitivity. 13

perceived communication overhead, with subsequent works extending this paradigm across hybrid parallelism strategies [21, 30]. The proposed tuner complements these scheduling techniques by directly reducing collective execution latency under network contention, allowing the performance benefits of both approaches to compose cleanly. Collective Communication Library Stacks and Tuning. Vendor and open-source stacks such as RCCL [13] and MSCCL [18] share foundational design elements with NCCL, while extensions like MSCCL++ [39], NCCLX [41], and PCCL [21] introduce programmable kernels, optimized transport paths, and peer-to-peer execution abstractions. For Mixture-of-Experts (MoE) workloads, NCCL EP [14] provides specialized dispatch and combine primitives built directly on NCCL’s Device API. In the domain of configuration tuning, AutoCCL [50] demonstrated the suboptimality of NCCL’s default heuristics and motivated the NCCL Tuner Plugin API. Subsequent studies [3] have explored this configuration space, while AFNFA [47] improves bandwidth prediction through offline profiling. However, as highlighted in Section 2, existing autotuners suffer from measurement bias by optimizing for past or current observations rather than future network states, a limitation our dynamic tuner directly addresses through continuous runtime contention detection and proactive adaptation. Learning-Based Systems Optimization. Reinforcement learning (RL) has proven effective across diverse network and systems optimization tasks [28,34]. While standard RL excels in static environments, capturing dynamic network behavior requires temporal context. Our dynamic tuner enhances standard RL by incorporating an LSTM layer to embed runtime network history directly into the RL agent’s state representation. Combined with a novel multi-agent decomposition, this approach explicitly addresses the partially observable and temporally dependent nature of network contention, distinguishing our dynamic tuner from prior RL applications aimed at stateless systems.

7

strates that treating network state as a first-class control variable yields speedups of up to 65% under network congestion, effectively recovering performance lost to static library heuristics. Our results demonstrate that as LLM training and inference scale across shared, heterogeneous infrastructure, the communication stack must adapt dynamically alongside the workloads it serves. While this work focuses on contentionaware configuration tuning, its modular, multi-agent architecture provides a extensible framework to balance communication latency, energy efficiency, and in-network computing hardware such as NVIDIA SHARP or NVSwitch. We opensource our dynamic tuner to provide the community with a robust foundation for building autonomous, network-aware distributed communication systems.

References [1] Saurabh Agarwal, Hongyi Wang, Shivaram Venkataraman, and Dimitris Papailiopoulos. On the utility of gradient compression in distributed training systems. Proceedings of Machine Learning and Systems, 4:652– 672, 2022. [2] Youhui Bai, Cheng Li, Quan Zhou, Jun Yi, Ping Gong, Feng Yan, Ruichuan Chen, and Yinlong Xu. Gradient compression supercharged high-performance data parallel DNN training. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, SOSP ’21, page 359–375, New York, NY, USA, 2021. Association for Computing Machinery. [3] Majid Salimi Beni, Ruben Laso, Biagio Cosenza, Siegfried Benkner, and Sascha Hunold. Exploring NCCL tuning strategies for distributed deep learning. In 2025 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 59–62, 2025.

Conclusion

[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.

Collective communication libraries have long operated under the assumption of a static, unperturbed network, fixing communication parameters at runtime based on idealized initial conditions. As distributed AI workloads scale across multi-tenant clusters, background traffic introduces stochastic, time-varying network contention that severely degrades performance. This work introduces a dynamic tuner that shifts collective communication from static, snapshot-based heuristics to continuous, stateful adaptation. By decoupling contention detection (LSTM+SPRT agent) from configuration selection (D3QL agent), the proposed tuner provides a low-overhead mechanism to navigate the execution trade-offs of modern network fabrics like Slingshot-11. Our evaluation across six production AI workloads demon-

[5] Zixian Cai, Zhengyang Liu, Saeed Maleki, Madanlal Musuvathi, Todd Mytkowicz, Jacob Nelson, and Olli Saarikivi. Synthesizing optimal collective algorithms. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 62–75, 2021. [6] Qiaoling Chen, Qinghao Hu, Guoteng Wang, Yingtong Xiong, Ting Huang, Xun Chen, Yang Gao, Hang Yan, 14

Yonggang Wen, Tianwei Zhang, et al. AMSP: Reducing communication overhead of ZeRO for efficient LLM training. arXiv preprint arXiv:2311.00257, 2023.

Yin, et al. NCCL EP: Towards a unified expert parallel communication API for NCCL. arXiv preprint arXiv:2603.13606, 2026.

[7] Yahya Chetouani. A sequential probability ratio test (sprt) to detect changes and process safety monitoring. Process Safety and Environmental Protection, 92(3):206– 214, 2014.

[15] Khaled Hamidouche, John Bachan, Pak Markthub, PeterJan Gootzen, Elena Agostini, Sylvain Jeaugey, Aamir Shafi, Georgios Theodorakis, and Manjunath Gorentla Venkata. GPU-initiated networking for NCCL. arXiv preprint arXiv:2511.15076, 2025.

[8] NVIDIA Corporation. [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.

[9] Meghan Cowan, Saeed Maleki, Madanlal Musuvathi, Olli Saarikivi, and Yifan Xiong. Mscclang: Microsoft collective communication language. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2023, page 502–514, New York, NY, USA, 2023. Association for Computing Machinery.

[17] Sepp Hochreiter and Jürgen Schmidhuber. Long shortterm memory. Neural Computation, 9(8):1735–1780, 1997. [18] Qiang Hu, Haoquan Qi, Shengzhi Du, and Pengwei Wang. MSCCL: A framework for enhancing mashup service clustering with contrastive learning. IEEE Transactions on Services Computing, 18(3):1588–1601, 2025.

[10] Daniele De Sensi, Tommaso Bonato, David Saam, and Torsten Hoefler. Swing: short-cutting rings for higher bandwidth allreduce. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation, NSDI’24, USA, 2024. USENIX Association.

[19] Zhiyi Hu, Siyuan Shen, Tommaso Bonato, Sylvain Jeaugey, Cedell Alexander, Eric Spada, James Dinan, Jeff Hammond, and Torsten Hoefler. Demystifying NCCL: An in-depth analysis of gpu communication protocols and algorithms. In 2025 IEEE Symposium on High-Performance Interconnects (HOTI), pages 48–59. IEEE, 2025.

[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019.

[20] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.

[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.

[21] Ziyang Jia, Laxmi N. Bhuyan, and Daniel Wong. PCCL: Energy-efficient LLM training with power-aware collective communication. In 2024 IEEE 42nd International Conference on Computer Design (ICCD), pages 84–91, 2024.

[13] Wenkai Du, gilbertlee (AMD), Sylvain Jeaugey, Bertan Dogancay, dependabot[bot], Nilesh M Negi, akollias (AMD), Nusrat Islam, Corey Derochie, Stanley Tsang, Mustafa Abduljabbar, Ziyue Yang, Pedram Alizadeh, Arm Patinyasakdikul, Saad Rahim (AMD), Edgar Gabriel, Atul Kulkarni, Boris Fomitchev, Nathan Luehr, Tim, isaki001, mberenjk, Wen-Heng (Jack) Chung, Ke Wen, Jeffrey Novotny, David Addison, alex-breslowamd, Avinash, Kapil S. Pawar, and Eiden Yoshida. Rocm/rccl. https://github.com/ROCm/rccl, jan 17 2026.

[22] Andrej Karpathy. nanogpt: The simplest, fastest repository for training/finetuning medium-sized gpts. https: //github.com/karpathy/nanoGPT, 2022. GitHub repository. [23] Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. Lessons learned from the Chameleon testbed. In Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC ’20). USENIX Association, July 2020.

[14] Amos Goldman, Nimrod Boker, Maayan Sheraizin, Nimrod Admoni, Artem Polyakov, Subhadeep Bhattacharya, Fan Yu, Kai Sun, Georgios Theodorakis, Hsin-Chun 15

[24] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179–1191, 2020.

[33] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, et al. Deep learning recommendation model for personalization and recommendation systems. arXiv preprint arXiv:1906.00091, 2019.

[25] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. PyTorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020.

[34] Md S. Q. Zulkar Nine, Tevfik Kosar, Muhammed Fatih Bulut, and Jinho Hwang. Greennfv: Energy-efficient network function virtualization with service level agreement constraints. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23, New York, NY, USA, 2023. Association for Computing Machinery.

[26] Shigang Li and Torsten Hoefler. Chimera: efficiently training large-scale neural networks with bidirectional pipelines. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–14, 2021.

[35] NCCL Nvidia. Optimized primitives for collective multigpu communication, 2015.

[27] Zhiqi Lin, Youshan Miao, Guanbin Xu, Cheng Li, Olli Saarikivi, Saeed Maleki, and Fan Yang. Efficient schedule construction for distributed execution of large DNN models. IEEE Transactions on Parallel and Distributed Systems, 35(12):2375–2391, 2024.

[36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.

[28] Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula. Resource management with deep reinforcement learning. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks, HotNets ’16, page 50–56, New York, NY, USA, 2016. Association for Computing Machinery.

[37] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020.

[29] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928– 1937. PmLR, 2016.

[38] Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv:1802.05799, 2018. [39] Aashaka Shah, Abhinav Jangda, Binyang Li, Caio Rocha, Changho Hwang, Jithin Jose, Madan Musuvathi, Olli Saarikivi, Peng Cheng, Qinghua Zhou, et al. Msccl++: Rethinking gpu communication abstractions for cutting-edge ai applications. arXiv preprint arXiv:2504.09014, 2025.

[30] Adel Nabli, Louis Fournier, Pierre Erbacher, Louis Serrano, Eugene Belilovsky, and Edouard Oyallon. ACCO: Accumulate while you communicate for communication-overlapped sharded LLM training. arXiv preprint arXiv:2406.02613, 2024.

[40] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.

[31] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM symposium on operating systems principles, pages 1–15, 2019.

[41] Min Si, Pavan Balaji, Yongzhou Chen, Ching-Hsiang Chu, Adi Gangidi, Saif Hasan, Subodh Iyengar, Dan Johnson, Bingzhe Liu, Regina Ren, et al. Collective communication for 100k+ GPUs. arXiv preprint arXiv:2510.20171, 2025.

[32] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on GPU clusters using Megatron-LM. In Proceedings of the international conference for high performance computing, networking, storage and analysis, pages 1–15, 2021.

[42] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using DeepSpeed and Megatron to 16

train Megatron-Turing NLG 530B, a large-scale generative language model. arXiv preprint arXiv:2201.11990, 2022.

[52] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, et al. GSPMD: General and scalable parallelization for ML computation graphs (2021). URL http://arxiv. org/abs/2105.04663, 2021.

[43] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

[53] Jie You, Jae-Won Chung, and Mosharaf Chowdhury. Zeus: Understanding and optimizing GPU energy consumption of DNN training. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 119–139, Boston, MA, April 2023. USENIX Association.

[44] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.

[54] Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. Alpa: Automating inter-and intra-operator parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 559–578, 2022.

[45] Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, and Yuxiong He. Zero++: Extremely efficient collective communication for giant model training. arXiv preprint arXiv:2306.10209, 2023. [46] Hao Wang, Sreeram Potluri, Miao Luo, Ashish Kumar Singh, Sayantan Sur, and Dhabaleswar K Panda. Mvapich2-gpu: optimized gpu to gpu communication for infiniband clusters. Computer Science-Research and Development, 26(3):257–266, 2011. [47] Zibo Wang, Yuhang Zhou, Chen Tian, Xiaoliang Wang, and Xianping Chen. AFNFA: An approach to automate NCCL configuration exploration. In Proceedings of the 7th Asia-Pacific Workshop on Networking, APNet ’23, page 204–205, New York, NY, USA, 2023. Association for Computing Machinery. [48] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995– 2003. PMLR, 2016. [49] Udayanga Wickramasinghe and Andrew Lumsdaine. A survey of methods for collective communication optimization and tuning. arXiv preprint arXiv:1611.06334, 2016. [50] Guanbin Xu, Zhihao Le, Yinhe Chen, Zhiqi Lin, Zewen Jin, Youshan Miao, and Cheng Li. {AutoCCL}: Automated collective communication tuning for accelerating distributed and parallel DNN training. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 667–683, 2025. [51] Qifan Xu and Yang You. An efficient 2d method for training super-large deep learning models. In 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 222–232. IEEE, 2023. 17

Related documents

• Purlin: Separating Orchestration from the Datapath of Collectives#1132155
• Purlin: Separating Orchestration from the Datapath of Collectives#1122262
• Computing with Coupled Oscillator Networks: Optimization, Memory, and Modern AI Primitives#1491098
• Bridging the Deployment Gap: Integrating AI into Accelerator Control Systems#403519
• MCCL: Measurement-based Collective Communication Library#1341971