Characterizing the Scalability and Performance of Large-Scale AI Training Under Multi-Tenancy

Characterizing the Scalability and Performance of Large-Scale AI Training Under Multi-Tenancy 1st Jacopo Raffi

2nd Thomas Pasquali

3rd Lorenzo Piarulli

4th Filippo SPiga

5th Marco Faltelli

arXiv:2609.00817v1 [cs.DC] 1 Sep 2026

University of Trento University of Trento Sapienza University of Rome NVIDIA ENEA jacopo.raffi@unitn.it thomas.pasquali@unitn.it piarulli@di.uniroma1.it fspiga@nvidia.com marco.faltelli@enea.it

6th Andreas Herten

7th Domenico Siracusa

8th Daniele De Sensi

9th Flavio Vella

Forschungszentrum Jülich GmbH a.herten@fz-juelich.de

University of Trento domenico.siracusa@unitn.it

Sapienza University of Rome desensi@di.uniroma1.it

University of Trento flavio.vella@unitn.it

Abstract—Characterising AI workload performance on modern HPC systems requires understanding both their scalability in isolation and their behaviour under concurrent execution. However, the interplay among parallelisation strategies, network congestion, compute capability, and interconnect technologies remains poorly understood. This work investigates the performance and scalability of AI models up to 2400 GPUs. We quantify the communication overheads and their impact across different interconnects by evaluating scale-up, scale-out, and rack-scale configurations under multiple allocation schemes. Finally, we study how multiple concurrent training jobs interfere with each other by designing a realistic noise model. We design a benchmark suite of AI models to evaluate the performance of five distinct parallelisation strategies across different supercomputing clusters, including Alps, Leonardo, LUMI, JUPITER, NVL72 GB300, and DGX A100. Our work provides a systematic characterization of the scalability and execution efficiency of distributed AI training, while offering key insights into performance behavior under realistic multi-tenant scenarios. Index Terms—Deep learning, High performance computing, Scalability.

I. I NTRODUCTION The rapid growth of modern AI models has made large-scale distributed training a central challenge for high-performance computing systems. Training workloads now span hundreds to thousands of GPUs, driven by increasingly large models such as GPT-3 and LLaMA [1], [2]. While compute capabilities have grown rapidly, achieving efficient scaling remains difficult, as communication and network behavior often become the dominant factors in overall performance [3]–[5]. Distributed training relies on GPU-centric communication libraries, such as the NVIDIA Collective Communication Library (NCCL) [6], the ROCm Communication Collectives Library (RCCL) [7], and the oneAPI Collective Communications Library (oneCCL) [8]. As models scale, the cost of communication and synchronization increases, often limiting scalability despite abundant compute resources [3], [5]. In practice, training performance is determined by the interaction of multiple system and workload characteristics. Parallelization strategies define communication patterns and synchronization points; network topology and interconnect technologies determine how efficiently these patterns are executed; and job

placement and co-scheduled workloads influence the level of contention in shared environments. Prior work has provided important insights into individual aspects of this problem, including communication overheads and scalability limits [3], [5], topology-aware optimizations [4], and congestion effects in large-scale systems [9], and fine-grained characterization of power, performance, and hardware utilization on single- and multi-GPU nodes [10]. However, these studies typically consider only a subset of factors, often focusing on a single system, simplified network conditions, or isolated workloads. As a result, there is still limited understanding of how distributed AI training behaves under realistic conditions that combine large-scale, diverse parallelization strategies, complex network topologies, and multi-tenant execution. This limitation is particularly relevant in light of recent studies showing that production workloads and network traffic exhibit significant heterogeneity and complexity, which are not captured by commonly used models and assumptions [11], [12]. In this paper, we present a systematic study of distributed AI training performance across systems, scales, and architectures. We evaluate platforms ranging from tightly coupled nodelevel systems to rack-scale designs such as NVIDIA GB300 NVL72, and up to leadership-class supercomputers including Alps, JUPITER, Leonardo, and LUMI. Across these environments, we compare communication libraries from AMD and NVIDIA, and analyze a diverse set of interconnect technologies and topologies, including Dragonfly and Dragonfly+. This breadth allows us to identify performance trends that are consistent across systems, as well as those that are specific to particular architectural choices. To enable this analysis, we developed DLNetBench, a benchmark framework that reproduces the communication behavior and synchronization structure of modern distributed training workloads. DLNetBench directly issues the collective operations induced by each parallelization strategy, while approximating the computation–communication balance of workloads through hardware-aware timing models. This design enables precise control over job placement, reproducible experiments across systems, and systematic exploration of

concurrent workload configurations, which are difficult to achieve with existing deep learning frameworks. Using this framework, we study how network topology, job placement, and workload concurrency interact to determine the performance of distributed training workloads. Rather than analyzing these factors independently, we focus on their combined effects across systems and scales, and on how they shape both scalability and performance variability in realistic multitenant environments. Our results show that scaling behavior observed under isolated conditions can be misleading, as interconnect characteristics, job placement, and multi-tenant interference can fundamentally reshape performance at scale. By systematically exposing these effects across systems, this work provides practical guidance for designing, deploying, and evaluating large-scale AI training workloads in realistic environments, and helps identify the design choices that matter most for scalable and efficient training in next-generation AI systems and Giga-Factory-scale infrastructures. II. R ESEARCH Q UESTIONS AND M ETHODOLOGY To study distributed AI training under realistic conditions, we design a methodology that models three interacting factors: communication structure, computation time, and network interference. These factors are explicitly controlled to reproduce the execution behaviour of distributed training workloads while enabling systematic comparisons across systems, placements, and configurations. This methodology is motivated by the needs of two distinct user communities. The first comprises AI researchers and practitioners, who typically view the underlying infrastructure as an abstracted compute resource. From this perspective, the key challenge is understanding how network bottlenecks and communication overheads impact the scalability of different parallelization strategies. The second comprises system designers and operators, for whom the central question is how system-level choices influence application performance. In this setting, job placement, topology, and concurrent job interactions become first-order factors, as they can significantly influence scalability and communication efficiency, depending on both workload characteristics and infrastructure design. The methodology and experiments are designed to answer four main research questions. R1. How do compute capability and network characteristics bound the scalability of various parallelization strategies? R2. How does job placement within the network topology impact the scaling efficiency of different parallelization strategies? R3. How do system noise and contention affect training performance, and to what extent can placement choice mitigate or amplify these effects? R4. Are these phenomena visible only at large supercomputer scale, or do they already emerge at node-level and rack-scale systems? To address these questions, we model their interaction in a unified experimental setting. Communication is not modelled as a generic traffic pattern; instead, it is derived directly

from the distributed training strategy, thereby capturing the effects of different parallelization strategies, as these induce distinct collectives, message sizes, communicators, and synchronization points. At the same time, computation determines when communication operations are triggered and how they overlap in time. To capture this interaction and maintain a controlled setup, we approximate compute phases using a hardware-aware model based on roofline analysis [13], preserving the balance between computation and communication phases without relying on full framework execution (e.g., PyTorch). This model calibrates execution delays based on the arithmetic intensity of transformer layers and the theoretical peak throughput and memory bandwidth of the target GPU. Finally, network interference is introduced through concurrent job configurations that reflect the shared, multi-tenant nature of modern supercomputing and AI platforms, allowing us to study how contention affects performance under realistic workload conditions. We consider five primary parallelization strategies that span the design space of modern distributed training. Data Parallelism (DP) [14] replicates the model across devices and synchronizes gradients globally. Fully Sharded Data Parallelism (FSDP) [15] reduces memory footprint via hybrid sharding, distributing parameters within a group of devices and materializing them on demand. Combined Data and Pipeline Parallelism (DP+PP) [16] partitions the model sequentially across devices, while combined Data, Pipeline, and Tensor Parallelism (DP+PP+TP) [17] combines pipelining with intranode parameter sharding. Finally, combined Data, Pipeline, and Expert Parallelism (DP+PP+EP) [18] targets Mixtureof-Experts architectures, where tokens are dynamically routed to specialized sub-networks across different GPUs, introducing intensive all-to-all communication phases. Each strategy dictates a distinct set of collective operations. We categorize these communications into two domains. Intra-Replica (IR) operations handle the data exchanges required to execute a single model instance, such as passing activations between pipeline stages. Extra-Replica (ER) operations synchronize state, typically gradients, across multiple parallel instances of the model. These collectives are detailed in Table I. To ensure realistic communication volumes, we pair each strategy with representative model architectures, such as Vision Transformers (e.g., ViT-H [19]) and Large Language Models (e.g., LLaMA [20], Minerva [21], and Mixtral [22]), summarized in Table II. We implement these AI workloads in DLNetBench, a benchmark framework designed to expose and control the communication behaviour of distributed training workloads. DLNetBench directly issues the collective operations prescribed by each parallelization strategy, making communication patterns fully observable and reproducible across systems. Unlike wrappers around existing frameworks such as JAX [23] or DeepSpeed [24], DLNetBench enables explicit and finegrained control over job placement, communication structure, and concurrency, while avoiding framework-specific effects that prevent systematic comparisons.

DLNetBench does not execute real tensor operations. Instead, it replaces each compute phase with a calibrated sleep whose duration is derived from the roofline performance model [13]. This approach provides a hardware-aware estimate that captures the effective throughput limit imposed by either arithmetic throughput or memory bandwidth, while remaining independent of framework internals whose behaviour can vary across versions and configurations. For each target GPU architecture, we take peak floating-point throughput and peak memory bandwidth for the bfloat16 data type directly from the vendor datasheet, establishing a single, reproducible baseline that does not depend on empirical microbenchmarks. Consider a transformer layer with batch size B, sequence length N , hidden dimension d, feed-forward dimension H, number of experts E, number of active experts k, and element size s. For element size, we use s = 2 for bfloat16 all other parameters are determined by the model in use (see Table II). The per-layer costs are as follows: attn : FLOPs = 8BN d2 + 4BN 2 d, mlp : FLOPs = 4BN dHk,

Bytes = 4d2 s + 2BN d s, Bytes = 2dHsE + 2BN d s.

The arithmetic intensity I = FLOPs/Bytes determines whether execution is compute- or memorybound. The estimated per-layer compute time is t =
FLOPs/ min peak flops, I · bandwidth . The forward time sums tattn and tmlp over all L layers; the backward time is approximated as 2× the forward time. Since DLNetBench replaces real computation with a roofline-based timing model, we compared it against reference implementations in production frameworks. For example, on a single LUMI node (8 GPUs) the PyTorch DP implementation is approximately ≈ 2.5 slower end-to-end than DLNetBench under the same configuration. This is expected, since production frameworks may execute many more ML operators that our compute model intentionally abstracts away. Despite this mismatch in endto-end time, the dominant compute cost (GEMM) and the communication pattern of each parallelisation strategy remain fully controlled and consistent, which is sufficient for our goal of comparable, reproducible results and realistic performance estimation across systems. Across all parallelization strategies, we use a batch size of 16 and 16 microbatches; for pure data parallelism, gradients are partitioned into 50 buckets; for FSDP, we use 16 sharding units with a sharding factor of 8; and for the hybrid configurations, we use a pipeline parallelism degree of 8, combined with a tensor parallelism degree of 4 or an expert parallelism degree of 8. Here, the parallelism degrees specify the number of processes participating in each respective parallelization dimension, while the number of buckets or units determines the granularity at which gradients or model parameters are communicated. The suite supports multiple communication backends, including (CUDA-aware) MPI, NCCL [6], RCCL [7], and oneCCL [8]. To assess scalability, the relevance of network performance, and the impact of multi-tenancy, we conduct two classes of experiments: baseline experiments, in which a single instance

Strategy

Collectivescope

Per-rank msg size

DP

IA LL R EDUCE W

25.3 MB

FSDP

IA LL G ATHER IR R EDUCE S CATTER IR IA LL R EDUCE ER

125.5 MB / 115.6 MB 125.5 MB / 115.6 MB 125.5 MB / 115.6 MB

DP+PP

A LL R EDUCE ER S END R ECV IR

2.008 GB / 1.850 GB 67.1 MB / 33.6 MB

DP+PP+TP

A LL R EDUCE IR A LL R EDUCE ER S END R ECV IR

33.6 MB 4.4 GB 134.2 MB

DP+PP+EP

A LL R EDUCE ER A LL R EDUCE IR A LLT OA LL IR S END R ECV IR

905.3 MB 200.7 MB 8.0 MB 33.6 MB

TABLE I: Collectives, communicator scopes, and per-rank message sizes for each parallelism strategy. For FSDP and DP+PP, sizes are reported as Llama3-8B / Minerva-7B. Subscripts denote the communicator scope: W (world, all ranks), IR (intra-replica, within a model replica), and ER (extrareplica, across model replicas).

of DLNetBench runs alone, and concurrent experiments, in which multiple instances are co-scheduled. We perform both classes of experiments under two settings: first, in isolation on reserved portions of the system, and second, through the production SLURM queue, reflecting the experience of a typical user. In the reserved setting, each experiment is allocated only a portion of the system (3 Dragonfly groups), and the resulting network traffic is generated by DLNetBench instances. In contrast, when jobs are submitted through the regular SLURM queue, resource allocations are determined by SLURM, and the network is shared with traffic generated by workloads from other users. Concurrent configurations vary in both workload mix and scale, enabling us to emulate realistic contention scenarios based on the heavily skewed workload distributions observed in production AI clusters [11], [25], where a small number of massive pre-training jobs coexist with numerous smaller allocations. We treat each co-scheduled job simultaneously as a potential source of congestion and as a subject of measurement, departing from the conventional aggressor–victim dichotomy [26], [27] to capture the bidirectional interference inherent in shared AI infrastructure. Experiments construction, job placement strategies, measurements, and metrics are detailed in Section IV. DLNetBench and all related scripts are publicly available at: https: //github.com/HicrestLaboratory/DLNetBenchSC26/tree/main III. S YSTEMS D ESCRIPTION In the following, we describe the main characteristics of the systems used in our experimental campaign. For the analysis of scale-up scalability, we consider two node-level systems: an NVIDIA DGX A100 platform and a LUMI-G compute node. For rack-scale experiments, we use the NVIDIA NVL72 GB300 platform. For the supercomputer-level analysis, we

Strategy DP [14] (D) FSDP [15] (FSDP) DP+PP [16] (DP) DP+PP+TP [17] (DPT) DP+PP+EP [18] (DPE)

Model(s) ViT-H [19] (ViT-H) LLaMA3-8B [20], Minerva-7B [21] (LaM-8) LLaMA3-8B [20], Minerva-7B [21] (Minv) LLaMA3-70B [20] (LaM-70) Mixtral-8x7B [22] (Mxt)

TABLE II: Parallelization strategies and associated models used in DLNetBench. Between brackets, we report the short name version we will use during results analysis in Section V. focus on Alps, Leonardo, LUMI, and JUPITER. The summary of the main systems’ characteristics are reported in Table III. A. NVIDIA DGX A100 The NVIDIA DGX A100 [29], [30] is a node-scale 8GPU platform designed for AI workloads. It integrates eight NVIDIA A100 Tensor Core GPUs interconnected through third-generation NVLink and six second-generation NVSwitch chips, forming a fully connected intra-node GPU fabric. Each A100 GPU exposes twelve NVLink connections, providing up to 300 GB/s per direction to the NVSwitch fabric. By terminating GPU links on the NVSwitches rather than directly on peer GPUs, the DGX A100 provides a switched all-to-all topology with uniform communication characteristics across all eight accelerators. As a result, any GPU can communicate with any other at node-local bandwidth, and the system can sustain up to 4.8 TB/s of aggregate bidirectional GPU-to-GPU traffic when all eight GPUs communicate concurrently at peak rate. B. NVIDIA NVL72 GB300 For the rack-scale experiments, we consider an NVIDIA GB300 NVL72 system, which integrates 72 Blackwell Ultra GPUs and 36 Grace CPUs, delivering 360 PFLOP/s FP16/BF16, within a single liquid-cooled NVLink domain. The rack is composed of 18 compute trays and 9 NVLink switch trays: each compute tray hosts 2 GB300 Superchips (2 Grace CPUs and 4 GPUs), while each switch tray hosts 2 NVLink switch chips, for a total of 18 switches. Using fifthgeneration NVLink and the NVLink Switch System, the rack delivers 130 TB/s of aggregate NVLink bandwidth and fullmesh connectivity across all 72 GPUs. Each GPU exposes 18 NVLink ports, one to each switch chip, providing up to 1.8 TB/s of low-latency GPU-to-GPU bandwidth (Fig. 1). As a result, all GPU communication remains within a single racklocal high-bandwidth fabric [31]. C. Alps Alps is a 434.90 PFlop/s supercomputer deployed by CSCS [32]. The system is composed of HPE Cray Supercomputing EX254n blades, each hosting two nodes. Node architecture. Each node is composed of four GH200 Grace Hopper Superchips [33], [34] connected in an all-to-all topology using NVLink 4.0. Six 200 Gb/s links connect each GH200 pair, for a total of 1.2 Tb/s between any GPU pair.Inter-node connectivity. Each node has one HPE Cray Cassini-1 200 Gb/s Network Interface Card (NIC) for each GH200. Nodes

Fig. 1: NVIDIA GB300 NVL72 architecture, view of NVLink connectivity from a single compute tray. are connected in a Dragonfly topology [35] through an HPE Cray Slingshot-11 network [27], [36]. Each switch has 16 ports to endpoints, 31 ports to other switches within the same Dragonfly group, and 17 ports to switches in other Dragonfly groups. D. Leonardo Leonardo [37] is a EuroHPC pre-exascale supercomputer owned by the EuroHPC Joint Undertaking and hosted by CINECA. In the November 2025 TOP500 list, it ranks 10th with an HPL performance of 241.2 PFlop/s. We focus on its Booster GPU partition, comprising 3456 compute nodes. Node architecture. Each Booster node comprises a singlesocket 32-core Intel Xeon® 8358 CPU and four NVIDIA A100 Tensor Core GPUs [30], for a total of 13 824 GPUs. The GPUs are interconnected through NVIDIA NVLink 3.0, while communication with the host CPU and the NIC is provided through PCIe® Gen4 x16 links. Inter-node connectivity. Nodes are interconnected by an InfiniBand HDR200 network. Each Booster node is equipped with two dual-port NVIDIA ConnectX-6 adapters, providing four 100 Gb/s network ports, which we model as four independent NICs. The interconnect follows a Dragonfly+ topology, in which each group contains 180 nodes and is organized as a two-level fat tree with 18 leaf and 18 spine switches. Each switch provides 40 ports at 200 Gb/s, configurable as dual 100 Gb/s links. Within a group, each leaf switch connects 10 nodes through 40 100 Gb/s ports and to the spine layer through 18 200 Gb/s uplinks, leaving two 200 Gb/s ports unused; each spine switch connects to the 18 leaf switches and uses the remaining 22 200 Gb/s ports for inter-group links. E. LUMI LUMI is a 380 PFlop/s supercomputer owned by the EuroHPC Joint Undertaking and hosted by CSC [38]. This paper considers the LUMI-G GPU partition consisting of 2978 nodes. Node architecture. Each node is built around a 64-core AMD EPYCTM 7A53 “Trento” CPU with 128 GB of DDR4 memory, and four AMD MI250X GPUs [39]. Each MI250X

TABLE III: Comparison of supercomputers used in this manuscript. Between brackets, we report the TOP500 [28] rank. Alps (#8)

JUPITER (#4)

Leonardo (#10)

LUMI (#9)

CPU GPU NIC

4× NVIDIA Grace (72 core) 4× NVIDIA Hopper 4× HPE Cray 200 Gbit/s Cassini-1 10 644 PFLOP/s NVLink 4.0, 6× links (1200 Gbit/s) between any GPU pair HPE Cray Slingshot 11, Dragonfly topology

Software Environment

prgenv-gnu/26.3:v1 uenv: GCC 14.3.0, Cray MPICH 9.1.0, libfabric 2.4.0, CUDA 13.1.1, NCCL 2.29.21 (with AWS-OFI-NCCL 1.17.2)

Intel Ice Lake Xeon 8358 (32 core) 4× NVIDIA Ampere A100 2× dual-port NVIDIA Connect-X6 (100 Gbit/s per port) 4313 PFLOP/s NVLink 3.0, 4× links between any GPU pair (800 Gbit/s) NVIDIA InfiniBand HDR, Dragonfly+ topology (23 groups, each a 2level fat-tree) GCC 12.2.0, Open MPI 4.1.6, CUDA 12.4, NCCL 2.18.5

AMD Trento EPYC 7A53 (64 core) 4× AMD MI250X (8 GCDs) 4× HPE Cray 200 Gbit/s Cassini-1

bfloat16 Intra-node Interconnect Inter-node Interconnect

4× NVIDIA Grace (72 core) 4× NVIDIA Hopper 4× NVIDIA Mellanox 200 Gbit/s Connect-X7 23 300 PFLOP/s NVLink 4.0, 6× links (1200 Gbit/s) between any GPU pair NVIDIA InfiniBand NDR, Dragonfly+ topology (25 groups, each a 2level fat-tree) GCC 14.3.0, Open MPI 5.0.8, CUDA 13.0.0, NCCL 2.29

module consists of two Graphics Compute Dies (GCDs), so that a single node contains eight GCDs in total. Every GCD is paired with a 64 GB HBM memory stack, for an aggregate of 128 GB of HBM memory per MI250X module. Because the two GCDs of each MI250X are exposed as separate devices in the software stack, we model a LUMI-G node as an 8GPU node. From an interconnect perspective, each GCD is attached to a CPU NUMA domain through a 288 Gb/s AMD Infinity FabricTM link, while pairs of GCDs are interconnected by one to four 400 Gb/s Infinity Fabric links. Inter-node connectivity. Each MI250X module is connected to a 200 Gb/s Cassini-1 NIC. Nodes are connected in a Dragonfly topology through an HPE Cray Slingshot-11 interconnect composed of 24 groups, with 124 nodes per group. There are 31 switches per group, connected in an all-to-all manner. Each switch has 64 ports, 16 connected to the nodes and 48 connected to other switches. Each node is connected to two different switches in the same group. Each switch has 16 ports to endpoints, 31 ports to other switches within the same Dragonfly group, and 17 ports to switches in other Dragonfly groups. F. JUPITER JUPITER is a EuroHPC JU supercomputer hosted at Forschungszentrum Jülich, Germany. Currently in buildup [40], it is the first exascale system in Europe and ranks #4 on the TOP500 list, achieving a performance of 1 EFLOP/s. Node architecture. JUPITER uses 23 536 NVIDIA GH200 superchips, a tight combination of a 72-core ARM-based Grace CPU (120 GB LPDDR5 memory) and a Hopper GPU (96 GB HBM3 memory), connected with a 900 GB/s NVLink C2C bus. Each node features four GH200 superchips, with direct connections between all GPUs (300 GB/s pair-wise, bidirectional bandwidth) and all CPUs (200 GB/s pair-wise, bidirectional bandwidth). Inter-node connectivity. The nodes of JUPITER are interconnected by an InfiniBand NDR-200/400 network in Dragonfly+ topology. Within a local Dragonfly group, up to 240 nodes are connected in a fat-tree across two levels of switches, 15 lower L1 switches and 16 upper L2 switches. A node features four InfiniBand adapters, which connect to L1 switches with 200 Gbit/s bandwidth per link. Between all switches, the link bandwidth is 400 Gbit/s. In total,

4562 PFLOP/s Between one and four 400 Gbit/s Infinity Fabric links to other GCDs HPE Cray Slingshot 11, Dragonfly topology (24 groups) Cray clang 16.0.1, Cray MPICH 8.1.27, libfabric v1.15.2, ROCm 6.3.4, RCCL 2.21.5 (with AWSOFI-RCCL 1.4)

25 of these groups form the overall JUPITER topology, with 30 direct links available between each pair of groups, and further bandwidth available through adaptive routing. IV. E XPERIMENTAL S ETUP This section describes how we instantiate the methodology from Section II into concrete experiments. We define how we construct baseline experiments, concurrent configurations, and how jobs are placed. Finally, we define the terminology and metrics (e.g., slowdown) used throughout the results analysis in Section V. A. Experiment Construction 1) GPU counts per strategy: Not every strategy is meaningful at every scale. This variation arises from fundamental architectural constraints and convergence trade-offs. For multi-dimensional strategies, a single model replica often spans many devices. For instance, a DP+PP+TP configuration with 8 pipeline stages and 4 tensor parallel shards requires 32 GPUs just to host one model replica; applying data parallelism on top of this requires scaling in multiples of 32. Similarly, a DP+PP+EP setup with 8 pipeline stages and 8 experts requires a minimum baseline of 64 GPUs per replica. Conversely, pure Data Parallelism (DP) is impractical at massive scales for models like ViT-H. Scaling pure DP linearly increases the global batch size, which can severely degrade training convergence. Therefore, we define the feasible GPU counts for each strategy as follows: DP ∈ {2, 4, 8, 16, 32, 64}; FSDP ∈ {4, 8, 16, 32, 128, 256}; DP+PP ∈ {4, 8, 16, 32, 64, 128, 256}; DP+PP+TP ∈ {256, 512, 1024}; and DP+PP+EP ∈ {512, 1024, 2048}. In these sets, bold denotes GPU counts used in concurrent experiments, italics indicate rack/node-scale-only runs, and plain text represents baselines only. 2) Allocation patterns: An allocation pattern specifies how aPsystem’s G GPUs are divided into job-slots (g1 , . . . , gk ) with 1≤j≤k gj ≤ G. We generate patterns from two families. Uniform patterns assign the same GPU count to every slot: (g, g, . . . , g). The tier-sampled patterns draw slot sizes from a weighted tier distribution that explicitly categorizes jobs into three probabilistic tiers: small (75% probability, 8 GPUs), medium (20%, 16, 32, 64, and 512 GPUs), and large (5%,

1024 GPUs). A pattern is constructed by repeatedly sampling a tier, then a GPU count within that tier, and appending the slot if it fits within the remaining budget, stopping once the budget is exhausted. On LUMI, the GPU counts are doubled since it has 8 GPUs per node instead of 4. 3) Strategy Assignment and Configuration Diversity: Given a pattern (g1 , . . . , gk ), a strategy assignment ϕ maps each job-slot j to a strategy Si ∈ {DP, . . . , DP+PP+EP} with gj being a feasible GPU count for that strategy. The resulting set of valid assignments is exponentially large, so we sample it in a stratified manner using a mixture entropy score that captures P how evenly slots P are distributed across strategies: Hϕ = − i p̃i log p̃i , p̃i ∝ j | ϕ(j)=Si gj . Hϕ = 0 when all slots use the same strategy; Hϕ = log m (with m strategies) when GPU usage is perfectly balanced across strategies. Because evaluating every possible strategy assignment is computationally infeasible, we group the entropy scores, [0, logm], into three bins representing low, medium, and high diversity. By randomly selecting a few configurations from each bin, we can efficiently evaluate the entire spectrum of system behaviours, ranging from isolated strategies to complex, mixed workloads, without needing to test every single permutation. 4) Network Topology and Job Placement: We model each system as a multi-level switching hierarchy: a node groups 4 or 8 GPUs connected all-to-all; an L1 switch aggregates typically 10 to 16 nodes; a group connects multiple L1 switches (roughly 120 to 240). Concrete parameters for each system have been given in Section III. For a job occupying g GPUs, we define its placement class based on how the GPUs are distributed across the network hierarchy: (i) I NTRA -L1: GPUs span multiple nodes within one L1-switch domain. (ii) I NTRA G ROUP: GPUs span multiple L1 switches within one group. (iii) I NTER -G ROUP: GPUs span two or more groups. (iv) We also consider a deep-learning-motivated variant in which GPUs are arranged in blocks of n nodes each pinned to the same L1 switch (I NTRA /I NTER -G ROUP -S AME -L1-n); the latter are topology-aware allocations that aim at keeping Intra-Replica traffic local. To compare placement classes numerically, we assign each a locality score (1 through 3 for the three classes above, with fractional values for the S AME -L1 variants) that approximates the average network distance between GPUs. For each strategy assignment, we vary the distribution of job-slots across the system topology. Each job-slot j is assigned a placement class κj . The mean locality score of a Pk configuration, s̄ = k1 j=1 locality score(κj ), summarizes its aggregate network locality. Analogously to the entropy bins, we partition the range [1, 3] into three locality bins and draw a small, fixed number of placements per bin per assignment, covering the spectrum from predominantly I NTRA -L1 to predominantly I NTER -G ROUP configurations. B. Measurements, Baselines and Slowdown Metric Each DLNetBench application assigns one MPI rank per GPU. Before any measurement is taken, one warmup iteration is performed and excluded from all results to avoid cold-start effects. After the warmup, all DLNetBench instances perform

multiple iterations. The number of iterations has been chosen depending on the parallelization strategy and system to ensure neither too short nor too long runtimes, for instance, DP performs from 6 to 10 iterations while DP+PP+EP from 2 to 4. For each iteration, we collect the total runtime and instrument all communication operations: blocking collectives are timed directly, whereas for non-blocking collectives we record the time spent at the synchronization barrier, capturing the portion of communication that is not hidden behind computation. Even though the timing of a blocking “send” does not necessarily correspond to the full duration of the underlying data transfer, we still measure and report the time associated with that collective call, as it reflects the overhead experienced by the application. In multi-tenant scenarios, we set a global timeout (around 4 minutes), and we respawn DLNetBench instances as soon as they finish on the same nodes. We keep the results produced by all runs and iterations (excluding warmups). 1) Baselines: Every valid (strategy, GPU count, placement) triple is measured in both isolation (spanning at most 3 Dragonfly groups) and asking SLURM to allocate resources as a normal user would do. The objective is to characterize the per-strategy performance and scalability before any deliberate concurrent workload is introduced. 2) Communication Relevance: To characterize the compute-to-communication balance of each workload on each system, we compute for each baseline the fraction of total runtime spent in communication. This ratio serves as a proxy for how network-sensitive a given strategy and model combination is: workloads with a high communication fraction are expected to be more susceptible to network congestion, while those dominated by compute are less so. The communication time per iteration is defined as the sum of all blocking collective durations plus, for non-blocking collectives, the time spent waiting at the synchronization point, that is, the residual communication cost that overlapping computation could not absorb. We call this value sync time. We compute this metric by first computing all sync time/tot time ratios, for each run we take the maximum across ranks, and we then compute the arithmetic mean across runs [41]. 3) Throughput: Training throughput is reported in samples per second, where one sample corresponds to a single training instance (an image or a fixed-length token sequence). Across all parallelization strategies, the total number of samples processed per iteration is simply the global batch size, which is given by the local batch size multiplied by the total number of model replicas. In our setup, each rank reports the total throughput based on its iteration runtime. For each iteration, we select the minimum throughput (that belongs to the rank that finished last) [41]. Then, we gather the throughput of all runs. For the scalability analysis in Section V, we report the geometric mean of iteration throughputs. 4) Slowdown: For a job running concurrently with others, slowdown is defined as the ratio of its baseline throughput baseline to its throughput under concurrent execution: σ = TTconcurrent

σ = 1 indicates no degradation; (σ − 1) × 100 is the percentage throughput loss, often referred to as congestion impact [26]. For a given strategy Si , GPU count g, and placement class κ within a concurrent configuration, we gather all available throughput measurements according to what was just described in Section IV-B3, then we compute slowdowns using the corresponding baseline throughput (geometric mean of its iterations’ throughputs).

computation. Hybrid strategies (DP+PP, DP+PP+TP, and DP+PP+EP) show overall stable communication costs generally in the 30–50% range, indicating a stronger dependence on network performance. In contrast, DP shows the highest communication percentages, with fractions that grow rapidly as the system scales due to its reliance on global IA LL R E DUCE operations, often exceeding 70% of time spent for communication on larger scales.

V. R ESULTS A NALYSIS

Observation 1: These results directly inform R1: communication overhead is minimal intra-node but grows sharply once it crosses nodes, and systems with lower compute-tonetwork balance (e.g., Leonardo) overlap communication better.

This section analyzes the experimental results through the lens of our research questions (Section II), connecting observed performance trends to architectural characteristics, parallelization strategies, and placement decisions. We secured reservations spanning three Dragonfly(+) groups on all considered large-scale systems. In particular, 2400 GPUs on JUPITER, 1860 on Leonardo, 1320 on Alps (Clariden), and 2720 GPU modules on LUMI. In some cases, some faulty or underperforming nodes have been excluded as indicated by system administrators. The provided figures already consider this. Even within these reserved groups, adaptive routing in production environments may still introduce residual external interference. For our dedicated testbed systems, namely the NVL72 GB300, DGX A100, we omit user-perspective measurements (job submissions on the production SLURM queue), as their low utilization does not provide representative background traffic. On each system, we run all baselines and around 15-25 concurrent experiments constructed as described in Section IV. Collectively, we ran more than 450 experiments for a total of around 5,300 node-hours. All figures in this section use the short names of parallelization strategies and models provided in Table II. A. Communication Relevance Fig. 2 reports the fraction of time spent in communication (detailed in Section IV-B2) across all systems and strategies. On node- and rack-scale systems (DGX A100 and NVL72 GB300), communication overhead is generally limited, although it is not negligible at larger scales. DGX A100 consistently remains below roughly 10% communication time, indicating that execution is largely compute-bound. NVL72, while still benefiting from high-bandwidth NVLink/NVSwitch interconnects, exhibits higher variability, with communication fractions increasing with scale and reaching up to approximately 41–58% in some configurations. On supercomputers, communication becomes a dominant factor even at small scale. With only a few nodes, communication already accounts for roughly 30% to over 60-80% of total runtime. This highlights the cost of leaving the node boundary: scale-out communication is significantly slower, leading to both higher absolute communication time and a larger fraction of the total execution. Across strategies, communication sensitivity varies significantly. FSDP consistently exhibits the lowest communication fraction (always below 10%), demonstrating its ability to effectively overlap communication (IA LL G ATHER) with

B. Scalability and Parallel Efficiency Fig. 3 shows that near-ideal scaling is achievable only in regimes where communication remains either negligible or perfectly provisioned. On the DGX A100, all strategies achieve ≥99% efficiency, confirming that NVSwitch effectively removes communication bottlenecks. The fact that the DGX A100 NVSwitch can perfectly provision communication may be due to the computenetwork balance discussed in Observation 1. NVL72 largely preserves this behavior; however, DP degrades to 80–85% at larger GPU counts (32–64 GPUs). This drop is driven by the increasing number of participants in the IA LL R EDUCE W . The efficiency drop is coherent with the 20% of time spend in communication shown in Fig. 2. Regarding FSDP and DP+PP, the percentage of time spent in communication remains stable and so does the parallel efficiency. This shows that the high-bandwidth interconnect can perfectly satisfy the communication needs of the hybrid approaches while it is not able to perfectly provision for concurrent IA LL R EDUCE. Note that this effect it is probably due to the really high compute capabilities of the Blackwell GPUs, which limit the amount of communication overlapped with compute, thus fully saturating the available bandwidth. On supercomputers, scalability becomes strongly strategydependent. FSDP, DP+PP, DP+PP+TP, and DP+PP+EP maintain high efficiency, typically above 80–95%. On JUPITER, the efficiency trends are less stable; in particular, the efficiency of DP+PP, DP+PP+TP, and DP+PP+EP drops to around 75%. DP efficiency always degrades significantly. Every time the number of GPUs is doubled, the efficiency drops by 10–30%, in some cases going below 50% when running on 64 GPUs. Scalability is fundamentally bounded by the interaction between communication patterns and the interconnect, even with lower compute capabilities. The root cause lies in the communication structure. For pure DP on large scales, the communication overhead of global IA LL R E DUCE operations increases, reducing the overlap between communication and computation and severely reducing scaling. In contrast, FSDP maintains high efficiency because its IA LL G ATHER operations achieve good overlap with the

NVL 72

Leonardo

Alps (Clariden)

37-48 37-48

43 44 44

49-51

JUPITER

44 44 51 51

LUMI

44-46 44-47 48-50

35 36 36 41 41 41

51-56 51-59 52-56

Communication Time (%)

DGX A100

Strategy

DPE Mxt DPT LaM-70 DP Minv DP LaM-8 FSDP Minv FSDP LaM-8 D ViT-H 0 2

80

7

8

56

56

58

39 41 42 43 43

47-51 50-64 54-64 54-58 56-62

46-48 49-55 55-59 55-60 56-61

7 8 34 36 37 37 38

60

6

7

42

41

41

36 37 37 38 38

39-44 42-46 43-46 44-46 45-48

40 42-43 43-46 45-46 43-46

7 7 32 33 33 33 34

40

1

1

2

2

3

5-7

6-7

6-7

6-7

6-7

5

6-7

6-10

6

6-9

5 1 1 1 1 1 1

0

0

3

3

3

3

3

3

3

3

3

3

1 4

1 8

GPUs

4 8

3

4

4

4

4

4

4

4

2

2

2

2

2

2

1

1

1

2

2

9 16

19 32

22 64

3-6

32-35

GPUs

8

43 48 16 32 64 128 256 512 1K

GPUs

43-44 65-86 47-67 59-80

16

8

64 128 256 512 1K

32

GPUs

20

2 1 1 1 1 1 1

40-47 59-64 63-67 69-75

8 16 32 64 128 256 512 1K 2K

GPUs

1 1 2 74-7981-83 86 2 4 8 16 32 64 128 256 512 1K 2K

GPUs

Throughput (samples/s)

Fig. 2: Percentage of runtime spent in communication (detailed in Section IV-B2) . Ranges represent the minimum and maximum values obtained with different placement classes (when available and relevant). DGX A100

2K 0.5K

82K

42K 23K

14 7

100

13 6

36

5

2

8

4

DGX A100

0.4K

0.5K

0.2K

0.1K

98 0.1K

8

100

99

0.2K 0.3K

0.9K

0.3K

0.4K 0.5K

64

0.2K

16 146

0.1K

60

16

32

NVL 72

64

2

4

GPUs D - ViT-H - Intra Gr D - ViT-H - L1 D - ViT-H - Multi Gr D - ViT-H - N/A

8

25 32 15

Leonardo

70 60 8

DP - LaM-8 - Intra Gr DP - LaM-8 - Intra Gr 2N/S DP - LaM-8 - L1 DP - LaM-8 - Multi Gr

16

GPUs

32

DP - LaM-8 - Multi Gr 2N/S DP - LaM-8 - N/A DP - Minv - Intra Gr DP - Minv - Intra Gr 2N/S

64

Alps (Clariden)

32K 8K 3K 2K

14K 12K 8K 4K 2K 2K 1K

0.5K 0.1K

89 87

32

39 47 46 45 22

8K 3K

0.7K 0.6K 0.4K 0.3K 0.7K 0.6K 0.2K 0.2K 0.3K 0.3K 79 0.2K 0.2K 89 88 87 86 85 84 83 36 3235 31 44 43 42

0.1K 0.1K

60

8 16 32 64 128 256 512 1K

32

Alps (Clariden)

22 21

100

JUPITER

90

60

GPUs

DPT - LaM-70 - Intra Gr 4N/S DPT - LaM-70 - Multi Gr DPT - LaM-70 - Multi Gr 4N/S DPT - LaM-70 - N/A

100

LUMI

80 70 60

50 8 16 32 64 128 256 512 1K

1

2 4 8 16 32 64 128256512 1K 2K

90

80

40

DPE - Mxt - Intra Gr DPE - Mxt - Multi Gr DPE - Mxt - N/A DPT - LaM-70 - Intra Gr

89 88 46 39 50 45

1K 0.7K 0.6K 0.4K 0.3K 0.3K 0.6K 0.2K 0.2K 0.1K 0.3K 0.3K 92 79 0.2K 0.2K 69 0.1K 0.2K 89 83 82 81 82 81 80 79 78 3135 59 42 41

LUMI

16K 4K 1K 1K 1K 1K 1.0K 0.8K 0.6K 0.6K 0.5K 1K 0.4K 0.7K 0.7K 0.3K 0.3K 0.1K 0.1K 0.3K 72 62 0.2K 36 31 86 64 33 87 80 68 18 15 43 16 44 40 34 9 7 22 20 16 78 17 4 3 11 10 8 4 5 14 3

8 16 32 64 128 256 512 1K 2K

70

GPUs

DP - Minv - L1 DP - Minv - Multi Gr DP - Minv - Multi Gr 2N/S DP - Minv - N/A

0.1K

60

20

5K

0.5K 70

16K

9K

2K

1K

8 16 32 64 128 256 512 1K

100

JUPITER

32K

21K 17K 10K

80

80

85

99

12 16 7

1K 0.9K 0.8K 0.5K 0.4K 0.2K 0.2K 0.1K 0.1K 0.3K 51 50 0.1K 0.1K 26 64 60 47 38 30 1113 23 19

90

90

99

57 28

9K

5K

8 16 32 64 128 256 512 1K

100

95

99

3K

1K

2K

8

Leonardo

16K 4K 2K

12K

8K

32

99

NVL 72

131K 32K

0.1K

2

Parallel Efficiency (%)

2K

1K 0.6K

8 16 32 64 128 256 512 1K 2K

FSDP - LaM-8 - Intra Gr FSDP - LaM-8 - Intra Gr 2N/S FSDP - LaM-8 - L1 FSDP - LaM-8 - Multi Gr

GPUs

FSDP - LaM-8 - Multi Gr 2N/S FSDP - LaM-8 - N/A FSDP - Minv - Intra Gr FSDP - Minv - Intra Gr 2N/S

2 4 8 16 32 64 128256512 1K 2K

GPUs

FSDP - Minv - L1 FSDP - Minv - Multi Gr FSDP - Minv - Multi Gr 2N/S FSDP - Minv - N/A

Fig. 3: Parallelization strategies scalability and parallel efficiency. Efficiency is computed with respect to the lowest GPU count run; an exception is made for LUMI multi-node experiments, which are normalized on 16 GPUs’ throughput . We report the geometric mean of throughputs computed as explained in Section IV-B3. The dotted lines indicate the ideal scaling. The y-axis ranges are fitted to the data for a better visualization. compute phase, and its R EDUCE S CATTER operations are confined to small communication groups (spanning only 8 GPUs, typically across two nodes, or a single node on LUMI). For the hybrid strategies, although DP+PP+TP introduces more communication overhead compared to DP+PP, its tensor parallel shards are strictly intra-node. Furthermore, the significantly larger compute workload of LLaMA-3-70B, compared to smaller models like LLaMA-3-8B or Minerva-7B, increases the compute-to-communication ratio, reducing the relative impact of these overheads on overall efficiency. DP+PP+EP benefits from this same favorable ratio. However, while its all-to-all communication is logically restricted to small 8GPU expert groups, scaling up the total number of GPUs increases the number of model replicas. Because the model is divided into 8 pipeline stages, and each stage operates its own expert communicator, every newly added replica introduces 8 additional independent all-to-all communicators (each fixed at 8 GPUs) to the system. We hypothesize that this multiplication of concurrent transfers might introduce network noise, as these numerous logically independent communicators could begin to disturb one another. This speculation is supported by an observation on JUPITER efficiency in Fig. 3. For instance, a DP+PP+EP configuration localized within 3 Dragonfly

groups (purple rhombus) achieves noticeably higher efficiency than a scaled-out configuration spanning 17 groups (purple circle). Similarly, the efficiency drop observed for DP+PP+TP is for runs spanning 3 and 18 groups. Absolute throughput is primarily driven by raw compute capability, with systems equipped with more powerful GPUs consistently achieving higher peak values, an ordering that holds clearly across all platforms. On node- and rack-scale systems, throughput scales smoothly with GPU count and remains close to the expected behaviour, indicating that communication does not significantly limit performance at this scale. On supercomputers, scaling remains broadly consistent with compute capability, but some deviations emerge at larger scale, with higher variability across systems suggesting that communication effects begin to play a role. A comparison between JUPITER and Leonardo illustrates the dominant role of compute: despite sharing a Dragonfly+ topology, JUPITER achieves around twice the throughput for all parallelization strategies thanks to more powerful GPUs and interconnect bandwidth, while both systems follow similar scaling trends. Comparing JUPITER and Alps instead isolates the effect of the interconnect: with similar compute capability, their throughput is broadly comparable across scales, with no consistent advan-

Observation 2: Network placement has little to no impact on scaling trends, addressing R2. C. Impact of Network Congestion and Multi-Tenancy We now analyze the distribution of slowdown ratios σ under co-scheduling. These results expose how contention interacts with communication patterns and job placement. In Figs. 4 to 8, each x-axis tick corresponds to a specific class of DLNetBench instances, defined by the combination of parallelization strategy, model, scale (number of GPUs), and placement class (in total we consider 32 distinct combinations). When multiple placement classes are available, they are grouped, as indicated by horizontal brackets below the tick labels. Where reserved and production-queue runs are both available for a given configuration, the corresponding tick is split into two adjacent violins, one per class. The y-axis reports the throughput slowdown ratio σ (defined in Sections IV-B3 and IV-B4). For each configuration, we visualize the distribution of slowdowns using boxplots overlaid with violin plots. Boxes represent the interquartile range (IQR, 25th–75th percentile), with the median shown as a solid black line and the mean as a white diamond. Whiskers extend to the most extreme values within 1.5 × IQR. Dashed brackets alongside each violin partition the samples into disjoint y-axis intervals, annotated with the count (or count/total) of runs falling within each. To preserve readability, the y-axis is clipped to a fixed range and, for some systems, further split into multiple panels with independent sub-ranges (indicated by diagonal break marks) so that widely separated clusters of slowdown values can each be shown at legible resolution. For configurations with values exceeding the visible range entirely, a red annotation above the plot indicates both the fraction of clipped points and the maximum observed slowdown. Colors denote the parallelization strategy. Configurations for which no slowdown outside the range ±10% was measured are omitted from the plots; the absence of a given strategy/model/scale/placement configuration therefore indicates that no slowdown was observed for it. 1) Scale-Up Systems: On DGX A100 (Fig. 4), slowdowns are generally below 10–15%, confirming that intra-node NVSwitch fabrics are not significantly impacted by congestion. However, some extreme outliers (up to 21×) indicate that even in tightly coupled systems, transient interference or system noise can disproportionately affect communication-heavy workloads (further details are discussed in Section V-C4).

7% 11% (6.4x) 3k (21.2x) 940

Slowdown

tage for either system, indicating that the Dragonfly+ topology does not translate into a clear throughput difference between the two. The throughput scaling trend is largely unaffected by placement across all systems (Fig. 3), with curves from different placement classes remaining close to one another. Instead, compute capability and interconnect bandwidth are the primary factors shaping throughput. The limited impact of placement arises because communication saturates the links bandwidth, effectively masking the additional latency associated with placements spanning greater network distances (in the scenarios in which baselines are run in isolation).

2% (0.2x) 28/120

12% 14/62

6%

60

30

0%

48/62

D ViT-H 2 GPU

D ViT-H 4 GPU

FSDP LaM-8 4 GPU

FSDP Minv 4 GPU

92/120

DP LaM-8 4 GPU

DP Minv 4 GPU

Fig. 4: Distribution of slowdown on individual application runs when co-scheduled with other concurrent jobs on DGX A100.

Negative slowdowns (speedups) are due to transient noise in the measurements. In contrast, on the NVIDIA NVL72, we do not observe significant slowdowns. All slowdowns are below 1.5%, with extremely rare exceptions reaching 4%. Despite its significantly higher compute performance and the corresponding increase in communication demand, the NVSwitch-based fabric consistently sustains all concurrent workloads without observable degradation, effectively eliminating congestion as a performance factor. Observation 3: To answer R4, congestion effects already emerge at node-level scale, though far less severely than at large supercomputer scale: the DGX A100 shows generally low slowdowns with some extreme outliers, whereas the rack-scale NVL72 shows no meaningful slowdown. 2) JUPITER: In Fig. 5, we observe that when all GPUs are located under the same switch (Intra L1), performance remains mostly unaffected by congestion (≤15%). While some outliers in 8/16-GPU data parallelism exhibit higher slowdowns, these are likely due to transient system noise rather than network congestion, since intra-switch traffic does not traverse the oversubscribed core fabric. In contrast, intragroup and inter-group placements are sometimes severely impacted, particularly in less regular allocations; for 8-GPU data parallelism, we observe slowdowns reaching 8.6× and 23.6× for intra-group and inter-group cases, respectively. The effects of measurements noise for DP are further analyzed in Section V-C4. The benefit of regular and application-specific allocation is clearly demonstrated by DP+PP+TP (DPT) on LLaMA3-70B (LaM-70) in 224-GPU runs across different inter-group scenarios. Specifically, in a standard inter-group allocation, executions experience an average slowdown of approximately 15%. By contrast, using a 4-nodes-per-switch allocation in an inter-group configuration reduces the average slowdown to only 5%. A similar trend can be observed for DP+PP instances, where 2-nodes-per-switch placements strongly reduce, or even eliminate (violins not shown), the variability in slowdowns. In most cases, runs on the production SLURM queue exhibit greater variance than those executed within 3 reserved groups, as expected: allocated nodes may span multiple Dragonfly+ groups, and network traffic can

0.08% (0.4x)

Slowdown

30% 15% 0%

4k

2% (8.6x) 10k

3%

1%

0.40%

4k (19.8x) 84 8k (23.6x) 508 (0.4x)

14k 937/4k

186

716

20/36

3/24

17 446

294

726 10

16/36

3k/4k

Normal Queue Reservation

210 7 15

15

52

21/24 4/5

16

64

42

7/8

74

-15% D ViT-H 8 GPU

D ViT-H 16 GPU

D ViT-H 8 GPU

Intra L1

D ViT-H 16 GPU

Intra Group

DP Minv 16 GPU

D ViT-H 8 GPU

D ViT-H 16 GPU

DP Minv 16 GPU

DP Minv 32 GPU

Intra Group 2 Nodes/Switch

DP Minv 64 GPU

DPT LaM-70 224 GPU

DPE Mxt 512 GPU

DPE Mxt 1K GPU

DPT LaM-70 224 GPU

Inter Group

DPT LaM-70 512 GPU

Inter Group 4 Nodes/Switch

Fig. 5: Distribution of slowdown on individual application runs when co-scheduled with other concurrent jobs on JUPITER. Each x tick represents one configuration, defined by its strategy, model, and placement (see Table II and Section IV-A4). 0.04% (106.4x)

D ViT-H 8 GPU

D ViT-H 16 GPU

D ViT-H 8 GPU

Intra L1

2/7

80%

3 36k

11k/24k 40/139 4k

3k

1k

5/7 398

14k/24k 99/139

D ViT-H 16 GPU

D ViT-H 8 GPU

Intra Group

D ViT-H 16 GPU

DP Minv 16 GPU

Inter Group

4

DPE Mxt 512 GPU

3

DPE Mxt 1K GPU

Slowdown

Fig. 6: Distribution of slowdown on individual application runs when co-scheduled with other concurrent jobs on Leonardo. 2.4x 1.6x 75% 60% 45% 30% 8k/11k 15% 0% 3k/11k -45% -60% -75%

0.01% (4.3x)

0.05% (13.7x) 4k

Normal Queue Reservation

43k/82k

Slowdown

Slowdown

0.00%

(2.6x) 70% Normal Queue 65% Reservation 42k/107k 30% 24% 18% 15k/32k 12% 14k/29k 14k/29k 5k 2k 6% 2k 15k/29k 66k/107k 0% 17k/32k 15k/29k

0.08% (1.1x)

40%

626/4k

5k 501/835 859/3k

40/200

0%160/200 -40%

585/2k 682/2k

D ViT-H 16 GPU

1k/2k

2k/3k 334/835

Normal Queue Reservation

D ViT-H 32 GPU

Intra L1

D ViT-H 16 GPU

652/2k

D ViT-H 32 GPU

Intra Group

1k/4k

213/355 1k/2k

189/315

2k/4k 126/315

142/355

D ViT-H 16 GPU

D ViT-H 32 GPU

Inter Group

Fig. 8: Distribution of slowdown on individual application runs when co-scheduled with other concurrent jobs on LUMI.

170/300 18k/38k

4k/28k 23k

989

21k

24k/28k 22k/82k 16k/82k 130/300

78

648

19k/38k

1k

D ViT-H 8 GPU

D ViT-H 16 GPU

Intra L1

D ViT-H 8 GPU

D ViT-H 16 GPU

Intra Group

D ViT-H 8 GPU

D ViT-H 16 GPU

FSDP LaM-8 32 GPU

Inter Group

FSDP Minv 32 GPU

DPE Mxt 512 GPU

Fig. 7: Distribution of slowdown on individual application runs when co-scheduled with other concurrent jobs on Alps.

be affected by other users on the system. Interestingly, this pattern does not hold for DP+PP. In the reservation scenario, approximately 25% of runs exhibit higher slowdowns. This behavior may be attributed to the strategy’s “pipeline” peerto-peer communication pattern, which, under high interconnect utilization, can become vulnerable to delays affecting individual messages along the communication pipeline, resulting in a drop in the overall throughput. 3) Leonardo: The results for the Leonardo supercomputer are reported in Fig. 6. In contrast to the results observed on JUPITER, Leonardo generally exhibits a lower overall impact from network congestion. This difference is largely due to Leonardo’s reliance on A100 GPUs, which, being less performant than the H200 used in JUPITER, as discussed in Section III-D, results in a lower relevance of the communica-

tion performance. There are, however, two notable exceptions to this trend. First, data parallelism on 8 GPUs with intergroup allocation shows a noticeably higher variance, with some runs experiencing slowdowns as high as 106×. Second, DP+PP+EP (DPE) at a 1K-GPU scale in the “no-reservation” scenario shows a significant performance hit, with average slowdowns reaching 65%. These runs have been automatically allocated by SLURM on more than 10 Dragonfly+ groups. As more groups are spanned, the probability of being affected by congestion on inter-group links increases. This effect is further exacerbated by the fact that Leonardo’s interconnect provides only half the bandwidth of JUPITER’s. Moreover, as shown in Fig. 2, DP+PP+EP spends almost half of the runtime time in communication. However, we only observe slowdowns when allocations exceed 3 Dragonfly+ groups. 4) Alps: As becomes evident from Fig. 5, DP seems to have high variability in the throughput measurements which reflect on slowdowns directly. Therefore, we analyzed this aspect further. Pure Data Parallelism (DP) is an intrinsically simple strategy: each step performs 50 global non-blocking A LL R EDUCE operations (one per gradient tensor), followed by a global synchronization on all 50 reductions. Because the underlying model (ViT-H) is comparatively small, per-step compute time is short, leaving little room for communication overhead to be hidden behind computation and making DP especially exposed to network-side variability. To isolate this

4

Time (ms)

Alps (Clariden) - 4 nodes - 4 ranks/node 75

Time (ms)

Alps (Clariden) - 2 nodes - 4 ranks/node

50

2

25

0

1000

1500

0

2000

0

500

1000

1500

2000

500

1000

1500

2000

LUMI - 4 nodes - 8 ranks/node

Time (ms)

30

Time (ms)

20

500

LUMI - 2 nodes - 8 ranks/node

20

10

10

0

500

1000

1500

AllReduce Run

2000

0

AllReduce Run

Fig. 9: Runtimes of A LL R EDUCE collective on Alps and LUMI. Note that y axis is fitted to the data.

effect, we ran an A LL R EDUCE microbenchmark on Alps and LUMI, using the same message sizes as DLNetBench DP, over each system’s native collective backend (NCCL on Alps, RCCL on LUMI). This microbenchmark issues 2000 A LL R E DUCE collectives, one at a time, and reports the runtime of the slowest rank. Results are shown in Fig. 9. Intra-node performance is stable on both systems, but beyond one node it becomes unstable, with fluctuations amplifying at larger node counts and both systems showing extreme transient outliers. Notably, LUMI’s fluctuations decay and stabilize after around 500 runs, whereas Alps continues to show persistent, nondecaying variability. Because DP’s per-step runtimes are so small, even minor variance in collective performance translates into large swings in measured throughput. We conjecture this stems partly from rank desynchronization, compounded by our throughput methodology, which uses the last-arriving rank at each step and so is particularly sensitive to any single rank’s transient slowdown. These effects are already visible on DGX A100, JUPITER and Leonardo but worsen on Alps and LUMI. Fig. 7 shows that DP slowdowns variance is consistently high among all scales and placement classes on the Alps system. Accordingly to microbenchmark results, such high variance is due to the variability in runtime of A LL R EDUCE communications. Such instability, does not allow to measure a proper baseline, thus, reported absolute slowdowns values are difficult to interpret. Although Fig. 2 indicates that communication accounts for only 3–7% of the total runtime in FSDP, this percentage appears sufficient to trigger, in a few isolated cases (8 runs), slowdowns of up to 60% due to orders of magnitude increase in communication time. Notably, this sensitivity is only observed for Minerva-7B. DP+PP+EP shows a few ≥10% slowdowns, though exclusively when scaling to 512 GPUs. Note that these slowdowns are only observed in runs on the production queue. Despite DP+PP and DP+PP+TP spending 40% to 60% of their total runtime in communication (Fig. 2), they do not seem to be significantly affected by network congestion showing that Alps interconnect can effectively support their communication patterns even in presence of network traffic.

5) LUMI: Similarly to Alps, hybrid strategies spend 30% to 40% of their total runtime in communication. Still, Fig. 8 shows that on LUMI all hybrid strategies exhibit no significant slowdown and are therefore omitted from the figure entirely. This is probably because LUMI features 8 GPUs per node, so the intra-replica communications stays on the scaleup network, and only the extra-replica A LL R EDUCE traffic crosses onto the scale-out network. DP, in contrast, shows high variance; this is because RCCL requires a substantial number of iterations to stabilize (see Fig. 9), and in our experiments it had not yet reached a stable regime, inflating the observed slowdown variance. System noise and contention affect training performance unevenly, depending on communication pattern and scale, rather than uniformly across all workloads. Congestion is chiefly a scale-out network phenomenon: intra-switch/intranode traffic is largely unaffected, while cross-group traffic can produce severe slowdowns. Sensitivity is highest for strategies with many small/medium messages and low per-step compute to hide latency (DP and DP+PP+EP) or with chained peer-topeer dependencies (DP+PP), yet the final impact is strongly system-dependent. System noise, distinct from network contention, also plays a role: unstable collective-library behavior (NCCL/RCCL not yet converged) drives high-variance slowdowns even for intra-switch DP traffic. Placement choice is an effective but bounded mitigation. Regular, topology-aware allocation (e.g., fixed nodes-per-switch) can cut average slowdown several-fold or eliminate variability entirely. Reserved allocations generally reduce variance relative to productionqueue scheduling, but not universally: pipeline-parallel jobs can show the opposite trend due to their sensitivity to individual message delays. Overall, placement can strongly amplify or suppress contention-driven slowdowns, but its mitigating power depends on the communication pattern. Noise-driven variance from unstable interconnect software remains largely unaffected by placement altogether. Observation 4: To answer R3, contention and noise affect performance unevenly across communication patterns and scales; topology-aware placement can mitigate contention up to a certain limit. VI. R ELATED W ORK A. Impact of Network Congestion Network congestion caused by concurrent jobs sharing the interconnect fabric has emerged as a critical source of performance variability and a scalability limit. Hoefler et al. [42], [43] provided an early characterization of how both operating system and network noise degrade parallel application performance, demonstrating that even minor perturbations can create substantial scaling bottlenecks. Similarly, Bhatele et al. [44] demonstrated variability of up to 2× for communicationintensive applications, directly correlating these fluctuations with contention from co-running workloads. This landscape is further complicated by the shift toward cloud-based HPC;

research indicates that the impact of network noise in cloud environments is even more pronounced than that observed in traditional, on-premise HPC systems [45]. The interaction between network topology and job management also plays a significant role in congestion. Jain et al. [46] explored how different routing strategies and job placement policies affect throughput on Dragonfly networks, highlighting that randomized placement can help mitigate hotspots. Other studies [47] observed that adaptive routing, while intended to alleviate congestion by using non-minimal paths, can paradoxically generate additional traffic and increase interference for other applications. To address these challenges, specialized tools have been designed to systematically detect and quantify the impact of network congestion: Jha et al. [48] developed Monet to characterize the spatial extent of congestion hotspots, while the GPCNeT [26] aggressor–victim methodology has become widely used for measuring latency and bandwidth degradation by inducing synthetic contention in shared environments [27], [49]. B. Network Contention in AI Clusters In AI-centric datacenters, the impact of congestion is intensified by the bursty, bulk-synchronous traffic patterns inherent to distributed training [50], [51]. Inter-job contention is particularly damaging in GPU-accelerated clusters. The Crux system [52] demonstrated that when a 64-GPU LLM training job and a 16-GPU language model share network resources, both GPU utilization and training throughput degrade significantly. The severity of these issues is further explained by the highly skewed nature of AI workloads. Jeon et al. [25] first identified a power-law distribution in jobs runtimes on Microsoft’s Philly cluster, where a small fraction of multi-GPU jobs accounts for the vast majority of GPU time. This observation was reinforced at a larger scale by Hu et al. [11] in the Acme datacenter, where LLM pretraining jobs represent only 3.2% of the total job count but consume 94.0% of GPU resources, while evaluation jobs constitute 92.9% of the count but use merely 0.8% of resources. By integrating these insights on power-law workload distributions with established aggressor– victim methodologies, we have developed a new congestion evaluation strategy (Sec. IV). This approach reflects realistic multi-tenant allocation patterns to faithfully reproduce the conditions observed on production AI supercomputers. VII. C ONCLUSION AND F UTURE W ORK This work provides a systematic, large-scale characterization of distributed training performance across six systems spanning node-, rack-, and supercomputer-scale interconnects. Our results show that communication overhead is minimal within a node or rack but becomes highly relevant once workloads cross the node boundary, accounting for 30% to over 70% of runtime even at small supercomputer scales. Strategy choice strongly mediates this transition: FSDP consistently overlaps communication with computation and sustains high efficiency, pure DP degrades sharply with scale due to its reliance on global IA LL R EDUCE, and hybrid strategies fall

in between, with robustness depending on how their communication groups map onto system topology. Regarding placement, network placement has little to no impact on aggregate throughput scaling: absolute throughput is governed primarily by compute capability and interconnect bandwidth. This decoupling breaks down under contention. Our multi-tenancy analysis shows that congestion and system noise affect performance unevenly. Congestion is chiefly a scale-out phenomenon: intra-switch and intra-node traffic remains largely unaffected across all systems, while crossgroup traffic can produce severe, highly variable slowdowns. Independent of contention, unstable collective-library convergence (NCCL/RCCL) is itself a source of high-variance slowdowns that placement cannot mitigate. Where congestion is the driver, topology-aware placement is an effective but bounded mitigation: regular, application-specific allocation can cut average slowdowns several-fold, and reserved allocations generally reduce variance relative to production-queue scheduling, though pipeline-parallel strategies can show the opposite trend. At node scale, DGX A100 is not significantly affected by co-scheduling, with only a handful of isolated outliers that are more plausibly attributable to transient system noise than to genuine network congestion, while NVL72’s NVSwitch fabric essentially eliminates congestion as a performance factor despite higher compute and communication demands. Together, these findings indicate that scalable, robust largescale training performance requires jointly reasoning about parallelization strategy, communication pattern, interconnect characteristics, and topology-aware placement, since no single factor predicts performance under multi-tenancy and in isolation. Several aspects remain open. Our evaluation uses a limited set of models with fixed parameters, constraining analysis of compute-to-communication balance and achievable overlap. Our Alps/LUMI microbenchmarks suggest collective-library convergence behavior warrants further study to disentangle noise from genuine contention. Future work will broaden models and scales to separate model-specific from strategylevel trends, vary DLNetBench parameters to probe computecommunication balance, extend comparisons to additional collective libraries, and run fully controlled allocations at larger scale to better characterize worst-case contention across many inter-group links. ACKNOWLEDGMENT This work has received funding from the European HighPerformance Computing Joint Undertaking (EuroHPC JU) under grant agreement No 101175702. The project received access to the JUPITER supercomputer, which is funded by the EuroHPC JU and the German BMFTR and MKW-NRW, through the JUPITER Research and Early Access Program (JUREAP). We acknowledge the CINECA award under the ISCRA initiative, CSCS, and CSC for the availability of highperformance computing resources and support.

R EFERENCES [1] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” 2020. [Online]. Available: https://arxiv.org/abs/2005.14165 [2] A. G. et.al., “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783 [3] Q. Anthony, B. Michalowicz, J. Hatef et al., “Demystifying the communication characteristics for distributed transformer models,” in IEEE International Symposium on High-Performance Interconnects (HOTI). IEEE, 2024, pp. 57–65. [4] W. Wang, M. Khazraee, Z. Zhong, M. Ghobadi, Z. Jia, D. Mudigere, Y. Zhang, and A. Kewitsch, “{TopoOpt}: Co-optimizing network topology and parallelization strategy for distributed training jobs,” in 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023, pp. 739–767. [5] Z. Zhang, C. Chang, H. Lin et al., “Is network the bottleneck of distributed training?” in Workshop on Network Meets AI & ML (NetAI). ACM, 2020, pp. 8–13. [6] NVIDIA, “Nccl,” https://developer.nvidia.com/nccl, 2023. [7] AMD, “RCCL Developer Guide,” https://rocm.docs.amd.com/projects/ rccl/en/latest/, 2025, accessed: 2025-02-20. [8] Intel, oneAPI Collective Communications Library (oneCCL), UXL Foundation, 2025, oneCCL Documentation 2021.17.0, last updated 21 October 2025, accessed 4 April 2026. [Online]. Available: https://uxlfoundation.github.io/oneCCL/index.html [9] S. Rajasekaran, M. Ghobadi, G. Kumar, and A. Akella, “Congestion control in machine learning clusters,” in ACM Workshop on Hot Topics in Networks (HotNets). ACM, 2022, pp. 235–242. [10] A. A. E. Elsayed, A. A. Al-Obaidi, and H. E. Farag, “Characterization of high-resolution ai data center training workloads on single and multiple gpu nodes,” Scientific Data, 2026. [11] Q. Hu, Z. Ye, Z. Wang, G. Wang, M. Zhang, Q. Chen, P. Sun, D. Lin, X. Wang, Y. Luo, Y. Wen, and T. Zhang, “Characterization of large language model development in the datacenter,” in 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). Santa Clara, CA: USENIX Association, Apr. 2024, pp. 709–729. [Online]. Available: https: //www.usenix.org/conference/nsdi24/presentation/hu [12] S. Huang, L. Peng, M. Wang, Y. Liu, Z. Liu, X. Wang, and Y. Cui, “Datacenter network deserves better traffic models,” in Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, ser. HotNets ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 124–130. [Online]. Available: https://doi.org/10.1145/3626111.3628209 [13] O. Antepara, L. Oliker, and S. Williams, “Roofline analysis of tightly-coupled CPU-GPU superchips: A study on MI300A and GH200,” in Proceedings of the SC ’25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC Workshops 2025, St Louis, MO, USA, November 16-21, 2025. ACM, 2025, pp. 1205–1216. [Online]. Available: https://doi.org/10.1145/3731599.3767497 [14] S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala, “Pytorch distributed: Experiences on accelerating data parallel training,” Proc. VLDB Endow., vol. 13, no. 12, pp. 3005–3018, 2020. [Online]. Available: http://www.vldb.org/pvldb/vol13/p3005-li.pdf [15] Y. Zhao, A. Gu, R. Varma, L. Luo, C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y. Hao, A. Mathews, and S. Li, “Pytorch FSDP: experiences on scaling fully sharded data parallel,” Proc. VLDB Endow., vol. 16, no. 12, pp. 3848–3860, 2023. [Online]. Available: https://www.vldb.org/pvldb/vol16/p3848-huang.pdf [16] S. Fan, Y. Rong, C. Meng, Z. Cao, S. Wang, Z. Zheng, C. Wu, G. Long, J. Yang, L. Xia, L. Diao, X. Liu, and W. Lin, “DAPPLE: a pipelined data parallel approach for training large models,” in PPoPP ’21: 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Virtual Event, Republic of Korea, February 27March 3, 2021, J. Lee and E. Petrank, Eds. ACM, 2021, pp. 431–445. [Online]. Available: https://doi.org/10.1145/3437801.3441593

[17] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on GPU clusters using megatron-lm,” in International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, November 14-19, 2021, B. R. de Supinski, M. W. Hall, and T. Gamblin, Eds. ACM, 2021, p. 58. [Online]. Available: https://doi.org/10.1145/3458817.3476209 [18] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538, 2017. [19] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=YicbFdNTTy [20] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” CoRR, vol. abs/2302.13971, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2302.13971 [21] R. Orlando, L. Moroni, P. H. Cabot, S. Conia, E. Barba, S. Orlandini, G. Fiameni, and R. Navigli, “Minerva llms: The first family of large language models trained from scratch on italian data,” in Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024), Pisa, Italy, December 4-6, 2024, ser. CEUR Workshop Proceedings, F. Dell’Orletta, A. Lenci, S. Montemagni, and R. Sprugnoli, Eds. CEUR-WS.org, 2024. [Online]. Available: https://ceur-ws.org/Vol-3878/76\ main\ long.pdf [22] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de Las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mixtral of experts,” CoRR, vol. abs/2401.04088, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.04088 [23] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne et al., “Jax: composable transformations of python+ numpy programs,” 2018. [24] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He, “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,” in Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, 2020, pp. 3505– 3506. [25] M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, and F. Yang, “Analysis of large-scale multi-tenant GPU clusters for DNN training workloads,” in 2019 USENIX Annual Technical Conference, ser. USENIX ATC ’19. USENIX Association, 2019. [26] S. Chunduri, T. Groves, P. Mendygral, B. Austin, J. Balma, K. Kandalla, K. Kumaran, G. Lockwood, S. Parker, S. Warren, N. Wichmann, and N. Wright, “GPCNeT: Designing a benchmark suite for inducing and measuring contention in HPC networks,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’19. New York, NY, USA: ACM, 2019. [27] D. De Sensi, S. Di Girolamo, K. H. McMahon, D. Roweth, and T. Hoefler, “An in-depth analysis of the slingshot interconnect,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’20. IEEE Press, 2020. [28] TOP500, “TOP500 Supercomputer Sites,” https://top500.org/, 2026, accessed: 2026-04-08. [29] J. Choquette, W. Gandhi, O. Giroux, N. Stam, and R. Krashinsky, “ NVIDIA A100 Tensor Core GPU: Performance and Innovation ,” IEEE Micro, vol. 41, no. 02, pp. 29–35, Mar. 2021. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/MM.2021.3061394 [30] NVIDIA, NVIDIA A100 Tensor Core GPU Architecture, 2020. [Online]. Available: https://images.nvidia.com/aem-dam/en-zz/ Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf [31] ——, NVIDIA DGX SuperPOD: Next Generation Scalable Infrastructure for AI Factories Reference Architecture Featuring NVIDIA DGX GB300 Systems, NVIDIA, 2025, accessed: 2026-04-04. [Online]. Available: https://docs.nvidia.com/pdf/dgx-spod-gb300-ra.pdf

[32] CSCS, “Alps Supercomputer,” [Accessed 01-04-2026]. [Online]. Available: https://www.cscs.ch/computers/alps [33] NVIDIA, NVIDIA Grace Hopper Superchip Architecture Whitepaper, 2024. [Online]. Available: https://resources.nvidia.com/en-us-grace-cpu/ nvidia-grace-hopper [34] G. Schieffer, J. Wahlgren, J. Ren, J. Faj, and I. Peng, “Harnessing integrated cpu-gpu system memory for hpc: a first look into grace hopper,” in Proceedings of the 53rd International Conference on Parallel Processing, ser. ICPP ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 199–209. [Online]. Available: https://doi.org/10.1145/3673038.3673110 [35] J. Kim, W. J. Dally, S. Scott, and D. Abts, “Technology-driven, highly-scalable dragonfly topology,” in Proceedings of the 35th Annual International Symposium on Computer Architecture, ser. ISCA ’08. USA: IEEE Computer Society, 2008, p. 77–88. [Online]. Available: https://doi.org/10.1109/ISCA.2008.19 [36] K. S. Khorassani, C.-C. Chen, B. Ramesh, A. Shafi, H. Subramoni, and D. K. Panda, “High performance mpi over the slingshot interconnect,” Journal of Computer Science and Technology, vol. 38, no. 1, pp. 128– 145, 2023. [37] M. Turisini, G. Amati, and M. Cestari, “LEONARDO: A Pan-European Pre-Exascale Supercomputer for HPC and AI applications,” Journal of Large Scale Reasearch Facilities, vol. 8, no. A186, 2024. [38] EuroHPC Joint Undertaking, “LUMI supercomputer,” [Accessed 19-03-2024]. [Online]. Available: https://lumi-supercomputer.eu/ [39] Advanced Micro Devices, Inc., “AMD CDNA 2 Architecture White Paper,” AMD, Tech. Rep., 2021. [Online]. Available: https://www.amd.com/content/dam/amd/en/documents/ instinct-business-docs/white-papers/amd-cdna2-white-paper.pdf [40] A. Herten, S. Achilles, D. Alvarez, J. Badwaik, E. Behle, M. Bode, T. Breuer, D. Caviedes-Voullième, M. Cherti, A. Dabah, S. E. Sayed, W. Frings, A. Gonzalez-Nicolas, E. B. Gregory, K. H. Mood, T. Hater, J. Jitsev, C. M. John, J. H. Meinke, C. I. Meyer, P. Mezentsev, J.O. Mirus, S. Nassyr, C. Penke, M. Römmer, U. Sinha, B. v. S. Vieth, O. Stein, E. Suarez, D. Willsch, and I. Zhukov, “Applicationdriven exascale: The jupiter benchmark suite,” in SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, 2024, pp. 1–45. [41] T. Hoefler and R. Belli, “Scientific benchmarking of parallel computing systems: twelve ways to tell the masses when reporting performance results,” in Proceedings of the international conference for high performance computing, networking, storage and analysis, 2015, pp. 1–12. [42] T. Hoefler, T. Schneider, and A. Lumsdaine, “Characterizing the influence of system noise on large-scale applications by simulation,” in Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’10. USA: IEEE Computer Society, 2010, p. 1–11. [Online]. Available: https://doi.org/10.1109/SC.2010.12 [43] ——, “The impact of network noise at large-scale communication performance,” in 2009 IEEE International Symposium on Parallel & Distributed Processing, 2009, pp. 1–8. [44] A. Bhatele, K. Mohror, S. H. Langer, and K. E. Isaacs, “There goes the neighborhood: Performance degradation due to nearby jobs,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’13. New York, NY, USA: ACM, 2013. [45] D. De Sensi, T. De Matteis, K. Taranov, S. Di Girolamo, T. Rahn, and T. Hoefler, “Noise in the clouds: Influence of network performance variability on application scalability,” Proc. ACM Meas. Anal. Comput. Syst., vol. 6, no. 3, Dec. 2022. [Online]. Available: https://doi.org/10.1145/3570609 [46] N. Jain, A. Bhatele, X. Ni, N. J. Wright, and L. V. Kale, “Maximizing throughput on a dragonfly network,” in SC ’14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2014, pp. 336–347. [47] D. De Sensi, S. Di Girolamo, and T. Hoefler, “Mitigating network noise on dragonfly networks through application-aware routing,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’19. New York, NY, USA: ACM, 2019. [48] S. Jha, A. Patke, J. Brandt, A. Gentile, B. Lim, M. Showerman, G. Bauer, L. Kaplan, Z. Kalbarczyk, W. Kramer, and R. Iyer, “Measuring congestion in high-performance datacenter interconnects,” in 17th USENIX

Symposium on Networked Systems Design and Implementation, ser. NSDI ’20. USENIX Association, 2020. [49] D. De Sensi, L. Pichetti, F. Vella, T. De Matteis, Z. Ren, L. Fusco, M. Turisini, D. Cesarini, K. Lust, A. Trivedi, D. Roweth, F. Spiga, S. Di Girolamo, and T. Hoefler, “Exploring gpu-to-gpu communication: Insights into supercomputer interconnects,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, ser. SC ’24. IEEE Press, 2024. [50] T. Bonato, A. Kabbani, D. De Sensi, R. Pan, Y. Le, C. Raiciu, M. Handley, T. Schneider, N. Blach, A. Ghalayini et al., “Fastflow: Flexible adaptive congestion control for high-performance datacenters,” arXiv preprint arXiv:2404.01630, 2024. [51] Y. Zhang, Q. Meng, C. Hu, and F. Ren, “Revisiting congestion control for lossless ethernet,” in 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). Santa Clara, CA: USENIX Association, Apr. 2024, pp. 131–148. [Online]. Available: https://www.usenix.org/conference/nsdi24/presentation/zhang-yiran [52] J. Cao, Y. Li, and D. Zhuo, “Crux: GPU-efficient communication scheduling for deep learning training,” in Proceedings of the ACM SIGCOMM 2024 Conference, ser. SIGCOMM ’24. New York, NY, USA: ACM, 2024.

A PPENDIX DLNetBench Pseudocode and Constants TABLE IV: Experimental Setup and Parameters Strategy DP FSDP DP+PP DP+PP+TP DP+PP+EP

Batch Size 16 16 16 16 16

Microbatches – – 16 16 16

PP Degree – – 8 8 8

TP Degree – – – 4 –

EP Degree – – – – 8

Specific Parameters num_buckets=50 num_units=16, sharding_factor=8 – – –

Algorithm 1 DP: Pure Data Parallelism 1: function RUN DATA PARALLEL 2: C OMPUTE(forward time) 3: 4: 5: 6:

▷ Forward pass for each bucket do ▷ Backward pass C OMPUTE(backward time per bucket) IA LL R EDUCE W (bucket) WAITA LL W

Algorithm 2 FSDP: Fully Sharded Data Parallelism 1: function RUN FSDP 2: A LL G ATHER IR (unit 0) 3: for u = 0 to num units − 2 do 4: IA LL G ATHER IR (unit u + 1) 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17:

▷ Forward pass

C OMPUTE(forward time) WAIT IR (unit u + 1) for u = num units − 1 to 0 do ▷ Backward pass if u > 0 then IA LL G ATHER IR (unit u − 1) C OMPUTE(backward time) R EDUCE S CATTER IR (unit u) if num replicas > 1 then IA LL R EDUCE ER (unit u) if u > 0 then WAIT IR (unit u − 1) if num replicas > 1 then WAITA LL ER

Algorithm 3 DP+PP: Data and Pipeline Parallelism (GPipe) 1: function RUN DATA PIPELINE PARALLEL 2: for each microbatch do ▷ Forward pass 3: if first stage then 4: C OMPUTE(microbatch forward time) 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24:

S END IR (next stage) else if last stage then R ECV IR (previous stage) C OMPUTE(microbatch forward time) else R ECV IR (previous stage) C OMPUTE(microbatch forward time) S END IR (next stage) for each microbatch do ▷ Backward pass if first stage then R ECV IR (next stage) C OMPUTE(microbatch backward time) else if last stage then C OMPUTE(microbatch backward time) S END IR (previous stage) else R ECV IR (next stage) C OMPUTE(microbatch backward time) S END IR (previous stage) A LL R EDUCE ER (gradients) ▷ Data parallel gradient synchronization

Algorithm 4 DP+PP+TP: Data, Pipeline and Tensor Parallelism 1: function RUN DATA PIPELINE TENSOR PARALLEL 2: for each microbatch do ▷ Forward pass 3: if first stage then 4: for each layer do 5: C OMPUTE(attention forward time) 6: A LL R EDUCE IR (tensor parallel) 7: C OMPUTE(MLP forward time) 8: A LL R EDUCE IR (tensor parallel) 9: S END IR (next stage) 10: else if last stage then 11: R ECV IR (previous stage) 12: for each layer do 13: C OMPUTE(attention forward time) 14: A LL R EDUCE IR (tensor parallel) 15: C OMPUTE(MLP forward time) 16: A LL R EDUCE IR (tensor parallel) 17: else 18: R ECV IR (previous stage) 19: for each layer do 20: C OMPUTE(attention forward time) 21: A LL R EDUCE IR (tensor parallel) 22: C OMPUTE(MLP forward time) 23: A LL R EDUCE IR (tensor parallel) 24: S END IR (next stage) 25: for each microbatch do ▷ Backward pass 26: if first stage then 27: R ECV IR (next stage) 28: for each layer do 29: C OMPUTE(attention backward time) 30: A LL R EDUCE IR (tensor parallel) 31: C OMPUTE(MLP backward time) 32: A LL R EDUCE IR (tensor parallel) 33: else if last stage then 34: for each layer do 35: C OMPUTE(attention backward time) 36: A LL R EDUCE IR (tensor parallel) 37: C OMPUTE(MLP backward time) 38: A LL R EDUCE IR (tensor parallel) 39: S END IR (previous stage) 40: else 41: R ECV IR (next stage) 42: for each layer do 43: C OMPUTE(attention backward time) 44: A LL R EDUCE IR (tensor parallel) 45: C OMPUTE(MLP backward time) 46: A LL R EDUCE IR (tensor parallel) 47: S END IR (previous stage) 48: ▷ Data parallel gradient synchronization 49: A LL R EDUCE ER (gradients)

Algorithm 5 DP+PP+EP: Data, Pipeline and Expert Parallelism 1: function RUN DATA PIPELINE EXPERT PARALLEL 2: for each microbatch do ▷ Forward pass 3: if first stage then 4: for each layer do 5: C OMPUTE(attention forward time) 6: A LLT OA LL IR (expert parallel) 7: C OMPUTE(MLP forward time) 8: A LLT OA LL IR (expert parallel) 9: S END IR (next stage) 10: else if last stage then 11: R ECV IR (previous stage) 12: for each layer do 13: C OMPUTE(attention forward time) 14: A LLT OA LL IR (expert parallel) 15: C OMPUTE(MLP forward time) 16: A LLT OA LL IR (expert parallel) 17: else 18: R ECV IR (previous stage) 19: for each layer do 20: C OMPUTE(attention forward time) 21: A LLT OA LL IR (expert parallel) 22: C OMPUTE(MLP forward time) 23: A LLT OA LL IR (expert parallel) 24: S END IR (next stage) 25: for each microbatch do ▷ Backward pass 26: if first stage then 27: R ECV IR (next stage) 28: for each layer do 29: C OMPUTE(attention backward time) 30: A LLT OA LL IR (expert parallel) 31: C OMPUTE(MLP backward time) 32: A LLT OA LL IR (expert parallel) 33: else if last stage then 34: for each layer do 35: C OMPUTE(attention backward time) 36: A LLT OA LL IR (expert parallel) 37: C OMPUTE(MLP backward time) 38: A LLT OA LL IR (expert parallel) 39: S END IR (previous stage) 40: else 41: R ECV IR (next stage) 42: for each layer do 43: C OMPUTE(attention backward time) 44: A LLT OA LL IR (expert parallel) 45: C OMPUTE(MLP backward time) 46: A LLT OA LL IR (expert parallel) 47: S END IR (previous stage) 48: ▷ Data parallel gradient synchronization 49: A LL R EDUCE IR (non-expert gradients, expert parallel) 50: A LL R EDUCE ER (gradients, data parallel)

Related documents

• Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development#451337
• Official Review · ICLR 2025#787132
• Explore barriers and risks that hinder AI integration into global talent management: A Systematic Review.#685910
• One Size Does Not Fit All: RevisitingWorld Models and Neurosymbolic AI#680658
• The New Associationism: Lessons from Deep Learning#691728