arXiv CSopen access

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

networking, internet, protocols, distributed systems

arXiv:2604.16088v1 [cs.NI] 17 Apr 2026

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks MIGUEL SÁNCHEZ DE LA ROSA, Universidad de Castilla-La Mancha (UCLM), Spain GABRIEL GOMEZ-LOPEZ, Universidad de Castilla-La Mancha (UCLM), Spain ALEJANDRO BAVIERA, Universitat Politècnica de València (UPV), Spain JOSE DURO, Universitat Politècnica de València (UPV), Spain FRANCISCO J. ANDÚJAR, Universidad de Valladolid (UVA), Spain JESUS ESCUDERO-SAHUQUILLO, Universidad de Castilla-La Mancha (UCLM), Spain PEDRO J. GARCIA, Universidad de Castilla-La Mancha (UCLM), Spain FRANCISCO J. ALFARO, Universidad de Castilla-La Mancha (UCLM), Spain MARIA E. GOMEZ, Universitat Politècnica de València (UPV), Spain JULIO SAHUQUILLO, Universitat Politècnica de València (UPV), Spain JOSÉ L. SÁNCHEZ, Universidad de Castilla-La Mancha (UCLM), Spain FRANCISCO J. QUILES, Universidad de Castilla-La Mancha (UCLM), Spain The interconnection network is a key component of Supercomputers and Data centers, and its design must cope with the increasing communication demands of current applications and services; otherwise, it may become a system bottleneck. The most challenging network design issues are the topology, routing algorithm, flow control, and power efficiency. However, even the most efficient interconnection networks may suffer severe performance degradation due to congestion, especially under specific network traffic patterns generated by communication operations in high-performance computing (HPC), deep learning training, or online dataintensive services. In this context, characterizing and modeling these communication operations and the network traffic patterns they generate is a fundamental challenge for studying their impact on network performance. This paper presents a methodology, based primarily on the VEF Traces framework, to characterize, model, and simulate the communication patterns of representative computing- and data-intensive applications. More precisely, we have extended the VEF traces framework with tools that enable us to characterize network congestion, either directly from VEF traces or via simulations. We have analyzed a set of VEF traces obtained from runs of NEST, GROMACS, LAMMPS, and PATMOS on several Supercomputers. In these studies, we identify potential congestion scenarios that arise in realistic network configurations when certain collective operations are performed. Authors’ Contact Information: Miguel Sánchez de La Rosa, Universidad de Castilla-La Mancha (UCLM), Albacete, Spain, [email protected]; Gabriel Gomez-Lopez, Universidad de Castilla-La Mancha (UCLM), Albacete, Spain, [email protected]; Alejandro Baviera, Universitat Politècnica de València (UPV), Valencia, Spain, [email protected]; Jose Duro, Universitat Politècnica de València (UPV), Valencia, Spain, jodugo1@disca. upv.es; Francisco J. Andújar, Universidad de Valladolid (UVA), Valladolid, Spain, [email protected]; Jesus Escudero-Sahuquillo, Universidad de Castilla-La Mancha (UCLM), Albacete, Spain, [email protected]; Pedro J. Garcia, Universidad de Castilla-La Mancha (UCLM), Albacete, Spain, [email protected]; Francisco J. Alfaro, Universidad de Castilla-La Mancha (UCLM), Albacete, Spain, [email protected]; Maria E. Gomez, Universitat Politècnica de València (UPV), Valencia, Spain, [email protected]; Julio Sahuquillo, Universitat Politècnica de València (UPV), Valencia, Spain, [email protected]; José L. Sánchez, Universidad de Castilla-La Mancha (UCLM), Albacete, Spain, [email protected]; Francisco J. Quiles, Universidad de Castilla-La Mancha (UCLM), Albacete, Spain, [email protected]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM Manuscript submitted to ACM

Sanchez de la Rosa et al.

Additional Key Words and Phrases: High-performance interconnection networks, Communication operations, Network congestion, Traffic modeling, VEF traces framework, OMNeT++. ACM Reference Format: Miguel Sánchez de La Rosa, Gabriel Gomez-Lopez, Alejandro Baviera, Jose Duro, Francisco J. Andújar, Jesus Escudero-Sahuquillo, Pedro J. Garcia, Francisco J. Alfaro, Maria E. Gomez, Julio Sahuquillo, José L. Sánchez, and Francisco J. Quiles. 2025. Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks. ACM Trans. Arch. Code Optim. 0, 0, Article 0 ( 2025), 24 pages. https://doi.org/XXXXXXX.XXXXXXX

Introduction

Simulation allows network designers to model high-performance interconnection networks and evaluate their performance under workloads generated by the system’s end nodes when running specific benchmarks and applications. It is therefore essential to accurately characterize the communication patterns of these workloads in real systems, so that realistic network traffic can be fed into network simulators. Although this approach is not new and some simulation frameworks were using it in the past (e.g., DUMPI traces in SST [1] or VEF traces[2]), it has gained popularity in recent studies (e.g., Astra-sim [3] or ATHLAS [4]) because it enables network designers and researchers to identify bottlenecks and other undesired conditions before any real deployment. To characterize specific communication patterns, we have developed a methodology that extends and leverages the open-source framework VEF Traces [2, 5], consisting on i) profiling MPI-based applications communication and recording this information in network traffic traces, ii) modeling realistic network workloads considering the information of network communication patterns made publicly available by HPC systems and Data-center users, iii) reproducing these communication patterns in a specific simulation tool, and iv) performing different types of analyses (called static or dynamic) using the characterized patterns. Figure 1 shows a general overview of the workflow of our proposed methodology, including the most prominent tools of the VEF Traces framework (in yellow). In detail, the VEF Traces framework generates traffic traces that capture the communication operations of MPIbased parallel applications. To do that, the VEF Prospector suite provides a set of libraries and tools for profiling these operations. This process requires that the target application and VEF Prospector run concurrently on a real HPC cluster. VEF Prospector leverages the Profiling MPI (PMPI) library to capture the MPI calls (either point-to-point or collective) per MPI rank. The MPI call information per MPI rank is stored locally on each end node in the cluster and later combined into a single VEF trace. Although trace profiling is a reliable way of modeling communication operations, its lack of scalability is its main weakness. Note that a real HPC cluster is required to run the application and collect the traces. Moreover, traces cannot be generated for applications with MPI task counts exceeding the number of cluster cores. On the other hand, in data centers, there are applications other than MPI-based ones whose workloads are of interest for congestion characterization. However, in this case, we cannot use VEF Prospector or its profiling functionality to collect traces. To overcome this issue, assuming profiling tools similar to PMPI are unavailable, another option is to use application-level information to characterize communication patterns, such as the flow size distribution, the sourcedestination distribution, and the time consumed by these patterns in the network. Sometimes, this information is publicly available from several sources [6, 7], so it is possible to configure a set of parameters (e.g., packet destination distribution, traffic flow size distribution, number of traffic flows, generation period, traffic classes, etc.) to build a trustworthy communication model, which could eventually be scaled out to evaluate its impact on the largest interconnection networks. The VEF traces framework also supports this approach through the “extended” VEF traces [5], which permits scaling out the behavior of a set of communication patterns to thousands of nodes. This approach allows us to model Manuscript submitted to ACM

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks

the communication patterns of widely used applications in Data centers. Examples of these patterns include those derived from online data-intensive services (e.g., those used in Facebook Data centers) or deep learning training models (e.g., those run on top of the Horovod distributed framework). To replay the behavior recorded in VEF traces in a simulated network environment, the VEF framework provides another open-source software library, called VEF TraceLib, which generates communication messages for a VEF trace and manages dependencies among them. TraceLib is simulator-agnostic; it does not model network-specific behavior, which is left to the simulators. TraceLib provides a dedicated API with primitives that any simulator can use to inject messages into the network and be notified when they arrive at their destination. There is an implementation reference of TraceLib in the VEF-Traces RED-SEA blog post1 and the tracetor tool2 . Network simulation tools that use TraceLib and VEF traces should provide a set of performance metrics to help network designers analyze communication patterns and their impact on network performance, including degradation caused by bottlenecks, reliability issues, or power consumption. Hereafter, we refer to the analysis of these specific metrics using simulation results as dynamic analysis (see Figure 1), since the simulated network behavior and the resulting performance metrics vary over time and depend on the network configuration parameters and the nature of applications’ communication operations. This paper describes a methodology (see Figure 1) to leverage the VEF Traces framework and network simulation tools to characterize the communication patterns of real applications. After generating the VEF traces, we analyze the communication patterns they capture in two ways. First, we have extended the VEF Prospector tool with a toolchain that helps perform a static analysis using communication metrics, such as the operation type (P2P or collective), the MPI calls count, number of generated bytes, source and destination end-nodes engaged in the communication, etc. This toolchain generates a set of plots that can be combined into a single-file report. For the dynamic analysis, we 1 The VEF-Traces RED-SEA blog post: https://redsea-project.eu/the-vef-traces-framework/ 2 Source code of the tracetor tool https://gitraap.i3a.info/fandujar/VEF-TraceLIB/-/blob/master/test/main.c

Fig. 1. Diagram of our traffic-modeling methodology. Yellow text squares indicate the applications and tools of the VEF Traces framework. Manuscript submitted to ACM

Sanchez de la Rosa et al.

have used the SAURON [8] simulator, based on the well-known OMNeT++ framework [9]. SAURON has been widely used and tested in the last decade, modeling specific, state-of-the-art interconnect technologies (e.g., InfiniBand [10] or BXI [11, 12]). We have extended the SAURON simulator to provide metrics and plots that help analyze the impact of congestion on network performance. The main contributions of this paper are the following: • We have studied the communication operations generated by several parallel applications widely used in Supercomputers and Data centers, such as LAMMPS, NEST, GROMACS, PATMOS, or DIAPASOM, which were run in real cluster infrastructures to record VEF traces. These traces have been uploaded to the public VEF traces repository3 , so the community can use them. • We have extended the VEF traces framework with additional tools that help analyze the information stored in the trace files. These tools enable us to perform a static analysis to understand applications’ traffic patterns behavior and to identify potential bottlenecks (i.e., congestion scenarios) that they may generate. • We have performed the static analysis on a set of VEF traces, analyzing the behavior of collective operations, the expected bandwidth required from the network, the source and destination distribution, etc. • We have also performed a dynamic analysis by running the aforementioned VEF traces in the SAURON network simulator, under realistic network technologies, such as InfiniBand or BXI. This analysis, based on performance metrics such as flow completion time (FCT) and network resource utilization, helps us assess the impact of network congestion on the simulated scenarios. We expect that the proposed methodology will help network designers devise and configure efficient high-performance interconnection networks. The paper is organized as follows. Section 2 describes a specific use case where we have used the proposed framework. Section 3 details the experiments and analyses (i.e., static and dynamic) that we performed on the used VEF traces. Section 4 provides related works on this topic. Finally, some conclusions are drawn in Section 5. 2

Case Study: Network Workloads from EuroHPC-JU funded projects

As described before, the VEF traces framework allows MPI-based applications to be profiled using VEF Prospector, so the MPI calls (P2P or collective) are recorded into a VEF trace file. We can analyze the communication pattern recorded in VEF traces using the set of tools described later, and replay this trace in a network simulator. This section describes the tasks performed using the proposed methodology (see Figure 1) during an international collaboration among several European R&I projects (funded by EuroHPC-JU), such as RED-SEA4 , DEEP-SEA5 , IO-SEA6 , and MAELSTROM7 . This cross-collaboration has permitted several partners to gather VEF traces from HPC applications and benchmarks, such as NEST [13], GROMACS [14], LAMMPS [15], PATMOS [16], DIAPASOM [17], and Weather Forecasts (based on machine learning) [18]. These traces have been uploaded to a public repository [19], so they can be used to feed network simulators whenever they use the TraceLib library API. In the following, we focus on four representative applications (NEST, GROMACS, LAMMPS, and PATMOS) and describe their computational models, domain decomposition, and parallelization strategy. We also detail how the static and dynamic analyses enabled by the VEF framework have been performed on the VEF traces generated from different runs of these applications.

3 Repository URL: https://gitraap.i3a.info/jesus.escudero/vef-traces-repository 4 Project homepage: https://redsea-project.eu/ 5 Project homepage: https://www.deep-projects.eu/ 6 Project homepage: https://iosea-project.eu/ 7 Project homepage: https://www.maelstrom-eurohpc.eu/

Manuscript submitted to ACM

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks 2.1

Applications description

2.1.1 NEST. The NEural Simulation Tool (NEST) [13] models spiking neural networks of any size with focus on the dynamics, size, and structure of neural systems rather than on the exact morphology of individual neurons. Examples of these models include studies on spike-timing, dependent plasticity in extensive simulations of cortical networks, verification of mean-field models, and models of Alzheimer’s disease, Parkinson’s disease, and tinnitus. To simulate these models, NEST defines a neural system (also called the NEST network) composed of a potentially large number of neurons and their connections. In a NEST network, different neuron and synapse models can coexist. Any two neurons can have multiple connections with several properties. Thus, in general, connectivity cannot be described by a weight or connectivity matrix, but rather by an adjacency list. The NEST computational model uses a hybrid parallelism approach that combines shared- and distributed-memory architectures. NEST assigns neurons to virtual processes (i.e., MPI tasks or OpenMP processes) that compute the evolution of a corresponding fraction of the total number of neurons. Specifically, NEST uses OpenMP for processes on a single node that communicate “all-to-all” using shared memory. The MPI processes are mapped to different nodes in the computing cluster, and they communicate all-to-all in a distributed-memory architecture. Mapping OpenMP and MPI processes to computing cluster resources is left to NEST users and the cluster’s scheduling policies. This extensive communication between OpenMP and MPI processes requires distributing the different NEST model fractions and synchronizing them to maintain consistency. Therefore, each MPI process keeps its status synchronized with the other processes, exchanging data via extensive MPI_Alltoall calls after each model-integration step. Moreover, calls to MPI_AllReduce and MPI_Barrier are required for synchronization. These collective operations may be prone to generating congestion scenarios, as we show later.

2.1.2 GROMACS. The GROningen MAchine for Chemical Simulations (GROMACS) [14] is one of the most widely used open-source software codes in chemistry. It performs molecular dynamics simulations using Newtonian equations of motion for systems with hundreds to millions of particles. GROMACS is primarily designed for biochemical molecules such as proteins, lipids, and nucleic acids, which involve numerous complex bonded and non-bonded interactions. In addition to GROMACS, other options for molecular dynamics (MD) simulations include LAMMPS, as described later. Typically, an MD user selects an initial molecular configuration, specifies the atomic interactions and model physics, runs a simulation, and observes the trajectory. Such simulations evaluate the millions of particle interactions over billions of time steps, which can require extraordinary amounts of computational hardware and time. GROMACS exploits MPI to distribute the simulation across computing nodes, OpenMP for multi-threading within each node, and vector extensions on CPUs (AVX2) and GPUs to achieve maximum efficiency. GROMACS splits the problem into independent work units and distributes them across ensembles of simulations, multiple program paths, and domains. Spatial domain decomposition efficiently partitions the simulation to preserve reference locality within each domain. Specifically, a spatial domain is assigned to each rank, and the equations of motion for particles within its local domain are integrated. This data parallelization maps each domain to an MPI rank, which can access CPUs and GPUs within each cluster computing node. In general, GROMACS has been thoroughly optimized to reduce communication among MPI ranks. However, model scalability threatens this principle because interatomic interactions (short- and long-range) scale with the number of atoms. So it increases the volume of all-to-all communication among the MPI ranks. Although this communication involves multiple MPI collective operations, MPI_Broadcast and MPI_AlltoAll dominate the amount of exchanged information, which may stress the computing cluster interconnection network. Manuscript submitted to ACM

Sanchez de la Rosa et al.

2.1.3 LAMMPS. The Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) [15] is a classical molecular dynamics (MD) code with a focus on materials modeling8 , such as solid-state materials (metals, semiconductors), soft matter (biomolecules, polymers), and coarse-grained or mesoscopic systems. It can be used to model atoms or, more generally, to serve as a parallel particle simulator at the atomic, meso, or continuum scale. It is also well known for its ease of compiling and running on several different computer architectures (from laptops to large clusters). As the model’s computational cost scales linearly with the number of simulated atoms, LAMMPS is designed for distributed-memory MPI-based parallelism, so with a few hundred atoms/core, most of its models are scalable to millions of CPU cores. Specifically, LAMMPS defines a 2-D or 3-D simulation box to model the interaction of atoms (or particles), then partitioning this box into variable-sized and non-overlapping subdomains containing the same number of atoms. These subdomains are distributed across the cluster’s memory and assigned to different processors. Each processor stores information (positions, velocities, etc.) for the subset of (own) atoms within its subdomain, and stores copies of some of that information for other subdomain atoms (i.e., ghost) within a cutoff distance. This enables each processor to calculate short-range interactions that involve its own and ghost atoms. However, copies of ghost atoms must be updated, which requires communication with the processors that own them during long-range interactions among atoms. This information exchange among processors defines the communication pattern of LAMMPS that contains point-to-point (P2P) communications to transfer information to nearby domains, plus collective operations (mostly MPI_Broadcast and MPI_AllReduce) to update the domain population, and MPI_AlltoAll to perform fast-fourier transform (FFT) operations involving long-range atom interactions. As we demonstrate later, based on static analysis of a LAMMPS VEF trace, communication is dominated by the MPI_AllReduce collective operation, which can cause network congestion. 2.1.4 PATMOS. It is a Monte Carlo neutron transport code developed at CEA (France) that prototypes applications for nuclear safety and radiation shielding [16]. Its final goal is to perform pin-by-pin full-core depletion calculations for large nuclear power reactors with realistic temperature fields. PATMOS has been conceived to support two levels of parallelism (i.e., hybrid parallelism). The first level corresponds to distributed-memory parallelism and relies on MPI. The second level corresponds to shared-memory parallelism using OpenMP or POSIX threads. In PATMOS, simulations are divided into batches, even if all source particles are independent (for instance, one million histories are typically simulated as 1000 batches of 1000 particles each). Because of their size, the only non-duplicate mutable structures are the scores, which are concurrently modified as shared objects via OpenMP atomic adds. PATMOS is multi-threaded, so particles in the batch/cycle are dispatched in equal numbers to available threads, which simulate the particle history and compute their contributions to the score. This parallelization procedure is deterministic, thus assuring reproducibility and facilitating debugging. It is possible to use dynamic dispatch of particles to available threads to ensure load balancing, but in this case, the result is non-deterministic. Each thread has its own random number generator, initialized independently at the start of the simulation. Each MPI process executes an independent simulation. At the end of the run, the results of all simulations are aggregated via MPI reduction operations (i.e., MPI_AllReduce for computing the mean, followed by a call to MPI_Reduce for the variance calculation).

8 https://www.lammps.org/

Manuscript submitted to ACM

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks 2.2

Static analysis

As mentioned above, VEF traces can be analyzed offline before their use in a network simulator. VEF prospector provides two tools for this purpose: • tracetor is a simple application written in the C language to validate a VEF trace after it was generated with VEFProspector. It is included in the VEF-TraceLib repository under the test/ directory as a simulator-independent implementation reference. This application uses the TraceLib library to execute all the communication operations recorded in the trace and insert them in an ideal network (i.e., a bus with infinite bandwidth). In this way, tracetor analyzes the number of communication operations (and the messages in which they are decomposed) at first glance, before reproducing them in a simulation environment. Most importantly, tracetor also enables testing that all messages are generated and received correctly and that there are no failures in the dependencies between messages recorded in the VEF trace file. • offline-vef-analysis.sh is a script that characterizes the VEF trace by providing an extensive number of plots, text files, and PDF reports, with specific information about message generation, destination generation distribution, workload size, number, and types of collective operations, etc., which are gathered in a VEF trace generated through VEF-Prospector. For MPI collective operations, we assume the 𝑁 delivery algorithm in TraceLib. That is, for one-to-many collectives (e.g., MPI_Broadcast or MPI_Scatter), the root task generates one message for each remaining task in the communicator. For instance, for a MPI_Broadcast operation on a communicator with 𝑁 tasks, the total bytes sent by the root task in bytes is 𝑁 − 1 × 𝑠𝑖𝑧𝑒 𝑏𝑐𝑎𝑠𝑡 bytes. Subsequently, many-to-one collectives (e.g., MPI_Reduce or MPI_Gather) generate as many messages as the communicator has non-root tasks. For point-to-point operations, applications select tasks at runtime. In any case, our analysis tools do not alter the applications’ communication patterns. Using these tools, we performed static analyses on all the traces obtained. We uploaded one report per VEF trace to the already-mentioned public repository. Although static analysis provides valuable information, it is insufficient to explain how this traffic will behave under specific network configurations or the congestion scenarios it may generate. For this reason, we need additional insights from network simulation tools that use VEF traces and analyze specific metrics to characterize congestion, i.e., dynamic analysis. 2.3

Dynamic analysis

To replay VEF trace behavior in a network simulation tool and perform the dynamic analysis, we first need to integrate the simulator with the TraceLib library. The TraceLib library API provides functions for reading VEF traces, mapping tasks to end nodes, managing trace execution, and communicating with the simulator, so the network simulator only needs to ask the library to supply the application messages recorded in a VEF trace. For each obtained message, the network simulator splits it into packets according to the network MTU, injects these packets into the network, models how these packets are transferred from a source end node to a destination end node, composes messages again when packets arrive at the receiving node, and returns these messages to TraceLib, which finally processes them. Depending on the dependencies among messages, as recorded in a VEF trace, message reception at TraceLib may automatically generate dependent messages. For each generated message, TraceLib provides the simulator with the source and destination end nodes, the message length, and the message ID (used to uniquely identify the message within TraceLib). Note that the simulator does not need to distinguish whether a specific message was generated due to a point-to-point or collective communication Manuscript submitted to ACM

Sanchez de la Rosa et al.

operation, or whether an application task is stopped waiting for messages that the simulator has not yet provided to TraceLib. Moreover, TraceLib includes features for simulator developers, such as the ability to simulate multiple, even simultaneous, traces, a flexible mapping scheme for MPI tasks to end-nodes, and the ability to implement specific collective communication functions (e.g., those implemented by OpenMPI or MVAPICH). It is also possible to have idle end-nodes and end-nodes with multiple tasks allocated (even tasks from different traces). It is worth noting that MPI collective communication has been implemented in TraceLib based on the algorithms available in the OpenMPI driver, and these algorithms can be extended and improved by the community. In our proposed framework, TraceLib has been integrated with the OMNeT++-based SAURON simulator [8]. Note that the network simulator has also been extended to provide multiple metrics for dynamic analysis of VEF trace behavior across the configured network scenarios. SAURON models communication operations at a packet level, meaning it performs message packetization. This feature allows us to observe the evolution of traffic patterns over time in the VEF trace, using key network metrics such as application execution time, FCT, and switch queue occupancy. We have also added a new feature to the SAURON simulator: a GUI that colors network links based on buffer occupancy. This coloring can be observed “over time” for a specific VEF trace run in the simulator. This feature, combined with the static and dynamic analyses, can be used to characterize the network congestion. 3

Performance Evaluation

Based on the methodology described previously (see Figure 1), this section presents simulation results from experiments using the previously mentioned applications. These experiments allow us to characterize realistic communication patterns and congestion scenarios. In the following sections, we describe the experiment configuration, static analysis, and dynamic analysis of these traces using the SAURON network simulator. 3.1

Experiments configuration

We have selected VEF traces from NEST, GROMACS, LAMMPS, and PATMOS applications, available in the VEF traces repository [19] (see Section 2). Specifically, we have chosen VEF traces configured with 64 and 256 MPI ranks. We have performed the static analysis of these traces, examining several metrics such as the number of collective calls per operation, the amount of traffic generated by these operations, the message and byte count versus execution time, and the source/destination message and number of bytes exchanged. These results are shown in Section 3.2. Next, we used these VEF traces to feed the SAURON simulator. These results can be seen in Section 3.3. In more detail, we have configured SAURON to model realistic network technologies. Table 1 shows two configurations for the network architecture. Network configuration #1 has features similar to high-speed Ethernet networks, InfiniBand, or next-generation BXI, while network configuration #2 defines features used in network devices from previous generations (e.g., 100 Gbps link bandwidth), which are still utilized in some HPC systems and Data centers today. It is worth mentioning that our objective is to understand how communication operations recorded in VEF traces affect different network configurations, rather than to demonstrate which configuration is faster. Table 2 shows the network topologies used in our dynamic analysis: a 256-node Fat-tree and a 288-node Megafly [24]. These topologies are illustrated in Figure 2. We only show two terminal nodes connected to each leaf switch for clarity. Note that the Fat Tree is widely used in HPC systems and data centers. However, new topologies based on Dragonfly [25] ones have been proposed, providing better scalability as the number of nodes increases. One problem with Dragonfly networks is that the minimum-length path diversity is lower than that of Fat Trees. Therefore, Megafly [24] Manuscript submitted to ACM

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks

Network parameters Configuration #1 Configuration #2 Switch architecture Combined Input-Output Queues (CIOQ) [20] Input-Queued (IQ) [21] Input buffer size (KiB) 128 128 Output buffer size (KiB) 48 Flow control Priority Flow Control (PFC) [22] Credit-based [23] MTU (Bytes) 9600 (JUMBO frames) 4096 Variable packet size Yes No Link bandwidth (Gbps) 400 100 Link length (m) 3 3 Link latency (ns/m) 5 5 Number virtual channels 1 1 Table 1. Network configurations used in the Sauron simulator.

Number of terminal nodes Number of terminal nodes in each leaf switch Number of leaf switches Switch radix Number of Switches Network diameter Number of switch ports used

Fat Tree 256

Megafly 288

16 32 32 3 768

36 16 72 4 1152

Table 2. Network topologies configuration parameters.

Switch

...

Terminal node ...

...

Simple Link

...

Double Link

...

(a) 288-nodes Megafly.

(b) 256-nodes Fat Tree.

Fig. 2. Topologies used for the dynamic analysis experiments.

(a.k.a. Dragonfly+ [26]) topologies have been proposed to overcome this issue. As we can see in Figure 2a, the number of paths between two terminal nodes in different groups is multiple, compared to that in a Fat tree (see Figure 2b). Manuscript submitted to ACM

Sanchez de la Rosa et al. Regarding the routing algorithm, we have configured the network to use 𝐷-mod-𝐾 deterministic routing [27], which

efficiently balances traffic across available paths in the topology, making it a good option for all-to-all communication. Note that this deterministic routing algorithm performs similarly to adaptive routing algorithms under certain traffic patterns [28]. Unfortunately, other traffic patterns generated by collective communication in the network, such as all-to-one or all-to-many, may cause congestion, degrading routing performance. Note that the study of other routing algorithms aimed at alleviating congestion problems, such as oblivious or adaptive, is out of the scope of this paper. The SAURON simulator provides a range of metrics to evaluate the performance of the configured networks. Among the previously defined scenarios, the most useful for dynamic analysis are the execution (i.e., simulated) time of a VEF trace and the Flow Completion Time (FCT), which measures the average time from when an application message is generated until it is completely received at the destination end node. SAURON also provides metrics to analyze switch buffer occupancy over time, helping identify communication bottlenecks and characterize network congestion. 3.2

Static analysis results

The main objective of the static analysis is to study the behavior of a VEF Trace before it is used in a simulated environment. This analysis is intended to show the number of messages exchanged among end nodes, the source/destination distribution, expected bytes exchanged, etc., so it does not assume a specific network topology or implementation for collective operations message exchange. This section presents results obtained using the scripts described in Section 2.2 on VEF traces from NEST, GROMACS, LAMMPS, and PATMOS applications. Figure 3 shows the static analysis results for the NEST application. Figures 3a and 3b depict the number of messages and bytes generated in the network per communication operation, which provides information about which operations certainly dominate the network communication. The byte counts in the figures reflect the MPI buffer sizes when calling the collective or P2P operations. This means that, for example, a Broadcast operation has 𝑠𝐵 bytes sent by the root task, while the others (𝑠𝑖𝑧𝑒𝑐𝑜𝑚𝑚 ) receive 𝑠𝐵 bytes. Of course, the real traffic generated would be 𝑠𝐵 × (𝑠𝑖𝑧𝑒𝑐𝑜𝑚𝑚 − 1) bytes if employing the 𝑁 delivery algorithm, or 𝑠𝐵 × 𝑙𝑜𝑔 (𝑠𝑖𝑧𝑒𝑐𝑜𝑚𝑚 −1) bytes if using the logN algorithm. As we can see in Figure 3a, out of the four collective operations found in the trace, All2All is the most predominant in the number of calls, which is three orders of magnitude larger than that of AllReduce. Figure 3b shows that the All2All contribution to traffic in the number of bytes is more than five orders of magnitude higher than the second one. Also, the number of bytes generated by AllGather operations exceeds that of AllReduce. Figures 3c and 3d show the number of messages and bytes generated in the network as a function of time, which permits identifying when a bottleneck may appear during the application execution, and which specific communication operation type is responsible for creating this situation. As we can see in Figure 3c, the application starts with a burst of AllGather operations, until it reaches around 350 seconds of runtime, where AllGather and AllReduce calls precede a period dominated by All2All operations; after that period, the application ends. In Figure 3d, the traffic generation in the number of bytes follows the trend observed in the collective operation calls. However, as noted earlier, AllGather operations generate more traffic with fewer calls than AllReduce, as shown before and after the period when All2All is predominant. Note that the All2All traffic burst after 350 seconds of runtime is combined with traffic of AllReduce and AllGather, so traffic flows from the latter may suffer extra latency due to congestion. Figures 3e and 3f depict the number of messages and bytes generated in the network as matrices showing source-todestination MPI rank pairs, which enables identification of which source-destination communications are most popular and may create congested areas. In Figure 3e, we can see that overall message exchange among MPI ranks is aligned with the operations found in the trace, where all-to-all communications are the predominant ones. This figure also Manuscript submitted to ACM

100 k

Number of Bytes

Number of Messages

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks

10 k 1k 100 10 1

1T 10 G 100 M 1M 10 k 100 1

(b) Traffic generated by each operation.

AllGather

Barrier

10 M 1M 100 k 10 k 1k 100 10 1

All2All

10 G Number of Bytes

Number of Messages

All2All

AllGather

AllReduce

All2All

Barrier

AllGather

AllReduce

(a) Number of collective calls for each operation. AllReduce

100 M 1M 10 k 100 1

100

200

300

400

500

600

100

200

Task (dst) 60 80

400

100 120

Task (dst) 60 80

100 120

12.0 k

3.5 M 120 3.0 M

10.0 k 100

2.5 M

8.0 k 6.0 k

Task (src)

100

600

(d) Collective operation byte count.

120

500

Time (secs)

300

80 2.0 M 60

1.5 M

4.0 k

1.0 M

2.0 k

500.0 k

0.0

(e) Number of exchanged messages.

(f) Number of exchanged bytes.

Fig. 3. NEST static analysis results.

shows that NEST communication flows uniformly to and from every rank. Note that this behavior corresponds to the NEST computation model, described in Section 2.1.1, where the model neurons are distributed among the application Manuscript submitted to ACM

Sanchez de la Rosa et al.

processes and communicate all-to-all (i.e., using All2All collective calls) their status to synchronize the data distributed throughout the system. Figure 4 shows the static analysis results for the GROMACS application, organized in subfigures, as we have described for the NEST application. Specifically, Figures 4a and 4b show that All2All collective dominates the number of operation calls and generated bytes, followed by the Broadcast (one order of magnitude less) and P2P (almost three orders of magnitude less). Figures 4c and 4d show the number of generated messages and bytes per communication operation. We can see that the application calls Broadcast and AllReduce at the beginning, and there are short bursts of AllReduce, Broadcast, and Scatter until past 20 seconds, where P2P traffic is exchanged. After 40 seconds, All2All and P2P calls generate most of the network traffic and last for around 10 seconds, so congestion may occur during this period. After that, some Reduce and AllReduce bursts mark the end of the simulation. Figures 4e and 4f show the message and bytes exchanged among the MPI tasks, where we can see that communication occurs within groups of nodes in the network. This is because, as described in Section 2.1.2, GROMACS splits the problem into independent units of work and distributes them across ensembles of simulations. Moreover, the spatial domain decomposition partitions the application into multiple MPI communicators to group tasks, so all-to-all and broadcast communication can be restricted to exploit locality and reduce communication. Figure 5 shows the static analysis results for the LAMMPS application. Figures 5a and 5b show the number of calls and bytes generated by the collective and point-to-point operations. As we can see, AllReduce is the most predominant operation, followed by P2P and Broadcast. Figures 5c and 5d show that LAMMPS starts broadcasting information among the nodes. Before one second passes, Reduce operations are called, and afterward, AllReduce operations are invoked. After a while, more data is broadcast, and another reduction happens. At two seconds of runtime, a burst of P2P and AllReduce operations are called, and, finally, some hundreds of Broadcast and a million AllReduce messages mark the end of the execution. Figure 5d shows that the generated bytes per collective are proportional to the collective call count. Note that the communication pattern of LAMMPS, described in Section 2.1.3, uses P2P operations to transfer data between two nearby domains, while Broadcast and, mainly, AllReduce collective operations are used to update the domain population. Moreover, AlltoAll calls implement FFT operations required by long-range particle interactions. Figures 5e and 5f show that most traffic is either one-to-many or many-to-one, with MPI rank 0 as the root. Note that AllReduce, while following a many-to-many communication pattern, behaves like a reduction and a broadcast with MPI task 0 as the root rank. As with the NEST trace, most P2P messages are self-messages. Figure 6 shows the static analysis results for the PATMOS application. Figures 6a and 6b show that the most predominant operation is Broadcast, in terms of number of generated messages and bytes, followed by AllReduce and Reduce. The communication of this application, as seen in Figure 6c, starts with two bursts of Broadcast with large amounts of data, after which nodes engage in computation using shared memory, so that no MPI calls are observed during a considerable period. More than 1200 seconds later, AllReduce and Reduce operations are performed. Figure 6d shows that the number of messages generated correlates with the number of generated bytes. As described in Section 2.1.4, the computational model of PATMOS uses a hybrid parallel approach, where data is distributed among the processes that use multi-threading to operate on shared objects within each process. These threads compute their contributions to the particle’s history. Only at the end of execution are the results of all processes shared among MPI ranks (i.e., via AllReduce and Reduce operations). Figures 6e and 6f show the messages and bytes exchanged by source and destination MPI ranks. As we can see, the most common source/destination combination is from MPI rank 0 to the rest of the tasks, which means that most of the traffic is generated by the Broadcast operations. Manuscript submitted to ACM

100 k

Number of Bytes

Number of Messages

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks

10 k 1k 100 10 1

1T 10 G 100 M 1M 10 k 100 1 P2P

All2All

Scatter

Gather

AllReduce

Reduce

Bcast

P2P

All2All

Barrier

Scatter

Gather

AllReduce

Reduce

Bcast

(a) Number of collective calls for each operation.

(b) Traffic generated by each operation.

Broadcast

Reduce

AllReduce

Gather

Scatter

Barrier

All2All

P2P

10 G Number of Bytes

100 k 10 k 1k 100 10 1

100 M 1M 10 k 100 10

Time (secs)

Task (dst) 60 80

100 120

Task (dst) 60 80

120 140.0 M 100

2.5 k 2.0 k 1.5 k

Task (src)

3.0 k

120.0 M

100.0 M 80.0 M

60.0 M 40

1.0 k 20 0

100 120

3.5 k

160.0 M

120

(d) Collective operation byte count.

4.0 k

100

Time (secs)

Task (src)

Number of Messages

500.0 0.0

(e) Number of exchanged messages.

40.0 M 20

20.0 M

0.0

(f) Number of exchanged bytes.

Fig. 4. GROMACS static analysis results. Manuscript submitted to ACM

Sanchez de la Rosa et al.

100 k

Number of Bytes

Number of Messages

10 k 1k 100 10 1

100 M 10 M 1M 100 k 10 k 1k 100 10 1

(b) Traffic generated by each operation.

Reduce

AllReduce

Barrier

P2P

10 M 1M 100 k 10 k 1k 100 10 1

10 G Number of Bytes

Number of Messages

P2P

AllReduce

Broadcast

Reduce

Bcast

P2P

Barrier

AllReduce

Reduce

Bcast

(a) Number of collective calls for each operation.

100 M 1M 10 k 100 1

0.5

1.5

2.5

0.5

Task (dst) 60 80

1.5

(d) Collective operation byte count.

100 120

Task (dst) 60 80

100 120

12.0 k

1.2 M

120

120 10.0 k

1.0 M

100 8.0 k 6.0 k

Task (src)

100 80

Time (secs)

80 60

800.0 k 600.0 k

4.0 k

400.0 k

2.0 k

200.0 k

0.0

(e) Number of exchanged messages.

(f) Number of exchanged bytes.

Fig. 5. LAMMPS static results.

Manuscript submitted to ACM

2.5

1.0 k

1.0 T Number of Bytes

Number of Operations

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks

100.0 10.0 1.0

100.0 G 10.0 G 1.0 G 100.0 M AllReduce

Collective Operation

(a) Number of collective calls for each operation. Broadcast

Reduce

Bcast

Barrier

AllReduce

Reduce

Bcast

Collective Operation

Reduce

(b) Traffic generated by each operation. AllReduce

Barrier

10 G Number of Bytes

1k 100 10 1

100 M 1M 10 k 100 1

200

400

600

800 1000 1200 1400

200

400

Task (dst) 60 80

600

800 1000 1200 1400

Time (secs)

(d) Collective operation byte count.

100 120

Task (dst) 60 80

100 120

80.0

250.0 M

120

120 70.0

80 60

60.0 50.0 40.0 30.0

Task (src)

100 Task (src)

Number of Messages

10 k

100

200.0 M

150.0 M

60 100.0 M 40

20.0 20 0

10.0 0.0

(e) Number of exchanged messages.

50.0 M

20 0

0.0

(f) Number of exchanged bytes.

Fig. 6. PATMOS static results.

Manuscript submitted to ACM

Sanchez de la Rosa et al.

3.3

Dynamic analysis results

The dynamic analysis evaluates network performance metrics when VEF traces are used to feed a network simulator, such as simulated runtime or FCT, and charts such as the cumulative distribution function (CDF) or buffering occupancy at network devices (see Section 3.1). In this section, we analyze a set of metrics from the SAURON simulator, based on the results of the experiments described in Section 3.1. Figure 7 shows the simulated execution time when we feed the SAURON simulator with VEF traces from the NEST, GROMACS, LAMMPS, and PATMOS applications. We have configured SAURON to model a 288-node Megafly and 256-node Fat-tree (see Table 2) and two network configurations defined in Table 1.

(a) 288-node Megafly.

(b) 256-node Fat-Tree.

Fig. 7. Execution time (seconds) for VEF traces of NEST, GROMACS, LAMMPS, and PATMOS, configured with 256 MPI ranks and comparing network configurations #1 and #2.

Note that the Execution time per application is the time the simulator takes to run completely the VEF trace, and that this is virtually identical to that observed in the static analysis (see Figures 3, 4, 5, and 6). This confirms that the execution times measured in the dynamic analysis match those predicted by the static analysis. Since a large portion of application execution time is spent in idle wait rather than active network use, network configurations and topologies generally have little impact on execution time. But there are some exceptions, for instance, when simulating the LAMMPS application. Figure 8 illustrates the speedup achieved in execution time when comparing configuration #1

1.40

1.35

1.30

1.25

1.25 Speedup

Speedup

against configuration #2.

1.20 1.15

1.10

1.05

1.00

0.95

0.95 NEST

Gromacs Lammps experiment

(a) 288-node Megafly.

Patmos

NEST

Gromacs Lammps experiment

Patmos

(b) 256-node Fat-Tree.

Fig. 8. Speedup obtained in configuration #1 compared against configuration #2. Manuscript submitted to ACM

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks

As shown, there are negligible differences in execution time across network configurations and topologies because the traffic volume generated by these traces does not cause excessive network contention. LAMMPS is the only application in which a significant speedup is observed between the two configurations, reaching up to 1.35×. This occurs because it has the shortest execution time; consequently, the impact of the network architecture is more pronounced. Indeed, the time between messages recorded in the VEF trace (i.e., the time spent processing each MPI call) depends on the machine’s CPU architecture. Figure 9 illustrates the mean FCT for the same scenarios described above. For every VEF trace and network topology, configuration #1 achieves a lower mean FCT because it is faster than configuration #2. It has a higher link data rate and MTU, and it uses variable packet size, allowing packets to be smaller than the MTU for short messages. In other words, the payload size for packets is only as long as required. Also, we note that for some applications, such as LAMMPS, the Fat-Tree topology achieves a slightly lower mean FCT than the Megafly results, since the former has a smaller diameter than the latter.

(a) Mean FCT in a 288-node Megafly

(b) Mean FCT in a 256-node Fat-Tree

(d) Maximum FCT in a 288-node Fat-Tree

Fig. 9. Mean and Maximum FCT (in seconds) for VEF traces of NEST, GROMACS, LAMMPS, and PATMOS applications configured with 256 MPI ranks, comparing network configurations #1 and #2.

We observe a higher FCT value for the NEST application on the 288-node Megafly topology, due to the higher diameter. Despite having more switch ports available for distributing trace load in the 288-node Megafly topology, latency is nearly the same in both topologies. This means that processing time at the nodes is dominated by the computing times recorded in the trace file, which depend on the node architecture used to collect the specific VEF Manuscript submitted to ACM

Sanchez de la Rosa et al.

trace. However, we can appreciate that using Configuration #1 significantly reduces the maximum FCT compared to Configuration #2. Figure 10 shows the Cumulative Distribution Function (CDF) of the FCT metrics per application for each network configuration. Configuration #1 obtains better FCT values than Configuration #2, regardless of the network topology. Moreover, the Megafly topology yields slightly worse results than the Fat Tree. For the NEST application (see Figure 10a), Configuration #1 obtains significantly better FCT values than Configuration #2. For Conf. #1, 100% of flows finish in less than 5ms. For GROMACS (see Figure 10b), similar conclusions can be drawn, with a special focus on the step produced in the FCT series after 70% of the flow completion, where latency moderately increases for Conf. #1 while significantly increases for Conf. #2, since GROMACS generates a moderate load that delays the completion of medium-sized and large flows. Similar to NEST, we can see for LAMMPS (see Figure 10c) and PATMOS (see Figure 10d) applications that Conf. #1 completes 100% of its traffic flows with significantly lower FCT values than those obtained by Conf. #2, and

1.0

0.8

0.6

CDF

the tail latency of the latter is by far the largest one.

0.4

0.4 #1 - 256-node Fat Tree #2 - 256-node Fat Tree #1 - 288-node Dragongfly+ #2 - 288-node Dragongfly+

0.2 0.0 0.00000

0.00005

0.00010 0.00015 0.00020 Flow Completion Time (s)

0.00025

#1 - 256-node Fat Tree #2 - 256-node Fat Tree #1 - 288-node Dragongfly+ #2 - 288-node Dragongfly+

0.2 0.0

0.00030

(a) NEST

4 6 Flow Completion Time (s)

10 1e 5

(b) GROMACS 1.0

CDF

0.8 0.6 0.4 #1 - 256-node Fat Tree #2 - 256-node Fat Tree #1 - 288-node Dragongfly+ #2 - 288-node Dragongfly+

0.2 0.0 0.000

0.001

0.002

0.003 0.004 0.005 Flow Completion Time (s)

0.006

0.007

(d) PATMOS

Fig. 10. CDF of FCT for each trace, comparing topology and network architecture configuration.

To extend the dynamic analysis, we must examine additional FCT metrics to identify why tail latency increases significantly in specific network scenarios. The following section extends our study by characterizing congestion episodes in the applications we analyzed. Manuscript submitted to ACM

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks 3.4

Congestion characterization

An indicator of the severity of network congestion is the occupancy of the queues at switches’ ports. In this section, we analyze the occupation of input ports in switches using the new GUI in the SAURON simulator with VEF traces. This GUI depicts the topology’s switches and links in a smooth layout, making it easy to identify hotspots and congestion dynamics induced by the traffic patterns at a given time instant. The input buffer occupancy is shown by coloring half of the links; a link connected to a switch or NIC is colored based on the occupancy of the input port buffer for that switch or NIC. The logic behind the link coloring is as follows: green indicates empty buffers, blue indicates low occupancy, and red indicates the highest occupancy level in a period of time. Any color displayed between those is calculated using a gradient for higher granularity. This way, we can visualize the presence of congestion trees and their potential backpropagation across the network. Figure 11 shows the maximum input occupation of the Fat-tree and Megafly topologies (see Table 2) and network architectures #1 and #2 (see Table 1) for the NEST application.

(a) Fat Tree (Conf. #1).

(b) Fat Tree (Conf. #2).

(d) Megafly (Conf. #2).

Fig. 11. Maximum input occupation in NEST from 358.9 to 359 seconds.

According to the static analysis (see Figure 3), after 358 seconds of execution time, the communication is dominated by AlltoAll collective operations, producing a moderate load. For network architecture #1, we can see that links are mostly colored in blue, meaning that there is traffic exchange with low buffer occupancy (see Figures 11a and 11c). Manuscript submitted to ACM

Sanchez de la Rosa et al.

Note that we use 400 Gbps links in this configuration. However, in network architecture #2, i.e., small buffering and lower link bandwidth (100 Gbps), we can see that occupancy is high in the Fat Tree (see Figure 11b), at the input buffers of the switches at the second stage. We can also see end nodes that are visited more than others, i.e., those with links colored in red, indicating that congestion originates at those nodes. By contrast, in the Megafly topology and network architecture #2 (see Figure 11d), we observe congestion in the input buffers of the first-level switches and in some inter-group communications. To analyze the impact of congestion in greater detail, we combined the previously analyzed Nest trace with a synthetic traffic burst. In this scenario, 64 nodes send a 10 MiB message to a single destination node simultaneously. Figure 12 shows the maximum input port occupancy during this interval. As observed, this burst has a more significant impact on configuration #2 across both topologies. It is also notable that the maximum occupancy in the remaining ports is considerably lower. This suggests that the congestion tree generated by incast traffic reaches the source nodes, thereby negatively impacting overall network performance. This observation is corroborated by Figure 13, which shows that although the mean FCT is unaffected by incast traffic, the maximum FCT increases substantially across both topologies and configurations. The highest maximum FCTs are observed in the Megafly topology with configuration #2. We provide congestion characterization for the remaining traces in the supplementary PDF. 4

Related Work

Reproducing the traffic patterns of real workloads in simulators is vital for comparing our simulation tools against real systems. Several tools exist to model and characterize traffic, such as Vampir [29], Score-P [30], Scalasca [31], DUMPI [1], or Extrae [32]. However, these tools may not be suitable for network simulations due to trace size or the use of absolute timestamps. For instance, traces that contain more information than necessary (e.g., message source, destination, and size) may be slower or more challenging to process. Moreover, using absolute timestamps means that time-related information is tightly tied to the system and network on which traces were collected, and this is reflected in the traffic patterns. To address this, Andujar et al. proposed VEF self-related traces [33], which include only information used in network simulators. Relative timestamps and self-related messages help extract application traffic patterns, leaving network behavior to the simulator. The framework consists of VEF-Prospector [34], a profiling program that generates MPI calls for running applications, and VEF-TraceLib [35], a library for use in simulators. VEF-TraceLib can also synthesize patterns from other traffic models [6]. After capturing MPI calls, VEF-Prospector also includes a static analysis tool, as shown in this paper. On the other hand, interconnection network simulators provide designers with insights into expected network performance that would otherwise be difficult to obtain using other methodologies or tools. Several network simulators are available. TOPAZ [36] is a general-purpose interconnection network simulator that can model a lot of different switches and gives the functionality to sacrifice precision for the sake of speed. NS-3 [37] is a discrete-event network simulator for Internet systems, targeted primarily for research and educational use. It is primarily based on NS-2, which was initially devised to model TCP. Ns-3 was designed from the ground up to convey higher detail and realism, bringing actual software implementations closer to the real-world models. The Structural Simulation Toolkit (SST) [38] is a modular simulation framework for modelingdesigned to model high-performance computing systems at multiple levels of abstraction. It facilitates hardware and software co-design by enabling detailed simulations of processors, memory hierarchies, and interconnects. SST supports parallel execution and scalability, allowing researchers to explore large-scale architectural configurations. Its component-based architecture promotes extensibility and integration with third-party tools. SST is widely used in architectural research for performance prediction and design-space exploration. Manuscript submitted to ACM

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks

(a) Fat Tree (Conf. #1).

(b) Fat Tree (Conf. #2).

(d) Megafly (Conf. #2).

Fig. 12. Maximum input occupation in NEST from 358.9 to 359 seconds, when the incast traffic is injected.

OMNeT++ [9] is a general simulation environment designed to provide a simulation framework for developers and researchers in distributed systems, computer networks, and multiprocessors. Specific simulation models can be built in OMNeT++ to simulate domains such as wireless networks, business processes, P2P networks, or high-performance interconnection networks. Some well-known OMNeT-based models include INET [39] and the SAURON simulator [8], which were used in this study. The SAURON simulator has recently been integrated with other simulators, such as COSSIM, which models the system-node architecture with higher accuracy [40], enabling us to generate VEF traces at different message-generation rates. In this paper, we have assumed constant message-generation rates for the systems used to collect VEF traces. Recently, other simulation frameworks, such as Astra-Sim [3] or ATLAHS [4], enable systematic investigation of modern HPC and deep learning workloads, supporting the identification of bottlenecks and the design of efficient methodologies for large-scale DNN models across diverse future platforms. Through its APIs, users can seamlessly integrate any custom-made network, compute, or memory simulator backend in a plug-and-play manner. Integrating the methodology proposed in this paper with the Astra-Sim or ATLAHS APIs is left for future work. Additionally, the DUMPI, GOAL, and Chakra trace-file formats used in SST, ATLAHS, and Astra-sim, respectively, Manuscript submitted to ACM

Sanchez de la Rosa et al.

Mean FlowCompletionTime [us]

#1 #2

40 30 20 10 0

With Incast, FatTree

With Incast, Megafly Without Incast, FatTree Without Incast, Megafly incast, network (a) Mean Flow Completion Time

#1 #2

Max FlowCompletionTime [ms]

2.5 2.0 1.5 1.0 0.5 0.0

With Incast, FatTree

With Incast, Megafly Without Incast, FatTree Without Incast, Megafly incast, network (b) Maximun Flow Completion Time

Fig. 13. Mean and Maximum FCT for NEST, configured with 256 MPI ranks and comparing network configurations #1 and #2, in both topologies, and with and without incast.

should be considered and compared with the VEF format to discuss the advantages and disadvantages of these three formats. 5

Conclusions

In this paper, we present an open-source framework and methodology for characterizing communication patterns from real MPI-based applications running on HPC systems and converting these patterns into specific-purpose VEF traces, enabling network designers and researchers to use them to feed network simulators. We have analyzed a set of VEF traces from a public repository, which were collected through collaboration across several EU-funded projects Manuscript submitted to ACM

Characterization of Real Communication Patterns and Congestion Dynamics in HPC Interconnection Networks

(EuroHPC-JU). These traces were obtained from real runs of NEST, GROMACS, LAMMPS, and PATMOS applications in different HPC systems. To analyze the behavior of these traces, we performed static analysis to measure key metrics, such as the amount of data generated by communication operations, which may indicate network congestion. We also performed dynamic analysis of network performance metrics using the OMNeT++-based SAURON interconnection network simulator. The analysis results provide insight into congestion generated in the network by the mentioned applications. The traces repository and the open-source framework are available to the community to either populate that repository with VEF traces from new application runs or use the available ones in their network simulators. Acknowledgment This work has been supported by the RED-SEA project, which received funding from the European High-Performance Computing Joint Undertaking (EuroHPC-JU) under grant agreement No. 955776. For this project, the EuroHPC-JU receives support from the European Union’s Horizon 2020 research and innovation programme of Spain under grants PCI2021-121934 (UPV) and PCI2021-121976 (UCLM), among other countries. References [1] “Structural Simulation Toolkit (SST) DUMPI Trace Library,” (Accessed July 5, 2024). [Online]. Available: https://github.com/sstsimulator/sst-dumpi [2] F. J. Andujar, J. A. Villar, F. J. Alfaro, J. L. Sánchez, and J. Escudero-Sahuquillo, “An open-source family of tools to reproduce mpi-based workloads in interconnection network simulators,” J. Supercomput., vol. 72, no. 12, pp. 4601–4628, 2016. [Online]. Available: https://doi.org/10.1007/s11227-016-1757-0 [3] W. Won, T. Heo, S. Rashidi, S. Sridharan, S. Srinivasan, and T. Krishna, “Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale,” in 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2023, pp. 283–294. [4] S. Shen, T. Bonato, Z. Hu, P. Jordan, T. Chen, and T. Hoefler, “Atlahs: An application-centric network simulator toolchain for ai, hpc, and distributed storage,” 2025. [Online]. Available: https://arxiv.org/abs/2505.08936 [5] F. J. Andújar, M. S. de la Rosa, J. Escudero-Sahuquillo, and J. L. Sánchez, “Extending the VEF traces framework to model data center network workloads,” J. Supercomput., vol. 79, no. 1, pp. 814–831, 2023. [Online]. Available: https://doi.org/10.1007/s11227-022-04692-0 [6] B. Montazeri, Y. Li, M. Alizadeh, and J. Ousterhout, “Homa: a receiver-driven low-latency transport protocol using network priorities,” in Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, ser. SIGCOMM ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 221–235. [Online]. Available: https://doi.org/10.1145/3230543.3230564 [7] L. Gonzalez-Naharro, J. Escudero-Sahuquillo, P. J. García, F. J. Quiles, J. Duato, W. Sun, X. Yu, and H. Zheng, “Modeling traffic workloads in data-center network simulation tools,” in 17th International Conference on High Performance Computing & Simulation, HPCS 2019, Dublin, Ireland, July 15-19, 2019. IEEE, 2019, pp. 1036–1042. [Online]. Available: https://doi.org/10.1109/HPCS48598.2019.9188099 [8] P. Yebenes, J. Escudero-Sahuquillo, P. J. Garcia, and F. J. Quiles, “Networks of exascale systems with omnet++.” in Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, 2013, pp. 203–207. [9] A. Varga and R. Hornig, “An overview of the omnet++ simulation environment.” ICST, 5 2010. [10] P. Yébenes, G. Maglione-Mathey, J. Escudero-Sahuquillo, P. J. García, and F. J. Quiles, “Modeling a switch architecture with virtual output queues and virtual channels in hpc-systems simulators,” in 2016 International Conference on High Performance Computing & Simulation (HPCS), 2016, pp. 380–386. [11] G. Gomez-Lopez, M. S. de la Rosa, J. Escudero-Sahuquillo, P. J. García, F. J. Quiles, and P. Lagadec, “Hybrid congestion control for bxi-based interconnection networks,” in Euro-Par 2024: Parallel Processing - 30th European Conference on Parallel and Distributed Processing, Madrid, Spain, August 26-30, 2024, Proceedings, Part II, ser. Lecture Notes in Computer Science, J. Carretero, S. Shende, J. García-Blas, I. Brandic, K. Olcoz, and M. Schreiber, Eds., vol. 14802. Springer, 2024, pp. 242–256. [Online]. Available: https://doi.org/10.1007/978-3-031-69766-1_17 [12] M. S. de la Rosa, G. Gomez-Lopez, F. J. Andújar, J. Escudero-Sahuquillo, J. L. Sánchez, F. J. Alfaro-Cortés, and P. Lagadec, “Quality-of-service provision for bxiv3-based interconnection networks,” J. Supercomput., vol. 81, no. 4, p. 601, 2025. [Online]. Available: https://doi.org/10.1007/s11227-025-07069-1 [13] M.-O. Gewaltig and M. Diesmann, “Nest (neural simulation tool),” Scholarpedia, vol. 2, no. 4, p. 1430, 2007. [Online]. Available: https://doi.org/10.4249/scholarpedia.1430 [14] M. J. Abraham, T. Murtola, R. Schulz, S. Páll, J. C. Smith, B. Hess, and E. Lindahl, “Gromacs: High performance molecular simulations through multi-level parallelism from laptops to supercomputers,” SoftwareX, vol. 1-2, pp. 19–25, 2015. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2352711015000059 [15] A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, P. J. in ’t Veld, A. Kohlmeyer, S. G. Moore, T. D. Nguyen, R. Shan, M. J. Stevens, J. Tranchida, C. Trott, and S. J. Plimpton, “LAMMPS - a flexible simulation tool for particle-based materials modeling at the Manuscript submitted to ACM

Sanchez de la Rosa et al.

atomic, meso, and continuum scales,” Comp. Phys. Comm., vol. 271, p. 108171, 2022. [Online]. Available: https://doi.org/10.1016/j.cpc.2021.108171 [16] E. Brun, S. Chauveau, and F. Malvagi, “Patmos: A prototype monte carlo transport code to test high performance architectures,” 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:237524013 [17] M. Barnaba, “Diapasom,” https://github.com/exactlab/diapasom, 2022. [18] A. Bahra, “Managing work flows with ecflow,” pp. 30–32, 2011 2011. [Online]. Available: https://www.ecmwf.int/node/17434 [19] “The VEF Traces Repository homepage,” (Accessed August 1, 2025). [Online]. Available: https://gitraap.i3a.info/jesus.escudero/vef-traces-repository [20] A. Awan and R. Venkatesan, “Design and implementation of enhanced crossbar CIOQ switch architecture,” in Canadian Conference on Electrical and Computer Engineering 2004 (IEEE Cat. No.04CH37513), vol. 2, 2004, pp. 1045–1048 Vol.2. [21] M. Karol, M. Hluchyj, and S. Morgan, “Input versus output queueing on a space-division packet switch,” IEEE Transactions on communications, vol. 35, no. 12, pp. 1347–1356, 1987. [22] 802.1Qbb, “IEEE Standard for Local and Metropolitan Area Networks—Virtual Bridged Local Area Networks – Amendment: Priority-based Flow Control.” IEEE, 2011. [Online]. Available: https://1.ieee802.org/dcb/802-1qbb/ [23] N. Kung and R. Morris, “Credit-based flow control for atm networks,” IEEE Network, vol. 9, no. 2, pp. 40–48, 1995. [24] M. Flajslik, E. Borch, and M. A. Parker, “Megafly: A topology for exascale systems,” in High Performance Computing: 33rd International Conference, ISC High Performance 2018, Frankfurt, Germany, June 24-28, 2018, Proceedings 33. Springer, 2018, pp. 289–310. [25] J. Kim, W. J. Dally, S. Scott, and D. Abts, “Technology-driven, highly-scalable dragonfly topology,” in 2008 International Symposium on Computer Architecture, 2008, pp. 77–88. [26] A. Shpiner, Z. Haramaty, S. Eliad, V. Zdornov, B. Gafni, and E. Zahavi, “Dragonfly+: Low Cost Topology for Scaling Datacenters,” in 2017 IEEE 3rd International Workshop on High-Performance Interconnection Networks in the Exascale and Big-Data Era (HiPINEB), 2017, pp. 1–8. [27] E. Zahavi, “Fat-tree routing and node ordering providing contention free traffic for mpi global collectives,” Journal of Parallel and Distributed Computing, vol. 72, no. 11, pp. 1423–1432, 2012, communication Architectures for Scalable Systems. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0743731512000305 [28] C. Gómez, F. Gilabert, M. E. Gómez, P. López, and J. Duato, “A HoL-blocking aware mechanism for selecting the upward path in fat-tree topologies,” The Journal of Supercomputing, vol. 71, no. 7, pp. 2339–2364, Jul. 2015. [Online]. Available: https://doi.org/10.1007/s11227-014-1303-x [29] A. Knüpfer, H. Brunst, J. Doleschal, M. Jurenz, M. Lieber, H. Mickler, M. S. Müller, and W. E. Nagel, “The vampir performance analysis tool-set,” in Tools for High Performance Computing, M. Resch, R. Keller, V. Himmler, B. Krammer, and A. Schulz, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 139–155. [30] A. Knupfer, C. Rossel, D. an Mey, S. Biersdorff, K. Diethelm, D. Eschweiler, M. Geimer, M. Gerndt, D. Lorenz, A. Malony, and W. E. Nagel, “Score-p: A joint performance measurement run-time infrastructure for periscope, scalasca, tau, and vampir.” 8 2012. [Online]. Available: https://www.osti.gov/biblio/1567522 [31] M. Geimer, F. Wolf, B. J. N. Wylie, E. Ábrahám, D. Becker, and B. Mohr, “The scalasca performance toolset architecture,” Concurrency and Computation: Practice and Experience, vol. 22, no. 6, p. 702–719, apr 2010. [32] “Extrae documentation — Extrae 3.8.3 documentation,” (Accessed July 5, 2024). [Online]. Available: https://tools.bsc.es/doc/html/extrae/index.html [33] F. J. Andujar, J. A. Villar, J. L. Sánchez, F. J. Alfaro, and J. Escudero-Sahuquillo, “VEF traces: A framework for modelling MPI traffic in interconnection network simulators,” in 2015 IEEE International Conference on Cluster Computing, CLUSTER 2015, Chicago, IL, USA, September 8-11, 2015. IEEE Computer Society, 2015, pp. 841–848. [Online]. Available: https://doi.org/10.1109/CLUSTER.2015.141 [34] “VEF-Prospector repository homepage,” (Accessed July 5, 2024). [Online]. Available: https://gitraap.i3a.info/fandujar/VEF-Prospector [35] “VEF-TraceLib repository homepage,” (Accessed July 5, 2024). [Online]. Available: https://gitraap.i3a.info/fandujar/VEF-TraceLIB [36] P. Abad, P. Prieto, L. G. Menezo, A. Colaso, V. Puente, and J.-A. Gregorio, “Topaz: An open-source interconnection network simulator for chip multiprocessors and supercomputers,” in 2012 IEEE/ACM Sixth International Symposium on Networks-on-Chip, 2012, pp. 99–106. [37] G. F. Riley and T. R. Henderson, The ns-3 Network Simulator. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 15–34. [Online]. Available: https://doi.org/10.1007/978-3-642-12331-3_2 [38] A. F. Rodrigues, K. S. Hemmert, B. W. Barrett, C. Kersey, R. Oldfield, M. Weston, R. Risen, J. Cook, P. Rosenfeld, E. Cooper-Balis, and B. Jacob, “The structural simulation toolkit,” SIGMETRICS Perform. Eval. Rev., vol. 38, no. 4, p. 37–42, Mar. 2011. [Online]. Available: https://doi.org/10.1145/1964218.1964225 [39] J. Vejražka, Z. Csaba, and A. Varga, “The INET Framework,” in Proceedings of the 6th International ICST Conference on Simulation Tools and Techniques (SIMUTOOLS ’13). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2013, pp. 1–10. [40] N. Tampouratzis, I. Papaefstathiou, G. Gomez-Lopez, M. S. de la Rosa, J. Escudero-Sahuquillo, and P. J. García, “Distributed fast and accurate simulation platform for advanced ARM- and risc-v-based HPC systems,” J. Supercomput., vol. 81, no. 16, p. 1484, 2025. [Online]. Available: https://doi.org/10.1007/s11227-025-07972-7

Manuscript submitted to ACM

Record · ID 31230 · SHA-256 c645cf46c80df86f
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.