Exploiting the Interplay of Compute- and Memory-Bound kernels in MPI Applications

arXiv:2610.01587v1 [cs.DC] 1 Oct 2026

Exploiting the Interplay of Compute- and Memory-Bound kernels in MPI Applications Ayesha Afzal

Krishna Manda

Georg Hager

Erlangen National High Performance Computing Center (NHR@FAU) Erlangen, Germany ORCID: 0000-0001-5061-0438 Email: ayesha.afzal@fau.de

University of Bonn Germany ORCID: 0000-0002-9744-3080 Email: kmanda@uni-bonn.de

Erlangen National High Performance Computing Center (NHR@FAU) Erlangen, Germany ORCID: 0000-0002-8723-2781 Email: georg.hager@fau.de

Abstract—Parallel applications are often designed for synchronous, lock-step execution, treating communication stalls as performance hazards. Yet, in a communication-light application without frequent synchronization points that alternates between compute-bound memory-bound execution, an MPI communication stall can act as an unintentional relief on memorybandwidth contention. We demonstrate this using a Parallel Optical Flow Solver, which combines a compute-bound Ray Tracing kernel with a memory-bound Optical Flow Solver kernel and negligible inter-process communication. This program shows considerable speedup via desynchronization and automatic overlap between compute- and memory-bound phases, showing that natural desynchronization is an architecture-aware optimization. An optimal speedup is achieved when the number of processes concurrently executing the memory-bound phase on a ccNUMA domain is near the bandwidth saturation point. We also show a case where reducing communication overhead using MPI asynchronous progress significantly degrades performance because it allows too many ranks to contend for memory bandwidth simultaneously. In order to study the dynamics under more controlled conditions, we develop a tunable dual-kernel microbenchmark, with which we show that significant application or system noise (natural or injected) is required to achieve full desynchronization. Finally, we also validate these results using a bandwidth-aware, model-based simulator. Index Terms—MPI Parallel Applications, Desynchronization, Memory-/compute-bound, Performance Modeling, Simulation

I. I NTRODUCTION AND RELATED WORK High-Performance Computing (HPC) applications are often designed with synchronous, lock-step execution, alternating between computation and communication phases. On real systems, this ideal is rarely achieved: Workload imbalance, system noise, and bottlenecks cause processes to diverge, a behavior known as desynchronization [1]. It is usually considered a performance hazard, since a stalled MPI process means idle hardware, and the standard engineering response is to remove whatever is causing the stall. This paper shows a case where this assumption can fail. In a communicationlight multi-kernel application combining compute-bound and memory-bound kernels, an MPI communication stall can act as an unintentional relief on memory-bandwidth contention; removing it by any means can reduce throughput. This behavior arises because compute-bound and memory-bound kernels scale fundamentally differently: Compute-bound phases con-

tinue to benefit from additional cores, whereas memory-bound phases quickly saturate the shared bandwidth on a ccNUMA domain. Their interaction can therefore create spontaneous desynchronization even when communication is negligible. In addition, desynchronization can provide a performance boost by preventing excessive simultaneous demand on shared bottlenecks [2]. Related Work: Prior research on desynchronization has focused mainly on communication-heavy workloads, where synchronization delays propagate as idle waves [3]–[5] and analytic models describe their evolution [1]. Studies of largescale applications have also analyzed synchronization inefficiencies and wait states [6], while resilience techniques for extreme-scale stencil applications address execution disruptions through local recovery and failure masking [7]. Asynchronous update methods further explore relaxing lock-step execution constraints to improve scalability [8] Related performance models include the Execution-Cache-Memory (ECM) model [9], the Roofline model [10], and communication-cost models such as Hockney’s model [11]. These models support the analysis on real hardware or simulation via DisCostiC [12], [13], but they primarily target single-kernel or communicationdriven behavior. In memory-bound workloads, prior work introduced the concept of saturation cores, showing that once too many cores compete for shared memory bandwidth, performance degrades regardless of available compute capacity, and that desynchronization can emerge naturally from bandwidth variation across ccNUMA domains [14]. Other studies investigate communication-computation overlap using non-blocking communication [15], [16], but communication remains the primary source of desynchronization. Research Gap: This phenomenon extends beyond singlekernel solvers. Many HPC and simulation pipelines combine compute-intensive and memory-intensive phases, such as rendering integrated with physics simulations or machine learning preprocessing and post-processing around memory-bound inference. These heterogeneous kernels exhibit fundamentally different scaling behavior, yet existing desynchronization studies largely attribute performance effects to communication rather than kernel interaction. Unlike the communicationheavy halo-exchange workloads dominating prior desynchro-

nization studies, these applications may communicate little between processes but can exploit controlled desynchronization as a performance lever. This work addresses the missing case of desynchronization driven by heterogeneous kernel interaction rather than communication or bandwidth variance alone. Specifically, we ask whether kernel heterogeneity itself can become a beneficial source of desynchronization, and under which combinations of memory-to-compute workload fraction and memorybandwidth saturation, and communication behavior this effect improves application throughput. While prior work demonstrated that desynchronization effects can be induced and exploited [2], [15], we show that the same bandwidth threshold governs desynchronization in communication-negligible applications where kernel heterogeneity is the driver. An incidental or deliberate stall can keep execution near the memorybandwidth saturation threshold, turning a phenomenon traditionally viewed as a performance hazard into an optimization opportunity. Contributions: We study desynchronization in multikernel applications: the MPI Parallel Optical Flow Solver and a tunable, controlled Pisolver-Triad benchmark. Each combines a compute-bound kernel (Ray Tracing or Pi Solver) with a memory-bound kernel (Optical Flow or STREAM Triad) and negligible inter-process communication. Concretely, we: quantify the performance impact of naturally occurring desynchronization relative to a forced lock-step baseline, and identify the memory-to-compute workload fraction that maximizes this gain; • show that eliminating MPI communication stalls via asynchronous progress can degrade performance, by removing an incidental reliefing effect that had been reducing memory-bandwidth contention; • analyze the impact of problem size on L3 bandwidth utilization to identify the best sustain desynchronization; • validate the real hardware traces against simulated traces using DisCostiC to characterize where and why the idealized simulation diverges from real hardware behavior.

•

These findings are distilled into a simple design rule: performance is maximized when concurrent memory-bound ranks remain near the hardware’s measured bandwidth-saturation threshold, whether regulated by workload ratio, communication stalls, or deliberate noise injection. Open-source Data Source: All data required to reproduce this work are available in our performance artifact repository1 . The repository is organized by figure numbers, with each directory containing the corresponding raw results, key numeric data used for plotting, runtime logs, and supporting scripts required for reproduction and figure generation. The artifact also includes input files and documentation of the hardware and software environments used for the experiments, enabling fully reproducible results. 1 https://go-nhr.de/Multi-Kernel-MPI-Analysis

II. M ETHODOLOGY AND A PPLICATIONS This section derives the theoretical upper bound on desynchronization speedup, experimental applications, and methodology used to isolate kernel-driven desynchronization from other performance effects. A. Theoretical Upper Bound on Desynchronized Speedup t t Let Pasync and Psync be maximum total performance under natural desynchronization and forced synchronization (bart rier overhead excluded from Psync ). The desynchronization speedup [17] is:

Sd =

t t − Psync Pasync × 100 [%], t Psync

(1)

which isolates the gain from desynchronization itself, excluding effects such as reduced communication. Sd = 0% t t means no gain, Sd < 0% a loss, and Pasync /Psync ≈ 1+Sd /100. In the dual-kernel program communication is negligible (tcomm → 0); each iteration has compute-bound time tCB and memory-bound time tM B , with tsync = tCB + tM B and memory-to-compute fraction as: ρ=

tM B tM B + tCB

∈ [0, 1],

(2)

reported as a percentage (e.g., ρ = 0.55 → 55%). Desynchronization overlaps kernels, reducing the effective memorybound time to tM B′ ≤ tM B while tCB is unchanged. The ideal per-iteration time is therefore max(tM B′ , tCB ), minimized when tM B′ = tCB . Substituting tM B = ρtsync and tCB = (1 − ρ)tsync gives Sdmax (ρ) = 100 ·

ρ [%], 1−ρ

(3)

valid only while tM B′ can shrink to match tCB without limit. In practice tM B′ is bounded: within the bandwidth-saturation region (c ∈ [Csat , N ] active processes per ccNUMA domain), aggregate bandwidth is roughly constant, so tM B′ (c) = tM B · B MB c/N , giving a floor tM · Csat /N at
c = Csat . min = t CB The ideal time becomes max t , tM B Csat /N , with Eq. 3 holding until this floor dominates, at ρ = ρ∗ = N/(N +Csat ). Combining both regimes:  ρ  ρ ≤ ρ∗ 100 · 1 − ρ , max   Sd (ρ) = N  100 · − 1 , ρ > ρ∗ ρ Csat

(4)

continuous at ρ∗ , where both branches equal 100·N/Csat . For our platform (N = 18, Csat = 10; Figs. 2b, 6c), ρ∗ ≈ 64%, and the measured best case ρ = 55% (< ρ∗ ) has a theoretical desync speedup limit – consistent with the observed behavior of speedup peaking near the best-case region and declining toward both the compute- and memory-heavy extremes.

B. Parallel Optical Flow Solver

TABLE I P ISOLVER -T RIAD TUNING PARAMETERS ( SINGLE - NODE )

2

The Parallel Optical Flow Solver is an MPI application that synthesizes images via ray tracing and then computes their optical flow fields. Each MPI process independently performs a configurable number of repetitions; within each repetition, it alternates between the compute-bound Ray Tracing (RT) kernel and the memory-bound Optical Flow (OF) kernel across several sub-steps. Because the two kernels place demands on different hardware resources and communication between MPI processes is minimal, natural imbalances in kernel execution time cause processes to drift out of lock-step as execution proceeds. Two application parameters let us tune the balance between the two kernels. The OF image size is governed by the multigrid level m, with side length 2m − 1; a higher multigrid level increases the memory-bound OF workload. The RT input size is the OF image size scaled by a supersampling factor s, RT i.e., nRT = nOF = nOF x x ·s and ny y ·s; a higher supersampling factor increases the compute-bound RT workload. Sweeping these two parameters lets us move the application’s execution profile continuously from compute-bound to memory-bound. Each MPI process computes the same global image sequence, but ranks are staggered: rank r begins at frame r · dt · numTimeSteps rather than frame zero, so that at steady state all ranks are working on different, evenly-spaced frames of the same sequence. Because sphere placement in the synthesized scene follows a deliberately non-uniform (“hourglass”shaped) spatial distribution, different frames contain different numbers of spheres, which is a secondary, load-imbalancedriven source of inter-process timing variation, distinct from the kernel-driven desynchronization that is the focus of this paper. Below, we describe how this secondary effect is neutralized, allowing the two sources of inter-process variation to be studied independently. Isolating Kernel-Driven Effects in Flow solver from Load Imbalance: Before attributing performance changes to kerneldriven desynchronization, we first isolate and neutralize two independent sources of initial load imbalance in the application: the non-uniform, hourglass-shaped spatial distribution of spheres across frames (controlled by sinPrefactor), and residual differences between the staggered starting frames of different ranks (controlled by timestep size dt). We quantify per-rank workload imbalance via the total number of doubleprecision floating-point operations (FLOPs) after a fixed number of iterations, X FLOPs = i Fi , (5) i∈{1,2,4,8}

where Fi denotes the number of double-precision floatingpoint instructions operating on i values simultaneously (scalar, 128-bit, 256-bit, and 512-bit SIMD instructions, respectively). The weighting factor i accounts for the number of doubleprecision values processed per instruction. The instruction counts are obtained using likwid-perfctr. Box plots 2 https://github.com/SebastianKuckuk/ray-tracing-optical-flow

Cases

Pi calculation steps

Triad array elements

6.12 × 109 6.8 × 108 3.6 × 109

2.2 × 108 2.0 × 109 1.36 × 109

Compute heavy: ρ = 18% Memory heavy: ρ = 91% Best case: ρ = 55%

of per-rank Flop counts across 100 repetitions show that variability shrinks monotonically as dt is reduced, confirming that a sufficiently small dt (we use 10−5 ) yields near-identical per-process frame data and hence temporal load balance; setting sinPrefactor = 0 removes the spatial imbalance directly. With both parameters set accordingly, an ITAC trace of the first iteration shows a lock-step startup across all MPI processes. All results use this mitigated configuration, so that any performance change observed thereafter is attributable to kernel-driven desynchronization rather than residual workload imbalance. C. Pisolver-Triad Benchmark Because desynchronization in the flow solver application sets in within the first few iterations, it is difficult to study its onset and progression directly. We therefore use the MPIparallel Pisolver-Triad benchmark3 , a lightweight benchmark that reproduces the same dual-kernel structure: a computebound π-calculation kernel (Pisolver) and a memory-bound STREAM Triad kernel. Each iteration follows the same interleaving as the application’s RT/OF sub-steps: a Pisolver step, a non-blocking MPI_Isend to the successor rank, three further interleaved Pisolver–Triad sub-steps, a blocking MPI_Recv from the predecessor, and a final Triad step. As in the application, communication is periodic and lightweight: each rank sends to its predecessor and receives from its successor, with the time spent in communication is matched to the application’s best-case configuration (using a message size of (200 KB). Table I lists the tuned kernel parameters (per-node; scaled linearly by node count for multi-node runs) for the three configurations used throughout this paper. The relative weight and array sizes of each kernel, as well as artificial noise, can be injected at specified ranks, timesteps, and ccNUMA domains – control that is not available in the full application. III. E XPERIMENTAL S ETUP This section presents the hardware and software environment, measurement methodology, and simulation framework used throughout the evaluation. A. Testbed and Experimental Methodology Experiments were run on the Fritz and Helma3 . Table II summarizes the hardware. On Fritz, five nodes (360 MPI processes) were used for multi-node experiments, at 3 Fritz (https://doc.nhr.fau.de/clusters/fritz) (https://doc.nhr.fau.de/clusters/helma)

and

Helma

TABLE II H ARDWARE CONFIGURATION OF F RITZ AND H ELMA . Component

Fritz3

Helma3

Processor Microarchitecture Cores/node NUMA domains/node L3 cache (shared/socket) Memory / node Theor. MEM BW/socket Interconnect

Intel Xeon Platinum 8360Y Ice Lake 72 (dual-socket) 4 54 MiB 256 GiB, 8×DDR4-3200 204.8 GB/s HDR100 InfiniBand, fat tree

AMD EPYC 9965 Zen5c 384 (dual-socket) 8 384 MiB 768 GiB, 24×DDR5-4800 460.8 GB/s NDR InfiniBand

a fixed clock frequency of 2.4 GHz with Hyper-Threading disabled. Experiments on Helma used a single node with 384 MPI processes. The application, benchmark, and simulator were built with Intel oneAPI (version 2025.2) and Intel MPI (version 2021.16) at -O3 -xHost (Cluster A) and -O3 -mavx2 -mfma (Cluster B). Execution traces were collected with the Intel Trace Analyzer and Collector (ITAC) [18] and Vampir [19]; memory bandwidth and floating-point rates were measured with LIKWID’s likwid-perfctr [20], using the MEM_DP hardwareevent group to compute per-process floating-point operation counts (Eq. 5 in Section II-B). Intel MPI asynchronous progress was enabled using I_MPI_ASYNC_PROGRESS=1 and I_MPI_ASYNC_PROGRESS_THREADS=1 with one progress thread per MPI process. Because MPI asynchronous progress is incompatible with ITAC, the real-side comparison trace in this experiment was instead captured via manual high-resolution timestamping (JSON, visualized with chrome://tracing) with asynchronous progress enabled. High-resolution timings were captured with std::chrono around the kernels of interest, after discarding warm-up iterations (10 for the application, 80 for the benchmark, followed by an MPI_Barrier) to avoid first-call overhead. Variations characterization used 100 repetitions per timestep-size configuration and reported as a full distribution, not a point estimate. Benchmark experiments ran for 10,000 iterations. Barrier overhead itself (8.03 µs for our 5-node configuration, measured with the Intel MPI Benchmarks) is subtracted from forced-synchronous timings so reported speedups reflect only the effect of desynchronization, not instrumentation cost. We report single representative runs per configuration, selected after confirming qualitative consistency across preliminary trials. For process timeline plots, during execution, LIKWID monitored all cores on each node using the MEM_DP event group with 200 ms sampling intervals to capture memory bandwidth and floating-point activity. B. DisCostiC Simulation Framework DisCostiC (Distributed Cost in Clusters) [12], [13] is a DSEL-based simulation framework that predicts MPI application performance using first-principles analytic models rather than executing application code. Application behavior is expressed in a Domain-Specific Embedded Language (DSEL) that mirrors the structure of the original MPI program,

the same sequence of compute kernels and point-to-point communication calls, so a trace can be generated without re-executing the application itself. Per-kernel runtimes in DisCostiC are estimated with the Execution-Cache-Memory (ECM) model [9], which predicts a kernel’s cycles-per-cacheline cost as the maximum of the in-core execution time and the non-overlapping data-transfer time through the cache and memory hierarchy: TECM = max(TOL , TnOL +Tdata ). For the compute-bound Pisolver kernel, each divide operation costs ≈ 4 cy/iteration; with 8 double-precision values per 64-byte cache line, TOL = 32 cy/CL and all data-transfer terms are negligible, giving 1.667 ns/iteration at 2.4 GHz. For the memory-bound Triad kernel, we derive the data-transfer time per cache line by summing the non-overlapping in-core cost and the transfer times across each cache level: Tdata = TnOL + TL1L2 + TL2L3 + TL3MEM ≈ 22.5 cy/CL, (6) with TL3MEM ≈ 5.5 cy/CL derived from a measured attainable bandwidth of 111.69 GB/s, consistent with Kerncrafttool estimates [21] and measured timings on Fritz. We scale this cost by the number of cores concurrently active in a ccNUMA domain to capture bandwidth contention – the same saturation effect responsible for the empirically observed 7core threshold (Fig. 1b and Fig. 6b in later Sections). For the OF kernel, whose per-iteration cost varies with the recursive multigrid problem size, we fit a logistic curve to execution time vs. problem size and calibrate its scale to match the measured best-case runtime, then reuse the fitted ECM parameters across problem sizes. We support two-kernel modeling within DisCostiC: each rank’s per-iteration runtime is composed from independent ECM estimates for both kernels, combined with the bandwidth-contention model, so a rank’s simulated progress depends on how many other ranks are simultaneously executing the memory-bound phase – precisely the coupling that drives desynchronization in the real application. This lets us reproduce and study desynchronization dynamics for both the benchmark and the full application under fully controlled, noise-free conditions. IV. R ESULTS : O PTICAL F LOW S OLVER We identified the OF kernel’s bandwidth-saturation using likwid-perfctr measurements within a single ccNUMA domain (multigrid level 11, supersampling factor 2, chosen such that the working set exceeds cache capacity; dt = 10−5 , sinPrefactor = 0), defining the saturation core count as the point at which throughput reaches 95% of its maximum. Fig. 1(a, b) shows RT scaling consistently with core count (compute-bound) while OF throughput saturates at 7 active cores per ccNUMA domain, sustaining ≈ 80 GB/s, confirming OF’s memory-bound classification. With load imbalance mitigated, execution begins in a synchronized, lock-step state; as execution proceeds, this synchrony erodes naturally because RT and OF have different bottlenecks. We compare this naturally desynchronized run against a forced-synchronous variant with a single MPI_Barrier inserted at the midpoint of each iteration after two RT/OF sub-steps (a barrier

(a) Performance: OF and RT kernels

(b) Memory bandwidth: OF kernel

(c) Performance: full application

Fig. 1. (a) RT and OF kernel performance (GFlop/s) versus active cores within a ccNUMA domain; RT scales linearly, while OF saturates. (b) OF memory bandwidth versus active cores, saturating at ≈80 GB/s beyond 7 cores. (c) Mean per-process performance timeline for the naturally desynchronized run on five Intel Ice Lake nodes; shaded region shows the min–max range across processes. Performance increases with desynchronization and stabilizes at Pasync .

(a) Memory-to-compute fraction ρ

(b) Associated desync speedup Sd

Fig. 2. (a) Heatmap of the memory-to-compute fraction ρ (Eq. 2 for the flow solver under lock-step execution across multigrid levels and supersampling factors. (b) Corresponding normalized desynchronization speedup (Eq. 1) from end-to-end runtime (maximum MPI process time) on five Intel Ice Lake nodes; peak speedup (≈ 46%) occurs near ρ ≈ 55%.

after full-iteration is infeasible, as it deadlocks against the application’s four-sub-step structure). Fig. 1(c) shows mean per-process performance (Flops/s) over time for the default run, with a shaded band showing the spread across MPI processes. Performance is low in the first iterations, while processes are still tightly coupled and OF has not yet benefited from reduced bandwidth contention. As desynchronization propagates globally, fewer processes execute OF concurrently, each obtaining a larger share of memory bandwidth, and performance rises to a sustained plateau. In the final iterations, only the few remaining lagging processes still run OF, each with near-exclusive bandwidth access. Overall, the naturally desynchronized execution achieves 278.5 GF/s, compared with 221.7 GF/s for the forced-synchronous baseline, providing a ≈ 25.5% performance improvement due to desynchronization.

A. Effect of the memory-to-compute fraction Sweeping the multigrid level and supersampling factor lets us sweep ρ from compute-heavy to memory-heavy. Fig. 2a shows ρ as a function of the two parameters; Fig. 2b shows the corresponding normalized desynchronization speedup (Eq. 1, using the first-iteration lock-step performance as the synchronous baseline). The largest speedup, ≈ 46%, occurs near the balanced ρ = 55% configuration (multigrid level 9, supersampling factor 2), which represents our best-case configuration. Fig. 3(a)-(c) compares per-process timeline performance for a compute-heavy (ρ = 18%), a memory-heavy (ρ = 91%) and the best-case (ρ = 55%) configuration. The computeheavy case shows uniform per-process performance with minor iteration-to-iteration variation and little benefit from desynchronization, since OF barely contributes. The memoryheavy case shows wider variability but limited net speedup: too many processes saturate bandwidth simultaneously. The

(a) Compute-heavy: ρ = 18%

(b) Memory-heavy: ρ = 91%

(c) Best case: ρ = 55%

(d) Best case (async progress)

Fig. 3. Per-process performance timelines for the flow solver on five Intel Ice Lake nodes: (a) compute-heavy ρ = 18%, (b) memory-heavy ρ = 91%, and best-case ρ = 55% with (c) disabled and (d) enabled asynchronous progress.

0

0

7

7

MPI ranks

MPI ranks

1

14 21 28

3.68 3.84 4.00 4.16 Time [s] (a) Best case trace

1 the theoretical ceiling (Eq. 3), with the remaining gap due to imperfect process staggering, boundary-rank effects, and residual serialization discussed in Section IV-B. B. The Cost of Removing Communication Stalls

14 21 28 3.60 4.00 4.40 4.80 5.20 Time [s]

(b) Best case trace (async progress)

Fig. 4. ITAC traces over four timesteps for the best-case ρ = 55% of the flow solver on five Intel Ice Lake nodes (a) without and (b) with asynchronous progress (pink: RT, yellow: OF, red: MPI); long MPI_Recv (red) stalls largely disappear with asynchronous progress.

best-case configuration shows the widest per-process spread and the largest net gain, as processes stagger execution and use memory bandwidth more efficiently. To explain why the best-case configuration wins, we observe active cores per ccNUMA domain at the execution midpoint and endpoint in each configuration. In the memory-heavy case, midpoint active-core counts reach 16–18 per domain, more than twice the 7-core saturation threshold, causing sustained over-contention; counts drop only in the final iterations, too late to recover the lost throughput. In the compute-heavy case, active-core counts stay below the threshold throughout, so bandwidth is never the bottleneck, but there is also no memory-bound workload for desynchronization to exploit. The best-case configuration keeps midpoint active-core counts close to the 7-core threshold and drops below it by the final iterations, the only configuration that spends most of its runtime near the saturation boundary from both directions. We note that desynchronization onset is rapid across all configurations, within approximately five iterations in every case, which limits our ability to resolve the transition from synchronized to desynchronized behavior in fine temporal detail. For the best configuration ρ = 55%, the application achieves less than

Trace analysis of the best-case configuration (Fig. 4(a)) revealed that, despite using non-blocking MPI_Isend, several processes still block at MPI_Recv in early iterations – a side effect of the MPI eager/rendezvous protocol switch for messages exceeding the eager threshold, under which a nominally non-blocking send behaves quasi-blockingly while it waits for receiver readiness. These stalls behave like idle waves [5] that propagate into later iterations, in some cases persisting for nearly the duration of an entire OF sub-step. We eliminated them by enabling MPI asynchronous progress, offloading communication to a background thread. At a single timestep, removing the stalls only eliminates idle time as expected. However, over longer execution with a larger timestep size (4), enabling asynchronous progress reduces performance by nearly 80%; see Fig. 3d and 4b). With the stalls present, only a limited subset of processes execute OF at a time, and those processes benefit from higher per-core bandwidth; removing the stalls lets all processes make uniform forward progress, but this increases the number of processes concurrently competing for memory bandwidth, pushing the system well past the 7-core saturation threshold and reducing per-process throughput. In this workload, the natural partial serialization induced by communication stalls was, in effect, an unintentional but beneficial throttling mechanism, the same mechanism responsible for the gains, now removed. C. Cross-Architecture Validation on AMD CPU To evaluate whether the observed desynchronization benefits are specific to the Intel Ice Lake platform, we repeated the best-case experiment on Helma, an AMD EPYC system with a different microarchitecture, memory subsystem, and NUMA topology. Using the same memory-to-compute balance (ρ = 55%), the Optical Flow Solver achieves an 88% performance improvement from natural desynchronization compared with the forced lock-step execution. Fig. 5(a) shows the corresponding per-process performance timeline, where performance in-

1

MPI ranks

47 40 30 20 10 0

(a) Per-process performance

0

0.2

0.4 0.6 Time [s]

0.8

1

(b) Process timeline

Fig. 5. Best-case memory-to-compute fraction (ρ = 55%) on a single AMD CPU node (384 MPI processes) of Helma. (a) Per-process performance timeline. (b) Initial 1 s execution trace of one NUMA domain (48 MPI processes); the full-node trace is available at https://go-nhr.de/AMDnode.

creases as processes gradually desynchronize and memorybound phases overlap less frequently. The execution trace of one NUMA domain (Fig. 5(b)) shows the same qualitative behavior observed on Intel: desynchronization reduces concurrent memory-bandwidth pressure and allows remaining OF kernels to execute with higher throughput. Although the absolute speedup differs from the Intel system due to differences in core count, NUMA organization, and memory-bandwidth characteristics, the same mechanism governs the performance gain. The result demonstrates that beneficial desynchronization is not a hardware- or vendor-specific artifact, but a general effect arising from the interaction between heterogeneous kernel scaling behavior and shared memory-bandwidth saturation. D. Simulation Validation with DisCostiC Using the ECM-derived parameters from Section III-B, we configured the two-kernel DisCostiC simulator to reproduce the full solver flow application, and compared simulated traces against real hardware traces. We replicated the bestcase configuration in DisCostiC using RT/OF kernel timings matched to the real execution, so that divergence reflects system-level rather than computational differences. We enabled asynchronous progress to match the noise-free, stallfree conditions assumed by the simulator. Both simulated and real traces exhibit early desynchronization and comparable iteration completion trends (blue line in Fig. 7). However, the real trace additionally reflects boundary-rank effects: DisCostiC’s DSEL representation assumes a periodic communication topology, whereas the real implementation skips MPI_Recv/MPI_Isend at domain-edge ranks, introducing idle stalls or premature progress that are not captured by the idealized simulation. Despite these differences, DisCostiC reproduces the desynchronization behavior and timing trends of the full application within < 3% difference, validating its use for controlled and repeatable studies of this phenomenon. The results also identify boundary-process modeling as a concrete direction for improving simulation fidelity. V. R ESULTS : P ISOLVER -T RIAD B ENCHMARK We first characterize the benchmark’s unperturbed behavior across the three tuned configurations (Table I) on a single ICX

node for 10,000 iterations. As in the application, Fig. 6(a,b) shows Pi scaling with core count (compute-bound), while Triad saturates at 7 active cores per ccNUMA domain with ≈80 GB/s bandwidth. Unlike the application, the benchmark shows negligible spontaneous desynchronization benefit (at most 5.7%): per-process performance remains nearly constant over 10k iterations (Fig. 6(c)). The sweep of nine memory-tocompute ratios (ρ ≈ 18%–95%) using end-to-end normalized speedup (desynchronized vs. forced-synchronous) shows the same ratio-dependent trend, with a peak of ≈5.7% near ρ ≈ 55%. However, the gain is much smaller than in the full application (5.7% vs. 46%) because persistent synchronization in at least one ccNUMA domain limits the benefit. Trace inspection confirms that one ccNUMA domain remains synchronized throughout execution for every configuration (Fig. 9a), although the affected domain varies across runs. The changing message activity from compute- to memoryheavy cases confirms that the tuning controls the intended workload balance, but additional desynchronization alone does not improve throughput while a synchronized domain remains the bottleneck. This motivates deliberate intervention. A. Periodic, Spatially Distributed Noise Injection We first tested whether a single localized delay could break persistent synchrony by injecting an additional 2-second π calculation at one process. The disturbance propagates through neighbors but is absorbed faster in memory-heavy cases due to bandwidth contention. All configurations resynchronize, showing that isolated delays are insufficient. We then injected distributed noise: at every kth timestep, a random process performs an additional Pisolver iteration. We evaluated three spatial granularities: one perturbed process per fivenode group, per node, and per ccNUMA domain. Grouplevel noise dissipates too quickly, while per-node noise leaves intra-domain synchrony. Only per-ccNUMA-domain injection provides full spatial coverage and removes the persistent synchronized domain. With one perturbed process per ccNUMA domain, noise every fourth timestep improves performance by 20% (Fig. 8a), while every second timestep increases the gain to 25%(Fig. 8b). Fig. 8 shows the corresponding performance timelines: higher injection frequency produces stronger and more sustained desynchronization. Thus, both spatial coverage and injection frequency are required to sustain beneficial desynchronization in workloads that do not naturally desynchronize. B. Simulation Validation with DisCostiC For the unperturbed benchmark (single ICX node, 300 iterations, best-case configuration ρ = 55%), DisCostiC reproduces the baseline execution pattern and message-exchange behavior observed in the real trace (Fig. 9). The main difference lies in the desynchronization onset: the real execution diverges earlier, whereas the noise-free simulation develops divergence gradually due to differences in intra- and inter-NUMA-domain communication latency. Nevertheless, the simulation captures the key behavior of the real system, with at least one ccNUMA

(a) Performance: Pi and Triad kernels

(b) Memory bndwidth: Triad kernel

(c) Desync Speedup: Full benchmark

Fig. 6. PI-Triad bench: (a) Performance (GFlop/s) of the Pi and STREAM Triad kernels vs. active cores within a single ccNUMA domain; Pi scales linearly, STREAM saturates. (b) STREAM memory bandwidth vs. active cores, saturating at ≈80 GB/s beyond 7 cores. (c) Corresponding normalized speedup from desynchronization (end-to-end performance with maximum time among all the MPI processes). Peak speedup (≈5.7%) occurs near ρ ≈ 55%.

71 32 34 36 38 40 42 44 46 Time [s] (a) Real trace

71

(b) Simulated DisCostiC trace

Fig. 7. Execution traces of the Optical Flow Solver for the best-case configuration (29–37 s) on five Intel Ice Lake nodes: (a) real trace with asynchronous progress enabled and (b) simulated DisCostiC trace. One selected iteration completion time is highlighted in blue.

(a) k=4

0 10 20 30 40 50 60 71

29 31 33 35 37 Time [s]

(b) k=2

Fig. 8. Per-process performance timelines for the best-case configuration of Pisolver-Triad Benchmark with one perturbed process per ccNUMA domain at noise injection frequencies (a) k=4 and (b) k=2.

domain remaining internally synchronized, while predicting runtime with only a 0.5% error. C. Best Case: Rapid Desynchronization When Data Fits in L3 Previously, the benchmark showed negligible spontaneous speedup (≤ 5.7%, rather than the 20–63% seen elsewhere)

1

MPI ranks

0 10 20 30 40 50 60

1

MPI ranks

0 10 20 30 40 50 60

MPI ranks

MPI ranks

1

0 10 20 30 40 Time [s] (a) Real trace

55

0 10 20 30 40 50 60 71

0 10 20 30 40 Time [s]

55

(b) Simulated DisCostiC trace

Fig. 9. a) Real and (b) simulated DisCostiC traces for the best-case configuration of Pisolver-Triad Benchmark on two Intel Ice Lake chips.

because at least one ccNUMA domain remains synchronized when the Triad working set resides in main memory, regardless of workload fraction (Fig. 6(c)). We therefore tested whether this persistent synchrony is a property of main-memory accesses rather than cache-resident execution. We fixed the memory-to-compute fraction at the best-case value (ρ ≈ 55%, Table I) and varied only the absolute problem size by scaling the Pisolver iteration count and per-rank Triad array size together, preserving ρ. Eleven configurations span Triad arrays from 2.5 × 106 to 1.28 × 109 elements, covering per-domain L2, L3, and main-memory working sets (Fig. 10). To obtain sufficient runtime, the benchmark was extended to 70,000 iterations. Figure 10 shows that the benefit depends on the shared resource under contention. When the working set fits in the private L2 cache, bandwidth is not shared across cores, so desynchronization provides little benefit. The largest speedup (≈ 70%) occurs when the Triad working set fits in the shared L3 cache (6×106 elements), where desynchronization reduces concurrent L3 bandwidth demand. As the working set grows into main memory, the speedup decreases monotonically to ≈ 5% because memory-bandwidth contention dominates and the persistent synchronized ccNUMA domain re-emerges.

by keeping the number of ranks simultaneously executing the memory-bound phase near the platform’s bandwidth-saturation threshold, whether achieved through workload balance, incidental communication stalls, or deliberate perturbation. A direct test of this principle is to implement a mechanism that enforces this condition and compare its performance with the accidental and injected desynchronization studied here. Future work will extend this analysis to larger and more complex applications with multiple interacting kernels, such as climate simulations, and generalize DisCostiC to support arbitrary multi-kernel workloads while improving trace fidelity. We will also investigate energy-to-solution, using per-node power measurements such as LIKWID’s RAPL-based energy counters, to determine whether desynchronization-based runtime improvements translate into energy savings. ACKNOWLEDGMENTS

Fig. 10. Natural desynchronization speedup for the best-case workload fraction (ρ ≈ 55%) as the per-process Triad array size spans L2-, L3, and main-memory-resident working sets on five ICX nodes over 70,000 iterations. For comparison, figures 6 and 9 use a main-memory-resident array (1.36 × 109 elements), yielding a 5.7% speedup. Here, n denotes the number of Pi calculation steps.

Thus, the benchmark again requires deliberate noise injection to sustain desynchronization. VI. C ONCLUSION AND F UTURE W ORK This work studied desynchronization in a regime largely unexplored by prior work: negligible inter-process communication, with desynchronization instead driven by the interaction between compute-bound and memory-bound kernels within a single application. Using the Parallel Optical Flow Solver, we showed that natural desynchronization improves performance by 25.5% over a forced-synchronous baseline, increasing to 46% for a near-balanced 55% memory-to-compute fraction. We further showed that removing MPI communication stalls through asynchronous progress can reduce performance by 80%, because these stalls unintentionally limit memorybandwidth contention. In contrast, the Pisolver-Triad benchmark does not naturally desynchronize under main-memorybound execution and requires periodic ccNUMA-domaintargeted idle-wave injection to sustain desynchronization, achieving up to 25% speedup. However, when the working set fits in shared L3 cache, the benchmark desynchronizes naturally and achieves substantially higher speedup, showing that bottleneck characteristics, in addition to multi-kernel structure, determine desynchronization behavior. Using DisCostiC, we reproduced the qualitative desynchronization trends of both the benchmark and the full application, while identifying boundary-rank modeling and secondary idle-wave generation as key directions for improving simulation fidelity. These results lead to a simple design rule: maximize performance

The authors gratefully acknowledge support from the German Research Foundation (DFG), project number 545776403, FOR 5880: Holistic Energy and Performance Modeling for Sustainable Computing (Mod4Comp). The authors acknowledge the HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) at FAU Erlangen-Nürnberg. NHR funding is provided by the German Federal Ministry of Education and Research and the state governments participating on the basis of the resolutions of the GWK for the national high-performance computing at universities by federal and Bavarian state authorities. The NHR@FAU hardware is partially funded by the German Research Foundation (DFG), grant number 440719683. R EFERENCES [1] A. Afzal, G. Hager, and G. Wellein, “Analytic modeling of idle waves in parallel programs: communication, cluster topology, and noise impact,” in Proceedings of the International Conference on High Performance Computing (ISC High Performance 2021). Springer, 2021, pp. 351–371. [Online]. Available: https://doi.org/10.1007/978-3-030-78713-4 19 [2] ——, “The role of idle waves, desynchronization, and bottleneck evasion in the performance of parallel programs,” IEEE Transactions on Parallel and Distributed Systems (TPDS), vol. 34, no. 2, pp. 623–638, 2022. [Online]. Available: https://doi.org/10.1109/TPDS.2022.3221085 [3] ——, “Propagation and decay of injected one-off delays on clusters: a case study,” in 2019 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 2019, pp. 1–10. [Online]. Available: https://doi.org/10.1109/CLUSTER.2019.8890995 [4] N. Losada, A. Bouteiller, and G. Bosilca, “Asynchronous receiverdriven replay for local rollback of MPI applications,” in 2019 IEEE/ACM 9th Workshop on Fault Tolerance for HPC at eXtreme Scale (FTXS). IEEE, 2019, pp. 1–10. [Online]. Available: https: //doi.org/10.1109/FTXS49593.2019.00006 [5] S. Markidis, J. Vencels, I. B. Peng, D. Akhmetova, E. Laure, and P. Henri, “Idle waves in high-performance computing,” Physical Review E, vol. 91, no. 1, p. 013306, 2015. [Online]. Available: https://doi.org/10.1103/PhysRevE.91.013306 [6] D. Böhme, M. Geimer, L. Arnold, F. Voigtlaender, and F. Wolf, “Identifying the root causes of wait states in large-scale parallel applications,” ACM Transactions on Parallel Computing, vol. 3, no. 2, pp. 11:1–11:24, Jul. 2016. [Online]. Available: https: //doi.org/10.1145/2858782

[7] M. Gamell, K. Teranishi, M. A. Heroux, J. Mayo, H. Kolla, J. Chen, and M. Parashar, “Local recovery and failure masking for stencil-based applications at extreme scales,” in SC ’15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2015, pp. 1–12. [Online]. Available: https://doi.org/10.1145/2807591.2807672 [8] A. Kolakowska and M. A. Novotny, “Desynchronization and speedup in an asynchronous conservative parallel update protocol,” arXiv preprint cs/0409032, 2004. [Online]. Available: https://arxiv.org/abs/cs/0409032 [9] J. Treibig and G. Hager, “Introducing a performance model for bandwidth-limited loop kernels,” in International Conference on Parallel Processing and Applied Mathematics (PPAM). Springer, 2009, pp. 615– 624. [Online]. Available: https://doi.org/10.1007/978-3-642-14390-8 64 [10] S. Williams, A. Waterman, and D. Patterson, “Roofline: an insightful visual performance model for multicore architectures,” Communications of the ACM, vol. 52, no. 4, pp. 65–76, 2009. [Online]. Available: https://doi.org/10.1145/1498765.149878 [11] R. W. Hockney, “The communication challenge for MPP: intel paragon and meiko CS-2,” Parallel Computing, vol. 20, no. 3, pp. 389–398, 1994. [Online]. Available: https://doi.org/10.1016/S0167-8191(06)80021-9 [12] A. Afzal, G. Hager, and G. Wellein, “DisCostiC: Simulating mpi applications without executing code,” 2024, poster at the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC24, November 17–22, 2024, Atlanta, GA, USA. [Online]. Available: https://sc24.supercomputing.org/proceedings/poster/ poster pages/post167.html [13] ——, “DisCostiC: Digital twin performance simulations unlocking hardware-software interplay,” 2025, first Price Best Research Poster at the International Conference for ISC High Performance 2025, June 10–13, 2025, Hamburg, Germany. [Online]. Available: https://isc.app.swapcard.com/widget/event/isc-high-performance-2025/ planning/UGxhbm5pbmdfMjU4NDU1Mg== [14] ——, “Desynchronization and wave pattern formation in MPIparallel and hybrid memory-bound programs,” in Proceedings of the International Conference on High Performance Computing (ISC High Performance 2020). Springer, 2020, pp. 391–411. [Online]. Available: https://doi.org/10.1007/978-3-030-50743-5 20 [15] A. Afzal, G. Hager, S. Markidis, and G. Wellein, “Making applications faster by asynchronous execution: slowing down processes or relaxing MPI collectives,” Future Generation Computer Systems (FGCS), vol. 148, pp. 472–487, 2023. [Online]. Available: https: //doi.org/10.1016/j.future.2023.06.017 [16] A. Afzal, G. Hager, G. Wellein, and S. Markidis, “Exploring techniques for the analysis of spontaneous asynchronicity in MPI-parallel applications,” in International Conference on Parallel Processing and Applied Mathematics (PPAM). Springer, 2022, pp. 155–170. [Online]. Available: https://doi.org/10.1007/978-3-031-30442-2 12 [17] A. Afzal, A Holistic White-Box Approach to Performance Modeling for Supercomputing, ser. FAU Studien aus der Informatik, B. Eskofier, R. Lenz, A. Maier, M. Philippsen, L. Schröder, W. Schröder-Preikschat, M. Stamminger, and R. Wanka, Eds. Erlangen: FAU University Press, 2026, vol. 22. [Online]. Available: https://doi.org/10.25593/ 978-3-96147-940-5 [18] C. Iwainsky and D. an Mey, “Comparing the usability of performance analysis tools,” in Euro-Par 2008 Workshops – Parallel Processing, ser. Lecture Notes in Computer Science, vol. 5415. Springer, 2009, pp. 315– 325. [Online]. Available: https://doi.org/10.1007/978-3-642-00955-6 36 [19] A. Knüpfer et al., “Score-p: A joint performance measurement run-time infrastructure for periscope, scalasca, tau, and vampir,” in Tools for High Performance Computing 2011. Springer, 2012, pp. 79–91. [Online]. Available: https://doi.org/10.1007/978-3-642-31476-6 7 [20] J. Treibig, G. Hager, and G. Wellein, “LIKWID: a lightweight performance-oriented tool suite for x86 multicore environments,” in 2010 39th International Conference on Parallel Processing Workshops. IEEE, 2010, pp. 207–216. [Online]. Available: https: //doi.org/10.1109/ICPPW.2010.38 [21] J. Hammer, J. Eitzinger, G. Hager, and G. Wellein, “Kerncraft: a tool for analytic performance modeling of loop kernels,” in Tools for High Performance Computing 2016. Springer, 2017, pp. 1–22. [Online]. Available: https://doi.org/10.1007/978-3-319-56702-0 1

Related documents

• Exploiting the Interplay of Compute- and Memory-Bound kernels in MPI Applications#1156858
• Using the Transformational Approach to Build a Safe and Generic Data Synchronizer#175892
• CVE-2026-77762 — HIGH 8.1#1080982
• FPGA Acceleration of Fully Homomorphic Encryption with Adaptive Key Switching#987172
• Mixed-Precision Communication-Avoiding SGD for Generalized Linear Models on GPUs#287105