Efficient and Portable Support for Overdecomposition on Distributed Memory GPGPU Platforms Aditya Bhosale∗ , Anant Jain† , Shourya Goel† , Ritvik Rao∗ , Peddoju Sateesh Kumar† , Laxmikant Kale∗ ∗ University of Illinois Urbana-Champaign, Urbana, IL, USA
Emails: {adityapb, rsrao2, kale}@illinois.edu † Indian Institute of Technology, Roorkee, Roorkee, India
arXiv:2605.12734v1 [cs.DC] 12 May 2026
Emails: {anant j, shourya g, sateesh}@cs.iitr.ac.in
Abstract—Overdecomposition has emerged as a powerful and sometimes essential technique in parallel programming. Many application domains or frameworks, including those based on adaptive mesh refinements, or tree codes use it. Charm++ is a parallel programming system which has demonstrated the utility of overdecomposition for many applications and in multiple contexts. However, the emergence of GPGPUs as a dominant compute component has created some real and perceived challenges for this paradigm, especially regarding the higher overhead brought about by overpartitioning — having multiple objects assigned to the same GPGPU device. We address this issue as well as the issue of portability by developing techniques and software that demonstrate that overdecomposition can be efficiently and productively supported on combinations of GPU vendor types, and interconnection networks. Index Terms—performance, portability, GPGPUs, overdecomposition, Kokkos, Charm++
I. I NTRODUCTION Modern HPC has entered an era of unprecedented architectural diversity. The leading systems on the TOP500 list are distributed across at least three GPU vendors: Frontier and El Capitan are powered by AMD accelerators, Aurora by Intel, and Jupiter Booster, Eagle, and Alps by NVIDIA GPUs. Each vendor ships its own native programming model, HIP, SYCL, and CUDA respectively, and each uses a distinct network stack. For application developers and runtime system designers, this diversity has turned portability from a convenience to a first-class requirement. Real applications increasingly exhibit dynamic behavior, for example, adaptive mesh refinement, multi-physics simulations, and irregular algorithms whose loads are data dependent and hard to statically predict. Hardware contributes another source of imbalance, from performance variability between identical GPUs, to thermal throttling, and network contention. The bulk-synchronous model that dominates large-scale GPU codes offers no inherent mechanism for tolerating this dynamism. Asynchronous Many Task (AMT) runtimes such as Charm++ [1], [9], HPX [8], Legion [2], etc. address this gap by decoupling the application’s expression of parallelism from its hardware mapping.
Charm++ addresses adaptivity through overdecomposition, ie. dividing the computation into many more migratable objects (chares) than processing elements (PEs), and letting the runtime schedule, overlap, and migrate them with minimal application intervention. These benefits are well-established on CPU-based systems and have been validated at scale in production codes such as NAMD [10] and ChaNGa [7]. However, extending the same model to GPUs has a fundamental challenge: GPUs achieve peak efficiency with large, highoccupancy kernels and bulk data transfers, while overdecomposition deliberately partitions work into many smaller chunks. The concern is that the resulting fine granularity will manifest as reduced occupancy, increased kernel launch overheads, and synchronization costs that will together outweigh the benefits of overdecomposition. Previous work from Choi et al. [4] quantified these costs for GPU-aware Charm++ applications on NVIDIA Summit, using a single-PE-per-GPU configuration, and proposed applicationlevel mitigations such as kernel fusion to coalesce fine-grained packing/unpacking kernels with the computation kernel, and CUDA Graphs to amortize kernel launch overhead for iterative applications. While these techniques are effective, they require the application developer to restructure code around the runtime system’s granularity. Their study was also limited to a single GPU vendor and a single communication stack. Whether overdecomposed GPU execution can be supported portably from a single source across multiple GPU vendors and communication stacks, whether its overhead can be managed transparently at the runtime layer instead of the application layer, and how it scales with the overdecomposition factor on modern accelerators remain open questions. In this paper we take a step towards answering them. We extend Charm++ with a portable GPU communication layer built on the Lightweight Communication Interface (LCI) [11] that operates over both ibverbs and libfabric, paired with Kokkos [5] for execution portability. The result is a singlesource Charm++/Kokkos environment in which an overdecomposed application runs unchanged on NVIDIA and AMD GPUs.
Within this environment we develop two application agnostic improvements aimed at managing the cost of fine-grained execution on GPUs. The first uses non-blocking GPU streams so that each chare’s fine-grained packing/unpacking kernels overlap with the computation kernels of other chares mapped to the same device. The second generalizes the runtime’s execution model into a hierarchy in which multiple processes are mapped to each GPU and multiple worker threads (PEs) are mapped to each process. Mapping multiple PEs to a shared GPU allows kernels to be submitted from these PEs in parallel, mitigating the launch side serialization that a single-PE design incurs as the overdecomposition factor grows. Both techniques are implemented entirely in the runtime requiring no application level restructuring, and complement the application-level techniques explored in prior work. On top of this foundation, we conduct a detailed empirical study of overdecomposition cost, measuring how the application overhead scales with the overdecomposition factor across vendor platforms, and characterizing the trade-offs across execution model configurations. This paper makes the following contributions: 1) A portable GPU runtime for Charm++ built on LCI and Kokkos, supporting NVIDIA and AMD GPUs over both libverbs and libfabric from a single source. 2) Two application-agnostic, runtime-level techniques for managing the cost of overdecomposition on GPUs: nonblocking streams that overlap each chare’s pack/unpack with other chares’ computation, and a hierarchical execution model with multiple processes per GPU and multiple worker threads per process that enables parallel kernel submission to a shared device. 3) A detailed empirical study of overdecomposition cost on AMD and NVIDIA GPUs, attributing per-chare overhead to its sources, characterizing how it scales with the overdecomposition factor, and analyzing the trade-offs across execution model configurations. II. BACKGROUND AND R ELATED W ORK A. Performance portability frameworks Kokkos [5], RAJA [3], and SYCL provide single-source abstractions that compile to vendor-native backends, allowing parallel kernels to target CPUs and GPUs from any of the major vendors without source modification. These frameworks have been adopted at scale across the DOE exascale machines. By design, they scope themselves to within-node execution and do not address inter-node communication or runtime adaptivity. Our work uses Kokkos for execution portability and complements it with a portable communication layer. B. Asynchronous many-task runtimes on GPUs Several AMT runtimes have been extended to multivendor GPU execution. Uintah [6] has been ported to AMD, NVIDIA, and Intel GPUs through Kokkos backends, demonstrating single-source portability for an asynchronous manytask framework across all three GPU vendors.
PE 0
PE 2
PE 4
PE 6
PE 1
PE 3
PE 5
PE 7
GPU 0
GPU 1
PE 8
PE 10
PE 12
PE 14
PE 9
PE 11
PE 13
PE 15
GPU 2
GPU 3
Process
Physical Node
Fig. 1: Execution model for GPU-aware Charm++ applications
C. Charm++ and Overdecomposition The specific AMT we extend here is Charm++, which pioneered the idea of overdecomposition. In charm++, programmer decomposes the work and data into C++ objects called chares, that are organized into one or more multidimensional collections. Chares are assigned to processors by the Charm++ runtime. The application developed does not need to be aware of the location (processor) where a chare lives — A chare communicates with another chare using its collection name and index, sending an asynchronous method invocation towards it. So, a call such as A[i,j].foo(params) immediately returns, while a message containing the params data is send towards the last known location where the (i,j) member of chare collection A lives by the system, where the foo method is eventually scheduled by a user-space queue based scheduler. The ability to migrate chares across processors confers the ability to dynamically balance loads to an introspection (instrumentation) based runtime system. III. E XECUTION M ODEL We organize the runtime into a hierarchy of user-level threads, processes, and GPUs, illustrated in Figure 1. Each GPU is shared by one or more processes, and each process contains one or more worker threads. Chares are scheduled onto PEs by the Charm++ runtime, and each chare issues its GPU kernels to the GPU associated with its process. PEs within the same process share a CUDA or HIP driver context, enabling direct device-to-device copies between chares. Crossprocess GPU messages uses CUDA or HIP IPC on the same node and GPU-aware RDMA across nodes.
Effect of Overdecomposition on Kernel Runtime
A. Managing Overdecomposition Average End-to-End Kernel Runtime (µs)
The idea of overdecomposition means there will be many objects (Chares), each with its own kernels running on each GPU and each PE. How many should there be? In the nonGPU context, Charm++ typically needs about 8-16 chares per PE (a PE is roughly the same as a core. It is a core that has a charm scheduler running on it). This allows the load balancer adequate flexibility in moving some number of chares into or away from a core to restore load balance. Any smaller than that, and one suffers the effects of quantization. (An analogy is distributing rocks to buckets, to keep all the buckets of equal weight. Heavy chunks would hinder the ability to balance buckets). An important question arises when we consider GPUs: should a core still be the right unit of scheduling? Since there are hundred+ cores on modern nodes, it will be an extremely fine-grained computation if we require 8-16 chares per PE. An important insight is that computing load in modern GPU based applications is carried mostly on GPUs, and not on CPU cores. The cores on the host are used more as a coordination unit and to do small unavoidable computations. Thus, we conclude that overdecomposition should be done with respect to GPUs – putting 8-16 chares per GPU device. Our objective, for this paper, then becomes an easier one: can we sustain an adequately fine grainsize to generate 8-16 chares per GPU without undue overhead? What are the obstacles, and what techniques can be developed to overcome them. It has been known that a single kernel latency on modern GPUs is large – of the order of 10-20 microseconds. That may seem like a short time, but especially for overdecomposed kernel, that overhead may be prohibitive. However, we hypothesize that this latency is composed of parts that can be overlapped in various ways. In particular, there may be latency associated with the work done on the host core, and separately in device runtime which is scheduling and initializing each kernel. To assess the potential of such overlap, we construct a microbenchmark to study the effect of overdecomposition on end-to-end kernel runtimes. Here, the work is divided into a variable number of kernels. To isolate GPU runtime overhead, all kernels are enqueued into their respective streams up front and then released simultaneously by having each stream wait on a host-mapped synchronization flag, using cuStreamWaitValue32. The resultant data is compiled in Figure 2 with the total number of threads meaning the number of CUDA threads in CUDA kernel launch. The Results show that overdecomposition, in this highly controlled situation has pretty low overhead. For the workload size as low as 256K threads , the completion times remain broadly similar across configurations, ranging from 388 µs when the work is launched from 1 chare to 402 µs when the work is launched from 64 chares, indicating that overdecomposition introduces relatively little overhead in this controlled setting. For example, the smallest configuration completes in about 108 µs, increasing only to about 120 µs
2500
2355
2355
2355
2359
2361
2364
2366
1232
1231
1232
1232
1234
1237
1237
669
668
666
666
668
672
673
388
387
385
385
387
201
200
199
205
217
389 240
402 307
120
140
181
8
16
32
2000
Total Number of threads launched 64 K threads 128 K threads 256 K threads 512 K threads 1024 K threads 2048 K threads
1500
1000
500
0
108
1
105
2
108
4
Number of Chares
295
64
Fig. 2: Effect of overdecomposition on Kernel runtimes
when the same work is split across 8 kernels, and to 296 µs at 64 kernels. The larger increase at the highest ODFs is likely due to CUDA runtime overheads, including scheduler pressure and limits on kernel concurrency. Submitting kernels ahead of time to streams, and only releasing them on signal, is a synthetic mechanism to demonstrate the potential. In reality, an asynchronous task-based runtime will be submitting kernels at arbitrary times based on availability of data. Further, if there is a component latency borne by the CPU core, it may be better to use multiple CPUs for each GPU. After all, many more cores exist on a node than GPUs. To study this aspect, we ran a new benchmark. This benchmark involves rapidly firing kernels from a single process to a single GPU, while varying the number of threads and the number of chares per thread. The benchmark shows how overdecomposition and multithreading can increase the use of a GPU for fine-grained kernels. In our benchmark, each kernel is a Kokkos kernel that occupies all the SMs/compute units of a device, but otherwise does not actually do any work. When a kernel completes, we use a stream (1 per chare) to trigger a HAPI callback that will fire the next kernel. Fig. 3 shows the number of kernels launched per second with 1-4 chares per PE and 1-4 PEs. From this experiment, there is a clear advantage to using 2 chares per thread as opposed to 1, as the Charm++ scheduler can switch to the next chare and fire a kernel while the first chare waits on a callback. However, using 4 or more chares per thread shows no advantage. This may be because of the overhead of the Charm++ scheduler (running on the host CPUs) is large enough that the first kernel launch will likely have returned by the time a 4th chare’s kernel is ready to launch. B. Using Kokkos in charm++ runtime Kokkos is primarily designed around a synchronous execution model such as MPI, and integrating it effectively with Charm++ requires additional considerations to preserve asynchronous execution and overlap. One source of overhead arises from implicit synchronization in operations such as temporary buffer deallocation and APIs like Kokkos::resize, which internally invoke Kokkos::fence because of Kokkos’s reference-counting-based memory management. In a multithreaded Charm++ execution model, these implicit fences can
Pure Transfer
Kernel Launch Rate vs. Chares per Thread 250000
ppn 1 ppn 2 ppn 4
225000
104 175000
Iteration Time ( s)
Launches per Second
200000
150000 125000 100000
47942 Message Size 512 KB 1024 KB 239802048 KB 4096 KB 8192 KB 16384 KB 12000 32768 KB 65536 KB 131072 KB
47945
47974
48023
46768
46435
46622
47280
23983
24011
24061
23492
23392
23686
24385
12004
12029
12078
11848
11899
12243
13034
6012
6019
6037
6087
6022
6139
6532
3014
3016
3042
3080
3084
3283
4411
1520
1525
1546
1553
767
776
785
822
397
400
416
212
214
1
2
1640
916 570
1
2
Chares per Thread
4
Fig. 3: Rate of Kokkos kernel launches on A40 vs. number of chares per thread
1889
461
3668 3009 2251 1570
1171
103
75000
7199
2317 2011 1748
1235 1023
841 647
395
289 236
4
8
ODF
16
32
64
128
Fig. 4: Effect of overdecomposition on communication unnecessarily stall unrelated chares. To avoid such synchronization points, we make frequently used Kokkos::Views persistent members of the class implementation and obtain scratch space from preallocated buffers instead of dynamically allocating temporary buffers during execution. Another challenge stems from Kokkos’s functor caching behavior, which can introduce additional synchronization across kernel launches. Since our applications do not launch identical kernels in a tightly consecutive manner, the caching mechanism provides almost no benefit while increasing synchronization. We mitigate this behavior using Kokkos’s lightweight hints. To support concurrent execution across multiple streams, we use separate Kokkos execution spaces. In doing so, several considerations become important. For example, Kokkos::deep_copy operations issued without an explicit execution space can introduce host-side synchronization, Kernels issues without an execution Space run on the default stream, which synchronizes implicitly with the rest of the streams. Additionally, initializing an execution space itself is relatively expensive because it involves device initialization and associated memory copies from host to device. To minimize this overhead, execution spaces are created once during initialization and reused throughout the lifetime of the application. C. Performance trade-offs Wider processes, ie. more PEs per process and fewer processes per GPU, keeps more message traffic within a single driver context, where it benefits from direct device-todevice copies. Smaller processes routes more traffic through the IPC mechanism which requires two intermediate copies into pre-allocated buffers on both the sender and receiver. Wider processes, however, incur an overhead due to thread contention on the host for kernel launches. On NVIDIA hardware, multi-process configurations require the Multi-Process Service (MPS) for concurrent kernel execution across processes. Without MPS, kernels from different contexts time-slice. ROCm provides no equivalent
service, so multi-process configurations on AMD hardware time-slice rather than execute concurrently. The benefits of multi-process-per-GPU configurations are therefore vendorasymmetric, while the costs are not. D. Pipelined communication We evaluate GPU-direct communication and computation–communication overlap on 2 A40 GPUs connected at 21 GB/s peak one sided bandwidth. The benchmark transfers a fixed total data volume split across a varying number of sender receiver pairs. ODF is the number of sender-receivers pairs. To reflect realistic application behavior we append a small kernel launch at the end of each receive. The total end-to-end time thus captures both increase in transfer time because of overdecomposition and benefit due computation communication overlap. Figure 4 shows for the data sizes for a typical application, the communication only starts to climb up when the amount of overdecomposition goes beyond 16, which is enough for load balancing to work effectively. Figure 5 shows the effect of adding an O(n) computation kernel at the end of the transfer. Total walltime for most sizes beyond 2048 KB are reduced for some higher degree of overdecomposition depending on the data size. IV. P ORTABLE C OMMUNICATION L AYER When one overdecomposes the work and data units, naturally the amount of data communicated is fragmented into smaller but more numerous messages. For example, in a 2D stencil computation, If you over-decompose by a factor of 4, each message is of half the size as before. To complicate mattes further, some of these messages may go to chares on the same device, some may go to a neighboring device on the same physical node and some may go to another node. To ensure overdecomposition overheads are reduced, all three paths must be individually optimized. The charm++ runtime can identify the destination chare’s location — this requires a runtime option that pre-fills the location table. So it is possible
Transfer + Small Kernel
Iteration Time ( s)
104
51740 Message Size 512 KB 1024 KB 258882048 KB 4096 KB 8192 KB 16384 KB 12955 32768 KB 65536 KB 131072 KB
49859
49254
48668
47244
46644
46686
47265
24945
24681
24361
23717
23511
23674
24350
12486
12423
12235
11951
11936
12182
12994
6491
6260
6213
6162
6087
6128
6484
3253
3144
3144
3129
3114
3293
1641
1589
1591
1595
1677
1809
828
812
809
853
918
Process 0
Csrc
2205
1122
103
596 427
420
430
484
1517 1203 1002
src buffer (GPU)
3009
223
2
596
415 Node A
242
4
8
dst buffer (GPU)
(a) Intra-process GPU communication
783
Process 0
1
D2D memcpy
2325 2009 1769
318 223
Cdst
7177 4389
3584
ctrl message (src)
ODF
16
32
64
Process 1
ctrl message (offset, event_idx)
128 Csrc
Cdst
Fig. 5: Effect of Communication-Computation Overlap src buffer (GPU)
D2D memcpy
IPC comm buffer
dst buffer (GPU)
NVLink
to know which code path to use and send it via the best protocol. A. Transport selection When a chare sends a device zerocopy message, the runtime selects one of three transport paths based on the location of the source and destination chares. The three transport paths are illustrated in Figure 6. 1) Intra-process transfers – When source and destination chares reside on the same process, the transfer is done by a single asynchronous device-to-device copy on the stream specified in the post entry method. We use the native CUDA/HIP APIs for async memory copy to implement this. 2) Intra-node, inter-process transfers – When the source and destination chares are in different processes on the same physical node, the transfer is done using IPC. To avoid the high cost of opening an IPC memory handle for every message, we stage the IPC messages through a communication buffer. Each process preallocates a shared device communication buffer and a pool of cross process synchronization events at startup. At startup, every process also maps every peer’s communication buffer and events into its own address space using POSIX shared memory to exchange handles. The transfer then involves two device-to-device copies source buffer to the source’s communication buffer, and source’s communication buffer to the destination buffer. An IPC event is used on the destination process to wait for the source copy to complete before the destination copy. The chare-level control message contains the offset into the source communication buffer and the index of event pool used for synchronization. 3) Inter-node transfers – For source and destination chares on different nodes, we use one-sided RDMA. The source PE registers the source buffer and sends a control message to the destination PE with the source buffer address. The destination PE then issues a one-sided get
(b) Intra-node, inter-process GPU communication
Node A
register memory
src buffer (GPU)
ctrl message (src, rmr)
Csrc
PCIe
NIC
Node B post RDMA get
NIC
Cdst
PCIe
register memory
dst buffer (GPU)
Network
(c) Inter-node GPU communication
Fig. 6: GPU communication protocols in Charm++
operation that pulls the data from the source GPU into its own. V. E XPERIMENTAL R ESULTS We evaluate weak and strong scaling performance of overdecomposition across three mini-applications—Jacobi2D, MiniMD, and LULESH. We compare our Charm++/Kokkos implementations of these mini-applications against MPI/Kokkos implementations across a range of ODF configurations. The nvidia runs are done on NCSA delta’s A40 GPU compute nodes. Each compute node had 4 nvidia A40 GPUs, connected via PCIe Gen4. Different nodes are connected via HPE/Cray’s Slingshot 11 interconnects. The AMD runs are carried out on frontier. Each compute node on frontier has 4 AMD Instinct MI250X GPUs; each MI250X contains 2 Graphics Compute Die(GCD), so Frontier exposes 8 GCDs per node. The system uses HPE Slingshot networking interfaces. 1) jacobi2d: Jacobi2d is a simple stencil based application that applies the Jacobi iterative method on a 2D grid. The
2
4
8
50.9
50.8
Charm-ODF-2
Charm-ODF-4
Charm-ODF-8
Charm-ODF-16
strong Scaling Performance of Jacobi2D
16
8
16
32
6.7
6.5
6.4
6.7
4
32
6.5
8
6.5
12.9
12.8
12.8
12.7
12.8
25.7 13.0
16
Number of GPUs
64
Number of GPUs
Fig. 7: jacobi2d weak scaling on A40
Fig. 8: jacobi2d strong scaling on A40 MPI Charm ODF 1
Charm ODF 2 Charm ODF 4
Jacobi2D Weak Scaling
Charm ODF 8 Charm ODF 16
Time per step (ms)
20
15
10
5
0
1
2
4
8
16
32
GPU count
Fig. 9: jacobi2d weak scaling on MI250X
observed for Jacobi2D, with runtimes remaining comparable across MPI and Charm++ implementations with different ODF configurations. However, as shown in Figure 11, Charm++ begins to outperform MPI at larger scales. At 32 GPUs, higher ODF configurations achieve a 6.5% − 9.5% improvement in runtime compared to MPI, suggesting that overdecomposition provides benefits through improved communication– computation overlap. The performance difference between ODF-1 and MPI, however, still requires further analysis. Strong scaling on NVIDIA. We perform strong scaling experiments with a fixed domain size of 300 × 300 × 300, scaling from 4 GPUs (1 node) to 64 GPUs (8 nodes). Similar to the weak scaling results, the MPI and Charm++ implementations with different ODF configurations exhibit broadly comparable performance across the scaling range. However, at MPI Charm ODF 1
Charm ODF 2 Charm ODF 4
Jacobi2D Strong Scaling
35
Charm ODF 8 Charm ODF 16
30
Time per step (ms)
grid is divided among charm++ chares/MPI ranks, which communicate by halo exchanges. The application is run for 100 iterations without convergence checks. Weak scaling on Nvidia. Figure 7 presents weak scaling results for Jacobi2D using a base domain size of 32768×32768. As the number of GPUs doubles, the grid dimensions are alternately increased to maintain a constant workload per GPU. The experiments scale up to 32 GPUs across 8 nodes. Across the evaluated range, the different ODF configurations closely match the MPI implementation, indicating that overdecomposition introduces little additional overhead while still enabling features such as dynamic load balancing. Strong scaling on Nvidia. For strong scaling experiments, we use a fixed domain size of 131072 × 98304 and scale from 8 GPUs (2 nodes) to 64 GPUs (16 nodes). As shown in Figure 8, the ODF configurations continue to track MPI performance closely even at larger scales, consistent with the weak scaling observations. Weak scaling on AMD. On AMD systems, weak scaling is evaluated using the same base domain size as the Nvidia experiments, scaling from 1 GPU to 32 GPUs across 4 nodes. The weak scaling experiments show that time per step only has a small increase even as the number of GPUs increases, just like with the A40 runs. However, using overdecomposition shows clear benefits in execution time compared to running MPI or Charm without overdecomposition. Strong scaling on AMD. Strong scaling on AMD uses the same fixed domain sizes as the Nvidia experiments and scales from 8 GPUs(1 node) up to 32 GPUs(4 nodes). 2) MiniMD: MiniMD is a molecular dynamics proxy application that follows many of the same design principles as the parallel MD code LAMMPS. MiniMD uses spatial decomposition MD, where each Charm++ chare or MPI rank arranged in a grid owns a subset of the simulation domain. The application also exhibits a mixture of communication patterns due to its different communication routines. We run MiniMD for 100 iterations, disabling reverse communicate and safeexchange checking. Weak scaling on Nvidia. We start with a base domain size of 100 × 100 × 200 and alternately scale each dimension at every scaling step, scaling from 1 GPU to 32 GPUs across 4 nodes. The overall trend for 1-16 GPUs is similar to that
25.5
25.4
25.4
25.6
32
25.4
Average time per step (ms)
16
1
Charm-ODF-1
34.7 34.3 33.9 33.8 33.9 34.0
24
8
MPI 64
50.7
Charm-ODF-16
50.6
Charm-ODF-8
34.1 33.9 33.9 33.9 33.9 34.2
31.3 31.3 30.7 30.6 30.7 30.8
31.1 31.0 30.6 30.6 30.7 30.7
30.5 30.5 30.6 30.6 30.6 30.7
Average time per step (ms)
32
Charm-ODF-4
51.6
Charm-ODF-2
weak Scaling Performance of Jacobi2D
51.3
Charm-ODF-1
30.8 30.9 30.7 30.6 30.7 30.8
MPI 48
25 20 15 10 5 0
8
16
32
GPU count
Fig. 10: jacobi2d strong scaling on MI250X
Charm-ODF-16
32
8
16
1
8
16
43.2
41.9
41.2
42.9
40.2
43.3
40.9
40.8
42.6
40.7
41.3
16
32
64
Number of GPUs
Charm-ODF-2
Charm-ODF-4
Charm-ODF-8
Charm-ODF-16
strong Scaling Performance of MiniMD
142.8
156.5
75.5
78.2
78.0
77.2
128
8
4
8
16
16.0
16.8
16.1
17.5
16
15.9
35.6
34.9
32
34.9
34.8
64
18.0
13.3 13.1 12.3 12.5 12.2 14.1
24.4 22.7 22.2 22.4 22.0 22.2
16
Average time per step (ms)
48.0 45.0 44.4 44.3 42.9 43.2
32
4
Charm-ODF-1 148.5
MPI 256
131.4 131.5 131.7 140.1 136.0 137.4
64
8
Charm-ODF-16
145.1
Charm-ODF-8
165.4
Charm-ODF-4
strong Scaling Performance of MiniMD
84.5 86.7 84.5 84.5 84.8 85.3
Average time per step (ms)
128
Charm-ODF-2
8
Fig. 13: miniMD weak scaling on MI250X
144.5
Charm-ODF-1
39.8
2
Number of GPUs
Fig. 11: miniMD weak scaling on A40 MPI
39.6
16
Number of GPUs
256
37.1
35.9
39.1
36.1
38.6
36.2
34.8
32
8
32
Charm-ODF-16
64
36.6
4
Charm-ODF-8
128
35.2
2
Charm-ODF-4
76.5
1
Charm-ODF-2
weak Scaling Performance of MiniMD
76.2
16
Charm-ODF-1
35.4
56.5 52.8 51.7 52.6 51.9 51.1
51.5 53.1 52.3 52.9 52.3 52.1
51.0 52.4 51.7 50.7 50.9 50.7
40.1 40.1 40.2 43.5 42.5 42.7
39.7 40.0 39.9 40.1 41.0 41.0
64
MPI 256
35.6
Charm-ODF-8
128
38.5 38.4 38.7 38.7 39.2 40.1
Average time per step (ms)
Charm-ODF-4
34.4
Charm-ODF-2
weak Scaling Performance of MiniMD
34.3
Charm-ODF-1
Average time per step (ms)
MPI 256
32
Number of GPUs
Fig. 12: miniMD strong scaling on A40
Fig. 14: miniMD strong scaling on MI250X
32 and 64 GPUs, the ODF-16 configuration begins to perform slightly worse than intermediate ODFs. This behavior is likely due to the much smaller per-GPU problem sizes at higher scales, where excessive overdecomposition introduces runtime overheads that are not sufficiently hidden. Weak scaling on AMD. We use the same base domain size as in the NVIDIA experiments and scale MiniMD from 1 GPU to 16 GPUs (2 nodes). As shown in Figure 13, there is no noticeable performance gap between the Charm++ and MPI implementations, and the behavior is similar to what was observed for Nvidia. Strong scaling on AMD. We perform strong scaling experiments with a fixed domain size same as the Nvidia runs, scaling from 1 GPU to 16 GPUs (2 nodes). The results in Figure 14 indicate that Charm++ and MPI perform on par with each other, a trend that aligns with the Nvidia observations. 3) LULESH: LULESH (2.0) is a shock-hydrodynamics mini-application that solves the Sedov blast wave problem. It initializes a 3D hexahedral mesh of arbitrary size and partitions the domain across Charm++ chares or MPI ranks, with each rank responsible for a subdomain and for exchanging state across boundary elements. The application can also model load imbalance across domains, making it a useful benchmark for future load-balancing studies. The current MPI and Charm++ implementations are limited to decompositions with a perfectcube number of domains. Because of this perfect-cube constraint, the ODF often takes on fractional values. For example, running on 32 GPUs with
a target ODF of 4 would ideally require 128 chares. Since 128 is not a perfect cube, the application instead uses the nearest perfect cube, 53 = 125, resulting in an effective ODF of 125/32 ≈ 3.91. We run LULESH for 50 to 100 iterations depending on the scaling setup. Additionally, the global mesh is constrained to have equal side lengths in all √ three dimensions, requiring each dimension to be scaled by 3 2 (≈ 1.25) whenever the compute is doubled during weak scaling. This restricts the MPI runs we can get to only perfect-cube number of GPUs. We still do charm++ runs for the entire range of scaling. Weak scaling on NVIDIA. We evaluate weak scaling by increasing the grid dimensions as described earlier starting from a grid size of 256 × 256 × 256 to maintain a constant workload per GPU, scaling from 1 GPU to 32 GPUs across 8 nodes. The integral ODF configurations closely follow MPI performance across the scaling range, exhibiting trends similar to those observed for Jacobi2D and MiniMD. In contrast, the fractional ODF configurations consistently perform worse at most scales, indicating possible imbalance introduced by chare placement and mapping. This behavior warrants further investigation. Strong scaling on NVIDIA. For strong scaling, we use a fixed domain size of 480 × 480 × 480 and scale from 4 GPUs (1 node) to 64 GPUs (16 nodes). At smaller scales, the Charm++ implementations continue to track MPI performance closely across different ODF configurations. However, at 32 and 64 GPUs, the higher integral ODFs begin to show a slight
weak Scaling Performance of LULESH
weak Scaling Performance of LULESH
32 16
1
2
4
8
16
8
32
4
190.2 ODF: 3.91 320.2 ODF: 6.75 277.2 ODF: 16 241.0
strong Scaling Performance of LULESH
16
64
Number of GPUs
53.8
ODF: 16
ODF: 6.75
31.4 ODF: 2
ODF: 3.91
60.4
84.8
101.8
92.6 57.8 ODF: 4
ODF: 7.81
57.6 ODF: 1.69
32
ODF: 13.5
129.4 ODF: 15.62
104.2 ODF: 8
91.4 ODF: 1
64
ODF: 3.38
92.0 MPI
160.2
199.6
226.4 ODF: 6.75
175.6 ODF: 2
112.4
128
MPI
Average time per step (ms)
59.6 ODF: 15.62
31.8
33.4 ODF: 8
ODF: 3.38
49.1
32
28.1
30.6 MPI
8
ODF: 1
62.3
16
ODF: 1.95
66.2
61.9
ODF: 16
51.0
ODF: 3.91
ODF: 2
32
ODF: 6.75
ODF: 32
199.0
256
ODF: 16
269.9
311.9 181.2
ODF: 7.81
ODF: 13.5
215.2 ODF: 1.69
ODF: 4
208.3
166.2 ODF: 8
ODF: 15.62
286.8 166.6
ODF: 3.38
167.6 MPI
ODF: 1
433.0
64
4
32
512
332.6
330.1
ODF: 6.75
ODF: 2
ODF: 16
128
16
Fig. 17: lulesh weak scaling on MI250X
strong Scaling Performance of LULESH 512
8
Number of GPUs
Fig. 15: lulesh weak scaling on A40
Average time per step (ms)
ODF: 2
32
Number of GPUs
8
ODF: 1.69 247.6 ODF: 4 212.8 ODF: 7.81 350.8 ODF: 13.5 249.0
64
16
8
256
158.0 164.8 ODF: 3.38 302.2 ODF: 8 173.2 ODF: 15.62 205.4
2
MPI ODF: 1
1
ODF: 2 209.8 ODF: 6.75 282.6 ODF: 16 224.4
128
ODF: 4 215.4 ODF: 13.5 232.0
64
256
Average time per step (ms)
ODF: 2 212.7 ODF: 3.91 271.4 ODF: 6.75 231.2 ODF: 16 213.0
ODF: 1.69 216.8 ODF: 4 180.9 ODF: 7.81 311.9 ODF: 13.5 269.6 ODF: 32 199.3
189.3 188.4 ODF: 3.38 310.4 ODF: 8 186.8 ODF: 15.62 239.6 MPI ODF: 1
191.4 189.8 ODF: 4 ODF: 13.5
ODF: 2 194.7 ODF: 6.75 249.1 ODF: 16 194.7
194.3 193.7 192.2
128
MPI ODF: 1 ODF: 8
Average time per step (ms)
256
MPI 202.2 ODF: 1 195.4 ODF: 8 191.2
512
512
16 8
1
4
8
16
32
Number of GPUs
Fig. 16: lulesh strong scaling on A40
Fig. 18: lulesh strong scaling on MI250X
degradation in performance relative to intermediate ODFs, similar to the behavior observed in MiniMD. The reduced perGPU problem size at these scales likely makes the overheads from excessive overdecomposition more pronounced. Fractional ODF configurations again remain consistently slower throughout the scaling range. Weak scaling on AMD. On AMD systems, weak scaling experiments are performed from 1 GPU to 32 GPUs (4 nodes) with the base domain size of 320×320×320. Figure 17 shows almost similar runtimes between charm++ and MPI with some performance loss shown with higher ODFs. This behavior may be partially explained by limitations in the event-based IPC signaling mechanism used for intra-node HIP communication, which currently requires additional synchronization. Strong scaling on AMD. Strong scaling on AMD uses a fixed domain size same as the Nvidia runs, while scaling from 4 GPUs to 32 GPUs (4 nodes). The overall behavior mirrors the AMD weak scaling results, with Charm++ continuing to exhibit noticeable overheads relative to MPI for larger ODFs.
demonstrated it is possible to run overdecomposed kernels at very low grainsizes. Communication-wise, for overdecomposed programs to run efficiently, it is essential to efficiently handle multiple cases of source-destination pairs without requiring moving data via host. We implemented device-to-device data transfer in all code-paths in Charm++, based on the runtime’s knowledge of the destination location, which normally is pushed to the lowest level of communication layer, where the layer, such as UCX may use its internal mechanism to effect the data transfer often with copying. We demonstrated that pipelined communication, and data driven scheduling allows us to support overdecomposition effectively. The three mini - applications demonstrated that overdecomposed applications run with about the same performance as non-overdecomposed ones, including MPI-only as well. There is still more research and evaluation needed to make progress along this direction. First, we need to study highly fine-grained benchmarks, such as biomolecular simulations, and astronomy codes, such as n-body gravity codes. These may require additional mechanisms to mitigate overheads. More importantly, all of this only establishes the potential of overdecomposition. To realize and demonstrate its utility, specifically for dynamic load balancing and shrink/expand in the cloud context, it is necessary to support chare migration efficiently, and to do accurate load estimation, each with its own challenges.
VI. D ISCUSSION AND F UTURE W ORK We explored the overhead of overdecomposition in its computational and communication aspects. For computation, overdecomposition leads to smaller computation per kernel call. However, by employing efficient asynchronous techniques for kernel completion and continuations, utilizing multiple cores and chares to pipeline and overlap launch overhead, we
The contributions of this paper create a foundation on which these future capabilities can be built. VII. ACKNOWLEDGMENTS This work used Delta at NCSA through allocation ASC050025 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. This research used the Delta advanced computing and data resource which is supported by the National Science Foundation (award OAC 2005572) and the State of Illinois. Delta is a joint effort of the University of Illinois Urbana-Champaign and its National Center for Supercomputing Applications. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Advanced Scientific Computing Research programs in the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. R EFERENCES [1] Bilge Acun, Abhishek Gupta, Nikhil Jain, Akhil Langer, Harshitha Menon, Eric Mikida, Xiang Ni, Michael Robson, Yanhua Sun, Ehsan Totoni, Lukasz Wesolowski, and Laxmikant Kale. Parallel Programming with Migratable Objects: Charm++ in Practice. SC, 2014. [2] Michael Bauer, Sean Treichler, Elliott Slaughter, and Alex Aiken. Legion: expressing locality and independence with logical regions. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’12, Washington, DC, USA, 2012. IEEE Computer Society Press. [3] David A. Beckingsale, Jason Burmark, Rich Hornung, Holger Jones, William Killian, Adam J. Kunen, Olga Pearce, Peter Robinson, Brian S. Ryujin, and Thomas RW Scogland. Raja: Portable performance for large-scale scientific applications. In 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), pages 71–81, 2019. [4] Jaemin Choi, David F. Richards, and Laxmikant V. Kale. Improving scalability with gpu-aware asynchronous tasks. In 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 569–578, 2022. [5] H. Carter Edwards, Christian R. Trott, and Daniel Sunderland. Kokkos: Enabling manycore performance portability through polymorphic memory access patterns. Journal of Parallel and Distributed Computing, 74(12):3202 – 3216, 2014. Domain-Specific Languages and High-Level Frameworks for High-Performance Computing. [6] John K. Holmen, Marta Garcı́a, Allen Sanderson, Abhishek Bagusetty, and Martin Berzins. Lessons learned and scalability achieved when porting uintah to doe exascale systems. In Euro-Par 2024: Parallel Processing Workshops: Euro-Par 2024 International Workshops, Madrid, Spain, August 26–30, 2024, Proceedings, Part I, page 231–242, Berlin, Heidelberg, 2024. Springer-Verlag. [7] Pritish Jetley, Filippo Gioachin, Celso Mendes, Laxmikant V. Kale, and Thomas R. Quinn. Massively parallel cosmological simulations with ChaNGa. In Proceedings of IEEE International Parallel and Distributed Processing Symposium 2008, 2008. [8] Hartmut Kaiser, Patrick Diehl, Adrian S. Lemoine, Bryce Adelstein Lelbach, Parsa Amini, Agustı́n Berge, John Biddiscombe, Steven R. Brandt, Nikunj Gupta, Thomas Heller, Kevin Huck, Zahra Khatami, Alireza Kheirkhahan, Auriane Reverdell, Shahrzad Shirzad, Mikael Simberg, Bibek Wagle, Weile Wei, and Tianyi Zhang. Hpx - the c++ standard library for parallelism and concurrency. Journal of Open Source Software, 5(53):2352, 2020. [9] L.V. Kalé and S. Krishnan. CHARM++: A Portable Concurrent Object Oriented System Based on C++. In A. Paepcke, editor, Proceedings of OOPSLA’93, pages 91–108. ACM Press, September 1993.
[10] James C. Phillips, Rosemary Braun, Wei Wang, James Gumbart, Emad Tajkhorshid, Elizabeth Villa, Christophe Chipot, Robert D. Skeel, Laxmikant Kalé, and Klaus Schulten. Scalable molecular dynamics with NAMD. Journal of Computational Chemistry, 26(16):1781–1802, 2005. [11] Jiakun Yan and Marc Snir. Lci: a lightweight communication interface for efficient asynchronous multithreaded communication. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’25, page 1043–1059, New York, NY, USA, 2025. Association for Computing Machinery.