Performance Portability Explained: Insights from accelerating ICON microphysics

Performance Portability Explained: Insights from accelerating ICON microphysics Faveo Hörold1,2,* , Georgiana Mania3,* Panagiotis Adamidis , Harshada Balasubramanian4 , Jörg Behrens3 , Yakup Budanaz1 , Alexandru Calotoiu1 , Nicoletta Farabullini5 , Andreas Herten2 , Torsten Hoefler1,5 , Yannik Koenneker, Ioannis Magkanaris5 Philip Müller5 , Ralf Müller3 , Edoardo Paone5 , William Sawyer5 , Bjorn Stevens4 , Hannes Vogt5 3

1

ETH, Zurich, Switzerland Forschungszentrum Jülich, Jülich, Germany 3 Deutsches Klimarechenzentrum (DKRZ), Hamburg, Germany 4 Max Planck Institute for Meteorology, Hamburg, Germany 5 Swiss National Computing Centre (CSCS), Zurich, Switzerland

arXiv:2611.14204v1 [cs.DC] 10 Nov 2026

2

Abstract—Scientific computing is under increasing pressure to adapt to HPC hardware that is shaped by AI workloads. In these modern ecosystems, extracting reasonable performance without investing major engineering effort is a central issue for domain scientists. Correctness constraints and intricate reference implementations represent further impediments to development. Instead of using a double-precision implementation as a reference, considering the underlying scientific problem directly opens more opportunities for performance improvements when updating and rewriting applications. We take seven different portability frameworks—covering pragma-based, DSL, and data-centric approaches—and evaluate them on a microphysics kernel of the ICON climate model. Our performance benchmarks show that each different HPC system benefits from a different combination of optimizations, so we develop the measure of framework expressivity to gauge how easily each framework can customize portable implementations in this way.

I. I NTRODUCTION It is imperative for scientific computing in the HPC context to adapt to hardware ecosystems that are increasingly shaped by AI workloads. In weather and climate modeling, this shift is occurring alongside a broader transformation in both the scientific ambition and computational organization of Earth-system simulations, as kilometer-scale models and ensemble-based workflows demand ever greater computational efficiency [1], [2]. Not only are scientific applications expected to be performance portable even on the newest exascale systems, but results must remain correct across different hardand software environments [3]. Evaluating correctness of new implementations is difficult, regardless of whether results are within tolerance for a certain reference, or physically correct for a set of standard experiments with known solutions [4]– [8]. In general, optimizing and porting scientific applications to modern accelerator architectures does not result in a faithful * co-lead, equal contribution. Email: fhoerold@ethz.ch

re-expression of an existing implementation, especially given recent trends that change floating-point arithmetic at the hardware level. Furthermore, for many real-world problems there is no reference solution to compare against, only a reference implementation which has shown to match expectations for a set of experiments empirically. Approximating the difference between the physical solution and a given implementation is a research field of its own. Instead, performance portability requires design choices at multiple levels for the efficient implementation of computations and, in tandem, the representation of the underlying problem in terms of physics and numerics. Here, we argue that a scientific code development cannot be defined through a sequence of discrete, independent decisions. Rather, the best approach is for scientists to jointly navigate the shared design space with a spectrum of abstraction levels covering physics, mathematics, and computer science. From the precise form of equations employed, to the applied numerical algorithms and discretization strategies, down to the order of operations chosen at the hardware level, each decision has ripple effects on the final application. The domain scientist generally makes decisions about physical models and the experiment setup at the highest level of the spectrum. The computational scientist bridges the gap between the domain scientist and the performance engineer by applying existing numerical methods to discretize the model [9], [10]. At the lowest level, the performance engineer uses her knowledge of computer science to introduce efficient optimizations. In a productive workflow, all three scientists will be in active communication and aware of the broad impact that decisions at any level of the design space can have. In order to demonstrate how a simple design decision at the low-level end of the design space can affect all levels, we run performance portability benchmarks with a small physics kernel from the weather and climate domain, called Graupel. Like many scientific codes, the Graupel kernel is sensitive to

perturbations in computations, which arise from the fact that addition and multiplication are not associative in floating-point arithmetic. While a + b + c + d may be a mathematically unambiguous equation to the domain scientist, the computational scientist may prescribe a specific associativity (((a+b)+c)+d) to best avoid numerical cancellation. At an even lower level, the performance engineer may prefer a different associativity ((a + b) + (c + d)) in order to utilize superscalar hardware more efficiently. Thus, the shift in semantics at a lower level may undermine design decisions that have already been made at a higher level. In this work, we use the Graupel kernel to argue that achieving performance on modern architectures is not simply a matter of re-expressing an existing implementation in a portable framework. Rather, the rewriting process frequently traverses several levels of abstraction, as implementation choices interact with numerical structure, latent assumptions in the reference code, and the acceptable interpretation of the underlying scientific problem. Different performance portability frameworks mediate this process in different ways: they expose different optimization opportunities, impose different constraints on the resulting implementation, and support different modes of collaboration between domain scientists and performance engineers. We accelerate Graupel using seven different frameworks: OpenMP [11], OpenACC [12], Kokkos [13], SYCL [14], C++ std::par, DaCe, and GT4Py [15]. The Graupel onemoment microphysics module is part of the ICON climate and weather simulation ecosystem, which has been the focus of an ongoing effort to enable efficient kilometer-scale simulations on exascale HPC systems [16]. Graupel, and by extension ICON, represents the perfect proving ground for testing our ideas about performance portability. The following contributions are made: We apply seven different parallelization, performance, and portability frameworks to Graupel and benchmark the resulting implementations on several HPC systems. • After identifying the optimal set of optimizations on each system, we discuss how the engineering effort required to port, parallelize, and optimize Graupel varies between frameworks. • We distill the overlapping features of each framework down to a measure of framework expressivity, which evaluates how easily each framework can produce the distinct sets of optimizations needed for the different HPC systems.

•

Moving forward, we introduce ICON and Graupel in Section II. Next, we explore the design decision spectrum using concrete examples from Graupel in Section III. Then, we introduce the seven frameworks in Section IV, and present our performance results in Section V, where we also field test our implementations in a full-scale distributed R2B8 ICON run. In Section VI, we introduce the concept of framework expressivity and examine how it can reduce engineering efforts. In Section VII, we summarize related frameworks and move

to a conclusion in Section VIII. II. ICON AND G RAUPEL ONE - MOMENT MICROPHYSICS ICON (ICOsahedral Non-hydrostatic) [17] is a code base that supports a variety of atmospheric and oceanic applications, mainly related to the domains of numerical weather prediction and climate modeling. All atmospheric applications share an underlying fluid dynamical solver, which can be configured for various geometries [18], e.g., on an icosahedral grid [19], and resolutions [20], including static adaptivity through nesting. The dynamical solver includes an energy equation coupled to mass conservation equations. Additionally, a transport module advects various phases and habits of water, as well as trace-gases or particulates. Diabatic processes, such as turbulence, radiation, or particulate microphysical processes are also modeled. The ICON code base contains different cloud-microphysical models, but the core approach is a one-moment bulk scheme called Graupel that describes the formation, growth, and sedimentation of water particles (hydrometeors). Graupel calculates transitions between vapor, liquid and solid water, whereby the liquid and solid phases are described by different habits— droplets and drops (rain) for liquid; ice, snow, and graupel for solid. Phase changes influence temperature and density. This in turn influences force balances in the dynamic equations both directly, and indirectly by virtue of how water is partitioned among phases and habits, and how these interact differently with radiant energy transfer. Graupel presented itself as an interesting case study for the study of performance portable design choices for two main reasons: Firstly, Graupel only involves grid dependencies in the vertical dimension, where all points are co-located on a process. No message-passing communication or horizontal reductions are required. This is an access pattern shared by most physical parameterizations in the weather and climate domains. Secondly, Graupel employs a sparse calculation in physical space, reflecting that clouds and hydrometeors are not everywhere: typically they cover only 5-25 percent of the domain. Making it more challenging is that this sparsity is dynamic and threshold dependent. To minimize computation on classical CPU architectures, various gather-scatter approaches are applied to condensate thresholds that themselves could be targets of optimization. For emerging architectures, however, a gather-scatter approach is not necessarily advantageous. With this in mind, alternative implementations only need to reproduce the results of the masked domain, with some flexibility in how conservatively to define the masks, mindful of the fact that the sparsity of this algorithm is the key challenge in creating performance-portable code. We take a closer look at these sparsity-aware optimizations and examine the actual effects that they have on performance for GPU architectures in Section VI-A. As a first step in this study, the original ICON Graupel implementation (still in ICON) for weather prediction was completely rewritten, still in Fortran, but avoiding advanced Fortran language features. The rewrite was done by a domain

Hardware constraints (FLOPs, Data size)

Domain Scientist Physical Constraints

E=n×...

∑xy⋅...

Discretization Approaches

Accuracy constraints MPI

Blas

Performance Engineer

Data layouts, Parallelization Approaches Stability Constraints

Computational Scientist

Fig. 1. Illustrating typical design choices at abstraction levels and coordination between experts. At the highest level, the domain scientist formulates a hypothesis and chooses the equations which model her problem. At an intermediate level, the computational scientist discretizes physical and temporal domains, introducing appropriate numerical methods to solve the scientist’s equations. At the lowest level, the performance engineer chooses data layouts and parallelization strategies to map the numerical methods to a given HPC architecture.

scientist, with the intent of removing unnecessary dependencies and unused legacy functionality, and separating specific processes that had become entangled over years of piecewise optimization. The rewrite resulted in a threefold reduction in lines of code, a much greater algorithmic transparency with no loss of efficiency on classic CPU architectures, and the elimination of some errors. This rewrite made the compact code base more accessible for computational scientists and performance engineers for subsequent analysis. In addition to the sequential Fortran implementation, a parallelized version of the Graupel code using OpenACC is available in the official ICON release. Since Graupel does not make use of any distributed reductions, optimizations can produce bit-identical results when built with a conservative compiler configuration. Nevertheless, those optimizations that adjust the handling of sparsity for accelerators fundamentally change the numerical results of Graupel. The necessity of discussing, handling, and accepting such changes is one of the main discussion points of this work. III. A SPECTRUM OF DESIGN CHOICES In this section, we build on our ideas from Section I and solidify how all design choices that go into designing, porting, and optimizing a scientific application can be mapped onto a spectrum passing through each of the domain scientist’s, computational scientist’s, and performance engineer’s domains. The highest levels of the design space reside in the domain of physics. This is where the domain scientist comes up with a hypothesis, constructs a physical model, and outlines an experiment to prove or disprove the hypothesis. The domain scientist is responsible for setting initial and boundary conditions for the experiment, accuracy constraints, and physical equations to simulate. The layers at a medium level of abstraction in the design space reside in the space of mathematics. Here, the computa-

tional scientist takes the experiment proposed by the physicist, discretizes it, and chooses the appropriate numerical methods to solve the physical equations while fulfilling the accuracy constraints imposed by the physicist. The computational scientist is responsible for imposing stability constraints, which will sometimes imply floating point accuracy constraints. The lowest abstraction levels of the design space dive into computer science. Here, the performance engineer will take the high-level implementation proposed by the computational scientist and look for opportunities to improve scheduling, data layout, and access patterns. They have to work under the constraint of the stability and accuracy constraints imposed by the domain and computational scientists, respectively. However, they can also ask the computational scientist to update their numerical model, which might be made more amenable to efficient data layouts if the discretization approach (e.g. grid, domain decomposition) is adjusted. The performance engineer’s feedback can even influence decisions about the experiment design at the highest abstraction levels; depending on hardware constraints like available FLOPs, memory, and input/output data storage, the physicist may need to update their experiment and the computational scientist may need to find more efficient numerical methods or propose a more granular domain decomposition to make better use of limited HPC node resources. In practice, no single person in a research team will have a deep understanding of every level in the abstraction spectrum. Rather, various scientists specialize in different regions of the spectrum and must communicate in order to choose the best approach that leads to a feasible design at the low level while delivering acceptable results at the high level. Across all of these levels, design choices may be semantically equivalent within the abstraction at which they are made, while nevertheless constraining the choices that remain available at lower levels. Crucially, once a decision has been fixed at a higher level, subsequent transformations at lower levels may be unable to recover either the numerical behavior or the performance potential that would have been enabled by a different choice at the higher level. At the same time optimizations such as reordering instructions at the lower level cannot be applied without fundamentally breaking numerical approaches at the higher level. These inter-dependencies and the need for coordination between experts is illustrated in Figure 1. When confronted with the need to rewrite or port an application to new hardware, the central question is no longer merely how to preserve an existing implementation, but how far one must ascend the ladder of abstraction in order to recover a sufficiently rich optimization space. The answer will, in general, depend on both the structure of the application and the characteristics of the target system and the definition of correctness. CLEO [21] illustrates the cumulative effect of design choices at different abstraction levels. It is a state-of-theart cloud microphysics model in ICON that uses Lagrangian particle tracking to explicitly compute microphysical pro-

cesses, thereby providing a potential alternative to the reference implementation used in this work. With the introduction of a more computationally intensive physical model by the domain scientist [22], both the computational scientist and the performance engineer focus on designing for and improving flexibility and performance, instead of merely optimizing an existing model. In this work, we focus on the central part of the design spectrum, encompassing implementation details and optimizations related to the HPC and node-local architectures. The design decisions required for parallelization and porting also fall within this region. Therefore, we explicitly do not consider the edges of the design spectrum: In our situation, the domain scientist has already fixed the physical model and the computational scientist has already decided on a numerical scheme and discretization strategy. However, we conferred with domain experts to confirm that propagating or masking out certain negligible physical contributions was acceptable for the results of the Graupel simulation. We explore the effects of these sparsity-exploiting optimizations in Section VI-A. On the other end of the spectrum, details of the ISA and hardware implementation have been fixed; we run our computations using IEEE-754 single- and double-precision floating-point arithmetic on various x86-64 and aarch64 processors, optionally with offloading to Nvidia and AMD GPU accelerators. IV. F RAMEWORKS FOR PARALLELIZATION , P ERFORMANCE , AND P ORTABILITY This section introduces the frameworks and APIs that are currently being evaluated, roughly ordered by historical appearance: OpenMP, OpenACC, Kokkos, SYCL, C++ Standard Parallelism (C++ std::par), DaCe (using its Fortran frontend) and GT4Py (using DaCe as a performance-enhancing backend). All frameworks aim to allow the same source code to target multiple hardware architectures while aiming for good performance during parallel execution, but choose different means to achieve this goal. Support of a certain programming model on a certain device type through a certain compiler is a complex and moving target. While we try to give a concise overview, we refer to the literature for more extensive discussions [23]. Despite the rewrites being largely independent, there are overall trends that can be identified across the different frameworks: In contrast to directive-based approaches, Kokkos, SYCL, and C++ std::par do not merely annotate an existing sequential implementation. Instead, they require developers to reformulate the computation in terms of framework abstractions. As a result, although the host language remains C++, achieving portability is not simply a matter of incremental annotation, but of recasting the algorithm so that parallelism is made explicit in the form expected by the framework. The resulting code is shaped not only by the scientific algorithm itself but also by the abstractions and limitations imposed by the respective frameworks. This can make implementations more challenging to read and reason about, particularly for computational and domain scientists

whose primary concern is the numerical method rather than the mechanics of heterogeneous execution. On the other hand, directive-based frameworks like OpenMP and OpenACC may require compile-time logic to differentiate between different acceleration approaches, which is also known to clutter code and make maintenance more challenging. Finally, DaCe and GT4Py side-step this issue by hiding optimizations from the code altogether. The potential downside is that the build process for optimized release binaries becomes more complex. We note that the initial translation of the Fortran code to sequential C++ using raw pointers served as the starting point for OpenACC, OpenMP, SYCL, and C++ std::par implementations, hence they share some of the technical decisions, i.e., to use linearized array indexing. While Kokkos, DaCe and GT4Py make use of linearized arrays in the backend/generated code, the 2d indexing is still used in the application code, which improves readability. A. OpenMP Introduced in 1997, OpenMP is arguably the most prominent parallel programming model for shared- and near-memory parallelism [24]. At its core, the OpenMP standard outlines an extensive set of compiler directives designed to facilitate parallel programming. These directives require compiler support for translation into parallel constructs, alongside a specialized runtime library that implements hardware-specific functionality. The OpenMP standard v4.0 introduced the omp target directive (extended with every subsequent release), which opens the path to offloading the computations to accelerators such as GPUs. [11] Although C++ and Fortran compilers provide well-optimized OpenMP support for CPU architectures, GPU implementations still lag in both functional capabilities and performance efficiency. [25] Hardware vendors develop specialized OpenMP implementations, finely tuned to their architectures. NVIDIA’s HPC SDK compiler suite stands out as a notable example. Complementing this, open-source compilers like GNU GCC and LLVM Clang also offer strong OpenMP support, [26] fostering accessibility and communitydriven innovation. The Graupel implementation using OpenMP was tailored for GPU execution and in view of exascale systems. Specific optimizations to reduce branching were implemented alongside a series of algorithmic changes. Nevertheless, these proved to hinder the CPU parallelism. This highlights a significant issue with pragma-based portability frameworks—it is challenging to fine-tune the implementation for heterogeneous set of architectures such as CPU and GPU without sacrificing readability and maintainability. B. OpenACC OpenACC [12] is a directive-based programming model that targets parallel accelerators. Supported mainly by NVIDIA, coverage beyond the NVIDIA HPC SDK compilers is limited, especially for CPUs. OpenACC is modeled closely after OpenMP, but specifically for more detached accelerators

(although CPU support is also available). In a time, when GPUs rapidly entered the HPC market, OpenACC was able to swiftly develop directive-based accelerator programming, and implement it in vendor compilers. Simple directives, like kernel allow to accelerate many lines of code at once. As support for the model beyond the NVIDIA ecosystem is minimal, other mature accelerator-enabled alternatives have emerged since. The Graupel C++ OpenACC implementation is straightforward and based on the sequential C++ version. Given the limited compiler support, no further optimizations were made, since we consider that C++ OpenACC is not a viable successor of the existing Fortran OpenACC in ICON. C. Kokkos Kokkos originated as part of the Exascale Computing Project and split off into an independent project in 2015. Currently, it is organized as an established open-source project of the High Performance Software Foundation1 . With powerful abstractions for data and execution modes, Kokkos enables scientists to separate the concerns between algorithm development and performance portability. The API is exposed as a C++ header-only library which translates the code to parallelization APIs native to each hardware platform, enabled by selecting the host and device backend at compile time. Currently, Kokkos supports three execution models for the host—a sequential one and two parallel ones, OpenMP and C++ Threads—and three execution models for accelerators, enabled by the device backends CUDA, HIP, and SYCL. Kokkos is designed to be compiler-agnostic and it can therefore be effortlessly embedded into diverse software build systems. The Kokkos::View abstraction for multi-dimensional arrays draws a conceptual parallel to Fortran’s array notation, easing ICON developers’ transition from Fortran to Kokkos. It also serves as the foundational blueprint for the upcoming std::mdspan support in the C++26 standard. The Kokkos Graupel implementation is a direct rewrite of the original Fortran code, with minimal changes, which preserves the initial algorithm structure for readability purposes. Therefore, it can trivially produce bit-identical results to Fortran in specific compilation setups on CPU platforms, while leaving room for future more aggressive optimizations. D. SYCL The SYCL standard was introduced by the Khronos Group in 2014 as an extension to C++, aiming to build on the device support and parallel execution capabilities of OpenCL. SYCL, much like Kokkos, occupies an intermediate position in the performance-portability landscape: it preserves a single-source programming model across heterogeneous architectures, while still exposing enough structure for downstream compilers and runtimes to specialize execution for a given target. 1 https://hpsf.io

Like Kokkos, SYCL has its own abstractions for execution and data management, imposing similar constraints. Additionally, unlike Kokkos, SYCL relies on compiler support, which provides added benefits such as SYCL-specific optimisations and just-in-time compilation for its C++ kernels, offering the flexibility of on-the-fly code generation for specific hardware—but of course at the cost of a strong compiler dependency. The microphysics implementation in SYCL, a direct translation of the original Fortran code, was tested using Intel’s oneAPI Data Parallel C++ (DPC++) compiler suite and AdaptiveCpp, which leverages LLVM to compile SYCL across different hardware platforms. On x86 hosts, the Intel compiler supports CUDA, ROCm/HIP, and Level Zero backends. Even so, separate plugins (e.g., for CUDA and ROCm/HIP) must be compiled and patched into the compiler manually, and are not always available on existing HPC systems. Similarly, AdaptiveCpp almost always needs to be built by the performance engineer, requiring additional effort in existing portability workflows. Moreover, on Arm CPUs, a heavy reliance on AdaptiveCpp as the single SYCL implementation remains a limiting factor. E. C++ std::par C++ Standard Parallelism, or C++ std::par, refers to the support of parallel execution directly in the C++ standard, enabled through the STL. Similar to Kokkos and SYCL, individual algorithmic building blocks, like for_each, can be used to express algorithmic patterns abstract on a high level. With the according execution policy, enabled algorithms can then be executed in parallel and unsequenced2 . C++ std::par needs compiler and library support, and is currently available in the NVIDIA HPC SDK or in the opensource AdaptiveCpp. The simplicity of clean C++ code—free from pragmas and explicit memory control—comes with a limitation: STL parallel loop lambdas can only capture host pointers, by value or reference. While this is not a problem for CPU runs, it becomes one for GPU execution. In ICON’s use case, data pointers are copied to the GPU before the time loop, processed across multiple timesteps, and copied back to the host at the end. This necessitates passing device pointers directly to the Graupel function to avoid CPU-GPU intermediate transfers. However, this is not supported by C++ std::par, leading to the high runtimes observed in Figure 3 due to host-device memory transfers at every invocation. F. DaCe DaCe is a data centric framework [27]–[32] that provides architecture-specific backends, which specialize a common intermediate representation—the Stateful Dataflow multiGraphs (SDFG) IR—to diverse target systems, including modern CPUs and GPUs. Its core contribution is to lift programs from control-centric languages such as Fortran and Python 2 Function calls with the unsequenced may be interleaved.

into the data-centric representation, where data movement and dependencies become explicit and latent parallelism can be exposed before the computation is specialized toward a particular hardware architecture. This separation is important: rather than embedding optimization decisions directly into the source code, DaCe enables transformations to be applied at the level of the representation, allowing the same computational intent to be retargeted across CPUs and accelerators. In this sense, DaCe does not merely offer another portability layer; it offers a structured pathway from legacy scientific code to architecture-aware implementations through a sequence of semantically equivalent program transformations in the performance engineering space. In particular, data-centric methods have been shown to support Gordon Bell Prize-winning work at SC19, [33] and the same broader line of work is now also associated with the SC25 Gordon Bell Prize for Climate through its role in enabling efficient kilometer-scale Earth-system simulations [20]. DaCe’s vision is to separate the concerns of domain scientists and performance engineers. Domain scientists should be able to express the problem to be solved in a form that remains close to the underlying scientific intent, while performance engineers operate on transformations that improve performance, exploit parallelism, and specialize the computation for different hardware targets. This division of labor improves maintainability and readability and avoids forcing either group to acquire deep expertise in the other’s discipline. As architectures increasingly privilege mixed precision, heterogeneous execution, and AI-oriented computational patterns, successful optimization cannot always remain confined to backend tuning alone. In many cases, achieving substantial gains will require revisiting the representation of the problem itself and, in some instances, reconsidering aspects of the numerical method. While DaCe allows the source code to stay in its original format, and thus makes collaboration with the domain scientist and computational scientist simpler, it does require the performance engineer to become familiar with the data-centric representation and learn how to write and apply data-centric code transformations, which is a significant challenge, different than optimizing imperative programming languages, further compounded by the currently growing, but small, community of experts. Here, we use F2DaCe to refer to the DaCe Fortran frontend, which lifts the Fortran source code directly into the SDFG IR. G. GT4Py GT4Py [15] is a Python library for generating high performance implementations of stencil kernels from a high-level definition using Python syntax by taking advantage of a range of backends to provide high performance on a variety of compute architectures. It follows a functional programming model, providing abstractions for parallel patterns common in atmospheric simulation, and is part of the GridTools [34] framework, a set of libraries and utilities used to develop performance portable applications for weather and climate modeling. Similar to DaCe with the Fortran frontend as

presented in Section IV-F, GT4Py attempts to separate the concerns of the domain scientist from those of the computational scientist and the performance engineer, even if a complete separation is not realistically possible in any framework. In contrast to DaCe, GT4Py builds on a Python implementation. This distinguishes GT4Py from the other frameworks; porting to Python represents a significant engineering effort and furthermore, making an incremental port is more difficult than for other frameworks because modules need to be interfaced with a different language. Nonetheless, we include the GT4Py framework in this work because it represents yet another approach to performance portability. Moreover, in the niche case where the reference implementation is already written in Python, this porting effort disappears, giving GT4Py a clear advantage over other frameworks. In GT4Py computations are expressed in a domain specific language, a subset of Python, which can be either executed natively in Python (mainly for debugging), or just-in-time (JIT) compiled to high-performance implementations for CPUs and GPUs utilizing DaCe as an optimization framework. On the first run of a GT4Py program, the user will see a potentially significant overhead from JIT compilation workflow implemented in Python, which is avoided in subsequent runs due to caching. On the other hand, JIT compilation allows to specialize the generated executable and apply more optimizations based on the parameters passed at runtime. The GT4Py Graupel port was implemented by a computational scientist who reproduced the original Fortran results. Performance engineers suggested refactoring that improved the performance marginally, however, most of their optimizations targeted GT4Py and the generated DaCe IR using custom transformations. These transformations take advantage of inplace updates to memory that reduced the number of copies necessary stemming from the functional representation of GT4Py for points that are masked-out by a where expression and impacts the performance due to the sparse characteristic of the problem. Conceptually this transformation could be implemented generically in the backend for all applications, but due to time constraints, this optimization was tailored to Graupel. In order to enable fair comparisons with other frameworks, we report the pure GT4Py timings along with those for this specialized optimization (GT4Py opt). Compared to standardized programming frameworks, the development of the GT4Py DSL is currently driven by a small community. V. R ESULTS In order to understand the characteristics of each framework, we evaluate their runtimes on several different systems with different CPUs and accelerators. On CPU, we use the full number of cores; 64 on EPYC, 24 on Intel, and 72 on Grace. For this reason, using our results to compare the performance of implementations on different hardware platforms is not straightforward. Instead, we emphasize the relative difference between implementations and how these compare between hardware platforms and execution instances.

CPU runtimes (s) OpenMP

SYCL

F2DaCe

GT4Py

GT4Py opt

10.5s

14.7s

std::par

6.0 4.0 2.0 G I NGGG G C I NGGG G GANGGG G C ANGGG

2.0

G C I NGGG

I

G I NGGG C C I NGGG G GANGGG G C ANGGG 2.5s

I

3.3s

I I N I GG

2.7s

1.5 1.0 0.5 I

0.53s

0.54s

0.55s

0.4

I I N I GG G N I N C GG

0.66s

0.0 G 0.5

C I NGGG G GANGGG G C ANGGG

G

I I NGGG G G I NGGG C C I NGGG G GANGGG C C ANGGG

G I I N I GG EPYC 7763

I

I

G I I NGGG EPYC 7763

0.57s

20 km resolution

Kokkos

8.0

0.0 G 2.5

40 km resolution

OpenACC

2.8s

10 km resolution

10.0

0.3 0.2 0.1 0.0

I I NGGG EPYC 7742

G I NGGG G GANGGG G C ANGGG Xeon 8168 Grace (1) Grace (2) Double precision

I

N I NGGG C G I NGGG G GANGGG C C AN C GG EPYC 7742 Xeon 8168 Grace (1) Grace (2) Single precision

Fig. 2. Runtimes for different implementations and grid sizes on CPU. Letters below the bars indicate the compiler that resulted in the fastest run: GNU, Clang/LLVM, Intel, Nvidia, Adaptive CPP. The left half shows double-precision runs, while the right side shows single-precision runs. Different grid sizes are grouped by rows; the top row is for the finest 10 km resolution, the middle row is for 20 km resolution, and the bottom row is for the coarsest 40 km resolution. The OpenACC implementation is omitted here because no compiler targets it to CPU with competitive performance.

GPU runtimes (ms) GT4Py opt 815.4ms

GT4Py 772.2ms

2948.8ms

914.2ms

846.6ms

F2DaCe

std::par

40

240.1ms

GGG C NG I NNGG C NNANNGG 229.6ms

835.1ms

847.3ms

C NG I NGGG C NN I NNGG C G I 147.0ms

GGG C NG I NGGG C NNANGGG

252.5ms

0 C NN I NGGG C NN I NNGG C G I 15

218.2ms

GGG C NN I NNGG C NNANGGG

40

I

162.8ms

C GI

30 20

74.9ms

GGG C NN I NGGG C NNANNGG 72.6ms

765.8ms

C NG I NGGG C NN I NNGG C G I 27.5ms

79.5ms

74.2ms

612.9ms

12

15.07ms

10

32.3ms

20 km resolution

SYCL

80

0 C NN I NGGG 50

40 km resolution

Kokkos 814.6ms

120

OpenACC

966.7ms

10 km resolution

160

OpenMP

9 6 3 0

C NN I NNGG C NG I NNGG C G I GGG C NG I NGGG C NGANGGG A100 80GB A100 40GB MI300 Hopper (1) Hopper (2) Double precision

C NG I NGGG C NN I NNGG C G I GGG C NG I NGGG C NGANGGG A100 80GB A100 40GB MI300 Hopper (1) Hopper (2) Single precision

Fig. 3. Runtimes for different implementations and grid sizes on GPU. This figure is structured identically to Figure 2: Double-precision runs are shown on the left, single-precision runs on the right. The top row is for 10 km, the middle for 20 km, and the bottom row for 40 km resolution. For A100 40GB (double-precision, left half) no timings are shown for the 10 km grid because this problem size does not fit on the device. SYCL automatically shuffles memory to the device asynchronously, effectively circumventing this limitation. For MI300, OpenACC and C++ std::par are missing due to inadequate compiler support.

TABLE I L IST OF HPC CENTERS AND HARDWARE USED FOR BENCHMARKS IN THIS WORK . System name

Organization

CPU

GPU

Levante DKRZa AMD EPYC 7763 Nvidia A100 80GB Jureca JSCb AMD EPYC 7742 Nvidia A100 40GB Juwels JSC Intel Xeon Platinum 8168 c Beverin CSCS AMD Instinct MI300 Santis CSCS Nvidia Grace Nvidia GH200 Hopper GPU Jupiter JSC Nvidia Grace Nvidia GH200 Hopper GPU a Deutsches Klimarechenzentrum (German Climate Computing Centre) b Jülich Supercomputing Centre c Centro Svizzero di Calcolo Scientifico (Swiss National Supercomputing Centre)

Each measurement reported here uses the compiler giving the best performance for each implementation. We provide the list of viable compilers for each combination of system, architecture, and implementation in Appendix A. There, we also provide a detailed description of compilation flags and software dependency versions. As shown by the variations in our benchmark data, performance shifts based on a number of factors, including not only compiler choice, framework, system, and architecture, but also distribution, contents, and sparsity of the input data. For this reason we introduce the term execution instance to signify that a specific implementation was run with a specific compiler, on a specific system and architecture, and with a specific test size, e.g. 20km resolution. A. Performance Benchmarks The main experiment is designed to give each performanceportability framework the best possible chance to perform well, while solving the same Graupel configurations on global grids at 40 km (referred to as ”R2B6” resolution), 20 km (”R2B7”), and 10 km (”R2B8”) resolution across a diverse set of CPU and accelerator platforms. The resources provided by each HPC system are listed in Table I. For every framework, we evaluate multiple compiler options and report results using the compiler that achieves the best performance on each platform, rather than imposing a uniform compiler choice across all systems. This setup is intended to assess each framework under favorable and realistic conditions, and to compare their ability to deliver strong performance across heterogeneous architectures. Our results show that the best choice of compiler is not only dependent on the framework, but also on the underlying hardware, power limits, and problem size. To reduce the influence of cold-start effects and capture runto-run variability, we execute the Graupel kernel 105 times for each measurement. We discard the first five runs and report the median of the remaining 100 executions. We have found that the performance measurements are repeatable to the extent that confidence intervals become effectively invisible in our plots, so we omit them altogether. This is largely due to the fact that the lack of MPI communication eliminates any network jitter and runs are performed in isolation on compute nodes, so influence from the OS is minimal.

Notes 8 NUMA domains per CPU socket GPUs not benchmarked CPUs not benchmarked 300W CPU, 580W GPU power cap

For the most part, the different frameworks perform within a factor of two of one another for a given device. This can be observed for the GPU runtimes shown in Figure 3. Larger performance gaps appear on CPU architectures, shown in Figure 2. On CPU, the presence of multiple NUMA nodes and various cache-coherence network layouts can exacerbate small implementation differences, which we observe in particular for the EPYC 7742 and Xeon CPUs, where Kokkos, C++ std::par, and GT4Py perform significantly worse than their counterparts (including optimized GT4Py). EPYC 7742 has core complexes with four cores each [35] and uses a huband-spoke layout for the cache-coherency layer, which is a part of Infinity Fabric™ [36]. In the EPYC 7763, core complexes have eight cores each [37], significantly reducing the number of cache-coherence bus requests that have to be fulfilled via the slower link to Infinity Fabric. The Intel 8168 CPU uses a mesh network interfacing all 24 cores in on NUMA domain [38]. Grace, which also uses a mesh network, called Scalable Coherency Fabric (SCF) [39], has a much higher bisection bandwidth than the older Intel CPU, again explaining the reduced impact of cache-coherence traffic. 3 Apart from a few outliers (C++ std::par for GPU, NUMA-slowdowns on CPU), our results show that the portability frameworks expose broadly similar performance ceilings. Thus, there is contention between the framework, and it is apparent that the framework which performs best varies between execution instances, even when the hardware of a single node is identical. For example on Hopper, Kokkos performs best for the largest grid in double-precision, but is beaten by GT4Py for smaller grids. Meanwhile, on A100, SYCL takes a clear lead for double precision, surpassed only by OpenACC for single precision. Instead of evaluating the best-performing framework, a more interesting observation here is that the relative performance of F2DaCe and GT4Py vary between problem sizes and precisions. For large grids, F2DaCe wins, but GT4Py clearly performs better on the smallest 40 km grid. Meanwhile in single precision, GT4Py wins over F2DaCe regardless of the problem size. Overall, C++ std::par shows comparatively weak performance on GPU; this is caused by the measurement problem 3 Nvidia reports a bisection bandwidth of 3.2 TB/s for SCF [39], while the older Intel CPU appears to have a bisection bandwidth of about 32 B/cycle × 2.4 GHz × 5 mesh rows = 384 GB/s [40]–[42].

CPU optimization runtimes

GPU optimization runtimes

F2DaCe

Jureca-DC (AMD EPYC 7742)

GT4Py

Jureca-DC (Nvidia A100 SXM 40GB)

1.8 Runtime (relative, lower is better)

Runtime (relative, lower is better)

1.4 1.0 0.6 Jupiter Booster (Nvidia Grace) 1.8 1.4 1.0 0.6 Juwels Cluster (Intel Xeon Platinum 8168) 1.8 1.4 1.0 0.6

1.4

F2DaCe

1.2

Fortran

1.0

GT4Py Jupiter Booster (Nvidia GH200 Hopper)

1.4 1.2 1.0 Beverin (AMD Instinct MI300A) 1.4 1.2 1.0

A B C D E F G H

A B C D E F G H

A B C D E F G H

SIG F LIP M ASK

A B C D E F G H

A B C D E F G H

A B C D E F G H

40 km

20 km

10 km

SIG F LIP M ASK

40 km

20 km

10 km

Fig. 4. Relative runtimes for the SIG, F LIP and M ASK optimizations for all problem sizes on CPU. The key at the bottom indicates which of the SIG, F LIP, or M ASK optimizations were applied. The heavy box highlights the configuration used by the original Fortran implementation. Runtimes are independently normalized so that the default implementation measures 1.0. Different colors show the effects of equivalent optimizations on different implementations.

described in Section IV-E. On CPU, where there is significantly more variance between the performance delivered by individual implementations than on GPU, we observe that F2DaCe is competitive on Grace and EPYC 7763 for both precisions, but is ultimately beaten by SYCL on Xeon 8186 and EPYC 7742. Unsurprisingly, SYCL, a framework pushed by Intel, performs better than the other frameworks on Intel hardware. However, the exception defines the rule: While rather lacking in performance on EPYC and Xeon, Kokkos makes up for it on Jupiter’s Grace CPU (Grace (2) in the figure) to become the best implementation for small problems in single precision. The significant difference in relative performance for the Kokkos framework between Santis and Jupiter Grace CPUs is due to different power caps set for those systems. This example clearly shows how drastic performance variations of a single framework implementation can be triggered by any minute difference in execution instance. The results support our findings that no single ranking between frameworks is possible, and a framework that performs adequately for one execution instance may incur significant performance losses on another system or beat all other implementations for a different problem size. Thus, we emphasize the need for an expressive framework that can expose different optimizations and variants to adapt to the problem and system at hand. B. Correctness An optimized implementation is considered valid only if it preserves numerical accuracy. Correctness is evaluated using the maximum absolute difference between double- and single-

Fig. 5. Relative runtimes for the SIG, F LIP and M ASK optimizations for all problem sizes on GPU. The key at the bottom indicates which of the SIG, F LIP, or M ASK optimizations were applied. The heavy box highlights the configuration used by the original Fortran implementation. Runtimes are independently normalized so that the default implementation measures 1.0. Different colors show the effects of equivalent optimizations on different implementations. There is no data for Fortran on Beverin due to lack of compiler support.

precision outputs of the original Fortran code for each field. Correctness checks were performed with low compiler optimization flags (-O0, -nofma). As we implied in the introduction (Section I), testing for bit-identical results is an unattainable undertaking, especially on GPUs. In fact, Fortran does not impose associativity guarantees, which may be exploited by compilers even with optimization turned off. In our Graupel case, the unordered reduction a + b + c is a concrete example of an expression that is interpreted differently by the Nvidia and GNU compilers, leading to differences in results. Moreover, different math libraries lead to slight numerical deviations on CPU and GPU for various functions such as log() and exp(). VI. F RAMEWORK EXPRESSIVITY AND ENGINEERING EFFORT

Our benchmark results show that all frameworks achieve good performance on most systems. In this section, we include combinations of optimizations that can go beyond semantic equivalence in the performance engineering space and require reasoning from the domain scientist and computational scientist as well. We focus our optimizations on how sparsity is exploited and what data access patterns are leveraged, and analyze their impact on performance. This leads us to consider an additional aspect when comparing performance portability frameworks, namely their ability to express different physically equivalent implementations and the ease with which they can navigate from one to another. This is the reason why we only focus on the F2DaCe and GT4Py frameworks

in this section: less engineering effort to express the different optimizations. A. Taking advantage of sparsity in Graupel The reference Graupel implementation makes use of two optimizations that exploit sparsity in the fundamental physical formulation of the Graupel model. The first optimization, which we call M ASK, completely disables computation for any cells where the physical properties of the air parcel do not lead to active microphysics processes. The second optimization, which we label SIG, disables calculations related to snow, ice, and graupel when those species are only present in negligible quantities and would thus make only negligibly small contributions to the parameterization. As mentioned in Section III, we discussed the impact of toggling these optimizations, as they change the numerical output of the simulation, even if only by a negligible amount. Domain experts confirmed that when contributions become small in local regions of the domain, there is no difference between counting them in or masking them out altogether. Thus enabling or disabling the sparse optimizations as we do here does not affect the physical correctness of the final results of the Graupel simulation: We toggle the M ASK and SIG optimizations on and off individually to understand their impact on the performance of Graupel on different systems. Hypothesis: On a CPU architecture, we expect each of these optimizations to reduce runtime as they reduce the computational workload without increasing data movement. Hypothesis: On a GPU architecture, we expect that these optimizations with their associated branches increase runtime because they cause thread divergence. On CPU, turning off the M ASK optimization consistently increases runtimes, as the workload on the CPU increases: this can be seen comparing A, B, C, D to E, F, G, H in Figure 4. Thus, our measurements confirm our hypothesis; M ASK is a beneficial optimization on CPU. Similarly, our measurements for GPU in Figure 5 show that the sparsity optimization M ASK also improves performance on GPU. The effect of the M ASK optimization can be seen by comparing A, B, C, D to E, F, G, H in Figure 5. In contrast to the CPU case, this similar result disqualifies our hypothesis; clearly, the branch condition is not complex enough for thread divergence to cause performance degradation. On Nvidia GPUs, the threads that are masked out wait to rejoin the warp after the active threads have completed their computations. Even in the worst case scenario, where every wavefront suffers from thread divergence, no additional work is done, so there is no loss of performance. Moreover, the distribution of masked threads is typically much better than the worst case because the computational domain is sparse (5-25% active cells) and exhibits high data locality due to the naturally clumped pattern of clouds. For this reason, the M ASK condition is homogeneous and completely on or completely off for entire thread warps, leading to an overall reduction of work for the GPU.

On GPU, the effects of the SIG when used in combination with M ASK are negligible. However, when M ASK is disabled (E-H in Figure 5), SIG is detrimental to performance, confirming our hypothesis, in part. On CPU, the effects of SIG are much more diverse, showing some dependence on problem size as well as CPU architecture. On Grace, removing the SIG optimization increases runtimes for the 20 and 40 km grids and has close to no effect otherwise, compare C to D in the middle row of Figure 4. On AMD, removing SIG decreases runtimes for the 10 and 20 km grids, and increases runtimes for the 40 km grid. An expressive framework is aware of minor implementation variants like M ASK and SIG, and can automatically toggle them on or off as needed for each execution environment. Implementing these variants is still up to the performance engineer, but transparently incorporating them into the simulation without increasing code complexity should be handled by the framework. F2DaCe and GT4Py take a step in this direction by making these variants invisible to the computational and domain scientists, thus keeping code complexity at a minimum. Disabling either of M ASK and SIG results in slight perturbations to the numerical results of the simulation compared to the reference. However, the reality is that the exploitation of sparsity in the Graupel reference implementation, and thus M ASK and SIG themselves, are optimizations. Masking out non-active cells and turning off physics for locally negligible processes results in an approximation of the true density field of microphysical species in the atmosphere. Thus, transforming the code by disabling M ASK or SIG, which are part of the reference, results in semantically equivalent implementations from the computational and domain scientists viewpoints. This underscores the need for collaboration in unlocking optimization opportunities. B. Improving access patterns Another optimization with sometimes counterintuitive effects concerns the ordering of a nested loop that appears in the last of the kernels in Graupel. The cache hierarchy and memory on CPU architectures generally leads to implementations that prioritize data locality, so we hypothesize that an arrangement with contiguous memory access in the inner loop would lead to the best performance. On GPU, memory loads are coalesced across warp threads, so the nested loops should be reordered. For the nested loop in question, the inner and outer iteration statements can be exchanged without changing the semantics of the computation, so we benchmark both variants. When this transformation, which we call F LIP, is applied, we expect better performance on GPU and worse performance on CPU. The F LIP transformation does improve performance on GPU, but not to the expected degree. For 40 km and 20 km grid sizes, performance does improve by up to 10%, but for the largest grid, the effect is negligible, compare C, D to A, B in Figure 5. Once again, this is different on the AMD GPU,

where the optimization introduces almost 20% speedup for 40 km and still reaches 10% speedup for the 10 km grid. Similar to our analysis of sparse optimizations in Section VI-A, more detailed analysis shows that in reality, the scientific code introduces additional factors which confound analytical performance predictions. Due to the kernel size, warp threads are starved of instructions, which can outweigh memory bandwidth bottlenecks on certain GPUs. C. Engineering effort The graupel microphysics component was implemented in each framework by users of the respective framework rather than by core developers or domain experts of the respective frameworks. This setup was chosen to better reflect a realistic adoption scenario, in which scientific application developers must work with new programming models and toolchains without direct access to the framework authors. For the OpenMP, SYCL, and F2DaCe implementations, the work was carried out by master’s students as part of their respective thesis projects, while OpenACC and C++ std::par implementations are based on initial versions created by students as part of the ISC Student Cluster Competition (SCC). For the F2DaCe implementation, which relied on a prototype version of the Fortran frontend, support from DaCe developers was limited to assistance with bug fixes in the tooling. Similarly, the GT4Py implementation received 3-4 person months of overall support, and some new Graupelspecific optimizations were added to the framework which is still evolving in its support for physical parameterizations. Beyond technical assistance, the objective was for each implementation to be developed as independently as possible by its assigned developer. Although a perfectly level playing field cannot be achieved in a study of this scale, particularly given the limited sample size and the differing maturity levels of the frameworks, all implementations were carried out under a common guiding principle: to obtain a best-effort solution within a reasonable development period. The intention was therefore not to compare idealized expert implementations, but rather to assess what can be achieved in practice by capable users working under realistic time constraints. Moreover, we can find one execution instance for each framework in which it outperforms all other implementations: OpenMP is best for MI300 single-precision but is outperformed by GT4Py on MI300 double-precision. On other GPUs, OpenACC outpaces the competitors for single-precision. Kokkos outperforms the others for single-precision on Grace, while SYCL takes the lead on the Intel Xeon CPU. Finally, F2DaCe leads the pack for double-precision on Grace. Thus, each framework is best at least once, regardless of differences in developer skill and invested time. The only exception is C++ std::par, which performs poorly across the board, mostly due to measurement problems (Section IV-E). While porting the applications to each framework took roughly similar amounts of work in each case, we observed

F2DaCe and GT4Py to be at an advantage when introducing additional variants as described in the introduction to Section VI-A. This is because the DaCe-based frameworks decouple fine-grained optimizations from implementation, enabling developers to make changes at a high level that are immediately reflected in the optimized executable. For other frameworks, the parallelization scheme is coupled to the code structure, so implementing changes like F LIP would require re-associating the relevant directives (OpenMP, OpenACC) or abstractions (Kokkos, SYCL, C++ std::par) with the correct loop body. However, DaCe and GT4Py are only robust against simple changes like SIG, F LIP, and M ASK; large scale restructuring will still cause brittle failures. In this way, F2DaCe and GT4Py are more expressive in decoupling fine-grained optimizations from high-level changes, allowing the computational scientist to evaluate the performance impact of high-level changes more rapidly. This is because in most cases, DaCe can simply carry over the finegrained optimizations introduced by the performance engineer without requiring any manual intervention. D. Framework expressiveness The challenge in predicting the actual performance impact across architectures of these reasonably simple optimizations highlights the need for performance portability frameworks to quickly and easily express such concepts, as only having the flexibility to produce multiple variants and test them can lead to finding the solution with the best performance. Thus, in an attempt to achieve excellent performance on every system, one would need to pick the most suitable from a number of implementations on every system. In many cases, this approach requires an infeasible engineering effort to develop implementations with all frameworks. Instead, a framework could allow the performance engineer to suggest, but not enforce possible performance optimizations, while exposing as many semantic constraints as are necessary to maintain correctness, but not more. The goal is to enable the performance engineer to be as expressive as possible when preparing the scientific application for the performance framework in order to enable the framework to best optimize for a target architecture. Kokkos and SYCL are expressive in that they let performance engineers encode semantic constraints through a range of abstract parallelization strategies, albeit at the cost of requiring domain and computational scientists to understand the application in those terms. GT4Py and F2DaCe similarly enable richer semantic information to reach the automatic optimization stage through their data-centric internal representations, while allowing scientific developers to continue working from a simpler source formulation. This greater expressiveness typically comes with substantially more complex internal machinery—as seen in Kokkos’ use of modern C++ compile-time mechanisms and, even more so, for GT4Py and F2DaCe, which effectively operate as compilers in their own right—but that complexity is precisely what can simplify

the overall development process and improve collaboration between domain scientists, computational scientists, and performance engineers. VII. R ELATED W ORK The tools covered in our work represent the state of the art when it comes to performance portability, given their ability to generate code for a majority of CPUs and GPUs used routinely in HPC.4 However, other tools, approaches, and explorations of performance portability exist. We list and discuss appropriate representatives here. Python provides Numba [43], an LLVM Just-in-Time compiler which transforms Numpy code into machine code for better performance. During the compilation process, Numba resolves pointer indirection and lowers a subset of high-level Python and Numpy features into efficient loops with performance comparable to other low-level compiled languages. In particular, Numba is able to apply aggressive transformations by limiting the scope of supported features and deviating from Python semantics. [43] Superficially, this is similar to the GT4Py DSL which uses DaCe SDFGs as an IR instead of Numba’s LLVM MLIR, and which has been specialized and optimized for stencil-based computations. Similar to our GT4Py implementation, using Numba would still require a complete re-write of the Graupel code in Python. Given that Numba provides less opportunities for influencing the optimization process, we expect performance to be at most comparable with GT4Py. In lieu of DSL features, a Numba port may also require more lines of code to achieve the same optimized stencil operations which are exposed as a language feature of GT4Py. The Alpaka [44] and RAJA [45] frameworks provide a C++11 abstraction layer for hierarchical parallelism. In this regard, they are similar to Kokkos and SYCL, though they require no specialized compiler, unlike the latter. For these reasons, we expect performance and implementation effort in RAJA and Alpaka to be comparable to Kokkos and SYCL. Added functionality, like loop chain abstractions [46] in RAJA, can offer additional performance benefits. The hardware vendors Nvidia and AMD offer the CUDA and HIP programming languages that can be used to implement kernels for their respective accelerators. These languages offer the most flexibility when it comes to extracting performance, but the engineering effort required to write an optimized program grows proportionally. Moreover, CUDA and HIP prioritize their respective accelerators5 and do not produce portable software outside a narrow range of devices. For these reasons, we do not compare our results against a CUDA/HIP baseline. Nonetheless, the DaCe backend for GPU generates CUDA code, so this can be interpreted as a baseline. 4 That is, with the exception of C++ std::par, which is young compared to the other frameworks and is limited by compiler support, enjoying no support on AMD GPUs. 5 Although HIP provides utilities to convert HIP from CUDA and even run HIP code on Nvidia devices.

Comparable initiatives to our own exist to evaluate and compare performance portability frameworks. Harrell et al. [47] develop a tool to facilitate the calculation and collection of performance portability data. Using their tool, they explore the effectiveness of multiple performance portability frameworks including OpenMP, Kokkos, and CUDA. Additionally, they characterize development effort using a subjective task load index [48]. Artigues et al. [49] reimplement a Particle-In-Cell (PIC) kernel in OpenMP, OpenACC, CUDA, Kokkos, and RAJA targeting CPU and GPU, finally turning to an in-depth discussion of the usability and performance of Kokkos and RAJA. Marowka [50] evaluates 324 case studies of OpenACC, OpenMP, Kokkos, and RAJA based on a novel performance portability metric. Deakin et al. [51] develop BabelStream, a framework to evaluate the maximum memory bandwidth that can be attained using different programming models, including Kokkos, OpenMP, OpenACC, and SYCL. Daniel et al. [52] benchmark various applications ported to Kokkos and OpenACC on CPU and on GPU, highlighting that the optimal framework depends on problem size, among other factors. This supports our results, driving home the idea that multiple implementations are required to extract optimal performance from an arbitrary system. Focusing on the effort needed to port codes to various frameworks, Heroux et al. [53] offer a comprehensive retrospective on portability frameworks used to port codes to GPU within the Exascale Computing Project (ECP). In a similar vein, Dubey et al. [54] give a perspective on the adoption of frameworks like OpenMP, Kokkos, and RAJA in ECP projects. The authors highlight the importance of close collaboration between domain scientists and computer science experts. Many publications inhabiting specific domains, like Roccon et al. [55] for computational fluid dynamics, Bauer et al. [1] for earth-system science, or Abdulah et al. [56] for statistical computing recognize the need for portability but focus on domain-specific challenges and requirements. To the best of our knowledge, the current literature lacks a discussion of framework expressivity and how it ties into development effort at all levels. VIII. C ONCLUSION The clear takeaway from our experiments is that portable applications must flexibly exchange optimizations in order to adapt to different hardware configurations and problem configurations to extract the highest possible performance. What remains to be investigated is how to achieve this level of expressivity, which is not yet offered by any existing framework. In addition to this, porting codes to any framework efficiently is an open research problem, especially when the framework in question is specially designed to make use of multiple implementation variants for some kernels. In a similar vein, Holmen et al. [57] propose an approach for adopting portability frameworks in a piecewise fashion. Building on these ideas, further research is required to work out the details of porting existing code to a framework without reimplementing and testing many sections of the algorithm in

different variants to achieve sufficient expressivity, regardless of human work hours or LLM tokens spent. Our evaluation shows that, despite relying on markedly different programming models, abstraction mechanisms, and optimization strategies, all considered performance-portability frameworks are ultimately capable of approaching a similar performance ceiling for the Graupel kernel on modern systems. This suggests that no single framework holds a fundamental monopoly on achievable performance. Rather, the main differences lie in how that performance is obtained: through different balances of manual intervention, compiler support, abstraction design, and framework-specific optimization opportunities. At the same time, our results indicate that these differences have important consequences for the development process itself. The effort required to obtain high performance, and the way this effort is distributed among domain scientists, computational scientists, and performance engineers, varies substantially across frameworks. Some approaches demand that semantic restructuring and optimization intent be made explicit early in the implementation, increasing the burden on those expressing the scientific kernel, while others defer more of this work to intermediate representations, compiler technology, or downstream transformation stages. Consequently, the choice of framework should not be assessed solely in terms of runtime performance, but also in terms of how it shapes collaboration, maintainability, and the division of expertise required to bring scientific applications efficiently onto new architectures. ACKNOWLEDGMENT We gratefully acknowledge Hussein Harake and the Swiss National Supercomputing Centre (CSCS) for providing access to computing resources via the RACKlette student cluster, which was instrumental to this work. The authors acknowledge the ICON partnership — including the German Weather Service (DWD), the Max Planck Institute for Meteorology, the German Climate Computing Center (DKRZ), the Karlsruhe Institute of Technology (KIT), MeteoSwiss, and ETH Zurich — for their continued development and improvement of the ICON model. The work of many members of this partnership over the years has laid the foundation for the results presented in this study. The further development of ICON for the efficient use of exascale HPC systems was provided by the German Federal Ministry of Research, Technology and Space (BMFTR) funded project WarmWorld, under grant number 01LK2203.This work used resources of the Deutsches Klimarechenzentrum (DKRZ) granted by its Scientific Steering Committee (WLA) under project ID bk1341. This work was supported by the WeatherGenerator project (grant agreement No. 101187947). This work was supported by the Swiss National Science Foundation (SNSF) through the National Centre of Competence in Research (NCCR CLIM+ – Climate Extremes and Society).

R EFERENCES [1] P. Bauer, P. D. Dueben, T. Hoefler, T. Quintino, T. C. Schulthess, and N. P. Wedi, “The digital revolution of earth-system science,” Nature Computational Science, vol. 1, no. 2, pp. 104–113, 2021. [Online]. Available: https://doi.org/10.1038/s43588-021-00023-0 [2] P. Bauer, A. Thorpe, and G. Brunet, “The quiet revolution of numerical weather prediction,” Nature, vol. 525, no. 7567, pp. 47–55, 2015. [Online]. Available: https://doi.org/10.1038/nature14956 [3] J. Shalf, “The future of computing beyond moore’s law,” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 378, no. 2166, 2020. [Online]. Available: https://doi.org/10.1098/rsta.2019.0061 [4] W. L. Oberkampf and T. G. Trucano, “Verification and validation in computational fluid dynamics,” Progress in Aerospace Sciences, vol. 38, no. 3, pp. 209–272, 2002. [Online]. Available: https: //doi.org/10.1016/S0376-0421(02)00005-2 [5] S. Rüdisühli, A. Walser, and O. Fuhrer, “Cosmo in single precision,” COSMO News Letter, vol. 14, no. 5.1, pp. 70–87, 2014, no DOI available. [Online]. Available: https://www.cosmo-model.org/content/m odel/documentation/newsLetters/newsLetter14/cnl14 09.pdf [6] P. D. Düben, A. Subramanian, A. Dawson, and T. N. Palmer, “A study of reduced numerical precision to make superparameterization more competitive using a hardware emulator in the OpenIFS model,” Journal of Advances in Modeling Earth Systems, vol. 9, no. 1, pp. 566–584, Mar. 2017. [Online]. Available: http://dx.doi.org/10.1002/2016MS000862 [7] W. F. van Gunsteren, X. Daura, N. Hansen, A. E. Mark, C. Oostenbrink, S. Riniker, and L. J. Smith, “Validation of molecular simulation: An overview of issues,” Angewandte Chemie International Edition, vol. 57, no. 4, pp. 884–902, 2018. [Online]. Available: https://doi.org/10.1002/anie.201702945 [8] H. Banderier, C. Zeman, D. Leutwyler, S. Rüdisühli, and C. Schär, “Reduced floating-point precision in regional climate simulations: An ensemble-based statistical verification,” EGUsphere, vol. 2023, p. 1–18, 2023. [Online]. Available: https://doi.org/10.5194/egusphere-2023-2263 [9] A. Sameh, G. Cybenko, M. Kalos, K. Neves, J. Rice, D. Sorensen, and F. Sullivan, “Computational science and engineering,” ACM Comput. Surv., vol. 28, no. 4, p. 810–817, Dec. 1996. [Online]. Available: https://doi.org/10.1145/242223.246865 [10] S. Cunningham, S. C. Pulliam, C. D. Swanson, and P. R. Turner, “Computational science and engineering: tools and techniques for teaching,” SIGCSE Bull., vol. 34, no. 1, p. 135–136, Feb. 2002. [Online]. Available: https://doi.org/10.1145/563517.563393 [11] OpenMP Architecture Review Board, “Openmp application programming interface version 4.0,” OpenMP Architecture Review Board, Tech. Rep., 2013, no DOI available. [Online]. Available: https://www.openmp.org/wp-content/uploads/OpenMP4.0.0.pdf [12] OpenACC Organization, The OpenACC® Application Programming Interface, Version 3.4, May 2025, accessed: 2026-04-01. [Online]. Available: https://www.openacc.org/specification [13] C. R. Trott, D. Lebrun-Grandié, D. Arndt, J. Ciesko, V. Dang, N. Ellingwood, R. Gayatri, E. Harvey, D. S. Hollman, D. Ibanez, N. Liber, J. Madsen, J. Miles, D. Poliakoff, A. Powell, S. Rajamanickam, M. Simberg, D. Sunderland, B. Turcksin, and J. Wilke, “Kokkos 3: Programming model extensions for the exascale era,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 4, pp. 805–817, 2022. [Online]. Available: https://doi.org/10.1109/TPDS.2021.3097283 [14] R. Reyes, G. Brown, R. Burns, and M. Wong, “Sycl 2020: More than meets the eye,” in Proceedings of the International Workshop on OpenCL, ser. IWOCL ’20. New York, NY, USA: Association for Computing Machinery, 2020. [Online]. Available: https://doi.org/10.1145/3388333.3388649 [15] E. G. Paredes, L. Groner, S. Ubbiali, H. Vogt, A. Madonna, K. Mariotti, F. Cruz, L. Benedicic, M. Bianco, J. VandeVondele, and T. C. Schulthess, “Gt4py: High performance stencils for weather and climate applications using python,” 2023. [Online]. Available: https://arxiv.org/abs/2311.08322 [16] P. Adamidis, E. Pfister, H. Bockelmann, D. Zobel, J.-O. Beismann, and M. Jacob, “The real challenges for climate and weather modelling on its way to sustained exascale performance: a case study using icon (v2.6.6),” Geoscientific Model Development, vol. 18, no. 4, pp. 905–919, 2025. [Online]. Available: https://doi.org/10.5194/gmd-18-905-2025

[17] ICON partnership (DWD and MPI-M and DKRZ and KIT and C2SM), “Icon release 2025.10,” 2025. [Online]. Available: https: //doi.org/10.35089/WDCC/IconRelease2025.10 [18] R. Fiévet, L. Linardakis, L. Kornblueh, M. Daniel-Lacombe, H. M. Gloeckner, T. Fiolleau, L. Kluft, C. Hohenegger, and B. Stevens, “Tropical convection and storm-resolving modeling: a case study based on the orcestra campaign,” ESS Open Archive, vol. 2026, no. 0310, 2026. [Online]. Available: https://doi.org/10.22541/essoar.177315810. 09022965/v1 [19] G. Zängl, D. Reinert, P. Rı́podas, and M. Baldauf, “The icon (icosahedral non-hydrostatic) modelling framework of dwd and mpi-m: Description of the non-hydrostatic dynamical core,” Quarterly Journal of the Royal Meteorological Society, vol. 141, no. 687, pp. 563–579, 2015. [Online]. Available: https://doi.org/10.1002/qj.2378 [20] D. Klocke, C. Frauen, J. F. Engels, D. Alexeev, R. Redler, R. Schnur, H. Haak, L. Kornblueh, N. Brüggemann, F. Chegini, M. Römmer, L. Hoffmann, S. Griessbach, M. Bode, J. Coles, M. Gila, W. Sawyer, A. Calotoiu, Y. Budanaz, P. Mazumder, M. Copik, B. Weber, A. Herten, H. Bockelmann, T. Hoefler, C. Hohenegger, and B. Stevens, “Computing the full earth system at 1km resolution,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 125–136. [Online]. Available: https://doi.org/10.1145/3712285.3771789 [21] C. J. A. Bayley, A. K. Naumann, F. Poydenot, R. Vogel, B. Stevens, and S.-I. Shima, “Cleo: The numerical methods of a new superdroplet model including a droplet breakup algorithm,” EGUsphere, vol. 2025, pp. 1–26, 2025. [Online]. Available: https://doi.org/10.5194/egusphere-2025-4399 [22] C. J. A. Bayley, T. Kölling, A. K. Naumann, R. Vogel, and B. Stevens, “Cleo: The fundamental design for high computational performance of a new superdroplet model,” EGUsphere, vol. 2025, pp. 1–33, 2025. [Online]. Available: https://doi.org/10.5194/egusphere-2025-4398 [23] A. Herten, “Many cores, many models: Gpu programming model vs. vendor compatibility overview,” in Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, ser. SC-W ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 1019–1026. [Online]. Available: https://doi.org/10.1145/3624062.3624178 [24] T. Kadosh, N. Hasabnis, T. Mattson, Y. Pinter, and G. Oren, “Quantifying openmp: Statistical insights into usage and adoption,” in 2023 IEEE High Performance Extreme Computing Conference (HPEC). Boston, MA, USA: IEEE, Sept 2023, p. 1–7. [Online]. Available: https://doi.org/10.1109/HPEC58863.2023.10363459 [25] A. Jarmusch, F. Cabarcas, S. Pophale, A. Kallai, J. Doerfert, L. Peyralans, S. Lee, J. Denny, and S. Chandrasekaran, “Ci/cd efforts for validation, verification and benchmarking openmp implementations,” in Advancing OpenMP for Future Accelerators, A. Espinosa, M. Klemm, B. R. de Supinski, M. Cytowski, and J. Klinkenberg, Eds. Cham: Springer Nature Switzerland, 2024, pp. 111–125. [Online]. Available: https://doi.org/10.1007/978-3-031-72567-8 8 [26] T. Huber, S. Pophale, N. Baker, M. Carr, N. Rao, J. Reap, K. Holsapple, J. H. Davis, T. Burnus, S. Lee, D. E. Bernholdt, and S. Chandrasekaran, “Ecp sollve: Validation and verification testsuite status update and compiler insight for openmp,” in 2022 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). Dallas, TX, USA: IEEE, Nov. 2022, p. 123–135. [Online]. Available: https://doi.org/10.1109/P3HPC56579.2022.00017 [27] T. Ben-Nun, J. De Fine Licht, A. N. Ziogas, T. Schneider, and T. Hoefler, “Stateful dataflow multigraphs: a data-centric model for performance portability on heterogeneous architectures,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. Denver Colorado: ACM, Nov. 2019, pp. 1–14. [Online]. Available: https://doi.org/10.1145/3295500.3356173 [28] A. N. Ziogas, T. Schneider, T. Ben-Nun, A. Calotoiu, T. De Matteis, J. De Fine Licht, L. Lavarini, and T. Hoefler, “Productivity, portability, performance: data-centric Python,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. St. Louis Missouri: ACM, Nov. 2021, pp. 1–13. [Online]. Available: https://doi.org/10.1145/3458817.3476176 [29] T. Ben-Nun, L. Groner, F. Deconinck, T. Wicky, E. Davis, J. Dahm, O. D. Elbert, R. George, J. McGibbon, L. Trümper, E. Wu, O. Fuhrer, T. Schulthess, and T. Hoefler, “Productive performance engineering for weather and climate modeling with python,” in Proceedings of the International Conference on High Performance Computing,

Networking, Storage and Analysis, ser. SC ’22. IEEE Press, 2022, (DOI 10.5555/3571885.3571982 does not seem to be registered beyond ACM). [Online]. Available: https://dl.acm.org/doi/10.5555/3571885.3571982 [30] O. Rausch, T. Ben-Nun, N. Dryden, A. Ivanov, S. Li, and T. Hoefler, “A data-centric optimization framework for machine learning,” in Proceedings of the 36th ACM International Conference on Supercomputing. Virtual Event: ACM, Jun. 2022, pp. 1–13. [Online]. Available: https://doi.org/10.1145/3524059.3532364 [31] A. Boudaoud, A. Calotoiu, M. Copik, and T. Hoefler, “DaCe AD: Unifying High-Performance Automatic Differentiation for Machine Learning and Scientific Computing,” Sep. 2025, arXiv:2509.02197. [Online]. Available: https://doi.org/10.48550/arXiv.2509.02197 [32] P. Schaad, T. Ben-Nun, and T. Hoefler, “C.A.T.S.: Memory and Control Flow Tracing for Whole-Program Performance Analysis,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. St. Louis MO USA: ACM, Nov. 2025, pp. 331–348. [Online]. Available: https: //doi.org/10.1145/3712285.3759797 [33] A. N. Ziogas, T. Ben-Nun, G. I. Fernández, T. Schneider, M. Luisier, and T. Hoefler, “A data-centric approach to extreme-scale ab initio dissipative quantum transport simulations,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. Denver Colorado: ACM, Nov. 2019, pp. 1–13. [Online]. Available: https://doi.org/10.1145/3295500.3357156 [34] A. Afanasyev, M. Bianco, L. Mosimann, C. Osuna, F. Thaler, H. Vogt, O. Fuhrer, J. VandeVondele, and T. C. Schulthess, “Gridtools: A framework for portable weather and climate applications,” SoftwareX, vol. 15, p. 100707, 2021. [Online]. Available: https: //doi.org/10.1016/j.softx.2021.100707 [35] A. Kashyap, Tuning Guide for AMD EPYC™7002 Series Processors, 56827th ed., Advanced Micro Devices, January 2020, accessed: 2026-09-17. [Online]. Available: https://web.archive.org/web/20221006 233837/https://developer.amd.com/wp-content/resources/56827-1-0.pdf [36] C. Lam, “Pushing amd’s infinity fabric to its limits,” November 2024, accessed: 2026-09-17. [Online]. Available: https://web.archive.org/web/ 20260909084536/https://chipsandcheese.com/p/pushing-amds-infinity-f abric-to-its [37] High Performance Computing (HPC) Tuning Guide for AMD EPYC™7003 Series Processors, 57091st ed., Advanced Micro Devices, March 2022, accessed: 2026-09-17. [Online]. Available: https://docs.amd.com/v/u/en-US/high-performance-computing-tuning-g uide-amd-epyc7003-series-processors [38] A. Kumar, New Intel®Mesh Architectyure: The ’Superhighway’ of the Data Center, Intel, 2023, accessed: 2026-09-17. [Online]. Available: https://web.archive.org/web/20231202191421/https://silix.com.br/pdf/I ntel/Intel Mesh Whitepaper.pdf [39] J. Evans, I. Finder, I. Goldwasser, J. Linford, V. Mehta, D. Ruiz, and M. Wagner, “Nvidia grace cpu superchip architecture in depth,” Nvidia Corporation, Tech. Rep., January 2023, accessed: 2026-09-17. [Online]. Available: https://web.archive.org/web/20260511182729/https://develo per.nvidia.com/blog/nvidia-grace-cpu-superchip-architecture-in-depth/ [40] M. HadiBrais, “What is the data width of the mesh in skx?” January 2020, accessed: 2026-09-17. [Online]. Available: https: //community.intel.com/t5/Software-Tuning-Performance/What-is-the-d ata-width-of-the-mesh-in-SKX/m-p/1181488 [41] Intel®Xeon®Processor Scalable Memory Family Uncore Performance Monitoring Reference Manual, 671389th ed., Intel, July 2017, accessed: 2026-09-17. [Online]. Available: https://web.archive.org/web/20231202 191421/https://silix.com.br/pdf/Intel/Intel Mesh Whitepaper.pdf [42] WIN11, “Intel xeon platinum 8168 @ 3392.36 mhz,” July 2025, accessed: 2026-09-17. [Online]. Available: https://valid.x86.fr/jreptz [43] S. K. Lam, A. Pitrou, and S. Seibert, “Numba: a llvm-based python jit compiler,” in Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC, ser. LLVM ’15. New York, NY, USA: Association for Computing Machinery, 2015. [Online]. Available: https://doi.org/10.1145/2833157.2833162 [44] E. Zenker, B. Worpitz, R. Widera, A. Huebl, G. Juckeland, A. Knupfer, W. E. Nagel, and M. Bussmann, “Alpaka – an abstraction library for parallel kernel acceleration,” in 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, May 2016, p. 631–640. [Online]. Available: http://dx.doi.org/10.1109/IPDPSW.2016.50 [45] D. A. Beckingsale, T. R. Scogland, J. Burmark, R. Hornung, H. Jones, W. Killian, A. J. Kunen, O. Pearce, P. Robinson,

and B. S. Ryujin, “Raja: Portable performance for large-scale scientific applications,” in 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). Denver, CO, USA: IEEE, Nov 2019, p. 71–81. [Online]. Available: https://ieeexplore.ieee.org/document/8945721/

[50] A. Marowka, “On the performance portability of openacc, openmp, kokkos and raja,” in International Conference on High Performance Computing in Asia-Pacific Region, ser. HPCAsia ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 103–114. [Online]. Available: https://doi.org/10.1145/3492805.3492806

63

42

Nvidia F2DaCe

10 km

0.4

20 km

0 1.2 0.8 0.4 0 1.2 0.8 0.4

Double precision

(2 )

)

ra c

e

(1

68

G

ra G

X eo

n

ce

81

77 C

C EP Y

EP Y

ce G ra

ra G

77

(2 )

) (1

8

ce

81 6 n

77 42

X eo

Y C

77

63

0 C

[49] V. Artigues, K. Kormann, M. Rampp, and K. Reuter, “Evaluation of performance portability frameworks for the implementation of a particle-in-cell code,” Concurrency and Computation: Practice and Experience, vol. 32, no. 11, p. e5640, 2020, e5640 cpe.5640. [Online]. Available: https://doi.org/10.1002/cpe.5640

Intel Kokkos

0.8

EP

[48] S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” in Human Mental Workload, ser. Advances in Psychology, P. A. Hancock and N. Meshkati, Eds. North-Holland, 1988, vol. 52, pp. 139–183. [Online]. Available: https://doi.org/10.1016/S0166-4115(08)62386-9

LLVM

1.2

40 km

[47] S. L. Harrell, J. Kitson, R. Bird, S. J. Pennycook, J. Sewall, D. Jacobsen, D. N. Asanza, A. Hsu, H. C. Carrillo, H. Kim, and R. Robey, “Effective performance portability,” in 2018 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). Dallas, TX, USA: IEEE, Nov. 2018, p. 24–36. [Online]. Available: https://doi.org/10.1109/P3HPC.2018.00006

GNU OpenMP

EP Y

[46] B. Neth, T. R. W. Scogland, B. R. de Supinski, and M. M. Strout, “Inter-loop optimization in raja using loop chains,” in Proceedings of the 35th ACM International Conference on Supercomputing, ser. ICS ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 1–12. [Online]. Available: https://doi.org/10.1145/3447818.3461665

CPU relative runtimes

Single precision

Fig. 6. Comparison of compiler-influenced performance variance for CPU. Bars show the relative difference between fastest and slowest compilers: ((max − min)/min = max/min − 1). Only OpenMP, Kokkos, and F2DaCe have a free choice of compiler on CPU (see Table II). Given that tallest bars show a relative difference of about 1, this plot confirms that overall, the best results have about half the runtime of the slowest.

[51] T. Deakin, J. Price, M. Martineau, and S. McIntosh-Smith, “Evaluating attainable memory bandwidth of parallel programming models via babelstream,” International Journal of Computational Science and Engineering, vol. 17, p. 247, 01 2018. [Online]. Available: https://doi.org/10.1504/IJCSE.2018.095847 [52] D. F. Daniel and J. Panetta, “On applying performance portability metrics,” in 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). Denver, CO, USA: IEEE, Nov. 2019, p. 50–59. [Online]. Available: https: //ieeexplore.ieee.org/document/8945591/

[56] S. Abdulah, M. L. O. Salvaña, Y. Sun, D. E. Keyes, and M. G. Genton, “High-performance statistical computing ( hpsc ): Challenges, opportunities, and future directions,” WIREs Computational Statistics, vol. 17, no. 4, p. e70052, Dec 2025. [Online]. Available: https://wires.onlinelibrary.wiley.com/doi/10.1002/wics.70052 [57] J. K. Holmen, B. Peterson, and M. Berzins, “An approach for indirectly adopting a performance portability layer in large legacy codes,” in 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC). Denver, CO, USA: IEEE, Nov. 2019, p. 36–49. [Online]. Available: https://doi.org/10.1109/P3HPC49587.2019.00009

Intel

Nvidia

ACPP F2DaCe

17.0

0.4

SYCL

0.1

12.3

0.04

0.4

10.1

0 0.06

14.0

0.02

8.1

0 0.06 0.04

0.3

0.02

11.1

10 km

0.04

0.02

Double precision

(1 ) er

00

op p

H

M I3

G B

A 10

0

80 10 0 A

40

G B

) (1

00 H

op

pe r

M I3

B 40 G

10 0 A

10 0

80 G

B

0

A

[55] A. Roccon, G. Amati, L. Brandt, D. Calhoun, P. Costa, W. Lu, S. Pirozzoli, D. Richter, M. Umair, D. You, T. Zahtila, and C. Marchioli, “Gpu-accelerated simulations of turbulence: Review of current applications and future perspectives,” Physical Review Fluids, vol. 11, no. 3, p. 034905, Mar 2026. [Online]. Available: https://link.aps.org/doi/10.1103/vz9c-bbzm

0.06

20 km

[54] A. Dubey, L. C. McInnes, R. Thakur, E. W. Draeger, T. Evans, T. C. Germann, and W. E. Hart, “Performance portability in the exascale computing project: Exploration through a panel series,” Computing in Science & Engineering, vol. 23, no. 5, p. 46–54, Sept 2021. [Online]. Available: https://ieeexplore.ieee.org/document/9495114/

GNU Kokkos

40 km

[53] M. A. Heroux, L. C. McInnes, J. Ahrens, T. Gamblin, T. C. Germann, X. S. Li, K. Mohror, T. Munson, S. Shende, R. Thakur, J. Vetter, and J. Willenbring, “Ecp libraries and tools: An overview,” The International Journal of High Performance Computing Applications, p. 10943420241271005, Sept 2024. [Online]. Available: https: //journals.sagepub.com/doi/10.1177/10943420241271005

GPU runtimes (ms)

Single precision

Fig. 7. Comparison of compiler-influenced performance variance for GPU. Bars show the relative difference between fastest and slowest compilers: ((max − min)/min = max/min − 1). Only Kokkos, SYCL, and F2DaCe have a free choice of compiler on GPU (see Table II). The relative difference between compilers is minimal compared to to CPU (Figure 6), staying mostly below 5%. The exception is formed by ACPP, which performs much more poorly than Intel.

TABLE II TABLE SHOWING COMPILERS TESTED FOR EACH IMPLEMENTATION ON EACH SYSTEM . T HE ABBREVIATIONS G, C, I, N, A CORRESPOND TO THE GCC, LLVM, I NTEL , N VIDIA , AND A DAPTIVE CPP COMPILERS .

Framework

x86 64 EPYC, Xeon G

C

I

N

aarch64 Grace A

G

C

I

sm 80 A100 N

A

G

C

I

sm 90 Hopper N

A

G

C

I

gfx942 MI300A N

A

G

C

I

N

A

OpenMP × × × × × × × × OpenACC × × Kokkos × × × × × × × × × × × × ∗ SYCL × × × × × × × std::par × × × × F2DaCe × × × × × × × × × × × × GT4Py × × × × × ∗ On ALPS/Santis, the legacy Intel DPC++ compiler is still available. At the time of writing, this compiler and the accompanying PTX plugin are neither supported nor served, making a new install on Jupiter Booster impossible. TABLE III C OMPILER FLAGS USED FOR EACH COMPILER AND IMPLEMENTATION . F LAGS PRINTED IN BOLD ARE NOT AVAILABLE ON AARCH 64. Compiler G C I N A

GCC LLVM Intel Nvidia ACPP

G C I N A

GCC LLVM Intel Nvidia ACPP

Flags Performance -O3 -ffast-math -O3 -ffast-math -ffp-contract=on -std=c++17 -march=native -fuse-ld=lld -O3 -std=c++17 -fma -march=core-avx2 -mtune=core-avx2 -O3 -std=c++17 -fast -O3 -ffast-math -ffp-contract=on -march=native Correctness -g -O0 -mpc-64 -g -O0 -fuse-ld=lld -g -O0 -fp-model=precise -m64 -g -O0 -nofma -Mnofprelaxed -Msingle -g -O0

TABLE IV S UMMARY OF CLUSTERS USED FOR BENCHMARKS IN THE PAPER . T HE TABLE INCLUDES : SYSTEM NAME , CPU HARDWARE , GPU HARDWARE , COMPILER VERSIONS FOR GCC, LLVM, O NE API, AND NVHPC. N/A IS GIVEN FOR ALPS/B EVERIN CPU H ARDWARE AND FOR J UWELS C LUSTER GPU H ARDWARE BECAUSE THOSE COMBINATIONS WERE NOT USED . F URTHERMORE , THE I NTEL O NE API COMPILER SUITE IS NOT AVAILABLE FOR ARM- BASED HARDWARE (ALPS/B EVERIN , ALPS/S ANTIS , J UPITER B OOSTER ). A DAPTIVE C PP ON ALPS/B EVERIN USES LLVM 18.1.0.

System Name

CPU Hardware

Levante AMD EPYC 7763 Jureca DC AMD EPYC 7742 ALPS/Beverin N/A ALPS/Santis Nvidia Grace Juwels Cluster Intel Xeon Platinum 8168 Jupiter Booster Nvidia Grace a 18.1.0 for AdaptiveCpp

GPU Hardware

GCC

LLVM

Compiler versions OneAPI NVHPC

Nvidia A100 SXM4 80GB Nvidia A100 SXM 40GB AMD Instinct MI300 Nvidia GH200 Hopper N/A Nvidia GH200 Hopper

12.3.0 13.3.0 13.4.0 13.2.0 13.3.0 13.3.0

17.0.4 18.1.8 20.0.0a N/A 20.1.8 18.1.8

2023.2.1 2024.2.0 N/A N/A 2025.2.0 N/A

TABLE V V ERSIONS FOR SOFTWARE DEPENDENCIES OF OUR TOOLCHAIN . T HE LAST COLUMN INDICATES CUDA VERSIONS FOR EVERY CLUSTER EXCEPT ALPS/B EVERIN , WHERE THE ROCM VERSION IS GENERALLY 7.1.0, BUT 6.3.3 FOR A DAPTIVE C PP ONLY.

System name

CMake

Levante 2.26.3 Jureca-DC 3.30.3 ALPS/Beverin 3.28.3 ALPS/Santis 3.27.9 Juwels Cluster 3.30.3 Jupiter Booster 3.29.3 a 6.3.3 for AdaptiveCpp

Software versions NetCDF C++ CDO CUDA/ROCm 4.3.1 4.3.1 4.3.1 4.3.1 4.3.1 4.3.1

2.2.2 2.6.0 N/A 2.4.0 2.6.0 2.6.0

12.5 12.6 7.1.0a 12.6 N/A 12.6

24.7 24.9 N/A 25.1 24.9 24.9

AdaptiveCpp N/A 25.10 25.10 25.10 N/A 25.10

A PPENDIX A C OMPILERS Table II shows the compiler-implementation combinations that were considered for each hardware platform. The compilation flags for each compiler are collected in Table III, while Table IV shows the version numbers for compilers on each system. For completeness, Table V gives the version numbers for software dependencies. Figures 6 and 7 visualize the differences in performance influenced by compiler choice. Finally, Table VI list the cases in which a particular framework is best-in-class for our experiment. This information is extracted from Figures 2 and 3.

TABLE VI I N WHICH EXECUTION INSTANCES DOES A FRAMEWORK OFFER BEST- IN - CLASS PERFORMANCE ? T HIS TABLE LISTS EXECUTION INSTANCES IN WHICH A PARTICULAR IMPLEMENTATION PERFORMS BETTER THAN THE OTHERS . N OTABLY, THERE IS AT LEAST ONE CASE FOR EACH IMPLEMENTATION , EXCEPT C++ S T D :: P A R . F URTHERMORE , THE LISTS ARE NOT COMPREHENSIVE , E . G . SYCL IS ALSO BEST- IN - CLASS FOR DOUBLE PRECISION ON EPYC 7742.

Implementation

Best-in-class performance for:

OpenMP OpenACC Kokkos SYCL C++ std::par F2DaCe GT4Py

MI300, single precision, all resolutions A100, Hopper single precision, all resolutions Grace (2), single precision, R2B6 Xeon 8168, single precision, all resolutions N/A, see Section IV-E Grace, double precision, all resolutions MI300, double precision, all resolutions

Related documents

• Twelve quick tips for designing AI-driven HPC workflows#793719
• Twelve quick tips for designing AI-driven HPC workflows#789715
• Adaptation Fidelity of SPEC CPU2026#781007
• Adaptation Fidelity of SPEC CPU2026#779632
• Performance modeling and optimization techniques for heterogeneous computing#957827