Enhancing Performance Insight at Scale: A Heterogeneous Framework for Exascale Diagnostics Dragana Grbic
arXiv:2605.03561v1 [cs.DC] 5 May 2026
Department of Computer Science Rice University Houston, TX [email protected]
Abstract—As exascale systems reach unprecedented concurrency, traditional performance analysis tools struggle with the overhead of massive-scale telemetry. We present an accelerated infrastructure for the hpcanalysis framework that leverages a high-performance C++ API and GPU parallelism to enable high-throughput diagnostics. Our C++ API achieves a 9.69-second ingestion time for 100,000 MPI ranks on Aurora. Furthermore, our GPU-accelerated layer achieves up to 314× speedup over CPU-based processing when analyzing 100,000 execution traces. Finally, we implement a topology-aware workflow that maps logical performance outliers to physical Slingshot interconnect coordinates, localizing network congestion across 22 distinct racks on Aurora. We also demonstrate how the framework’s advanced interface seamlessly integrates with external tools to provide sophisticated analytical models. We introduce a novel tri-dimensional performance model that ”re-materializes” iterative behavior from execution traces; using this model, we identified a 32.28% potential speedup for a GAMESS workload on Frontier. Index Terms—Exascale computing, Performance analysis, Heterogeneous frameworks, GPU acceleration, Interconnect congestion, Iteration-aware modeling.
GPU code. While HPCToolkit provides the hpcviewer [6] Graphical User Interface (GUI) for performance presentation, the recently introduced hpcanalysis framework [8] was specifically designed to enable scalable, efficient, and interactive analysis of exascale measurements. Exascale measurements can reach massive scales, rendering manual GUI-based inspection time-consuming and often impractical. In this paper, we focus on further enhancing the hpcanalysis framework to support interactive, highthroughput analysis of large-scale applications. Specifically, we refine the initial architecture to increase data ingestion speed, leverage GPU-accelerated parallelism to offload the bulk of the computational analysis to specialized hardware, and integrate hpcanalysis with external tools to enable advanced diagnostics. Finally, we demonstrate how this enhanced infrastructure efficiently interprets application performance at the full scale of modern supercomputers. The paper makes the following contributions: Heterogeneous Infrastructure: We present an accelerated C++ and GPU-based substrate for the hpcanalysis framework, enabling high-throughput ingestion and analysis of exascale performance measurements. • Tri-dimensional Modeling: We introduce a novel performance model to ”re-materialize” iterative behavior from execution traces, enabling the analysis of fine-grained imbalances and patterns typically lost in aggregate profiles. • Iteration-aware Diagnostics: We demonstrate finegrained analysis of GPU load imbalance in production workloads with repetitive behavior, providing mechanisms to predict significant performance gains. • Exascale Topology Mapping: We evaluate our accelerated infrastructure on 100,000 MPI ranks on Aurora, localizing logical performance outliers to physical Slingshot interconnect coordinates to identify network congestion. •
I. I NTRODUCTION Analyzing application performance on heterogeneous platforms remains a formidable challenge. While the HighPerformance Computing (HPC) community has developed numerous tools for measuring and analyzing the performance of parallel scientific applications, many struggle to interpret the behavior of complex codes within critical scientific domains. These applications are particularly difficult to analyze because their interpretation requires orchestration across disparate parallel execution contexts. A primary hurdle lies in the sheer complexity of exascale platforms, where applications exhibit extreme levels of heterogeneity—spanning thousands of MPI tasks, intensive CPU multi-threading, and offloading heavy computations to GPUs. Exascale supercomputers—such as Frontier [1], [2] at ORNL, Aurora [3], [4] at ANL, and El Capitan [5] at LLNL—provide cutting-edge infrastructures for executing diverse heterogeneous workloads. On such systems, users perform experiments across numerous parallel execution contexts, often distributed across thousands of compute nodes. The HPCToolkit [6], [7] performance suite was developed to efficiently measure applications on exascale platforms, collecting fine-grained, instruction-level measurements of CPU and
The remainder of the paper is organized as follows. Section II reviews existing tools for performance measurement and analysis, providing background on challenges at exascale and the initial hpcanalysis architecture. Section III details the design of our high-throughput C++ API for ingesting exascale measurements, while Section IV describes the GPU-
accelerated abstraction layer we implemented to accelerate the analysis of extracted data. Section V introduces the tridimensional performance model for iterative trace analysis, achieved by integrating hpcanalysis with Thicket [9]. Section VI presents our experimental results on the Frontier and Aurora supercomputers. Finally, Section VII concludes the paper. II. BACKGROUND : H P C A N A L Y S I S The HPC community has developed numerious tools for measuring and analyzing the performance of parallel scientific applications. NVIDIA Nsight Systems [10] enables systemwide tracing to identify communication and I/O bottlenecks, while NVIDIA Nsight Compute [11] enables low-level analysis of individual GPU kernels. The TAU Performance System [12] and Score-P [13] provide broad capabilities for instrumenting and collecting performance data across various programming models. Similarly, Caliper [14] and Extrae [15] provide mechanisms for intercepting runtime calls and integrating performance-aware annotations directly into source code. Scalasca [16] and VampirTrace [17] collect trace data for visual analysis in specialized GUIs such as Paraver [18] and Vampir [17]. Within this ecosystem, HPCToolkit stands out due to its powerful measurement methodology, which employs statistical sampling of hardware counters and call stack unwinding. This approach enables fine-grained, instruction-level measurement of CPU and GPU code with minimal execution overhead [19]. Such a sophisticated methodology facilitates the profiling and tracing of applications across thousands of compute nodes on heterogeneous exascale supercomputers—environments where traditional tools often struggle with the immense volume of measurement data. Several tools have emerged to facilitate programmatic and automated post-mortem analysis of performance measurements. Hatchet [20] is a Python library that can analyze performance profiles collected by various measurement tools. Hatchet models performance profiles as generic GraphFrame objects, which link a calling context tree to a Pandas [21] DataFrame that stores associated metrics. Thicket [9], built upon Hatchet, extends these capabilities to support ensemble analysis across multiple experiments, enabling users to compare application performance across diverse platforms and scales. Pipit [22] is a Python library that can analyze execution traces from various file formats, such as Projections [23] and Open Trace [24], by parsing them into consistent Pandas DataFrame structures. However, as demonstrated in [8], these tools exhibit critical inefficiencies when applied to fine-grained, large-scale HPCToolkit performance measurements. They load entire datasets into memory at once, which is infeasible for largescale analysis. Furthermore, they don’t provide techniques for pruning code regions with low information content or sampling subsets of massive performance profiles or execution traces. Notably, these tools store performance profiles in a dense
format, which is highly inefficient for sparse profiles for GPUaccelerated applications; prior studies [25] have shown that dense representations for these profiles can be over 1000× larger than their sparse counterparts. Finally, these tools often analyze profiles and traces in isolation, preventing complex analysis tasks such reconstructing a calling context tree for a specific trace interval. To bridge this gap, prior work [8] introduced the hpcanalysis framework. This Python-based framework provides an advanced interface for the interactive, scalable, and efficient processing of fine-grained and large-scale performance measurements. The framework incorporates techniques for pruning code regions with low information content, techniques for sampling subsets of massive performance profiles and execution traces via specialized query expressions, and a unified approach to analyzing both profiles and traces. Previous studies [8] have shown that hpcanalysis can efficiently process datasets from thousands of compute nodes by employing pruning and sampling. A. Architecture of the hpcanalysis Framework Fig. 1 illustrates the three-layered architecture of the hpcanalysis framework as introduced in [8]. The framework consists of the Read API, Query API, and Data Analysis layers. This design decouples low-level data access from middle-level query expressions and high-level analytical tasks, optimizing each layer for maximum scalability and efficiency. The Read API is a low-level API that parses and ingests requested data ”slices” into memory. For small metadata, such as metric descriptions and profile descriptions, the Read API parses the entire section (typically a few megabytes) upon the first request. For large metadata, such as the calling context tree, and large performance data, such as performance profiles execution traces, the Read API employs pruning and sampling techniques: • Pruning Large Calling Context Trees: HPCToolkit’s global calling context tree represents the union of calling contexts across all parallel execution contexts. It contains fine-grained information, including library internals, linelevel statements, and nodes with negligible inclusive costs. Because excessive detail can hamper analysis of critical bottlenecks, hpcanalysis enables users to declare strategies for pruning code regions with low information content from the global calling context tree. Pruning strategies are evaluated once on the summary profile - an aggregate of all parallel performance profiles - and then propagated across parallel profiles. The Read API leverages the internal structure of performance files, where calling context tree nodes and metrics are sorted by their IDs, to extract only requested ”slices” via selective access/binary search, without exploring massive profiles in their entirety. Furthermore, the Read API employs internal parallelism to parse individual profiles simultaneously, significantly accelerating data ingestion. • Sampling Massive Profiles and Traces: To facilitate analysis of large-scale datasets, the Read API enables
"rank(0-100:10)"
Map profile query
No Fetch entire file
Prune lowinformation code regions
Fetch entire file
Metric Descriptions
Profile Descriptions
Fetch slices specified by IDs
No
Table initialized?
Yes
Yes
"function(MPI_*)"
"cputime:prop (i)"
Map metric query
No
Table initialized?
Query API
Yes
CCT
Read API Execution Traces
Table initialized?
Map CCT query
Performance Profiles
DataFrame
DataFrame
DataFrame
Profile Descriptions
CCT
Metric Descriptions
Extract profile IDs
Extract CCT IDs
Extract metric IDs
Fetch slices specified by IDs
Slices cached?
Data Analysis
Automated Analysis
Performance Reports
Integration with External Tools
Visualization
No Yes DataFrame
DataFrame
Performance Profiles Slices
Execution Traces Slices
Fig. 1: Architecture of the hpcanalysis framework
users to sample subsets of massive performance profiles and execution traces to estimate results on smaller subsets. Performance profiles consist of local calling context trees annotated with metric values for their parallel execution context. Given their vast volume at scale, the Read API does not fetch the entire section upon first access; instead, it uses indices provided by the Query API to extract ”slices”. Users provides queries specifying execution contexts (ranges of MPI ranks/threads), calling contexts (specific call chains or functions), and metrics of interest, which are mapped to raw indices for the Read API. Within the database, profiles are organized as an array indexed by profile IDs, with internal calling context tree nodes and metrics sorted by their respective IDs. This enables the Read API to utilize indexing, binary search, and parallelism to perform sampling. Execution traces, representing series of timed calling context tree node events, are handled similarly. The Query API is a middle-level API that enables users to access performance data via specialized query expressions rather than the raw indices required by the Read API. It orchestrates out-of-core data handling through two primary mechanisms: •
Query Mapping: Users submit queries specifying ”slices” of interest. For instance, when analyzing an execution on 100,000 MPI ranks, users can submit queries "rank(0-100000:100)", "function(MPI_*)", and "cputime:prop (i)" to examine CPU performance of MPI routines for every 100th rank. These queries are mapped to raw indices using metadata tables and then processed by the Read API using parallelism, indexing, and binary search. Similarly, users can provide
rank queries and time intervals to perform sampling of execution traces. • Caching and Requesting: Extracted data ”slices” are stored in Pandas DataFrame tables for analysis. When data is requested, the Query API checks if the corresponding logical IDs are already present/cached in its internal DataFrame tables. If not, it requests the missing ”slices” from the Read API. Once loaded, users can leverage standard DataFrame operations to analyze program behavior and performance. The Data Analysis layer, built atop the Query API, initially provided utilities for generating flat profiles, detecting load imbalances, and visualizing calling context trees, alongside the specialized hpcreport utility [8]. hpcreport analyzes the summary profile to categorize the execution time into domains (e.g., MPI or GPU execution) and automatically infers whether an application is compute, memory, communication, or I/O bound. In the following sections, we describe our enhancements to the initial hpcanalysis’ infrastructure to enable more scalable and efficient processing of exascale measurements and introduce novel workflows for advanced, automated performance interpretation at scale. III. ACCELERATING R E A D API FOR P ERFORMANCE DATA E XTRACTION The initial architecture presented in Fig. 1 was implemented entirely in Python, including Read API, Query API, and Data Analysis layers. These APIs were designed to integrate seamlessly into Jupyter [26] notebooks, enabling users to write queries and specialized analysis tasks to investigate application performance. When processing massive exascale
measurements, the original implementation relied on techniques for pruning code regions with low information content and sampling subsets of massive performance profiles and execution traces via specialized query expressions. While they effectively reduce the volume of exascale measurements by sacrificing some level of detail, researchers often require full or larger datasets to perform high-fidelity diagnostics with greater confidence. Techniques for pruning calling context trees described in Section II-A are highly efficient, as they identify major bottlenecks on the summary profile and propagate them across massive parallel performance profiles via selective access/binary search. However, sampling subsets of massive performance profiles and execution traces can sometimes omit critical performance information, preventing a comprehensive examination of the execution. While prior work [8] successfully identified communication anomalies in a 64K-rank LAMMPS [27] execution on Frontier by sampling a small subset of ranks, this approach was largely motivated by the ingestion bottlenecks inherent in the Python-based Read API. To enable users to analyze their large-scale executions with a greater level of detail, we focus on providing an accelerated infrastructure for ingesting exascale measurements. Because data ingestion represented a bottleneck, our first enhancement to hpcanalysis was to re-implement the Read API as a shared C++ library. This new API follows the exact interface described in [8], ensuring backward compatibility while significantly improving throughput. The initial Python Read API relied on standard file operations (file.seek() and file.read()), which typically involve double-copying data: first from the disk to the kernel buffer, and then from the kernel buffer to the Python process memory space. In our new C++ API, we utilize the mmap [28] system call to map files directly into the process’s virtual address space. This enables the operating system to load data lazily upon access; subsequent accesses use simple pointer arithmetic, which is significantly faster than repeated file I/O calls. While Python provides an mmap module, it is notably more efficient in C++, where address jumps are performed using single machine instructions without the overhead of object creation or interpreter intervention. Furthermore, the C++ API leverages native OpenMP parallelism, which is effectively unavailable in Python due to the Global Interpreter Lock (GIL). The GIL is a mutex that prevents multiple native threads from executing Python bytecodes simultaneously. While joblib.Parallel used in the initial Python implementation was useful for I/O-bound tasks, where the GIL can be released during disk operations, for CPU-bound tasks joblib [29] often falls back to multiprocessing, incurring the overhead of process creation and interprocess communication. In contrast, our C++ API uses fast, shared-memory OpenMP threads for low-overhead parallel processing of multiple performance profiles and execution traces. By default, the C++ API utilizes maximum hardware concurrency to determine the number of OpenMP threads, while the user retains the flexibility to specify a custom value.
Finally, we use the pybind11 [30] library to convert native C++ objects to Python-compatible structures, ensuring that the bulk of the computational work is completed in C++ before any conversion overhead is incurred. In Section VI-B, we evaluate our C++ API on a massive-scale execution on Aurora, demonstrating that it can ingest exascale measurements with significantly higher throughput than the original Python-based Read API. IV. ACCELERATING Q U E R Y API FOR P ERFORMANCE DATA A NALYSIS After enhancing the low-level Read API, we turned our focus to the middle-level Query API, which enables users to access performance data via specialized query expressions. In hpcanalysis’ initial implementation, Query API stored extracted performance data in Pandas DataFrame tables for further analysis. While Pandas leverages efficient, vectorized operations through numpy [31], it is restricted to a singlethreaded CPU execution, hindering scalable processing of exascale measurements. To address this, we accelerated the analysis workflow by replacing Pandas DataFrame tables with a GPU-accelerated abstraction layer, enabling high-throughput parallel processing across heterogeneous architectures. We implemented the HpcDataFrame abstraction, a unified interface that the Query API utilizes to store and manipulate extracted performance data. By default, the HpcDataFrame executes on the CPU using a Pandas backend. However, if the user explicitly enables GPU acceleration, the framework dynamically selects a specialized backend based on the detected hardware. On systems with NVIDIA GPUs, the framework utilizes the cuDF [32] library, which provides a GPUaccelerated, Pandas-compatible interface. Similarly, on systems with AMD GPUs, the framework utilizes the hipDF [33] library. Because a native, Pandas-compatible library for Intel GPUs—comparable to cuDF or hipDF—is not available, we implemented our custom SYCL-based wrapper to provide GPU-accelerated support for Intel architectures. Utilizing the dpnp [34] library, we modeled the HpcDataFrame by storing each table column as a discrete array in Intel GPU memory. This wrapper implements the core functionalities required by the Query API by leveraging SYCL-accelerated kernels. While the SYCL wrapper currently focuses on singlecolumn operations, it provides the foundational infrastructure needed for high-throughput analysis on Intel GPUs. For complex multi-column operations that exceed the current SYCL wrapper’s capabilities, users retain the flexibility to cast the HpcDataFrame back to a CPU-based Pandas representation for final processing. HpcDataFrame abstraction enables the Query API to interact with a single, consistent API, while the abstraction itself manages hardware-specific execution. We evaluated the performance of the HpcDataFrame and its GPU-accelerated backends by analyzing execution traces from a large-scale execution of the AMG benchmark [35] on Aurora. We performed the execution across 1,000 compute nodes using 100 ranks per node for a total of 100,000 MPI ranks. The benchmark solved a large-scale Laplace problem
TABLE I: Performance Comparison of Pandas (CPU) and cuDF (NVIDIA GPU) DataFrame Operations on Polaris Number of Traces
Grouping (s)
Sorting (s)
Filtering (s)
Merging (s)
CPU
GPU
Speedup
CPU
GPU
Speedup
CPU
GPU
Speedup
CPU
GPU
Speedup
10
0.001667
0.000982
1.70×
0.000233
0.000581
0.40×
0.000372
0.003548
0.10×
0.006814
0.001739
3.92×
100
0.006041
0.001980
3.05×
0.001751
0.000520
3.37×
0.000508
0.001278
0.40×
0.008197
0.002199
3.73×
1,000
0.034233
0.002260
15.15×
0.024794
0.002224
11.15×
0.002622
0.002433
1.08×
0.028052
0.005326
5.27×
10,000
0.483654
0.003993
121.13×
0.352657
0.003165
111.44×
0.033414
0.003908
8.55×
0.318041
0.007574
41.99×
100,000
7.696614
0.024470
314.54×
4.856504
0.023193
209.40×
0.375433
0.008901
42.18×
3.125140
0.046568
67.11×
TABLE II: Performance Comparison of Pandas (CPU) and hipDF (AMD GPU) DataFrame Operations on Frontier Number of Traces
Grouping (s)
Sorting (s)
Filtering (s)
Merging (s)
CPU
GPU
Speedup
CPU
GPU
Speedup
CPU
GPU
Speedup
CPU
GPU
Speedup
10
0.000832
0.001015
0.82×
0.000207
0.000453
0.46×
0.000339
0.001405
0.24×
0.006900
0.001523
4.53×
100
0.003507
0.000976
3.59×
0.001751
0.000484
3.62×
0.000579
0.001384
0.42×
0.008702
0.001562
5.57×
1,000
0.036337
0.001744
20.84×
0.026613
0.000943
28.22×
0.003052
0.001680
1.82×
0.031599
0.002735
11.55×
10,000
0.575268
0.007235
79.51×
0.439099
0.003317
132.39×
0.056606
0.002421
23.38×
0.448289
0.008239
54.41×
100,000
9.342253
0.060572
154.23×
6.452984
0.029318
220.10×
0.814406
0.006581
123.75×
5.459309
0.065223
83.70×
TABLE III: Performance Comparison of Pandas (CPU) and SYCL (Intel GPU) Array Operations on Aurora Operation (s)
10 Traces
100 Traces
1,000 Traces
10,000 Traces
100,000 Traces
CPU
GPU
Speedup
CPU
GPU
Speedup
CPU
GPU
Speedup
CPU
GPU
Speedup
CPU
GPU
Speedup
Scalar Compare
0.000043
0.000122
0.35×
0.000071
0.000163
0.44×
0.000269
0.000134
2.01×
0.001706
0.000206
8.28×
0.013758
0.001196
11.50×
Vector Add
0.000062
0.000146
0.42×
0.000073
0.000142
0.51×
0.000624
0.000131
4.76×
0.000882
0.000590
1.49×
0.010708
0.001199
8.93×
In-place Multi
0.000075
0.000123
0.61×
0.000094
0.000121
0.78×
0.000790
0.000114
6.93×
0.004231
0.000167
25.34×
0.035281
0.000559
63.11×
Sorting
0.000036
0.000133
0.27×
0.000322
0.000151
2.13×
0.003722
0.000292
12.75×
0.034060
0.001461
23.31×
0.485844
0.020705
23.47×
Filtering
0.000013
0.000287
0.05×
0.000054
0.000289
0.19×
0.000441
0.000253
1.74×
0.003502
0.000712
4.92×
0.125687
0.002120
59.29×
Reduction Sum
0.000009
0.000076
0.12×
0.000015
0.000057
0.26×
0.000109
0.000058
1.88×
0.000872
0.000070
12.46×
0.022190
0.000324
68.49×
Cumulative Sum
0.000017
0.000097
0.18×
0.000105
0.000078
1.35×
0.000864
0.000088
9.82×
0.007594
0.000521
14.58×
0.120441
0.001474
81.71×
using a local grid size of 40 × 50 × 50 per rank, which generated a performance database of approximately 11 GB. We analyzed the processing of execution traces across three distinct platforms, leveraging vendor-specific software stacks for GPU acceleration: Polaris [36], [37] (NVIDIA A100 40GB GPUs) using CUDA 12.8 [38]; Frontier (AMD Instinct MI250X GPUs) using ROCm 7.0.2 [39]; and Aurora (Intel Max 1550 GPUs) using the Intel oneAPI [40] environment operating over the Level Zero [41] 12.60.7 driver interface. On each platform, we measured the time required to manipulate trace events using the standard CPU-based Pandas backend against the corresponding GPU-accelerated backend (cuDF for Polaris, hipDF for Frontier, and our SYCL wrapper for Aurora). We evaluated manipulating data subsets ranging from 10 to 100,000 MPI ranks. To ensure our results account for potential noise in tested environments, every operation reported in Tables I, II, and III was executed 10 times, with the presented metrics representing the calculated average across those runs. The experimental results demonstrate a significant performance advantage when utilizing GPU-accelerated backends for large-scale analysis. Across all platforms, we observe a
consistent trend: for smaller datasets (e.g., 10 to 100 traces), the CPU often outperforms the GPU due to the overhead associated with memory transfers and kernel launches. However, as the data volume scales toward 100,000 traces, the massive parallelism of the GPU becomes transformative. The most remarkable speedups were achieved with cuDF on Polaris and hipDF on Frontier; for high-complexity operations like grouping and sorting, we observed speedups exceeding 314× and 220×, respectively. For the Intel-based Aurora system (Table III), we evaluated speedups on the fundamental array operations that underpin our SYCL wrapper. Operations such as cumulative sum and reduction sum saw speedups of up to 81× and 68×, respectively. These findings validate the impact of our HpcDataFrame abstraction and serve as a powerful benchmark for GPU-accelerated libraries on realworld exascale performance data. V. T RACE A NALYSIS W ITH T HICKET In Section II, we noted that Hatchet [20] and Thicket [9] tools provide generic models for storing and analyzing calling context trees and performance profiles collected by various measurement tools. While Hatchet analyzes performance
A
B
D
Iteration /Node
A
B
C
D
E
F
G
I
t5-t1
t3-t1
t5-t3
t2-t1
t3-t2
t4-t3
t5-t4
II
t9-t6
t8-t6
t9-t8
t7-t6
t8-t7
t9-t8
NA
III
t13-t10 t11-t10 t13-t11
NA
t11-t10 t12-t11 t13-t12
C
E
F
G
Thicket Profile
A
Rank /Node
B
D A
A
C
E
F
G
A
B
C
D NA
E
F
I
t13-t10 t11-t10 t13-t11
II
t28-t25 t27-t25 t28-t27 t26-t25 t27-t26 t28-t27
G
t11-t10 t12-t11 t13-t12 NA
A
Thicket Profile B
D
C
E
B
F
G
D
C
E
B
F
C
E
F
G
A
D
RANK 1 t0
t1
E t2
F
G
t3
D
t4
t5
t6
E
F
t7
t8
E t9
F
t10
G
t11
t12
t13
t14
B
D
E
RANK 2 t15
t16
F t17
G t18
D t19
t20
E t21
F t22
G t23
D t24
t25
E t26
C
E
F
F t27
t28
t29
Fig. 2: Workflow for iteration-aware trace analysis. Traces are scanned to detect iterative boundaries, followed by reconstructing calling context trees for each interval. The resulting data is merged into a three-dimensional Thicket profile (nodes × traces × iterations) for fine-grained comparative analysis.
profiles from a single application run, Thicket provides a multi-dimensional model that can compare performance profiles across numerous experiments. Thicket utilizes a twodimensional infrastructure: it maintains a unified calling context tree representing the union of calling contexts across all experiments and maps the union tree to a DataFrame where each row corresponds to a specific experiment. However, as discussed in Section II, Hatchet and Thicket scale poorly for fine-grained, large-scale performance data, lack support for pruning and sampling exascale measurements, and do not provide native support for analyzing execution traces. In this work, we integrate the advanced hpcanalysis’ interface with Thicket’s multi-dimensional analytical model to provide workflows for advanced trace analysis. Our goal is to evolve Thicket’s initial two-dimensional model for analyzing performance profiles across experiments into a threedimensional model for analyzing fine-grained behaviors within and across execution traces, such as iterative patterns and load imbalances. By leveraging hpcanalysis’ interface to efficiently extract critical performance ”slices”, we can populate a Thicket model that enables users to compare execution phases within and across execution traces using a few lines of Python code. This integration demonstrates how the hpcanalysis infrastructure can extend external tools to provide sophisticated models for advanced performance analysis. We implemented an advanced workflow, shown in Fig. 2, that detects iterative phases within and across execution traces to construct a unified Thicket model. For a given set of execution traces, the workflow first identifies sequences of trace events that represent repeated iterative phases. Once they are identified, hpcanalysis reconstructs a distinct Hatchet profile for each individual iteration. This step is crucial
because HPCToolkit’s default performance profiles collapse the time dimension, aggregating the total cost of each calling context across the entire execution. To perform fine-grained iterative analysis, one must ”re-materialize” performance profiles for specific time intervals (iterations) from the execution traces—a core capability of our new workflow. After reconstructing Hatchet profiles for individual iterations, we aggregate them into a unified Thicket object. Unlike standard Thicket object designed to compare distinct experiments, our workflow constructs a tri-dimensional model that facilitates comparisons within and across execution traces. We achieve this by implementing a tri-dimensional index within Thicket’s DataFrame table, consisting of: (1) the event (calling context tree node), (2) the trace index, and (3) the iteration index. The DataFrame columns store the actual time spent in each event for the specific trace and iteration. Using standard DataFrame operations, users can slice this model by fixing a specific iteration to compare it across execution traces, or by fixing a specific trace to analyze performance evolution across its iterations. This enables the detection of load imbalances and the prediction of potential performance gains, which we discuss in Section VI-A. VI. C ASE S TUDIES In this section, we present two case studies demonstrating how the enhanced hpcanalysis infrastructure enables advanced analysis and interpretation of program performance at scale. First, we evaluate our novel performance model for analyzing iterative phases within and across execution traces using a real-world application. Second, we demonstrate how the framework’s accelerated infrastructure can process massivescale executions to discover critical performance inefficiencies. The first experiment was performed on the Frontier super-
GPU Kernels Identification: Because GPU performance profiles are sparse, hpcanalysis extracts GPU kernels by sampling the summary profile for calling contexts attributed with the identified GPU metrics. For the GAMESS execution on Frontier, it isolated six primary GPU kernels (e.g., gpu_rhf_j05_ppps_) and their parent gpu_ompmod_twoei_jk_. • GPU Imbalance Quantification: hpcanalysis examines parallel performance profiles to calculate a global balance ratio for each GPU kernel, defined as the average execution time across ranks divided by the maximum. As shown in Table IV, several GPU kernels exhibited severe imbalance, with ratios as low as 0.50 for gpu_rhf_j04_psps_ (where a ratio of 1.0 represents perfect balance). • Iterative Trace Analysis with Thicket: Because performance profiles aggregate costs, execution traces are required to understand variance over time. hpcanalysis scans GPU execution traces to detect iterative boundaries, reconstructs separate Hatchet profiles for each iteration, and merges them into a tri-dimensional Thicket model described in Section V. This enables the framework to orchestrate a complex analysis: first using the summary profile to detect major bottlenecks, then using parallel performance profiles to quantify global load imbalance, and finally using execution traces to understand behavior across iterations. Fig. 4 illustrates a single iteration (rank 0, iteration 0) within the Thicket model for the GAMESS execution on Frontier. While Thicket stores data for all ranks and iterations in its tridimensional DataFrame index, the visualization enables users to select specific ”slices”. This structure enables powerful, oneline statistical queries. For example: •
Fig. 3: Single-node GAMESS execution on Frontier. The varying GPU stream lengths and idle gaps illustrate significant load imbalance across MPI ranks.
computer (AMD EPYC 7A53 CPUs, AMD Instinct MI250X GPUs, HPE Slingshot 11) and the second on the Aurora supercomputer (Intel Xeon Max 9470 CPUs, Intel Max 1550 GPUs, HPE Slingshot 11). A. GAMESS GPU Utilization on Frontier
In our first case study, we analyzed a single-node execution of a Hartree-Fock energy calculation in GAMESS [42] on Frontier. GAMESS is a quantum chemistry suite used to calculate the electronic structure of nanoparticles. We performed the execution across 16 MPI ranks; in a typical GAMESS configuration, each computational rank offloads to one GPU and is paired with a data-server rank. Consequently, the total number of ranks is double the number of GPUs. We utilized this execution to evaluate our tri-dimensional model’s ability to examine iterative phases within and across execution traces 1 df_kernel.groupby("iteration")["time"]. agg( and quantify GPU utilization in detail. 2 ["mean", "min", "max", "std"] First, in Fig. 3, we visualized the GPU execution traces 3 ) for the eight computational MPI ranks using hpcviewer. Visual inspection revealed a significant load imbalance, with This query calculates load imbalance across MPI ranks several GPUs spending considerable time idle. This stems for every iteration for a specific GPU kernel. By swapping from a triangularly nested loop, where computational tasks are "iteration" for "rank", users can examine variance assigned varying amounts of work. While the GUI highlights within a rank over time. Using these queries, we calculated the gaps, it cannot quantify load balance ratios for specific the Coefficient of Variation (CV) for major GPU kernels, as GPU kernels or predict potential performance gains. We used shown in Table IV. We found that while CV across ranks this execution to demonstrate how hpcanalysis and our tri- reached as high as 46.31%, the CV within ranks remained dimensional model can automatically detect load imbalances under 9%, proving that the iterative behavior is stable but and convert them into actionable insights. poorly distributed across resources. We implemented a workflow that performs fine-grained Finally, we used the tri-dimensional model to predict poanalysis of GPU utilization for the GAMESS execution on tential performance gains under ideal load balancing. By Frontier. The process follows several key steps: calculating the difference between the average maximum time • GPU Metrics Identification: hpcanalysis detects and the average mean time per iteration for each GPU kernel which GPU metrics (e.g., kernel execution, data copies, (Table V), we estimated potential savings of 28.08 seconds memory allocation, or synchronization) contain non-zero across 11 iterations. Given the total application time of 87 secsamples. For the GAMESS execution on Frontier, it onds, resolving this imbalance could yield an approximately identified GPU kernel execution (gker) and explicit data 32.28% speedup. This demonstrates how hpcanalysis can copies (gxcopy) as the primary metrics of interest. transform raw execution traces into concrete optimization
TABLE IV: Analysis of global load imbalance and iteration variance for the major GPU kernels in GAMESS execution on Frontier. GPU Kernel
Execution Share (%)
Global Balance Ratio
Across-Rank Time CV (%)
Within-Rank Time CV (%)
gpu rhf j05 ppps gpu rhf j06 pppp gpu rhf j03 ppss gpu rhf j04 psps gpu rhf j02 psss gpu rhf j01 ssss
42.89% 35.93% 10.63% 6.99% 3.46% 0.10%
0.64 0.96 0.79 0.50 0.80 0.68
34.21 7.20 19.84 46.31 19.68 27.51
1.45 1.36 7.79 2.66 9.09 6.28
TABLE V: Estimated wall-clock time reductions through ideal load balancing of GPU kernels in GAMESS execution on Frontier. GPU Kernel
Avg. Mean Time (s)
Avg. Max Time (s)
Potential Savings (s / Iteration)
Total Estimated Reduction (s)
gpu rhf j05 ppps gpu rhf j06 pppp gpu rhf j03 ppss gpu rhf j04 psps gpu rhf j02 psss gpu rhf j01 ssss
2.963 2.483 0.734 0.483 0.239 0.007
4.650 2.599 0.945 0.958 0.300 0.010
1.687 0.116 0.211 0.475 0.061 0.003
18.557 1.276 2.321 5.225 0.671 0.033
Fig. 4: Thicket model for a single iteration (rank 0, iteration 0) for the GAMESS execution on Frontier.
targets. Our tri-dimensional model targets applications with regular, iterative patterns to analyze load imbalance within and across execution traces; it is not optimized for complex applications with irregular or highly variable execution phases. B. Exascale Network Congestion Analysis on Aurora In our second case study, we demonstrate how the accelerated hpcanalysis infrastructure enables efficient processing of massive-scale executions to isolate critical performance
inefficiencies. We analyzed the 100,000-rank execution of the AMG benchmark on Aurora, which served as a stress test for our HpcDataFrame abstraction in Section IV. This 100,000-rank execution was distributed across 1,000 compute nodes, with 100 MPI ranks per node. Here, we evaluate how our accelerated infrastructure can efficiently process exascale measurements to localize network-level bottlenecks. High-Throughput Ingestion of Exascale Data. The AMG execution on Aurora produced 100,000 parallel profiles and 100,000 execution traces. The global calling context tree—the union of local calling context trees across all MPI ranks—contained 97,833 distinct calling contexts. Consequently, the resulting performance dataset represents a massive, structured matrix of up to 100, 000 × 97, 833 calling contexts, each annotated with multiple inclusive and exclusive costs. After applying pruning strategies to remove MPI library internals, line statements, and negligible code regions (less than 1% of total runtime), our new C++ Read API ingested parallel profiles for the full 100,000-rank dataset in an average of 9.6878 seconds on Aurora. This represents a paradigm shift in ”time-to-insight”. By evaluating pruning logic once on the summary profile and propagating it via binary search across all MPI ranks, hpcanalysis efficiently imports critical code regions for massive-scale executions. Furthermore, our accelerated C++ API eliminates the sampling dependencies that limited prior work. Table VI compares the original Python-based Read API using joblib.Parallel and various task counts against our new C++ API utilizing OpenMP multi-threading. On Aurora, each compute node contains 104 cores with 2 hardware threads per core; thus, in Table VI, our C++ API utilized 204 OpenMP threads to maximize hardware concurrency. In Table VI, we measured performance when sampling up to 10,000 MPI ranks, as the Python API’s performance began to
significantly degrade at that scale—requiring several minutes to process data that the C++ API handled in approximately one second. The results indicate that the initial Python-based API suffered from poor scalability, whereas the C++ implementation scaled with high efficiency. Ultimately, the C++ API imported the entire 100,000-rank dataset in only 9.6878 seconds. All metrics in Table VI represent an average across 10 trials. While we performed the entire analysis on Aurora, users may also transfer collected measurements to systems with NVIDIA or AMD GPUs to leverage more mature GPUaccelerated backends. TABLE VI: Ingestion latency (s) for the 100,000-rank AMG execution on Aurora: Python (joblib) vs. accelerated C++ (OpenMP) across sampling scales. MPI Ranks Sampled
Implementation 10
100
1,000
10,000
Python (16 tasks) Python (32 tasks) Python (64 tasks) Python (128 tasks)
3.7252 0.2338 0.2335 0.2287
1.6838 1.8088 1.8954 1.7087
18.3799 16.2053 15.7189 16.2276
444.1241 166.0623 166.7260 160.7161
C++ (204 threads)
0.1152
0.1224
0.2674
1.1265
hypre_GMRESSetup → hypre_BoomerAMGSetup → hypre_ParCSRMatrixSetNumNonzeros_core. Node-Level Performance Correlation. To investigate whether the observed communication overhead was tied to specific hardware, we mapped rank-level performance clusters to their compute nodes. By joining the DataFrame containing metric costs with the metadata DataFrame containing profilelevel information (Fig. 1), we localized individual MPI ranks to logical compute nodes. When measuring applications across numerous compute nodes, HPCToolkit attributes nodes with their POSIX ID, enabling unique node-level attribution. For the critical MPI_Allreduce instance, we first applied DBSCAN [43] to the 100,000 MPI ranks, which partitioned the 1,000-node allocation into two distinct groups of 798 and 202 compute nodes with zero intersection. Interestingly, querying the total execution time across ranks (by replacing "function(MPI_*)" with "function(main)") yielded the exact same exclusive grouping: the first group averaged 3.12s, while the 202-node outlier group averaged 5.19s. We further tested cluster stability for the group of 798 compute nodes using K-Means [44] across 2 to 5 sub-groups, finding minimal intersections: • 2 groups: Only 2 compute nodes intersected. • 3 groups: 13 compute nodes intersected between groups 1 and 2; no others. • 4 groups: 13 compute nodes intersected between groups 1 and 2, 3 compute nodes between groups 2 and 3, and 13 compute nodes between groups 3 and 4. • 5 groups: Intersections were limited to 12 compute nodes between groups 2 and 3 and 16 compute nodes between groups 3 and 4. These minimal intersections across a 1,000-node allocation suggest a high correlation between elevated MPI latency and specific logical compute nodes. Interconnect Mapping and Congestion Analysis. To localize the observed 202 outlier compute nodes, we implemented a workflow that maps collected POSIX IDs to physical Slingshot [45] interconnect coordinates on Aurora. We first examined the MPI ranks to detect groups of similar behavior. We then utilized metadata DataFrame table to uniquely distinguish between compute nodes. While POSIX IDs enabled statistical grouping, they did not reveal the topographical location of the congestion. Therefore, we developed a workflow that maps POSIX IDs to Aurora’s hexadecimal physical addresses, which follow formats such as:
Quantifying Communication Imbalance. Following the ingestion, we utilized the Query API to examine communication overhead. By querying "summary", "function(MPI_*)", and "cputime:sum (i)", we identified primary MPI bottlenecks within the summary profile. A subsequent request for rank-level granularity using "rank", "function(MPI_*)", and "cputime:prop (i)" enabled us to calculate global balance ratios (average execution time across ranks divided by the maximum execution time across ranks) with only a few lines of Python code. Extracted performance data are represented as a DataFrame table where metric costs are attributed to calling context tree node IDs. Because a single MPI routine may be invoked from multiple locations, hpcanalysis utilizes calling context tree node IDs to distinguish unique call sites. Fig. 5 illustrates the performance distributions across ranks for the six primary MPI bottlenecks. While five routines were well-balanced (ratios near 1.0), the second instance of MPI_Allreduce (Fig. 5b) exhibited a critical imbalance with a ratio of only 0.39. We clustered the 100,000 MPI ranks into six groups; for the critical MPI_Allreduce instance, 1 "x4109c0s0b0n0" the brown cluster in Fig. 5b clearly identifies a significant performance outlier. where coordinates represent racks (x), chassis (c), and Obtaining Calling Context. To understand the source slots (s). All strings end with b0n0, representing the single of the observed communication overhead, hpcanalysis blade/node within the slot [46]. By mapping the 202 outlier automatically reconstructs the full calling context for compute nodes to the interconnect, we determined that the the calling context tree node ID. By maintaining congestion was not confined to a single rack or chassis. The a separate tree structure alongside the DataFrame, affected nodes were distributed across 22 different racks; in the framework performs highly efficient bottom-up some racks, entire chassis were affected, while in others, only traversal. Starting from the MPI_Allreduce leaf a few showed degradation. By identifying physical coordiID, the framework identified the specific call chain: nates of communication overhead at this scale, we prove that
MPI_Allreduce (Instance 1) Total Runtime Fraction: 11.53%, Global Variance: 0.80
MPI_Allreduce (Instance 2) Total Runtime Fraction: 6.92%, Global Variance: 0.39
0.50
0.26
0.6
0.24
Cluster 0 (n=1235) Cluster 1 (n=10629) Cluster 2 (n=11345) Cluster 3 (n=19774) Cluster 4 (n=34986) Cluster 5 (n=22031)
0.35
0.30
0
20000
40000
60000
80000
Cumulative MPI Ranks (Sorted by Cluster)
Time (seconds)
0.40
0.4 0.3
Cluster 0 (n=42588) Cluster 1 (n=11124) Cluster 2 (n=6476) Cluster 3 (n=6319) Cluster 4 (n=12381) Cluster 5 (n=20200)
0.2 0.1
100000
0
20000
40000
60000
80000
Cumulative MPI Ranks (Sorted by Cluster)
(a)
Time (seconds)
0.5
0.45
Time (seconds)
MPI_Allreduce (Instance 3) Total Runtime Fraction: 6.48%, Global Variance: 0.86
0.22 0.20
Cluster 0 (n=3159) Cluster 1 (n=6731) Cluster 2 (n=12149) Cluster 3 (n=24659) Cluster 4 (n=25042) Cluster 5 (n=28260)
0.18 0.16 0.14
100000
0
20000
(b)
MPI_Waitall Total Runtime Fraction: 5.50%, Global Variance: 0.61
0.20
40000
60000
80000
Cumulative MPI Ranks (Sorted by Cluster)
100000
(c)
MPI_Scan Total Runtime Fraction: 4.03%, Global Variance: 0.73
MPI_Finalize Total Runtime Fraction: 3.87%, Global Variance: 0.81
0.30
0.16 0.18 0.14
0.15
Cluster 0 (n=1489) Cluster 1 (n=2548) Cluster 2 (n=12281) Cluster 3 (n=54423) Cluster 4 (n=26522) Cluster 5 (n=2682)
0.10
0.05
0
20000
40000
60000
80000
Cumulative MPI Ranks (Sorted by Cluster)
100000
0.16
0.14
Cluster 0 (n=22242) Cluster 1 (n=14833) Cluster 2 (n=9262) Cluster 3 (n=6923) Cluster 4 (n=21202) Cluster 5 (n=25538)
0.12
0.10
0.08
0
20000
40000
60000
80000
Cumulative MPI Ranks (Sorted by Cluster)
(d)
(e)
100000
Time (seconds)
0.20
Time (seconds)
Time (seconds)
0.25
0.12 0.10
Cluster 0 (n=1341) Cluster 1 (n=4064) Cluster 2 (n=15440) Cluster 3 (n=23322) Cluster 4 (n=31557) Cluster 5 (n=24276)
0.08 0.06 0.04 0
20000
40000
60000
80000
Cumulative MPI Ranks (Sorted by Cluster)
100000
(f)
Fig. 5: Performance distributions across ranks for the major MPI bottlenecks in the AMG exascale execution on Aurora.
Cluster MPI ranks by performance similarity to identify outliers
Map MPI ranks to logical compute nodes to determine job-wide distribution
Project compute nodes onto physical Slingshot topology to localize interconnect congestion
Fig. 6: Topology-aware diagnostic workflow for localizing exascale interconnect congestion. The process maps logical performance outliers to physical interconnect coordinates. hpcanalysis can efficiently examine exascale executions to detect, quantify, and physically locate network congestion within a production interconnect. Fig. 6 illustrates the entire workflow we utilized to localize network congestion for the AMG exascale execution on Aurora. VII. C ONCLUSION As high-performance computing enters the exascale era, the sheer volume and complexity of performance telemetry have outpaced the capabilities of traditional, CPU-bound analysis tools. In this paper, we presented an accelerated, heterogeneous infrastructure for the hpcanalysis framework,
leveraging a high-performance C++ API and GPU parallelism to meet the throughput demands of exascale executions. With this enhanced foundation, we demonstrated the ability to ingest performance datasets for 100,000-rank executions in seconds and efficiently process them to detect critical performance inefficiencies. On the Aurora supercomputer, we integrated a topology-aware workflow that maps logical performance outliers to their physical locations within the Slingshot interconnect, localizing network congestion at the rack and chassis level during massive-scale executions. Beyond raw throughput, we demonstrated how the programmatic interface of hpcanalysis enables seamless integration with external tools to provide sophisticated analytical models. By integrating hpcanalysis with the Thicket framework, we introduced a novel tri-dimensional performance model (N ode × T race × Iteration) that ”rematerializes” iterative behavior within and across execution traces. This model enables the detection of iterative imbalances, information typically lost in aggregate performance profiles. With this model, we were able to perform finegrained analysis of GPU idleness in production workloads and estimate potential performance gains on Frontier. Collectively, our contributions provide a scalable and extensible substrate for performance diagnostics at the absolute limit of modern supercomputing. We believe that the hpcanalysis framework will have a transformative impact on the HPC community, driving critical advancements in the analysis and optimization of exascale computing systems.
R EFERENCES [1] Oak Ridge Leadership Computing Facilty, “Frontier,” 2026, Accessed: April 2026. [Online]. Available: https://www.olcf.ornl.gov/frontier [2] ——, “Frontier User Guide,” 2026, Accessed: April 2026. [Online]. Available: https://docs.olcf.ornl.gov/systems/frontier user guide.html [3] Argonne Leadership Computing Facility, “Aurora,” 2026, Accessed: April 2026. [Online]. Available: https://www.alcf.anl.gov/aurora [4] ——, “Aurora User Guide,” 2026, Accessed: April 2026. [Online]. Available: https://docs.alcf.anl.gov/aurora/getting-started-on-aurora [5] Lawrence Livermore National Laboratory, “El Capitan: Preparing for NNSA’s first exascale machine,” 2026, Accessed: April 2026. [Online]. Available: https://asc.llnl.gov/exascale/el-capitan [6] L. Adhianto, S. Banerjee, M. Fagan, M. Krentel, G. Marin, J. Mellor-Crummey, and N. R. Tallent, “HPCToolkit: Tools for performance analysis of optimized parallel programs,” Concurrency and Computation: Practice and Experience, vol. 22, no. 6, pp. 685–701, 2010. [Online]. Available: https://www.doi.org/10.1002/cpe.1553 [7] K. Zhou, L. Adhianto, J. Anderson, A. Cherian, D. Grubisic, M. Krentel, Y. Liu, X. Meng, and J. Mellor-Crummey, “Measurement and analysis of GPU-accelerated applications with HPCToolkit,” Parallel Computing, vol. 108, p. 102837, 2021. [Online]. Available: https://www.doi.org/10.1016/j.parco.2021.102837 [8] D. Grbic and J. Mellor-Crummey, “Analyzing the Performance of Applications at Exascale,” in Proceedings of the 39th ACM International Conference on Supercomputing, 2025, pp. 792–806. [Online]. Available: https://www.doi.org/10.1145/3721145.3730417 [9] S. Brink, M. McKinsey, D. Boehme, C. Scully-Allison, I. Lumsden, D. Hawkins, T. Burgess, V. Lama, J. Lüttgau, K. E. Isaacs et al., “Thicket: seeing the performance experiment forest for the individual run trees,” in Proceedings of the 32nd International Symposium on HighPerformance Parallel and Distributed Computing, 2023, pp. 281–293. [Online]. Available: https://www.doi.org/10.1145/3588195.3592989 [10] NVIDIA Corporation, “NVIDIA Nsight Systems,” 2026, Accessed: April 2026. [Online]. Available: https://developer.nvidia.com/nsight-sys tems [11] ——, “NVIDIA Nsight Compute,” 2026, Accessed: April 2026. [Online]. Available: https://developer.nvidia.com/nsight-compute [12] A. Malony, S. Shende, W. Spear, C. W. Lee, and S. Biersdorff, “Advances in the TAU performance system,” in Tools for High Performance Computing 2011: Proceedings of the 5th International Workshop on Parallel Tools for High Performance Computing, September 2011, ZIH, Dresden. Springer, 2012, pp. 119–130. [Online]. Available: https://www.doi.org/10.1007/978-3-642-31476-6 10 [13] D. A. Mey, S. Biersdorf, C. Bischof, K. Diethelm, D. Eschweiler, M. Gerndt, A. Knüpfer, D. Lorenz, A. Malony, W. E. Nagel et al., “Score-P: A unified performance measurement system for petascale applications,” in Competence in High Performance Computing 2010: Proceedings of an International Conference on Competence in High Performance Computing, June 2010, Schloss Schwetzingen, Germany. Springer, 2011, pp. 85–97. [Online]. Available: https://www.doi.org/10.1007/978-3-642-24025-6 8 [14] D. Boehme, T. Gamblin, D. Beckingsale, P.-T. Bremer, A. Gimenez, M. LeGendre, O. Pearce, and M. Schulz, “Caliper: performance introspection for HPC software stacks,” in SC’16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2016, pp. 550–560. [Online]. Available: https://www.doi.org/10.1109/SC.2016.46 [15] A. Munera, S. Royuela, G. Llort, E. Mercadal, F. Wartel, and E. Quiñones, “Experiences on the characterization of parallel applications in embedded systems with extrae/paraver,” in Proceedings of the 49th International Conference on Parallel Processing, 2020, pp. 1–11. [Online]. Available: https://www.doi.org/10.1145/3404397.3404 440 [16] M. Geimer, F. Wolf, B. J. Wylie, E. Ábrahám, D. Becker, and B. Mohr, “The Scalasca performance toolset architecture,” Concurrency and computation: Practice and experience, vol. 22, no. 6, pp. 702–719, 2010. [Online]. Available: https://www.doi.org/10.1002/cpe.1556 [17] W. Williams and H. Brunst, “Parallel performance engineering using Score-P and Vampir,” in Companion of the 2023 ACM/SPEC International Conference on Performance Engineering, 2023, pp. 121– 125. [Online]. Available: https://www.doi.org/10.1145/3578245.3583715 [18] B. J. Wylie, J. Giménez, C. Feld, M. Geimer, G. Llort, S. Mendez, E. Mercadal, A. Visser, and M. Garcı́a-Gasulla, “15+ years of
joint parallel application performance analysis/tools training with Scalasca/Score-P and Paraver/Extrae toolsets,” Future Generation Computer Systems, vol. 162, p. 107472, 2025. [Online]. Available: https://www.doi.org/10.1016/j.future.2024.07.050 [19] L. Adhianto, J. Anderson, R. M. Barnett, D. Grbic, V. Indic, M. Krentel, Y. Liu, S. Milaković, W. Phan, and J. MellorCrummey, “Refining HPCToolkit for application performance analysis at exascale,” The International Journal of High Performance Computing Applications, vol. 38, no. 6, pp. 612–632, 2024. [Online]. Available: https://www.doi.org/10.1177/10943420241277839 [20] A. Bhatele, S. Brink, and T. Gamblin, “Hatchet: Pruning the overgrowth in parallel profiles,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2019, pp. 1–21. [Online]. Available: https://www.doi.org/10.1145/3295 500.3356219 [21] J. Bernard, “Python data analysis with pandas,” in Python recipes handbook: A problem-solution approach. Springer, 2016, pp. 37–48. [Online]. Available: https://www.doi.org/10.1007/978-1-4842-0241-8 5 [22] A. Bhatele, R. Dhakal, A. Movsesyan, A. K. Ranjan, and O. Cankur, “Pipit: Scripting the analysis of parallel execution traces,” arXiv preprint arXiv:2306.11177, 2023. [Online]. Available: https://www.doi. org/10.48550/arXiv.2306.11177 [23] L. V. Kale, G. Zheng, C. W. Lee, and S. Kumar, “Scaling applications to massively parallel machines using projections performance analysis tool,” Future Generation Computer Systems, vol. 22, no. 3, pp. 347–358, 2006. [Online]. Available: https://www.doi.org/10.1016/j.futu re.2004.11.020 [24] A. Knüpfer, R. Brendel, H. Brunst, H. Mix, and W. E. Nagel, “Introducing the open trace format (OTF),” in International Conference on Computational Science. Springer, 2006, pp. 526–533. [Online]. Available: https://www.doi.org/10.1007/11758525 71 [25] J. Anderson, Y. Liu, and J. Mellor-Crummey, “Preparing for performance analysis at exascale,” in Proceedings of the 36th ACM International Conference on Supercomputing, 2022, pp. 1–13. [Online]. Available: https://www.doi.org/10.1145/3524059.3532397 [26] Project Jupyter, “Project Jupyter: Interactive Computing across Programming Languages,” 2026, Accessed: April 2026. [Online]. Available: https://jupyter.org [27] A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, P. J. In’t Veld, A. Kohlmeyer, S. G. Moore, T. D. Nguyen et al., “LAMMPS-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales,” Computer physics communications, vol. 271, p. 108171, 2022. [Online]. Available: https://www.doi.org/10.1016/j.cpc.2021.108171 [28] Linux Man-Pages Project, “mmap(2) — Linux Manual Page,” 2026, Accessed: April 2026. [Online]. Available: https://man7.org/linux/man -pages/man2/mmap.2.html [29] Joblib Development Team, “Joblib: Running Python Functions as Pipeline Jobs,” 2026, Accessed: April 2026. [Online]. Available: https://joblib.readthedocs.io [30] pybind11 Development Team, “pybind11: Seamless Operability Between C++11 and Python,” 2026, Accessed: April 2026. [Online]. Available: https://pybind11.readthedocs.io [31] C. R. Harris, K. J. Millman, S. J. Van Der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith et al., “Array programming with NumPy,” nature, vol. 585, no. 7825, pp. 357–362, 2020. [Online]. Available: https://www.doi.org/10.1038/s41586-020-2649-2 [32] RAPIDS Development Team, “cuDF: GPU-Accelerated Pandas-like DataFrames,” 2026, Accessed: April 2026. [Online]. Available: https://docs.rapids.ai/api/cudf/stable [33] AMD ROCm Development Team, “hipDF: HIP-based DataFrames for AMD GPUs,” 2026, Accessed: April 2026. [Online]. Available: https://rocm.docs.amd.com/projects/hipDF/en/latest [34] Intel oneAPI Development Team, “dpnp: NumPy-compliant Interface for Data Parallel C++ (DPC++),” 2026, Accessed: April 2026. [Online]. Available: https://intelpython.github.io/dpnp [35] Lawrence Livermore National Laboratory, “AMG: Algebraic Multi-Grid Parallel Iterative Solver Benchmark,” 2026, Accessed: April 2026. [Online]. Available: https://www.osti.gov/biblio/1389816 [36] Argonne Leadership Computing Facility, “Polaris,” 2026, Accessed: April 2026. [Online]. Available: https://www.alcf.anl.gov/polaris [37] ——, “Polaris User Guide,” 2026, Accessed: April 2026. [Online]. Available: https://docs.alcf.anl.gov/polaris/getting-started
[38] NVIDIA Corporation, “NVIDIA CUDA Toolkit,” 2026, accessed: April 2026. [Online]. Available: https://developer.nvidia.com/cuda-toolkit [39] Advanced Micro Devices, Inc., “ROCm: Open Software Platform for GPU Compute,” 2026, accessed: April 2026. [Online]. Available: https://rocm.docs.amd.com [40] Intel Corporation, “Intel oneAPI Toolkits,” 2026, accessed: April 2026. [Online]. Available: https://www.intel.com/content/www/us/en/develope r/tools/oneapi/overview.html [41] ——, “Intel oneAPI Level Zero Specification,” 2026, accessed: April 2026. [Online]. Available: https://oneapi-src.github.io/level-zero-spec [42] G. M. Barca, C. Bertoni, L. Carrington, D. Datta, N. De Silva, J. E. Deustua, D. G. Fedorov, J. R. Gour, A. O. Gunina, E. Guidez et al., “Recent developments in the general atomic and molecular electronic structure system,” The Journal of chemical physics, vol. 152, no. 15, 2020. [Online]. Available: https://www.doi.org/10.1063/5.0005188
[43] D. Deng, “DBSCAN clustering algorithm based on density,” in 2020 7th international forum on electrical engineering and automation (IFEEA). IEEE, 2020, pp. 949–953. [Online]. Available: https: //www.doi.org/10.1109/IFEEA51475.2020.00199 [44] K. P. Sinaga and M.-S. Yang, “Unsupervised K-means clustering algorithm,” IEEE access, vol. 8, pp. 80 716–80 727, 2020. [Online]. Available: https://www.doi.org/10.1109/ACCESS.2020.2988796 [45] D. De Sensi, S. Di Girolamo, K. H. McMahon, D. Roweth, and T. Hoefler, “An in-depth analysis of the slingshot interconnect,” in SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2020, pp. 1–14. [Online]. Available: https://www.doi.org/10.1109/SC41405.2020.00039 [46] Argonne Leadership Computing Facility, “Running Jobs on Aurora,” 2026, Accessed: April 2026. [Online]. Available: https://docs.alcf.anl. gov/aurora/running-jobs-aurora