ClusterBench: A Framework for Cluster-Wide Continuous Benchmarking and Regression Testing

arXiv:2608.10956v1 [cs.DC] 11 Aug 2026

ClusterBench: A Framework for Cluster-Wide Continuous Benchmarking and Regression Testing Aditya Ujeniya

Jan Eitzinger

Thomas Gruber

Erlangen National HPC Center Friedrich-Alexander-University Erlangen, Germany 0009-0000-1670-4131

Erlangen National HPC Center Friedrich-Alexander-University Erlangen, Germany 0009-0000-3350-3841

Erlangen National HPC Center Friedrich-Alexander-University Erlangen, Germany 0000-0001-5560-6964

Georg Hager

Gerhard Wellein

Erlangen National HPC Center Friedrich-Alexander-University Erlangen, Germany 0000-0002-8723-2781

Erlangen National HPC Center Friedrich-Alexander-University Erlangen, Germany 0000-0001-7371-3026

Abstract—Data centers require tooling that validates the stability and reliability of an entire installation rather than of individual nodes. Such validation is needed when a cluster is accepted into service and at regular intervals for the rest of its lifetime. This requires that identical benchmarks be dispatched, in a single submission, to every node of the cluster, which in turn requires a cluster-aware scheduling framework. This paper presents ClusterBench, a framework for cluster-wide continuous benchmarking with cluster-aware scheduling. A benchmark collection ships with ClusterBench to target each component: CPU, GPU, memory, interconnect, and I/O disk. Running this suite through ClusterBench tests each component within a node and collects the statistical variation across cross-specimen components in the whole cluster. Because measurements are repeated throughout the cluster’s lifetime, ClusterBench collects data across both space and time. Comparing measurements against earlier runs allows detection of performance regressions introduced by software changes, such as OS kernel updates or new library versions. The repeated measurements also constitute a comprehensive dataset for research on hardware variability. Across component-targeted benchmarks on the NHR@FAU clusters Helma, Alex, and Fritz, variation within a single component stays within 1%. Depending on the component and benchmark type, performance variation across specimens reaches up to 5%, despite nodes that are identical by specification. To better understand this variation, performance is further correlated with collected metrics such as power draw, frequency, and temperature. This correlation depends on how the data is viewed: combining space (i.e., specimens) and time presents a different picture than considering either dimension alone, and the relationship between temperature and performance also differs between air-cooled and liquid-cooled nodes. Index Terms—regression, continuous benchmarking, clusterwide, health check, stress test, reproducibility

I. I NTRODUCTION Data centers validate a cluster at multiple stages in its life: at acceptance, when the administrators must show that the delivered system meets its specification, and then at regular intervals for the rest of its service life. The latter is necessary because neither the hardware nor the software stack stays

fixed. Components age and eventually fail, firmware settings change, kernels are patched, and compilers, MPI implementations, and numerical libraries are replaced. Any of these can change the performance of end user applications. In both cases, validation must cover the entire cluster rather than any single node. The relevant question is not whether one node meets its expected performance, but whether all nodes do, and how widely the measured values are distributed. This matters for multi-node jobs, because the slowest process determines the runtime of the whole job. Validation must therefore measure every node, and do so in a way that makes the results comparable. Existing benchmarking frameworks make the second part difficult. A test is specified by the resources it requests, and the choice of nodes is left to the scheduler — appropriate for the question these frameworks were built to answer: how a benchmark performs at a given scale. Some frameworks support cluster-wide benchmarking through single-node, clusteraware scheduling, but not with the granularity needed to test every component within a node individually. Moreover, no existing framework provides collecting metrics such as power consumption, frequency, and temperature out of the box, which would be valuable for correlating with the performance of each individual component. Section II examines these limitations in detail and motivates a framework designed specifically to address them. This paper presents ClusterBench1 , a framework for clusterwide continuous benchmarking and regression testing. A test is defined once, for a cluster or one of its partitions. ClusterBench then submits the jobs, tracks their completion, and stores each result together with the node, the time, and the software environment. Coverage can be exhaustive: every node can be measured on its own, node pairs can be measured to characterize the interconnect, and on nodes with several accelerators, 1 https://github.com/ClusterCockpit/cc-clusterbench

each accelerator can be exercised in parallel. Tests can also run on a schedule, so results accumulate along two axes. Across nodes, the spread of a metric over nominally identical nodes reveals the state of the installation and highlights outliers. Over time, comparison with earlier runs of the same test exposes regressions from software changes, as well as the slow hardware degradation that can precede a component failure. Our benchmark collection targets each hardware component separately: CPU, GPU, memory, interconnects, and disks. A component-level benchmark points to one component, while an application-level result mixes them all. ClusterBench also wraps each benchmark in a metric collector, so power, clock frequency, and temperature are recorded over the same interval as the benchmark. These metrics can help explain part of the spread observed along both axes — spatial and temporal. The consequence is that exact reproducibility of time-based metrics such as Flops/s and Bytes/s cannot be guaranteed even on a correctly functioning node; reproducibility is a meaningful claim only when a margin of statistical deviation is allowed. Our contributions are as follows. • identify cluster-aware scheduling as the capability required to validate a whole cluster installation, and define it to cover not only one benchmark per node, but also node pairs for multi-node benchmarks and individual accelerators within a node, all derived from a single definition of an input test file. • report a component-based variability study of production clusters, separating within-component from crossspecimen variation for CPUs and GPUs. • correlate the observed performance variation with power, clock frequency, and temperature to see how strongly they correlate. Section II reviews benchmarking frameworks, benchmarking suites, and earlier studies of hardware variability. Section III describes in detail the design and features of ClusterBench, Section IV the benchmark suite and the measurement method, and Section V the results. Section VI further discusses ClusterBench’s current limitations and what it is not meant for. II. R ELATED W ORK A. Benchmarking and regression frameworks Running a benchmark reproducibly requires several steps: the environment must be prepared, the code built, a job submitted, and the figures of merit extracted. Several centres have combined these steps into frameworks. JUBE2 describes a benchmark in XML or YAML. It executes the benchmark as a sequence of dependent steps. Each step is instantiated once for every combination of a parameter space. Results are written as tables or into a SQLite database [1]. ReFrame3 defines a test as a Python class. Each test passes through a fixed pipeline. The pipeline builds the code, submits the job, polls its state, and checks sanity and performance. Each

session is stored in a SQLite database [2]. Pavilion24 describes tests in YAML, and most parts of the system can be extended through plugins [3], [4]. Similar frameworks include buildtest, the OLCF Test Harness, BenchPRO, and Ramble [5]–[8]. AutoBench [9] targets HPC testbeds. It defines knobs in layers for partition, node, OS, software stack, and benchmark. Every combination of these knobs becomes one benchmark instance. Jobs are generated from templates and started from CI. The extracted figures of merit are then joined with telemetry from the DCDB5 monitoring system. Benchpark6 describes a benchmark as a Spack package that builds it and a Ramble application that runs it. These specifications are held in a shared repository. An experiment defined at one centre can therefore be repeated at another [10]. Some of these frameworks can cover a whole machine, and they do so in a similar way. ReFrame can distribute a test to all nodes in a given state. Pavilion2 can divide the system into chunks and run a test on each chunk. AutoBench can name partitions, nodes, and components as knobs and emit one job for each. The node is a coordinate in the configuration space rather than a participant in one experiment. The authors state this limitation themselves and list large jobs across a cluster as future work. Benchpark differs for another reason. A benchmark exists there as a recipe for building the code. Its specifications also serve comparisons between centres rather than between the nodes of one machine. ReFrame is limited further with the --distribute mode, because it distributes only single-node tests, and only for some scheduler backends. An interconnect benchmark is therefore excluded. Covering the interconnect of a cluster requires forming node pairs from the node list; separating single-node from multinode benchmarks within one job allocation; or dispatching one instance per accelerator inside a node, with systematic collection of the respective output for all these componentlevel testing cases. None of these frameworks offer this out of the box, so users must configure and adapt the framework themselves, and component-level testing may not fit within a single definition file. This difference also changes how output is stored. In the common case, one job runs one benchmark on one node with one configuration, so writing the output once per job is enough to tell results apart. JUBE, ReFrame and AutoBench therefore organize their output by benchmark. ClusterBench cannot rely on this, because a single allocation may carry measurements for many nodes, for node pairs, and for individual accelerators. The job alone no longer identifies a result, so the placement must become part of the output path. Without it, the files of a run cannot be traced back to the units that produced them. ClusterBench therefore derives a fixed directory layout from the configuration and the placement. The extraction step uses the same layout to find an output file together with the metric samples that belong to it. 4 https://github.com/hpc/pavilion2

2 https://github.com/FZJ-JSC/jube

5 https://gitlab.lrz.de/dcdb/dcdb

3 https://github.com/reframe-hpc/reframe

6 https://github.com/LLNL/benchpark

The frameworks also differ in the metrics they collect during a run. ReFrame and Pavilion2 record only the figure of merit reported by the benchmark. Values such as power draw, frequency, and temperature are therefore not available. Such values sometimes explain a large part of the difference between runs. AutoBench does associate them with its results. It obtains them by querying DCDB. This monitoring database runs continuously on the target system and is maintained independently of the benchmarking platform. The metrics are therefore joined to a result after the run has finished. The approach also requires that monitoring is already deployed for the machine. The form in which results are kept is a further consideration. JUBE, ReFrame, and Pavilion2 use row-major stores such as SQLite. AutoBench writes the extracted metrics as CSV into measurement repositories. Both forms serve as a record of what has been run, and both are suitable for retrieving a single result. They are less suitable as a basis for aggregation. Measurements accumulate across many nodes, components, and repetitions over the lifetime of a system, and their analysis then benefits from a columnar organization of the data. All the existing frameworks are feature-rich and have mature ecosystem include compilation phase for benchmarks before executing. ClusterBench purposefully excludes it and we explain more about the reasons to exclude this in the Limitations and Future Work Section VI-A. B. Continuous benchmarking Another related work concerns repeating benchmarks over time. The motivation is that continuous integration checks correctness, while performance is typically left unchecked [11]. exaCB, developed at JSC for JUPITER, is the closest work to ours in this group [11]: it integrates benchmarking into CI/CD pipelines and delegates execution to JUBE framework, with its components connected through a versioned JSON protocol. Results that conform to this protocol are kept, so they can be read as time series, showing whether the system has regressed after a change to the software stack. AutoBench serves a similar purpose: daily runs started from CI are used to monitor the health of hardware components [9]. Further related work includes Thicket, which supports the analysis of performance data collected across many runs [12], and the reproducible-benchmarking principles proposed by Koskela et al. [13]. These frameworks are driven by changes in software. A pipeline runs when the code or the software stack changes, and the measurement shows whether that change affected performance. Here, the code is the object under test, and the machine is the fixture. Our approach is different. We hold the binary fixed. Instead, the environment around it changes: the operating system, drivers, firmware, and the state of the hardware over the machine’s lifetime. So a run starts on a weekly schedule, or after an event such as acceptance testing or the end of maintenance, rather than after a code commit. This changes what we measure. A single value per benchmark records that something has changed, but not where. A set of

values covering all nodes at one point in time separates a shift of the whole machine from the deviation of an individual node, and this distribution can be the quantity of interest. C. Benchmarking Suites Several benchmark suites already exist. The NAS Parallel Benchmarks are a small set of programs derived from computational fluid dynamics codes, made up of five kernels and three pseudo-applications, with problem sizes fixed in advance as classes [14]. The HPC Challenge suite adds seven tests, among them HPL, STREAM, DGEMM, PTRANS, RandomAccess, FFT, and a latency and bandwidth test [15], [16]. The JUPITER Benchmark Suite combines 16 application benchmarks with 7 synthetic ones and was built for the procurement of an exascale system [17]. CORAL-2 serves the same purpose, and asks vendors to tune its codes to show the value of the hardware they offer.7 Some of these suites share individual benchmarks with ours, and the JUPITER suite is planned for the acceptance procedure of the new system [17]. Acceptance testing, however, happens once. A health check happens throughout the lifetime of the cluster, daily or weekly, and must run on a machine that is otherwise in production. For this purpose an application benchmark is a poor probe. It touches many components at the same time, so a change in its result does not say which part of the node has degraded. A component-level benchmark gives a single performance number for a single piece of hardware, and that number can be compared against the value measured when the machine was new. In the proposed benchmark collection, each benchmark is tied to one component of a node: CPU, GPU, main memory, GPU memory, network interconnects, and local disk. Where a vendor-tuned build exists, that build is used, so the component runs at peak performance, good enough to stress the desired component. A portable reference (vanilla or non-optimized) code reaches only part of the peak performance and does not stress the component enough. D. Variation between identical hardware Measurements have repeatedly contradicted the assumption that nodes of the same specification perform identically. Rountree et al. reported power differences between processors of the same type [18], and Inadomi et al. measured about two thousand CPU and DRAM modules and attributed the observed variation to manufacturing variability [19]. The same effect has been described under Turbo Boost [20], across processor generations [21], in a production system at NERSC [22], and across several hundred GPUs [23]. Process variation changes the achievable clock frequency and the leakage current of individual dies, and frequency scaling turns this into a difference in performance. So far, all these studies have been conducted only with application benchmarks. A variability study of componentlevel benchmarks and the CPU/GPU related metrics has not been done yet. 7 https://asc.llnl.gov/coral-2-benchmarks

benchmark.json

Refresh/Update time in seconds Individual worker/functionality

Parse JSON file

B. Databases

Add benchmarks to BenchmarkPool

MachineState

Populate node list

Check running jobs

Create job scripts

Allocate nodes to benchmarks

Submit jobs

Metric collector script

if cron: repeat

Extract output

update

Compress files

scheduler.db

results.db

Fig. 1: Architecture of ClusterBench framework

III. C LUSTER B ENCH ARCHITECTURE AND FEATURES ClusterBench is designed to operate in daemon mode. The daemon mode requires several background workers — a cron submitter, a job tracker, and a node list populator — and is additionally necessary for the submission of benchmarks. Go was selected as the implementation language due to its strong concurrency support, as native goroutines allow workers to be spawned and managed with minimal code complexity. Furthermore, the framework benefits from Go’s robust builtin tooling, comprehensive standard library, and straightforward deployment. Benchmark definitions are accepted only in JSON, which is parsed by the standard libraries without any external dependence. Its grammar is also unambiguous, avoiding both the implicit type coercion of YAML and any dependence on indentation or whitespace. With reference to Figure 1, the architecture of ClusterBench is explained in the following sections. A. General pipeline ClusterBench takes a simple definition file as input. It parses this file and loads the applicable benchmarks into the BenchmarkPool, discarding any invalid benchmark definitions. At startup, ClusterBench also populates a list of the idle and allocated nodes across the different partitions; this list is then used to assign nodes to the jobs that will be submitted. The next step is to generate structured job files and submit them to the assigned nodes. Into each job a call to MachineState8 [24] is injected, which records the state of a node before the benchmarks begin. Capturing this state makes it possible to detect later changes in software, modules or kernel versions, 8 https://github.com/RRZE-HPC/MachineState

any of which may account for the deviations observed during regression testing. Once node assignment is complete, all jobs are submitted simultaneously and recorded in a bookkeeping database, from which other workers can retrieve the information they need.

ClusterBench ships with two persistent database objects. Embedded databases are preferable to external database services, as they avoid introducing an additional runtime dependency on the framework. 1) Book-keeping database: A persistent SQLite database records all submitted jobs and their state. Because SQLite is row-oriented, it is well suited for storing per-job records such as the partition and nodes a benchmark was submitted to and whether it completed or failed. 2) Analytics database: A persistent DuckDB database stores the output produced by the benchmarks. Because DuckDB is column-oriented, it supports the fast aggregate queries needed to build responsive dashboards. As metrics can be collected per benchmark, the output extractor records both the benchmark result itself and the associated metrics, such as frequency, temperature and power draw. With all of these values held in a single database, users can visualize them in the dashboarding tool of their choice; the persistent database object can be used directly as a data source in Grafana, Elasticsearch, or similar tools. C. Metric-collector script ClusterBench ships with a Perl-based metric collector. When metric collection is enabled for a benchmark in its definition file, the benchmark is wrapped in a collector that runs alongside it and writes the hardware metrics of the interval to a separate file. For CPUs, temperature and clock frequency are read from the sysfs interface, and power is read from the kernel powercap interface, which requires RAPL support on the processor. For GPUs, the metrics are queried from nvidia-smi or amd-smi. Because the collector is bound to the benchmark, the metrics cover exactly the interval that the benchmark was running for. Metric collection therefore depends on the node configuration. Power readings require that the powercap files are readable by the job user, which is not the default on recent kernels. Where a source is unavailable, the affected metric is omitted and the remaining ones are still recorded. D. Background workers 1) Node-list populator: This worker runs at startup and, by default, populates lists of the nodes that are idle or allocated in each available partition. The lists are refreshed every 30 seconds, which is particularly useful for periodically scheduled (cron) benchmarks. The worker can also be configured to track other node states, such as drained, down, reserved or under maintenance, although only the idle and allocated states are considered by default.

2) Running-job checker: ClusterBench offers two ways to check whether the benchmarks have completed or not: normal update mode, which runs only once during the startup of the framework, or a separate worker that checks and updates the benchmark every specified interval. A separate worker inspects the running jobs every 5 seconds (configurable) and updates their state in the database as either completed or failed. 3) Output extractor: This functionality selects all completed and failed jobs from the book-keeping database scheduler.db and searches their output for the values of interest. Extraction can be performed in two ways. The first is to specify a regular expression, from which the first numerical value in the matched pattern is taken. The second is to supply a path to a user-provided shell script, which receives inputs such as the output file name, the cluster name and the partition name, processes the output file and returns a single numerical value. The extracted values are then written to the analytics database results.db. Moreover, if the metric measurement was selected in benchmark definition file and if the metric measurement file exists for the respective benchmark, then it will also average all the metrics over time, and store an averaged value in the database. ClusterBench also offers to extract LIKWID9 [25] measurements over sysfs measurements. If the LIKWID measurements are present in the output file, then the user can configure to extract them instead of averaging the measurements from sysfs metrics. 4) File compressor: This functionality compresses the various output files once the extraction phase has completed; extraction is therefore a prerequisite for compression. Because the framework supports cluster-wide and component-level benchmarking and can additionally record per-benchmark metrics, the volume of output it produces grows rapidly. Compressing the output files on a per-benchmark basis reduces both the disk space consumed and the number of files retained. IV. B ENCHMARK COLLECTION The benchmark collection10 is a curated collection of nodelevel benchmarks assembled for continuous regression testing of production HPC systems. Its design follows a componentoriented principle: every benchmark is associated with exactly one hardware component within a single compute node, and it is configured such that it places the highest possible load or stress on that component. The relevant components are the CPU, the GPU, main memory, GPU device memory, the communication fabric, and the local disk. Since each measurement is intended to approach the attainable hardware limit rather than to represent a particular application, the resulting figures can be compared against vendor specifications and, more importantly, against earlier runs on the same hardware. The suite is deployed through ClusterBench, which schedules the benchmarks across nodes and collects the results. The individual benchmarks are described in the following.

a) HPL: The High Performance Linpack (HPL)11 benchmark solves a dense system of linear equations by LU decomposition and is the de-facto standard for reporting floatingpoint throughput of HPC systems. We use it to characterize the compute performance of the CPU, and rely on vendoroptimized builds because these reach peak floating-point performance at a comparatively small matrix size. A vanilla HPL build, by contrast, must be configured with a matrix that occupies roughly 80% of main memory, which results in run times well in excess of 30 minutes and is therefore impractical for health checking the nodes in production clusters. b) DGEMM and SGEMM: On the GPU we use dense matrix–matrix multiplication from the vendor BLAS libraries as the compute-bound counterpart of HPL, since it exhibits high arithmetic intensity and drives the floating-point units close to their limit. Depending on the ratio of FP64 to FP32 hardware units, either DGEMM (double precision) or SGEMM (single precision) is the appropriate choice. HPL itself is not suitable in this role: the VRAM capacity of a GPU is far smaller than the main memory of a node, so only modest matrix sizes can be allocated, and the resulting run completes within a few minutes. Such a short run is sufficient to report a peak figure but leaves the device under load for too little time to serve as a stress test. A GEMM kernel, in contrast, can be repeated for a reasonable duration at a constant load level. c) TheBandwidthBenchmark: This benchmark12 comprises a set of streaming kernels that differ in their data traffic and in their ratio of floating-point operations to transferred bytes. It is executed on both the CPU and the GPU and yields the saturated bandwidth of main memory and of GPU device memory, respectively. Recent server-grade NVIDIA GPU architectures such as Blackwell and Rubin provide an increasingly wide memory interface, and saturating such an interface requires a sufficient number of bytes in flight. TheBandwidthBenchmark therefore offers vectorized load and store variants of its kernels, which are necessary to reach the attainable bandwidth on such devices. d) OSU Micro-Benchmarks: The OSU MicroBenchmarks13 cover a comprehensive range of MPI communication patterns and also support GPU device buffers. They are straightforward to deploy and include dedicated tests for GPU-aware communication. We use the multi-rank CPU-only benchmarks to characterize and stress test the inter-node communication fabric. For the intra-node GPU interconnect, such as NVLink and NVSwitch, we use the all-to-all communication pattern, since it stresses all links simultaneously and thus reveals the aggregated latency of the interconnect. e) fio: The Flexible I/O Tester (fio)14 generates synthetic I/O workloads from a job description that specifies the access pattern, block size, number of concurrent jobs, and queue depth, and reports bandwidth, I/O operations per second, and 11 https://www.netlib.org/benchmark/hpl 12 https://github.com/RRZE-HPC/TheBandwidthBenchmark

9 https://github.com/RRZE-HPC/likwid

13 https://mvapich.cse.ohio-state.edu/benchmarks

10 https://github.com/RRZE-HPC/benchmark-collection

14 https://github.com/axboe/fio

TABLE I: Benchmarks in the collection, the node component they are aimed at, and the reported metric. Metric Floating-point performance Floating-point performance Memory bandwidth Bandwidth and latency I/O bandwidth, IOPS, latency

* Vendor-optimized implementation.

450 300 150 0

latency distributions. We apply it to the node-local disk with direct I/O enabled, so that the page cache is bypassed and the measurement characterizes the storage device itself. Since read and write paths differ in cost, the reported bandwidth strongly depends on the mix of the two. Currently, we use a ratio of 20% reads and 80% writes, because the lifespan of an SSD is governed by the volume of data written before its performance degrades, and a write-dominated workload is therefore the more demanding and the more relevant case. Table I summarizes the benchmarks together with the component they target and the metric they report. All benchmarks are integrated as Git submodules and are built through a common Makefile, so that the whole suite can be obtained and compiled with a single recursive clone. This keeps the individual benchmarks at their upstream sources and makes the exact version used for a given measurement reproducible. V. C LUSTER - WIDE STATISTICAL VARIABILITY OF CPU AND GPU A. Testbed and methodology Table II summarizes the details of the CPUs, GPUs, and interconnects within each cluster and partition used for the tests presented in this study. Our benchmarking methodology involves testing the entire cluster every two weeks using ClusterBench. For different clusters and multiple repeated runs, we use a fixed set of precompiled binaries, reusing the same binary across repeated benchmarks to eliminate build-to-build variance; configuration settings (e.g., cluster and partition parameters) are adjusted according to the specific cluster and partition being tested. Each data point in the correlation plots, and each count in the distribution plots, represents one end-to-end benchmark run. We run each benchmark for at least 10 minutes to stress the components long enough. Benchmark performance numbers are obtained through reported performance numbers. For metric data points, we obtain a single data point by averaging the metrics collected during that benchmark run. Since the benchmarks run with stable workloads for a long enough time, the average converges to the expected value of the metric. The averaged data point is therefore a reliable summary of the benchmark run. In the normalized plots, the data has been normalized with respect to the range of observed metric values.

median = 3.812 ± 2.45%

600

Count

Benchmark Target component HPL* CPU DGEMM/SGEMM* GPU TheBandwidthBenchmark CPU and GPU memory OSU Communication Micro-Benchmarks fabric fio Local disk

750

3.6

3.7

3.8

3.9

HPL score [TFLOP/s]

4.0

Fig. 2: Per-node CPU HPL performance distribution across 992 nodes of singlenode partition on Fritz (Inteloptimized, May 2025–July 2026).

In the distribution plots, the median value is reported. Along with the median value, the percentage fluctuation between the 5th and 95th percentiles of the distribution is also reported. B. Results The performance variation of dual-socket HPL runs is first examined on the largest CPU-only cluster at NHR@FAU. The singlenode partition on Fritz consists of 992 nodes with dual-socket Intel Ice Lake SP processors. Figure 2 shows a performance variation of up to 5% around a mean of 3.8 TFlop/s. The peak throughput of the dualsocket Ice Lake chips in this partition is approximately 5.5 TFlop/s. This gap is explained by core frequency throttling: the base core frequency is 2.4 GHz, while the sustained frequency during HPL execution is 1.85 GHz. The distribution aggregates data from all nodes over one year, combining spatial and temporal variation. The absence of extreme outliers indicates that the majority of nodes are healthy. A closer examination of two individual nodes from the singlenode partition provides further insight. Figure 3 shows the per-node distribution of performance and related metrics. Each node exhibits a performance variation of less than 1%, and each metric shows a distinct pattern of fluctuation. Node f0383 shows a slightly higher power draw and more frequency throttling than node f0384, which results in lower performance. These distributions indicate that each node has a characteristic operating signature. Because HPL is compute bound and stresses the CPU, it is necessary to determine whether the same nodes behave differently under memory-bound stress. We use repeated calls to the Schoenauer Triad kernel (a[i] = b[i] + c[i] * d[i]) from TheBandwidthBenchmark, where a, b, c, and d are arrays. Such streaming kernels are sufficient to stress the memory interface. Figure 4 shows that the memory-bound code produces different characteristics on the same two nodes. The fluctuation in memory power draw is smaller than for

TABLE II: Hardware specifications across NHR@FAU clusters and different partitions Cluster

Alex

Helma

Fritz

Partition

a100

a40/a100

h100

cpu

singlenode

spr2tb

Target

GPU

Host CPU

GPU

CPU

CPU

CPU

NVIDIA AMD EPYC NVIDIA AMD EPYC Intel Xeon Intel Xeon A100 7713 H100 9965 8360Y 8470 Microarch. Ampere Zen 3 Hopper Zen 5c Ice Lake Sapphire Rapids SMs / Cores 108 SMs 64 Cores 132 SMs 192 Cores 36 Cores 52 Cores Per Node 8 GPUs 2 Sockets 4 GPUs 2 Sockets 2 Sockets 2 Sockets # Nodes 20 / 18 82 96 312 992 48 Device TDP 400 W 225 W 700 W 500 W 250 W 350 W Base Freq. 1.09 GHz 2.00 GHz 1.6 GHz 2.25 GHz 2.40 GHz 2.00 GHz Boost Freq. 1.41 GHz 3.68 GHz 1.98 GHz 3.70 GHz 3.50 GHz 3.80 GHz Peak FP64 FLOPS 19.4 T 2.05 T 33.45 T 6.90 T 2.76 T 3.33 T 40 GB / 1 TB / Mem. Capacity 94 GB 768 GB 256 GB 2 TB 80 GB 2 TB Mem. BW 1.5 TB/s / 2 TB/s 204.8 GB/s 2.41 TB/s 460.8 GB/s 204.8 GB/s 307.2 GB/s Mem. Tech HBM2 DDR4 HBM2e DDR5 DDR4 DDR5 Mem. Clock 1215 MHz / 1593 MHz 3200 MT/s 1593 MHz 5600 MT/s 3200 MT/s 4800 MT/s Interconnect 2 x HDR200 4 x NDR200 1 x NDR200 1 x HDR100 1 x HDR100 Cooling Air Air DLC DLC DLC DLC

34.41 ± 2.91% 35.47 ± 2.86%

34

35

36

37

Memory power draw [W]

3.85

240 1,808 ± 0.67% 1,835 ± 0.45% 200 160 120 80 40 0 1810 1820 1830 1840

Frequency [MHz]

Fig. 3: Distribution of CPU HPL performance and CPU metrics for nodes f0383 (blue) and f0384 (orange) of singlenode partition on Fritz (Intel-optimized, May 2025–July 2026).

HPL, although the spread of power draw remains similar; the shape of the distribution changes. The higher power draw on node f0383 leads to greater frequency fluctuation, whereas node f0384 draws less power, remains further from its TDP limit, and maintains a more stable frequency. For both compute-bound and memory-bound code, node f0383 underperforms node f0384 slightly, consistent with its higher power draw. These observations motivate an examination of correlations between metrics. In Figure 5, each node occupies a distinct region in the correlation plot, forming a separate cluster. A positive correlation between frequency and performance is expected, since reduced frequency limits the throughput of compute-bound code. For the nodes in the singlenode partition, this relationship holds and confirms the expectation.

492.1 ± 0.26% 497.0 ± 0.02%

Count

Count 3.80

750 600 450 300 150 0

490.0 492.5 495.0

Power draw [W]

150 120 90 60 30 0

49.00 ± 0.27% 49.76 ± 0.24%

48.5

49.0

49.5

150 125 100 75 50 25 0

304.9 ± 0.26% 305.6 ± 0.25%

304

306

Triad score [GB/s]

Count

180 150 120 90 60 30 0

3.841 ± 0.46%

HPL score [TFLOP/s]

Count

Count

Power draw [W]

180 3.750 ± 0.46% 150 120 90 60 30 0 3.75

Count

494.9 ± 0.02% 495.2 ± 0.02% 240 200 160 120 80 40 0 494.5 495.0 495.5 496.0

Count

Count

Chip Name

50.0

Memory power draw [W]

1250 3,081 ± 0.08% 3,093 ± 0.00% 1000 750 500 250 0 3080 3085 3090

Frequency [MHz]

Fig. 4: Distribution of CPU Schoenauer Triad kernel performance and CPU metrics for nodes f0383 (blue) and f0384 (orange) of singlenode partition on Fritz (May 2025–July 2026).

However, data from two additional nodes in the spr2tb partition illustrate a risk in correlating metrics and performance at cluster level than at individual level. At the level of an individual node, frequency and performance remain positively correlated, as expected. When data from both nodes are combined across time, however, the aggregate correlation appears negative, which does not reflect the true relationship. This shows that correlation data at the individual node level might be different from the correlating data across the whole cluster at once. The outlier nodes in the cluster may sway the correlation to a certain extent. It also shows that each CPU has a characteristic operating signature, visible as a distinct cluster in the scatter plot. Dual-socket HPL runs were also performed on the AMD

spr2tb

0.8

0.8

0.6 0.4

Count

all: r = − 0.462

2 0

0.4

70

Temperature [ ◦ C]

80

0.4

0.6

HPL score (normalized)

0.8

0.00.0

1.0

0.2

0.4

0.6

HPL score (normalized)

0.8

1.0

3.10

3.15

3.20

HPL score [TFLOP/s]

3.25

1.0 r = 0.793

1.0

0.8

0.8

0.4 0.2

r = − 0.873

0.6

Count

0.6

0.4 0.2

0.2

0.4

0.6

HPL score (normalized)

0.8

1.0

0.00.0

Fig. 7: Distribution of per-node CPU HPL performance and CPU temperature across 64 nodes in the a40 and a100 partition of Alex (AMD-optimized, 31 Jan 2026). 180 150 120 90 60 30 0380 100 80 60 40 20 0

396.5 ± 2.13%

385

390

395

Power draw [W]

0.4

0.6

Power draw (normalized)

0.8

1.0

Fig. 6: Correlation between per-node CPU HPL performance and CPU metrics across 300 nodes in the cpu partition on Helma (AMD-optimized, 2 July 2026).

Turin Dense nodes (384 cores per node) in the cpu partition of the Helma cluster. Figure 6 presents a single-timestamp snapshot of the cpu partition, with one data point per node. Given the sample size is big enough, this confirms the expected positive correlation between frequency and performance. It also shows that higher power draw is associated with greater frequency throttling, producing a strong negative correlation between power draw and frequency for compute-bound code. Because each node forms a distinct cluster in the correlation plot, restricting the analysis to one point in time per node provides a more reliable method for computing cluster-wide correlations between metrics and performance. The Alex cluster contains two partitions, a40 and a100, named for their NVIDIA A40 and NVIDIA A100 GPUs. The cluster is air-cooled. We evaluated the host CPUs of both partitions under stress testing. Figure 7 shows that, despite using identical CPU models, the CPUs in the a100 partition run approximately 10 degrees hotter than the CPUs in the a40 partition. Higher temperature would be expected to increase throttling and reduce performance; instead, the CPUs in the a100 partition perform slightly better. This result is unexpected, and the underlying cause remains unclear.

80

240 200 160 120 80 40 0

65

70

75

19.12 ± 1.77%

18.5

19.0

GPU DGEMM score [TFLOP/s]

71.68 ± 9.46%

Temperature [ ◦ C]

0.2

400

200 160 120 80 40 0

Count

0.2

Frequency (normalized)

Frequency (normalized)

60

3.187 ± 4.45% 3.225 ± 0.75%

0.2

Fig. 5: Correlation between CPU HPL performance and frequency for the singlenode partition (nodes f0383 (blue), f0384 (orange)) and spr2tb partition (nodes f2183 (blue), f2184 (orange)) on Fritz (Intel-optimized, May 2025–July 2026).

0.00.0

4

15 12 9 6 3 0 3.05

Count

0.00.0

64.36 ± 8.93% 74.05 ± 9.01%

6

f2184: r = 0.188

Count

0.2

0.6

8

Count

1.0 f2183: r = 0.694

Frequency (normalized)

Frequency (normalized)

singlenode

1.0 all: r = 0.915

1,391 ± 1.78%

1325

1350

1375

Frequency [MHz]

1400

Fig. 8: Distribution of per-GPU DGEMM performance and GPU metrics for all 304 GPUs (38 nodes) of the a100 partition on Alex (CuBLAS-optimized, Nov 2025–July 2026).

We also examined compute-bound and memory-bound codes on GPUs, which show different performance variability depending on code type. Figure 8 shows the performance variability of the compute-bound GEMM kernel on the A100 GPUs. An identical DGEMM binary was submitted to each individual GPU within the node. Performance varies by less than 5% across space and time. The distribution of SM frequency closely matches the distribution of DGEMM performance, indicating a strong correlation between the two. GPU temperature shows the highest variability of the measured metrics, consistent with the air-cooled design of these nodes. Figure 10 examines the correlation between performance, frequency, and temperature in more detail. Before turning to these correlation plots, however, the distributional analysis of memory-bound code on GPUs merits further discussion. To stress the memory hierarchy and components of the GPU, the Schoenauer Triad kernel from TheBandwidthBenchmark is again used. Figure 9 shows stable Schoenauer Triad performance for the GPUs. This performance stability, observed across all 160 GPUs (20 nodes) over time, suggests that these GPUs retain substantial power headroom. Although the distribution of GPU power draw shows high variation, the GPUs do not reach their TDP limit. Consequently, SM frequency does not throttle, and Schoenauer Triad performance

50

55

Temperature [ ◦ C]

GPU Triad score [TB/s]

1.400

1405

1410

Frequency [MHz]

0.8

Temperature (normalized)

Frequency (normalized)

0.8

0.00.0

0.6 0.4 0.2

0.6 0.4 0.2

0.2

0.4

0.6

0.8

0.00.0

1.0

GPU DGEMM score (normalized)

0.2

0.4

0.6

0.8

GPU DGEMM score (normalized)

1.0

1.0 r = − 0.581

1.0 r = − 0.494

0.2

0.8

0.00.0

1.0 r = 0.990

0.4

0.8

1,410 ± 0.00%

1415

Fig. 9: Distribution of per-GPU Schoenauer Triad performance and GPU metrics for all 160 GPUs (20 nodes) of the a100 partition on Alex (Nov 2025–July 2026).

0.6

1.0 r = − 0.038

Memory power draw (normalized)

53.17 ± 10.87%

Count

Count

Power draw [W]

75 60 45 30 15 0

1.395

1.0 r = 0.877

Temperature (normalized)

60

900 750 600 450 300 150 0

230

1.395 ± 0.02%

Frequency (normalized)

240

900 750 600 450 300 150 0 1.390

Count

Count

213.6 ± 6.88% 90 75 60 45 30 15 0200 210 220

0.8 0.6 0.4 0.2 0.00.0

0.6

0.2

0.4

0.6

0.8

GPU DGEMM score (normalized)

1.0

Fig. 11: Correlation between per-GPU DGEMM performance and GPU metrics across all 384 GPUs (96 nodes) in the h100 partition on Helma (CuBLAS-optimized, Jan 2026–July 2026).

0.4 0.2

0.2

0.4

0.6

0.8

GPU DGEMM score (normalized)

1.0

0.00.0

0.2

0.4

0.6

0.8

GPU DGEMM score (normalized)

1.0

Fig. 10: Correlation between per-GPU DGEMM performance and GPU metrics across all 304 GPUs (38 nodes) in the a100 partition on Alex (CuBLAS-optimized, Nov 2025–July 2026).

remains stable. GPU temperature again shows high variability, consistent with the DGEMM results. Such stable Schoenauer Triad performance with less than 0.05% variability was also observed on other GPU types in the tested clusters, including the NVIDIA A40, H100, and H200. Correlation plots allow us to test two assumptions drawn from the GPU DGEMM distribution plots above: the dependence of the DGEMM performance on SM frequency and GPU temperature. Figure 10 confirms both. SM frequency correlates strongly and positively with performance, consistent with expectation. GPU temperature correlates negatively with performance, a relationship that is more complex and cluster dependent. Because the a100 partition is air cooled, thermal throttling is a contributing factor: higher temperature increases SM frequency throttling, which in turn reduces the DGEMM performance. This relationship does not hold for the GPUs in the h100 partition. The NVIDIA H100 GPUs in the Helma cluster are water cooled and show a different relationship between GPU temperature and DGEMM performance. Figure 11 shows no corre-

lation between GPU temperature and DGEMM performance in this case, even though these GPUs also reach their TDP limit and experience SM frequency throttling. The median for the H100 GPU temperature was (53 ± 2) ◦ C. The strong correlation between SM frequency and benchmark performance observed previously also holds for the H100 GPUs. The correlation between memory power draw and DGEMM score reveals an additional pattern. Because GPU memory power draw is included in the total GPU power budget, an increase in GPU memory power draw reduces the power available to other components, lowering SM frequency and, consequently, performance [26]. This produces the observed negative correlation between DGEMM performance and GPU memory power draw. This relationship does not hold for codes that do not reach TDP and therefore do not experience SM frequency throttling; in those cases, GPU memory power draw and performance show no correlation. These results raise several questions for future investigation. How frequently should component-level health checks be performed: daily, weekly, or monthly? How does the observed distribution evolve over time, whether across years or over the operational lifetime of a cluster? What machine learning or statistical methods are best suited to capture performance fluctuations or degradation? A complete treatment of these questions requires additional time and data and is left for future work.

VI. L IMITATIONS AND F UTURE W ORK FOR C LUSTER B ENCH ClusterBench was designed for a narrow purpose: repeated, component-level measurement of every node in a production system, whether for cluster validation or ongoing health monitoring. Several capabilities that a general-purpose benchmarking framework would offer are therefore absent by design in ClusterBench rather than by oversight. This section states those boundaries, the reasoning behind them, and the work planned beyond them. A. Compilation phase We have deliberately kept the compilation phase out of this framework. Since the target is to perform cluster-wide component-level testing, the burden to provide correct binaries falls onto the user. The user needs to provide the binary and provide the correct path to it. Compilation phase implementation adds lots of complexity, for which this framework was never designed. Production environments do not change rapidly, so the dependent modules or compilers would be present for a single binary. Recompiling the benchmarks for every run is unnecessary when the health check is executed weekly. It adds build overhead to each cycle. Compilation behavior is also not the target of the measurement: the goal is to stress the components, not to characterize the build. The binaries are therefore built once and reused across runs. MachineState allows capturing the software environment to a certain extent, but documentation of the configuration management for the binaries, e.g., benchmark version, build environment, dependencies (MPI, CUDA), etc., is not captured. If required, there are existing frameworks that can be coupled with ClusterBench to function as an end-to-end regression testing framework. B. Parameter space exploration ClusterBench does not sweep input parameters or ranges. A component-level health check answers a narrower question, namely whether a node behaves like its peers under one fixed configuration, and holding that configuration constant is what makes results comparable across nodes and across time. A sweep would multiply the node count by the size of the parameter space, which is difficult to justify against a production allocation for routine monitoring. However, the capability is useful for characterization work and its addition is planned. C. Scheduler support Slurm is the most widely used scheduler in HPC environments today, and ClusterBench currently supports only Slurm. The scheduler-facing logic is limited to three tasks: submission, state polling, and accounting queries. Because this interface is narrow, support for other schedulers can be added later as separate backends without changing the core logic of the framework. PBS is an older scheduler and is used less often now. Flux is newer and not yet widely used in production systems; support for either can be added if a clear need arises.

HTCondor comes from the AI community and is mature, so it may be worth supporting if it becomes more common in HPC settings. VII. C ONCLUSION This paper presented ClusterBench, a framework for clusterwide continuous benchmarking and health monitoring of production systems. The framework treats the cluster, rather than a single node, as the object under test. A test definition is set up once for a cluster or a partition, and ClusterBench derives from that definition the submission of identical benchmarks to every node, to node pairs for interconnect measurements, and to each accelerator within a node. Each result is stored together with the node, the timestamp, and the software environment, and every benchmark is wrapped in a metric collector that records power draw, core frequency, and temperature over the same interval. Results therefore accumulate along two axes, across the nodes and across their lifetime. Using the accompanying component-level benchmark suite, we observed the NHR@FAU clusters Fritz, Alex, and Helma over more than one year. Performance variation within a single component stays below 1%, while variation across component specimens of the same type reaches up to 5% on nodes that are identical by specification. The distributions of performance and of the accompanying metrics are narrow and reproducible per node, which indicates that each component has a characteristic operating pattern rather than a single nominal value. The absence of extreme outliers in these distributions is itself the result a data center needs from an acceptance or health check. The correlation between performance and collected metrics strongly depends on how the data is aggregated. Within a single node, frequency and performance are positively correlated, as expected for compute-bound code. Once measurements from multiple nodes are combined across time, however, this correlation can invert, no longer reflecting the underlying pernode relationship. A snapshot taken at one point in time, with one measurement per node, is therefore the more reliable basis for such an analysis. The relationship between temperature and performance further depends on the node design: it is clearly negative on the air-cooled A100 nodes of Alex and absent on the water-cooled H100 nodes of Helma. Memory-bound code on GPUs behaves differently again compared to CPUs, remaining stable where the TDP limit is not hit and frequency does not throttle. Comparisons across runs are therefore easier to interpret when the observed performance and metric spread for the given cluster is known, and continuous measurement is one way to obtain it. The dataset accumulated in this way also serves a second purpose beyond validation, as a long-term record for the study of hardware variability and degradation. Future work will extend the framework along the directions outlined in Section VI and use the growing dataset to determine suitable intervals for component-level health checks and advanced statistical/machine learning methods for detecting gradual degradation.

A RTIFACT E VALUATION The ClusterBench software, all the statistical data from the Results section, the compiler flags used to build and the exact benchmark parameters to run the benchmarks in this paper are all available through Zenodo at https://doi.org/10.5281/ zenodo.21870554. For verifying the statistical data, a simple dashboard and the duckdb database object is also shipped in the given DOI repository. Detailed information can be further provided upon request. ACKNOWLEDGMENT The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National HPC Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the BayernKI project v111dc. BayernKI funding is provided by Bavarian state authorities. The authors gratefully acknowledge the scientific support and HPC resources provided by NHR@FAU of the FAU under the NHR project b104dc. NHR funding is provided by federal and Bavarian state authorities. NHR@FAU hardware is partially funded by the German Research Foundation (DFG) – 440719683. R EFERENCES [1] S. Lührs, D. Rohe, A. Schnurpfeil, K. Thust, and W. Frings, “Flexible and generic workflow management,” in Parallel Computing: On the Road to Exascale (ParCo 2015), vol. 27 of Advances in Parallel Computing, pp. 431–438, IOS Press, 2016. [2] V. Karakasis, T. Manitaras, V. Rusu, R. Sarmiento-Pérez, C. Bignamini, M. Kraushaar, A. Jocksch, S. Omlin, G. Peretti Pezzi, J. Augusto, B. Friesen, Y. He, L. Gerhardt, B. Cook, Z.-Q. You, S. Khuvis, and K. Tomko, Enabling Continuous Testing of HPC Systems Using ReFrame, pp. 49–68. 03 2020. [3] Los Alamos National Laboratory, “Pavilion2: A framework for running and analyzing tests targeting HPC systems.” https://github.com/hpc/ pavilion2, 2017. [4] J. K. Green, N. C. Sly, M. F. T. R. Lapid, J. A. Ogas, K. J. Everson, D. J. Magee, C. D. Seamons, and P. S. Ferrell, “Acceptance testing the chicoma hpe-cray ex supercomputer,” Los Alamos National Laboratory (LANL), 06 2021. [5] S. Siddiqui, “Buildtest: A software testing framework with module operations for hpc systems,” in Tools and Techniques for High Performance Computing (G. Juckeland and S. Chandrasekaran, eds.), (Cham), pp. 3– 27, Springer International Publishing, 2020. [6] D. Dietz et al., “OLCF test harness,” 2025. [7] Texas Advanced Computing Center, “BenchPRO.” https://github.com/ TACC/benchpro, 2023. [8] D. Jacobsen and R. F. Bird, “Ramble: A flexible, extensible, and composable experimentation framework,” in SC’23 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 600–608, 2023. [9] A. Parab, A. Raoofy, L. Spörl, S. Dimitrov, M. Tovey, and J. Weidendorfer, “Autobench: A holistic platform for automated and reproducible benchmarking in hpc testbeds,” in Companion of the 16th ACM/SPEC International Conference on Performance Engineering, ICPE ’25, (New York, NY, USA), p. 207–214, Association for Computing Machinery, 2025. [10] O. Pearce et al., “Towards collaborative continuous benchmarking for HPC,” in SC’23 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 627– 635, 2023. [11] J. Badwaik, M. Bode, M. Rajski, and A. Herten, “exaCB: Reproducible continuous benchmark collections at scale leveraging an incremental approach,” arXiv preprint arXiv:2603.22251, 2026.

[12] S. Brink et al., “Thicket: Seeing the performance experiment forest for the individual run trees,” in Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’23), pp. 281–293, 2023. [13] T. Koskela, I. Christidi, M. Giordano, E. Dubrovska, J. Quinn, C. Maynard, D. Case, K. Olgu, and T. Deakin, “Principles for automated and reproducible benchmarking,” in Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W ’23, (New York, NY, USA), p. 609–618, Association for Computing Machinery, 2023. [14] D. H. Bailey, E. Barszcz, J. T. Barton, D. S. Browning, R. L. Carter, L. Dagum, R. A. Fatoohi, P. O. Frederickson, T. A. Lasinski, R. S. Schreiber, H. D. Simon, V. Venkatakrishnan, and S. K. Weeratunga, “The NAS parallel benchmarks,” The International Journal of Supercomputing Applications, vol. 5, no. 3, pp. 63–73, 1991. [15] P. Luszczek, D. Bailey, J. Dongarra, J. Kepner, R. Lucas, R. Rabenseifner, and D. Takahashi, “Introduction to the HPC challenge benchmark suite,” tech. rep., Lawrence Berkeley National Laboratory, 2005. OSTI 860347. [16] J. Dongarra and P. Luszczek, “HPC challenge: Design, history, and implementation highlights,” in Contemporary High Performance Computing: From Petascale toward Exascale (J. S. Vetter, ed.), vol. 1, ch. 2, Chapman and Hall/CRC, 2013. [17] A. Herten, S. Achilles, D. Alvarez, et al., “Application-driven exascale: The JUPITER benchmark suite,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC’24), 2024. [18] B. Rountree, D. H. Ahn, B. R. de Supinski, D. K. Lowenthal, and M. Schulz, “Beyond DVFS: A first look at performance under a hardware-enforced power bound,” in IEEE 26th International Parallel and Distributed Processing Symposium Workshops and PhD Forum, pp. 947–953, 2012. [19] Y. Inadomi, T. Patki, K. Inoue, M. Aoyagi, B. Rountree, M. Schulz, D. Lowenthal, Y. Wada, K. Fukazawa, M. Ueda, M. Kondo, and I. Miyoshi, “Analyzing and mitigating the impact of manufacturing variability in power-constrained supercomputing,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC15), pp. 78:1–78:12, 2015. [20] B. Acun, P. Miller, and L. Kale, “Variation among processors under Turbo Boost in HPC systems,” in Proceedings of the 2016 International Conference on Supercomputing (ICS), 2016. [21] A. Marathe, Y. Zhang, G. Blanks, N. Kumbhare, G. Abdulla, and B. Rountree, “An empirical survey of performance and energy efficiency variation on Intel processors,” in Proceedings of the 5th International Workshop on Energy Efficient Supercomputing (E2SC’17), 2017. [22] S. Bhalachandra, B. Austin, and N. Wright, “Understanding power variation and its implications on performance optimization on the Cori supercomputer,” 2021. [23] K. Yoshida, R. Sageyama, S. Miwa, H. Yamaki, and H. Honda, “Analyzing performance and power-efficiency variations among NVIDIA GPUs,” in Proceedings of the 51st International Conference on Parallel Processing (ICPP), 2022. [24] T. Gruber, Jan, Julian, christiealappatt, B. Muite, and Hamza-Ghenai, “Rrze-hpc/machinestate: Machinestate-0.6.0,” June 2025. [25] J. Treibig, G. Hager, and G. Wellein, “LIKWID: A lightweight performance-oriented tool suite for x86 multicore environments,” in Proceedings of PSTI2010, the First International Workshop on Parallel Software Tools and Tool Infrastructures, 2010. [26] A. Ujeniya, J. Eitzinger, G. Hager, and G. Wellein, “Architectural tradeoffs in the energy-efficient era: A comparative study of power-capping nvidia h100 and h200,” 2026.

Related documents

• EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments#343496
• Predictive models for computing clusters: a case study based on Ilifu#416039
• ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction#822473
• A Framework for Testing Machine Learning Systems: Data-Box and Black-Box Perspectives#211402
• A unified test for regression discontinuity designs#635513