arXiv CS open access

Application Failures and Machine Computational Efficiency

2026 · arxiv_cs

arXiv CS · Papers · 2026

cloud distributed-computing parallel-computing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Application Failures and Machine Computational Efficiency∗

arXiv:2608.05408v1 [cs.DC] 5 Aug 2026

Carlo Graziani †

Bethany Lusch∗

Abstract. We present a framework for evaluating uptime efficiency of Exascale-class scientific computers when application failure rates are appreciable. This is the situation that confronts current leadership-class scientific computing platforms and large AI training installations. What distinguishes scientific computing platforms is the heterogeneity of their applications. We argue that this diversity requires that failure rates and mean intervals between failures should be specified in terms of usage (e.g. node-hours) rather than time, as is currently customary. We consider the usage loss terms due to failures, to checkpointing, and to restart costs, and update the framework of Daly (2006) allowing users to specify optimal checkpointing usage intervals that minimize such losses. We derive the machine computational efficiency, which specifies the expected fractional resource allocation that is available for scientific computation. We illustrate the methodology using one year of production runtime data from the Frontier supercomputer at Oak Ridge National Laboratory. 1 Introduction The study of the effects of faults in shared computing systems, and of resilient computation in the presence of hardware and software faults have been an active area of research for many decades now [13, 9, 15, 4, 10, 14]. This research is growing in relevance to scientific high-performance computing (HPC), because with the introduction of the newest Exascale computing platforms has come a reckoning with the fact that applications running at the largest scales face mean times between failure (MTBF) measured in hours, rather than in days or weeks. Adapting applications and runtime environments to this new landscape of frequent faults is necessarily a high priority for both developers of scientific codes and for HPC facilities. As of this writing, many datacenters belonging to technology companies are dedicated to training AI models at very large scales, and there is some current liter∗ The U.S. Government retains a nonexclusive, royalty-free license to publish or reproduce the published form of this contribution, or allow others to do so, for U.S. Government purposes. † Argonne National Laboratory, Lemont, IL, USA (cgraziani@anl.gov, blusch@anl.gov). ‡ Oak Ridge National Laboratory, Oak Ridge, TN, USA (bronson@ornl.gov)

O. E. Bronson Messer ‡

ature describing the experiences of such centers in confronting the same issues of frequent computing faults [5, 7]. It is noteworthy, however, that not all these experiences translate exactly into the the types of challenges faced by facilities concerned with scientific HPC. The reason is that scientific HPC facilities grant time to a far more heterogeneous collection of application codes, including AI loads, but also computational fluid dynamics codes, Earth system model codes, molecular dynamics, cosmology, quantum chemistry, protein folding, and many others (e.g., see [12, 11, 8, 6]). Many of these applications stress computing nodes in different ways, exercising CPU and GPU hardware, memory, interconnects, and other hardware according to a variety of computing patterns, and also requiring differing job sizes (node counts) to run optimally. Large-scale jobs at a scientific HPC facility are not atypically scheduled for planned runtimes of 24 hours, and face considerable challenges in environments where the MTBF corresponding to their job size is shorter than that time. Designed-in resilience in the face of such fault rates is a simple necessity for such codes, and for the facility itself. The most basic and longest-standing measure adopted to ensure resilience is prophylactic checkpointing, wherein the code periodically stops the computation in order to write out a summary of its state in sufficient detail that the computation may be resumed from that point in time in the event of a later failure. Checkpoint cadence optimization has been studied in some detail, [13, 3, 5]. To date, however, this has always been done in a context where the fault rate is considered a constant common to all applications, which is not really the case when application jobs can differ in node count. If we idealize job failures as being largely caused by single-node faults, then failure risk is clearly a function of job size. In this case, a 2000-node job that runs for one hour is at the same failure risk as a 1000-node job that runs for two hours. The established frameworks for considering optimal checkpointing need to be updated to reflect this reality. There is a separate issue that is closely tied to the challenges of application failures and resilience: what fraction of a supercomputer’s capacity is actually available for scientific computing, after computing opera-

tions have been optimally hardened against failures? This is an important question facing HPC facilities, for at least two reasons: it is important to attach realistic performance numbers to HPC platforms, and those numbers are affected by the fraction of FLOPS employed to implement resilience measures rather than on domain science computation; and facilities need to know what fraction of the node-hours allocated to projects are employable for scientific computation, so that they may provision computing resources to projects in a rational manner. Our contributions to these subjects are twofold. In the first place, we broaden previous work on checkpointing and resilience by changing their operating context from time to usage (that is, node-hours). Re-expressing operating costs and failure losses in node-hours rather than merely in computing time allows us to assimilate jobs running at different scales for different times, yielding consistent notions of optimal checkpointing cadence and of the amount of computational resources required to complete a calculation To this end, we introduce the notion of per-unit-usage failure rate R0 , a parameter with units of failures per node-hour. This parameter is the reciprocal of Mean Usage Between Failures (MUBF), the analog of MTBF in usage space. MUBF is an analogous quantity to "FLOPS per MTBF" [10]. We prefer R0 to MUBF in the technical development that follows below, since it seems to lead to simpler and more natural discussion, but readers more comfortable with MTBF can easily see the connection with R0 −1 . In the second place, we directly apply the optimal resilience results for applications at HPC facilities to the estimation of computational efficiency, which we define as the average fraction of total usage available for practical computation. This is similar in conception to “Effective Training Time Ratio” [7], but differs in the important respect that it is an average over a computational load that is representative of typical scientific HPC loads. The plan of the rest of the paper is as follows: in §2 we summarize the mathematical treatment of computing costs incurred as a consequence of a fixed rate of node-local application faults. In §3, we discuss the adaptation of classical results on optimal checkpointing from the time domain to the usage domain, from the perspective of domain scientists attempting to complete a specific computation. In §4, we discuss the costs and losses associated with an application whose computations are partitioned into discrete jobs. In §5 we adopt the perspective of an HPC facility running an ensemble of application jobs, and show how the previous results can be used to estimate the computational efficiency of an HPC platform. We discuss our implementation in

code of the analysis in §6. In §7 we show an example analysis using data from the Oak Ridge National Laboratory machine Frontier to furnish a fault rate and an ensemble of jobs characteristic of a scientific HPC facility’s computational load. We discuss the implications of our analysis methods and results in §8. 2 Summary of Failure-Related Costs Suppose that a large shared machine runs heterogeneouslysized jobs concurrently. The machine is composed of a large number of compute nodes which fail randomly, due to hardware or other system faults. Node faults cause application jobs to fail at some rate R0 , measured in failures per unit usage (i.e. failures/node-hr). The reciprocal of R0 is the mean usage between failures (MUBF), the usage analog of the mean time between failures (MTBF) that features in typical discussions of application failures (e.g. [3, 13]). MUBF is an analogous quantity to "FLOPS per MTBF" [10]. The reason for working with rates-per-unit-usage rather than per-unittime is that failure risk scales with job size: a 1000-node job that runs for 1 hour is at the same failure risk from node faults as a 500-node job that runs for two hours. To protect computations, applications checkpoint at a certain rate. Since the failure risk is measured in usage, the checkpoint intervals are also measured in usage: a 1000-node job runs up failure risk at twice the time rate of a 500-node job, and must accordingly checkpoint twice as frequently. The checkpoint usage cadence is denoted uc node-hours. This is the amount of usage dedicated to scientific computing between checkpoint writes. The costs associated with failures and checkpointing are as follows: • Each failure costs the loss of all usage consumed by computation or checkpoint writing since the last successful checkpoint; • Each checkpoint takes a certain amount of time, and hence consumes a certain amount of usage; • Each restart of a failed job costs a certain amount of usage for setup—including the potential cost of a node failure during startup. There is an additional cost incurred when a node experiences a non-recoverable hardware failure and needs to be reset manually or replaced during a maintenance period. We neglect this cost, since even if it applied to a few nodes per week, the associated cost in node-hours would be small. 2.1 Checkpointing Cost The principal abatement strategy to protect computations from failures is checkpointing: application codes periodically record Copyright © 2026 Copyright retained by principal author’s organization

their states, writing it to some non-volatile storage, so that if a failure occurs, only the output since the last checkpoint is lost. There are two types of checkpoints: global checkpoints, which effectively pause computation until the state of the application on all nodes is written out to a unitary checkpoint file on a global storage resource; and node-local checkpoints, which save a copy of the data locally, either to node-local storage or to on-node volatile memory (in which case a CPU on the node is dedicated to routing the data asynchronously to global storage), before proceeding with the computation. Clearly, for an application code to checkpoint locally to volatile memory, it must (a) checkpoint amounts of memory small enough to fit, and (b) be explicitly coded to forward its data to the on-node service responsible for handling checkpoints. Suppose a job runs on N nodes. Suppose also that each node must checkpoint a certain amount M of application memory (something like 20% of available node memory is typical). Assuming fixed write bandwidth Bw to its storage, this results in a checkpointing time τc = M/Bw . Note that this assumes that the entire write bandwidth is available to the node, that is, there is no network contention with other nodes writing their own checkpoints. Note also the significant approximation that all applications checkpoint the same amount of memory, M . In the case of node-local checkpointing, the data need only be written out in a time τchk = τc to the local storage, and the computation can resume immediately. In this case, the usage consumed by the checkpoint is (2.1)

uchk = N τchk (node-local storage).

The value of uF is a realization of a truncated exponential random variable YF ∼ Exp(R0 ) with distribution density (2.3) ( R0 exp(−R0 uF ) 0 ≤ uF ≤ uc + uchk πYF (uF ) = 1−exp(−R0 (uc +uchk )) 0 otherwise. The expected cost from such a failure is uF = EYF (YF ), given by Z uc +uchk uF = duF uF × πYF (uF ) 0 R0 (uc + uchk ) −1 (2.4) = R0 1− exp (R0 (uc + uchk )) − 1 2.3 Restart Cost Restarting a job involves reading a checkpoint file in to all the nodes in a job, and some fixed overhead. Assuming read bandwidth Br from storage is related to write bandwidth Bw to storage by Bw = αBr , the cost of a successful restart is (2.5)

ur = αuchk + τ0 N,

where τ0 is a fixed setup time. A restart may not be successful, however, since a node failure may occur while the restart is in process. The probability of a successful restart is ps = exp (−R0 ur ), and the probability of a failed restart is 1 − ps . Given that a restart has failed, the probability density that the failure resulted in a usage loss uF,r is analogous to Equation (2.3): ( R exp(−R u ) 0 0 F,r 0 ≤ uF,r ≤ ur 1−exp(−R0 ur ) πR (uF,r ) = . 0 otherwise

By contrast, in the case of global checkpointing, all nodes must ship their data to the storage device, contending for the available bandwidth to the device, Analogously to Equation (2.4), the expected loss due to before the computation may resume. The write time to a single restart failure is storage is thus τchk = N τc , where τc is the time required R0 ur −1 u = R 1 − . F,r 0 for one node to write out its checkpoint data. The usage exp (R0 ur ) − 1 consumed by the checkpoint is The random variable NF representing the number of (2.2) uchk = N × N τc = N 2 τc (shared storage). restart failures prior to a successful restart is govThe scaling with job size N is significantly different erned by the geometric distribution, Prob(NF ) = (1 − N in the two cases: O(N ) in the case of node-local check- ps ) F ps . By a standard result [1, Chapter 6], the ex2 pointing, versus O(N ) in the case of global checkpoint- pected value of NF is ing. Large jobs have a substantially larger checkpoint1 − ps = exp (R0 ur ) − 1. E(NF ) = ing usage burden in the shared storage case than they ps would in the node-local storage case. We thus obtain for the expected restart cost 2.2 Job Failure Cost A job that checkpoints each time the application computes for a usage interval uR = ur + E(NF ) × uF,r uc runs for a usage uc + uchk between checkpoints. A (2.6) = R0 −1 [exp (R0 ur ) − 1] . job that fails costs a usage uF equal to the elapsed usage since the last checkpoint, so 0 ≤ uF ≤ uc + uchk . In the limit R0 ur ≪ 1, this becomes uR = ur . Copyright © 2026 Copyright retained by principal author’s organization

Figure 3.1: Optimal checkpointing. The figures show the relation between the dimensionless quantities zc = R0 uc and zchk = R0 uchk . The left panel illustrates graphically the solution of Equation (3.3) for different values of zchk , indicated by the star markers. The dotted lines show the right-hand side of Equation (3.3), whereas the solid line shows the left-hand side of the equation. The right panel shows the dependence of zc on zchk , with the star markers indicating the corresponding solutions from the left panel. 3 Optimum Checkpoint Cadence Consider a project that is expected to require a total usage U to complete its scientific objectives. This will in general require many restarts to complete. The number of successful checkpoints required to complete the computation is U NS = . uc In order to complete NS successful checkpoints, the project will experience NF failures. This is a random variable governed by the negative binomial distribution, NF ∼ NB(pS , NS ), with probability mass function (3.1)

P (NF ) =

(NF + NS − 1)! NS N pS (1 − pS ) F , NF ! (NS − 1)!

where (3.2)

pS = exp (−R0 (uc + uchk ))

is the probability that the computation successfully completes a usage uc + uchk . Equation (3.1) can be understood simply, as the binomial probability of obtaining NF failures and NS − 1 successes in the first NF + NS − 1 attempts, multiplied by the probability of a success on the (NS + NF )-th attempt. By a standard result [1, Chapter 7], the expected number of failures is NS (1 − pS ) pS = NS [exp (R0 (uc + uchk )) − 1] .

N F = E(NF ) =

Each success comes at the cost of a checkpoint write. The total expected cost is thus L = N F (uF + uR ) + NS uchk = U uc −1 (exp (R0 (uc + uchk )) − 1) R0 −1 + uR − 1 . The optimal uc is obtained by minimizing this cost with respect to uc : dL duc = U R0 −1 + uR uc −2

× [1 − (1 − R0 uc ) exp (R0 (uc + uchk ))] , from which we obtain the relation (3.3)

1 − R0 uc = exp (−R0 (uc + uchk )) .

This formula is analogous to Equation (21) of [3], with the difference that costs here are measured in usage (node-hours) rather than in time. We may solve Equation (3.3) for uc numerically by a 1-dimensional root-find. Setting zc ≡ R0 uc , zchk ≡ R0 uchk , the solution is illustrated in the two panels of Figure 3.1. Once zc is determined as illustrated in the Figure, we obviously have uc = zc /R0 . There are two instructive limits: When R0 → 0, we have zc → 0 and zchk → 0, and Equation (3.3) is approximately

Each failure results in an expected cost that is the so that sum of the expected job failure cost and the restart cost.

1 − zc ≈ 1 − zc − zchk + zc ≈

√

1 2 (zc + zchk ) , 2

2zchk − zchk ≈

√

2zchk ,

to leading order in zchk . Hence we have r 2uchk (3.4) uc ≈ R0

the checkpoint cost, failed jobs incur failure costs, amounting in expectation to NJ × P (Failure) × uF , as well as restart costs, amounting in expectation to NJ × P (Failure) × uR . The sum of these costs is the computational loss due to machine instability to jobs of which is the well-known first-order result of [13], transtype (N, T ). Dividing this cost by NJ (an uninteresting lated from time to usage. The checkpoint cadence thus normalization) gives the cost per job. gets larger as the failure rate decreases, as accords with intuition. Another noteworthy property of this limiting 4.1 Expected Per-Job Checkpointing Loss case is that if checkpointing is performed to a central A successful job of type (N, T ) will comprise Nchk checkstorage server, then according to Equation (2.2) we have points, including the final, non-prophylactic checkpoint, 2 uchk ∝ N , and uc ∝ N . Hence the checkpointing time where cadence tchk = uchk /N tends to a constant independent of N in this limit. NT In the opposite limit R0 → ∞, it is clear that . (4.1) Nchk = zchk → ∞, zc → 1. We may rewrite Equation (3.3) uc + uchk as (1 − zc ) = exp (1 − zc ) × exp (−1 − zchk ) , The notation ⌈x⌉ denotes the smallest integer larger and expanding the first exponential we get than x, i.e. the “round up” function. The probability that an individual checkpoint is zc ≈ 1 − exp (−1 − zchk ) , successfully written is pS , given in Equation (3.2). Let us define the random variable K, representing the which is to say number of successful checkpoints prior to failure. K −1 −1 takes values k = 0, . . . , Nchk ,where K = Nchk is the uc ≈ R0 − R0 exp (−1 − R0 uchk ) . event “Success” and K < Nchk is the event “Failure”. Thus, as the failure rate increases, the optimal check- We have point time approaches R0 −1 , the mean usage between failures. chk P (Success) = pN S 4 Per-Job Losses We determined the optimal chk P (Failure) = 1 − pN . S checkpoint cadence uc by considering usage from the point of view of a single project, according to a model in which the project simply restarts every failed job until K has a truncated geometric distribution, with probait reaches its required computational usage. On real- bility mass function world supercomputers, applications compute in discrete ( units of jobs, each confined to some number N of pkS (1 − pS ) k < Nchk nodes and scheduled for some time T (typically limited P (K = k) = chk pN k = Nchk . S to 12 or 24 hours). We will refer to such a job as being of “type” (N, T ). Within that scheduled usage, PNchk the application checkpoints at its optimal rate for P (K = k) = 1 It is straightforward to verify that k=0 prophylactic purposes, i.e., to abate loss. If the job using the formula for summation of a geometric series. succeeds, it writes the final checkpoint before exiting, The loss to checkpointing for a single job of type so that it may restart later from that stage of the (N, T ) with K = k is computation. Each of the prophylactic checkpoints takes up a usage uchk which counts as a loss, since ( on a fault-free system they would be unnecessary. The kuchk k < Nchk Lchk (N, T, k) = final checkpoint would be required even on a fault-free (Nchk − 1)uchk k = Nchk , system, and therefore does not count as a loss. Suppose that a project requires NJ jobs of type (N, T ) to reach its computational objective. All jobs since we don’t count a checkpoint written at K = Nchk incur checkpointing costs irrespective of whether they as a loss. The per-checkpoint usage uchk is given by succeed or fail. If the failure probability of a job Equation (2.1) or (2.2), depending on the checkpointing is P (Failure), then we expect NJ × P (Failure) failed strategy (node-local or global). The expected per-job loss to checkpointing is therejobs, each of which will incur usage costs: besides Copyright © 2026 Copyright retained by principal author’s organization

fore

5 Machine-Level Loss A supercomputer will run a mix of different jobs of different types during a X given period. This could be described by a schedule Lchk (N, T ) = uchk kP (K = k) (Nl , Tl ), l = 1, . . . , NSched of NSched jobs, with the k=0 # l-th such jobs described by the type (Nl , Tl ). This + (Nchk − 1)P (K = Nchk ) formulation might be appropriate if one is estimating loss for a production machine based on past usage " Nchk data, or for a new machine based on scaled-up jobs X−1 = uchk (1 − pS ) kpkS from an existing production machine. Alternatively, k=0 a supercomputer facility may use its annual allocation # programs to get an idea of a distribution ρ(N, T ) chk + (Nchk − 1)pN (4.2) . S representing the number of jobs expected to run in a bin of width (∆N, ∆T ) centered on (N, T ) on a production The sum in this expression may be evaluated by machine during (say) a fiscal year. Either of these recognizing that it is proportional to the derivative formulations may be used to estimate expected loss. with to pS P of the sum of a geometric series, Here we adopt the schedule approach. Replacing a P respect d k k schedule with a binned distribution is straightforward, k (pS ) . We find k k(pS ) = pS dpS and we will not belabor the required changes in formulas chk pS − pN S here. (4.3) . Lchk (N, T ) = uchk 1 − pS A schedule (Nl , Tl ), l = 1, . . . , NSched represents a 4.2 Total Expected Per-Job Loss The per- total expected usage job loss to failure is NX Sched chk (4.4) LF (N, T ) = P (Failure) × uF = 1 − pN uF , U T ot = U (Nl , Tl ). S chk −1

l=1

where uF is given by Equation (2.4). The per-job loss to restart is (4.5) LR (N, T ) = P (Failure) × uR =

1 − pSNchk

uR ,

where uR is given by Equation (2.6). The total expected per-job loss is L(N, T ) = Lchk (N, T ) + LF (N, T ) + LR (N, T ).

The expected loss during the course of this schedule

NX Sched

L(Nl , Tl )

l=1

NX Sched

Lchk (Nl , Tl ) + LF (Nl , Tl ) + LR (Nl , Tl ) .

4.3 Expected Per-Job Usage In the compul=1 tation of machine efficiency, we will also require the expected usage of a job. This is not the same as its The remaining usage, U T ot − L is actual scientific scheduled usage, because of the possibility of job fail- computation. The efficiency e of the machine over the ure. course of this schedule is The usage of a job of type (N, T ) with K = k is ( U T ot − L L e≡ =1− . k × (uc + uchk ) + uF,k+1 k < Nchk U T ot U T ot U (N, T, k) = , Nchk (uc + uchk ) k = Nchk This is the key parameter to be recovered by this where uF,k+1 is the random failure loss in the (k + 1)-th exercise. It tells us the degree to which allocations must usage interval. We already know that E (uF,k+1 ) = uF , be “stretched” in order to accomplish their required given by Equation (2.4). We take the expectation of usage: If a project requires usage U node-hours to U (N, T, k): accomplish their science, the required allocation UAlloc Nchk satisfies U = eUAlloc , so that X−1 U (N, T ) = P (K = k) × k × (uc + uchk ) U k=0 (5.1) UAlloc = . e +P (Failure) uF +P (Success) Nchk (uc + uchk ) (4.6)

chk pS (1 − pN ) S chk )uF . = (uc + uchk ) + (1 − pN S 1 − pS

A perfect machine has e = 1, and allocates UAlloc = U . As the efficiency decreases, the required allocation increases. If over a period of time (e.g. a fiscal year) Copyright © 2026 Copyright retained by principal author’s organization

the maximum available usage is Umax , then the usage available for scientific computation is eUmax . The specific losses that contribute to this efficiency are also of interest. We define PNSched Lchk (Nl , Tl ) (5.2) , lchk ≡ l=1 U T ot PNSched LF (Nl , Tl ) (5.3) lF ≡ l=1 , U T ot PNSched LR (Nl , Tl ) (5.4) lR ≡ l=1 . U T ot Then we obviously have e = 1 − lchk − lF − lR . 6 Implementation We have implemented the above analysis as a small collection of Python functions, which are invoked by a Jupyter notebook to produces the results that we present below. The code is in a publicly available GitHub repository at https://github.com/ CarloGraziani/HPC_Computational_Efficiency. The repository also makes available the scheduling data from the Frontier supercomputer, described in §7, which we used to create a schedule (Nl , Tl ), l = 1, . . . , NSched as described in §5. The code calculates an optimal checkpointing usage cadence for each individual job, and calculates the resulting overall losses and machine efficiency that result from the assumption that all jobs are checkpointed at that rate. One subtlety that we noticed in the process of analyzing the data is that the job schedule distribution is strongly-clustered in job sizes and job durations, and this fact interacts with the discrete truncation of the per-job checkpoint number Nchk given in Equation (4.1) to produce slight non-smooth artifacts in the efficiency and loss curves shown in Figures 7.2, 7.4, and 7.5. The code offers an option to compute Nchk as a continuous function (by refraining from rounding up), which is possible because the loss formulas in §4 work irrespective of whether or not Nchk is integer-valued. Triggering this option removes the artifacts. We do not avail ourselves of that choice below, preferring instead to explain their origin while keeping the (more correct) formulation in terms of an integer-valued Nchk .

7.1 Computational Efficiency of Frontier Frontier is an exascale supercomputer maintained at the Oak Ridge Leadership Computing Facility (OLCF). Frontier is currently Number 2 on November 2024’s Top500 list with a High-Performance Linpack (HPL) score of 1.353 EFlop/s. The system is comprised of 9,408 HPE Cray EX235a nodes, each with one 64core AMD EPYC 7A53 CPU and four AMD MI250X GPUs. Each MI250X contains 2 Graphics Compute Dies (GCDs), each of which appears as a single logical device for most applications. Each compute node has 512 GB of DDR4 memory and 512 GB of highbandwidth memory (HBM2E), 64 GB per GCD. The CPU is connected to the GPUs via AMD’s Infinity Fabric which delivers a bandwidth of 36+36 GB/s. All GCDs on a Frontier node are interconnected via Infinity Fabric delivering up to 50+50 GB/s for GCDs across GPUs, and up to 200+200 GB/s for GCDs on the same GPU. Compute nodes on Frontier are interconnected via HPE’s Slingshot 11 interconnect. 7.2

System Parameters

7.2.1 Application Failure Rate Node failures on Frontier are detected and logged, and are found to occur almost exclusively in conjunction with the failure of applications that are utilizing them at the time of failure—that is, nodes unstressed by an application rarely fail spontaneously. We therefore estimate the parameter R0 based on the node failure rate. In this way −1 we obtain the estimate R0 = 2.0×10−5 (node-hours) . The resulting failure rate is necessarily a slight underestimate of the true R0 , because a job may occasionally also fail due to some system issue that does not cause a node failure. We neglect this error in this study.

7.2.2 Storage Bandwidth Frontier utilizes a central storage server mounting a Lustre filesystem (dubbed Orion) composed of a Flash-based performance tier of 5,400 nonvolatile memory express (NVMe) devices providing 11.5 petabytes (PB) of capacity at peak read-write speeds of 10 TB/s. In practice, most jobs see equal read and write bandwidths of roughly Bw = Br = 6 TB s−1 . Since read and write bandwidths are equal, we set the parameter α in Equation (2.5) to α = 1. We assume an average checkpoint size of 20% of on-node 7 Results From Frontier Usage Data We memory, amounting to M = 200 GB. This leads to a now describe the application of this framework for per-node checkpoint time τ = 9.26 × 10−6 hrs. In pracc analyzing machine efficiency to data from the Frontier tice, of course, different applications may checkpoint difsupercomputer. After a description of the machine ferent amounts of memory, so that the assumption of a and a discussion of the data and model parameters, shared value of M is a significant approximation. we provide analyses of computational efficiency, of the effect of suboptimal checkpointing, and of the influence 7.2.3 Fixed Setup Time On Frontier, the on efficiency of job size and duration, of application setup time τ0 in Equation (2.5) is 2s. This is a negfailure rate, and of bandwidth to storage. ligible term, which we include in our analysis for the Copyright © 2026 Copyright retained by principal author’s organization

Figure 7.1: Distribution of job characteristics from Frontier used in this study. sake of completeness.

Figure 7.2: Effect of suboptimal checkpoint interval selection on efficiency. The x-axis shows the common 7.3 Job Schedule We use a database of 331,640 fraction by which all checkpoint intervals are shifted jobs from one year (CY 2024) of operations on Frontier. from their respective optimal value. The job selection criteria used for the analysis required that the jobs examined be from one of the facility’s two 7.5 Importance of Proper Checkpointing major allocation programs, the Innovative and Novel The computational efficiency given above corresponds Computational Impact on Theory and Experiment (IN- to a situation in which all applications checkpoint at CITE) program and the ASCR Leadership Computing their own optimal rate. We now investigate the effect Challenge (ALCC) program. In addition, jobs that were of suboptimal checkpointing on efficiency. run on a single node were excluded, since such jobs are In the first place, what happens if applications on very typically interactive or small debugging runs. The Frontier do not do any prophylactic checkpointing at job characteristics are shown in Figure 7.1. all? To answer this question, we repeat the analysis Each job is characterized by its size Nl in nodes, while setting the checkpointing interval uchk of each job its requested time, and its actual runtime. Many jobs equal to its total requested usage. The result is that ran for an actual runtime that was substantially less the computational efficiency of Frontier would then be than its requested time: the ratio of mean runtime to e = 0.914, a drop of 4.3% from its optimal efficiency. For mean requested time is r = 0.46. Since most jobs ter- reference, with this efficiency projects would need to be minate normally, this ratio is indicative of conservative overprovisioned with node-hours by a factor of 9.4% in scheduling by users rather than of a high failure rate. order to achieve their requested targets on node-hours This presents a problem for our analysis, because the used for science computations. requested times Tl are not really representative of the In Figure 7.2 we show the effect of shifting checktime required for successful job completion. We account pointing cadence of all jobs by a common factor. The for this fact by multiplying all requested times by the left panel of the figure shows efficiency as a function of ratio r, so as to adjust the requested times to be closer checkpointing cadence, while the right panel shows the to what the applications actually required to complete dependence on checkpointing cadence of the individual a job. loss terms. The left panel of the figure shows the points Note that we do not use (or even know) the termi- at which the efficiency drops by 5% and by 10% from nation status of any of these jobs. We only use this data peak. If the checkpoint interval is displaced from opfor the purpose of establishing a plausible distribution timality by a factor of 5 or so in either direction, the of job sizes and requested durations that is representa- result is a loss of efficiency of 5%, and if the cadence is tive of a realistic computational load due to scientific about a factor of 10 too frequent, the loss can be as large applications running on Frontier. as 10%. Despite appearances, these kinds of inefficien7.4 Computational Efficiency of Frontier cies are not implausible mistakes to make. Consider, Based on the parameters given in §7.2 and the job for example an N = 2000 node job on Frontier, which, schedule described in §7.3, we find that Frontier has according to Equation (2.2) has a checkpointing usage a computational efficiency e = 0.957, due to costs and uchk = 37.04 node-hours. Since zchk = R0 uchk ≪ 1, we losses lchk = 0.012, lF = 0.028, lR = 0.003. The may use the formula in Equation (3.4) to compute the high efficiency on Frontier suggests that if users on checkpoint usage cadence uc = 1925 node-hours, and the machine in fact checkpoint optimally, the OLCF hence a time cadence uc /N ≈ 1 hr. Hence, checkpointneed only overprovision projects with node-hours by a ing at a time rate of every 12 minutes, or every 5 hours, fractional amount e−1 − 1 = 4.5% compared to their would lead to an efficiency degradation of 5% for this type of job. approved allocation requests. Copyright © 2026 Copyright retained by principal author’s organization

Figure 7.4: Efficiency as a function of failure rate.

Figure 7.3: Efficiency and relative losses as a function of requested duration. 7.6 Efficiency Versus Job Size and Duration Figure 7.3 shows the efficiency and relative losses expected for jobs of different node sizes and requested durations. Efficiency is shown in the top-left panel, while the loss terms due to faults, checkpointing, and startup are shown in the remaining panels. Quite generally, we see that the efficiency declines with increasing requested duration, saturating at large durations due to the saturation of checkpointing cost. The latter occurs because as duration becomes large so does Nchk , and it is clear from Equations (4.3), (4.6), and (5.2) that in this limit the checkpointing loss saturates. Intuitively, this is explainable in terms of the increasing certainty of a failure with increasing duration, which leads to a finite number of expected checkpoint writes during the course of a finite expected usage. The losses lF and lR due respectively to failures and restarts are constants as a function of requested job duration. Algebraically this is because, according to Equations (4.4), (4.5), and (4.6), the quantities LF (N, T ), LR (N, T ), and U (N, T ) all depend on duration T through the common multiplicative factor (1 − pS Nchk ), which therefore cancels in the computation of lF and lR according to Equations (5.3–5.4).

indicated in the figure, the efficiency drops from 95.7% when R0 = 2 × 10−5 to about 90.9% when R0 = 10−4 , and to 88.6% when R0 = 2 × 10−4 . These latter rates are shown for the purpose of illustrating how a new machine’s efficiency might be expected to improve as it progresses in stability from the early stages of deployment to mature operations, during which time failure modes are identified and abated. Note also that these rates are quite sensitive to the bandwidth available for transferring data to storage, as we discuss next. 7.8 Efficiency Versus Bandwidth to Storage Figure 7.5 displays the dependence of efficiency on bandwidth to storage, assuming a shared storage device. It is clear from the figure that bandwidth is a critically important consideration when designing a machine that is to support resilient computation in the presence of frequent machine faults, because there is a striking drop in efficiency with decreasing bandwidth. This degradation of efficiency is even more striking for the higher failure rates displayed in the figure. This is largely a consequence of the O(N 2 ) scaling of uchk that occurs for checkpointing to shared storage in Equation (2.2), which severely penalizes checkpointing for large jobs. Note

Figure 7.5: Efficiency as a function of bandwidth to 7.7 Efficiency Versus Failure Rate Figure 7.4 storage. displays the decline of efficiency with failure rate. As Copyright © 2026 Copyright retained by principal author’s organization

that we can partly understand the effect of applicationspecific variations in the checkpointable memory M by treating it as equivalent to application-specific bandwidth to storage with fixed, shared M . Figure 7.5 may be viewed as illustrating the fact that efficiency can be very sensitive to amount of checkpointable memory, especially at higher failure rates. 8 Discussion We have presented a framework for evaluating the computational efficiency of a scientific HPC machine in the presence of frequent node faults. The chief novelty here is the transposition of some previous results on checkpoint cadences [3, 13] from the time domain to the usage domain, as this is more relevant to the situation facing modern scientific HPC, because of the heterogeneity in job sizes and durations that is characteristic of such platforms. We can anticipate that future machines that move beyond the current Exascale class will experience even higher failure rates per node-hour than the current generation, due to the growing complexities of node design and of system software, and to growing transistor counts on nodes. This observation leads to some considerations that seem relevant to the design of such machines, and of their software stacks. In the first place, it seems likely that checkpointing will remain the principal application failure abatement strategy in future. While there exist checkpoint-free resilience strategies [2], they principally apply to single solver algorithms, and would be challenging to apply to the kinds of multi-physics, multi-solver codes that typify scientific HPC. This being the case, it seems clear to us that it is imperative that future machines should be designed so as to avoid the O(N 2 ) scaling of usage consumed by checkpoint reads and writes described by Equation (2.2), because that scaling is likely to become increasingly toxic with increasing node counts and fault rates. Node-local storage seems like an obvious design strategy to avoid this scaling. Alternatively, on-node memory and hardware specialized for the purpose of receiving checkpointable data and transmitting it to central storage while the computation proceeds on CPUs and GPUs could be a viable choice. Future machines could also provide explicit system support for checkpointing, making the process more standardized and requiring less specialized applicationlevel infrastructure development by science application developers. Some thought could be given to resilient strategies for system-level dynamic deletion of older checkpoints to make space for new ones, a consideration of increasing importance as checkpointable application memory seems likely to grow with machine size. In addition, better facilities for resilient computing could be supplied, so that an application that suffers a

node failure does not necessarily exit immediately and go back to the queue, but rather can restart gracefully from the last checkpoint and complete the computations of its planned job duration. While this sort of resilience is currently easy to obtain for some applications, such as AI loads that merely lose an inessential minibatch’s worth of gradient components when a node is lost [5], other applications require more resilience engineering at present. For example, a computational fluid dynamics simulation or an Earth system model cannot survive the loss of the part of their computational domain lost to a node failure without rolling back to the previous checkpoint. Job queuing software could be designed to recognize application failures due to node faults, and to provision replacement nodes and facilitate replacement MPI communicators (for example) which allow the application to recover gracefully from the last checkpoint. Replacement nodes could be pre-allocated to job partitions, based on the expected number of node faults during each job. In this work, we have used a single failure rate R0 as characteristic of all applications running on a machine. However, it seems very likely that different types of applications fail at different rates, depending on the manner in which they stress the node hardware and/or trigger problems due to system software. It would benefit individual application developers to establish estimates of their applications’ individual failure rates, because this could lead to more refined estimates of their optimal checkpointing cadence, with measurable benefits for their computational efficiency. Incorporation of application-specific checkpointable memory M into the formalism for computing optimal checkpoint intervals would also bring substantial benefits, because this is analogous to varying the bandwidth to storage for fixed M , and as Figure 7.5 illustrates, efficiency can be a very sensitive function of bandwidth, especially at higher failure rates. From the perspective of HPC facilities, it seems extremely desirable that node fault rates and application failure rates should be measured and monitored over time. This is not an easy task, because the relevant log files accumulate vast amounts of data, most of which is irrelevant to this task, and a good deal of it can wind up in different data silos: kernel messages, application standard error files, and scheduler logs are an example of this. Moreover, some of the highest-quality information concerning application faults is in application output files, which application teams do not necessarily share by default with the facility. And, unfortunately, job exit status can be an unreliable indicator of job success or failure, depending on the design of the job submission script. It would be well-worth designing instrumenta-

tion for capturing and identifying such faults and failures with good specificity and sensitivity, and building it into the machine from the outset, rather than relying on clever datamining of logfiles. Good capture of failure rates, possibly broken out by application type, would benefit HPC facilities by allowing them to accurately overprovision node-hours to application teams by factors suggested by the relevant computational efficiencies. This overprovisioning, described by Equation (5.1) would simplify the transactions between facilities and application teams, since the latter would not require compensating nodehours for each failed job, having been, in a sense, “pre-compensated” for machine faults by the expected amount. For this to work well, HPC facilities would certainly require accurate metering of fault and failure rates. One important gap in this work is that we have only considered node-local application failures here. These appear to constitute the majority of failures, but do not make up the whole story. Some failures are collective, such as slowdowns or hangs due to network congestion. In addition, currently an application that hangs rather than exiting cleanly consumes usage to no purpose until it is explicitly killed. This is an effect that could be modeled, but of course a better solution would be platform-level software (for example in the scheduler) to recognize hangs and stop the offending jobs.

the 20th international symposium on High performance distributed computing, 2011, pp. 73–84. [3] J. T. Daly, A higher order estimate of the optimum checkpoint interval for restart dumps, Future generation computer systems, 22 (2006), pp. 303– 312. [4] S. Di, H. Guo, E. Pershey, M. Snir, and F. Cappello, Characterizing and understanding hpc job failures over the 2k-day life of ibm bluegene/q system, in 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), IEEE, 2019, pp. 473–484. [5] A. Erben and E. Erdil, Hardware failures won’t limit ai scaling, 2024, https://epoch.ai/blog/ hardware-failures-wont-limit-ai-scaling. Accessed: 2025-03-21. [6] K. Heitmann, T. Uram, N. Frontiere, S. Habib, A. Pope, S. Rizzi, and J. Insley, The new worlds simulations: Large-scale simulations across three cosmologies, 2024, https://arxiv. org/abs/2406.07276. [7] A. Kokolis, M. Kuchnik, J. Hoffman, A. Kumar, et al., Revisiting reliability in large-scale machine learning research clusters, in 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), IEEE, 2025, pp. 1259–1274.

Acknowledgments We gratefully acknowledge the use of the resources of the Argonne Leadership Computing Facility at Argonne National Labratory. This [8] K. Nguyen-Cong, J. T. Willman, J. M. Gonwork was supported by the Office of Advanced Scientific zalez, A. S. Williams, et al., Extreme metaComputing Research, Office of Science, U.S. Departstability of diamond and its transformation to the ment of Energy, under Contract DE-AC02-06CH11357. bc8 post-diamond phase of carbon, The Journal of This research used resources of the Oak Ridge LeaderPhysical Chemistry Letters, 15 (2024), pp. 1152– ship Computing Facility at the Oak Ridge National Lab1160. oratory, which is supported by the Office of Science of [9] L. Svobodova, Resilient distributed computthe U.S. Department of Energy under Contract No. DEing, IEEE Transactions on Software Engineering, AC05-00OR22725. We are grateful to Nwamaka Okafor (1984), pp. 257–268. (ANL/ALCF), Shilpika (ANL/ALCF), and Scott Simmerman (ORNL/OLCF) for valuable discussions about [10] A. Taherin, T. Patel, G. Georgakoudis, log files and failure detection and to Mike Papka, VenkaI. Laguna, and D. Tiwari, Examining failures tram Vishwanath, Eric Pershey, Susan Coghlan, and and repairs on supercomputers with multi-gpu comMichael Zhang (ANL/ALCF) for discussion of issues pute nodes, in 2021 51st Annual IEEE/IFIP Interraised by application failures for HPC facilities. national Conference on Dependable Systems and Networks (DSN), IEEE, 2021, pp. 305–313. References [1] N. Balakrishnan and V. Nevzorov, A Primer [11] M. Taylor, P. M. Caldwell, L. Bertagna, C. Clevenger, et al., The simple cloudon Statistical Distributions, Wiley, 2004, https: resolving e3sm atmosphere model running on the //books.google.com/books?id=JIfk5kBdLGIC. frontier exascale system, in Proceedings of the In[2] Z. Chen, Algorithm-based recovery for iterative ternational Conference for High Performance Commethods without checkpointing, in Proceedings of puting, Networking, Storage and Analysis, SC Copyright © 2026 Copyright retained by principal author’s organization

’23, New York, NY, USA, 2023, Association for Computing Machinery, https://doi.org/10.1145/ 3581784.3627044. [12] A. Verma, J. E. Huffman, A. Rodriguez, M. Conery, et al., Diversity and scale: Genetic architecture of 2068 traits in the va million veteran program, Science, 385 (2024), p. eadj1182. [13] J. W. Young, A first order approximation to the optimum checkpoint interval, Communications of the ACM, 17 (1974), pp. 530–531. [14] X. Zhao, Z. Jiang, and J. Ma, A survey of deep anomaly detection for system logs, in 2022 International Joint Conference on Neural Networks (IJCNN), IEEE, 2022, pp. 1–8. [15] Z. Zheng, L. Yu, W. Tang, Z. Lan, R. Gupta, N. Desai, S. Coghlan, and D. Buettner, Coanalysis of ras log and job log on blue gene/p, in 2011 IEEE international parallel & distributed processing symposium, IEEE, 2011, pp. 840–851.

Application Failures and Machine Computational Efficiency

Analyze & explore related research in Conceptio

Related documents