TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training Shujie Han1 , Feng Jiang1 , Patrick P. C. Lee2 , Xiao Zhang1 , Zhijie Huang1 , Nannan Zhao1 , Xiaonan Zhao1 , and Lichen Pan3 1 Northwestern Polytechnical University 2 The Chinese University of Hong Kong 3 National University of Defense Technology
arXiv:2605.17821v1 [cs.DC] 18 May 2026
Abstract
globally consistent state after a failure. Recent work has extensively optimized checkpointing along three axes: (i) checkpointing overhead, which minimizes I/O blocking overhead to maximize training throughput via asynchronous persistence [12, 20, 23]; (ii) checkpoint size reduction, which compresses states [39] or exploits redundancy elimination [18] to enable higher checkpointing frequencies; and (iii) recovery efficiency, which leverages host CPU memory to rapidly restore training states and minimize downtime [35]. However, existing checkpointing systems apply an all-ornothing persistence strategy that largely ignores the heterogeneity of failures. In-memory systems (e.g., Gemini [35]) prioritize fast recovery by saving snapshots in CPU memory, but are vulnerable to node crashes. Conversely, systems that write to local or remote persistent storage (e.g., CheckFreq [23]) offer stronger fault tolerance but suffer from severe I/O bottlenecks during recovery. Even advanced techniques that reduce checkpoint sizes via incremental saving (e.g., [18, 39]) are still bottlenecked either by the computational cost of sequential state replay [39] or the overhead of redundant state detection [18] during checkpoint restoration. With the tight binding of a monolithic storage backend to diverse failure modes, these systems fail to strike an optimal balance between low-overhead persistence and fast recovery. To address this gap, an intuitive approach is to decouple checkpoint persistence across a hierarchy of storage layers. However, realizing a multi-tiered architecture introduces three critical system-level challenges spanning the entire checkpoint lifecycle:
Large Language Model (LLM) training is frequently interrupted by a heterogeneous spectrum of failures, from common GPU crashes to catastrophic cluster-wide outages. Existing checkpointing systems rely on monolithic, single-tier storage backend, forcing a trade-off between state-saving overhead and recovery speed. We propose TierCheck, a cluster-aware tiered checkpointing system that aligns storage placement with failure heterogeneity. TierCheck adopts a three-tier design that maintains lightweight differential checkpoints in local and peer memory for fast localized recovery, while asynchronously migrating heavyweight base checkpoints to remote persistent storage. It also ensures strict global consistency across tiers without stalling training, and achieves fast cluster-aware checkpoint restoration during recovery. Evaluations on models up to 40 billion parameters show that TierCheck achieves low training overhead, reduces end-to-end checkpointing time to under 10 s, and supports high-frequency checkpointing, ultimately striking an optimal balance between low-overhead persistence and fast recovery.
1
Introduction
In Large Language Model (LLM) training, failures are both prevalent and diverse. As model sizes scale to trillions of parameters [1, 10], training requires thousands of GPUs running continuously for months [13, 15], turning unexpected failures into routine daily occurrences. For instance, during a 54-day period of Llama-3 training at Meta [10], 419 of 466 job interruptions are unexpected failures at GPU and node levels, of which GPU failures alone account for 58.7%. Modern data centers also experience rack failures where multiple nodes within the same rack are unavailable simultaneously, albeit rarely [8, 40]. This skewed statistical distribution reveals the heterogeneity of failures in large-scale LLM training. To mitigate the loss of training progress, periodic checkpointing remains the standard fault-tolerance mechanism [7]. In LLM training, massive training states (e.g., model weights and optimizer states) are intricately partitioned across thousands of GPUs via multi-dimensional parallelisms (e.g., Tensor Parallelism [32], Pipeline Parallelism [11], and ZeRO Data Parallelism [26]). By regularly saving distributed training states, a training job can resume from the most recent
• Checkpoint saving: Offloading massive training states from GPUs competes for system resources. The challenge is to efficiently capture and route these states to appropriate tiers without stalling ongoing training. • Checkpoint retrieval: Because failures vary in frequency and severity, the recovery path must be optimized for the most common incidents (e.g., GPU and node failures) while still guaranteeing a robust fallback for rare, catastrophic events (e.g., rack-level power outages). • Checkpoint reclamation: Extreme bandwidth disparities between tiers inevitably cause slower storage to lag behind fast volatile memory. The system must determine precisely when it is safe to garbage-collect volatile checkpoints across the cluster. 1
We design TierCheck, a cluster-aware tiered checkpointing system aiming to tolerate heterogeneous failures in LLM training. TierCheck distributes checkpoint data across three tiers: Tier-1 (local volatile memory), which retains local checkpoints for fast recovery, Tier-2 (neighbor volatile memory), which asynchronously replicates checkpoints to a peer node for tolerating single-node failures, and Tier-3 (remote persistent storage), which durably migrates checkpoints to a remote backend for tolerating rack failures. To overcome the challenges of multi-tier checkpointing, TierCheck optimizes the checkpoint lifecycle across three key phases. First, for checkpoint saving, TierCheck decouples the persistence of training states. It frequently captures and compresses incremental updates into fast volatile memory (Tier-1 and Tier-2) to prevent bursty I/O from stalling training, while periodically buffering complete states at a much lower frequency and asynchronously migrating them to remote storage in the background. Second, for checkpoint retrieval, TierCheck dynamically routes the restoration path. It prioritizes fast recovery from Tier-1 and Tier-2 to quickly resolve common incidents (e.g., software and node failures), while seamlessly falling back to remote storage (Tier-3) for catastrophic rack failures. Third, for checkpoint reclamation, TierCheck safely governs the lifecycle of historical states. It strictly purges volatile replicas only after the corresponding base checkpoint is durably committed to remote storage, guaranteeing global consistency without risking data loss. We implement TierCheck atop DeepSpeed [28], achieving seamless compatibility with 3D parallelism strategies, including Tensor Parallelism [32], Pipeline Parallelism [11], and ZeRO Data Parallelism [26]. Our implementation natively supports the entire checkpoint lifecycle, systematically orchestrating asynchronous saving, a highly optimized retrieval path, and watermark-driven reclamation. We evaluate TierCheck on a testbed of 16 NVIDIA A800 (80 GiB) GPUs provisioned from a distributed cloud platform. Experimental results demonstrate that TierCheck reduces the checkpointing overhead by 62.8-82.7% compared to state-ofthe-art systems (e.g., CheckFreq [23] and Gemini [35]). Furthermore, TierCheck achieves a 3.6-6.9× recovery speedup for software failures and 1.8-3.5× for node failures. Even under worst-case rack failures, it remains 1.3-2.5× faster than baselines, all without compromising global model convergence. We will open-source the TierCheck prototype in the final paper.
2
Background and Motivation
2.1
LLM Training
second moments) to update model weights. Training state size. Checkpointing overhead scales directly with training state size. Modern mixed-precision training creates an asymmetry between the lightweight representation used for computation and the heavier training state that must be checkpointed for exact recovery. Let Φ denote the total number of model parameters. In mixed-precision training, model weights are stored in 16-bit half-precision floating-point format (FP16), consuming 2Φ bytes. To prevent numerical underflow, the Adam optimizer additionally maintains a 32-bit full-precision floating-point (FP32) master copy of the weights alongside its first and second moment tensors, which together consume 12Φ bytes. Thus, a full training snapshot requires 14Φ bytes in total, dominating both memory and storage footprints at scale. Parallel mechanisms. To accommodate trillion-parameter LLMs, modern training frameworks combine multiple parallelism strategies. Data Parallelism (DP) replicates the model to process independent data batches concurrently, often augmented by the Zero Redundancy Optimizer (ZeRO) [26]. ZeRO progressively shards training states across DP ranks (i.e., individual GPU workers) to minimize memory footprints: ZeRO-1 shards optimizer states, ZeRO-2 adds gradient sharding, and ZeRO-3 further shards model weights. Tensor Parallelism (TP) [32] partitions individual parameter matrices across GPUs, while Pipeline Parallelism (PP) [11] assigns consecutive model layers to different nodes for concurrent micro-batch processing. 2.2
Checkpointing
Training failures. Failures in large-scale LLM training are heterogeneous in both frequency and severity. We categorize training failures into three classes: (i) Software failures are software-triggered incidents such as CUDA kernel crashes, runtime bugs, or malformed input data [41]. Since the underlying hardware remains healthy, the process can simply be restarted on the same node, preserving local host memory contents. (ii) Node failures are host- or device-level hardware crashes caused by CPU, DRAM, motherboard, attached GPU devices, or local power faults, and dominate hardwarerelated incidents in production clusters [10]. Since the affected hardware becomes unavailable, recovery requires replacing the failed component or node, and all volatile state on that hardware is lost. (iii) Rack failures, such as rack-level power outages, top-of-rack switch failures, or cooling faults, simultaneously affect multiple nodes within the same rack. They are rare but catastrophic: they wipe out entire failure domains, including all local and peer volatile memory, leaving only remote durable storage as a recovery source. Checkpointing classification. LLM training systems tolerate failures by periodically saving training states to remote persistent storage (e.g., Lustre [31], Ceph [36], Amazon S3 [5]), which serves as the ultimate recovery source. Existing checkpointing mechanisms differ along two dimensions. By
Basics. The LLM training process is an iterative optimization loop. Each training iteration comprises a forward pass that computes the loss over a data batch and a backward pass that computes the gradients. The optimizer (e.g., Adam [14]) uses these gradients along with its internal states (e.g., first and 2
Table 1. Comparison of checkpointing systems and their recovery mechanisms under heterogeneous failures. ✓ indicates efficient and exact recovery; ∼ indicates recovery with significant trade-offs (e.g., slow performance, approximation, or massive hardware redundancy); × indicates inability to recover without catastrophic fallback or failure. Systems CheckFreq [23], DataStates -LLM [20], PCcheck [33], AsymCheck [22] LowDiff [39] AdaCheck [18] Gemini [35] CheckFree [4] Swift [42] FT-HSDP [30] TierCheck (Ours)
Software failures
Node failures
Rack failures
∼ Full remote checkpoint fetch (Bounded by slow persistent storage I/O) ∼ Remote compressed fetch + Sequential differential replay ∼ Remote half-precision fetch + Decompression overhead ✓ Local node fetch ✓ Peer node fetch × Volatile replicas wiped out ∼ Pipeline averaging ∼ Pipeline averaging × Not designed for multi-node failures ✓ DP group replicas ✓ DP group replicas × Hardware redundancy overwhelmed ∼ Drop failed replica ∼ Drop failed replica ∼ Remote fetch (Extremely costly) ✓ Tier-1 Local memory ✓ Tier-2 Peer memory ✓ Tier-3 Base anchor + Differential replay
placement, checkpoints are either in-memory (i.e., local or peer volatile memory for low-latency recovery) or persistent (i.e., remote durable backends for strong failure isolation). By format, checkpoints are either full (capturing the complete state for direct, exact restoration) or differential (recording only the incremental updates from training iterations to reduce checkpoint volume). Full checkpoints enable 𝑂 (1) recovery but incur severe I/O overhead; differential checkpoints greatly reduce save costs, but recovery requires sequentially replaying an update chain from the most recent full checkpoint, incurring 𝑂 (𝐿) replay cost proportional to the chain length 𝐿. While 𝐿 is typically bounded to tens to hundreds by the base interval, replaying even a short sequence requires repeated computation and communication over the massive training state, making the cumulative recovery time highly non-trivial. Because training states are partitioned across ranks, checkpoints are materialized as per-rank shards rather than single globally visible files. In this paper, a base checkpoint denotes a single rank’s shard of a full snapshot, collectively serving as the recovery anchor (i.e., the start point for differential checkpoint replay). A differential checkpoint denotes one rank’s shard of the incremental updates. The number of iterations between consecutive base checkpoints defines the base checkpoint interval. Compressing differential checkpoints. Differential checkpoints are further compressed as raw updates are too large for high-frequency saving. One approach is sparse compression (e.g., Top-K selection), which extracts only a small subset of large-magnitude entries rather than storing the full dense update [2, 39]. While effective for large tensors with heavily concentrated updates, sparse compression incurs index metadata overhead that can outweigh its spatial benefits on smaller tensors [29]. An alternative approach is quantization-based compression [3], which maps high-precision floating-point numbers to a limited set of discrete levels (e.g., 8-bit integers (INT8)). This preserves the dense tensor layout while uniformly reducing the bit width
of each element, offering a more efficient trade-off for small parameter vectors. Existing checkpointing systems often optimize for a single placement tier, a single checkpoint format, or a single dominant failure class, as shown in Table 1. They excel for a subset of failures but underperform or fail for others. Limitations of single-tier checkpointing. We define a tier as a storage backend characterized by a specific durability model and bandwidth/latency profile (e.g., local volatile memory, peer volatile memory, or remote persistent storage). No single tier can simultaneously satisfy the requirements imposed by all three failure classes. A remote-only design provides durable recovery during catastrophic outages, but forces even the most common software and single-node failures to incur full remote-storage fetch latency. A volatileonly design enables fast, localized restoration for common failures, but offers no protection once the failure destroys the hosting failure domain, as in node or rack failures. A differential-heavy design reduces checkpoint volume, but the recovery cost grows linearly with replay length and is limited by the availability of a recent full checkpoint. This mismatch is most apparent through the lens of failure survivability: after a software failure, local memory states remain available; after a node failure, peer memory states still survive; after a rack failure, only remote durable storage can be leveraged. Since the cheapest surviving recovery source changes with failure scope, no single-tier strategy can simultaneously provide locality, durability, and low overhead. The case for tiered checkpointing. Instead of coupling all checkpoints to a single backend, checkpoint placement should be aligned with failure domains such that each tier handles the failure scenarios it can optimally serve. A tiered architecture allows (i) fast local volatile storage to absorb frequent software failures, (ii) peer memory to recover from node failures without remote I/O, and (iii) a remote durable tier to recover from rack failures. This separation of fast persistence, localized recovery, and durable fallback preserves low steady-state overhead while guaranteeing exact recovery 3
3.1
across the full failure spectrum.
2.3
Figure 1 shows the architectural overview of TierCheck. We consider a training cluster composed of multiple machines, each equipped with multiple GPUs and interconnected through InfiniBand NICs. To align storage with heterogeneous failures (§2.2), TierCheck organizes checkpoint data into three tiers. Tier-1 is local volatile memory on the same node, used for the fastest checkpoint writes and local recovery. Tier-2 is peer memory on a physically adjacent node, used to survive single-node failures. Here, a peer is defined physically rather than by logical DP/TP/PP groups: each rank (as defined in §2.1) backs up its checkpoint to the corresponding rank on an adjacent machine through a node-level ring mapping, so replicas leave the local node’s failure domain. In particular, a Tier-2 peer intentionally resides in the same rack as its Tier-1 node. This design is a deliberate, failure-frequency-aware trade-off: rack failures are exceedingly rare, whereas single-node failures are the dominant hardware-level incident class [6, 8, 40]. Intra-rack peer placement thus eliminates the latency and bandwidth costs of cross-rack network transfers, which would otherwise slow the high-frequency Tier-2 replication path, at the expense of protection against the rare rack failures. Crucially, this residual risk is fully absorbed by Tier-3, a remote persistent storage backend that serves as the durable fallback for rack failures of any scope, including rack-level events. This cluster-aware design optimizes for the common case without sacrificing correctness for the rare case. On top of these tiers, TierCheck saves training state through two independent streams rather than naive threeway replication. The first stream contains high-frequency differential checkpoints, i.e., per-rank incremental updates that remain on the fast volatile path for fast recovery. The second stream contains low-frequency base checkpoints that periodically refresh the full-checkpoint recovery anchor. These base checkpoints are migrated asynchronously toward more durable tiers. This asymmetric organization preserves low overhead on the critical training path and still maintains an exact recovery anchor. TierCheck builds on three cooperating modules:
Challenges of Tiered Checkpointing
Realizing a tiered checkpointing architecture in large-scale LLM training clusters introduces three challenges. Checkpoint saving. A tiered system must decide how to distribute checkpoint data across local memory, peer memory, and remote storage without degenerating into naive three-way replication, which amplifies storage and bandwidth overhead and stalls the training loop behind massive checkpoint I/O. This is compounded by the deep sharding of model weights, gradients, and optimizer states across TP, PP, DP, and ZeRO partitions, making the materialization of a full snapshot inherently expensive. The challenge is thus to construct tier-specific checkpoint contents of appropriate granularity and schedule their transfers asynchronously, so that each tier stores only the data it needs while keeping the critical training path short. Checkpoint retrieval. Under heterogeneous failures, the system must quickly identify and exploit the cheapest surviving recovery path rather than blindly restoring from a fixed backend. The optimal path depends on which tiers remain intact: a software failure may be served entirely by local memory, a node failure by peer memory, and a rack failure by the remote persistent tier. Moreover, the total recovery cost is determined not only by data retrieval latency but also by the amount of replay or recomputation required after loading. The challenge is to jointly minimize pull time and re-execution time while ensuring seamless, exact fallback when faster tiers are unavailable, stale, or incomplete. Checkpoint reclamation. Since remote persistence is orders of magnitude slower than fast-tier writes, recovery should retrieve a local or peer memory copy whenever possible. However, retaining checkpoints in local and peer memory indefinitely is infeasible, as these volatile resources are shared with the training process and will quickly become a bottleneck. The challenge is to determine precisely when a checkpoint becomes globally safe to reclaim, given the asynchronous and heterogeneous completion times across all tiers, and how to perform coordinated garbage collection across the cluster without creating data gaps.
3
Architectural Overview
• Checkpoint saving. This module addresses how checkpoints are created and placed across tiers. Its key techniques include: (i) adaptive gradient compression, which shrinks high-frequency differentials before they enter the I/O path; (ii) in-memory checkpoint interception, which intercepts the optimizer’s internal serialization path to extract checkpoint data as reusable in-memory payloads without reconstructing the full state; and (iii) asymmetric transmission scheduling, which applies distinct batching and chunking strategies to the two checkpoint streams to avoid bursty interference with training communication. • Checkpoint retrieval. This module decides how TierCheck resumes from the most efficient surviving tier
Design of TierCheck
TierCheck is a cluster-aware tiered checkpointing system designed for tolerating heterogeneous failures in LLM training. The design follows a simple principle: separating checkpoint contents by size, production frequency, and failure coverage, then binding each class of data to the storage tier that serves it most efficiently. We first present the overall architecture (§3.1), and then describe how TierCheck saves (§3.2), retrieves (§3.3), and reclaims (§3.4) checkpoints. 4
InfiniBand network
§3.2 Checkpoint saving Node0
G2
Node0 Bn
Un Rank Ranks 1
Gn
Node1
Node1 Rank Ranks 1
Tier-1: Local volatile memory Differential checkpoints
Differential checkpoints
Base checkpoint
Base checkpoint
1. Adaptive gradient compression
3. Asymmetric transmission scheduling
Watermark-driven global reclamation
§3.4 Checkpoint reclamation Forward
B Backward
U Update
F
G Gradient
Control flow Differential checkpoint flow Base checkpoint flow
Asynchronous transmission
Asynchronous transmission
2. In-memory checkpoint interception
Migration in daemon
Tier-2: Peer volatile memory Differential checkpoints Differential checkpoints
Base checkpoint
…
G1
U2 … Fn
B2
…
U1 F2
…
B1
…
F1
§3.3 Checkpoint retrieval U1 U2 … Un G1 G2
Gn
3. Fused multi-step differential checkpoint replaying Recover from software failures 2. Cluster-aware checkpoint loading 1. Global consensus on latest checkpoint
Base checkpoint
Tier-3: Remote persistent storage Differential checkpoints
Differential checkpoints
Base checkpoint
Base checkpoint
Recover from node failures
Recover from rack failures
Figure 1. Architectural overview of TierCheck.
after a failure. Its key techniques include: (i) global consensus on latest checkpoint, which identifies the latest globally recoverable version across ranks; (ii) cluster-aware checkpoint loading, which maps checkpoint shards to the current cluster topology and avoids unnecessary data movement; and (iii) fused multi-step differential checkpoint replaying, which reconstructs the latest state by replaying a bounded sequence of differential checkpoints efficiently. • Checkpoint reclamation. This module decides when old volatile checkpoints can be safely removed. Its key mechanism is watermark-driven global reclamation, in which Tier-1 and Tier-2 histories are reclaimed only after the corresponding Tier-3 base checkpoint is globally safe. 3.2
differentials are complex to track, TierCheck leverages gradients to form the high-frequency stream [18, 39]. By intercepting the optimizer-visible partitions already exposed in ZeRO training [26], TierCheck captures and compresses these gradients before they enter the I/O pipeline. The compression strategy adapts to tensor size. For small tensors (e.g., fewer than 100K elements), where sparse compression offers little space reduction (§2.2), TierCheck applies dense INT8 quantization [3]. For large tensors, TierCheck targets a sparsification ratio of 𝑘 = 0.01 [39]. Conventional Top-K selection is prohibitively expensive due to multiple globalmemory passes [2]. Instead, TierCheck employs a thresholdbased sparse kernel: it estimates a magnitude threshold via sampling, then uses a single fused pass to extract and compact surviving entries into FP16 values and INT32 indices. To prevent index overflows in massive models, where a single tensor may contain billions of elements that exceed the INT32 range, oversized tensors are chunked before compression and safely rebased during reassembly. Ultimately, this adaptive design minimizes both the differential checkpoint size and the computational overhead of compression. In-memory checkpoint interception. The standard training framework’s base checkpointing path is suboptimal for tiered storage. For example, DeepSpeed [28] treats persistence as a synchronous, terminal operation, writing serialized checkpoints directly to storage. This opaque design forces a tiered system to incur redundant disk I/O, i.e., writing the full checkpoint only to immediately read it back for peer transmission. To eliminate this write-then-read penalty, TierCheck intercepts the internal torch.save path, cap-
Checkpoint Saving
Design goal. The saving path must resolve the tension that frequent saves are necessary to reduce rollback, but pushing the full training state through tiers on every save incurs significant overhead. TierCheck separates the save path into two asymmetric streams. A lightweight differential stream is generated frequently, exposed first on Tier-1 and Tier-2 for fast recovery, and transferred asynchronously to Tier-3, where explicit completion tracking ensures the checkpoint is fully committed to ensure broader failure coverage. A heavyweight base-checkpoint stream is generated less often and drained away from the critical training loop in the background. The key design goal is to generate each stream once, place it on the tiers that need it, and keep protection off the critical training path. Adaptive gradient compression. Since precise optimizer 5
Iteration
turing serialized base checkpoints directly as in-memory byte payloads. This zero-copy interception allows a single reconstructed state to be simultaneously dispatched to local memory (Tier-1), peer replication (Tier-2), and background remote migration (Tier-3), completely isolating the critical training path from synchronous storage stalls. Asymmetric transmission scheduling. Moving state across tiers without stalling training is challenging. As shown in Figure 2, TierCheck employs distinct strategies for the two checkpoint streams. For lightweight differential checkpoints (typically megabytes in size), TierCheck batches multiple iterations into a compact byte stream using a configurable batching length 𝑁 , where one batch denotes the differential checkpoints from 𝑁 consecutive iterations grouped together for batched saving and replaying [39] (e.g., 𝑁 = 2 in Figure 2). Conversely, monolithic replication of gigabyte-scale base checkpoints would severely block training. To mitigate this, TierCheck applies chunk-based transmission exclusively to Tier-2 base replication, dividing each base checkpoint into paced micro-chunks under a bounded per-iteration bandwidth budget (e.g., completing over 7 iterations as shown in Figure 2). The peer target is deterministically selected via a physical cross-node ring topology, mapping each rank directly to its counterpart on an adjacent machine. This one-to-one mapping ensures that replicas escape the local failure domain without centralized placement logic. To keep peer protection from turning into a blocking transfer, replication must finish before the next base checkpoint is generated. Peer ranks first exchange their serialized payload sizes to establish the maximum bilateral transfer volume. TierCheck then evenly divides this volume across the available training iterations within the base checkpoint interval, reserving a brief safety margin. Consequently, the chunk size adapts dynamically: longer intervals yield finer, less intrusive chunks, while tighter intervals demand larger chunks. To avoid foreground interference, TierCheck caps the maximum chunk size (e.g., 256 MiB). If the base checkpoint interval is too short to drain a massive payload under this cap, TierCheck extends the background transfer schedule across more iterations. As a fallback, if this extended transfer spills over into the next scheduled base checkpoint generation, TierCheck temporarily stalls the foreground training loop to synchronously flush the remaining micro-chunks, guaranteeing memory safety and strict replica consistency before a new checkpoint overwrites the buffer. To prevent such synchronous stalls from degrading training performance, the base checkpoint interval is configured to a sufficiently large value (e.g., 100 iterations), ensuring that even massive payloads can fully drain in the background. Ultimately, this strategy exploits the natural computational gap in the common case while providing a safety net for extreme edge cases.
0
1
2
3
4
5
6
7
8
9
10
Training (Foreground) Differential checkpoint Base checkpoint
C 1
C 2
C 3
C 4
C 5
C 6
C 7
C 1
Save base checkpoint at iteration 0
Figure 2. Adaptive cross-tier transmission without stalling foreground training.
3.3
Checkpoint Retrieval
Design goal. The retrieval path must prioritize both recovery speed and state consistency. Upon failure, TierCheck must identify the latest globally consistent state before resuming from the fastest surviving storage tier. To quantify checkpoint freshness, TierCheck defines a checkpoint version by its corresponding training iteration index. Its retrieval comprises three steps: (i) establishing this latest globally recoverable state, (ii) determining which surviving checkpoint files each rank should consume under the current TP/PP/DP/ZeRO placement, and (iii) replaying the remaining differential checkpoints with bounded overhead. Global consensus on latest checkpoint. After a failure, TierCheck’s recovery re-establishes the communicator across the restarted job and derives global consensus directly from surviving data, so that the consensus protocol does not depend on messages from the crashed process; it runs among the surviving or replacement ranks of the relaunched job. Specifically, each rank first identifies the highest base checkpoint version recoverable from its available tiers (local, peer, or remote). The cluster then computes a global minimum over these local observations to select the latest universally accessible base checkpoint. TierCheck applies this identical distributed reduction to differential checkpoints to establish the maximum valid replay boundary. This decentralized consensus is necessary because durable protection inherently lags behind rapid, volatile saves; ranks often observe locally fresh but globally unrecoverable versions. By aggregating local recoverability via global minima, TierCheck safely filters out transient inconsistencies and deterministically identifies the optimal recovery point. To systematically categorize recovery scenarios, TierCheck models the availability of a version’s base checkpoint as a boolean tuple (𝑆 1, 𝑆 2, 𝑆 3 ) ∈ {0, 1}3 (Figure 3). Here, 𝑆 1 indicates whether the base checkpoint remains available in Tier-1 (local volatile memory), 𝑆 2 denotes its availability in Tier-2 (the designated peer replica), and 𝑆 3 signifies whether it has been durably committed to Tier-3 (remote persistent storage). This availability tuple not only determines whether a specific version can serve as a valid recovery anchor, but also dictates the most cost-effective fallback source following a software failure, a node failure, or a rack failure. 6
Tier-1 Local S1=1?
Yes
Tier-1: Local-anchor recovery (1, *, *)
checkpoint version, each rank independently determines the most recent sequence of differential checkpoints available across its local, peer, and remote tiers. A subsequent global minimum operation then establishes the safe replay boundary for the entire cluster. In essence, the tuple (𝑆 1, 𝑆 2, 𝑆 3 ) dictates whether a version can serve as the exact recovery anchor, whereas the availability of differential checkpoints determines how far recovery can advance beyond that anchor. During the loading phase, differential checkpoints adhere to the same cost-ordered retrieval cascade: local memory first, followed by the designated peer, and finally Tier-3 for any remaining unavailable differential checkpoints. Cluster-aware checkpoint loading. Once the target checkpoint version is chosen, TierCheck needs to address training parallelism in checkpoint loading. Because modern training frameworks combine multiple parallelism strategies (TP/PP/DP/ZeRO), a recovered job cannot simply open a single shared checkpoint directory. Instead, each rank must selectively assemble the specific checkpoint shards dictated by its role in the active parallel layout and retrieve the associated metadata, such as version manifests and commit markers, to correctly map the distributed model and optimizer states to the target version. To address this, TierCheck performs cluster-aware checkpoint loading, following a tiered cascade ordered by retrieval cost. First, a recovering rank attempts to reuse its own local volatile checkpoints (Tier-1) if they survived the failure. If these are unavailable, the rank pulls the missing files directly from its assigned cross-node peer (Tier-2), mirroring the ring topology used during checkpoint saving. Next, for model states shared across multiple ranks under the active parallel layout, TierCheck fetches them from surviving intra-cluster peers. Only the remaining unavailable files are fetched from Tier-3. Even during a durable fallback to remote storage, loading remains highly selective: a single leader process on each node enumerates exactly the files required by its local ranks and acts as a proxy to fetch only those specific shards, avoiding a full directory-wide remote download. Fused multi-step differential checkpoint replaying. Even with efficient cluster-aware loading, differential checkpoint replay remains on the critical recovery path. Because the Adam optimizer is stateful, iterations cannot simply be accumulated or parallelized. Each iteration rigorously depends on the first and second moments produced by all preceding iterations. Consequently, a naive recovery must invoke the standard distributed optimization routine for every replayed iteration. As shown in Figure 4(a), this naive approach incurs massive penalties at every iteration: it needlessly triggers expensive distributed communications (e.g., cluster-wide collective operations to synchronize sharded gradients), repeatedly reads and writes the entire 14Φ-byte training state to GPU memory, and suffers from severe CPU-to-GPU dispatch overhead caused by launching numerous fine-grained, per-parameter operators from the native Python runtime.
No Tier-2 Peer S2=1?
Yes
Tier-2: Peer-anchor recovery (0, 1, *)
No Tier-3 Remote S3=1?
Yes
Tier-3: Remote-anchor only recovery (0, 0, 1)
No Global miss (0, 0, 0)
Figure 3. Availability of base-checkpoint anchors across different storage tiers under heterogeneous failures.
• Local-anchor recovery (𝑆 1 = 1). This category covers states (1, 0, 0), (1, 1, 0), (1, 0, 1), and (1, 1, 1). When a software failure terminates training while the host node remains healthy, TierCheck restores the base checkpoint directly from the local volatile copy, bypassing slower tiers entirely. The presence of additional peer or remote copies merely broadens the fallback space; it does not alter this optimal first choice. • Peer-anchor recovery (𝑆 1 = 0, 𝑆 2 = 1). This category covers states (0, 1, 0) and (0, 1, 1). When a local node fails but its designated peer replica survives, TierCheck reconstructs the missing base checkpoint from Tier-2. This volatile replica is significantly cheaper to access than a remote fetch. Even if a durable remote copy exists (𝑆 3 = 1), Tier-3 acts strictly as a secondary backup rather than the preferred recovery path. • Remote-anchor-only recovery (𝑆 1 = 0, 𝑆 2 = 0, 𝑆 3 = 1). If both volatile copies are lost but the base checkpoint has been durably committed to Tier-3, TierCheck falls back to this persistent remote base checkpoint. This scenario typically arises during rack failures and serves as the lastresort recovery path for any state that cannot be retrieved from the faster tiers. • Global miss (𝑆 1 = 0, 𝑆 2 = 0, 𝑆 3 = 0). A catastrophic outage eradicates the latest base checkpoint across all tiers before a durable commit completes. Such events are exceedingly rare in practice, as they require a correlated cluster-wide failure to strike exactly during the narrow window of an asynchronous transfer to Tier-3. To safeguard against this, TierCheck relies on a commit watermark to track durable completions; upon detecting an incomplete state, the system safely rolls back to the most recent base checkpoint version that is globally available and verified by the watermark. The availability of differential checkpoints is evaluated independently of the base checkpoint availability tuple. Once the cluster reaches consensus on the latest recoverable base 7
Load differential checkpoint
Distributed communication Repeated N iterations High overhead
Memory I/O (read 14Φ bytes)
Load differential checkpoints: 1… N
Node A
Customized fused operator Memory I/O (read 14Φ bytes once)
Memory I/O (write 14Φ bytes once)
Memory I/O (write 14Φ bytes)
Native optimizer update with N-th differential checkpoint
Removed
Trigger to reclaim
Node A
Node B
Node B
V-1
V
V-1
V
Tier-1
V-1
V
V-1
V
Tier-2
V-1
V
V-1
V
Tier-2
V-1
V
V-1
V
V-1
V
V-1
V
Tier-3
V-1
V
V-1
V
(a) Naive
(b) Watermark-driven
Figure 5. Comparison of naive and watermark-driven global reclamation.
newest locally visible state is incorrect under heterogeneous failures, as local recency provides no guarantee of clusterwide redundancy. Reclamation must thus be governed by global recoverability rather than per-node progress. Watermark-driven global reclamation. TierCheck resolves this synchronization challenge by deferring garbage collection using a strictly monotonic global watermark. This watermark tracks the highest checkpoint version whose base checkpoint is fully persisted to Tier-3, thereby establishing a globally stable replay anchor. To illustrate the necessity of this mechanism, consider the naive reclamation in Figure 5(a). A fast node (Node A) locally completes base checkpoint 𝑉 and eagerly reclaims the volatile recovery chain rooted at 𝑉 − 1. This premature deletion includes its Tier-1 and Tier-2 base copies of 𝑉 − 1, as well as any subsequent differential checkpoints. Meanwhile, a slower node (Node B) still holds 𝑉 − 1 in its volatile tiers. Because transferring massive model states takes significant time, the new base checkpoint 𝑉 may still be uploading to the slow Tier-3 storage, leaving 𝑉 − 1 as the only complete, durably committed remote anchor. If Node A fails, its local Tier-1 memory is lost, and the system must fall back to the remote anchor at 𝑉 − 1. However, the sequence of differential checkpoints bridging 𝑉 − 1 and 𝑉 is now fragmented. While some incremental updates may have already reached Tier-3 or survived on other nodes, asynchronous propagation means the full cross-rank chain is no longer intact. Once Node A’s local volatile copies are reclaimed, this missing data breaks the globally continuous chain required for exact replay. Consequently, the latest recoverable state is severely truncated: in the worst case, recovery falls back to the bare Tier-3 anchor at 𝑉 − 1. Figure 5(b) shows that TierCheck eliminates this vulnerability by decoupling local creation from deletion. The watermark for version 𝑉 advances only after its base checkpoint has been durably flushed to Tier-3 on all participating nodes (Nodes A and B). A designated central coordinator (e.g., the process with global distributed rank 0) then broadcasts this safe watermark, enforcing a unified global boundary for state reclamation across all ranks. Once the watermark advances to version 𝑉 , each rank safely purges obsolete states across all tiers: (i) strictly preceding differential checkpoints located in local, peer, and remote storage; (ii) local base checkpoints older than 𝑉 ; and (iii) base checkpoints replicated to peer
(b) Fused multi-step replay
Figure 4. Comparison of native and fused multi-step differential checkpoint replaying.
To eliminate these overheads, TierCheck introduces fused multi-step differential checkpoint replaying. Rather than replaying the entire historical chain in a single monolithic pass, TierCheck streams the recovered differential checkpoints and replays them in bounded batches. Each replay batch contains 𝑁 consecutive differential checkpoints (§3.2). As shown in Figure 4(b), all 𝑁 checkpoints in a batch are first loaded onto the recovery path. A customized fused operator then consumes the first 𝑁 − 1 checkpoints together: it reads the model weights, first moments, and second moments exactly once, applies the corresponding 𝑁 − 1 incremental gradients in temporal order locally on each rank, and writes the final results back to memory. Because these updates are applied directly to the local parameter partitions on each rank, this design seamlessly supports ZeRO-1/2/3 without requiring intermediate cross-rank synchronizations. While minor floating-point variations inherent to fused kernels and gradient compression preclude strict bitwise exactness, TierCheck maintains strong algorithmic equivalence to the baseline optimizer, paying communication costs exactly once per batch to accelerate replay by orders of magnitude. 3.4
Saved
Tier-1
Tier-3
Continuous optimizer update on local ranks with differential checkpoints 1… (N-1)
Optimizer update
(a) Native replay
Pending to save
Checkpoint Reclamation
Design goal. The reclamation path must resolve a tension in tiered checkpointing: volatile checkpoints cannot be retained indefinitely due to Tier-1 and Tier-2 memory constraints, yet they cannot be prematurely discarded based solely on local freshness. As checkpoint saving proceeds asynchronously, a newly generated base checkpoint may manifest in the volatile tiers of a fast node while slower nodes are still processing it and remote commits remain in flight. Consequently, a naive local-age policy that eagerly reclaims historical differential checkpoints preceding the 8
nodes older than 𝑉 . The base checkpoint at 𝑉 is preserved as the current globally safe anchor.
4
loads its shards and writes a local completion marker; the designated coordinator (e.g., rank 0) then verifies all markers before publishing a global _COMMITTED signal. Any checkpoint lacking this signal is ignored during recovery. Once committed, the coordinator advances the global watermark, broadcasting it to trigger asynchronous garbage collection on all ranks. The cleanup routine purges: (i) differential checkpoints whose replay range precedes the watermark from Tier-1 and Tier-2, and (ii) base checkpoints older than the watermark from both local and peer storage. Differential checkpoints in Tier-3 are retained for one additional base checkpoint interval to ensure freshly provisioned replacement ranks can always reconstruct state from Tier-3 alone.
Implementation
We implement TierCheck as a fault-tolerance layer atop PyTorch [24] and DeepSpeed [28] in 12 K lines of code. TierCheck hooks directly into existing optimizer, serialization, RPC, and communication interfaces without rewriting the underlying frameworks. High-level coordination logic is written in Python; performance-critical compression and replay primitives are accelerated via custom CUDA extensions. Saving path. TierCheck generates differential checkpoints by extracting incremental updates directly from rank-local gradient buffers exposed by the training framework. The compression pipeline dynamically selects between dense INT8 quantization [3] for small tensors and sampledthreshold sparse compression [39] for large ones. All deviceside captures, host-device staging, and serialization tasks are offloaded to low-priority CUDA streams and pinned host memory buffers for fully asynchronous execution. For base checkpoints, TierCheck intercepts the internal serialization routine (torch.save) to capture each emitted shard as a zerocopy in-memory byte payload, which is then simultaneously dispatched to Tier-1 local memory, replicated to Tier-2 peer memory, and queued for asynchronous migration to Tier-3. Tier-2 replication uses isolated communication groups that decouple the RPC-based metadata control plane from the tensor-based chunk data plane, and an adaptive chunk-based transmission scheduler that prevents gigabyte-scale base checkpoints from stalling training. Retrieval path. Upon restart, each rank independently identify its newest recoverable base checkpoint across surviving tiers via global consensus, then materializes only the shards required by its active DP/TP/PP/ZeRO placement following the cost-ordered cascade (i.e., prioritizing Tier-1, then Tier-2 peer replicas, and accessing Tier-3 as a last resort). Differential checkpoints undergo an identical consensus process to establish the safe replay boundary. Compressed updates are reconstructed directly on the recovery GPUs: sparse payloads are fused into dense tensors and INT8 payloads are dequantized from their stored scales. The fused multi-step replay operator consolidates the first 𝑁 − 1 differential steps in a replay batch into a single device-side update pass before re-exposing the final step to the native optimizer path, which is necessary to correctly trigger the post-update communication and optimizer state transitions (e.g., gradient synchronization and loss-scaler updates) that the fused kernel does not replicate. Tier-3 commit and reclamation. Tier-3 persistence is offloaded to a dedicated background daemon. As remote object stores lack atomic directory operations, the daemon employs a two-phase marker-based commit protocol: each rank up-
5
Evaluation
We evaluate TierCheck and summarize our findings below: • Training efficiency. TierCheck achieves lower training overhead and higher checkpointing frequencies than stateof-the-art checkpointing systems (Exp#1-3). • Recovery efficiency. TierCheck reduces recovery time for heterogeneous failures by leveraging Tiers-1/2’s checkpoints, and provides robust fallback to Tier-3 for rack failures (Exp#4). • Scalability and compatibility. TierCheck maintains its performance advantages across different model sizes (Exp#5) and varying parallelism configurations (Exp#6). • Microbenchmarks. TierCheck’s fused multi-step replay accelerates differential checkpoint recovery (Exp#7); its model convergence is preserved without accuracy degradation (Exp#8); and its multi-tier global reclamation strictly bounds the storage footprint without risking data loss (Exp#9). 5.1
Experimental Setup
Testbed environment. We conduct evaluations on a 16GPU testbed provisioned from a distributed cloud platform. The testbed comprises four compute nodes, each equipped with four NVIDIA A800 (80 GiB) GPUs and 512 GiB of CPU memory, interconnected via a 200 Gbps InfiniBand network. The remote Tier-3 storage is backed by NVMe SSD-based Lustre (v2.14.0) over an aggregated 400 Gbps Ethernet network. The software stack includes Ubuntu 22.04, PyTorch v2.6.0, DeepSpeed v0.16.4, CUDA 12.4, and NCCL v2.21.5. Models and workloads. We evaluate TierCheck using representative LLM architectures by adjusting hidden/intermediate sizes, attention heads, and layers [22, 35], as shown in Table 2. We train BERT on SQuAD [27] and other models on WikiText-103 or WikiText-2 [21] using the Adam optimizer [14]. While WikiText-2 is used solely to verify the convergence of the GPT2 124M model (Exp#8), all other evaluations employ billion-scale models (up to 40B parameters). Packing these 40B-parameter models into just 16 GPUs serves as a stress test that maximizes per-node mem9
BERT 40B
RoBERTa 40B
BLOOM 40B
Figure 6. (Exp#1) Average training time per iteration.
0
GPT2 20B
BERT 20B
0
11
11
20 10
22
30
5
10
CheckFreq 21
20
DataStates-LLM
12
Checkpointing time (s)
30
Gemini
11
TierCheck
ory, network, and storage I/O pressure. A system resilient to this extreme contention will naturally scale to larger clusters, where the per-GPU checkpoint payload is diluted. Baselines. We compare TierCheck against three checkpointing baselines: CheckFreq [23], Gemini [35], and DataStates-LLM [20], and against the no-checkpoint baseline DeepSpeed ZeRO-3 [26, 28]. Since Gemini is not opensourced, we have re-implemented it to the best of our ability based on its original publication. We also extended the original CheckFreq to support ZeRO-3 for fair comparison. Default setups. By default, TierCheck captures gradients at every training iteration to construct differential checkpoints. For adaptive gradient compression, we set the small-tensor quantization threshold to 100K elements and the large-tensor compression ratio to 𝑘 = 0.01. The Tier-2 base checkpoint replication uses a maximum chunk size of 256 MiB to avoid foreground interference. The batching length for saving differential checkpoints is set to 𝑁 = 5 [39]. The base checkpoint interval is fixed at 50 iterations, except in Exp#8 where it is extended to 100 iterations to test the replay efficiency of differential checkpoints. We enable ZeRO-3 and evaluate the compatibility of supporting different parallelisms in Exp#6. Failure injection. We use failure injection in Exp#4 and Exp#8. For Exp#4, we emulate three recovery scenarios. For software failures, we terminate the training processes while keeping all nodes alive, and then relaunch the job on the same cluster. For node failures, we assume any one of the four nodes fails, replace it with a fresh node, and restart training from the surviving checkpoints. For rack failures, we conservatively assume that all four nodes in our 16-GPU testbed reside in the same rack and experience a correlated power outage; after power is restored, all four nodes are rebooted and training resumes from the recovered checkpoint state. For Exp#8, to emulate frequent interruptions during longrunning training, we periodically inject software failures by terminating the training processes every ∼3,000 iterations and relaunching the job on the same cluster. 5.2
GPT2 40B
CheckFreq
5
128
Ckpt. frequency (iterations)
40
20.0
20480
13.8
5120
11.5
12 46 64
Gemini DataStates-LLM
3.2
12 40 40
19.6
3072 10240 20480
16.7
768 2048 5120
10 8 6 4 2 0
12.9
GPT2 124M 10B (GPT2) 20B (GPT2, BERT) 40B (GPT2, BERT, RoBERTa, BLOOM)
Hidden Size Intermediate #AH #Layers
No Ckpt. TierCheck
3.1
Model
Training time (s)
Table 2. Model configurations used in the evaluation. #AH is the number of attention heads. #Layers is the number of layers.
GPT2 20B
BERT 20B
Figure 7. (Exp#2) Checkpointing Figure 8. (Exp#3) Checkpoint time. frequency.
These performance gaps stem from fundamental differences in checkpointing architectures. CheckFreq incurs the highest overhead because it snapshots unpartitioned training states, causing severe synchronous stalls. While DataStatesLLM divides checkpoints into three large partitions, these remain difficult to overlap efficiently and inevitably stall forward passes [20, 22]. Gemini mitigates this by fragmenting states into smaller 32 MiB chunks to fit memory copies within the training pipeline’s idle timespans. TierCheck bypasses these heavy memory transfers entirely, minimizing foreground overhead by directly intercepting and adaptively compressing native gradients (§3.2). (Exp#2) Checkpointing time. We define checkpointing time as the duration required to persist training states to storage tiers. We conduct evaluation on 20B models to prevent prohibitive storage demands. Figure 7 shows that TierCheck completes checkpointing in 3.1-3.2 s per iteration on average for two 20B models. Existing baselines exhibit higher persistence latencies: DataStates-LLM requires 11.5-12.9 s, Gemini takes 13.8-16.7 s, and CheckFreq incurs the highest latency at 19.6-20.0 s per checkpoint. These performance gaps stem from differences in underlying persistence mechanisms. CheckFreq flushes unpartitioned full model states directly to disk and incurs heavy synchronous I/O overhead. Gemini uses small partitions to hide foreground interference, but leaves backend disk flushing unoptimized, resulting in a persistence bottleneck. DataStates-LLM implements asynchronous I/O optimizations in C++. In contrast, TierCheck persists differential checkpoints in an optimized manner rather than persisting massive full states frequently (§3.2). (Exp#3) Checkpoint frequency. We evaluate the maximum checkpointing frequency achievable by each method under a 3.5% training speed degradation bound [22, 23]. We report this frequency as the minimum required checkpoint interval (i.e., the number of training iterations between consecutive saves); a smaller interval signifies a higher check-
Macrobenchmarks
(Exp#1) Average training time per iteration. We evaluate the average training time over 50 iterations using 40B models with per-iteration snapshots. Figure 6 shows that TierCheck incurs only a 10.7-15.1% overhead, outperforming state-of-the-art baselines by 52.3-60.8% (Gemini), 56.970.4% (DataStates-LLM), and 68.2-82.7% (CheckFreq). 10
Failures
𝑇𝑟𝑜𝑙𝑙𝑏𝑎𝑐𝑘 𝑇𝑟𝑒𝑟𝑢𝑛 Total
Model: GPT2 20B CheckFreq All Scenarios 17.8 s Gemini All Scenarios 15.4 s DataStates-LLM All Scenarios 16.0 s Software failures 4.5 s TierCheck Node failures 9.6 s Rack failures 18.4 s Model: BERT 20B CheckFreq All Scenarios 17.8 s All Scenarios 16.4 s Gemini DataStates-LLM All Scenarios 16.4 s Software failures 5.5 s TierCheck Node failures 9.8 s Rack failures 19.8 s
42.9 s 17.9 s 19.9 s 4.3 s 8.8 s 5.5 s 45.8 s 17.8 s 18.0 s 4.1 s 8.3 s 5.9 s
60.5 s 33.3 s 35.9 s 8.8 s 18.4 s 23.9 s
No Ckpt. TierCheck
Gemini DataStates-LLM
CheckFreq
8 6 4 2 0
Training time (s)
Systems
Training time (s)
Table 3. (Exp#4) Recovery time under different failure scenarios.
3
TierCheck
2 1 0
GPT2 10B GPT2 20B GPT2 30B GPT2 40B
No Ckpt.
DP=16 DP=4 (ZeRO-3) PP=4
DP=4 TP=4
PP=4 DP=4 TP=4 TP/PP=2
Figure 9. (Exp#5) Scalability of dif- Figure 10. (Exp#6) Compatiferent model sizes. bility with parallelisms.
TierCheck’s localized recovery advantages would be even more pronounced in deployments with constrained remote bandwidth. (Exp#5) Scalability of different model sizes. We evaluate the scalability of different model sizes by measuring the average training time per iteration with GPT2 models ranging from 10B to 40B parameters. As shown in Figure 9, TierCheck consistently outperforms the evaluated baselines across all model scales. While the checkpointing overhead of existing systems grows significantly as the model size increases, TierCheck remains closely aligned with the nocheckpoint baseline, incurring a minimal training overhead of only 10.6-15.1%. This indicates that TierCheck’s asynchronous transmission and adaptive compression mechanisms maintain consistent performance advantages across varying model dimensions and scale effectively to larger models. (Exp#6) Compatibility with parallelisms. We evaluate five parallelism configurations: DP=16 (ZeRO-3), DP=4/PP=4, DP=4/TP=4, PP=4/TP=4, and balanced 3D parallelism with DP=4/PP=2/TP=2. Since the other checkpointing systems do not support such 3D-parallel training configurations, we compare only against the no-checkpoint DeepSpeed baseline. Figure 10 shows that TierCheck remains consistently close to the no-checkpoint DeepSpeed baseline across all settings, demonstrating its compatibility with both pure data parallelism and hybrid 3D parallelism. The training-time overhead of TierCheck over DeepSpeed is 11.8% for DP=16, 15.8% for DP/PP=4, 15.2% for DP/TP=4, 15.3% for PP/TP=4, and 12.6% for DP=4/PP=2/TP=2. Overall, these results indicate that TierCheck does not rely on a particular parallelism strategy and maintains low additional overhead across diverse 3D-parallel deployments.
63.6 s 34.2 s 34.4 s 9.6 s 18.0 s 25.7 s
pointing frequency and thus stronger resilience to failures. Figure 8 shows that TierCheck achieves the highest frequency, requiring an interval of 5 iterations across the evaluated 20B models, which is also consistent to the batching length 𝑁 = 5. In contrast, existing methods demand longer intervals to satisfy the same degradation bound: Gemini requires 11 iterations, DataStates-LLM requires 11-12 iterations, and CheckFreq requires 21-22 iterations. (Exp#4) Recovery time under different failure scenarios. We evaluate recovery time, comprising rollback time (𝑇𝑟𝑜𝑙𝑙𝑏𝑎𝑐𝑘 ) to load the saved checkpoint and rerun time (𝑇𝑟𝑒𝑟𝑢𝑛 ) to replay lost iterations. Specifically, 𝑇𝑟𝑒𝑟𝑢𝑛 is the average of best-case (zero iterations lost) and worst-case (a full interval of iterations lost) re-execution times based on achievable checkpoint frequencies (Exp#3). For TierCheck, 𝑇𝑟𝑜𝑙𝑙𝑏𝑎𝑐𝑘 denotes the time to fetch the base checkpoint, while 𝑇𝑟𝑒𝑟𝑢𝑛 includes asynchronously pulling differential checkpoints and replaying lost iterations, with the worst-case replay length bounded by its base checkpoint interval of 50 iterations. Table 3 shows that TierCheck recovers in 8.8-9.6 s for software failures and 18.0-18.4 s for node failures. Baselines are significantly slower; their 𝑇𝑟𝑒𝑟𝑢𝑛 alone exceeds TierCheck’s total software recovery time. Gemini flushes its volatile buffers to the shared file system upon failure, degrading all its recoveries to slow Tier-3 fetches. TierCheck’s speedup stems from its tiered architecture. For 𝑇𝑟𝑜𝑙𝑙𝑏𝑎𝑐𝑘 , TierCheck rapidly fetches the base checkpoint from Tier-1 local memory (4.5-5.5 s) for software failures and Tier-2 peer memory (9.6-9.8 s) for node failures, only incurring higher latencies (18.4-19.8 s) during rack failures due to sequential probing before falling back to Tier-3. For 𝑇𝑟𝑒𝑟𝑢𝑛 , lightweight differential checkpoints reduce replay overhead to 4.1-8.8 s for software failures (§3.2). 𝑇𝑟𝑒𝑟𝑢𝑛 is slightly higher for node failures than rack failures (8.3-8.8 s vs. 5.5-5.9 s) because the former relies on the 200 Gbps training network while the latter fetches directly over the 400 Gbps Tier-3 path. Thus,
5.3
Microbenchmarks
(Exp#7) Fused multi-step differential checkpoint replay. Figure 11 evaluates the recovery time on a GPT2 20B model, comparing TierCheck’s fused multi-step replay against sequential differential checkpoint replay and standard DeepSpeed recovery (i.e., rerunning lost iterations). TierCheck’s fused multi-step replay consistently outperforms the sequential baseline across all evaluated iteration scales. For a recovery chain of 100 iterations, TierCheck reduces the rerun time from 26.0 s to 16.6 s (i.e., 36.2% reduction) over sequential replay. In contrast, DeepSpeed’s recovery spends 239.3 s to process the same interval, making 11
101 20
40
60 Iterations
80
100
Baseline TierCheck Crash & Recovery
6 4 2 0 0.0
0.5
1.0
1.5 2.0 Iterations
2.5
3.0
3.5
×104
Figure 11. (Exp#7) Fused multi-step differen- Figure 12. (Exp#8) Convergence accuracy. tial checkpoint replay.
it 14.4× slower than TierCheck. This performance gain is attributed to the fused operator’s ability to consolidate multiple optimizer updates into a single pass, thereby mitigating redundant memory I/O and cross-rank communication synchronizations. (Exp#8) Convergence accuracy. To verify algorithmic correctness without the prohibitive cost of 40B-scale end-to-end training, we fine-tune a GPT2 (124M) model on WikiText-2. This model size features a diverse tensor distribution that simultaneously triggers INT8 quantization (e.g., for small biases) and Top-K sparsification (e.g., for large weights). By injecting failures every ∼3,000 iterations, Figure 12 shows TierCheck closely tracks the fault-free baseline and even achieves a slightly lower final loss. This improvement likely occurs because the TopK compression of differential checkpoints naturally filters out insignificant gradient fluctuations [2]. Crucially, because larger models contain a vast number of redundant parameters and naturally exhibit even stronger gradient sparsity (i.e., more gradients are close to zero) [18, 39], this algorithmic robustness effectively scales to 40B configurations without accuracy loss. (Exp#9) Storage overhead. Figure 13 shows a traditional baseline linearly accumulating states to 1,140 GiB by iteration 100. Conversely, TierCheck strictly bounds storage via watermark-driven global reclamation. For a 20B model on 16 GPUs, the steady-state footprint peaks at 342.1 GiB for global Tier-3 and 171.1 GiB for per-node Tier-1/2. Because a newly generated base checkpoint momentarily coexists with the previous base checkpoint and accumulated differential checkpoints, the system incurs brief transient spikes (570.1 GiB in Tier-3, 228.1 GiB in Tier-1/2) during handoff. Post-reclamation, these footprints rapidly reset to 228 GiB and 57 GiB, respectively. This demonstrates that TierCheck prevents continuous storage exhaustion, ensuring sustainable training under constrained resources.
6
Storage overhead (GiB)
TierCheck TierCheck-native DeepSpeed
Training loss
Rerun time (s)
102
w/o reclamation Tier-3 storage overhead (all 4 nodes) Tier-1/2 storage overhead (per node)
800 600 400 200 0
0
50
100 Iterations
150
200
Figure 13. (Exp#9) Storage overhead.
ing for massive models. To address multi-dimensional parallelism, ByteCheckpoint [34] introduces a unified representation to decouple checkpointing from specific parallelism modes, while Universal Checkpointing [17] enables seamless resumption under reconfigurable parallelism. At the network level, FlowCheck [12] extracts gradients from mirrored traffic using dedicated CPU nodes, and AsymCheck [22] employs asymmetric partitioned checkpointing to exploit varying communication idleness across training phases. While these approaches reduce training overhead or enhance checkpoint flexibility, they share a common limitation: failure recovery remains bottlenecked by slow remote storage I/O, since all persistent state ultimately resides in a single remote tier. TierCheck bypasses remote storage entirely for the common single-node and GPU failures. Minimizing checkpoint size. Reducing checkpoint size reduces storage and network I/O pressure. LowDiff [39] persists incremental states via compressed gradients; AdaCheck [18] adapts to various parallelism modes by profiling tensor redundancy to store only half-precision gradients; MoEvement [9] introduces sparse checkpointing for Mixture-ofExperts (MoE) models by incrementally snapshotting active expert subsets. However, aggressive size reductions incur recovery overhead: LowDiff incurs sequential replay overhead over accumulated differences; AdaCheck imposes runtime redundancy-detection costs; MoEvement requires localized recomputation for unsnapshotted states. TierCheck applies lightweight Top-K threshold estimation to keep differential payloads small without expensive profiling, and uses fused multi-step operators to accelerate sequential replay. Fast recovery. In-memory architectures like Gemini [35] and Transom [37] preserve snapshots in host CPU memory to achieve near-zero recovery for isolated failures. Swift [42] leverages data-parallelism group replicas combined with selective recomputation, while FT-HSDP [30] implements replica-level fault tolerance. CheckFree [4] explores checkpoint-free recovery via weighted averaging of neighboring pipeline stages. MoEvement [9] enables localized recovery for MoE models via upstream activation logging. TierCheck targets heterogeneous failures and maintains fast recovery via cluster-aware tiered checkpointing.
Related Work
Optimizing checkpointing efficiency. Prior work buffers training states in local RAM-disks or GPU multi-level caches [19, 38], or exploits persistent memory for zero-copy checkpointing [16]. To overlap I/O with computation, CheckFreq [23], DataStates-LLM [20], and PCcheck [33] pipeline partitioned checkpoints into host CPU buffers. PyTorch Distributed Checkpoint [25] provides standardized sharded sav12
7
Conclusion
Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al. 2024. MegaScale: Scaling large language model training to more than 10,000 GPUs. In Proc. of USENIX NSDI. [14] Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR. [15] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Thomas Wolf, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, and Colin Raffel. 2022. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv preprint arXiv:2211.05100 (2022). [16] Yuanhao Li, Tianyuan Wu, Guancheng Li, Yanjie Song, and Shu Yin. 2024. Portus: Efficient DNN checkpointing to persistent memory with zero-copy. In Proc. of ICDCS. [17] Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, and Minjia Zhang. 2025. Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Parallelism. In Proc. of USENIX ATC. [18] Weijie Liu, Shengwei Li, Zhiquan Lai, Keshi Ge, Qiaoling Chen, Peng Sun, Dongsheng Li, and Kai Lu. 2026. AdaCheck: An Adaptive Checkpointing System for Efficient LLM Training with Redundancy Utilization. In Proc. of USENIX FAST. [19] Avinash Maurya, M Mustafa Rafique, Thierry Tonellot, Hussain J AlSalem, Franck Cappello, and Bogdan Nicolae. 2023. GPU-enabled asynchronous multi-level checkpoint caching and prefetching. In Proc. of HPDC. [20] Avinash Maurya, Robert Underwood, M Mustafa Rafique, Franck Cappello, and Bogdan Nicolae. 2024. DataStates-LLM: Lazy asynchronous checkpointing for large language models. In Proc. of HPDC. [21] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 (2016). [22] Zhangqiang Ming, Yuchong Hu, Zhiyuan Luo, Patrick P. C. Lee, Yuanhao Shu, Wenxiang Zhou, and Dan Feng. 2026. AsymCheck: Asymmetric Partitioned Checkpointing for Efficient Large Language Model Training. In Proc. of ACM/IEEE DAC. [23] Jayashree Mohan, Amar Phanishayee, and Vijay Chidambaram. 2021. CheckFreq: Frequent, Fine-Grained DNN Checkpointing. In Proc. of USENIX FAST. [24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proc. of NeurIPS. [25] PyTorch Team. 2024. Distributed Checkpoint (DCP) — PyTorch Tutorials. https://docs.pytorch.org/tutorials/recipes/distributed_ checkpoint_recipe.html. [26] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory optimizations toward training trillion parameter models. In Proc. of SC. [27] Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822 (2018). [28] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proc. of KDD.
We present TierCheck, a cluster-aware tiered checkpointing system that tolerates heterogeneous failures in large-scale LLM training. By aligning checkpoint placement with failure domains, TierCheck decouples state persistence into a high-frequency differential checkpoint stream and a lowfrequency base checkpoint stream, distributing them across local memory, peer memory, and remote persistent storage. To minimize training stalls, it introduces adaptive gradient compression and asymmetric transmission scheduling. During recovery, it leverages decentralized consensus and fused multi-step differential checkpoint replaying to enable fast, localized restoration. Furthermore, a watermark-driven global reclamation mechanism ensures strict cross-tier consistency and reliably bounds storage footprints. Evaluations on up to 40B-parameter models demonstrate that TierCheck effectively reduces both checkpointing overhead and recovery time against state-of-the-art baselines.
References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv (2023), arXiv preprint arXiv:2303.08774. [2] Alham Fikri Aji and Kenneth Heafield. 2017. Sparse Communication for Distributed Gradient Descent. In Proc. of EMNLP. [3] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2017. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. In Proc. of NeurIPS. [4] Nikolay Blagoev, Oğuzhan Ersoy, and Lydia Yiyu Chen. 2026. All is Not Lost: LLM Recovery without Checkpoints. In Proc. of EuroMLSys. [5] James Bornholt, Rajeev Joshi, Vytautas Astrauskas, Brendan Cully, Bernhard Kragl, Seth Markle, Kyle Sauri, Drew Schleit, Grant Slatton, Serdar Tasiran, Jacob Van Geffen, and Andrew Warfield. 2021. Using lightweight formal methods to validate a key-value storage node in Amazon S3. In Proc. of ACM SOSP. [6] J. Dean. 2009. Designs, lessons and advice from building large distributed systems. Keynote talk at LADIS. [7] Elmootazbellah Nabil Elnozahy, Lorenzo Alvisi, Yi-Min Wang, and David B Johnson. 2002. A survey of rollback-recovery protocols in message-passing systems. Comput. Surveys 34, 3 (2002), 375–408. [8] Daniel Ford, François Labelle, Florentina I Popovici, Murray Stokely, Van-Anh Truong, Luiz Barroso, Carrie Grimes, and Sean Quinlan. 2010. Availability in Globally Distributed Storage Systems. In Proc. of USENIX OSDI. [9] Swapnil Gandhi and Christos Kozyrakis. 2026. Sparse Checkpointing for Fast and Reliable MoE Training. In Proc. of USENIX NSDI. [10] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 herd of models. arXiv (2024), arXiv preprint arXiv:2407.21783. [11] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Proc. of NeurIPS. [12] Zimeng Huang, Hao Nie, Haonan Jia, Bo Jiang, Junchen Guo, Jianyuan Lu, Rong Wen, Biao Lyu, Shunmin Zhu, and Xinbing Wang. 2025. FlowCheck: Decoupling Checkpointing and Training of Large-Scale Models. In Proc. of EuroSys. [13] Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, 13
[29] Cedric Renggli, Saleh Ashkboos, Mehdi Aghagolzadeh, Dan Alistarh, and Torsten Hoefler. 2019. SparCML: High-Performance Sparse Communication for Machine Learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC). ACM. [30] Omkar Salpekar, Rohan Varma, Kenny Yu, Vladimir Ivanov, Yang Wang, Ahmed Sharif, Min Si, Shawn Xu, Feng Tian, Shengbao Zheng, et al. 2026. Training LLMs with Fault Tolerant HSDP on 100,000 GPUs. arXiv preprint arXiv:2602.00277 (2026). [31] Philip Schwan. 2003. Lustre: Building a file system for 1,000-node clusters. In Proc. of Linux Symposium. [32] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv (2019), arXiv preprint arXiv:1909.08053. [33] Foteini Strati, Michal Friedman, and Ana Klimovic. 2025. PCcheck: Persistent Concurrent Checkpointing for ML. In Proc. of ACM ASPLOS. [34] Borui Wan, Mingji Han, Yiyao Sheng, Yanghua Peng, Haibin Lin, Mofan Zhang, Zhichao Lai, Menghan Yu, Junda Zhang, Zuquan Song, Xin Liu, and Chuan Wu. 2025. ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development. In Proc. of USENIX NSDI. [35] Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xinwei Fu, TS Eugene Ng, and Yida Wang. 2023. Gemini: Fast failure recovery in distributed training with in-memory checkpoints. In Proc. of SOSP.
[36] Sage A Weil, Scott A Brandt, Ethan L Miller, Darrell DE Long, and Carlos Maltzahn. 2006. Ceph: A scalable, high-performance distributed file system. In Proc. of USENIX OSDI. [37] Baodong Wu, Lei Xia, Qingping Li, Kangyu Li, Xu Chen, Yongqiang Guo, Tieyao Xiang, Yuheng Chen, and Shigang Li. 2023. Transom: An efficient fault-tolerant system for training LLMs. arXiv (2023), arXiv preprint arXiv:2310.10046. [38] Wubiao Xu, Xin Huang, Shiman Meng, Weiping Zhang, Luanzheng Guo, and Kento Sato. 2024. An Efficient Checkpointing System for Large Machine Learning Model Training. In Proc. of SC Workshops. [39] Chenxuan Yao, Yuchong Hu, Feifan Liu, Zhengyu Liu, and Dan Feng. 2025. LowDiff: Efficient Frequent Checkpointing via Low-Cost Differential for High-Performance Distributed Training Systems. In Proc. of SC. [40] Mi Zhang, Shujie Han, and Patrick P. C. Lee. 2019. SimEDC: A Simulator for the Reliability Analysis of Erasure-Coded Data Centers. IEEE Transactions on Parallel and Distributed Systems 30, 12 (2019), 2836–2848. [41] Ru Zhang, Wencong Xiao, Hongyu Zhang, Yu Liu, Haoxiang Lin, and Mao Yang. 2020. An empirical study on program failures of deep learning jobs. In Proc. of ACM/IEEE ICSE. [42] Yuchen Zhong, Guangming Sheng, Juncheng Liu, Jinhui Yuan, and Chuan Wu. 2024. Swift: Expedited Failure Recovery for Large-Scale DNN Training. IEEE Transactions on Parallel and Distributed Systems 35, 9 (2024), 1644–1656.
14