SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training

SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training Zhuang Wang∗

arXiv:2608.11034v1 [cs.DC] 11 Aug 2026

Abstract

production study found fail-slows delaying large jobs by 1.34× on average [49]. Detecting a failure is insufficient to resume training because recovery must also localize where to act. A timeout, slowdown, or numerical anomaly does not say which host or communication path to replace or which checkpoint to restore. A wrong decision can restart the job on the same faulty resource, quarantine a healthy one, or restore corrupted state [3, 5]. Minder [3] reports that manually locating a faulty machine took more than half an hour on average and sometimes days. We use detection for deciding that the job is unhealthy and localization for identifying the component on which recovery should act. Localization lies on the critical path from the initial alarm to resumed, useful training. Synchronous execution makes localization difficult by collapsing rank-local causes into shared symptoms. If one rank never enters a collective, every rank that did enter waits, and a healthy rank may be the first to report the timeout [4, 26]. A synchronous iteration finishes only after its required peers finish, so its duration reflects the slowest dependency rather than identifying whether a delay came from compute, communication, input, or workload imbalance [23, 49, 51]. A reduction such as AllReduce combines tensors contributed by multiple ranks; after one wrong contribution has been incorporated into the result and returned to every rank, the corrupted value no longer identifies its source [19, 28, 55]. Distinct software and hardware faults can consequently produce the same job-wide symptom [3–5, 26], and the first reporter may be a victim. Synchronization therefore leaves multiple causes and candidate ranks consistent with the same alarm, so localization requires additional evidence that distinguishes ranks [3, 4, 26]. Failures are latent when synchronization exposes a job-wide symptom without an authoritative signal that identifies the component on which recovery should act. Although ranks synchronize during training, they remain independent operating-system processes; an out-of-memory kill, uncaught exception, or device error can terminate one rank before its peers [34, 38]. Such fail-stop failures fall outside this category because the launcher or devicehealth interface supplies an authoritative rank or device signal. This paper focuses on three representative latent manifestations. Silent data corruption (SDC) returns an incorrect numerical value without reporting an error; a recurring straggler continues to make progress but does so abnormally slowly; and a hang leaves all visible participants stopped without naming the rank that initiated the stall. These manifestations are not exhaustive; we emphasize them because their symptoms do not identify a culprit and each requires a different form of evidence. Existing mechanisms provide essential detection, observability, and recovery functions, but their evidence does not always identify where to act for the target latent failures. Launchers identify exited processes [38], collective watchdogs identify non-completion [26],

In LLM pre-training, synchronization propagates rank-local stalls, slowdowns, and numerical errors into job-wide symptoms, obscuring their origin. Existing diagnosis often relies on in-process monitors that cannot report after the trainer blocks or terminates, or on post-mortem logs that preserve only synchronized symptoms; offline health tests lose the workload and operating conditions that triggered the failure. We present SCOUT, a unified runtime failure-localization framework built on one design principle: identify outliers through strict-majority consensus among equivalent replicas. SCOUT aligns replica progress, timing, and numerical evidence, then uses its Consensus Collective Communication (C3) abstraction to identify ranks whose compact signatures disagree with their peers. An out-of-band CPU observer remains responsive when training hangs, whereas in-situ replay exercises recurring stragglers and silent data corruption (SDC) beside the live job with its model state, kernels, allocations, communication path, and thermal and memory pressure present. Collective fingerprints expose rank-local protocol divergence. Clean replay coverage certifies checkpoint numerical integrity, preventing recovery from selecting state corrupted by SDC. SCOUT integrates with PyTorch, TorchTitan, Megatron-Core, and DeepSpeed without training-loop or framework-source modifications. SCOUT is open source at https://github.com/LMResiliency/lm-resiliency.

1

Introduction

Modern LLM pre-training is a distributed systems workload spanning tens of thousands of accelerators. MegaScale reports training a 175-billion-parameter model on 12,288 GPUs [14]. Meta reports pre-training the nearly two-trillion-parameter Llama 4 Behemoth with 32,000 GPUs [30]. Because model and optimizer state exceed one accelerator’s memory, training distributes them across worker processes, or ranks. Model and optimizer state are partitioned across ranks with data parallelism [21], tensor parallelism [42], pipeline parallelism [11], expert parallelism [20], and state sharding [40, 54]. These hybrid-parallel jobs are typically synchronous: within every optimization iteration, ranks exchange tensors through collective communication and wait at data or control dependencies before advancing [14, 40, 42]. At this scale, reliability directly determines delivered throughput because one delayed or failed component can idle many healthy accelerators. While a synchronous job waits for diagnosis or rollback, its reserved accelerators continue accruing GPU-hours without completing training steps. An O (10,000)-GPU cluster recorded 100–230 critical failures per week [5], and another large distributedtraining environment observed up to two faults per day for an individual job using thousands of machines [3]. Performance faults are comparably costly: a five-month trace found stragglers in 42.5% of jobs, wasting 10.4% of allocated GPU-hours [23], while a separate ∗ Independent Researcher; wangzhuangowl@gmail.com.

1

device telemetry reports known hardware errors [34], and checkpoint systems restore previously stored state [6, 46]. Tracing, online profiling, metric models, and post-mortem log correlation improve visibility [3, 5, 10, 14, 26, 51], but they do not supply a general localization oracle for latent failures. Instrumentation may depend on framework internals, monitors inside the training process may become unable to report when that process blocks or terminates, and synchronized logs can preserve the shared symptom rather than its origin. Generic offline health tests run a different workload and operating regime; production evidence shows that they can miss defects triggered by particular training inputs and execution units [50, 55]. SDC also complicates recovery: corruption can perturb parameter updates and persist in model state across steps [1, 29], so a byte-consistent checkpoint may already contain corrupted state. The newest checkpoint may reintroduce corruption, whereas conservative rollback discards healthy work. These gaps motivate SCOUT, a runtime Symmetric Consensus OUTlier detection framework for localizing different failure manifestations. Its design follows from three observations about synchronous training. First, failures are rare relative to healthy ranks: an individual failure usually leaves most ranks operating normally [3, 5, 14]. Second, practical LLM pre-training jobs typically use data parallelism to create equivalent replicas that execute the same model computation graph at the same training coordinate [9, 14, 54]. Third, a rank-local failure breaks the behavioral symmetry among these equivalent replicas. The affected rank may schedule collectives in a different order, take longer to execute, or produce numerical values that differ from healthy ranks [26]. These observations suggest a simple design principle: localize failures as outliers through strict-majority consensus among equivalent replicas. SCOUT groups replicas that share a computation graph and aligns their training coordinate, input, and randomness. Within each group, majority behavior provides a live, workload-relative reference for progress, elapsed time, or numerical output. The principle is simple but general: it converts redundancy already present in distributed training into online diagnostic evidence. It requires neither an absolute health threshold nor a preselected golden rank, exercises the production model rather than a displaced health test, and produces an outlier verdict without offline reconstruction from synchronized logs. Turning this principle into continuous runtime diagnosis raises three fundamental systems challenges. First, SCOUT must establish cross-rank comparability before majority voting is meaningful. Progress is meaningful only when ranks report it in the same training coordinate system, timing is meaningful only for equivalent work, and numerical output is comparable only when benign input and randomness differences are removed. Second, SCOUT must keep diagnostic evidence available when the training path itself is blocked or suspect. A hang therefore requires an observer that remains responsive when the trainer blocks, whereas straggler and SDC diagnosis must exercise the live accelerator to preserve production execution conditions. Third, SCOUT must minimize the system overhead of continuous diagnosis. Duplicating full iterations or exchanging full tensors would make continuous diagnosis impractical. SCOUT realizes this principle by making replica evidence comparable, collecting it through failure-appropriate paths, and limiting

the work performed for diagnosis. For comparability, SCOUT aligns training coordinates and inputs, executes equivalent work, and encodes each manifestation in a dedicated evidence schema. Hang schemas use ordered collective fingerprints, straggler schemas use elapsed time under equivalent work, and SDC schemas use deterministic numerical signatures. The Consensus Collective Communication (C3) abstraction completes this path by evaluating the resulting payloads against the strict majority while preserving which ranks disagree. For evidence availability and fidelity, SCOUT combines an out-of-band CPU observer with in-situ replay. An outof-band CPU process remains available to compare progress after the training group hangs, while in-situ replay exercises recurring stragglers and SDC beside the live job so that model state, kernels, allocations, communication, and operating pressure remain present. For low overhead, SCOUT combines coverage-based replay scheduling, layer sampling, configurable cadence, rotating coverage, and compact-signature exchange. Coverage-based scheduling selects representative shapes for recurring execution paths in dynamic MoE workloads instead of continuously replaying every observed shape. These controls avoid duplicated full iterations and full-tensor exchange while accumulating diagnostic coverage over time. Together, these mechanisms let SCOUT localize hangs, stragglers, and SDC online. For a hang, SCOUT reports a rank when its collective sequence or metadata differs from the majority. For a straggler, SCOUT identifies a rank that remains slower than equivalent peers under controlled replay. For SDC, SCOUT reports the divergent rank and replay surface whose numerical signature disagrees with the healthy majority. SCOUT also uses replay verdicts to certify model checkpoints, allowing failure recovery to use in-memory checkpoints when numerically trusted and otherwise fall back to the latest verified checkpoint. These localization and recovery verdicts are produced online without a separate offline log-analysis step. SCOUT supports PyTorch [37], Megatron-Core [36], TorchTitan [22], and DeepSpeed [40] through public interfaces. It modifies neither training loops nor framework source. SCOUT also integrates checkpoint certification with GEMINI [46]. These choices make SCOUT an additive service, not a second training runtime. In summary, this paper makes the following contributions: • We introduce the design principle of identifying outliers through strict-majority consensus among equivalent replicas to localize latent training failures online. • We design C3, an out-of-band hang observer, and in-situ replay to turn this principle into actionable evidence for hangs, stragglers, and SDC. • We use clean replay coverage to certify recovery checkpoints, excluding candidates whose numerical integrity remains unverified. • We implement SCOUT across training frameworks and integrate it with GEMINI.

2 Background 2.1 Resilient LLM pre-training LLM pre-training iteratively optimizes model parameters over massive token corpora, and a complete run can occupy a cluster for months [9, 53]. Each iteration computes forward and backward 2

Hangs obscure which rank stopped useful progress first. If one rank blocks before entering a collective, every peer that has entered the collective waits, and a healthy peer may be the first to report a timeout. A rank that issues a different collective sequence or incompatible metadata can produce the same visible stall as a blocked host thread, device kernel, or communication endpoint [4, 26]. The timeout identifies a collective that did not complete, but it does not by itself identify the initiating rank or distinguish protocol divergence from a failure along an otherwise compatible execution path. Stragglers obscure which rank is intrinsically slow. Because a synchronized dependency completes no faster than its slowest participant, one delayed rank increases the observed iteration time of all ranks that wait for it. End-to-end iteration time therefore detects lost throughput but cannot separate slow computation or communication from time spent waiting on another rank. Raw perrank durations are also ambiguous because legitimate input, routing, and workload differences can change the amount of work assigned to a rank. Production studies accordingly report heterogeneous causes, including compute degradation, communication delay, input stalls, and workload imbalance [23, 49, 51]. Localization requires evidence that a rank remains slower than peers while they perform equivalent work. SDC obscures both the source of corruption and the safe recovery point. An SDC event produces a wrong but valid numerical value without raising an exception or device error. When a collective incorporates one corrupted contribution, its result can distribute the error to healthy ranks and erase the identity of the source [19, 28, 55]. Subsequent optimizer updates can persist the error in model state, allowing a successfully written checkpoint to contain numerically corrupted parameters [1, 29]. Loss curves or activation statistics may reveal that execution is anomalous, but evidence collected after propagation need not identify the rank that introduced the error or the newest checkpoint preceding it. Safe recovery therefore requires both source localization and positive evidence that the selected checkpoint remains clean. Production traces quantify both prevalence and cost: stragglers affected 42.5% of jobs and wasted 10.4% of allocated GPU-hours, while manual faulty-machine localization took over 30 minutes on average and sometimes days [3, 23]. For latent failures, localization therefore remains a prerequisite for recovery because no component-level signal supplies an action. Checkpoint and restart mechanisms cannot select a resource or clean recovery point until diagnostic evidence narrows the culprit. Hangs, stragglers, and SDC require rank-distinguishing evidence tailored to the information that each symptom obscures.

passes, aggregates gradients, and updates the model and optimizer state. Different parallelism dimensions synchronize through process-group communication: data and tensor parallelism use reduction and gather collectives, context parallelism exchanges attention state, expert parallelism routes tokens with AllToAll, and pipeline parallelism transfers activations and gradients between stages [11, 20, 21, 36, 42, 54]. Because each communication dependency waits for all required participants, an iteration advances no faster than its slowest required rank and stops if any required rank stops participating [26]. At training scale, failures are routine: production studies report software exceptions, host and accelerator faults, network failures, and performance degradation [3, 5, 14, 23]. These events fall into two natural categories based on the required recovery action: software failures such as process crashes or input-pipeline exceptions can often be cleared by restarting processes, whereas hardware faults require the affected resource to be replaced, repaired, or excluded [46]. Either class can stall one rank and thereby idle every rank that depends on it. Resilient pre-training organizes recovery into three interdependent stages: checkpointing, diagnosis, and restart. Checkpointing periodically captures a globally consistent recovery point containing the model, optimizer, and other state required to resume training [6]. Diagnosis detects that execution is unhealthy and localizes the rank, host, accelerator, or communication path on which recovery should act. Restart applies that action, reconstructs the distributed job, and restores all ranks from the latest checkpoint known to be complete and clean [38]. High-frequency in-memory checkpointing shifts the recovery bottleneck toward diagnosis. When a rank stops participating, its peers cannot commit the current iteration, and the job must resume from a mutually consistent model and optimizer state. Frequent memory-resident checkpoints reduce checkpoint creation and retrieval time while bounding the computation lost to rollback [46]. Diagnosis remains unavoidable: until it identifies a remediation target and a clean recovery point, the accelerator reservation cannot return to useful work. This paper therefore focuses on reducing this diagnostic interval.

2.2

Latent failures and their localizations

Many failures are easy to detect and localize because they produce an explicit component-level error. For example, an out-of-memory kill or uncaught exception terminates a particular process, a scheduler revocation names a host, and device-health telemetry reports a specific accelerator fault [34, 38]. The launcher or platform monitor can map such a signal directly to the resource that recovery should restart or replace. A failure is latent when the runtime observes an unhealthy job but receives no authoritative component-level signal for recovery. Synchronization can propagate a rank-local effect into a shared symptom, making the same underlying fault explicit when reported and latent when only its effect is visible. This paper focuses on three representative latent manifestations: hangs, stragglers, and silent data corruption (SDC), each of which obscures different information required for recovery.

3

SCOUT Overview

This section reframes latent failures as behavioral outliers and derives SCOUT from that perspective. It validates the conditions required for consensus detection and then presents SCOUT’s design principle and architecture.

3.1

Latent Failures Are Outliers

Existing diagnosis systems demonstrate the value of relative evidence for distributed training. Minder [3] identifies failure-related 3

Manifestation Hang Straggler SDC

Spatial behavioral outlier One rank reports different progress or collective intent after the group stalls. One rank is persistently slower than its peers on controlled equivalent work. One rank produces a different deterministic value for the same computation.

Model Megatron 1T [32] MegaScale 175B [14] GLM-130B [52] Llama 3 405B [9] Skywork-MoE [48] MegaScale-MoE [15]

GPUs 3,072 12,288 768 16,384 1,536 1,440

Parallelism TP8, PP64, DP6 TP8, PP8, DP192 TP4, PP8, DP24 TP8, PP16, DP128 PP12, EDP4, DP32 PP15, EP8, DP12

Peer degree 6 192 24 128 32 12

Table 1: Spatial behavioral outliers for latent failures.

Table 2: Published dense and MoE configurations with equivalent peer groups.

metric patterns, Holmes [51] compares irregularities across workers, and EROICA [10] correlates differential observations across the stack. Their localization accuracy, however, depends on a predefined set of telemetry signals and fault-specific analysis rules. A fault is attributable only when it creates a rank-specific pattern in exported metrics, logs, or traces; when synchronization makes healthy and faulty ranks expose the same stall or slowdown, these systems may observe the job-wide symptom without identifying its source. SCOUT instead defines the localization target independently of any particular telemetry source. A latent failure makes a rank a behavioral outlier when its progress, execution time, or numerical result departs from what it would have produced without the fault under the same work and operating conditions [4, 19, 51], as listed in Table 1. Because this counterfactual is unobservable, localization requires an observable reference with the same expected behavior as the rank being evaluated. Such a reference can be temporal, using the same rank’s history, or spatial, using concurrent ranks. Temporal comparison can in principle expose the three latent failure manifestations by detecting a stalled progress signal, a latency increase, or a changed result when the same computation recurs. In LLM training, however, expected progress, latency, and numerical output vary across layers, inputs, model states, and MoE routing decisions. Consequently, a temporal detector must model these phases and set phase-specific decision boundaries to avoid confusing legitimate variation with a failure. Temporal history is therefore useful for corroborating trends and shared slowdowns, but it is an unstable primary reference for a phase-varying workload. SCOUT instead uses spatial comparison as its primary localization mechanism. Concurrent peers performing equivalent work provide the reference; when a strict majority agrees, their result estimates the evaluated rank’s fault-free behavior. Because the comparison is concurrent, it does not rely on measurements collected under an earlier model state, input regime, or cluster load. Spatial comparison is valid only among ranks with the same expected behavior. For progress comparison, peers occupy the same tensor-parallel shard, pipeline stage, context-parallel shard, and expert assignment while belonging to different data-parallel or FSDP instances. For timing and numerical replay, SCOUT additionally aligns model state, input, randomness, and execution conditions. SCOUT’s spatial comparison requires three conditions: a failure creates a rank-local divergence in a compared behavior, faulty observations remain a minority within a peer group, and training provides equivalent peers. Table 1 establishes the first condition:

hangs, stragglers, and SDC create progress, timing, and numerical outliers, respectively. Production reports indicate that the faulty ranks implicated by an individual incident typically form a small minority, even when synchronization spreads the symptom across the job. ByteRobust [45] reports that large-scale training failures normally occur independently on individual nodes, that simultaneous multi-node failures are extremely rare, and that only one or two nodes are typically faulty in a 9,600-GPU job. MegaScale [14] likewise documents how one faulty GPU worker can cascade into a cluster-wide NCCL stall. SCOUT encodes this operational pattern as a strict-majority requirement instead of assuming exactly one faulty rank. Hybrid parallelism routinely leaves equivalent peers in both dense and MoE pre-training, as listed in Table 2. Tensor, pipeline, context, and expert parallelism partition model work, whereas data parallelism repeats the resulting partitions. The non-data-parallel coordinates of a rank are its tensor shard, pipeline stage, context shard, and assigned expert set. Ranks that match at each of these positions but differ along the data-parallel or FSDP dimension execute corresponding model partitions and therefore form candidate peer groups. Recent frontier MoE systems reinforce this pattern: Kimi K2.5 [16] uses ZeRO-1 data parallelism, GLM-5 [8] shards gradients across data-parallel ranks, and DeepSeek-V4 [2] synchronizes MoE gradients across data-parallel ranks, although their reports do not disclose a fixed pre-training DP degree.

3.2

Architecture Overview

The behavioral-outlier formulation yields one design principle: localize failures as outliers through strict-majority consensus among equivalent replicas. Within an aligned peer group, the majority supplies a live reference for expected progress, execution time, or deterministic numerical output, and minority behavior identifies localization candidates. SCOUT needs neither an absolute health threshold nor a preselected golden rank. SCOUT organizes runtime diagnosis into three layers, as shown in Figure 1. The integration layer connects SCOUT to each framework and determines which ranks perform equivalent work. The evidence layer keeps failure-specific observations available under the conditions that produce them. The decision layer compares them and produces localization and checkpoint decisions. Integration layer. Strict-majority consensus is meaningful only among ranks expected to perform equivalent work, so this layer normalizes framework structure and constructs peer groups. A framework adapter uses public module-hook, optimizer, processgroup, and checkpoint interfaces to expose the model architecture, 4

PyTorch

Megatron-Core

TorchTitan

DeepSpeed

Integration layer Framework adapters

public module, optimizer, process-group, and checkpoint interfaces

Topology manager (Sec. 4)

parallel coordinates → equivalent peer groups

Evidence layer In-situ layer replay (Sec. 6) captured layer → live replay numerical signatures and execution time

Trainer-independent OOB observer (Sec. 7) shared progress → CPU comparison progress and collective fingerprints

Decision layer Consensus Collective Communication (Sec. 5) exact majority and robust timing outliers

Evidence record (Sec. 5) healthy, rank outlier, or group stall

Checkpoint gate (Sec. 8) clean numerical replay required

External recovery policy continue, restart, quarantine, or diagnose

Figure 1: SCOUT separates runtime failure localization into three layers. The integration layer establishes equivalent peers across frameworks, the evidence layer preserves numerical, timing, and hang observations, and the decision layer turns peer comparisons into localization and checkpoint decisions. C3 returns healthy when scheduled comparisons agree and an evidence record naming the disagreeing ranks when a strict majority identifies an outlier. For hangs, divergent progress or collective fingerprints expose a visible software or control-flow mismatch, whereas agreement after a progress timeout classifies the incident as an equal-progress runtime or transport stall for further hardware or fabric diagnosis. An external recovery policy takes no action on a healthy result, restarts the job in place for a localized software or control-flow failure, and replaces or quarantines a resource for a confirmed hardware failure. An SDC that reaches a parameter or optimizer update becomes part of subsequent training state, so a newly captured checkpoint can preserve the corruption and reintroduce it after restart. SCOUT also uses replay evidence to decide whether recovery can use the latest checkpoint or must fall back to a verified checkpoint (Section 8).

communication groups, optimizer boundary, and checkpoint boundary. The topology manager combines this structure with each rank’s parallel coordinates to group ranks that execute the same model partition at the same training coordinate (Section 4). The application continues to issue its ordinary forward, backward, collective, and optimizer operations without framework-source changes or diagnostic branches in the training loop. Evidence layer. SCOUT uses two observation mechanisms because SDC and straggler diagnosis must preserve live accelerator conditions, whereas hang diagnosis must survive a blocked trainer. SCOUT diagnoses SDC and stragglers through bounded in-situ replay on the live accelerator (Section 6). Replay preserves the training job’s model state, kernels, allocations, communication activity, thermal conditions, and compute and memory pressure while yielding compact numerical signatures and execution times for peer comparison. A hang may block both the trainer and its NCCL communicator, so neither can reliably participate in diagnosis. SCOUT handles hangs with an out-of-band (OOB) CPU observer that reads progress and pending-collective fingerprints published by training hooks through shared memory (Section 7). The observer remains responsive on a control communicator separate from the blocked training communicator and compares peers after a progress timeout. Decision layer. The decision layer distinguishes healthy execution, localized rank divergence, and group-scoped stalls. SCOUT’s Consensus Collective Communication (C3) applies exact majority comparison to progress, collective fingerprints, and numerical signatures, and robust statistical comparison to replay timing (Section 5).

4

Forming Equivalent Peer Groups

SCOUT forms equivalent peer groups by mapping frameworkspecific parallelism into logical mesh addresses. Peers retain the same model-parallel role while differing only along an eligible datareplica or state-shard dimension. LLM training maps ranks onto replica, state-sharding, tensor-, pipeline-, context-, and expert-parallel dimensions [9, 11, 20, 21, 42, 54]. Let R be the set of training ranks, and represent each rank by the logical parallelism-mesh address 𝑟 = (𝑑𝑟 , 𝑠𝑟 , 𝑡𝑟 , 𝑝𝑟 , 𝑐𝑟 , 𝑒𝑟 ). 5

Here, 𝑑𝑟 is the data-replica coordinate; 𝑠𝑟 is the FSDP state-shard coordinate; and 𝑡𝑟 , 𝑝𝑟 , 𝑐𝑟 , and 𝑒𝑟 are the tensor-parallel position, pipeline stage, context-parallel position, and expert partition, respectively. Ordinary data parallelism varies 𝑑 and has a degree-one 𝑠 dimension, whereas pure FSDP varies 𝑠 and has a degree-one 𝑑 dimension. Any other dimension absent from a configuration also has degree one and imposes no constraint. The address is logical rather than framework-native: SCOUT reads it from a PyTorch device mesh [22] or derives it from the process groups configured by Megatron-Core [36] and DeepSpeed [40]. Ranks can be compared only when the coordinates that determine their assigned computation match. Matching 𝑡 selects the same tensor shard, matching 𝑝 selects the same layers, and matching 𝑒 selects the same expert partition. Matching 𝑐 is also necessary because context-parallel ranks hold different sequence shards and participate in position-specific attention communication even though they replicate model parameters. SCOUT therefore compares only ranks at the same context-parallel position. With these model-parallel coordinates fixed, group formation has two cases: SCOUT varies 𝑑 when natural replicas exist and varies 𝑠 when state sharding is the only source of peers. Natural replica peers. For a rank 𝑟 in a job whose data-replica dimension has degree greater than one, SCOUT forms

result, allowing recovery to rule out both ranks. A group of at least three can identify minority outliers whenever healthy observations form a strict majority. For a group of size 𝑁 , rank attribution therefore requires more than 𝑁 /2 healthy members.

5

Consensus Collective Communication

Existing collectives exchange rank-local values, but they do not decide which ranks disagree with their peers. SCOUT instead uses Consensus Collective Communication (C3), a diagnostic collective abstraction that combines a standard AllGather with deterministic consensus to return the disagreeing ranks.

5.1

Evidence and verdict interface

C3 maps one diagnostic object from each rank of an equivalent peer group to a result 𝑅 = (𝑠, 𝐵, 𝐸). Let 𝐺 = (𝑟 0, . . . , 𝑟 𝑁 −1 ) be an ordered peer group, and let 𝑥𝑖 be any supported diagnostic object supplied by 𝑟𝑖 . C3 first converts 𝑥𝑖 to a comparable evidence 𝑒𝑖 (§ 5.2) and an AllGather gives every participant 𝐸 = (𝑒 0, . . . , 𝑒 𝑁 −1 ) before each rank applies the same deterministic comparison rule locally. The result contains a status 𝑠, an 𝑁 -bit outlier bitmap 𝐵, and the gathered evidence 𝐸; 𝐵 [𝑖] = 1 attributes the divergence to peer 𝑟𝑖 . This decentralized evaluation avoids a coordinator and ensures that all responsive peers derive the same result. C3 distinguishes three statuses in 𝑠 so that an all-zero bitmap is not ambiguous. Agree means that the evidence satisfies the selected agreement rule and 𝐵 = 0𝑁 . Attributed means that the rule identifies one or more outliers in 𝐵. Inconclusive means that peers disagree but the evidence does not justify rank attribution, such as an exact comparison without a strict majority; C3 returns 𝐵 = 0𝑁 rather than naming an arbitrary culprit. SCOUT stores the result with the ordered peer-to-global-rank mapping so its failure handler can map bitmap positions to training ranks. C3 deliberately separates comparison from interpretation. The same interface accepts progress coordinates, collective fingerprints, tensors or structured tensor outputs, and execution times. Algorithm 1 lists the common protocol and makes the returned result explicit.

𝐺 rep (𝑟 ) = {𝑞 ∈ R | 𝑠𝑞 = 𝑠𝑟 , 𝑡𝑞 = 𝑡𝑟 , 𝑝𝑞 = 𝑝𝑟 , 𝑐𝑞 = 𝑐𝑟 , 𝑒𝑞 = 𝑒𝑟 }. The unconstrained 𝑑𝑞 coordinate selects corresponding ranks across data-parallel replicas while every state-shard and model-parallel coordinate remains fixed. For example, consider eight global ranks arranged as a four-way data-parallel by two-way tensor-parallel mesh with logical shape (𝑑, 𝑠, 𝑡, 𝑝, 𝑐, 𝑒) = (4, 1, 2, 1, 1, 1). Under row-major placement, the rank at logical position (𝑑, 𝑡) has global rank 2𝑑 + 𝑡. Therefore, the two equivalent peer groups are 𝐺 rep (0) = {0, 2, 4, 6} and 𝐺 rep (1) = {1, 3, 5, 7}. State-shard peers. When the data-replica dimension has degree one, no natural replica group exists, so SCOUT instead forms peers along the FSDP state-shard dimension. FSDP ranks can form an equivalent peer group because they execute the same forward and backward computation after the standard parameter AllGather [54]. For pure FSDP, SCOUT forms

5.2

Producing comparable evidence

C3 can take scalars or tensors as its payload. Small fixed-size values, including progress coordinates, execution times, and collective fingerprints, are supplied as comparable evidence directly. Gathering a large GPU-resident tensor, however, would make diagnostic traffic scale with model-state or activation size. In addition, element-wise comparison among multiple tensors on each GPU can incur costly overheads. For each checked tensor 𝑇𝑖 —such as a layer output, input gradient, parameter gradient, or updated weight—rank 𝑟𝑖 therefore supplies

𝐺 shard (𝑟 ) = {𝑞 ∈ R | 𝑡𝑞 = 𝑡𝑟 , 𝑝𝑞 = 𝑝𝑟 , 𝑐𝑞 = 𝑐𝑟 , 𝑒𝑞 = 𝑒𝑟 }. The unconstrained 𝑠𝑞 coordinate selects the ranks that shard the same model-parallel partition. DP and FSDP expose the same computation graph but differ in what each rank holds before an operation begins. DP ranks already hold corresponding model state, whereas FSDP ranks hold different parameter and optimizer-state shards. SCOUT bridges this difference by synchronizing the captured input and the state used by the selected diagnostic operation, so every state-shard peer executes that operation with the same values. Peer-group size. Peer-group size determines how precisely SCOUT can act on a disagreement. A singleton group provides no comparison. A two-rank group can detect disagreement but cannot identify the faulty rank. SCOUT therefore returns a group-scoped

𝑒𝑖 = 𝐻 𝑤 (dtype(𝑇𝑖 ), shape(𝑇𝑖 ), bytes(𝑇𝑖 )), where 𝐻 𝑤 is a deterministic, position-sensitive hash that produces a 𝑤-bit signature over the tensor metadata and bytes. C3 is independent of signature width, while SCOUT’s current implementation uses 64-bit signatures. SCOUT performs the bulk 6

Algorithm 1: Consensus Collective Communication Input: Diagnostic object 𝑥𝑖 from rank 𝑟𝑖 ; comparison mode mode ∈ {Exact, Statistical}; ordered peer group 𝐺 = (𝑟 0, . . . , 𝑟 𝑁 −1 ) Output: Result 𝑅 = (𝑠, 𝐵, 𝐸) containing a status, an outlier bitmap, and the gathered evidence 1 Function C3(𝑥𝑖 , mode, 𝐺): 2 𝑒𝑖 ← produce comparable evidence from 𝑥𝑖 3 𝐸 ← AllGather(𝐺, 𝑒𝑖 ) 4 if mode = Exact then 5 (𝑒 ∗, 𝑐 ∗ ) ← most frequent value in 𝐸 and its count 6 if 𝑐 ∗ ≤ |𝐺 |/2 then 7 return (Inconclusive, 0𝑁 , 𝐸) 8 9 10 11 12 13

14 15

5.4

𝐵 [𝑖] = 1{|𝜏𝑖 − 𝑚| > 𝜅𝑑 }. The robust scale 𝑑 = RobustScale(𝑇 ) estimates ordinary peer-topeer timing dispersion without being dominated by a small number of extreme observations. The sensitivity multiplier 𝜅 specifies how many units of that dispersion a timing may deviate from the median before C3 marks it as an outlier; a larger 𝜅 requires a more extreme deviation [12]. SCOUT first estimates 𝑑 from the median absolute deviation (MAD), median𝑖 |𝜏𝑖 −𝑚|, and falls back to the interquartile range and then a scaled observed range when an estimate is zero. Consequently, RobustScale(𝑇 ) returns zero only when every timing is identical; Lines 11–12 then report agree because there is no timing divergence to attribute. The median remains a useful spatial reference only while degraded observations remain a minority. Statistical C3 intentionally misses a fleet-wide slowdown because no peer differs from the others, and deviations within 𝜅𝑑 remain unreported. Temporal baselines can detect shared or gradual shifts, but without a concurrent spatial differential they do not add rank attribution.

else (𝑚, 𝑑) ← (median(𝐸), RobustScale(𝐸)) if 𝑑 = 0 then return (Agree, 0𝑁 , 𝐸) // 𝜅 > 0 is the sensitivity multiplier foreach 𝑟 𝑗 ∈ 𝐺 do 𝐵 [ 𝑗] ← 1{|𝑒 𝑗 − 𝑚| > 𝜅𝑑 } if ∃𝑗 : 𝐵 [ 𝑗] = 1 then return (Attributed, 𝐵, 𝐸)

18

return (Agree, 0𝑁 , 𝐸)

6 signature fold on the tensor’s accelerator and transfers only compact intermediate state for finalization, avoiding a full device-tohost copy or a full-tensor exchange. Once each rank has produced comparable evidence, C3 uses exact consensus for deterministic values and statistical consensus for timing values.

5.3

Statistical consensus

Healthy peers can report slightly different execution times, so C3 uses robust statistical consensus for timing evidence. For peer timings 𝑇 = (𝜏0, . . . , 𝜏𝑁 −1 ), C3 computes their median 𝑚 and a robust scale 𝑑, then marks

foreach 𝑟 𝑗 ∈ 𝐺 do 𝐵 [ 𝑗] ← 1{𝑒 𝑗 ≠ 𝑒 ∗ }

17

16

peer agree remains invisible. Section 4 therefore makes the healthymajority assumption and the available attribution scope explicit before C3 runs.

In-Situ Training Replay

A pre-training iteration proceeds through data loading, forward pass, backward pass, and the optimizer update [22]. Periodically, a selected iteration adds checkpoint I/O after the optimizer update [6, 46]. SCOUT observes data-loading and checkpoint-I/O delays at phase boundaries, but replays selected work within other phases because progress alone cannot distinguish ranks. Data loading and checkpoint I/O expose explicit entry and exit boundaries, so SCOUT localizes their hangs and stragglers from progress reported by the out-of-band (OOB) observers in Section 7. At the configured sampling interval, the framework adapter marks when a rank starts and finishes fetching a training batch. On checkpoint iterations, checkpoint hooks similarly mark the start and completion of checkpoint reads and writes. If one rank remains inside either phase after peers advance, C3 localizes that rank as an outlier.

Exact consensus

Exact consensus compares evidence whose healthy values should be identical. Progress coordinates, collective fingerprints, and deterministic numerical signatures all satisfy this requirement. Algorithm 1 lists the exact-consensus procedure. SCOUT first selects the most frequent value 𝑣 ∗ and its count 𝑐 ∗ (Line 4). If 𝑐 ∗ > 𝑁 /2, SCOUT marks every rank whose evidence differs from 𝑣 ∗ (Lines 7–8). The common return logic reports attributed when at least one bit is set or agree when none is set (Lines 15–17). If 𝑐 ∗ ≤ 𝑁 /2, SCOUT returns inconclusive with a zero bitmap (Lines 5–6) because the evidence does not identify a strict-majority reference. This rule handles several distinct minority values without assuming exactly one faulty rank, while refusing to choose a side in a two-rank disagreement. Exact consensus is a sparse-fault attribution rule, not Byzantine agreement [17] or a correctness oracle. It assumes that responsive ranks report one value each and that the healthy value holds a strict majority; it does not establish agreement under arbitrary behavior. A faulty value held by a strict majority is indistinguishable from the expected value, and common-mode corruption that makes every

6.1

Dense-model replay

Progress observation cannot reveal an incorrect result or distinguish slow local work from waiting at synchronization. SCOUT therefore replays selected forward, backward, and optimizer work to produce rank-distinguishing numerical and timing evidence. Fixed local tensor shapes make dense-model replay reusable and bounded. Under a fixed training configuration, tensor shapes remain constant even as their values change. SCOUT builds recipes for the embedding layer, hidden layers, the language-model output layer, and the optimizer update. The dense Transformer models we 7

Algorithm 2: Ordinary FSDP forward and SCOUT replay Input: Module 𝑀 with shard 𝜃 𝑟 ; invocation 𝑥𝑟 ; shard group 𝑆𝑟 ; pre-forward RNG state 𝜌𝑟 ; equivalent peers 𝐺 1 Function FSDP(𝑀, 𝑥𝑟 , 𝑆𝑟 ): 2 𝜃 ← AllGatherParameters(𝑆𝑟 , 𝜃 𝑟 ) 3 𝑦𝑟 ← 𝑀 (𝑥𝑟 , 𝜃 ) 4 return 𝑦𝑟

SCOUT broadcasts the source peer’s pre-forward CPU, accelerator, and framework-specific model-parallel RNG states in 𝜌𝑟 with the input (Line 6), so healthy peers reproduce the same stochastic masks and outputs [35, 39]. After replay, each peer restores its saved state, so diagnosis neither advances the training RNG streams nor requires deterministic training. Input variation. Value variation broadens SDC coverage without changing the fixed tensor shapes. One captured invocation tests only one value range, yet SDC-defective GPUs can fail only for particular training inputs [55]. Periodic capture changes activations, gradients, and parameters as training progresses. Optionally, transforms such as {0.1𝑥, 𝑥, 10𝑥 } widen the numerical ranges of floating-point inputs before Line 6 broadcasts them. Each variant uses the same RNG state across peers, and C3 keeps output and gradient signatures separate rather than concatenating their tensors. A complete layer recipe applies exact C3 to its output and backward gradients, and statistical C3 separately to every timed communication or computation interval. After the real optimizer update, the optimizer recipe repeats the update on a copied slice and compares the results through exact C3, avoiding a copy of the complete model or optimizer. System overhead. Sampling one hidden layer at a configurable cadence makes the per-rank layer-equivalent replay estimate independent of the job’s rank count. For a model with 𝑁 repeated hidden layers, replaying one input variant through one hidden layer’s forward and backward passes costs roughly 1/𝑁 of one training iteration time. If SCOUT replays 𝑉 input variants once every 𝐼 iterations, the amortized layer-equivalent overhead is therefore approximately 𝑉 /(𝐼 𝑁 ). For 𝑉 = 3, 𝐼 = 20, and 𝑁 = 50, this estimate is 3/(20 × 50) = 0.3%.

Function Replay(𝑀, 𝑥𝑟 , 𝑆𝑟 , 𝜌𝑟 , 𝐺): (𝑥, 𝜌) ← BroadcastSource(𝐺, 𝑥𝑟 , 𝜌𝑟 ) 7 𝑅broadcast ← C3((𝑥, 𝜌), Exact, 𝐺) 8 (𝜃, 𝑡 gather ) ← TimedAllGather(𝑆𝑟 , 𝜃 𝑟 ) 9 (𝑦, 𝑡𝑀 ) ← TimedCall(𝑀, Copy(𝑥, 𝜃 )) 10 𝑅coll ← C3(𝜃, Exact, 𝐺) 11 𝑅sdc ← C3(𝑦, Exact, 𝐺) 12 𝑅gather ← C3(𝑡 gather, Statistical, 𝐺) 13 𝑅module ← C3(𝑡𝑀 , Statistical, 𝐺) 14 return (𝑅broadcast, 𝑅coll, 𝑅sdc, 𝑅gather, 𝑅module ) 5

6

target stack hidden layers with the same module architecture and local tensor shapes [9, 53], allowing SCOUT to reuse one replay recipe across them. Each recipe’s replay interval can be configured independently. When a recipe is due, SCOUT executes it between iterations so that replay does not affect the training result. The embedding and language-model output recipes broadcast captured token indices or a captured hidden-state tensor plus the incoming output gradient, respectively, then compare output and gradient signatures. The framework adapter preserves vocabulary and tensor-parallel partitions, so C3 compares only peers with the same model-parallel coordinates. 6.1.1 Hidden-layer replay. Algorithm 2 uses an FSDP-sharded hidden layer to contrast ordinary forward execution with replay. Let 𝑆𝑟 denote the FSDP shard group that materializes rank 𝑟 ’s parameters, and let 𝐺 denote the equivalent peer group whose evidence C3 compares. Replay() preserves normal FSDP parameter materialization but broadcasts the source input and RNG state, computes on copied state, and compares the materialized parameters and module output outside the training result. Two techniques make replay evidence comparable, while input variation broadens coverage. Identical inputs. A module call may pass tensor inputs by position, such as a hidden state, or by name, such as an attention mask. SCOUT broadcasts these tensors from one source rank (Line 6), then applies exact C3 to the received inputs and RNG state (Line 7) to verify that every peer starts replay from identical values. Non-tensor arguments remain local and must already agree. For backward replay, the source broadcasts incoming output gradients, and each peer uses autograd.grad to return diagnostic input and parameter gradients without changing the training states. For optimizer replay, the source broadcasts the copied pre-update parameter slice, effective gradient, optimizer state, and update configuration. The source values need not be correct because spatial comparison requires only identical starting values across peers. Deterministic execution. Comparable replay of stochastic operators requires a common random stream and deterministic kernels.

6.1.2 Separating computation and communication. A single replay duration cannot distinguish slow module computation from time spent in collective communications, yet the two cases require different localization evidence. SCOUT therefore times computation and communication separately. Lines 8–9 in Algorithm 2 separately record 𝑡 gather for parameter AllGather and 𝑡𝑀 for module computation. Lines 10–13 then compare the outputs of both the collective communication and the module computation with exact C3 and statistical C3. An outlier in 𝑡𝑀 localizes the slowdown to module 𝑀 and an outlier in 𝑡 gather instead identifies FSDP parameter communication as the affected surface. Figure 2 shows why the two timings yield different spatial evidence. Figure 2 organizes eight ranks as a 4×2 DP–FSDP mesh, where each row is an FSDP shard PG and each column contains C3-equivalent replicas. In Figure 2(b), only 𝑟 4 has an elevated 𝑡𝑀 , so C3 directly identifies 𝑟 4 as the computation straggler. In Figure 2(c), both members of 𝑆 2 = {𝑟 4, 𝑟 5 } observe an elevated 𝑡 gather because the AllGather synchronizes them. The communication timing therefore identifies the affected PG 𝑆 2 , but not which member caused the slowdown. SCOUT uses cross-PG validation to convert affected PGs into an actionable machine-level diagnosis. Practical LLM pre-training combines multiple parallelism strategies and confines each TP group within one machine because TP communicates heavily [14, 42]. 8

DP dimension 𝑑

(a) Logical 4×2 mesh 𝑟0

𝑟1

(𝑑 0 , 𝑠 0 )

(𝑑 0 , 𝑠 1 )

𝑟2

𝑟3

(𝑑 1 , 𝑠 0 )

(𝑑 1 , 𝑠 1 )

𝑟4

𝑟5

(𝑑 2 , 𝑠 0 )

(𝑑 2 , 𝑠 1 )

𝑟6

𝑟7

(𝑑 3 , 𝑠 0 )

(𝑑 3 , 𝑠 1 )

(b) Module-time outlier

(c) AllGather-time outlier

𝑆0

𝑟0

𝑟1

𝑟0

𝑟1

𝑆1

𝑟2

𝑟3

𝑟2

𝑟3

𝑟4

𝑟4

𝑟5

𝑆2

high 𝑡𝑀

𝑟5

high 𝑡 gather

high 𝑡 gather

𝑆3

𝑟6

𝑟7

𝑟6

𝑟7

Only 𝑟 4 differs across DP-equivalent peers ⇒ computation straggler on 𝑟 4

FSDP dimension 𝑠

𝑆2

Both members of 𝑆 2 report slow AllGather ⇒ affected collective group 𝑆 2 , not an endpoint

Figure 2: Separate module and communication timers identify a computation straggler directly and localize a communication straggler to an affected PG. TP dimension 𝑡

DP dimension 𝑑

C3-equivalent replicas

TP slice 0

𝑟0

𝑟1

𝑟8

𝑟9

𝑟2

𝑟3

𝑟 10

𝑟 11

𝑟4

𝑟5

𝑟 12

𝑟 13

𝑟6

other hybrid-parallel configurations whenever independently observed PGs provide overlapping membership evidence for the same machine.

TP slice 1

𝑟7

𝑟 14

6.2

MoE-layer replay

Compared with a dense Transformer block, an MoE layer adds three stages: a router assigns tokens to experts, an AllToAll exchanges routed tokens across expert-parallel ranks, and expert computation processes the received tokens [15, 20, 31]. For a fixed batch size and sequence configuration, the router still receives a fixed-shape input, so the dense replay rules apply to it. AllToAll communication. SCOUT separates AllToAll diagnosis by failure manifestation. Replay targets SDC and software- or hardware-induced stragglers, while SCOUT’s OOB observers in Section 7 handle collective hangs. Routing can legitimately produce uneven token splits and slow a healthy collective, so workloadinduced imbalance is not itself failure evidence [23, 47]. To separate targeted failures from this workload variation, corresponding EP positions in independent replicas execute the same communication plan. As in Algorithm 2, C3 compares their numerical outputs and durations to expose numerical and timing outliers. GPUs hosting experts also participate in process groups from other parallelism dimensions, such as TP or DP. Cross-PG validation (§ 6.1.2) can therefore intersect slow groups to narrow the fault to a shared machine or communication endpoint. SCOUT also generates representative traffic matrices for AllToAll replay to broaden coverage across dynamic input shapes. Shape-dependent failures in Expert computation. Routing determines the number of tokens processed by each expert, which is one dimension of the expert’s input shape. Changing this token count changes the amount of computation and can change how a grouped kernel schedules it [7, 33]. Because production SDCs can depend on the input data and the affected GPU functional unit, while grouped-kernel latency depends on scheduling, a numerical error or slowdown may appear at one expert-input shape but not another [33, 55]. Replaying only the shape observed in one layer of one iteration can therefore miss a recurring failure that appears under another routed token distribution. Exhaustively replaying every admitted shape online is also impractical because the search

𝑟 15

FSDP dimension 𝑠

Slow FSDP {𝑟 4 , 𝑟 5 } ∩ slow TP {𝑟 4 , 𝑟 12 } = 𝑟 4 ⇒ replace the machine hosting {𝑟 4 , 𝑟 12 }

Figure 3: Cross-PG validation localizes the faulty machine: the slow FSDP and TP groups intersect at 𝑟 4 , so SCOUT selects the machine hosting {𝑟 4, 𝑟 12 } for replacement.

Each rank consequently participates in multiple PGs, allowing abnormal PGs from different parallelism dimensions to identify their common endpoint and its host machine. Figure 3 extends Figure 2’s 4×2 DP–FSDP mesh with TP degree two, yielding 16 ranks arranged in two TP slices. Ranks with the same DP and FSDP coordinates across the slices form an intramachine TP group, while the blue DP-aligned columns remain C3 comparison groups. In the example, the slow FSDP group {𝑟 4, 𝑟 5 } and slow TP group {𝑟 4, 𝑟 12 } intersect only at 𝑟 4 . Because {𝑟 4, 𝑟 12 } occupies one machine, SCOUT localizes that machine for replacement rather than reporting 𝑟 4 as the recovery unit. The joint TP–FSDP latency pattern separates four high-level diagnoses. Slow TP and intersecting FSDP PGs identify a machinelocal problem. A slow TP PG with normal FSDP latency identifies an intra-machine communication problem on the TP group’s machine. Normal TP latency with slow FSDP PGs indicates an intermachine communication problem; SCOUT reports the affected PGs and leaves link- or device-level localization to existing datacenter network-diagnosis systems [41, 43]. Only when many independent PGs spanning multiple machines slow simultaneously and no single machine explains them does SCOUT report a job-level software issue or cluster-wide contention. The same principle applies to 9

recipe cycle. With replay interval 𝐼 , a recipe cycle takes at most 𝐾𝐼 training steps and supplies the evidence required for checkpoint certification in Section 8.

space can contain thousands of shapes, increasing diagnosis latency and runtime overhead. Opportunity. SCOUT considers shape compression only after verifying that candidate shapes select the same kernel and scheduling configuration. A GPU GEMM divides its output matrix into tiles, and a grouped-kernel scheduler assigns those tiles to thread blocks [33]. An execution path is a distinct sequence of kernel operations used to process one scheduled unit of work, such as an interior tile, a boundary tile, or a tail tile. A single input shape can exercise several execution paths and may execute each path many times. Suppose shape 𝛼 executes every qualified path at least as many times as another shape 𝛽 with the same kernel and scheduling configuration. Any persistent fault that appears after a particular number of executions under 𝛽 can also appear under 𝛼 because 𝛼 executes the same execution path at least that many times. SCOUT can therefore retain 𝛼 and omit 𝛽 from the catalog. Shape coverage. SCOUT compresses the search space of shapedependent replay by omitting shapes whose execution paths are already covered by other shapes. The execution fingerprint 𝐹 (𝛼) summarizes shape 𝛼’s kernel and scheduling configuration. The path-count vector c(𝛼) records how many times 𝛼 exercises each execution path in that configuration. When two fingerprints match, corresponding components of their path-count vectors refer to the same execution paths and can be compared directly. Shape 𝛼 covers shape 𝛽 when their fingerprints match and 𝛼’s path-count vector is no smaller in any component: 𝛼⪰𝛽

⇐⇒

𝐹 (𝛼) = 𝐹 (𝛽) ∧ c(𝛼) ≥ c(𝛽)

7

Out-of-Band Hang Localization

Hang localization requires evidence that remains available after the trainer or its collective communications stop making progress. SCOUT therefore separates progress observation from the training process and from its communicators. The mechanism turns a jobwide timeout into either a rank-local progress divergence, a visible collective-protocol mismatch, or a group-scoped runtime stall. Why communication hangs. A collective timeout is a catch-all symptom rather than evidence that the communication library itself failed. PyTorch’s fleet analysis groups the common causes into four categories: (1) CPU-side stalls or control-flow divergence, (2) a preceding GPU-kernel hang, (3) incompatible collective arguments, and (4) network or hardware faults [26]. At the watchdog, these causes appear in two ways: either the collective kernel itself does not finish before the timeout, or ranks have desynchronized and are no longer executing compatible collective operations. The study reports that almost all observed timeouts arise from desynchronization, so the rank that reports the timeout and the collective visible at that instant may both be victims rather than causes. Comparable progress. SCOUT primarily targets hangs caused by desynchronization. Framework hooks publish a monotonic training coordinate consisting of the optimizer step and an operation identifier at repeated-layer, DataLoader, and supported collective boundaries. Immediately before a supported collective, SCOUT also publishes a fingerprint of the process group, collective type, and arguments that healthy callers must share, including fixed shape, dtype, reduction, and root metadata. Legitimately rank-dependent values, such as variable expert-parallel AllToAll splits, require a compatibility rule and are not compared by simple equality. The progress schema records intent before a rank enters the operation that may block it. A lagging rank therefore leaves a different coordinate, while ranks already waiting in the collective retain the later coordinate and pending fingerprint. This ordering avoids relying on the rank that first reports the training-communicator timeout, which may be a healthy victim. An independent observer. Each rank launches a CPU-only observer process that reads the latest progress record from shared memory. Observers communicate through a Gloo group created separately from the training communicator, so a blocked trainer and NCCL communicator do not directly block progress consensus. The training path only updates local shared memory and signals a local event. The observer enters C3 only after visible progress remains silent beyond a configured threshold. This event-triggered design avoids periodic diagnostic collectives during healthy execution while ensuring that the observers of synchronously blocked peers eventually enter the same comparison. Guiding recovery. The OOB classification guides the choice between restarting the job on the same resources and inspecting or replacing hardware before restart. When a minority of ranks publishes a different collective fingerprint, SCOUT attributes the hang to rank-local software or control-flow divergence. Recovery can

componentwise.

For example, if c(𝛼) = (10, 1), and c(𝛽) = (6, 1), then 𝛼 covers 𝛽. For each candidate 𝛼, the set of shapes it can cover is C(𝛼) = {𝛽 | 𝛼 ⪰ 𝛽}. A representative is a retained shape selected to cover part of the admitted set. SCOUT retains the fewest representatives whose coverage sets together contain every admitted shape. If SCOUT cannot determine an execution fingerprint or path-count vector reliably, it disables compression and retains every shape as a representative. Offline discovery. SCOUT builds the representative catalog before training so exhaustive profiling does not consume training resources or lengthen online diagnosis. Offline discovery is valid only for a fixed training environment, so each catalog is bound to the GPU type, software stack, and model configurations that determine kernel selection. Any environment drift invalidates the catalog and requires rediscovery. To build the catalog, SCOUT enumerates the admitted expert-input shapes for each represented layer, expert, and expert-parallel position and runs the production expert implementation. SCOUT records the execution fingerprint and path-count vector of each shape; it then derives the minimum set of representatives that cover the entire search space. Online replay. Online replay rotates through the offline representatives instead of testing every admitted shape at once. At each scheduled check, SCOUT selects the next representative and constructs identical inputs of that shape for equivalent peers. The peers replay the expert computation, and C3 compares their outputs and gradients to localize any disagreement. After a successful check, SCOUT advances to the next representative until the catalog is covered. We call one full rotation through the 𝐾 representatives a 10

Algorithm 3: Checkpoint saving and retrieval Input: Accepted-recipe count 𝑞 = 0; recipe-catalog size 𝐾; latest in-memory checkpoint 𝐶𝐾𝑃𝑇𝑙𝑎𝑡𝑒𝑠𝑡 ; current candidate 𝐶𝐾𝑃𝑇𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 ; newest verified checkpoint 𝐶𝐾𝑃𝑇𝑣𝑒𝑟𝑖 𝑓 𝑖𝑒𝑑 1 Function CheckpointSave(): 2 𝑓 ← job-wide SCOUT outcome 3 if 𝑓 ∈ {SDC, Hang} then 4 return CheckpointRetrieval(f)

then correct the protocol error and restart the job on the same hardware. When all ranks report matching progress and fingerprints, SCOUT instead classifies a group-scoped runtime or transport stall. Recovery then runs device and fabric diagnostics and replaces a resource only if they confirm a hardware fault.

8

Certifying Recovery Checkpoints

A latent SDC can corrupt model or optimizer state and contaminate recent checkpoints before replay exposes it, so restarting from the newest checkpoint can reintroduce corrupted state. A checksum verifies that stored bytes match the captured bytes, but cannot establish that the captured training state was numerically correct. SCOUT couples replay evidence to checkpoint eligibility and resumes training jobs only from certified checkpoints that preserve numerical trust. To reduce recovery latency, SCOUT pairs with GEMINI [46], which asynchronously copies post-optimizer training state from GPU to CPU memory for frequent capture and fast retrieval. Because dense models have static shapes during training, every inmemory checkpoint captured after an accepted SCOUT check is certified by that check. However, MoE routing produces dynamic replay shapes, so one accepted check does not cover the entire recipe catalog. SCOUT therefore persists the latest rank-consistent GEMINI checkpoint as a durable candidate only after a complete cycle supplies accepted evidence for all 𝐾 recipes. The checkpoint protocol for MoE models tracks three checkpoint states: latest, GEMINI’s newest complete inmemory checkpoint; candidate, the persisted checkpoint awaiting complete validation; and verified, the checkpoint whose numerical trust is supported by SCOUT’s replay evidence. Checkpoint saving during training. We define a replay boundary as the point immediately after the optimizer update in an iteration selected by SCOUT’s configured replay cadence. At every replay boundary, SCOUT invokes CheckpointSave() in Algorithm 3; its out-of-band observer enters the same procedure when it detects a communication hang. An SDC reported by any comparison group or a communication hang immediately invokes CheckpointRetrieval(). A healthy result or completed straggler instead permits GEMINI to capture the current state and increments 𝑞, the number of accepted recipes in the current cycle. SCOUT reports the straggler separately because slowness alone is not numerical divergence. When 𝑞 reaches 𝐾, SCOUT resets the counter, promotes the prior candidate to verified, and persists the latest GEMINI checkpoint as the new candidate. One cycle is insufficient because a fault can corrupt training state after its triggering recipe has already run. The checkpoint captured at that cycle boundary can therefore contain corruption that the first cycle did not test. The next cycle replays all 𝐾 recipes after capture, allowing SCOUT to expose such corruption before promotion. Thus, SCOUT promotes a candidate only after two consecutive accepted cycles, comprising 2𝐾 checks. Checkpoint retrieval for failure recovery. The CheckpointRetrieval() procedure in Algorithm 3 uses the failure class, machine accessibility, and, when necessary, the full-catalog replay result to choose between the latest and verified checkpoints. A confirmed SDC selects the verified checkpoint. If a machine is disconnected, SCOUT cannot inspect its latest shard or establish that the latest

5 6 7 8 9 10 11

𝐶𝐾𝑃𝑇𝑙𝑎𝑡𝑒𝑠𝑡 ← newly captured in-memory checkpoint 𝑞 ←𝑞+1 if 𝑞 = 𝐾 then 𝑞←0 𝐶𝐾𝑃𝑇𝑣𝑒𝑟𝑖 𝑓 𝑖𝑒𝑑 ← 𝐶𝐾𝑃𝑇𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 persist 𝐶𝐾𝑃𝑇𝑙𝑎𝑡𝑒𝑠𝑡 𝐶𝐾𝑃𝑇𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 ← 𝐶𝐾𝑃𝑇𝑙𝑎𝑡𝑒𝑠𝑡

Function CheckpointRetrieval(𝑓 ): if 𝑓 = SDC ∨ ¬MachinesAccessible() then 14 return 𝐶𝐾𝑃𝑇𝑣𝑒𝑟𝑖 𝑓 𝑖𝑒𝑑

12

13

15 16

if 𝑓 = Straggler then return 𝐶𝐾𝑃𝑇𝑙𝑎𝑡𝑒𝑠𝑡

19

𝑅 ← ReplayFullCatalog() if HasSDC(𝑅) then return 𝐶𝐾𝑃𝑇𝑣𝑒𝑟𝑖 𝑓 𝑖𝑒𝑑

20

return 𝐶𝐾𝑃𝑇𝑙𝑎𝑡𝑒𝑠𝑡

17 18

job-wide checkpoint is numerically clean, so it also selects the verified checkpoint. A straggler selects the latest GEMINI checkpoint because slowness alone does not indicate numerical corruption. For any other failure, SCOUT runs the complete compressed recipe catalog and selects the latest checkpoint only if the sweep completes without SDC; otherwise, it selects the verified checkpoint. The emergency sweep replays only the selected layers and captured inputs; it neither restores nor re-executes the full model checkpoint. GEMINI retrieves the selected checkpoint for newly added machines that replace faulty ones [46].

9

Implementation

SCOUT is implemented as a Python library over PyTorch [37]. Applications invoke enable_resiliency with the model, optimizer, and optional replay, checkpoint, and parallelism configuration. Framework adapters discover repeated layers, optimizer wrappers, and replica groups, then install public module and optimizer hooks without modifying the training loop or framework source. Adapters support TorchTitan, Megatron-Core [42], and DeepSpeed with ZeRO [40]. Structured replay. Forward and backward hooks retain the sampled layer’s pytree arguments, tensor-leaf gradient outputs, and input requires_grad mask. Dynamic-shape rounds clone captured tensors; other rounds retain detached views only through the optimizer boundary. SCOUT broadcasts tensor leaves and reconstructs 11

the pytree, while adapters must make non-tensor leaves equivalent. Replay temporarily installs the source peer’s CPU, CUDA, and Megatron model-parallel RNG states, scopes PyTorch deterministicalgorithm mode to the replay, and restores local state afterward. SCOUT computes replay gradients with autograd.grad. It never passes live parameter gradients to backward. TorchTitan materializes only the sampled FSDP2 layer; HSDP first compares corresponding local shards, then replays training-shaped ReduceScatter and replica AllReduce with scratch tensors. Pure FSDP omits shardcorrectness checks because no corresponding owner copy exists. Optimizer boundary. For replicated PyTorch optimizers, SCOUT compares sampled weights after the real optimizer.step; DeepSpeed and Megatron adapters hook their underlying PyTorch Optimizer. On a scheduled check, the pre-hook clones a rotating 64K-element slice of the parameter, effective gradient, group configuration, optimizer slots, and optimizer-owned tensors. After the real step runs unchanged, SCOUT applies the same base optimizer to the clone and compares the two updated slices. This surface requires corresponding replicated ownership and remains disabled for uniquely owned sharded updates. Progress path and communication. Each rank launches a separate CPU observer. Training hooks publish step, operation ID, pending-collective fingerprint, and sampled DataLoader timing to a lock-free shared-memory record and signal visible progress. The observers communicate through an independently rendezvoused Gloo group, so a blocked training communicator does not directly block hang localization. Replay uses a DP-aligned Gloo group for scalar and object metadata and a separate NCCL group for tensor broadcast and signature exchange. SCOUT folds contiguous tensor bytes to at most 512 64-bit values on the source device, finalizes one 64-bit signature on the CPU, and AllGathers only signatures; multiple numerical surfaces share one exchange but retain separate outlier bitmaps. Reports preserve peer ranks, per-surface SDC bitmaps, replay timing, layer, mode, shape, progress, collective fingerprints, stall duration, and attribution scope so recovery policy can distinguish rank outliers from group-scoped stalls. Recovery integration. SCOUT’s checkpoint coordinator converts replay results into GEMINI [46] checkpoint operations. Before mutating checkpoint state, it combines missing-result, SDC, and recipe-cycle flags across the job; incomplete evidence skips capture, and disagreement about a cycle boundary raises an error. At a cycle boundary, GEMINI completes any peer replication, verifies through a collective minimum that every checkpoint rank holds the same generation, and writes each local shard together with its received peer replica when explicit replication is enabled. Loading performs the same rank-consistency check before selecting an in-memory or persisted generation. Per-shard status metadata, temporary-file writes, fsync, and atomic replacement preserve the selected recovery role across restart.

10

experiments answer these questions on a focused two-host testbed and dedicated qualification harnesses.

10.1

Methodology and small-testbed setup

Hardware and software. Our main testbed comprises two hosts, each with eight NVIDIA A100-SXM4-40GB GPUs, for 16 training ranks. NVLink connects GPUs within a host, while inter-host NCCL and Gloo traffic uses TCP since the testbed has no RDMA devices. The framework campaign uses PyTorch 2.13.0 with CUDA 13.0, NCCL 2.29.7, TorchTitan 0.2.2, Megatron-Core 0.18.2, and DeepSpeed 0.19.4. Workloads. The experiments train a deterministic three-block Transformer with AdamW under DDP, FSDP2, and a 4×4 HSDP mesh. The broader integration matrix exercises framework-native pipeline, tensor, context, sequence, data, and expert parallelism, including Megatron/Transformer Engine and DeepSpeed MoE layers with routed forward and backward execution.

10.2

Fault-injection and recovery effectiveness

Protocol and metrics. SDC tests run three steps: one with injection disabled, one with a fault at the scheduled replay boundary, and one after disabling the injector again. The injection-free first step must produce no false positive; after the one-step fault is removed, the third step must again produce no SDC report, showing that SCOUT no longer flags the rank. Checkpoint-recovery tests instead run two accepted checks before injecting persistent SDC, while compute-straggler tests require the same rank in two consecutive timing rounds. We record false positives, the reported rank or group, fault class, checkpoint decision, and whether a fresh runtime restores the selected state bitwise. Fault model. Table 3 groups injections by manifestation and experiment purpose. The dense SDC row couples an architecturespecific numerical corruption to checkpoint exclusion and restart, while the dense numerical SDC row applies a broader replay-only perturbation matrix. A compute straggler delays sampled-layer execution, whereas a communication straggler delays a collective. MoE SDC corrupts expert state during routed training, while MoE kernelrole SDC perturbs selected semantic work inside the instrumented kernel. Hang and input-stall cases stop visible progress through a frozen process, incompatible collective metadata, or delayed input fetching. Table 3 reports measured coverage for each modeled injection. The current two-host training-job cells use one deterministic seed and inject rank 9 or 15, so their success fractions cover only the stated injections, not physical failures in general. These software injections validate the mechanism and its stated localization scope, not recall over arbitrary physical failures. The current checkpoint protocol recovers the correct state in the tested fault cells. Across DDP, FSDP2, and HSDP, all nine 16GPU cells for checkpoint recovery, SDC exclusion, and computestraggler localization passed. Each SDC-contaminated boundary was excluded globally, and a fresh runtime restored the preceding model, AdamW, caller-owned, CPU-RNG, and CUDA-RNG state bitwise. An eight-GPU, two-recipe test further made step 2 a candidate, promoted it after the clean check at step 4, injected SDC at

Evaluation

We evaluate SCOUT by asking three questions: (1) Can SCOUT localize different types of latent failures; (2) does its evidence prevent recovery from selecting a contaminated checkpoint; and (3) how much can qualified MoE execution regimes compress replay? The 12

Fault

Injected trigger

Observed coverage

Dense SDC

Rank-9 DDP parameter after backward, FSDP2 replay output, or HSDP gradient shard before the optimizer step Forward-output or parameter perturbation, including 64 near-invisible cases Rank-9 sampled-layer forward delayed by 250 ms

Exact rank in 3/3 cells; contaminated boundary excluded and prior state restored bitwise 344/344 localized; 30/30 fault-free runs stayed clean

Dense numerical SDC Compute straggler Communication straggler MoE SDC Hang and input stall MoE kernel-role SDC

Rank-15 tensor-parallel collective delayed Persistent rank-15 expert-weight corruption at 128- and 512-row routed shapes Frozen progress, divergent operation metadata, or delayed input fetch Every selected semantic work-item occurrence perturbed

Exact rank in 3/3 cells after two confirmations; fault-free replay below 1.6 ms Affected rank-14/15 group localized; no compute-straggler or SDC report; next step clean Exact rank at both shapes after one clean recipe cycle 120/120 primary and 30/30 focused launches passed 5,960/5,960 changed output or gradient; persistent faults localized ranks 7 and 15 on 8 and 16 GPUs

Table 3: Fault-injection coverage with 16-GPU training jobs.

Configuration

Admitted

Selected

Reduction

3,457 2,048 1,024 2,048

16 10 18 19

99.54% 99.51% 98.24% 99.07%

recorded execution-path occurrences across the admitted shapes, and all 5,960 injected selected-role occurrences changed the forward output, input gradient, or weight gradient. These grouped results cover uniform per-expert row counts; arbitrary heterogeneous routing vectors require separately qualified templates.

Single-expert projections 128×128 6144×10752 5120×1536 2048×1024

11

Grouped kernel, uniform 128×128 expert GEMMs 2 experts per kernel 4 experts per kernel 8 experts per kernel 16 experts per kernel

2,048 2,048 2,048 2,048

18 27 42 48

99.12% 98.68% 97.95% 97.66%

Table 4: Coverage-based compression in MoE replay.

step 5, and recovered step 2. These results establish recovery-state correctness for the injected scenarios.

10.3

Discussion and Limitations

Evaluation scale. The current evaluation exercises software fault injection and dense and MoE mechanisms on at most 16 A100 GPUs across two hosts. A production-scale study should additionally quantify detection accuracy, false positives, and time to evidence across multiple racks, hybrid-parallel layouts, and RDMA or multirail fabrics, including cross-process-group endpoint localization. It does not measure end-to-end throughput and resource overhead across replay cadences, or report recovery time and rollback distance. Paired in-situ and offline diagnosis and ablations of majority consensus, temporal confirmation, the independent OOB observer, controlled replay, and catalog compression remain future work. The present results therefore establish mechanism behavior for the stated injections and configurations, while comparative effectiveness, production-scale cost, and end-to-end recovery savings remain to be measured. Replay coverage. Replay covers the layers, values, shapes, and communication paths exercised by its rotating checks. SCOUT targets permanent faults and intermittent faults that recur under a similar operating regime; a one-shot fault that does not recur during replay falls outside this coverage contract. MoE catalog compression is qualified for a particular backend, hardware and software environment, and admitted shape domain. The current measurements cover uniform expert row counts, while arbitrary heterogeneous routing vectors require separately qualified templates. For dynamic expert-parallel AllToAll, SCOUT claims compressed coverage only if profiling shows that the admitted traffic matrices map to a bounded set of communication execution paths, physical resources, and load regimes; the present evaluation does not establish that condition. Diagnosis scope. SCOUT localizes an actionable rank, GPU, node, peer group, or conditionally an HCA/NIC endpoint. Identifying

MoE replay coverage and compression

We profile an instrumented BF16 Triton grouped-GEMM on A100 GPUs, executing every admitted expert row count three times and requiring a stable kernel fingerprint. The compressor retains representatives that jointly cover every recorded semantic work-item count within each execution and pressure regime. Single-expert experiments vary the projection dimensions; grouped experiments fix each expert projection at 128×128, assign the same row count to every physically executed expert, and vary the number of experts per kernel. In Table 4, “admitted” counts exhaustively profiled row counts and “selected” counts the compressed replay plans. Table 4 shows that the exhaustive 128×128 campaign compressed 3,457 admitted shapes into 16 representatives, while the other singleexpert projections retained 10–19. For uniform grouped execution with two to 16 experts per kernel, the 2,048-shape catalogs retained 18–48 representatives. The growth reflects new pressure regimes and non-dominated work-item counts as each kernel executes more experts. The grouped campaigns passed 4,757 selected executionpath injections, and host-swapped repetitions reproduced their catalogs. The 16 single-expert representatives covered all 1,134,224 13

Numerical correctness and SDC diagnosis. TrainCheck [13] infers and checks training invariants online, while TrainVerify [27] formally verifies a distributed plan’s equivalence to the model specification. AEGIS [19] separates lightweight corruption sensing from definitive verification; SDCHunter [55] replays the triggering workload to isolate a defective GPU and resume from a verified checkpoint. OpGuard [56] localizes the first bitwise operator mismatch, Dr. DNA [28] checks activation distributions inline, and SuperBench [50] exercises infrastructure with controlled diagnostics. SCOUT derives a live reference by equalizing inputs and randomness across an equivalent peer group and applying strict-majority consensus to numerical signatures. TrainCheck’s semantic invariants and TrainVerify’s plan equivalence validate complementary correctness properties; SCOUT adds controlled peer comparison that preserves rank identity and binds clean replay coverage to recovery-state eligibility. AEGIS [19] and SDCHunter [55] use SDC evidence to support rollback; SCOUT ties completed, clean replay coverage to a candidate, promotes it after all required ranks agree, and points recovery to the newest checkpoint supported by that evidence. Across these categories, SCOUT connects progress, timing, and numerical evidence through one peer-consensus interface. Its evidence complements recovery orchestration, collective-internal visibility, and production SDC diagnosis at different layers of the LLM training stack.

a lower-level cause, such as a kernel instruction, cable, port, or switch, requires external component telemetry or fabric diagnostics. SCOUT’s temporal reference detects equal-progress hangs through a progress timeout and peer-group-wide slowdowns relative to clean history, although either symptom requires additional evidence for source attribution. Moreover, public framework interfaces do not expose every collective launched inside FSDP, DTensor, fused kernels, or compiled graphs. Version-specific launchboundary adapters can extend this visibility; without one, SCOUT preserves a group-scoped diagnosis, and the current implementation does not automatically time every parallel process group.

12

Related Work

End-to-end training resilience. ByteRobust [45] combines lifecycle monitoring, fault isolation, replay, warm standbys, hot updates, and high-frequency checkpointing to sustain production LLM training. TrainMover [18] instead prepares elastic or standby machines through delta-based communication-group setup and sandboxed warmup, reducing downtime when an interrupted job changes membership. Check-N-Run [6], GEMINI [46], and Bamboo [44] provide complementary checkpointing and fault-tolerance mechanisms for restoring execution after known failures. SCOUT supplies culprit and checkpoint-eligibility evidence for latent failures, which complements these systems’ resource replacement, reconfiguration, and state-loading mechanisms. Hangs and stragglers. Mycroft [4] traces internal collectivecommunication states and reconstructs control and data dependencies to localize failures and stalls that coarse framework traces hide. The OSDI straggler study [23] uses counterfactual what-if analysis over production traces to quantify straggler impact and investigate temporal, spatial, and causal patterns. Minder [3] and Aegis [5] use job-level telemetry and diagnostic rules to detect faulty machines. Holmes [51], GREYHOUND [49], and EROICA [10] use cross-layer performance evidence to localize training slowdowns. SCOUT complements historical and collective-internal diagnosis with a controlled rank differential: its OOB observers compare progress and collective fingerprints after a trainer blocks, while in-situ replay compares equivalent computation and communication among live peers. Infrastructure and component diagnosis. NetBouncer [43] actively probes datacenter paths to localize failed devices and links, while passive network diagnosis combines path information with end-host statistics to localize partial network failures [41]. For accelerators, DCGM exposes GPU health telemetry and active checks [34], while SuperBench runs controlled validation workloads across AI infrastructure [50]. For CPUs and host memory, Linux RAS exposes machine-check and EDAC reports for processor, cache, and memory errors [25]. Linux In-Field Scan additionally runs circuit-level tests on CPU cores for defects that parity and ECC do not detect [24]. These mechanisms identify a replaceable component when a counter, probe, or diagnostic workload exposes the fault. SCOUT addresses complementary latent cases whose training symptom lacks such an authoritative component signal; its rank- or peer-group verdict narrows the endpoint that infrastructure tools can probe before remediation.

13

Conclusion

SCOUT makes runtime diagnosis additive to distributed training. Its OOB service remains available when a trainer blocks, while in-situ replay preserves the production model, kernels, allocations, and operating pressure. Equivalent peers turn progress, timing, and numerical outputs into evidence for distinguishing protocol divergence, group-scoped stalls, and rank outliers; topology and majority evidence bound attribution. Clean replay coverage also certifies the checkpoint eligible for recovery. Together, graceful attachment, production-condition replay, conservative localization, and checkpoint certification form SCOUT’s central design.

References [1] Anton Altenbernd, Philipp Wiesner, and Odej Kao. 2026. Exploring Silent Data Corruption as a Reliability Challenge in LLM Training. arXiv:2604.00726 [2] DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348 [3] Yangtao Deng, Xiang Shi, Zhuo Jiang, Xingjian Zhang, Lei Zhang, Zhang Zhang, Bo Li, Zuquan Song, Hang Zhu, Gaohong Liu, Fuliang Li, Shuguang Wang, Haibin Lin, Jianxi Ye, and Minlan Yu. 2025. Minder: Faulty Machine Detection for LargeScale Distributed Model Training. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, Philadelphia, PA, 505–521. [4] Yangtao Deng, Lei Zhang, Qinlong Wang, Xiaoyun Zhi, Xinlei Zhang, Zhuo Jiang, Haohan Xu, Lei Wang, Zuquan Song, Gaohong Liu, Yang Bai, Shuguang Wang, Wencong Xiao, Jianxi Ye, Minlan Yu, and Hong Xu. 2025. Mycroft: Tracing Dependencies in Collective Communication Towards Reliable LLM Training. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. ACM, New York, NY, USA, 254–269. doi:10.1145/3731569.3764848 [5] Jianbo Dong, Kun Qian, Pengcheng Zhang, Zhilong Zheng, Liang Chen, Fei Feng, Yichi Xu, Yikai Zhu, Gang Lu, Xue Li, Zhihui Ren, Zhicheng Wang, Bin Luo, Peng Zhang, Yang Liu, Yanqing Chen, Yu Guan, Weicheng Wang, Chaojie Yang, Yang Zhang, Man Yuan, Hanyu Zhao, Yong Li, Zihan Zhao, Shan Li, Xianlong Zeng, Zhiping Yao, Binzhang Fu, Ennan Zhai, Wei Lin, Chao Wang, and Dennis Cai. 2025. Evolution of Aegis: Fault Diagnosis for AI Model Training Service in Production. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, Philadelphia, PA, 865–881. 14

[23] Jinkun Lin, Ziheng Jiang, Zuquan Song, Sida Zhao, Menghan Yu, Zhanghan Wang, Chenyuan Wang, Zuocheng Shi, Xiang Shi, Wei Jia, Zherui Liu, Shuguang Wang, Haibin Lin, Xin Liu, Aurojit Panda, and Jinyang Li. 2025. Understanding Stragglers in Large Model Training Using What-if Analysis. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). USENIX Association, Boston, MA, 483–498. [24] Linux Kernel Contributors. 2026. In-Field Scan. Linux Kernel Documentation. https://docs.kernel.org/arch/x86/ifs.html Accessed 2026-08-10. [25] Linux Kernel Contributors. 2026. Reliability, Availability and Serviceability (RAS). Linux Kernel Documentation. https://docs.kernel.org/admin-guide/RAS/main. html Accessed 2026-08-10. [26] Phillip Liu, Uttam Thakore, Junjie Wang, and Justin Yang. 2026. Flight Recorder: A New Lens for Understanding NCCL Watchdog Timeouts. PyTorch Blog. https://pytorch.org/blog/flight-recorder-a-new-lens-for-understandingnccl-watchdog-timeouts/ Accessed 2026-08-03. [27] Yunchi Lu, Youshan Miao, Cheng Tan, Peng Huang, Yi Zhu, Xian Zhang, and Fan Yang. 2025. TrainVerify: Equivalence-Based Verification for Distributed LLM Training. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. ACM, New York, NY, USA, 237–253. doi:10.1145/3731569.3764850 [28] Dongning Ma, Fred Lin, Alban Desmaison, Joel Coburn, Daniel Moore, Sriram Sankar, and Xun Jiao. 2024. Dr. DNA: Combating Silent Data Corruptions in Deep Learning Using Distribution of Neuron Activations. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. ACM, New York, NY, USA, 239–252. doi:10.1145/3620666.3651349 [29] Jeffrey Ma, Hengzhi Pei, Leonard Lausen, and George Karypis. 2025. Understanding Silent Data Corruption in LLM Training. arXiv:2502.12340 [30] Meta AI. 2025. The Llama 4 Herd: The Beginning of a New Era of Natively Multimodal AI Innovation. Meta AI Blog. https://ai.meta.com/blog/llama-4multimodal-intelligence/ Accessed 2026-08-03. [31] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. 2024. OLMoE: Open Mixture-of-Experts Language Models. arXiv:2409.02060 [32] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, St. Louis, MO, Article 58, 15 pages. doi:10.1145/3458817.3476209 [33] NVIDIA. 2026. CUTLASS Grouped Kernel Schedulers. CUTLASS Documentation. https://github.com/NVIDIA/cutlass/blob/main/media/docs/cpp/grouped_ scheduler.md Accessed 2026-08-07. [34] NVIDIA Corporation. 2026. Health Monitoring. NVIDIA Data Center GPU Manager Documentation. https://docs.nvidia.com/datacenter/dcgm/latest/learn/ modules/health-monitoring.html Accessed 2026-08-03. [35] NVIDIA Corporation. 2026. Megatron-Core Tensor-Parallel Random-State API. NVIDIA Documentation. https://docs.nvidia.com/megatron-core/developerguide/latest/apidocs/core/core.tensor_parallel.random.html Accessed 2026-0808. [36] NVIDIA Corporation. 2026. Megatron-Core User Guide. NVIDIA Documentation. https://docs.nvidia.com/megatron-core/developer-guide/latest/ Accessed 202608-04. [37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc., Vancouver, Canada, 8024–8035. [38] PyTorch Contributors. 2026. Elastic Agent. PyTorch Documentation. https: //docs.pytorch.org/docs/stable/elastic/agent.html Accessed 2026-08-03. [39] PyTorch Contributors. 2026. Reproducibility. PyTorch Documentation. https: //docs.pytorch.org/docs/stable/notes/randomness.html Accessed 2026-08-08. [40] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, Atlanta, GA, USA, 1–16. doi:10.1109/SC41405.2020. 00024 [41] Arjun Roy, Hongyi Zeng, Jasmeet Bagga, and Alex C. Snoeren. 2017. Passive Realtime Datacenter Fault Detection and Localization. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, MA, 595–612. https://www.usenix.org/conference/nsdi17/technicalsessions/presentation/roy [42] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053

[6] Assaf Eisenman, Kiran Kumar Matam, Steven Ingram, Dheevatsa Mudigere, Raghuraman Krishnamoorthi, Krishnakumar Nair, Misha Smelyanskiy, and Murali Annavaram. 2022. Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX Association, Renton, WA, 929–943. https://www.usenix.org/conference/nsdi22/presentation/eisenman [7] Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2022. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts. arXiv:2211.15841 [8] GLM-5 Team. 2026. GLM-5: From Vibe Coding to Agentic Engineering. arXiv:2602.15763 [9] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [10] Yu Guan, Zhiyu Yin, Haoyu Chen, Sheng Cheng, Chaojie Yang, Kun Qian, Tianyin Xu, Pengcheng Zhang, Yang Zhang, Hanyu Zhao, Yong Li, Dennis Cai, and Ennan Zhai. 2026. EROICA: Online Performance Troubleshooting for LargeScale Model Training. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, WA, 1113–1130. [11] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism. In Advances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc., Vancouver, Canada, 103–112. [12] Boris Iglewicz and David C. Hoaglin. 1993. How to Detect and Handle Outliers. ASQC Basic References in Quality Control: Statistical Techniques, Vol. 16. ASQC Quality Press, Milwaukee, WI. [13] Yuxuan Jiang, Ziming Zhou, Boyu Xu, Beijie Liu, Runhui Xu, and Peng Huang. 2025. Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). USENIX Association, Boston, MA, 313–329. https://www.usenix.org/conference/osdi25/presentation/jiang [14] Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin, and Xin Liu. 2024. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, Santa Clara, CA, 745–760. [15] Chao Jin, Ziheng Jiang, Zhihao Bai, Zheng Zhong, Juncai Liu, Xiang Li, Ningxin Zheng, Xi Wang, Cong Xie, Qi Huang, Wen Heng, Yiyuan Ma, Wenlei Bao, Size Zheng, Yanghua Peng, Haibin Lin, Xuanzhe Liu, Xin Jin, and Xin Liu. 2026. MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-ofExperts Models in Production. In Proceedings of the 21st European Conference on Computer Systems. ACM, Edinburgh, UK, 17 pages. doi:10.1145/3767295.3769325 [16] Kimi Team. 2026. Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276 [17] Leslie Lamport, Robert Shostak, and Marshall Pease. 1982. The Byzantine Generals Problem. ACM Transactions on Programming Languages and Systems 4, 3 (1982), 382–401. doi:10.1145/357172.357176 [18] ChonLam Lao, Jiaqi Gao, Jiamin Cao, Zhipeng Zhang, Pengcheng Zhang, Jiangfei Duan, Zhilong Zheng, Yu Guan, Yichi Xu, Yong Li, Zhengping Qian, Aditya Akella, Minlan Yu, Ennan Zhai, Dennis Cai, and Jingren Zhou. 2026. TrainMover: An Interruption-Resilient Runtime for ML Training. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association, Seattle, WA, 2047–2064. https://www.usenix.org/conference/osdi26/ presentation/lao [19] Kinman Lei, Liyan Zheng, Xiang Li, Hongmin Chen, Yun Zhang, Gaohong Liu, Zuquan Song, Zixuan Ma, Zhiyu Xue, Minghui Yu, Shuguang Wang, Wencong Xiao, Haibin Lin, Yuyang Jin, Jidong Zhai, Bo Liu, and Xin Liu. 2026. Safeguarding LLM Training at Scale: Online SDC Detection and Insights from 35 Million GPU Hours. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association, Seattle, WA, 1369–1384. [20] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv:2006.16668 [21] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. 2020. PyTorch Distributed: Experiences on Accelerating Data Parallel Training. Proceedings of the VLDB Endowment 13, 12 (2020), 3005–3018. doi:10.14778/ 3415478.3415530 [22] Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, ChienChin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. 2025. TorchTitan: One-Stop PyTorchNative Solution for Production-Ready LLM Pre-Training. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum? id=SFN6Wm7YBI 15

[43] Cheng Tan, Ze Jin, Chuanxiong Guo, Tianrong Zhang, Haitao Wu, Karl Deng, Dongming Bi, and Dong Xiang. 2019. NetBouncer: Active Device and Link Failure Localization in Data Center Networks. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). USENIX Association, Boston, MA, 599–614. https://www.usenix.org/conference/nsdi19/presentation/tan [44] John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. 2023. Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 17 pages. [45] Borui Wan, Gaohong Liu, Zuquan Song, Jun Wang, Yun Zhang, Guangming Sheng, Shuguang Wang, Houmin Wei, Chenyuan Wang, Weiqiang Lou, Xi Yang, Mofan Zhang, Kaihua Jiang, Cheng Ren, Xiaoyun Zhi, Menghan Yu, Zhe Nan, Zhuolin Zheng, Baoquan Zhong, Qinlong Wang, Huan Yu, Jinxin Chi, Wang Zhang, Yuhan Li, Zixian Du, Sida Zhao, Yongqiang Zhang, Jingzhe Tang, Zherui Liu, Chuan Wu, Yanghua Peng, Haibin Lin, Wencong Xiao, Xin Liu, and Liang Xiang. 2025. Robust LLM Training Infrastructure at ByteDance. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. ACM, New York, NY, USA, 186–203. doi:10.1145/3731569.3764838 [46] Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xinwei Fu, T. S. Eugene Ng, and Yida Wang. 2023. GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints. In Proceedings of the 29th ACM Symposium on Operating Systems Principles. ACM, New York, NY, USA, 364–381. doi:10.1145/ 3600006.3613145 [47] Cunyang Wei and Abhinav Bhatele. 2026. Skew-aware Adaptive All-to-allv Algorithms for Dynamic Deep Learning Workloads. In Proceedings of the 40th ACM International Conference on Supercomputing. ACM, Belfast, UK, 450–462. doi:10.1145/3797905.3800541 [48] Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lü, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, Xiaokun Wang, Yutuan Ma, Rui Hu, Shuicheng Yan, Han Fang, and Yahui Zhou. 2024. Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models. arXiv:2406.06563 [49] Tianyuan Wu, Wei Wang, Yinghao Yu, Siran Yang, Wenchao Wu, Qinkai Duan, Guodong Yang, Jiamang Wang, Lin Qu, and Liping Zhang. 2025. GREYHOUND: Hunting Fail-Slows in Hybrid-Parallel Training at Scale. In 2025 USENIX Annual Technical Conference (USENIX ATC 25). USENIX Association, Boston, MA, 731– 747. [50] Yifan Xiong, Yuting Jiang, Ziyue Yang, Lei Qu, Guoshuai Zhao, Shuguang Liu, Dong Zhong, Boris Pinzur, Jie Zhang, Yang Wang, Jithin Jose, Hossein Pourreza, Jeff Baxter, Kushal Datta, Prabhat Ram, Luke Melton, Joe Chau, Peng Cheng, Yongqiang Xiong, and Lidong Zhou. 2026. SuperBench: A Proactive Validation System for Improving Reliability of Cloud AI Infrastructure. ACM Transactions on Computer Systems 44, 2 (2026), 1–38. doi:10.1145/3767334 [51] Zhiyi Yao, Pengbo Hu, Congcong Miao, Xuya Jia, Zuning Liang, Yuedong Xu, Chunzhi He, Hao Lu, Mingzhuo Chen, Xiang Li, Zekun He, Yachen Wang, Xianneng Zou, and Junchen Jiang. 2025. Holmes: Localizing Irregularities in LLM Training with Mega-Scale GPU Clusters. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, Philadelphia, PA, 523–540. [52] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. GLM-130B: An Open Bilingual Pre-Trained Model. arXiv:2210.02414 Published at ICLR 2023. [53] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01068 [54] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. Proceedings of the VLDB Endowment 16, 12 (2023), 3848– 3860. doi:10.14778/3611540.3611569 [55] Wenxin Zheng, Wenxiao Wang, Yun Zhang, Mingcong Han, Bin Xu, Jinyu Gu, Xingda Wei, Haibo Chen, Zuquan Song, Gaohong Liu, Yucheng Nie, Zhe Nan, Zhuolin Zheng, Huan Yu, Shuguang Wang, Ziming Zhou, Hang Zhu, Wencong Xiao, and Xin Liu. 2026. SDCs in the Wild: Characterizing and Diagnosing SDC-Defective GPUs in Production LLM Training. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association, Seattle, WA, 1349–1367. [56] Ziming Zhou, Yinjie Zhao, Hang Zhu, Wenxiao Wang, Zhihao Bai, Yun Zhang, Shuguang Wang, Haibin Lin, and Peng Huang. 2026. OpGuard: Bitwise Alignment for Precise and General Debugging of Production LLM Training. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association, Seattle, WA, 1385–1406. https://www.usenix.org/

conference/osdi26/presentation/zhou-ziming

16

Related documents

• Consensus Tree Estimation with False Discovery Control via Partially Ordered Sets#652139
• Consensus Tree Estimation with False Discovery Control via Partially Ordered Sets#650686
• Library Drift: Diagnosing and Fixing a Silent Failure Mode in Self-Evolving LLM Skill Libraries#204848
• Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus#120532
• Ensemble Kalman Inversion as an Inertial Interacting Particle System#1084236