arXiv CSopen access

ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions

ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions Ioanna Vavelidou

Subho S. Banerjee

Eric X. Liu

Mike Fuller

Stanford University

Google LLC

Google LLC

Google LLC

Subhasish Mitra

Caroline Trippel

Stanford University

Stanford University

arXiv:2605.15638v1 [cs.AR] 15 May 2026

Abstract

efforts rely on proprietary, hyperscaler- or vendor-specific tests. While some tests have been open-sourced by hyperscalers and vendors [22, 33], these public suites are generally smaller in scope, and vendor tests especially tend to be structurally distinct from proprietary variants. Thus, published findings from existing fleet studies are difficult to replicate, validate, or improve. Two bodies of work attempt to overcome the limitations of public functional test suites through automated test generation. The first generates tests that expose defects as captured by a specific fault model on a specific hardware implementation [46]. The second employs fuzzing to generate tests that thoroughly exercise a specific hardware implementation [34, 35, 37]. The first approach is limited by the inaccuracy of current fault models for defects [39, 50, 53] and the scalability challenges of fault-model-driven test generation [36, 68]. More problematic, both approaches generally require below-ISA hardware specifications, which are hardware-vendorproprietary and unavailable to end-users including hyperscalers. Thus, they are inapplicable for independent, in-datacenter fleet evaluation. The one exception is SiliFuzz [37], which operates at the ISA level, but generates short tests that have been deemed less effective than proprietary tests [53], a finding we corroborate in §8. The case for arbitrary programs as tests. Exercising hardware thoroughly is essential for surfacing any defect-induced errors that the hardware is vulnerable to. However, without below-ISA specifications, end-users cannot control or measure hardware coverage [37]. Instead, this paper proposes heuristically approximating this goal by deriving functional tests from arbitrary programs. This strategy is well motivated: real datacenter programs expose defects that synthetic tests miss [16, 53], by exercising complex microarchitectural features in ways that are difficult for synthetic tests to imitate and necessary for defects to manifest as errors [11]. The barrier imposed by existing checks. The key to transforming arbitrary programs into functional tests is instrumenting them with checks for defect-induced errors. Looking to recent work that describes or publishes functional test content [11, 14, 15, 22, 33– 35, 37, 51, 82, 83], three error checking methods are used: comparing (i) inputs and outputs of invertible computations (e.g., encryption and decryption) [22, 33], (ii) outputs of the same computation run on different threads/cores [11, 22, 33, 37]; and (iii) the output of a computation to a golden value computed on presumed-healthy hardware [11, 14, 15, 33–35, 37, 51, 82, 83]. Notably, none of these methods compares the architectural outputs of two instances of the same instruction within the same thread, given the same architectural inputs. This design choice seems to reflect a shared implicit, or explicit [51], assumption that defects induce consistent errors: two such instruction instances always produce the same wrong output.

Hyperscaler reports of silent data corruptions (SDCs)—presumed to be caused by silicon manufacturing defects—have motivated the development of functional tests for detecting defective CPUs and their use in hyperscaler fleet studies. Interestingly, all such tests seem to assume that defects induce consistent errors: two instances of the same instruction within the same thread, given the same architectural inputs, always produce the same wrong architectural output. We find that this assumption unnecessarily restricts which programs can serve as tests—biasing which defect-induced errors can manifest and get detected—and limits identification of affected instructions to those impacted by errors that short or targeted tests can reproduce—biasing how errors are characterized. We present ITHICA, which automatically generates functional tests for defect-induced errors from arbitrary programs by inserting intra-thread, instruction-level error checks, primarily leveraging instruction duplication and output comparison. Our key insight, challenging the assumption above, is that the most pernicious defects—those most likely to escape manufacturing testing—cause inconsistent errors: two executions of the same instruction within the same thread, given the same inputs, can produce different architectural outputs depending on the execution context in which they run. By exploiting this insight, ITHICA enables arbitrary programs to serve as tests and identifies affected instructions upon error detections, overcoming both aforementioned limitations of prior functional tests. We use ITHICA to transform industrial hyperscaler test programs (our baseline), datacenter workloads, and common libraries into functional tests, and evaluate them on over 3,000 CPU servers. ITHICA error checks detect 39% more defective servers than native checks within the ITHICA tests derived from our baseline programs, and enable novel findings on defect behavior that challenge conclusions drawn by prior hyperscaler fleet studies.

1

Introduction

Hyperscalers are reporting silent data corruptions (SDCs)—presumed to be caused by silicon manufacturing defects, resulting in permanent (hard) faults—as a critical threat to datacenter reliability [14, 15, 20, 29, 53, 82, 83]. Such SDCs occur when a hardware error (§2.2) causes a system to output an incorrect result without any indication that the error occurred [10]. Hyperscalers are finding that defect-induced SDCs impact a substantial fraction of deployed hardware, roughly one silicon device per thousand [14, 15, 29, 53, 82]. To detect defective hardware, hyperscalers have begun subjecting their server fleets to frequent functional testing. This testing has enabled hyperscalers to publish several fleet-wide studies characterizing the SDC problem [11, 14, 15, 29, 53, 82, 83], but these 1

Ioanna Vavelidou, Subho S. Banerjee, Eric X. Liu, Mike Fuller, Subhasish Mitra, and Caroline Trippel

Such consistent error checks severely limit which programs can serve as tests. Few programs are composed of invertible computations, and most lack well-defined final outputs for golden-value or cross-thread/core checks. Even when such outputs exist, these checks require deterministic execution, making them impractical for programs involving multi-threading, network I/O, or other sources of non-determinism. These challenges are amplified if finer-grained (i.e., more frequent) checking is required in order to improve coverage (e.g., reduce logical error masking) or to achieve instruction localization (i.e., identify which instructions a defect may impact). For this reason, prior work performs instruction localization using short or targeted tests only [11, 51] or by heuristically attributing errors to instructions that frequently occur across failing tests [82, 83]. This Paper: Enabling arbitrary programs as tests with ITHICA. We present ITHICA (Intra-THread Instruction Checking Approach), an automatic approach and tool that generates functional tests from arbitrary programs by instrumenting them with intrathread, instruction-level checks, primarily leveraging instruction duplication and output comparison [30, 42, 43, 58, 59, 69, 72]. These checks eliminate the need for final program outputs and significantly reduce determinism requirements, enabling many more programs to serve as tests. Their instruction granularity makes them less susceptible to error masking and enables instruction localization concurrently with detection. ITHICA’s checking approach is enabled by our key insight: the most pernicious defects—those most likely to escape manufacturing testing into production—cause inconsistent errors: two instances of the same instruction within the same thread, given the same inputs, can produce different architectural outputs. Intuitively, such inconsistencies result from extensive microarchitectural and electrical state in modern processors, which creates a highly diverse execution context for each dynamic instruction instance, modulating how defects affect each instance architecturally. Unlike consistent errors that can be exposed from every execution context given the right inputs, inconsistent errors are sensitive to specific contexts drawn from an overwhelming space of possibilities (§3.1). Notably, inconsistent errors can arise even when a defect affects both instruction instances; for example, on one server we evaluate (§7.4), two identical instructions that strongly appear to interact with the same faulty hardware component produce different wrong outputs (a type of inconsistent error), enabling ITHICA to detect the errors. Our first contribution is ITHICA itself. To our knowledge, it is the first work to validate that defects cause inconsistent errors on real defective hardware and to exploit this phenomenon for defect detection. ITHICA primarily repurposes intra-thread, instruction-level checking techniques from prior work on soft error detection [58, 59, 69] and post-silicon validation [30, 42, 43, 72]. Additionally, it introduces a novel technique that proactively diversifies the execution context of memory instructions, by encouraging them to interact with different levels of the memory hierarchy. Implemented as LLVM compiler passes, ITHICA applies to arbitrary programs across multiple architectures. By relying neither on proprietary specifications nor fault models, it bridges the research gap between state-of-the-art functional tests deployed in datacenters and what is publicly available to researchers. Our second contribution is a large-scale testing campaign that demonstrates ITHICA’s effectiveness. From an industrial fleet of

multiple millions of CPU servers, over 3,000 suspect- and confirmeddefective servers—spanning at least 10 microarchitectures—were identified and subjected to multiple ITHICA test types: derived from representative open-source hyperscaler test programs [22] (our baseline), fleet-representative workloads [24], and common libraries [3, 40, 41, 61, 85]. ITHICA detects 100 defective servers, which we analyze in detail—more than 3× [82] and 5× [51] the number of the two most detailed prior studies. By exploiting inconsistent errors, ITHICA checks within the instrumented baseline programs detect 39% more defective servers than native final-output checks within the same binaries, and 69% more than SiliFuzz [37], the only other functional test generation approach operating at or above the ISA level. ITHICA is the first technique to be directly compared against open-source hyperscaler tests on real defective hardware. Our third contribution is a set of novel findings on defect behavior uniquely enabled by ITHICA’s instruction-level error checking within a variety of programs and the high number of servers these tests detect. Many of these findings challenge conclusions drawn by recent hyperscaler studies: (1) ITHICA’s unique ability to perform instruction localization within arbitrarily long tests enables our discovery that the sequence-driven execution context in which a vulnerable instruction executes is the primary predictor of error manifestation. In contrast, instruction usage stress (i.e., the dynamic frequency of a failing opcode in a test) [82, 83] and culprit inputs are insufficient predictors (§7.4) (2) ITHICA enables empirically demonstrating the extreme difficulty of reproducing errors with short tests (§7.4): among the defective servers we analyze, only one reproduces errors with single-instruction tests; the rest require sequences ranging from modestly longer to full programs. Prior work characterizing error trends based primarily on those reproducible with short tests [51] may bias error characterization toward the few cases for which short sequences are effective. (3) ITHICA’s ability to test a wide variety of instructions in diverse execution contexts enables our finding that the same defect often causes errors across multiple instruction types at markedly different rates (up to six orders of magnitude, §7.5). In one such case, the detecting hyperscaler baseline test is one targeting vector instructions, yet ITHICA also reveals failing non-vector instructions within the same test program (§7.5). This finding cautions against broad claims of hardware localization (i.e., attributing defects to hardware components) with functional tests in general, especially those that target few instruction types [11, 51, 82, 83]. Our work suggests effective SDC testing requires instructionlevel checking within long, diverse programs. The ideal—exercising every instruction in every possible execution context—remains an open challenge. ITHICA takes a meaningful step towards it: by relaxing the consistent-error assumption, ITHICA opens up a vast space of previously unusable programs as tests, each exposing instructions to diverse execution contexts that prior checking methods can not leverage.

2 Background 2.1 Hardware Faults Hardware faults are physical flaws or malfunctions. They can be permanent or transient. 2

ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions

Permanent Faults. Permanent (hard) faults include manufacturing defects, aging-related wear-out, and design bugs. Manufacturing defects are flaws introduced during silicon chip manufacturing, and typically impact different physical regions of each affected chip in different ways. A special category of defects is early-life failures (ELFs), which are characteristic of weak chips that pass pre-deployment testing [53, 70], but induce erroneous behaviors within the first few weeks to months of deployment [9]. Distinct from defects (extrinsic flaws) [70], aging-related faults are caused by intrinsic failure mechanisms (e.g., Bias Temperature Instability (BTI) [67] or Hot Carrier Injection (HCI) [77]), which result from chip wear-out over time. Chip designs incorporate voltage and speed margins to prevent errors from aging faults [4]. Finally, electrical and logic design bugs are flaws in a design; thus, they affect most or all chips in a manufacturing batch. Electrical bugs manifest under specific operating conditions, namely voltage, frequency, and/or temperature [66]. Logic bugs are flaws in a chip’s logical implementation. Transient Faults. Transient faults are malfunctions that occur when energetic particles, such as neutrons from cosmic rays [84] or alpha particles from packaging materials [48], generate electronhole pairs in semiconductor devices. The resulting charge can accumulate at a transistor’s source or diffusion nodes and flip the state of a logic device, a phenomenon referred to as transient (i.e., soft) error. Transient faults are not the result of permanent hardware flaws [55].

2.2

Figure 1: Classification of how hardware erInconsistent Consistent Unresponsive Tradition

stimuli to logic circuits and inspect their outputs, leveraging fault models and test metrics for systematically generating scan tests [25, 28, 47, 49, 73, 76, 79, 81]. Functional testing complements scan testing, but lacks scalably computable, high-quality coverage metrics [36, 39]. Manufacturing testing is highly limited in duration due to cost, making exhaustive testing approaches infeasible [39]. In-Datacenter Testing. Historically, in-datacenter testing concluded with software burn-in testing [14] (different from hardware burn-in [78]), assuming hardware faults would be caught from builtin error detection mechanisms. Upon finding that defect-induced SDCs impact a substantial fraction of servers, datacenter operators have begun subjecting them to periodic functional testing (§1) both in- and out-of-production [14, 37, 82].

3

Motivating ITHICA: The Inconsistent Manifestation of Pernicious Permanent Faults

Our key insight in designing ITHICA is two-fold: (1) defects can cause inconsistent architectural errors, i.e., errors that cause some instruction, executed from two different execution contexts on the same hardware, with identical architectural inputs, to produce different architectural outputs; and (2) such defects are the most pernicious. An execution context denotes the entire microarchitectural (including architectural) and electrical state of a chip. We first derive this insight from a recent theoretical result (§3.1). We then build intuition for how permanent faults can cause inconsistent errors with a pedagogical RTL-level fault injection example in simulation (§3.2).

Symptoms of Hardware Faults

A hardware fault can induce a hardware error, i.e., an incorrect bit stored in a flip-flop. Whether a permanent fault induces a hardware error may be: sequence-dependent, i.e., dependent on the order of circuit stimuli; or timing-dependent, i.e., a subset of sequence-dependent that depends on the speed of stimuli [38]. Timing-dependent faults are dependent on a chip’s electrical state (i.e., voltages and currents at transistor terminals) and thus influenced by external factors like power supply noise, frequency or temperature variations [5, 12]. A hardware error may or may not manifest as architectural error, i.e., an incorrect architectural state (§3.1). If it does not, the hardware fault is said to be architecturally masked [75] or benign [54, 63]. Architectural errors can present with various symptoms, including system crashes, system hangs, or silent data corruption (SDC) [16, 53]. An SDC occurs when an architectural error causes the system to output an incorrect result without any indication that the error occurred [10]. Some architectural errors may not affect the system output at all due to logical masking [54, 56, 63, 75], e.g., if the error is multiplied by zero, or a program’s output does not depend on it.

2.3

rors can manifest as three types of architectural errors [6, 45] (§3.1). ITHICA explicitly detects pernicious inconsistent errors and implicitly detects unresponsive errors.

3.1

Architectural Errors: Formal Classification

Part one of our insight follows from recent work on formal presilicon verification (to detect logic bugs, a type of permanent fault, §2.1), which proves that a hardware error can manifest as one of three types of architectural errors [6, 45]:1 (1) an inconsistent error, defined above; (2) a consistent error, which causes some instruction, executed from every execution context on the same hardware device, to always produce the same wrong output for specific architectural inputs; or (3) an unresponsive error, which causes some instruction, executed from some context, to fail to produce an output within the expected time frame (Fig. 1). Notably, this result holds regardless of how the hardware error arises, e.g., even if it is caused by a fault that is both timing- and sequence-independent. Now, consider such a fault alongside one that is timing- or sequence-dependent, where both faults are capable of inducing the same hardware error (§2.2). That is, the first fault induces the hardware error persistently, while the second does so only under specific conditions (i.e., intermittently). If the error manifests as inconsistent or unresponsive in the first case, it will do so in the second. If the error manifests consistently in the first case, it will newly manifest inconsistently in the second.

Testing Silicon Chips for Defects

The SDC phenomenon recently reported by hyperscalers is generally attributed to defects [14, 15, 29, 82, 83]. Standard and emerging approaches for detecting defective chips are manufacturing and in-datacenter testing, respectively. Manufacturing Testing. Manufacturing testing follows chip fabrication and subjects each chip to scan-based structural tests [17] and (non-scan) functional tests [71]. Scan-based testing uses designfor-testability (DFT) features to apply automatically generated test

1 Inconsistent, consistent, and unresponsive errors map to functional consistency, single-

action correctness, and response bound bugs in prior work [6]. 3

Ioanna Vavelidou, Subho S. Banerjee, Eric X. Liu, Mike Fuller, Subhasish Mitra, and Caroline Trippel

Figure 2: Given an input program (<name>.cpp), ITHICA applies one or more transformations—implemented in this paper for LLVM IR— configured with some BlockSize and Interleaving, and outputs a functional test (<name>-ITHICA).

Part two of our insight is that faults that induce inconsistent errors are harder to detect (more pernicious) than those that induce consistent errors. This is because consistent errors can be exposed by executing instructions with diverse inputs from any execution context, while inconsistent errors require executing instructions in many distinct execution contexts, drawn from an overwhelming space of possibilities. ITHICA is explicitly designed to detect inconsistent errors and can implicitly detect unresponsive errors, though not consistent ones. This tradeoff pays off empirically: ITHICA detects more defective hardware—that has already undergone manufacturing testing— than state-of-the-art tests that do not exploit our insight (§7).

Notably, the microarchitectural simplicity of CVA6 (and the abstractness of RTL) makes such a persistent hardware error more challenging for ITHICA to detect: there is less opportunity to observe a resulting inconsistent architectural error. Any increase in microarchitectural complexity makes inconsistent errors easier to observe, e.g., a processor with multiple execution units of the same type may dispatch the original and duplicate instructions to different units. However, execution unit redundancy is not a requirement: our real hardware results feature a case of an instruction exhibiting inconsistent errors, detected by ITHICA, despite very likely executing on the same faulty execution unit (§7.4).

3.3 3.2

Pedagogical RTL Fault Injection Example

To build intuition for the theory in §3.1, we present an RTL-level fault injection example featuring a timing- and sequence-independent fault—the case where consistent errors are theoretically more prevalent, and thus where ITHICA is least likely to detect the fault. Our fault injection targets the open-source RISC-V CVA6 processor [60], a 64-bit, 6-stage, single-, in-order-issue core with speculation. We inject a single (i.e., one-bit) stuck-at fault [50]—a common fault model used in the literature to study permanent faults—into CVA6’s SystemVerilog RTL. Specifically, we force the rs1_valid_o signal—which informs the next instruction to issue that its first operand (rs1) is valid—permanently high, i.e., causing a persistent hardware error. We then run a simple ITHICA test on the “faulty” design in Verilator simulation [74]. The test is an assembly program that features two multiplications (MUL) using the same architectural inputs, preceded by initialization instructions as shown below: 1 2 3 4 5

Execution Context Diversity

ITHICA exploits our key insight (§3) for defect detection by leveraging natural diversity, where the execution context varies passively between the original instruction and its validation (e.g., duplicate in §3.2) instruction (§4.2), and proactive diversity via the novel MemDiv transformation (§4.1), which deliberately perturbs microarchitectural state, and thus execution context, between original and validation instructions. ITHICA also varies input programs (§5.1) and transformations that turn the input programs into tests (§4.1) to more significantly vary the execution context of both original and validation instructions, thus increasing the likelihood of surfacing defect-induced errors (§3.1).

4

ITHICA: Intra-THread Instruction Checking Approach for Defect Detection

ITHICA2 takes as input a program and configuration options (§4.2) and outputs a functional test, produced by applying one or more of its transformations (§4.1) to the program (Fig. 2). This section describes test generation with ITHICA and provides details on its LLVM implementation (§4.3).

lw a5, %[bval] // Sets value for a5 lw a4, %[aval] // Sets value for a4 addi t0, a5, 0 // Dummy inst mul %[s1], a4, a5 // Original MUL inst mul %[s2], a4, a5 // Duplicate (validation) MUL inst

4.1

ITHICA Program Transformations

Each ITHICA transformation (Table 1) inserts additional program instructions to check for errors in the outputs of specific original program instructions, including: (i) validation instructions, which perform redundant computations; (ii) check instructions, which detect inconsistent errors between original/validation instructions; (iii) error-reporting branches, which redirect control flow to an errorreporting basic block when an inconsistent error is detected; and (iv) novel diversity instructions, which create proactive diversity in microarchitectural state between the execution of original and validation instructions.

We compare the original and duplicate MULs’ outputs once the simulation has run to completion. By forcing rs1_valid_o permanently high, both MULs are forced to perform forwarding of their rs1 operand value, without explicitly waiting for their producer to complete. Getting the value from the register file is not permitted, due to the presence of the in-flight lw producer (line 2) that writes to the same operand. As a result, the first MUL forwards a reset value (zero) for rs1, that was written in the scoreboard when the lw instruction issued. The second MUL forwards the correct value from the lw (line 2), which has completed. The same effect is achieved if the dummy instruction is placed between the two MUL instructions instead.

2We will make our repository public upon publication.

4

ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions Pass

Validation instructions, and diversity instructions when applicable, are inserted for every original instruction within a transformation’s scope. Comparison results within each basic block aggregate into a single error-handling branch. The first three transformations below are adapted from prior work on post-silicon validation for design bugs [30, 42, 43, 72] and soft error detection [58, 59, 69]. As they were not publicly available and described for assembly, we re-implement them for LLVM IR (§4.3). Arith instruments arithmetic (i.e., computational) instructions. Validation instructions duplicate the original ones and their results are checked for mismatches against them. Duplicates preserve original data dependencies, allowing error propagation and detection even when comparisons are inserted less frequently (§4.2). Mem instruments load and store instructions. Loads are duplicated and their outputs are compared against the originals. For stores, validation instructions are loads from the store’s address, and their outputs are compared against the original store’s value operand. Br instruments conditional branches to verify that they resolve to the correct target by recording the intended target address before the branch resolves, and validating it against the actual address taken. Br is more lightweight than prior work [13, 26, 58, 69]: it exclusively targets conditional branches, and focuses on wrongtarget misdirections between two valid targets, which are more likely to manifest silently. Branching to a rogue address is more likely to result in a non-silent architectural error, like a crash (§2.2), which would be detected without functional testing [8]. A limitation of Mem, shared by ITHICA and prior work [42, 43], is that validation loads typically retrieve data: via an L1 cache hit (for checking loads), possibly missing errors in load outputs during L1 misses; or a core-local store buffer (for checking stores), possibly missing errors in store outputs at lower memory hierarchy levels. To address this and increase coverage, ITHICA introduces the novel MemDiv transformation, representing the first use of proactive diversity (§3.3) to check for errors affecting memory instructions. MemDiv extends Mem by introducing diversity instructions, namely mfence (memory fence) and clflush (cache line flush) (though others are also possible). These instructions flush store buffers and caches by virtual address, respectively, encouraging original and validation memory instructions to interact with and exercise defects within different levels of the memory hierarchy. Specifically, mfence encourages retrieval via an L1 cache hit. The subsequent clflush forces data retrieval directly from main memory. Note that parity/ECC might not cover all data movement of a complex memory system, nor be able to detect all errors [52].

4.2

Arith

Mem

MemDiv

Br

Target Instructions Original Instruct. Validation Instruct. Logic, binary, bitwise, vector, unary, aggregate & conversion ops, r0 = op(s0,s1) r1 = op(s0,s1) GEP, icmp, fcmp, select, side-effect-free intrinsics & inline asm [44] Load (non-atomic, v0 = load(a0) v1 = load(a0) non-volatile) Store (non-atomic, store(v0,a0) v1 = load(a0) non-volatile) Load (non-atomic, non-volatile)

v0 = load(a0)

Store (non-atomic, non-volatile)

store(v0,a0)

Conditional branches

br(c0, t0,t1)

Inserted Check

eq(r0,r1)

eq(v0,v1) eq(v0,v1)

v1 = load(a0); eq(v0,v1) && clflush(a0); v2 = load(a0) eq(v0,v2) v1 = load(a0); eq(v0,v1) && mfence(); v2 = load(a0); eq(v0,v2) && clflush(a0); v3 = load(a0) eq(v0, v3) src: e = c0 ? t1:t0; src: None store(e,tmp); t0: v0 = load(tmp); t0: eq(v0, t0) t1: v1 = load(tmp); t1: eq(v1, t1)

Table 1: ITHICA Transformation Passes and Validation Rules Configuring Block Size. ITHICA’s block size controls the insertion frequency of comparison instructions. A block size of 𝑛 indicates that comparisons will be inserted for every 𝑛-th original instruction. Validation instructions are inserted for all relevant original instructions, ensuring that errors impacting validation instructions can propagate through dependency chains to comparison instructions (unless they get logically masked, §2.2).

4.3

ITHICA LLVM Implementation

We implement each of the four ITHICA transformations, as well as their combinations (Arith+Mem, Arith+MemDiv, Arith+MemDiv+Br) as distinct LLVM IR compiler passes. Each pass can be configured with different block size and interleaving values via command-line options. The resulting shared object file (.so) integrates directly into Clang-14’s built-in compiler pipeline. Libraries managed by the build system are automatically compiled with our passes, while those explicitly linked are manually compiled (§5.1). Implementing ITHICA at the LLVM IR level provides portability, as IR passes are target-agnostic and can be integrated into existing build systems without requiring modifications to the compiler. Correct ITHICA instrumentation requires validation instructions to be side-effect free (to preserve program behavior) and that original and validation instructions yield the same output in the absence of errors. All LLVM instructions instrumented by Arith satisfy these properties, and can therefore be safely duplicated. For LLVM intrinsics and inline assembly, also instrumented by Arith, we verify via attributes that they are side-effect free before duplication. Br inserts a validation stack store and load with no side-effects. For Mem and MemDiv, ITHICA excludes atomic/volatile loads and stores due to their memory side-effects. Other instructions excluded for the same reason are alloca, atomicrmw, and cmpxchg [44]. ITHICA also excludes fence, which does not have an architectural effect to check. Notably, even non-atomic/volatile loads and stores can exhibit data races in thread-unsafe multithreaded code. For example, OpenSSL intentionally permits certain data races [62], so ITHICA does not instrument it with Mem and MemDiv in our experiments (§7). Despite these restrictions that limit coverage (i.e., fraction of checked instructions, §8), Arith and Br safely instrument all code, and thread-safe code is safely instrumented by all passes.

ITHICA Configuration Options

ITHICA supports configuring the location of validation instructions and the insertion frequency of comparison instructions per basic block. Configuration options are available for all transformations except Br, which targets at most one instruction per basic block. Configuring Interleaving. ITHICA’s interleaving controls the insertion points of validation/diversity instructions relative to originals. An interleaving of 𝑛 indicates that 𝑛 original instructions will appear consecutively in program order before their corresponding validation/diversity instructions appear in the same order.

5

Ioanna Vavelidou, Subho S. Banerjee, Eric X. Liu, Mike Fuller, Subhasish Mitra, and Caroline Trippel

Metric Description Server Detections # servers with error detections Error Detection Rate (EDR) # runs with error detections / # all runs (%) # error detections / # all runs Error Frequency (EF) Time to Detection (TTD) time from test start to first error detection

D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 𝜇arch u1 u2 u3 u4 u5 u5 u2 u6 u1 u4 u3 u5 u1 u2 CPU Age 19 6 34 16 45 38 81 19 24 11 11 42 10 64

Table 2: DPool characteristics. CPU age is shown in months.

PC Sensitivity (per opcode) # failing program counters (PCs) / # all PCs (%) BB Sensitivity (per opcode) # failing basic blocks (BBs) / # all BBs (%) Input Breadth (per opcode) # unique failing inputs / # all failing inputs (%)

We apply ITHICA as the final IR-level pass, to minimize interference with compiler optimization passes. Despite this, backend compiler optimizations can still eliminate our inserted instructions. To prevent this without modifying compiler internals, we mark duplicated loads and stores as volatile. For arithmetic instructions, we pass their arguments through no-op volatile store-load pairs to achieve the same effect. Future versions of Clang may offer intrinsics to retain side-effect-free instructions more cleanly. ITHICA instrumentation can affect how the program’s original instructions are lowered to assembly. Therefore, to ensure a fair evaluation against our baseline (CC in §5.1), we primarily compare ITHICA checks against final output checks within the same ITHICA-compiled binary, rather than across independently compiled binaries (i.e., before and after ITHICA instrumentation).

5

Table 3: Evaluation Metrics. Each run spans one hour. PCs refer to unique static instruction instances in the program.

libraries, like we do for CC, but rely on FB’s build system (Blaze) to rebuild all relevant libraries (e.g., Abseil) with ITHICA.

5.2

Testing Campaign: ITHICA on Real Hardware

To test our insight (§3) and evaluate ITHICA tests generated from various programs (§5.1), we conduct a large testing campaign (§5.2).

5.1

Two-Pool Evaluation Strategy

We conduct experiments on two groups of CPU servers drawn from a hyperscale fleet of multiple millions of them: the Quarantine Pool (QPool) and the Defective Pool (DPool). The QPool consists of more than 3,000 servers quarantined by the hyperscaler for out-of-production testing due to suspicious behavior. They span at least 10 microarchitectures across two major CPU server vendors. It is a dynamic pool (i.e., new servers are regularly added and removed) containing both suspect- and confirmed-defective servers. Servers are offlined to the QPool via three mechanisms: (1) customer complaints about data corruption; (2) flagging by opportunistically-run hyperscaler- and vendordeveloped screening tests; and (3) forensics-based analysis of hardware and software exceptions (e.g., fail-stop and fail-slow errors including machine check exceptions, kernel or user crashes, invariant fails in user code). The DPool consists of 20 servers, originally part of the larger QPool, that we set aside for dedicated testing with ITHICA. All 20 have failed at least one functional test developed either by the hyperscaler or by the CPU vendor in their datacenter lifetime and exhibit diversity in terms of microarchitecture, age (Table 2), geographic location, and time spent in the QPool before being added to the DPool. Across all our experiments, no functional test (i.e., neither ITHICA nor CC) detects any error on six of them. Thus, we report results for the 14 remaining servers. Benefits of Two-Pool Evaluation. The DPool and QPool serve distinct roles. The DPool provides a controlled environment with confirmed-defective servers, enabling us to evaluate our key insight (§3). Its static nature and our exclusive access to it ensure sufficient runtime to evaluate all ITHICA transformations and configurations: we collect over 2,000 hours of testing time per DPool server, exceeding the allocated time of previous detailed studies by 1-2 orders of magnitude [51, 82, 83]. In the QPool, ITHICA tests integrate into the datacenter operator’s standard out-of-production testing pipeline alongside other industrial tests, enabling large-scale evaluation in a real-world in-datacenter setting.

Instrumenting Diverse Input Programs

We use ITHICA to instrument three input program types: (i) industryrepresentative functional test programs comprising Google’s cpucheck (CC) [22]; (ii) industry-representative workloads comprising Google’s Fleetbench (FB) [1, 24]; and (iii) libraries [3, 40, 41, 61, 85] used by CC and FB. When ITHICA generates a functional test from a program, we append “-”, then “ITHICA” or the name of the applied transformation (e.g., “Arith”) to the program name to get the new test name (e.g., CC-ITHICA or CC-Arith, respectively). cpu-check. CC is a suite of bespoke functional tests with standard final output checks (§1) for in-datacenter testing (§2.3). Test programs include: invertible computations in sequence (e.g., encryptiondecryption, compression-decompression) with input-output comparisons; checksum and hash computations on random, non-deterministic data with cross-core output comparisons; and AVX computations with vector lane output comparisons. Most of our experiments evaluate ITHICA tests generated from CC, since its native checks provide a comparison baseline for ITHICA, and its C++ implementation allows for microarchitecture-independent LLVM compilation (unlike SiliFuzz tests [37]). Fleetbench. FB consists of C++ microbenchmarks, representing hot functions in Google datacenters, including Proto, Swissmap, Libc, Tcmalloc, Hashing, Compression, and Stl-Cord, executed with deterministic data pre-sampled from a production distribution or randomly generated with a fixed seed. We exclude Compression due to its reliance on OpenMP, which we could neither statically compile nor access as a system library on our servers. Libraries. For both CC and FB, we instrument both top-level and library code. For CC, we apply ITHICA to Libc, Libc++, Abseil, OpenSSL, and Zlib. Due to tight coupling of Libc and Libc++ with GCC, we instead apply ITHICA to LLVM’s overlay libraries, Llvmlibc and Llvmlibcpp. For FB, we do not explicitly instrument

6

Experiments Overview

The experimental evaluation in this paper is partitioned into two subsections: testing on the DPool (§6.1) and testing on the QPool (§6.2). Across our experiments, we compare the efficacy of ITHICA tests to CC (our baseline) using four main metrics, defined in the first four rows of Table 3. The results are interpreted in detail in §7. 6

ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions

D1

D2

D3

D4

D5

D6

D7

D8

D9

D10

D11

D12

D13

D14

0 0 37 0 0 0 20 43 39 89 4 3 0 97 70 2 0 93 0 1 1 88 86 7 0 1 0 Arith 100 (9) (0) 0 (0) (1) 0 (3) (3) 2 (4) (4) 11 (0) (0) 0 (0) (0) 0 (91) (66) 3 (0) (0) 0 (0) (0) 0 (0) (0) 1 (1) (0) 3 (2) (0) 0 (0) (2) 12 (1) (0) 2 0 0 0 16 31 10 50 61 1 0 97 3 0 0 0 0 0 0 0 74 26 0 0 1 0 0 3 0 0 0 3 97 0 0 0 0 0 62 12 0 0 0 Mem (0) (15) (15) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (25) (0) (0) (0) 0 0 0 0 0 28 0 0 0 0 100 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 0 0 45 0 1 1 0 3 1 1 0 0 0 MemDiv (0) (0) (0) (2) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) 0 0 0 0 23 0 0 40 0 0 100 0 0 0 0 0 0 0 0 100 0 0 0 3 0 0 0 0 0 0 0 99 0 0 0 0 0 85 2 0 0 0 Br (0) (0) (0) (0) (10) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (39) (0) (0) (0) Arith+ 40 0 0 9 35 25 42 42 0 95 9 5 0 0 0 0 0 0 22 17 58 0 0 3 85 0 5 0 0 0 77 79 8 4 0 0 0 8 14 0 1 1 (9) (9) (8) (8) (9) (9) (13) (9) (1) (0) (0) (0) (3) (0) Mem (2) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (1) Arith+ 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 70 10 57 59 3 0 100 82 100 100 100 0 MemDiv (1) (0) 0 (0) (0) 28 (3) (3) 0 (2) (0) 0 (0) (0) 0 (0) (0) 0 (100) (100) 0 (0) (0) 0 (0) (0) 0 (0) (0) 0 (0) (0) 0 (0) (0) 0 (0) (0) 2 (0) (0) 0 Arith+ 10 0 0 0 0 10 60 10 0 100 76 0 0 0 0 0 0 0 100 100 0 0 0 1 99 0 0 0 0 0 54 58 0 0 0 0 0 0 0 0 0 0 (0) (0) (100) (100) (0) (0) (0) (1) (0) (0) MemDiv+Br (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) 0 0 0 0 0 0 73 15 95 98 2 55 1 100 0 CC (0) (6) 8 (1) 0 (0) 0 (0) 2 (0) 0 (3) 8 (0) 1 (0) 0 (0) 0 (0) 0 (0) 2 (0) 0 (0) 0 Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr Ith Nat Cr

Figure 3: EDR for different CC-ITHICA tests (with block size and interleaving of 1) and CC in DPool (D1–D14). Each triplet reports results for ITHICA (Ith), Native (Nat), and program crashes with no detection (Cr). For Ith and Nat, the subset of executions that crashed after a detection is shown in parentheses. D6* is uniquely detected by Arith for interleaving of 8 (§7.3). Instruction with Error

Original

Validation

Both

% across all DPool runs

31.3%

44.7%

24.0%

Table 4: Decomposition of instructions exhibiting errors in DPool into original, validation (ITHICA-inserted), or both.

block size and interleaving of 1, since this configuration yielded the most detections on the DPool for CC-ITHICA (Fig. 3, §7.1): five libraries used by CC and six FB workloads (§5.1). We evaluate each FB-ITHICA binary for 100 one-hour runs. Figure 4: Server detections across all CC-ITHICA runs in both pools.

6.1

6.2

Testing on the DPool

Testing on the QPool

Due to limited available execution time in the QPool compared to the DPool, we primarily run CC-Arith with block size and interleaving of 1, again due to its success for CC-ITHICA in the DPool. The total execution time accumulated per server varies, ranging from 20 to 100 hours due to the presence of other tests in the pipeline and the dynamic nature of the QPool. We secondarily run CC-Mem, CC-MemDiv and CC-Br with block size and interleaving of 1, reporting only their unique detections compared to CC-Arith due to their shorter runtime (at least 20 hours). The results are summarized in Fig. 4, alongside the aggregate DPool results. Our baseline remains Native checks within the same CC-ITHICA binaries. To ensure reliable analysis, these results exclude servers with less than 20 hours per test. Finally, as in the DPool, we evaluate CC’s five libraries and the six FB workloads transformed with Arith with block size and interleaving of 1 for 20 to 100 hours.

First, to compare CC-ITHICA to CC, we run eight functional tests on each DPool server. Seven are CC-ITHICA tests obtained by applying its four main transformations (§4.1) and three combinations thereof (Arith+Mem, Arith+MemDiv, Arith+MemDiv+Br) on CC, all with block size and interleaving set to 1. The eighth is the original CC (i.e., without ITHICA instrumentation) for comparison. Each test runs 100 times per server (100 runs) each lasting one hour unless interrupted by a crash. Between runs, the server is rebooted. Within an ITHICA test, we distinguish between Native checks (checks in the original program, i.e., CC’s final output checks) and ITHICA checks (inserted by ITHICA). If either check type detects an error before a crash, the detection counts. CC contains only Native checks; CC-ITHICA contains both. Each run executes hundreds of thousands of rounds of the CC-ITHICA program (and more rounds for CC), each with different randomly-generated inputs, yielding hundreds of thousands of inputs per run. Fig. 3 shows EDR (Table 3) for the seven CC-ITHICA tests and CC across all 14 DPool servers (D1–D14). Parenthetical values show the subset of detections followed by crashes; the Cr column shows crashes without any other detection. To compare different ITHICA interleaving and block size configurations (§4.2) for the same program, we conduct two other experiments. First, fixing block size to 1, we sweep interleavings of 1, 2, 4, 8, and max (basic block length). Second, fixing interleaving to 1, we sweep block sizes of 1, 2, 4, 8, and dep, which inserts comparisons at the end of each instruction’s dependency chain. We evaluate each configuration with 100 one-hour runs. Fig. 6 shows the results of these two experiments. Finally, we evaluate ITHICA test content generated from two types of non-test programs, using the Arith transformation, with

7

Results

We present ten key findings derived from the experiments in §6.1 and §6.2, as well as some follow-up experiments for further analysis.

7.1

Defects Manifest as Inconsistent Errors

First, consider Fig. 3, produced from running seven CC-ITHICA tests and CC in the DPool (§6.1). Obs. 1 The figure shows that CC-ITHICA tests collectively detect errors in all 14 DPool servers. Among the four base transformations, Arith is the most effective, detecting 11 out of 14 servers. This result may reflect survivorship bias in the tests used to flag servers for offline testing (§5.2). Br does not detect errors in the DPool, but it flags (non-unique) servers in the QPool (§6.2). Obs. 2 Four servers (D1, D5, D8, D9) are uniquely 7

Ioanna Vavelidou, Subho S. Banerjee, Eric X. Liu, Mike Fuller, Subhasish Mitra, and Caroline Trippel

Ratio ITHICA/Native Geom. Mean Arith. Mean Median Min Max Error Detection Rate (EDR) 1.78 4.50 1.17 0.52 100.00 Error Frequency (EF) 7.51 130.15 3.44 0.02 5396.00 Time to Detection (TTD) 0.68 0.77 0.56 0.33 1.72

Table 5: Comparison of ITHICA and Native detection metrics (EDR, EF, TTD) for 58 servers detected by both methods.

detected by ITHICA checks in CC-ITHICA tests, while missed by Native checks in the same binaries as well as by CC. Next, consider Fig. 4, produced from running four/seven CC-ITHICA tests in the QPool/DPool (§6.1/§6.2). Obs. 3 Of the 49 QPool servers detected by both ITHICA and Native checks in CC-ITHICA tests, 45 were flagged by Arith, 1 uniquely by Mem, and 3 uniquely by MemDiv. Obs. 4 An additional 26 QPool servers were only captured by ITHICA; 25 by Arith and 1 uniquely by MemDiv. Only six servers were detected by Native but not by ITHICA. Two likely explanations are coverage gaps in ITHICA’s implementation and errors that manifest consistently for small interleavings, discussed in §8. Across both pools, CC-ITHICA tests detect 89 total servers (75/14 in the QPool/DPool)—39% more than Native checks in the same binaries. Finally, ITHICA tests derived from FB (§6.1, §6.2) detect 11 additional servers in the QPool (§7.3), resulting in 100 servers (89 + 11) detected by ITHICA across both pools. Notably, inconsistent errors are not limited to cases where only one of the two instruction instances is affected by a defect: ITHICA can detect errors even when both are. Table 4 shows which of the two instructions—original or validation—exhibits an error when ITHICA detects one across DPool runs. Obs. 5 In 24% of cases, both produce incorrect but different outputs. These cases are particularly informative because they constrain the possible explanations for the inconsistency: since both instances are incorrect, both were affected by the same defective hardware component, yet still produced different outputs. In contrast, when only one instruction is incorrect, we cannot determine whether they interacted with the same component or different ones (i.e., one defective and one not).

Figure 5: Distribution of opcodes where ITHICA detects errors, with EF calculated across all CC-ITHICA runs in the DPool. Obs. 7 The data reveals a diverse range of affected instructions across servers, spanning arithmetic, floating-point, vector, and memory instructions. Average error frequencies also vary drastically, from less than one per run to over a thousand. Both observations reflect the fact that defects impact different physical regions of each chip in different ways (§2.1). While ITHICA can detect inconsistent errors regardless of their source, the observed error characteristics in our experiments narrow plausible causes. Obs. 8 Their frequency and repeatability (Fig. 5) make transient faults less plausible; their device-specific (Fig. 5) yet microarchitecture-agnostic (Table 2) nature points away from design bugs; their presence across devices of varying ages (Table 2) challenges aging effects as a sole cause. Taken together, these observations are consistent with defects as the likely cause (§2.1).

Finding 1: Defects, despite being permanent faults, manifest as inconsistent errors in nearly all servers flagged as defective in our testing campaign (Obs. 1–5). For the Fig. 4 experiment, Table 5 compares ITHICA and Native checks on several other metrics (Table 3). Obs. 6 Across 58 servers detected by both ITHICA and Native (49/9 in the QPool/DPool), ITHICA achieves 1.78× EDR improvement, 7.51× EF improvement and 0.68× TTD improvement (1.47× faster).

Finding 3: ITHICA-detected errors in our server pools are most consistent with defects as the underlying cause (Obs. 7, 8). Implications for Chip-Users: Intra-thread, instruction-level checking for inconsistent errors is highly effective for defect detection. This finding contradicts the apparent implicit assumption of existing functional tests [11, 14, 15, 22, 29, 33–35, 37, 51, 82, 83] (§1) that defects produce consistent errors and enables the use of arbitrary programs as tests, removing bias toward those compatible with consistent error checking techniques (§1).

Finding 2: ITHICA checks outperform final output checks used by hyperscaler tests across all evaluated metrics (Obs. 2, 4, 6). Fig. 5 shows the distribution of failing instructions and their average error frequencies across all CC-ITHICA tests run in the DPool with a block size of 1 and an interleaving of 1, except D6, which is uniquely detected at interleaving of 8 (§7.3). A “failing instruction” denotes one that exhibits an incorrect output; it does not imply a particular defective hardware unit, as discussed in §7.5. Cases where no specific instruction is identified (no-instr) can result from an error in an instrumentation instruction itself (e.g., icmp), or a crash before logging completes.

7.2

ITHICA Paves the Way for Online Testing

The permanent nature of defects has a direct practical implication for ITHICA instrumentation. For defect detection (i.e., identifying whether a server is defective), catching any one error per server suffices, unlike soft error protection which requires catching every corruption that affects an application’s output. This distinction enables ITHICA instrumentation to be reduced, trading EDR for lower 8

ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions

Figure 6: Impact of interleaving and block size on EDR for each DPool server (D1 to D14), for CC-Arith. The top row shows the effect of varying interleaving (m=max, length of the basic block), while the bottom row shows the effect of varying block size (d=dep, length of instruction dependency chain). The rightmost panels show the average EDR across all servers. ITHICA Pass Performance Binary Size

Arith (Block Size) (1) (2) (4) (8) (dep) 2.18 1.79 1.64 1.52 1.75 8.47 5.88 4.59 4.11 6.11

Mem

MemDiv

Br

1.17 3.06

53.67 13.12

1.17 1.24

Sensitivity to preceding instruction sequence. First, we examine how preceding instruction sequences influence ITHICA error detections across different instances of the same opcode within the same test. For each opcode-server combination that ITHICA detects an error on, we analyze what fraction of static program counters (PCs) executing the opcode exhibit errors to calculate PC Sensitivity (Table 3). Obs. 10 For most failing opcodes, PC Sensitivity is very low (often <1%, Table 7), revealing that the same opcode behaves correctly in the vast majority of its program locations. The same is true for basic block (BB) Sensitivity. A notable exception is vector double instructions, which exhibit high PC Sensitivity. However, for each of these opcodes, all PCs are contained within a single basic block, and therefore likely execute under similar execution context. Similarly, other opcodes with relatively high PC Sensitivity— namely floating-point and vector integer instructions—also exhibit high BB Sensitivity, with failing PCs concentrated within one of only two basic blocks containing them (Table 7). Next, we show that applying different ITHICA transformations that check at least one common opcode to the same input program varies the preceding instruction sequences of those opcodes, similarly affecting detections. Obs. 11 Per Fig. 3, combined passes (Arith+Mem, Arith+MemDiv, Arith+MemDiv+Br) have the benefit of detecting errors across multiple instruction types (e.g., both arithmetic and memory instructions for D12), but sometimes have lower EDR or entirely miss servers detected by individual passes. This can be attributed to the additional instrumentation instructions changing the instruction sequence compared to individual passes, and thus disrupting the execution context needed for error detection. Finally, instruction sequences preceding checked opcodes vary across input programs, likewise influencing detections. Table 8 shows detected servers across the top-level CC code in CC-ITHICA tests, its library code, and FB. Since CC comprises multiple subtests (§5.1), we separate results per subtest where ITHICA detects errors. For each, we report total server detections and unique server detections compared to other tests or subtests. Among CC subtests, AVX detects the most unique servers. This is unsurprising since it is the only test exercising large (4x, 8x) vector operations. Among the instrumented libraries, Zlib detects the most servers (23 total). Obs. 12 Of the 31 servers missed by Native in CC-ITHICA (Fig. 4), eight are detected exclusively by ITHICA

Personal, non-industrial, server — Intel Xeon Gold 6226R, 2.9GHz, 2 × 16 cores, 2 threads/core, L1: 1MB D+I, L2: 32MB, L3: 44MB, Mem: 500 GB

Table 6: Performance overhead and binary size increase for ITHICA transformations compared to CC. overhead, while maintaining high coverage (number of defective servers detected). Notably, for error characterization, fine-grained instrumentation at every instruction remains necessary (§7.5). We investigate one simple strategy for reduced instrumentation: increasing block size (i.e., decreasing comparison frequency). The results in Fig. 6 show that a block size of 1 performs best on average, likely due to reduced logical error masking. Obs. 9 However, larger block sizes demonstrate comparable effectiveness: all DPool servers with reproducible (i.e., more than one) detections at block size 1 (all except for D10 and D14) are also detected at larger block sizes. This indicates that checking frequency can be relaxed without significantly compromising coverage. The overhead of Arith decreases from 2.18× at block size 1 to 1.52× at block size 8 (Table 6). Finding 4: ITHICA instrumentation and runtime overhead can be reduced at high defect detection coverage (Obs. 9). Implications for Chip-Users: The permanent nature of defects can be exploited for relaxed instrumentation. ITHICA’s fine-grained, flexible checking—easily applied to new programs and tunable in frequency and scope—can support a wide variety of relaxation strategies. Establishing ITHICA’s effectiveness for defects is the first step toward low-overhead, online, in-production testing—important future work.

7.3

Execution Context Drives Defect Detection

ITHICA error detections appear sensitive to two mechanisms for varying execution context. First, the instruction sequence preceding an instance of an opcode determines whether it executes in a context where it can exhibit an inconsistent error. Second, execution context variation—via natural and proactive diversity (§3.3)—between an original instruction with that opcode and its validation instruction determines whether it does. 9

Ioanna Vavelidou, Subho S. Banerjee, Eric X. Liu, Mike Fuller, Subhasish Mitra, and Caroline Trippel

Failing Opcode add int and int lshr int or int Integer trunc int Arithmetic udiv int zext int getelementptr icmp int Comparisons select int fadd x86_fp80 FP Arithm fmul x86_fp80 Vector add <4xint> or <4xint> <4xint> fmul <4xdbl> Vector fneg <4xdbl> <4xdbl> fma <4xdbl> Category

Vector <8xdbl> Memory

Affected # Total # Failing PC Sensitivity # Total # Failing BB Sensitivity # Total # Unique Input Breadth Servers PCs PCs Geom. Mean (Std) BBs BBs Geom. Mean (Std) Failing Inputs Failing Inputs Geom. Mean (Std) D3 1,202 5 0.42% (0.00) 664 4 0.60% (0.00) 22 7 31.82% (0.00) D3 / D12 748 3/1 0.23% (0.13) 634 3/1 0.27% (0.00) 2/1 2/1 100.00% (0.00) D3 424 1 0.24% (0.00) 203 1 0.49% (0.00) 1 1 100.00% (0.00) D3 / D6 180 7/1 1.47% (1.67) 145 7/1 1.82% (2.07) 776 / 9 34 / 2 9.87% (8.92) D3 492 5 1.02% (0.00) 240 5 2.08% (0.00) 6 4 66.67% (0.00) D1 36 1 2.78% (0.00) 25 1 4.00% (0.00) 1 1 100.00% (0.00) D3 809 46 5.69% (0.00) 489 32 6.54% (0.00) 19,708 217 1.10% (0.00) D1 / D3 / D8 5,568 6/1/1 0.03% (0.04) 2,412 6/1/1 0.08% (0.10) 3/1/4 3/1/1 63.00% (35.36) D3 / D12 3,968 12 / 1 0.09% (0.14) 3,552 12/1 0.10% (0.15) 17 / 1 4/1 48.51% (38.24) D3 617 7 1.13% (0.00) 419 7 1.67% (0.00) 84 4 4.76% (0.00) D5 4 1 25.00% (0.00) 2 1 50% (0.00) 36 6 16.67% (0.00) D9 4 1 25.00% (0.00) 2 1 50% (0.00) 17.92M 1,330 0.01% (0.00) D1 11 2 18.18% (0.00) 2 1 50% (0.00) 1 1 100.00% (0.00) D1 2 1 50.00% (0.00) 2 1 50% (0.00) 16 4 25% (0.00) D10 / D11 8 3/8 61.24% (31.25) 1 1 100% (0.00) 6 / 740,916 4 / 1,570 3.76% (33.23) D10 8 5 62.50% (0.00) 1 1 100% (0.00) 2 2 100.00% (0.00) D10 / D11 8 7/5 73.95% (12.50) 1 1 100% (0.00) 77 / 303,198 19 / 803 2.56% (12.21) fmul <8xdbl> D4 / D11 8 8/8 100.00% (0.00) 1 1 100% (0.00) 20.10B / 169,199 342,023 / 838 0.03% (0.25) fneg <8xdbl> D4 / D11 8 8/1 35.36% (43.75) 1 1 100% (0.00) 15,547 / 1 280 / 1 13.42% (49.10) fma <8xdbl> D4 / D11 8 8/4 70.71% (25.00) 1 1 100% (0.00) 313.97M / 261,973 40,021 / 810 0.06% (0.15) load D2 / D3 / D12 4,481 1 / 66 / 2 0.11% (0.68) 2,754 1 / 37 / 2 0.15% (0.61) - / 4,064 / 1 - / 29 / 1 8.45% (49.64) store D2 / D3 / D12 / D13 3,015 1 / 19 / 1 / 1 0.07% (0.26) 1,367 1/14/1/1 0.14% (0.41) 1 / 266 / 1 / 1 1/8/1/1 41.64% (42.00)

Table 7: PC Sensitivity, BB Sensitivity and Input Breadth per opcode and per DPool server, across all CC-ITHICA runs. Opcodes with only one PC or one error are omitted. Most opcodes exhibit low PC and BB Sensitivity (errors concentrate in few PCs/BBs) and high Input Breadth (errors manifest across a variety of inputs). The high PC Sensitivity of double vector opcodes may be due to their concentration within a single BB.

Proto Libc Tcmalloc Hashing

7 0 0 0

4 1

8 3 9 11 7 3 0 3 1 3

3 0

Stl-Cord

Swissmap

Test Total 19 3 21 17 6 4 23 9 Unique 15 0 7 3 3 0 4 1

FB

Abseil Llvmlibc Llvmlibcpp

OpenSSL

Libraries

Zlib

Silkscreen Utils

Malign_Buff

AVX

Hasher Pattern_Gen

Top-level CC

within library code: six in Zlib only, one in OpenSSL only, and one in both Zlib and OpenSSL. This demonstrates that even if Native checks were manually constructed to be more fine-grained in the top-level code, they could still miss errors manifesting in library code and are masked before reaching the library function’s output. Obs. 13 FB tests (§6.1/§6.2) detect 24 servers (20/4 in the QPool/ DPool), including 11 unique QPool servers (not included in Fig. 4) missed by both top-level CC (both ITHICA and Native checks) and libraries, for a total of 100 servers detected by ITHICA. Importantly, none of the FB programs could be used as tests without ITHICA, as they lack built-in checkable outputs.

Table 8: Total and unique server detections among CC, libraries and FB for DPool & QPool servers across all binaries.

Finding 6: The detection of defect-induced inconsistent errors depends on (typically modest) execution context variation between original and validation instructions (Obs. 14, 15).

Finding 5: Whether an instruction exhibits a defect-induced error depends on whether it executes in a vulnerable context shaped by its preceding instruction sequence (Obs. 10–13).

Implications for Chip-Users: The key to executing instructions in diverse microarchitectural and electrical state contexts with ISAlevel control is varying this context implicitly, by (a) transforming arbitrary programs into tests, (b) varying and combining ITHICA transformations to alter the surrounding instruction mix within a test and (c) varying interleaving to exploit natural diversity within a test; and explicitly, by using MemDiv (or similar) for proactive diversity.

Sensitivity to natural or proactive diversity. Obs. 14 Fig. 6 shows that natural diversity with an interleaving of 1—where each validation instruction immediately follows its original—is sufficient to expose inconsistent errors for most servers in the DPool. However, larger interleavings further improve EDR for some servers (e.g., D3, D4). Notably, D6 is exclusively detected at an interleaving of 8. Beyond natural diversity, proactive diversity may be necessary to expose certain defects. Obs. 15 As shown in Fig. 3, D13 is uniquely detected by MemDiv; all other ITHICA transformations miss it. Errors are localized to the third validation load that checks an original store (and follows a clflush), while the previous two loads yield correct results. One additional server in the QPool is similarly detected exclusively by MemDiv (Fig. 4). This suggests that the clflush and mfence instructions inserted by MemDiv perturb the server’s execution context between original and validation memory instructions sufficiently to expose an inconsistent error. These observations show that, for arithmetic instructions, natural diversity (§3.3) from modest interleavings is generally sufficient to expose inconsistencies, possibly because relevant execution context is largely core-side. Proactive diversity (§3.3) is necessary to expose certain defects affecting memory instructions, as demonstrated by the MemDiv detections, possibly due to their relevant execution context spanning core and uncore components.

7.4

Other Reported Factors are Insufficient for Defect Detection and Reproducibility

As established in §7.3, variation in execution context for failing instructions across programs explains the program-sensitivity of defect detection—why different programs executing the same failing instruction do not all detect the same defects. Prior work, lacking instruction-level visibility across diverse programs, has instead proposed instruction usage stress—the dynamic frequency of a failing opcode in a test—as a predictor of detection, claiming that detecting tests execute the defective opcode more frequently than non-detecting ones [82, 83]. To investigate this claim, we focus on servers uniquely detected by a single test and for which at least one failing opcode also appears in at least one other test besides the detecting one. For each such server, if multiple failing opcodes are shared with the non-detecting 10

1.0

add <4xint> getelementptr fmul x86_fp80 icmp int or <4xint> fmul x86_fp80 udiv int or <4xint> fadd dbl add <4xint> add <4xint> getelementptr and int add <2xint> select int lshr int or int mul int fmuladd dbl getelementptr llvm.rint dbl icmp int

Normalized Frequency

ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions

0.5 0.0

CC Swiss Proto TcmallocHash Stl Figure 7: Normalized execution frequency of failing opcodes for

D3 D9 Instr. BB Instr. BB Failing Opcode None or <4xint> / trunc int / zext int fmul x86_fp80 fmul x86_fp80 17 / 13.5/ 25.0 Fail BB Length — — 4.0 12.0 / 1.0/ 16.2 Pass BB Length — — —

servers (columns) uniquely detected by one ITHICA program. Orange indicates the detecting program; gray indicates non-detecting programs executing the same opcode.

Figure 8: Top: EDR of original detecting CC-ITHICA programs compared to the single-instruction and basic-block reproducers (one per failing opcode/basic block, averaged). Reproducers are evaluated under both original failing inputs (light bars) and random inputs (dark bars). Bottom: Failing opcodes and lengths of failing vs. non-failing basic blocks for reproducers with non-zero EDR.

tests, we select the most frequently failing one. Fig. 7 shows the normalized execution frequency of this opcode across all tests that execute it. Obs. 16 In 13 of 22 cases (59%), the detecting test is not the one with the highest execution frequency of the failing opcode. This supports that execution context, not merely instruction usage stress, impacts error manifestation and therefore detection. Notably, prior work also uses instruction usage stress for instruction localization, heuristically flagging as suspect those instructions that execute most frequently across failing testcases [82, 83] (Table 9); Obs. 16 equally undermines this inference.

real failing inputs (from the original CC-ITHICA runs) and newlygenerated random inputs. We use Arith to generate tests for all servers except D2/D12, where we use Mem/MemDiv, which uniquely detected them. We run each test in a loop for 100 hours (iterating through real or random inputs) and report the EDR results in Fig. 8, alongside the EDR of the original CC-ITHICA program from Fig. 3; details for tests with detections are shown at the bottom. Obs. 18 Typically, neither single-instruction nor basic-block tests detect any errors, even with real failing inputs. There are two exceptions; D3 is not reproduced by single-instruction tests but is by basic-block tests, suggesting that modestly long sequences establish the necessary execution context for its detection. D9, which fails on fmul x86_fp80, achieves perfect reproduction with the single-instruction tests with real inputs. A potential explanation for D9 is reduced microarchitectural execution path non-determinism [31] for its failing LLVM IR instruction. Such non-determinism can arise from multiple sources: compilers may map an LLVM IR instruction to multiple assembly instructions; hardware may map an assembly instruction to multiple micro-ops; and hardware may assign micro-ops to one of multiple functional units. For D9, the first two sources are eliminated and the third reduced: inspection of the compiled single-instruction LLVM IR reproducer reveals a single assembly instruction, fmul x86_fp80; fmul x86_fp80 maps to a single micro-op per uops.info [2] and the server’s microarchitecture documentation (undisclosed); and that micro-op can execute on one of two floating-point units (FPUs). The frequency of both original and validation instructions producing incorrect, inconsistent outputs suggests they often interact with the same defective hardware component. Since ITHICA checks detect errors exclusively for fmul x86_fp80 instructions across the full CC-ITHICA test suite, this component is likely one of the FPUs. However, we emphasize that hardware localization remains inherently opaque at the ISA-level (§7.5). Notably, even with this reduced microarchitectural execution path non-determinism and likely error persistence (§3.1), ITHICA still detects the error, making this case similar to our fault injection example in §3.2. This experiment, along with the TTD variability observed under fixed program inputs (Obs. 17), indicates that microarchitectural and

Finding 7: The program-sensitivity of defect detection is not explained by the dynamic frequency of a failing opcode (Obs. 16). Even for the same program running on the same server, not all runs detect errors: as Fig. 3 shows, detecting CC-ITHICA tests have EDR less than 100% for most servers. Two factors determine runto-run error reproducibility: architectural (input-dependent) and non-architectural (non-input-dependent) execution context. In the former case, different runs use different randomly generated inputs, which influence error manifestation (since instructions’ inputs are part of their execution context) and detection (e.g., through logical masking), or even control flow (determining which instructions execute at all). In the latter case, microarchitectural and electrical state vary across runs of the same program on the same hardware [53], causing errors to manifest in some runs but not others. To assess the impact of non-architectural context on the reproducibility of detections across runs, we rerun CC-ITHICA on three servers with high EDR (D3, D4, D7) with fixed inputs across runs, using TTD (Table 3) as a proxy for cross-run error reproducibility. Since the program inputs that resulted in detections in earlier experiments are not directly accessible, we use newly-generated random inputs. If the same input, and thus architectural context, repeatedly causes the same error to manifest, TTD should remain constant. Obs. 17 We find that TTD distributions remain highly variable (e.g., for D3, ranging from seconds to nearly the full hour), demonstrating that non-architectural execution context contributes to error manifestation. Other servers did not exhibit any error detections in this experiment, precluding further analysis. To assess the length of instruction sequences required for reproducibility of defect-induced errors, we construct two additional types of tests per server: single-instruction tests and basic-block tests, each derived from original CC-ITHICA programs by isolating a failing instruction or basic block. For each test, we evaluate both 11

Testing Method

Evaluation

Ioanna Vavelidou, Subho S. Banerjee, Eric X. Liu, Mike Fuller, Subhasish Mitra, and Caroline Trippel

Figure 9: Failing instruction type combinations across ITHICA

Conclusions

tests. Bar height: number of servers with errors in that combination. Bar segments: average per-server breakdown of errors by instruction type. Pie chart: same breakdown aggregated across all servers.

electrical state (beyond architectural control, as discussed in §3.3) contribute to the manifestation of defect-induced errors. Finding 8: Reproduction of defect-induced errors generally cannot be done deterministically with ISA-level control (Obs. 17, 18). Having just established that supplying the right inputs to the right opcodes is insufficient for reproduction, we further characterize the distribution of instruction-level input values that result in error detections for each opcode-server combination for our full ITHICA programs, using Input Breadth (Table 3). Obs. 19 Table 7 shows that, for most opcodes and servers, the number of unique failing inputs and Input Breadth are high, indicating that errors are largely not confined to a few culprit input values.

1. Factors Determining Detection 2. Vulnerable HW Unit 3. Solution Proposed

ITHICA (this work) >3,000 100

Inconsistent (validated on real HW, §7.1) All (Arith, FP, Vec, Mem, CF) Intra-thread inst.-level Concurrent w/ detection Sequence-driven execution Inst. usage stress, Opcode, Input, context (§7.3); Inst. usage Temperature Temperature stress & opcodes/inputs alone are insufficient (§7.4) ALU, FPU, Vec.Unit Vector Shows prior work conclusions Cache, TrxMem multiplier are unreliable (§7.5) Diverse programs as tests (§7.3); Can focus on ABFT for Inst. localization concurrently vulnerable features matmul kernels with detection (§7.5)

Table 9: ITHICA comparison with related hyperscaler fleet studies. we newly observe this phenomenon at the instruction level and quantify the rate disparity between types. Obs. 21 Despite accounting for a large share of detected servers, arithmetic and memory instructions contribute only a small fraction of errors: 92.9% originate from floating-point and vector instructions. This suggests that these types of instructions may be easier to reproduce or, equivalently, have reduced microarchitectural execution path nondeterminism (§7.4). Obs. 22 Floating-point instructions exhibit errors six orders of magnitude more frequently than memory instructions, on average. Yet, floating-point instructions are affected in fewer servers than memory instructions (26 vs. 28). Similarly, prior work observes high error frequencies for vector instructions on few detected servers—millions of errors across only 18 servers [51] (Table 9). Our findings demonstrate that high error frequencies for particular instructions aggregated across a fleet may reflect ease of reproducibility and not per-server prevalence. ITHICA’s ability to detect errors across multiple instruction types within the same test illustrates why drawing hardware localization conclusions from tests that check only a narrow set of instructions is problematic. Obs. 23 For example, the only baseline test that detects D1 is the AVX subtest of CC—a vector-specialized test. From this test alone, one may blame a vector-specific hardware unit. Yet ITHICA’s instruction-level checks within the same test reveal nonvector instructions are also affected, suggesting a different cause. SEVI [51], for instance, attributes errors in vector instructions to vector units, using vector-specific tests alone. Obs. 24 In our experiments, however, vector instructions exhibit errors on 46 servers, 31 of which feature errors in other instruction types as well, challenging these recent hardware localization conclusions.

Finding 9: For vulnerable instructions, selecting the “perfect” input appears to be neither necessary nor sufficient to induce a defect-induced error (Obs. 17–19). Implications for Chip-Users: Prior hyperscaler studies use instruction usage stress as a predictor of defect detection and as a basis for instruction localization [82, 83]; others rely on short or targeted tests for instruction localization [11, 51]. Our findings contradict both: instruction execution frequency does not reliably predict which tests detect a defective server—and therefore cannot reliably localize failing instructions—and far fewer errors are reproducible with short tests than previously assumed, biasing error characterization toward those that are (Table 9, Conclusion 1).

7.5

Alibaba [82, 83] SEVI [51] Suspect-Defective Unclear >2,500 Pool Size Servers Detected 27-30 18 & Analyzed Error Consistent Consistent Assumption (implicit) (explicit) Inst. Types Checked — (not inst.-level) Vector only Error Check Final output w/ Inst.-level w/ Technique golden value golden value Instruction Post-detection via Concurrent w/ Localization inst. usage stress detection

Hardware Localization with ISA-Level Information is Opaque

Fig. 5 shows that some DPool servers exhibit errors concentrated in a single instruction type (e.g., D10), while others show errors across types (e.g., D1, D12). Fig. 9 extends this analysis by examining the distribution of errors across instruction types for 93 of all 100 ITHICA-detected servers (89 by CC-ITHICA and 11 unique to FB-ITHICA); seven are excluded from this analysis due to a crash that prevented the logging of the failing instruction from completing (no-instr, §7.1). For each combination of instruction types, the figure shows the number of servers exhibiting errors for that combination and the average per-server breakdown of number of errors by instruction type. Finally, the accompanying pie chart aggregates this breakdown across all 93 servers. Obs. 20 Among all 93 servers analyzed, 44% exhibit errors spanning multiple instruction types, rather than a single type—consistent with prior work observing multi-family test failures [11, 82, 83];

Finding 10: The same defect often causes errors across multiple instruction types, at markedly different rates (Obs. 20–24). More generally, while ITHICA’s instruction-level visibility enables scrutinizing hardware localization conclusions, it cannot definitively establish them. This is a limitation of all ISA-level testing approaches [11, 22, 33, 34, 37, 51, 82, 83], including ITHICA. A defect may reside anywhere along an instruction’s microarchitectural execution path or in physically adjacent transistors serving unrelated 12

ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions Check Type Evaluation Golden Cross- Invert. Intra-thr. Fault Real Spec Value Core Comp. Instruct. Inject. HW cpu-check [22] – – C/C++ ✗ ✓ ✓ ✗ OpenDCDiag [33] ✓ ✓ ✓ ✗ ✗ ✓ C/C++ SiliFuzz [37] – – ASM ✓ ✓ ✗ ✗ Harpocrates [34] ✓ ✓ ✗ ✗ ✓ ✗ Uarch ITHICA (this paper) ✗ ✗ ✗ ✓ ✗ ✓ LLVM Technique

Figure 10: Comparison of SiliFuzz, CC and ITHICA tests, across all commonly tested servers. Unique server detections for each test are shown in parentheses.

Table 10: Comparison with testing tools/generation techniques

9 hardware components, none of which is visible at the ISA-level. Moreover, the compiler’s mapping of LLVM IR to assembly, and the hardware’s mapping of assembly to micro-ops and micro-ops to functional units, introduce microarchitectural execution path non-determinism inherent to ISA-level approaches (§7.4). Implications for Chip-Users: Hardware localization is inherently opaque from ISA-level observations. Although ITHICA’s instructionlevel checking across diverse instructions and programs does not overcome this opacity, it surfaces the full range of LLVM instructions affected on each server, from which prior hardware localization conclusions [51, 82, 83] (Table 9, Conclusion 2) can be scrutinized and, in some cases, discredited. Incorrect hardware localization conclusions risk guiding future software and hardware mitigations toward the wrong or insufficient components (Table 9, Conclusion 3).

8

Related Work

Prior works most relevant to ITHICA fall into two categories: intrathread instruction checking techniques for soft error detection and post-silicon validation, and hyperscaler fleet studies of the CPU SDC problem. We also compare to existing functional testing suites [22, 33] and test generation techniques [23, 34] in Table 10. Soft Error Detection and Post-Silicon Validation. EDDI [59] is the first to introduce intra-thread instruction checks. CFCSS [58] is a comprehensive approach that validates control-flow at runtime against compile-time control-flow graphs. SWIFT [69] reduces EDDI’s overhead through optimized resource usage, and improves control-flow coverage. Several variations propose further performance optimizations [13, 18, 27, 32]. The QED family of techniques [30, 42, 43, 72] targets design bugs in post-silicon validation, adopting intra-thread checks from soft error techniques, as well as cross-core instruction checks. ITHICA adapts and re-implements the intra-thread instruction checks of these works for LLVM, and extends them with the novel MemDiv transformation. MemDiv can benefit post-silicon validation but is inapplicable for soft errors. Hyperscaler Fleet Studies. Like ITHICA, the Alibaba [82, 83] and SEVI [51] studies evaluate their techniques using pools of suspect-defective servers. Table 9 compares their testing methods and findings with ITHICA’s. Other works quantify the prevalence and report detection trends of hardware errors across large CPU fleets [11, 14–16, 29, 53] or study SDC vulnerability through fault injection [7, 21, 64, 65]. Among them, PinDrop [11] advocates for continuous testing with a variety of complex tests at scale—a philosophy that aligns with ITHICA’s goal of enabling arbitrary programs to be used as tests. Mitra et al. [53] finds that a significant fraction of defects are detected by production workloads in the field, motivating the instrumentation of such workloads with ITHICA.

Discussion

Fuzzing. Fig. 10 compares SiliFuzz [23, 37] (the only fuzzing tool operating at the ISA level or above), CC, CC-ITHICA, and FB-ITHICA by number of total and unique detected servers and average EDR; TTD and EF statistics are not collected for SiliFuzz. This analysis is limited to servers that have accumulated at least 20 hours of execution for all tests. SiliFuzz detects significantly fewer servers than CC-ITHICA (42 vs. 71). For the servers it detects, it achieves slightly worse average EDR than CC-ITHICA. Notably, FB-ITHICA detects seven servers missed by all other tests. Coverage Gaps. We identify three main sources of limited coverage in ITHICA’s implementation: (1) atomic/volatile memory instructions (§4.3), (2) memory instructions in thread-unsafe code (§4.3) and (3) uninstrumented parts of Libc and Libc++ libraries (§5.1). Regarding (2), ITHICA still successfully instruments most opcodes for this class of programs that are difficult to check with other techniques (§1). Due to (3), we leave kernel code executed as part of system calls unchecked. Consistent Errors. Although it is not possible to determine whether ITHICA missed the six servers in Fig. 4 due to the above coverage gaps or consistent errors, future work can extend ITHICA with functional diversity [19, 42, 57, 80] to target the latter. Effect of Temperature. Temperature is a well-known contributor to fault manifestation [51, 82, 83]. Although it is part of our execution context definition as a component of electrical state (§3), controlling or measuring it with sufficient precision during indatacenter testing is highly challenging. Therefore, ITHICA does not modulate temperature directly. Instead, the levers described in §7.3 induce internal electrical state fluctuations, modulating it implicitly.

10

Conclusion

We present ITHICA, an approach for automatically generating functional tests from arbitrary programs to detect defective CPUs. ITHICA exploits the insight that the most pernicious defects manifest as inconsistent errors—validated here for the first time on real hardware—to build intra-thread, instruction-level checks that outperform baseline industry checks and enable novel findings that challenge conclusions of prior hyperscaler studies.

Acknowledgements This work was supported in part by the National Science Foundation under award 2321489 and a Sloan Research Fellowship. We also gratefully acknowledge gifts from Google, Meta, and the Open Compute Project. Finally, Gemini/Claude were used in preparing this manuscript for minor edits, e.g., flagging typos and minor phrasing issues. Gemini was used to edit figure plotting scripts. 13

Ioanna Vavelidou, Subho S. Banerjee, Eric X. Liu, Mike Fuller, Subhasish Mitra, and Caroline Trippel

References

[20] Nishant George, Sudhanva Gurumurthi, Vilas Sridharan, Harish Dattatraya Dixit, Emel Goksu, Bharath Parthasarathy, Amber Huffman, Thiago Macieira, Arani Sinha, Dean Liberty, Lisa Minwell, and Robert S. Chappell. 2025. Silent Data Corruption in AI: A Growing Challenge for Large-Scale Machine Learning. IEEE Micro (2025), 1–7. https://doi.org/10.1109/MM.2025.3645670 [21] Dimitris Gizopoulos, George Papadimitriou, Odysseas Chatzopoulos, Nikos Karystinos, Harish D. Dixit, and Sriram Sankar. 2024. Silent Data Corruptions in Computing Systems: Early Predictions and Large-Scale Measurements. In 2024 IEEE European Test Symposium (ETS). 1–10. https://doi.org/10.1109/ETS61313. 2024.10567770 [22] Google. 2020. Google cpu-check torture test. https://github.com/google/cpucheck. [23] Google. 2021. Silifuzz. https://github.com/google/silifuzz. [24] Google. 2022. Fleetbench. https://github.com/google/fleetbench. [25] F. Hapke, R. Krenz-Baath, A. Glowatz, J. Schloeffel, H. Hashempour, S. Eichenberger, C. Hora, and D. Adolfsson. 2009. Defect-oriented cell-aware ATPG and fault simulation for industrial cell libraries and designs. In 2009 International Test Conference. [26] Zhengyang He, Yafan Huang, Hui Xu, Dingwen Tao, and Guanpeng Li. 2023. Demystifying and Mitigating Cross-Layer Deficiencies of Soft Error Protection in Instruction Duplication. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Denver, CO, USA) (SC ’23). Association for Computing Machinery, New York, NY, USA, Article 86, 13 pages. https://doi.org/10.1145/3581784.3607078 [27] Zhengyang He, Hui Xu, and Guanpeng Li. 2024. A Fast Low-Level Error Detection Technique. In 2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). 90–98. https://doi.org/10.1109/DSN58291.2024. 00023 [28] K. Heragu, J.H. Patel, and V.D. Agrawal. 1996. Segment delay faults: a new fault model. In Proceedings of 14th VLSI Test Symposium. [29] Peter H. Hochschild, Paul Turner, Jeffrey C. Mogul, Rama Govindaraju, Parthasarathy Ranganathan, David E. Culler, and Amin Vahdat. 2021. Cores That Don’t Count. In Proceedings of the Workshop on Hot Topics in Operating Systems. [30] Ted Hong, Yanjing Li, Sung-Boem Park, Diana Mui, David Lin, Ziyad Abdel Kaleq, Nagib Hakim, Helia Naeimi, Donald S. Gardner, and Subhasish Mitra. 2010. QED: Quick Error Detection tests for effective post-silicon validation. In 2010 IEEE International Test Conference. [31] Yao Hsiao, Nikos Nikoleris, Artem Khyzha, Dominic P. Mulligan, Gustavo Petri, Christopher W. Fletcher, and Caroline Trippel. 2024. RTL2M𝜇 PATH: Multi𝜇 PATH Synthesis with Applications to Hardware Security Verification. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). 507–524. https://doi.org/10.1109/MICRO61859.2024.00045 [32] Yafan Huang, Shengjian Guo, Sheng Di, Guanpeng Li, and Franck Cappello. 2022. Mitigating Silent Data Corruptions in HPC Applications across Multiple Program Inputs. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. 1–14. https://doi.org/10.1109/SC41404.2022.00022 [33] Intel. 2021. OpenDCDiag. https://github.com/opendcdiag. [34] Nikos Karystinos, Odysseas Chatzopoulos, George-Marios Fragkoulis, George Papadimitriou, Dimitris Gizopoulos, and Sudhanva Gurumurthi. 2024. Harpocrates: Breaking the Silence of CPU Faults through Hardware-in-the-Loop Program Generation. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). 516–531. https://doi.org/10.1109/ISCA59077.2024.00045 [35] Nikos Karystinos, George-Marios Fragkoulis, Odysseas Chatzopoulos, Dimitris Gizopoulos, and Sudhanva Gurumurthi. 2025. Harpocrates++: Automated Functional Program Generation against CPU Faults and Silent Data Corruptions. IEEE Micro (2025), 1–9. https://doi.org/10.1109/MM.2025.3640385 [36] S. Kundu, S. Sengupta, and R. Galivanche. 2000. Test challenges in nanometer technologies. In Proceedings IEEE European Test Workshop. [37] Doug Kwan, Kostik Shtoyk, Kostya Serebryany, Maxim L Lifantsev, and Peter Hochschild. 2021. SiliFuzz: Fuzzing CPUs by proxy. Technical Report. Google. [38] J.C.-M. Li and E.J. McCluskey. 2002. Diagnosis of sequence-dependent chips. In Proceedings 20th IEEE VLSI Test Symposium (VTS 2002). 187–192. https: //doi.org/10.1109/VTS.2002.1011137 [39] Wei Li, Chris Nigh, Danielle Duvalsaint, Subhasish Mitra, and R. D. Blanton. 2022. PEPR: Pseudo-Exhaustive Physically-Aware Region Testing. In International Test Conference. [40] libcllvm 2024. The LLVM C Library. https://libc.llvm.org/. [41] libcxxllvm 2024. LLVM libc++. https://github.com/llvm/llvmproject/blob/main/libcxx/include/concepts. [42] David Lin, Ted Hong, Yanjing Li, Farzan Fallah, Donald S Gardner, Nagib Hakim, and Subhasish Mitra. 2013. Overcoming post-silicon validation challenges through quick error detection (QED). In 2013 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 320–325. [43] David Lin, Ted Hong, Yanjing Li, Eswaran S, Sharad Kumar, Farzan Fallah, Nagib Hakim, Donald S. Gardner, and Subhasish Mitra. 2014. Effective Post-Silicon Validation of System-on-Chips Using Quick Error Detection. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 33, 10 (2014), 1573–1590. https://doi.org/10.1109/TCAD.2014.2334301

[1] Andreas Abel, Yuying Li, Richard O’Grady, Chris Kennelly, and Darryl Gove. 2024. A Profiling-Based Benchmark Suite for Warehouse-Scale Computers. In 2024 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 325–327. https://doi.org/10.1109/ISPASS61541.2024.00046 [2] Andreas Abel and Jan Reineke. 2019. uops.info: Characterizing Latency, Throughput, and Port Usage of Instructions on Intel Microarchitectures. In ASPLOS (Providence, RI, USA) (ASPLOS ’19). ACM, New York, NY, USA, 673–686. https: //doi.org/10.1145/3297858.3304062 [3] abseil 2024. Abseil. https://github.com/abseil/abseil-cpp. [4] Mridul Agarwal, Bipul C. Paul, Ming Zhang, and Subhasish Mitra. 2007. Circuit Failure Prediction and Its Application to Transistor Aging. In 25th IEEE VLSI Test Symposium (VTS’07). 277–286. https://doi.org/10.1109/VTS.2007.22 [5] Jonathan T.-Y. Chang, Chao-Wen Tseng, Chien-Mo James Li, Mike Purtell, and Edward Joseph McCluskey. 1998. Analysis of pattern-dependent and timing-dependent failures in an experimental test chip. Proceedings International Test Conference 1998 (IEEE Cat. No.98CH36270) (1998), 184–193. https: //api.semanticscholar.org/CorpusID:16286356 [6] Saranyu Chattopadhyay, Keerthikumara Devarajegowda, Bihan Zhao, Florian Lonsing, Brandon A. D’Agostino, Ioanna Vavelidou, Vijay D. Bhatt, Sebastian Prebeck, Wolfgang Ecker, Caroline Trippel, Clark Barrett, and Subhasish Mitra. 2023. G-QED: Generalized QED Pre-silicon Verification beyond Non-Interfering Hardware Accelerators. In 2023 60th ACM/IEEE Design Automation Conference (DAC). 1–6. https://doi.org/10.1109/DAC56929.2023.10247903 [7] Odysseas Chatzopoulos, Nikos Karystinos, George Papadimitriou, Dimitris Gizopoulos, Harish D. Dixit, and Sriram Sankar. 2025. Veritas - Demystifying Silent Data Corruptions: uArch-Level Modeling and Fleet Data of Modern x86 CPUs. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). 1–14. https://doi.org/10.1109/HPCA61900.2025.00012 [8] Odysseas Chatzopoulos, George Papadimitriou, Dimitris Gizopoulos, Harish D Dixit, and Sriram Sankar. 2025. From gates to sdcs: Understanding fault propagation through the compute stack. In 2025 Design, Automation & Test in Europe Conference (DATE). IEEE, 1–7. [9] Tze Wee Chen, Kyunglok Kim, Young Moon Kim, and Subhasish Mitra. 2008. Gate-Oxide Early Life Failure Prediction. In 26th IEEE VLSI Test Symposium (vts 2008). 111–118. https://doi.org/10.1109/VTS.2008.55 [10] Eric Cheng, Shahrzad Mirkhani, Lukasz G. Szafaryn, Chen-Yong Cher, Hyungmin Cho, Kevin Skadron, Mircea R. Stan, Klas Lilja, Jacob A. Abraham, Pradip Bose, and Subhasish Mitra. 2016. CLEAR: Cross-Layer Exploration for Architecting Resilience - Combining hardware and software techniques to tolerate soft errors in processor cores. In Proceedings of the 53rd Annual Design Automation Conference (Austin, Texas) (DAC ’16). Association for Computing Machinery, New York, NY, USA, Article 68, 6 pages. https://doi.org/10.1145/2897937.2897996 [11] Peter Deutsch, Harish Dixit, Gautham Vunnam, Carl Moran, Eleanor Ozer, and Sriram Sankar. 2026. PinDrop: Breaking the Silence on SDCs in a Large-Scale Fleet. 1–14. https://doi.org/10.1109/HPCA68181.2026.11408620 [12] Peter W. Deutsch, Vincent Quentin Ulitzsch, Sudhanva Gurumurthi, Vilas Sridharan, Joel S. Emer, and Mengjia Yan. 2024. DelayAVF: Calculating Architectural Vulnerability Factors for Delay Faults. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). 231–245. https://doi.org/10.1109/ MICRO61859.2024.00026 [13] Moslem Didehban and Aviral Shrivastava. 2016. nZDC: A compiler technique for near Zero Silent Data Corruption. In 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC). 1–6. https://doi.org/10.1145/2897937.2898054 [14] Harish Dattatraya Dixit, Laura Boyle, Gautham Vunnam, Sneha Pendharkar, Matt Beadon, and Sriram Sankar. 2022. Detecting silent data corruptions in the wild. arXiv:2203.08989 [cs.AR] https://arxiv.org/abs/2203.08989 [15] Harish Dattatraya Dixit, Sneha Pendharkar, Matt Beadon, Chris Mason, Tejasvi Chakravarthy, Bharath Muthiah, and Sriram Sankar. 2021. Silent Data Corruptions at Scale. CoRR abs/2102.11245 (2021). https://arxiv.org/abs/2102.11245. [16] Rhea Dutta, Harish Dattatraya Dixit, Rik Van Riel, Gautham Vunnam, and Sriram Sankar. 2025. Hardware Sentinel: Protecting Software Applications from Hardware Silent Data Corruptions. Association for Computing Machinery, New York, NY, USA, 482–497. https://doi.org/10.1145/3676641.3716258 [17] E. B. Eichelberger and T. W. Williams. 1988. A logic design structure for LSI testability. In Papers on Twenty-Five Years of Electronic Design Automation (25 years of DAC). Association for Computing Machinery, New York, NY, USA, 358–364. https://doi.org/10.1145/62882.62924 [18] Shuguang Feng, Shantanu Gupta, Amin Ansari, and Scott Mahlke. 2010. Shoestring: probabilistic soft error reliability on the cheap. ACM SIGPLAN Notices 45 (03 2010), 385. https://doi.org/10.1145/1735971.1736063 [19] Nikos Foutris, Dimitris Gizopoulos, Mihalis Psarakis, Xavier Vera, and Antonio Gonzalez. 2011. Accelerating microprocessor silicon validation by exposing ISA diversity. In Proceedings of the 44th Annual IEEE/ACM International Symposium on Microarchitecture (Porto Alegre, Brazil) (MICRO-44). Association for Computing Machinery, New York, NY, USA, 386–397. https: //doi.org/10.1145/2155620.2155666 14

ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions

[44] llvm-language-ref 2022. LLVM Language Reference Manual. https://llvm.org/ docs/LangRef.html. Accessed: 2022-10-19. [45] Florian Lonsing, Subhasish Mitra, and Clark W. Barrett. 2020. A Theoretical Framework for Symbolic Quick Error Detection. In 2020 Formal Methods in Computer Aided Design, FMCAD 2020, Haifa, Israel, September 21-24, 2020. IEEE, 1–10. https://doi.org/10.34727/2020/ISBN.978-3-85448-042-6_9 [46] Jiacheng Ma, Majd Ganaiem, Madeline Burbage, Theo Gregersen, Rachel McAmis, Freddy Gabbay, and Baris Kasikci. 2025. Proactive Runtime Detection of AgingRelated Silent Data Corruptions: A Bottom-Up Approach. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 4 (Hilton La Jolla Torrey Pines, La Jolla, CA, USA) (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 220–235. https://doi.org/10.1145/3622781.3674182 [47] S.C. Ma, P. Franco, and E.J. McCluskey. 1995. An experimental chip to evaluate test techniques experiment results. In Proceedings of 1995 IEEE International Test Conference (ITC). [48] Timothy C. May and Murray H. Woods. 1978. A New Physical Mechanism for Soft Errors in Dynamic Memories. In 16th International Reliability Physics Symposium. 33–40. https://doi.org/10.1109/IRPS.1978.362815 [49] E.J. McCluskey. 1993. Quality and single-stuck faults. In Proceedings of IEEE International Test Conference - (ITC). [50] E.J. McCluskey and Chao-Wen Tseng. 2000. Stuck-fault tests vs. actual defects. In Proceedings International Test Conference 2000 (IEEE Cat. No.00CH37159). [51] Yixuan Mei, Shreya Varshini, Harish Dixit, Sriram Sankar, and K. V. Rashmi. 2026. SEVI: Silent Data Corruption of Vector Instructions in Hyper-Scale Datacenters. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (USA) (ASPLOS ’26). Association for Computing Machinery, New York, NY, USA, 1711–1726. https://doi.org/10.1145/3779212.3790217 [52] Justin Meza, Qiang Wu, Sanjeev Kumar, and Onur Mutlu. 2015. Revisiting Memory Errors in Large-Scale Production Data Centers: Analysis and Modeling of New Trends from the Field. In 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. 415–426. https://doi.org/10.1109/DSN. 2015.57 [53] Subhasish Mitra, Subho Banerjee, Martin Dixon, Rama Govindaraju, Peter Hochschild, Eric X. Liu, Bharath Parthasarathy, and Parthasarathy Ranganathan. 2025. Silent Data Corruption by 10x Test Escapes Threatens Reliable Computing. arXiv:2508.01786 [cs.AR] https://arxiv.org/abs/2508.01786 [54] S. Mukherjee. 2008. Architecture Design for Soft Errors. https://doi.org/10.1016/ B978-0-12-369529-1.X5001-0 [55] S.S. Mukherjee, J. Emer, and S.K. Reinhardt. 2005. The soft error problem: an architectural perspective. In 11th International Symposium on High-Performance Computer Architecture. [56] S.S. Mukherjee, C. Weaver, J. Emer, S.K. Reinhardt, and T. Austin. 2003. A systematic methodology to compute the architectural vulnerability factors for a high-performance microprocessor. In Proceedings. 36th Annual IEEE/ACM International Symposium on Microarchitecture, 2003. MICRO-36. 29–40. https: //doi.org/10.1109/MICRO.2003.1253181 [57] N. Oh, S. Mitra, and E.J. McCluskey. 2002. ED4I: error detection by diverse data and duplicated instructions. IEEE Trans. Comput. 51, 2 (2002), 180–199. https://doi.org/10.1109/12.980007 [58] N. Oh, P.P. Shirvani, and E.J. McCluskey. 2002. Control-flow checking by software signatures. IEEE Transactions on Reliability 51, 1 (2002), 111–122. https://doi. org/10.1109/24.994926 [59] Nahmsuk Oh, Philip Shirvani, and Edward McCluskey. 2002. Error detection by duplicated instructions in super-scalar processors. IEEE Transactions on Reliability 51, 1 (2002), 63–75. https://doi.org/10.1109/24.994913 [60] OpenHW Group. 2019. CVA6 RISC-V CPU. https://github.com/openhwgroup/ cva6. [61] openssl 2024. OpenSSL. https://github.com/openssl/openssl. [62] openssl-manual [n. d.]. OPENSSL Debian Manpages. https://manpages.debian. org/testing/libssl-doc/OPENSSL_LH_doall_arg.3ssl.en.html. [63] George Papadimitriou and Dimitris Gizopoulos. 2023. AVGI: MicroarchitectureDriven, Fast and Accurate Vulnerability Assessment. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 935–948. https: //doi.org/10.1109/HPCA56546.2023.10071105 [64] George Papadimitriou and Dimitris Gizopoulos. 2023. Silent Data Corruptions: Microarchitectural Perspectives. IEEE Trans. Comput. 72, 11 (2023), 3072–3085. https://doi.org/10.1109/TC.2023.3285094 [65] George Papadimitriou, Dimitris Gizopoulos, Harish Dattatraya Dixit, and Sriram Sankar. 2023. Silent Data Corruptions: The Stealthy Saboteurs of Digital Integrity. 2023 IEEE 29th International Symposium on On-Line Testing and Robust System Design (IOLTS) (2023), 1–7. https://api.semanticscholar.org/CorpusID:261315246 [66] Priyadarsan Patra. 2007. On the cusp of a validation wall. IEEE Design & Test of Computers 24, 2 (2007), 193–196. https://doi.org/10.1109/MDT.2007.54 [67] Bipul C. Paul, Kunhyuk Kang, Haldun Kufluoglu, Muhammad A. Alam, and Kaushik Roy. 2007. Negative Bias Temperature Instability: Estimation and Design

for Improved Reliability of Nanoscale Circuits. IEEE Transactions on ComputerAided Design of Integrated Circuits and Systems 26, 4 (2007), 743–751. https: //doi.org/10.1109/TCAD.2006.884870 [68] Mahesh Prabhu and Jacob A. Abraham. 2012. Functional test generation for hard to detect stuck-at faults using RTL model checking. In 2012 17th IEEE European Test Symposium (ETS). [69] G.A. Reis, J. Chang, N. Vachharajani, R. Rangan, and D.I. August. 2005. SWIFT: software implemented fault tolerance. In International Symposium on Code Generation and Optimization. 243–254. https://doi.org/10.1109/CGO.2005.34 [70] Matthias Sauer, Young Moon Kim, Jun Seomun, Hyung-Ock Kim, Kyung-Tae Do, Jung Yun Choi, Kee Sup Kim, Subhasish Mitra, and Bernd Becker. 2013. Early-life-failure detection using SAT-based ATPG. In 2013 IEEE International Test Conference (ITC). 1–10. https://doi.org/10.1109/TEST.2013.6651925 [71] Jian Shen and Jacob A. Abraham. 1998. Native mode functional test generation for processors with applications to self test and design validation. Proceedings International Test Conference 1998 (IEEE Cat. No.98CH36270) (1998), 990–999. https://api.semanticscholar.org/CorpusID:14132281 [72] Eshan Singh, Clark W. Barrett, and Subhasish Mitra. 2017. E-QED: Electrical Bug Localization During Post-silicon Validation Enabled by Quick Error Detection and Formal Methods. (2017). [73] Gordon L. Smith. 1985. Model for Delay Faults Based upon Paths. In International Test Conference. [74] Wilson Snyder, Paul Wasson, and Duane Galbi et al. [n. d.]. Verilator. https: //verilator.org [75] Daniel Sorin. 2009. Fault Tolerant Computer Architecture. Vol. 4. https://doi.org/ 10.2200/S00192ED1V01Y200904CAC005 [76] T.M. Storey and W. Maly. 1990. CMOS bridging fault detection. In Proceedings. International Test Conference 1990. [77] E. Takeda and N. Suzuki. 1983. An empirical model for device degradation due to hot-carrier injection. IEEE Electron Device Letters 4, 4 (1983), 111–113. https://doi.org/10.1109/EDL.1983.25667 [78] Arman Vassighi, Richard Kacprowicz, Carlos Carranza, and Walter Riordan. 2008. Characterizing infant mortality in high volume manufacturing. In 2008 IEEE International Reliability Physics Symposium. 717–718. https://doi.org/10.1109/ RELPHY.2008.4559004 [79] R. L. Wadsack. 1978. Fault modeling and logic simulation of CMOS and MOS integrated circuits. The Bell System Technical Journal 57, 5 (1978), 1449–1474. https://doi.org/10.1002/j.1538-7305.1978.tb02106.x [80] Ilya Wagner and Valeria Bertacco. 2008. Reversi: Post-silicon validation system for modern microprocessors. In 2008 IEEE International Conference on Computer Design. 307–314. https://doi.org/10.1109/ICCD.2008.4751878 [81] John A. Waicukauski, Eric Lindbloom, Barry K. Rosen, and Vijay S. Iyengar. 1987. Transition Fault Simulation. IEEE Design & Test of Computers 4, 2 (1987), 32–38. https://doi.org/10.1109/MDT.1987.295104 [82] Shaobu Wang, Guangyan Zhang, Junyu Wei, Yang Wang, Jiesheng Wu, and Qingchao Luo. 2023. Understanding Silent Data Corruptions in a Large Production CPU Population. In Proceedings of the 29th Symposium on Operating Systems Principles. [83] Shaobu Wang, Guangyan Zhang, Junyu Wei, Yang Wang, Jiesheng Wu, and Qingchao Luo. 2024. Understanding Silent Data Corruption in Processors for Mitigating its Effects. ACM Trans. Archit. Code Optim. 21, 4, Article 84 (Nov. 2024), 27 pages. https://doi.org/10.1145/3690825 [84] James F Ziegler and William A Lanford. 1979. Effect of cosmic rays on computer memories. Science 206, 4420 (1979), 776–788. [85] zlib 2024. Zlib. https://zlib.net/.

15