DEFault++: Automated Fault Detection, Categorization, and Diagnosis for Transformer Architectures

Abstract Transformer models are widely deployed in critical AI applications, yet faults in their attention mechanisms, projections, and other internal components often degrade behavior silently without raising runtime errors. Existing fault diagnosis techniques often target generic deep neural networks and cannot identify which transformer component is responsible for an observed symptom. In this article, we present DEFault++, a hierarchical learning-based diagnostic technique that operates at three level of abstraction: it detects whether a fault is present, classifies it into one of 12 transformer-specific fault categories (covering both attention-internal mechanisms and surrounding architectural components), and identifies the underlying root cause from up to 45 mechanisms. To facilitate both training and evaluation, we construct DEFault-bench, a benchmark of 3,739 labeled instances obtained through systematic mutation testing. These instances are created across seven transformer models and nine downstream tasks using DEForm, a transformer-specific mutation technique we developed for this purpose. DEFault++ measures runtime behavior at the level of individual transformer components. It organizes these measurements through a Fault Propagation Graph (FPG) derived from the transformer architecture. It then produces an interpretable diagnosis using prototype matching combined with supervised contrastive learning. On DEFault-bench, DEFault++ exceeds an AUROC of 0.96 for detection and a Macro-F1 of 0.85 for both categorization and root-cause diagnosis on encoder and decoder architectures. In a developer study with 21 practitioners, the accuracy of choosing correct repair actions increased from 57.1% without support to 83.3% when using DEFault++. 1

Introduction

Transformer models are widely used across natural language processing (OpenAI, 2023; Gemini Team et al., 2023), code generation (Chen et al., 2021), software engineering (Qiu et al., 2025), and medical imaging (Esteva et al., 2017). Their effectiveness stems largely from the attention mechanism (Vaswani et al., 2017), which addresses limitations of sequential models such as Recurrent Neural Networks (RNNs) (Vaswani et al., 2017). Attention is implemented through model code, optimized kernels, mixed-precision arithmetic, and task-specific preprocessing. Small implementation mistakes in attention logic, masking, or kernel integration can silently alter model behavior without raising explicit errors (Jahan et al., 2025a; de Santana Correia & Colombini, 2022; Zhai et al., 2023a; Wei et al., 2025; Miller, 2023; Voita et al., 2019). For example, attention heads can degenerate or become redundant (Michel et al., 2019; Voita et al., 2019). These faults can reduce representation diversity and affect model performance without producing immediate errors. Even when a symptom becomes visible, such as poor output quality or unstable fine-tuning, it rarely identifies the responsible component. The same symptom can arise from projection errors, masking faults, positional encoding faults, or residual corruption (Jahan et al., 2026). Developers can inspect loss curves, gradients, attention maps, and parameter updates, but these measurements alone do not connect a visible symptom to the root cause or the responsible transformer component. Recent work has proposed automated debugging techniques for DNN programs. Rule-based techniques, such as AutoTrainer (Zhang et al., 2021), UMLAUT (Yuan et al., 2021), and DeepDiagnosis (Wardat et al., 2022), check training runs against fixed symptoms such as exploding gradients, vanishing gradients, or oscillating loss. Learned classifiers, such as DeepFD (Cao et al., 2022) and DEFault (Jahan et al., 2025b), train on runtime features to predict fault categories such as loss function, learning rate, or activation. Both lines of work are architecture-agnostic and remain useful for general training and model faults in transformer programs. However, they do not address the class of faults that arise from transformer-specific mechanisms (Jahan et al., 2025a; 2026). These techniques observe runtime behavior at the model level rather than inside individual transformer components. As a result, they cannot reliably distinguish transformer-specific faults from generic training problems, nor can they identify which component is responsible. Transformer-specific debuggers such as ATTNChecker (Liang et al., 2025) and AtPatch (Weng et al., 2026) target narrower failure modes, such as NaN or Inf values during attention or attention-map anomalies at the patch level. They do not cover the full taxonomy of transformer faults shown in Figure 3. Consequently, existing techniques report broad categories such as incorrect activation or incorrect loss, without identifying which transformer component caused the fault, such as QKV or masking, or which mechanism inside that component explains it, such as dimension mismatch or stale projection update. To address these gaps, we propose DEFault++, a learning-based diagnostic technique for transformer models. DEFault++ produces three levels of diagnosis. Level 1 detects whether a fault is present. Level 2 classifies the fault into its category, such as QKV or masking. Level 3 identifies the root cause within that category. Our key insight is that each transformer fault can leave a distinctive pattern inside the affected component, even when the overall training metrics appear normal. DEFault++ measures how each transformer component behaves during training and connects these measurements through a Fault Propagation Graph (FPG) built from the internal structure of the transformer. This structure allows the diagnostic model to trace a fault back to its category and explain the diagnosis using the runtime patterns that support it. On DEFault-bench, which contains 3,739 labeled instances from encoder and decoder architectures, DEFault++ outperforms 2

four existing DNN debugging techniques on Level 1 fault detection across AUROC, macro-F1, and accuracy. In our developer study (N = 21), repair-action accuracy improved from 57.1% without assistance to 83.3% with DEFault++ assistance. In summary, we make the following contributions: (a) DEFault++, a hierarchical technique that detects faults in transformer programs, classifies them into fault categories, and identifies the root cause within that category. (b) DEForm, a mutation technique for transformer models based on a taxonomy of real-world transformer faults. (c) DEFault-bench, a transformer fault detection benchmark of 3,739 labeled instances produced by DEForm. (d) A Fault Propagation Graph (FPG) that encodes how faults propagate through transformer components and provides the diagnostic model with the transformer’s architectural structure. (e) A training objective for root-cause diagnosis that combines supervised contrastive learning with prototype matching to separate root causes within the same fault category, and an FPG-based explanation that reports which feature groups support each diagnosis. (f) An evaluation of DEFault++ on 3,739 labeled instances covering both encoder and decoder architectures, on real-world transformer faults reproduced from public GitHub issues, and through a developer study with 21 participants. The remainder of the paper is organized as follows. Section 2 presents a motivating example. Section 3 describes the construction of DEFault-bench and the DEForm mutation technique. Section 4 describes the DEFault++ diagnostic technique. Sections 5 to 7 report the controlled benchmark evaluation, the real-world fault evaluation, and the developer study, respectively. Sections 8 and 9 discuss limitations, future directions, and threats to validity. Section 10 surveys related work, and Section 11 concludes the paper.

Motivating Example

To show the class of faults DEFault++ targets, consider the real-world fault in Listing 2.1, taken from the HuggingFace Diffusers library (Issue #119031 ). The method fuse_qkv_projections merges three projection layers (query q_proj, key k_proj, and value v_proj) into one fused layer (to_qkv) and redirects the model to route all computation through this fused layer. The original projection layers are retained because deleting them would cause LoRA to raise an error when it cannot find its adapter targets. However, retaining them lets LoRA attach to modules that are no longer part of the forward pass. In the example, LoRA targets q_proj and v_proj, even though the forward pass uses only the fused layer to_qkv. The adapters therefore have no effect on model output. The fault produces no visible error. The model runs without exception, the loss decreases, and no invalid values (e.g., NaN or Inf) appear. The only observable indication that something is wrong is that fine-tuning does not change model behavior. 1

HuggingFace Diffusers Issue #11903

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

class CrossAttention(nn.Module): def __init__(self, dim, heads=8): super().__init__() self.q_proj = nn.Linear(dim, dim) self.k_proj = nn.Linear(dim, dim) self.v_proj = nn.Linear(dim, dim) self.out_proj = nn.Linear(dim, dim) def fuse_qkv_projections(self): """Fuse Q/K/V into a single linear layer for efficiency.""" w = torch.cat([self.q_proj.weight, self.k_proj.weight, self.v_proj.weight]) self.to_qkv = nn.Linear(dim, 3 * dim) self.to_qkv.weight.data.copy_(w) # BUG: original q_proj, k_proj, v_proj NOT deleted def forward(self, x, context=None): q, k, v = self.to_qkv(x).chunk(3, dim=-1) # uses fused layer # ... attention computation with q, k, v ... return self.out_proj(attn_output) # --- Downstream: LoRA targets the stale projections --lora_config = LoraConfig( target_modules=["q_proj", "v_proj"], # inactive in forward pass ) model = get_peft_model(pipeline.unet, lora_config) model.train() # LoRA adapts stale layers; no effect on forward pass

Listing 2.1: Faulty QKV projection fusion from HuggingFace Diffusers Issue #11903 (simplified)

Static analysis misses this fault because the original layers still exist, are correctly typed, and remain reachable in the parameter list. Dynamic techniques miss it for one of two reasons. Rule-based tools (e.g., AutoTrainer, UMLAUT, DeepLocalize) (Zhang et al., 2021; Yuan et al., 2021; Wardat et al., 2021) look for symptoms such as oscillating loss, heuristic validation-trend violations, or NaN/Inf, none of which occurs in this case. Learned classifiers (e.g., DeepFD, DEFault) (Cao et al., 2022; Jahan et al., 2025b) use DNN-level fault categories that do not represent transformer components. ATTNChecker (Liang et al., 2025) targets temporary NaN/Inf in attention, which this fault does not produce. In contrast, DEFault++ measures runtime traces such as QKV alignment, attention entropy, and projection update activity during training. These features help characterize the faulty component (i.e., the QKV projection path). DEFault++ therefore detects the fault at Level 1, assigns it to the correct category (i.e., QKV) at Level 2, and identifies the root cause (i.e., a stale projection update path) at Level 3.

Benchmark Construction

DEFault-bench is built from clean and faulty fine-tuning runs of seven transformer models across nine downstream tasks. We construct DEFault-bench using DEForm (Section 3.2), our mutation technique for transformers. DEForm builds on DeepCrime’s idea of deciding whether a mutation has an effect by repeating the training run several times and applying a statistical test (Humbatova et al., 2021). DeepCrime was designed for generic DNNs and does not cover transformer-specific components. DEForm adds the parts that are missing for transformers: a 12-category fault taxonomy that includes attention-internal components and the decoder KV cache, 45 mutation operators 4

taken from documented fault root causes, two injection mechanisms (one for stored parameters, one for forward computations), and a structural check that runs before the statistical test. We also replace DeepCrime’s twenty-seed generalized linear model with a one-sided sign-flip permutation test at five matched seeds, which keeps the test exact at α = 0.05 and reduces the compute cost per mutant. Each fault is specified by a configuration C = (m, t, u, f, v, ℓ, σ) that fixes the model m, the finetuning task t, the target component u, the fault category f , the fault variant v, the target layer ℓ, and the severity σ. We then run paired clean and faulty fine-tuning under matched random seeds, validate each mutant statistically against its clean counterpart on the task metric, and retain the validated mutants together with their training-time measurements as labeled instances for the diagnostic model. Figure 1 shows the benchmark construction workflow. We organize this section into the fault taxonomy and mutation operators (Section 3.1), the injection mechanism (Section 3.2), the subject models and tasks (Section 3.3), mutant validation (Section 3.4), and DEFault-bench statistics (Section 3.5).

Figure 1: Workflow for constructing DEFault-bench

3.1

Fault Taxonomy and Mutation Operators

Each mutation operator matches a transformer fault root cause that has been reported in real-world projects. The attention-internal categories and root causes come from the attention fault taxonomy of Jahan et al. (2026), which classifies 555 attention faults collected from open-source projects. The non-attention categories (Embedding, FFN, LayerNorm, Residual, Output) come from prior DNN fault taxonomies that catalog real DL faults: Humbatova et al. (2020) and Islam et al. (2019), which report 970 faults across five DL libraries. Following DeepCrime (Humbatova et al., 2021), we treat this real-fault link as the validation for each operator. DeepCrime extracted 35 operators from prior real-fault studies and argued that operators derived from real faults do not require a separate realism check. We take the same view. The statistical mutation-killing procedure in Section 3.4 then checks whether each injected fault changes model behavior against a clean baseline, which is separate from whether the fault is realistic. 5

We focus on implementation errors that do not raise runtime errors, such as configuration mistakes and silent numerical issues. These faults require statistical observation against a clean baseline for detection. We exclude explicit faults because conventional tests or runtime checks often detect them. We decompose each transformer block into six units that match common implementation boundaries. For a standard transformer block (Vaswani et al., 2017): h = LayerNorm x + MHA(x)

(1)

(2)

y = LayerNorm h + FFN(h)

where x ∈ Rn×d is an input sequence, MHA is multi-head self-attention, and the position-wise feed-forward network (FFN) follows. From this structure, we define six units: embeddings (input representations), multi-head self-attention, FFN, layer normalization, residual connections, and the output projection head. The same unit boundaries apply under pre-norm variants. Only the ordering of LayerNorm changes. For each unit, we define one or more fault categories, and each category contains multiple variants that represent specific bug patterns. Figure 2 shows the fault categories grouped by transformer component. (b) Attention Internals

(a) Transformer Unit Output Projection Head

Output Faults

Linear 𝑊 𝑂

Softmax

LayerNorm

Feed-Forward Network (FFN)

Residual Faults

LayerNorm

Multi-Head Self-Attention

Embeddings

FFN Faults

Normalization Faults

Masking Faults

Mask Kernel Faults (affects whole block)

Score Comp. Faults

Scale

Attention Variant Faults

Embedding Faults

MatMul

QKV Faults

Linear 𝑊𝑄

Linear 𝑊𝐾

Linear 𝑊𝑉 KV-Cache Faults

Input Sequence x

Input (from previous layer)

Positional Encoding Faults

Figure 2: Fault categories organized by transformer component. Part (a) shows block-level categories; Part (b) shows attention-internal categories

Multi-head self-attention is the most complex unit and a frequent source of faults in practice (Jahan et al., 2025a; Michel et al., 2019; Voita et al., 2019). We write self-attention as !

QK ⊤ Attention(Q, K, V ) = softmax √ +M V dk 6

(3)

where Q, K, V are linear projections of the input, dk is the key dimension, and M is an additive mask. We treat each component of Equation (3) as a potential fault surface and group attention faults into seven categories. Six apply to both encoders and decoders. The seventh, KV Cache, is decoder-only. Attention masking faults corrupt the mask M through missing padding masks, inverted mask logic, or incorrect reshaping across batch and head dimensions. For decoder-only self-attention, this category also covers causal-mask violations that allow attention to flow from position i to future positions j > i. QKV projection faults corrupt the projection matrices W Q , W K , W V through shared or swapped projections, zero-initialized projections, or frozen parameters. Score √ computation faults corrupt the scaling and dropout around attention scores through missing 1/ dk scaling, misplaced dropout, or unsafe mixed-precision conversions. Positional encoding faults corrupt positional information through off-by-one errors, truncation beyond the supported sequence length, or omission of positional embeddings. Kernel faults stem from misconfiguration of attention kernels, including mismatched dropout probabilities and silent backend fallbacks. Attention variant faults represent the selection of a wrong attention variant, such as single-head instead of multi-head attention, or applying causal masking in a bidirectional encoder. KV Cache faults target key-value cache management in autoregressive decoders through stale caches, off-by-one indexing, truncated caches, or cross-request leaks (Adnan et al., 2024). The remaining five units have a single fault category each. Input embedding faults corrupt token and segment embeddings by zeroing subsets, swapping pairs, or scaling segment embeddings. FFN faults alter hidden-layer characteristics by scaling weights, permanently dropping neurons, or replacing the nonlinearity. Layer normalization faults modify LayerNorm parameters by scaling or zeroing the learned scale parameter γ, shifting the bias parameter β, or misconfiguring the numerical stability term ϵ (Xiong et al., 2020; Mosbach et al., 2021). Residual connection faults break skip connections by removing them, scaling the residual path, or injecting noise. Output projection faults corrupt prediction heads by scaling logits, zeroing rows for selected output classes, or reinitializing the projection. Figure 3 shows the complete fault taxonomy that defines the Level 2 and Level 3 label spaces. The seven attention-specific categories and their 25 root causes are derived from the attention fault taxonomy (Jahan et al., 2026). The five non-attention categories (Input/Embedding, Layer Normalization, FFN, Residual, and Output) and their 20 root causes are derived from prior DNN fault taxonomies (Humbatova et al., 2020; Islam et al., 2019) and the DeepCrime mutation operators (Humbatova et al., 2021). Each root cause name identifies the fundamental mechanism or parameter that is faulty (e.g., activation function, dropout rate, weight scaling) rather than a specific corrective action. The taxonomy contains 12 fault categories. Eleven apply to encoders and all 12 apply to decoders, since KV Cache is decoder-only. Together, the 12 categories define 40 root causes for encoders and 45 for decoders. Tables 1 and 2 list the mutation operators implemented in DEForm (Section 3.2), separated into attention-specific operators and architecture-level operators. Each operator is based on a transformer fault root cause associated with the units in Figure 2, following prior fault taxonomies (Jahan et al., 2025a; Humbatova et al., 2020; Islam et al., 2019; Humbatova et al., 2021). The operator name preserves the corresponding root-cause mechanism, following (Humbatova et al., 2021). However, the final benchmark includes only the configuration that can run successfully during training and evaluation.

Figure 3: Taxonomy of transformer fault categories and root-cause labels used by DEFault++

Some root causes can lead to either explicit failures or silent behavioral faults, depending on how they are parameterized. For example, changing an output dimension may cause an immediate shape error if the output head no longer matches the task loss. The same root cause may also keep the model runnable while corrupting the output mapping. We discard configurations that crash, produce invalid tensor shapes, call unsupported backend functions, or create out-of-range indices. The remaining configurations must pass structural verification, complete feature extraction, and produce the required training traces. Each operator has a three-letter ID. The first letter identifies the transformer component (E = Embedding, M = Masking, Q = QKV, S = Score, P = Positional, K = Kernel, V = Variant, C = KV Cache, F = FFN, N = LayerNorm, R = Residual, O = Output), and the next two letters abbreviate the action and its target. For example, QZQ reads as “in the QKV component, Zero the Q projection.” The Search Type (ST) column says how DEForm (Section 3.2) chooses values for an operator’s parameter. B means the operator has no parameter and is either applied or not (e.g., MZM zeroes the attention mask). EU means the operator takes a numeric parameter and DEForm runs it once at each value in a list we provide (e.g., ETZ zeroes a configurable percentage of token embeddings). EL means the operator takes a choice from a fixed categorical set and DEForm runs it once at each item (e.g., FCA replaces the activation function with each of {ReLU, GELU, Tanh, Sigmoid}). 3.2

DEForm: Fault Injection Mechanism

We denote the clean model parameters by θ and the injected mutant by θ′ . We evaluate θ and θ′ under matched random seeds and otherwise identical fine-tuning conditions to isolate the causal effect of C. The components of the configuration C = (m, t, u, f, v, ℓ, σ) are the model architecture, the downstream task, the target unit, the fault category, the fault variant, the affected 8

Table 1: Attention-specific mutation operators used to construct DEFault-bench Component Root cause Mask application Masking

Mask application Mask generation Dynamic mask

QKV Projection

Parameter initialization Parameter initialization Parameter initialization Head interaction

Operator

Zero attention mask

MZM Replace the valid attention mask with an all-zero mask

Invert attention mask Reshape mask incorrectly Causal-mask break (decoder)

MIM

of the same shape. Invert mask semantics while preserving the expected mask shape and dtype. axis: batch or head dimension to misalign; retained only if broadcast-compatible. visibility: fraction of future keys made visible within the valid mask tensor.

Zero Q projection

QZQ Zero the existing query projection weights without

MRM MCB

Zero K projection

QZK

Zero V projection

QZV

Swap Q ↔ K

QSW

Head interaction

Tie head weights

QTH

Dynamic parameter registration

Freeze QKV gradients

QFG

Normalization

√ Drop 1/ dk scaling

B EL EU B B B B EL B

B EU

Indexing

Omit positional embeddings Shift position indices Truncate positional support

POE Remove or zero the positional contribution while

Force slow backend Mismatched dropout probability Trigger valid fallback (dtype/layout)

KSB Force a valid non-optimized attention backend. KMD pcfg , pkern : valid training vs. kernel dropout settings.

B EU

KFT dtype or layout mismatch that triggers a supported

Variant Single effective head configuration attention Dynamic dispatch Causal mask in encoder

VSH Route, tie, or mask heads so only one effective head remains while preserving output shape. Apply a broadcast-compatible causal mask in an encoder attention layer.

Cache invalidation Stale cache

CST layers: subset of layers using stale but shape-compatible

Interpolation

KV Cache (decoder)

SUC

Relative position

Variant

SPD

shape. p: dropout probability applied to score tensors before softmax. Cast score computation to fp16; configurations producing non-finite traces are discarded.

Apply dropout pre-softmax Precision handling Unsafe fp16 cast

Positional

Kernel

changing tensor shape. Zero the existing key projection weights without changing tensor shape. Zero the existing value projection weights without changing tensor shape. Swap compatible query and key projection tensors or outputs. heads: subset of existing heads tied while preserving projection dimensionality. Disable updates to existing QKV parameters while preserving the forward path.

SDS Remove score scaling while preserving attention-score

Score Implementation

Executable Mutation Parameters

Silent fallback Feature constraints Hardware incompatibility

B B

preserving hidden-state shape. ∆: index shift restricted to valid supported positions.

PTL cutoff : maximum retained position index; out-of-range

PSI

indices are discarded.

fallback path.

VEC

Cache-position

Off-by-one indexing

COB

Memory layout

Truncate cache

CTR

Distributed synchronization

Cross-request leak

CLK

cached K, V states. shift: cache index offset restricted to valid cache positions. length: maximum retained cache length while preserving the expected cache interface. Reuse compatible cached states across requests without changing cache tensor shape.

EU EU B

Table 2: Architecture-level mutation operators used to construct DEFault-bench Component Root cause Embedding

Operator

Zero token embedding subset Swap embedding pairs Scale segment / type embedding

ETZ percentage: fraction of vocabulary entries zeroed while

FSW factor: multiplicative scale on existing W1 , W2 tensors. FDN percentage: fraction of hidden neurons zeroed while

Weight initialization

Scale FFN weights Permanently drop neurons Change activation function Change weight regularization Change weight initialization

Scale parameter Scale parameter

Scale γ Zero γ

NSG factor: multiplicative scale on existing γ parameters. NZG Zero existing γ parameters while preserving LayerNorm

Bias parameter Stability parameter

Shift β Change ϵ

NSB shift: additive offset on existing β parameters. NCE value: positive numerical-stability constant accepted by

Skip connection

Remove skip connection Scale residual branch Inject Gaussian noise Change gradient clipping

RRS Zero or bypass the residual branch while preserving

Scale output logits

OSL factor: multiplicative scale on logits while preserving

output dimension. classes: subset of existing output rows to zero.

ORI init: replacement initialization scheme applied to the

Input initialization Input type Input type Weight scaling Neuron dropout

FFN Activation function Regularization

LayerNorm

Residual

ESS

preserving embedding shape. percentage: fraction of token pairs swapped within the existing vocabulary. factor: multiplicative scale applied to existing segment or type embeddings.

ST EU EU EU EU EU

preserving FFN shape.

FCA activation: executable replacement nonlinearity FRG FWI

(e.g., GELU → ReLU). scheme: replacement weight-decay or L2 coefficient accepted by the optimizer. init: replacement initialization scheme applied to existing W1 , W2 tensors.

EL EU EL EU B

shape.

Residual scaling Residual path Gradient clipping Output scaling

Output

ESW

Executable Mutation Parameters

EU EU

LayerNorm.

Output dimension Zero rows for selected classes Output type Reinitialize output projection Output dimension Change output interface

sublayer output shape. factor: multiplicative scale on the residual path.

RIN σ: standard deviation of additive noise with matching

RSR

RGC

OZR

OOD

tensor shape. value: replacement max-norm clip threshold accepted by the training loop.

existing output projection. dim/map: root-cause-derived output-dimension or mapping fault; retained only when compatible with the task loss.

layer indices ℓ ⊆ {1, . . . , L} where L is the number of transformer blocks, and the severity level σ ∈ {low, medium, high}. The severity controls the magnitude of the injected change for numeric variants (e.g., scaling factors, dropout rates) and maps to a discrete intensity for non-numeric variants (e.g., visibility ratio in causal-mask breaking). We use single-fault configurations only (one target unit, one category, one variant per run) to preserve attribution clarity at Level 2 and Level 3. Transformer faults affect either stored parameters or runtime computations, which requires two distinct injection mechanisms (Mahmoud et al., 2020; Humbatova et al., 2021). Static faults modify parameter tensors at rest, before forward execution. For example, an FFN weight-scaling fault multiplies the targeted weight matrices by a scalar:

θi′ =

 α · θ θi

if i ∈ params(u, ℓ), otherwise 10

(4)

where params(u, ℓ) selects parameters of unit u in layers ℓ. Static faults include embedding corruption, parameter scaling, permanent neuron dropout, and LayerNorm parameter changes. Dynamic faults modify the forward computation path through wrappers or hooks at selected module call sites. For example, a mask-zeroing fault intercepts the attention call and replaces the mask M by zeros: forward′ (x, M ) = forward(x, 0|M | )

(5)

where 0|M | denotes a zero tensor of the same shape as the original mask M . Dynamic faults include mask manipulation, Q/K swapping, removal of score scaling, and forced kernel selection. In both cases, the transformer architecture is unchanged. Only the targeted computation or parameter values are modified. Figure 4 contrasts the clean and injected execution paths for one representative example of each mechanism. (a) Encoder (BERT): Correct

(b) Encoder (BERT): Static Injection

# clean encoder FFN forward pass layer = model.bert.encoder.layer[layer_idx] ffn = layer # same module mutated in (b) h = ffn.intermediate.dense(hidden_states) h = ffn.intermediate.intermediate_act_fn(h) h = ffn.output.dense(h)

# static fault: mutate FFN weights at rest layer = model.bert.encoder.layer[layer_idx] ffn = layer # same module used in (a) orig_w1 = ffn.intermediate.dense.weight orig_w2 = ffn.output.dense.weight self._backup = (orig_w1.clone(), orig_w2.clone()) with torch.no_grad(): orig_w1.mul_(self.alpha) # scale W1 orig_w2.mul_(self.alpha) # scale W2

(d) Decoder (GPT): Dynamic Injection

# clean decoder call (causal behavior intact) outputs = model( input_ids=input_ids, attention_mask=attention_mask, use_cache=True, output_attentions=False, )

# dynamic fault: wrap the same model’s # attention.forward to weaken causal mask self._backup_fwd = attention.forward def faulty_forward(*args, **kwargs): mask = _get_attention_mask(kwargs) if mask is not None: # unmask a fraction of future keys kwargs["attention_mask"] = \ _weaken_causal_mask( mask, visibility_ratio) return self._backup_fwd( *args, **kwargs) attention.forward = faulty_forward

Figure 4: Clean and injected execution paths for the same model. Panels (b,d) show static parameter mutation and dynamic forward wrapping

Decoder-only self-attention is causal by construction. The clean model enforces that token position i cannot attend to future positions j > i. To inject a causal violation, we weaken or remove the existing constraint so that queries can attend to future positions. We do not add an additional causal mask. We replace a configurable fraction of the −∞ entries in the upper-triangular region of the additive mask M with 0, controlled by the severity parameter σ, so each query can attend to a proportion of keys beyond its own position. We treat causal violations as decoder instances of attention masking faults. To isolate the effect of an injected fault, we keep the main experimental settings identical between θ and θ′ runs (e.g., dataset and split assignment, optimizer and schedule, epoch budget, evaluation 11

setup, logging instrumentation, random seed). Each faulty training run is paired with a clean training run under the same seed and otherwise identical training conditions. We implement fault injection with a context manager. The manager stores the original parameter tensors and forward methods before mutation, applies the fault specified by C, and restores the original state after evaluation. The context-manager scope prevents carryover effects across configurations. Before any statistical comparison, we apply a structural verification check. For static faults, we verify that the parameter difference is restricted to params(u, ℓ) and that its magnitude matches σ within a relative tolerance of 10−6 to account for floating-point rounding. For dynamic faults, we verify that hooks or wrappers attach only at the intended module call sites and that the original forward function is restored after the context manager exits. We also verify execution completeness by confirming that each configuration produces the required instrumentation logs. Configurations that fail any check are excluded as structurally invalid. 3.3

Subject Models and Tasks

We use four encoder-only and three decoder-only transformer models spanning 66M–125M parameters, 6–12 layers, and both masked and causal language modeling objectives (Devlin et al., 2019; Liu et al., 2019; Sanh et al., 2019; Radford et al., 2019; Black et al., 2021). We limit models to at most 125M parameters to keep all paired runs feasible within our compute budget. Encoders are fine-tuned on five GLUE classification tasks (Wang et al., 2019) (accuracy as task metric); decoders are fine-tuned on four language-modeling corpora (Paperno et al., 2016; Marcus et al., 1993; Merity et al., 2016; Gokaslan & Cohen, 2019) (log-perplexity as task metric). Table 3 summarizes the models and tasks. All models use the standard Hugging Face Transformers implementations and default tokenizers. Table 3: Models and Tasks used to construct the DEFault-bench Architecture Model

Encoder

Parameters Layers Tasks

BERT-base (Devlin et al., 2019) DistilBERT (Sanh et al., 2019) RoBERTa-base (Liu et al., 2019) DistilRoBERTa (Sanh et al., 2019) GPT-2 (Radford et al., 2019)

110M 66M 125M 82M 124M

DistilGPT-2 (Sanh et al., 2019)

82M

GPT-Neo-125M (Black et al., 2021)

125M

Decoder

12 6 12 6 12

SST-2, QNLI, RTE, MRPC, QQP SST-2, QNLI, RTE, MRPC, QQP SST-2, QNLI, RTE, MRPC, QQP SST-2, QNLI, RTE, MRPC, QQP LAMBADA, PTB, WikiText-2, OpenWebText 6 LAMBADA, PTB, WikiText-2, OpenWebText 12 LAMBADA, PTB, WikiText-2, OpenWebText

Encoder models are fine-tuned on GLUE classification tasks (Wang et al., 2019): SST-2 (sentiment), QNLI (natural language inference), RTE (textual entailment), MRPC (paraphrase detection), and QQP (duplicate question detection). Decoder models are fine-tuned using a language modeling objective on LAMBADA (Paperno et al., 2016) (word prediction), PTB (Marcus et al., 1993) (language modeling), WikiText-2 (Merity et al., 2016) (language modeling), and OpenWebText (Gokaslan & Cohen, 2019) (language modeling). All models use the standard Hugging Face Transformersa implementations and default tokenizers for each checkpoint. a

https://github.com/huggingface/transformers

We fine-tune each model with a single shared setup across tasks (AdamW (Loshchilov & Hutter, 2019) with linear warmup, mixed-precision) (Devlin et al., 2019; Mosbach et al., 2021) so that observed differences can be attributed to the injected fault rather than to hyperparameter variation. 12

Each configuration is run under five fixed seeds S = {42, 123, 456, 789, 101112} (Mosbach et al., 2021), the smallest design that admits an exact one-sided sign-flip permutation test at α = 0.05 (smallest attainable p-value 1/25 ≈ 0.031). Layer indices and severity levels low, medium, high are sampled uniformly to avoid concentration on any single layer or magnitude. After structural verification, we obtain 3,739 single-fault configurations: 1,891 for encoders and 1,848 for decoders, with all categories represented. Producing the dataset required 37,390 paired clean and faulty finetuning runs (∼18,600 GPU-hours on NVIDIA A100 and H100 GPUs). Full hyperparameter tables appear in Section A. 3.4

Mutant Validation

We adopt the statistical mutation-killing framework of Humbatova et al. (2021). The original network N and a mutant M are each retrained n times under matched random seeds, and the resulting performance distributions AN (TestS) = ⟨AN1 , . . . , ANn ⟩ and AM (TestS) = ⟨AM1 , . . . , AMn ⟩ are compared on a held-out test set TestS. The predicate isKilled decides whether the mutant is killed: (

isKilled(N, M, TestS) =

true false

if p-value AN (TestS), AM (TestS) < α, otherwise.

(6)

We use accuracy as the test-set performance metric for encoder classification tasks and logperplexity for decoder language modeling tasks (Jahan et al., 2025a). We set n = 5 matched seeds and α = 0.05. Humbatova et al. (2021) use a generalized linear model with n = 20 retrainings paired with Cohen’s d effect size; we instead use a one-sided paired sign-flip permutation test (Good, 2005) on the per-seed deltas, since seed-paired deltas are exchangeable under the null and five matched seeds are the smallest design that admits an exact one-sided test at α = 0.05. The smallest attainable p-value with five paired comparisons is 1/25 ≈ 0.031. We estimate false positives by applying isKilled to pairs of clean-vs-clean runs. Following Humbatova et al. (2021), we summarize each mutation operator’s effectiveness using the mutation score: MS(MO) =

|{c ∈ MO : isKilled(N, Mc , TestS)}| |MO|

(7)

where MO is the set of injected configurations of a mutation operator. The overall mutation score is the average of MS(MO) across all operators. For each killed mutant we attach a label y = (u, f, v, ℓ, σ) that records the target unit, fault category, fault variant, affected layer set, and severity. We omit m and t because each instance is computed within one model–task context. A surviving mutant, for which isKilled returns false, is still an injected fault, but its measured effect does not pass the killing test (p > α). We therefore use surviving mutants as negative examples only for Level 1 detection and exclude them from Level 2 and Level 3 training. Algorithm 1 summarizes the per-configuration dataset construction loop. Trivial mutants. A mutant is trivial when its effect is so large that almost any training run reveals it (Humbatova et al., 2021). These mutants are easy to detect because they often cause obvious failures, but they provide limited evidence for distinguishing one fault category from another (Humbatova et al., 2021). We therefore sample severity uniformly across {low, medium, high} to avoid concentrating the benchmark on overly aggressive faults. 13

Algorithm 1 Per-configuration fault injection and validation Require: Clean model θ, configuration C, seeds S Ensure: Label isKilled, instance feature vector x 1: θ ′ ← Inject(θ, C) ▷ static or dynamic mutation ′ 2: if VerifyStructural(θ, θ , C) = false then return discard 3: end if 4: for s ∈ S do 5: acs ← TrainAndEvaluate(θ, s); afs ← TrainAndEvaluate(θ′ , s) 6: end for 7: p ← SignFlipPermTest({acs }, {afs }) ▷ one-sided, paired 8: killed ← [ p ≤ α ] ▷ Equation (6) c f 9: x ← Aggregate({as }, {as }) ▷ clean-to-faulty deltas 10: return (killed, x)

3.5

Benchmark Statistics

We applied isKilled (Equation (6)) to all 3,739 structurally valid configurations and computed the mutation score (Equation (7)) per category and per architecture, following Humbatova et al. (2021). Table 4 reports the per-category mutation scores. Macro-averages assign equal weight to each model–task pair, while configuration-weighted totals count individual mutants directly. Table 4: Mutation scores by fault category and architecture under isKilled Encoder

Decoder

Category

Killed Surviving Killed Surviving

Variant Score QKV Positional FFN Residual LayerNorm Embedding Output Masking Kernel KV Cache

100.0% 93.8% 93.0% 89.3% 85.1% 84.5% 83.5% 80.9% 56.9% 56.8% 51.6% —

Macro-average 79.6% Configuration-weighted total 84.5%

0.0% 6.2% 7.0% 10.7% 14.9% 15.5% 16.5% 19.1% 43.1% 43.2% 48.4% —

28.8% 17.5% 26.5% 48.2% 58.9% 42.0% 54.1% 54.7% 25.9% 68.5% 68.0% 5.6%

71.2% 82.5% 73.5% 51.8% 41.1% 58.0% 45.9% 45.3% 74.1% 31.5% 32.0% 94.4%

20.4% 41.3% 15.5% 63.9%

58.7% 36.1%

Counted over individual mutants, isKilled returns true for 1,598 of 1,891 encoder configurations (84.5%) and 1,180 of 1,848 decoder configurations (63.9%). Encoder mutation scores are consistently high, with Variant at 100% and Score, QKV, Positional, FFN, Residual, LayerNorm, and Embedding all above 80%. Output, Masking, and Kernel show the lowest encoder mutation scores. For decoders, mutation scores are more variable: Masking and Kernel achieve the highest values (68.5% and 68.0%), while KV Cache shows the lowest (5.6%) because cache-related perplexity shifts fall close to the seed-to-seed variance of the LAMBADA corpus (Paperno et al., 2016). Per-(model, task) mutation scores are reported in Figure 17. We retain all 3,739 structurally valid configu14

rations as labeled instances. Killed mutants are labeled faulty and surviving mutants are labeled correct. Table 5 summarizes the resulting dataset composition. Table 5: Composition of DEFault-bench Metric

Encoder

Decoder

Total Faulty Correct

1,891 1,598 293

1,848 1,180 668

Category

Encoder

Decoder

Variant Score QKV Positional FFN Residual

24 278 247 180 288 130

51 142 184 179 203 167

Metric

Encoder

Decoder

11 40

12 45

Category

Encoder

Decoder

LayerNorm Embedding Output Masking Kernel KV Cache

210 176 192 70 96 —

185 165 180 126 63 203

Fault categories Root causes

Faulty instances are killed mutants; correct instances are surviving mutants. Fault-category counts include both faulty and correct instances; Level 2 and Level 3 evaluation use only the faulty subset.

Each instance carries labels at three levels of granularity. At Level 1 (fault detection), the label is binary, faulty or correct. At Level 2 (fault categorization), the label is the injected fault category. Encoders use 11 categories: Score, Positional, FFN, QKV, LayerNorm, Residual, Masking, Embedding, Kernel, Output, and Variant. Decoders use the same 11 plus KV Cache. At Level 3 (root-cause diagnosis), the label is the injected root cause within the fault category. Each category contains 2–7 root causes, and each root cause belongs to exactly one fault category. The encoder and decoder subsets are imbalanced. The two architectures differ in their proportion of faulty and correct instances, and the fault categories (11 on encoders, 12 on decoders) and their root causes also have different sample sizes. Section 5 describes the evaluation criteria and class-weighting strategy used to address this imbalance.

DEFault++ Diagnostic Technique

4.1

Overview

DEFault++ takes as input the labeled set of 3,739 averaged instances from the benchmark constructed in Section 3, where each instance records the runtime measurements collected for one configuration. The three-level hierarchy from Section 1 is realized as follows. Level 1 uses the full feature representation for binary fault detection. Only instances predicted as faulty proceed to Level 2, which assigns one of the architecture-specific fault categories (11 for encoders, 12 for decoders). Level 3 predicts the root cause within the Level 2 predicted category through prototypical classification (Snell et al., 2017) in an embedding space aligned with the transformer’s Fault Propagation Graph. The same embedding space also produces feature-group importance scores for each diagnosis. Figure 8 shows the three-level hierarchy. We organize this section into the Fault Propagation Graph (Section 4.2), the feature representation (Section 4.3), the hierarchical diagnostic model (Section 4.4), the training objective (Section 4.5), and the FPG-based explanation (Section 4.6). 15

4.2

Fault Propagation Graph

The fault categories and root causes defined in Section 3 identify where a fault originates, but they do not describe how a fault in one component affects other components during forward computation or backward gradient flow. A stale QKV projection, for example, not only corrupts the projection output but also shifts downstream attention scores, attention weights, and the residual stream. To capture these inter-component dependencies, we define the Fault Propagation Graph (FPG), a directed graph derived from the transformer forward pass, backward pass, architectural configuration, and autoregressive execution. The FPG serves two purposes in DEFault++: it structures the feature representation by mapping feature groups to transformer components (Section 4.3), and it provides the adjacency matrix for message passing in the diagnostic model (Section 4.4.2). Prior DL fault-diagnosis techniques use graph representations for different purposes. NeuraLint (Nikanjam et al., 2021) encodes a typed meta-model graph for pre-training structural checks, while FL4Deep (Morovati et al., 2024) ranks root causes using a static and dynamic knowledge graph. In contrast, the FPG models inter-component fault propagation during transformer training. We test whether the FPG structure matters in RQ3 (Section 5.6) by comparing it against rewired and random graphs of the same density. We distinguish seven dependency mechanisms. Mechanisms one to three (M1–M3) describe local forward propagation. Mechanism four (M4) captures repeated forward propagation across stacked layers. Mechanism five (M5) captures backward gradient dependencies. Mechanism six (M6) captures architecture-wide interventions that affect multiple components at once. Mechanism seven (M7) captures temporal dependencies introduced by decoder caching. The current FPG collapses these mechanisms into unlabeled directed edges. The graph is deterministic since it mirrors the fixed computational graph of the transformer. Each edge encodes a structural dependency through which a fault can propagate. Whether a specific fault produces an observable downstream effect depends on its magnitude, on intervening nonlinear operations (detailed below), and on the parameter values learned during training. The FPG therefore models possible dependency pathways within transformer architectures, with edges derived from dependencies in the transformer forward and backward pass. Section 10 contrasts the FPG with prior graph-based representations in DNN fault analysis. We define the FPG G = (V, E) where each node v ∈ V represents a transformer component. The component types are: embedding, positional encoding, QKV projection, score computation, attention masking, attention weights, attention output, residual connections, LayerNorm, FFN, and output head. Residual connections and LayerNorm appear independently for each sublayer (attention and FFN). Decoders additionally include a KV cache node. Each directed edge (vi , vj ) ∈ E indicates that vj depends on vi under the chosen computational, execution, or architectural abstraction, so a fault at vi can reach vj . An edge (vi , vj ) is included when a fault at vi can directly affect the output of vj through forward data flow, backward gradient flow, residual identity flow, autoregressive state reuse, or architecture-wide intervention under the chosen transformer abstraction. The FPG models only transformer components and their dependencies. Not all diagnostic feature groups are FPG nodes. Model-wide context groups and the cross-layer observation group participate in the group-level graph through self-loops only (Table 7). By analyzing the transformer’s forward and backward equations, we identify seven dependency mechanisms. Let δx denote a perturbation to variable x.

1. M1: Forward sequential propagation. If B = f (A), then δB = (∂f /∂A) δA. Perturbation propagates along the data flow path: embedding → projections → scores → masking → weights → output → residual → LayerNorm → FFN → residual → output head. 2. M2: Simultaneous propagation. QKV projections feed both score computation (via Q, K) and attention output (via V ) simultaneously. A single projection fault therefore perturbs scores and values at the same time. In decoders, K and V additionally enter the cache. 3. M3: Residual bypass. In h = x + MHA(x), a perturbation δx appears in h with unit gain through the skip path. The skip connection therefore carries any change at x to the residual output even when the sublayer transformation reduces it, although subsequent normalization may reshape it. 4. M4: Cross-layer propagation. The residual stream provides repeated identity paths across stacked layers (Roquet et al., 2024). A change at layer ℓ can therefore reach later representations, although intervening normalization and nonlinear operations may reduce or reshape it. 5. M5: Backward pass propagation. A fault that changes the loss L can change ∂L/∂θi for parameters whose computation contributes to that loss term, which can affect weight updates across multiple components during training. 6. M6: Architecture-wide intervention. Some fault types inherently affect multiple components at once. A variant fault (e.g., single-head instead of multi-head attention) simultaneously changes the attention computation pattern, parameter shapes, and kernel dispatch. Unlike Mechanisms 1–5, this is not a propagation relation but a joint structural effect in which a single architectural change alters multiple components at the same time. 7. M7: Cache propagation across time steps (decoder only). The KV cache stores K and V across generation steps. A cache fault affects the current token and all future tokens. Three nonlinear operations limit propagation at specific edges. Softmax maps deviations into [0, 1] with a sum-to-one constraint, which reduces large-magnitude shifts. LayerNorm (Ba et al., 2016) rescales outputs to zero mean and unit variance, which removes additive shifts. Activation functions (e.g., ReLU, GELU) suppress changes in their saturation regions and pass them in their linear regions. We annotate these limiting operations on the corresponding edges of the FPG. The seven mechanisms fall into three classes (see Table 6). Propagation relations (Mechanisms 1, 2, 3, 5, 7) describe how a fault at one component directly reaches another through forward data flow, residual identity, backward gradient flow, or autoregressive state reuse. Recursive composition (Mechanism 4) is the repeated application of propagation relations 1 and 3 across stacked layers. Structural intervention (mechanism six) is a joint structural effect in which a single architectural change alters multiple components rather than propagating from one to another. Relation-typed message passing that distinguishes these three classes is a natural extension. For use in the diagnostic model, we collapse the component-level FPG to a group-level adjacency matrix. Each transformer component maps to the feature group that measures it, as shown in Table 7. A group-level edge exists if any component in the source group has a propagation edge to any component in the target group. Components that map to the same group (e.g., attention masking, attention weights, and attention output all map to the Attention group) are merged. Structural groups receive FPG-derived neighbor edges. The cross-layer observation group (Representation drift) and model-wide context groups (Training dynamics, Validation performance) participate through self-loops only: they undergo the same 2

M3, M4, M5, and M6 denote dependency patterns that expand to multiple directed edges.

Table 6: Edge derivation for the Fault Propagation Graph (FPG) Source

Target

Mechanism Class

Scope

Details

Embedding Positional QKV Proj. QKV Proj. QKV Proj. Score Comp. Attn. Mask Attn. Weights Attn. Output Residual input Residual (attn)

QKV Projection QKV Projection Score Comp. Attn. Output KV Cache Attn. Mask Attn. Weights Attn. Output Residual (attn) Residual (attn) LayerNorm (attn) FFN

Forward prop. (M1) Forward prop. (M1) Simultaneous (M2) Simultaneous (M2) Simultaneous (M2) Forward prop. (M1) Forward prop. (M1) Forward prop. (M1) Forward prop. (M1) Residual bypass (M3) Forward prop. (M1)

Enc/Dec Enc/Dec Enc/Dec Enc/Dec Dec Enc/Dec Enc/Dec Enc/Dec Enc/Dec Enc/Dec Enc/Dec

Q, K, V = WQ,K,V E[x] Position added to embedding before projection √ S = QK ⊤/ dk ; Q, K from projection Attn = softmax(S) V ; V from projection K, V stored in cache at each step S′ = S + M α = softmax(S ′ ) Attn = α V h = x + Attn(x) Skip connection: δx passes with unit gain x̂ = LN(h)

Forward prop. (M1)

Enc/Dec

FFN input is post-norm representation

Residual (FFN) Residual (FFN) LayerNorm (FFN) Output Head

Forward prop. (M1) Enc/Dec Residual bypass (M3) Enc/Dec Forward prop. (M1) Enc/Dec

h′ = h + FFN(x̂) Skip connection: δh passes with unit gain ĥ′ = LN(h′ )

Forward prop. (M1)

Final representation enters output head

Layer ℓ+1 All components Attn. Weights Multi components

Cross-layer (M4) Enc/Dec Backward prop. (M5) Enc/Dec Cache-time (M7) Dec Intervention (M6) Enc/Dec

LayerNorm (attn) FFN Residual input Residual (FFN) LayerNorm (FFN) Layer ℓ residual All components KV Cache Variant config.

Enc/Dec

Repeated M1+M3 across stacked layers ∂L/∂θi couples all parameters Cached K, V affect future-step attention Architectural change alters multiple components

learned transformation as structural groups but do not receive information from neighboring groups. Groups that map to multiple components inherit all edges of their constituent components. Figure 6 shows the group-level adjacency matrix Â for the decoder diagnostic model (G = 13). The near-diagonal sparsity reflects the sequential data flow through the transformer block. The off-diagonal entries (e.g., QKV→Cache, Cache→Attention, FFN→Residual) capture the feedback and branching paths that the FPG encodes. Non-structural groups (Representation drift, Training dynamics, Validation performance) appear as isolated self-loops in the lower-right block. 4.3

Feature Representation

The Fault Propagation Graph defines which transformer components can affect each other, but the diagnostic model requires quantitative measurements of those components during training. Generic DNN fault features such as loss trajectories, gradient statistics, and activation counts do not distinguish transformer fault categories. Corrupted QKV projections, missing causal masks, and misconfigured positional encodings leave end-task metrics untouched but produce characteristic patterns in attention-level and component-level measurements. We therefore define a transformer-specific feature space that targets attention distributions, projection alignments, and residual-stream behavior (see Figure 7). Table 8 lists the feature groups, their metric types, and the collection branch through which each group enters the feature-construction process. A. Layer-internal metrics (Cint , collected per layer at each training step). 18

Embedding

Positional store 𝐾 ,𝑉

QKV Projection

KV Cache

V 𝑆 ′ =𝑆+𝑀

residual skip

Score Computation

Attention Masking softmax

cached 𝐾 ,𝑉

Attention Weights Attention Output

Attention Sub-graph

×𝑁 layers (M4)

⊕ FFN Sub-graph

LayerNorm residual skip

layer norm

Feed-Forward Network (FFN) activation

⊕ LayerNorm layer norm

Output Head ∇𝜃 ℒ propagates in reverse along all edges ▶ M1: Forward - - ▶ M2: Simultaneous - - ▶ M7: Cache-time Computation

▶ M3: Residual

{ M4: Cross-layer (brace) Normalization

⋅ ⋅ ⋅ ▶ M5: Gradient

M6: Intervention (not drawn)

Residual Junction (⊕)

I/O

Auxiliary

softmax , layer norm , activation = nonlinear operations that limit fault propagation at the annotated edge

Figure 5: Fault Propagation Graph (FPG) for transformer architectures. The loopback denotes M4; the dashed KV Cache return denotes decoder-only M7. M6 is omitted because it affects multiple components simultaneously. Table 7: Feature-group roles and message passing in DEFault++ Feature Group

Role

Scope

Attention Score FFN output LayerNorm Residual stream QKV alignment Embedding Positional Output Cache (decoder)

Structural Structural Structural Structural Structural Structural Structural Structural Structural Structural

Attn. masking, weights, output FPG edges Score computation FPG edges FFN FPG edges LayerNorm FPG edges Residual connections FPG edges QKV projection FPG edges Embedding FPG edges Positional encoding FPG edges Output head FPG edges KV cache FPG edges

Representation drift

Cross-layer observation Inter-layer residual path

Training dynamics Model-wide context Validation performance Model-wide context

Model-wide optimization Model-wide output quality

Message Passing

Self-loop only Self-loop only Self-loop only

Tr ain in gD yn Ta . sk Pe rf .

pr .D rift Re

he Ca c

sid Re

nt io At te

re Sc o

lS tre am ye rN or m FF N Ou tp ut Ou tp ut

m en Al ign

V QK

Po sit i

in g

Target group

Embedding Positional

Structural

QKV Alignment Score

Source group

Attention Residual Stream LayerNorm FFN Output

Non-structural

Output Cache Repr. Drift Training Dyn. Task Perf. FPG edge

Self-loop

Figure 6: Group-level adjacency matrix Â for the decoder diagnostic model

A1-Attention. We measure seven properties of the post-softmax attention distribution at each sampled layer (six for encoders, seven for decoders). –Attention entropy (Hattn ) computes the Shannon entropy of the attention distribution, averaged across heads and query positions (Voita et al., 2019). Low entropy indicates that a head concentrates on a single position. High entropy indicates a near-uniform distribution. –Padding attention mass (PadMass) computes the fraction of attention directed at padding tokens. Non-zero padding mass indicates a masking fault. a ·a

–Inter-head cosine similarity (HeadSim) computes pairwise cosine similarity sim(ai , aj ) = ∥ai ∥i ∥aj j ∥ between the flattened attention patterns of heads i and j within each layer. High pairwise similarity indicates redundant heads. 20

Table 8: Feature groups, metric types, and feature dimensions in DEFault++ Feature Group

Metric Types

Attention

Attention entropy, padding attention mass, inter-head cosine similarity, head utilization, attention 6/7a rank, cross-example leakage, future attention mass (decoder only) Pre-softmax score 1

Layer-internal metrics (Cint , collected at each training step, per layer)

Score FFN output

FFN output norm

LayerNorm

Scale parameter norm, post-norm distribution

Residual stream

Residual cosine similarity

Repr. drift

CKA layer similarity

QKV alignment

Q–K sim., Q–V sim., K–V sim.

Optimization

Gradient and update features (gradient norm, update ratio, update activity)

Embedding

Embedding norm, token-level variance

Positional

Positional sensitivity

Training dyn.

Loss trajectory, gradient noise scale, step time, peak memory allocation

Output

Prediction confidence, output entropy, margin statistics

Cache

Cache hidden similarity, cache distribution divergence

Validation perf.

Task accuracy / perplexity, calibration error

Gradient metrics (Copt , collected at each training step) 21

Behavioral metrics (Ctrain , collected at each training step)

Validation metrics (Ceval , collected at epoch end or validation checkpoints) 2

a 6 for encoders, 7 for decoders (future attention mass applies only to decoders).

–Head utilization is the fraction of heads whose attention entropy exceeds a minimum threshold, indicating the head attends to more than one position. Variant faults that reduce the effective head count lower this fraction (Voita et al., 2019; Clark et al., 2019). –Attention rank quantifies the effective dimensionality of each head’s attention weight matrix as the exponential of the Shannon entropy of its normalized singular value distribution (Roy & Vetterli, P 2007): EffRank(Ai ) = exp(− nk=1 σ̃k log σ̃k ), where σ̃k is the k-th singular value normalized to sum to one. A rank near 1 indicates a low-dimensional attention pattern dominated by a single singular component. A rank approaching n indicates a broader singular value spectrum across query positions. Variant faults that reduce effective multi-head capacity lower the metric, while faults that introduce more diffuse high-dimensional structure may raise it. –Cross-example leakage measures the total attention weight flowing between unrelated examples in the same batch, averaged over heads and layers (Vaswani et al., 2017; Dao et al., 2022). Under correct masking, this quantity is zero. Non-zero values indicate that the attention mechanism attends across example boundaries, which is a direct indicator of masking faults. –Future attention mass (decoder only) measures the total attention weight assigned from each query position to future key positions, averaged over heads and layers (Vaswani et al., 2017). Under correct causal masking, this quantity is near zero, and causal-mask violations increase it systematically.

A2-Score.

We record the raw attention logits before the softmax operation (Vaswani et al., 2017): 21

s = QK ⊤ / dk p

(8)

At each layer, we reduce the full query-key score matrix to a single scalar by computing its mean across all positions (Dong et al., 2021). This within-layer reduction precedes the cross-layer aggregation (Aℓ ) described above. High score variance across layers indicates that attention sharpness differs substantially between early and late layers, which can result from softmax saturation and reduced gradient flow (Dong et al., 2021; Zhai et al., 2023b). Pre-softmax scores are sensitive to Score and Positional faults because both fault types directly alter this computation. A3-FFN output. We record the ℓ2 norm of the feed-forward sublayer outputs at each layer, computed as ∥FFN(x)∥2 per token and averaged across the batch. Abnormally large norms indicate instability in the FFN transformation. Near-zero norms indicate that the FFN sublayer contributes little to the residual stream, which can occur when activation functions suppress most dimensions. A4-LayerNorm. We record two properties of layer normalization (Ba et al., 2016) at each layer. Scale parameter norm is the Frobenius norm ∥γ∥F of the learned scale vector. Post-norm distribution captures the moments of the normalized output: x̂ = γ ⊙

x−µ +β σ+ϵ

(9)

A LayerNorm fault disrupts the rescaling and produces post-norm moments or γ magnitudes outside the range seen in clean training runs. A5-Residual stream. residual sublayer:

We compute the cosine similarity between the input and output of each res_simℓ =

xℓ · hℓ ∥xℓ ∥ ∥hℓ ∥

(10)

where xℓ is the sublayer input and hℓ = xℓ + f (xℓ ) is the residual output at layer ℓ. In a correctly functioning transformer, the skip connection preserves the input direction while the sublayer f adds a bounded change, so cosine similarity remains close to 1.0. A Residual fault that corrupts the skip-connection path reduces this similarity. This metric measures the effect of a fault within a single sublayer (input vs. output of the same residual block). A6-Representation drift. We compute Centered Kernel Alignment (CKA) (Kornblith et al., 2019) between consecutive layer representations: CKA(Hℓ , Hℓ+1 ) =

⊤ H ∥2 ∥Hℓ+1 ℓ F ⊤ ⊤ ∥Hℓ Hℓ ∥F ∥Hℓ+1 Hℓ+1 ∥F

(11)

where Hℓ ∈ Rn×d is the centered representation matrix at layer ℓ. CKA measures similarity independent of rotation and isotropic scaling. A fault that propagates across multiple layers causes decreasing CKA between adjacent layers. We use CKA rather than direct cosine similarity because CKA is invariant to orthogonal transformations of the representation space. Unlike the Residual stream metric, which measures how a sublayer’s output differs from its input, CKA measures cumulative representational change between consecutive layers and therefore captures faults that propagate beyond the originating sublayer. 22

A7-QKV alignment. We compute pairwise cosine similarity between the query, key, and value projection outputs at each sampled layer: sim(Q, K) =

Q·K , ∥Q∥ ∥K∥

sim(Q, V ) =

Q·V , ∥Q∥ ∥V ∥

sim(K, V ) =

K ·V ∥K∥ ∥V ∥

(12)

where Q = WQ x, K = WK x, and V = WV x are the projection outputs for input x (Vaswani et al., 2017). A fault that corrupts one projection weight matrix shifts its output direction relative to the other two, producing a measurable drop in pairwise similarity.

B. Gradient metrics (Copt , collected per component at each training step). For each parameterized component (e.g., Attention, QKV, FFN, LayerNorm, Embedding), we compute three gradient-level measurements inspired by prior DNN fault detection work (Jahan et al., 2025b; Cao et al., 2022). Gradient norm computes the ℓ2 norm of the gradient vector for all parameters belonging to that component: gc =

∥∇θ L∥2

(13)

θ∈Θc

where Θc is the parameter set of component c. Update ratio measures the relative magnitude of weight change between consecutive training steps: (t)

uc =

(t−1)

∥Wc − Wc

∥F

(t−1) ∥Wc ∥F + ϵ

(14)

(t)

where Wc is the weight matrix of component c at step t. Update activity is a binary flag indicating whether the gradient norm exceeds a minimum threshold (gc > 10−6 ), which identifies components that have stopped learning. These 15 component-level measurements (three per component) together with six global gradient statistics form the Copt = 21 gradient metrics in Table 8. The Output group uses behavioral metrics only and does not add a sixth component-routed gradient set. Gradient metrics do not form a separate FPG node. Each component-level gradient measurement is appended to the feature group of its corresponding component (Attention, QKV alignment, FFN output, LayerNorm, Embedding). The six global gradient statistics are appended to the Training dynamics group. The message-passing graph therefore contains 12 groups for encoders and 13 for decoders, as listed in Table 7.

C. Behavioral metrics (Ctrain , collected at each training step). C1-Embedding. We record two properties of the input embedding vectors. Embedding norm P is the mean ℓ2 norm n1 ni=1 ∥ei ∥2 across all n input tokens, where ei = E[xi ] is the embedding of token xi . Token-level variance is Var({e1 , . . . , en }) computed across the hidden dimension. An Embedding fault that corrupts the embedding matrix E produces abnormal norms or collapses the variance across tokens. We compute both metrics on the first hidden state before the first transformer block. 23

C2-Positional. We measure positional sensitivity as the model’s response to a fixed non-zero shift in position indices on a held-out batch. In all experiments, we set ∆ = 1. Let Pmax denote the model’s maximum supported position index, and let I(X) be the set of valid non-padding token positions in batch X such that i + ∆ < Pmax . For each i ∈ I(X), we compare the hidden state at position i under the original indices with the hidden state at the same token position under the shifted indices. We define positional sensitivity as

Spos (X) =

L X X 1 (∆) hℓ,i (X) − hℓ,i (X) 2 L |I(X)| ℓ=1 i∈I(X)

(15)

where L is the number of instrumented layers. Tokens that would exceed the valid index range after shifting are excluded from the average; we do not clip, wrap, or reuse position indices. Larger values indicate stronger dependence of the model’s representations on positional information. Positional faults that omit, misalign, or truncate positional encoding shift this quantity away from the clean baseline. C3-Training dynamics. We record four optimization-level and infrastructure-level metrics at each training step. Loss trajectory is the scalar training loss per step. Gradient noise scale measures the ratio of gradient variance to the squared gradient norm (McCandlish et al., 2018): GNS =

Var(∥∇L∥) E[∥∇L∥]2

(16)

computed over a rolling window of 20 training steps. High gradient noise scale indicates that individual batches produce inconsistent gradient directions. Step time is the wall-clock duration of each training step. Sudden increases in step time can indicate memory pressure or scheduling faults. Peak memory allocation records the peak GPU memory reported by the CUDA allocator during each training step (NVIDIA, 2021; Chakrabarti et al., 2012). A forced fallback to a nonoptimized kernel or an unintended data-layout change can increase peak memory without raising an explicit error (Dao et al., 2022). We include peak memory alongside step time because both are infrastructure-level measurements that detect kernel and scheduling faults. C4-Output. We record three properties of the output head at each training step. Prediction P (i) confidence computes the mean maximum softmax probability N1 N i=1 maxc pc across all samples in the batch. Output entropy computes the mean Shannon entropy of the output distribution P P (i) (i) − N1 N i=1 c pc log pc . Margin statistics measure the gap between the correct-class logit and the strongest competing logit. For encoder classification tasks, we compute these metrics over the classification logits per batch sample. For decoder language modeling tasks, we average over nonpadding tokens within each sequence, where margin measures the gap between the logit of the true next token and the highest competing logit. Low confidence, high entropy, or negative margins indicate that the output head does not produce well-separated predictions, whether because of upstream faults or direct output-projection faults. C5-Cache (decoder only). We measure two properties of the key-value cache used during autoregressive generation. Cache hidden similarity computes the cosine similarity sim(Kcached , Krecomputed ) between cached and recomputed key states at each generation step. High similarity indicates a consistent cache. Low similarity indicates stale or corrupted cached states. 24

Cache distribution divergence computes the KL divergence: DKL (Pcache ∥Pfresh ) =

Pcache (v) log

Pcache (v) Pfresh (v)

(17)

between the next-token distribution produced using the cache (Pcache ) and the distribution produced by fresh computation (Pfresh ). Non-zero divergence indicates that cached states produce different predictions, which is a direct indicator of KV Cache faults.

D. Validation metrics (Ceval , collected at epoch end or validation checkpoints). We record two measurements at epoch end or at scheduled validation checkpoints. Task accuracy is classification accuracy for encoder tasks and perplexity for decoder language modeling tasks. Calibration error is the expected calibration error (Guo et al., 2017):

ECE =

B X |b| b=1

acc(b) − conf(b)

(18)

where B is the number of confidence bins. Faults that degrade downstream performance produce abnormal values in these metrics. Faults that shift internal behavior without affecting validation performance (task-invisible faults) produce normal values in this group but abnormal values in other groups. Since these metrics are computed at epoch boundaries rather than at each training step, they enter the feature-construction process after the epoch-level summary stage (Table 8).

Figure 7: DEFault++ feature construction (raw metric to the fixed-length feature vector)

4.3.1

Aggregation & Feature Vector Construction

We classify each metric into one of four collection branches based on its granularity (see Figure 7). Prior work has shown that attention distributions (Voita et al., 2019; Clark et al., 2019), representation geometry (Ethayarajh, 2019; Kornblith et al., 2019), normalization parameters (Ba et al., 25

2016; Xiong et al., 2020), and residual propagation (Roquet et al., 2024) vary qualitatively across layers. We therefore measure Cint = 15 (encoder) or Cint = 16 (decoder) layer-internal metric types at each layer independently (Table 8). The decoder count is higher because future attention mass applies only to decoder architectures. The remaining metrics are collected at the step level or epoch level. Gradient metrics (Copt = 21) record gradient norm, update ratio, and update activity across parameterized components and global gradients. Behavioral metrics (Ctrain = 10 for encoders, 12 for decoders) measure model-wide properties such as training loss, output-head behavior, positional sensitivity under controlled position shifts, and peak memory allocation. Validation metrics (Ceval = 2) record accuracy or perplexity and calibration error at epoch boundaries. To create a fixed-length representation regardless of model depth, we aggregate per-layer layerinternal metrics into five summary statistics (Aℓ = 5: early mean, early standard deviation, mid mean, mid standard deviation, and final-layer value). A 6-layer and a 12-layer model therefore produce the same number of features per group. Step-level metrics (gradient and behavioral) skip this aggregation. After layer aggregation produces dstep = Aℓ Cint + Copt + Ctrain step-level features, we apply two further aggregation stages to convert variable-length training runs into a fixed-length vector. Epoch-level summary (Ae = 3). Following prior work on step-level training metrics (Ben Braiek & Khomh, 2023; Wardat et al., 2021), we aggregate step-level values within each training epoch into three statistics: epoch mean, epoch standard deviation, and a burst statistic (the 95th-percentile or maximum value). These statistics capture micro-level training behavior within a single pass over the data. Abnormal within-epoch variance can reveal batch-level issues such as gradient instability (Ben Braiek & Khomh, 2023) or data-complexity mismatches (Wardat et al., 2021) that disappear when only the final epoch value is recorded. Validation metrics (Ceval = 2) enter at this stage because they are collected at epoch end or at validation checkpoints. The epoch-level vector has dimensionality depoch = Ae dstep + Ceval . Training-phase summary (Ap = 5). We partition the epoch sequence into three training phases (early, mid, final) and compute the mean within each phase, a linear-regression slope across phases, and the final-epoch value. Together these five statistics per metric compose Ap = 5. The slope captures the macro-level trajectory of the training process. A fault that causes learning-rate decay to stall (Zhang et al., 2021), or that traps the optimizer in a suboptimal region (Wardat et al., 2021), produces a characteristic slope pattern that a single-epoch snapshot would miss. Final values capture faults whose effect is visible only at convergence, independent of the trajectory shape. Feature vector length. Layer-internal metrics (Cint ) pass through layer aggregation (Aℓ = 5) before the epoch-level and training-phase summaries. Gradient and behavioral metrics skip layer aggregation. Validation metrics enter after the epoch-level summary. The raw feature vector length before filtering is

dfinal = Ap (Ae (Aℓ Cint + Copt + Ctrain ) + Ceval ) . For encoders, Cint = 15, Copt = 21, Ctrain = 10, and Ceval = 2, giving dfinal = 5 × (3 × (5 × 15 + 21 + 10) + 2) = 5 × (3 × 106 + 2) = 5 × 320 = 1600. 26

(19)

For decoders, Cint = 16 because decoder features add future attention mass, and Ctrain = 12 because they add two cache metrics, giving dfinal = 5 × (3 × (5 × 16 + 21 + 12) + 2) = 5 × (3 × 113 + 2) = 5 × 341 = 1705. We apply coefficient-of-variation filtering (CV < 0.01) to remove near-constant columns. The fixed-length feature vector serves as input to the diagnostic model (Section 4.4). 4.4

Diagnostic Model

Figure 8: DEFault++ inference hierarchy for fault detection, categorization, and root-cause diagnosis

Algorithm 2 summarizes the diagnosis and explanation flow for one input instance. It shows the three-level gating logic and the explanation derivation; the shared encoding is defined in Sections 4.4.1 to 4.4.2. 4.4.1

Feature-Group Encoding

We encode each feature group with a dedicated MLP. We keep groups structurally separate so that FPG message passing can mix information across them and the explanation step can examine each group’s embedding independently. We use trainable MLPs rather than a flat classifier because the prototype-distance objective and the contrastive separation loss both require gradients to flow through the group-level weights during training. As encoder and decoder architectures differ in the number of feature groups (G = 12 vs. G = 13), we train separate encoder and decoder diagnostic models. Given the feature vector z ∈ Rd partitioned into G groups {zg1 , . . . , zgG } (see Table 8), each group g is encoded by a dedicated MLP (Equation (20)). hg = MLPg (zg ) ∈ Rh ,

g = 1, . . . , G,

(20)

where h = 32 is the hidden dimension per group (selected via grid search on the inner validation fold). The group embeddings form the matrix H = [h1 ; . . . ; hG ] ∈ RG×h . A projection layer maps the concatenated group embeddings to a shared representation (Equation (21)). zproj = Wproj vec(H) ∈ Re , (21) 27

Algorithm 2 DEFault++ hierarchical diagnosis for one input instance Input: Feature vector x; shared encoder producing group embeddings {hg }G g=1 and projection zproj ; trained prototypes {π c,r } Output: Detection label ŷ; if faulty: fault category ĉ, root cause r̂, and per-group importance scores {wg }G g=1 Level 1: Fault Detection 1: Classify the instance as faulty or clean: ŷ ← Detect(zproj ) 2: if ŷ = clean then

return ŷ 4: end if

▷ clean instances exit here; Levels 2–3 are not called

Level 2: Fault Categorization 5: Assign the fault to a transformer component category: ĉ ← Categorize(zproj ) Level 3: Root-Cause Diagnosis 6: for each root cause r in the predicted category Rĉ do 2 Sum squared distances to prototype r across all feature groups: dr ← G g=1 ∥hg − π ĉ,r,g ∥ 8: end for 9: Select the nearest prototype as the predicted root cause: r̂ ← arg minr∈Rĉ dr

Explanation 10: Identify the nearest alternative root cause: r̃ ← arg minr∈Rĉ \{r̂} dr 11: for each feature group g = 1, . . . , G do

Compute how much group g favors r̂ over r̃: ∆g ← dg (π ĉ,r̃ ) − dg (π ĉ,r̂ ) Normalize positive contributions into importance score: P max(∆g , 0) / g′ max(∆g′ , 0) 14: end for 15: return ŷ, ĉ, r̂, {wg }G g=1 12:

13:

←

where e = 64 is the embedding dimension and vec(H) ∈ RGh denotes the row-major vectorization. Levels 1 and 2 use the pooled projection zproj for detection and categorization. Level 3 uses the group embedding matrix H directly to preserve group-level structure for root-cause separation and group-level explanation (Section 4.6). 4.4.2

Message Passing in the Fault Propagation Graph

We update each group embedding by aggregating information from neighboring groups in the FPG. We construct the group-level adjacency matrix A ∈ RG×G from the component-level FPG (Section 4.2) and the mapping in Table 7. Structural groups receive FPG-derived neighbor edges. The cross-layer observation group (Representation drift) and model-wide context groups (Training dynamics, Validation performance) receive self-loops only, meaning they undergo the learned transformation but do not aggregate information from neighboring groups. We compute updated group embeddings following graph convolutional networks (Kipf & Welling, 2017):

H′ = ReLU Â H Wmsg , 28

(22)

where Â is the row-normalized adjacency matrix (with self-loops) and Wmsg ∈ Rh×h . We use row normalization rather than symmetric normalization because structural feature groups differ in the number of FPG propagation connections. Row normalization prevents high-degree groups from over-averaging their neighbors. The updated group embeddings H′ replace the pre-message-passing H in both the projection and the Level 3 prototype matching. We apply three rounds of message passing, each with a dedicated learnable weight matrix. Three rounds is also the diameter of the FPG: any node can reach any other node within three steps along the propagation edges, so this depth is enough to combine information across the full graph without entering the deeper regime where graph convolutional networks tend to oversmooth (Li et al., 2018). We selected this number through grid search on the inner validation fold (Section B). For example, a QKV projection fault propagates to attention scores and the residual stream according to Mechanisms 1 and 2. Message passing along these edges aggregates that multi-group information into the QKV alignment embedding before classification. The model-wide context groups are transformed independently, without exchanging with neighbors, and contribute through the projection and prototype comparison steps.

4.4.3

Hierarchical Classification

All three classification levels share one encoder (group encoding, FPG message passing, and projection) rather than one per level. Sharing reduces parameters and training cost. Message passing over a 12–13-node graph with 32-dimensional embeddings is negligible relative to the upstream transformer training. The FPG also encodes structural dependencies between transformer components. A fault in one component (e.g., QKV projection) can shift measurements in downstream components (e.g., attention scores, residual stream). Message passing aggregates these cross-component dependencies into each group embedding, which benefits all three levels to varying degrees. Each level adds a task-specific head. Level 1: Fault detection. A two-layer MLP maps zproj to binary logits (faulty vs. correct). Level 2: Fault categorization. A two-layer MLP maps zproj to C-class logits, where C = 11 for encoders and C = 12 for decoders. During training, the class-weighted cross-entropy is computed only over samples labeled faulty. Level 3: Root-cause diagnosis. Level 3 discriminates among the 2–7 root causes within a fault category. Per-root-cause training support ranges from fewer than 10 samples (e.g., Variant encoder) to approximately 80, with an average of 42 per root cause for encoders and 26 for decoders. A standard softmax classifier risks overfitting to majority root causes in this setting. It also gives one prediction score for the whole input, which does not show how each feature group supports the diagnosis. We therefore combine a cross-entropy head with a prototypical classifier (Snell et al., 2017; Kamp & Partee, 1995). The prototypical classifier compares each sample with class-mean embeddings instead of learning a separate weight vector for each class. Its distance score can also be broken down by feature group, which supports the explanation step in Section 4.6. First, a per-category classification head maps zproj to root-cause logits within each category. During training, the cross-entropy loss is computed only over faulty samples whose ground-truth fault 29

category matches the category under training. Negative samples for Level 3 are the other root causes within the same category, not samples from different categories. Second, a prototypical classifier (Snell et al., 2017; Lin et al., 2022) operates in the group embedding space H ∈ RG×h (not the logit space). Distances in H are converted to class logits through the prototype-matching loss (Equation (28)). Let Dc,r denote the set of training samples with category c and root cause r. For each root cause r within category c, we compute a prototype π c,r as the mean group embedding (Equation (23)).

π c,r =

X 1 Hi ∈ RG×h . |Dc,r | i∈D

(23)

c,r

The distance from a sample to a prototype decomposes additively by group (Equation (24)).

d(H, π c,r ) =

G X

∥hg − π c,r,g ∥2 =

G X

dg ,

(24)

g=1

where dg = ∥hg − π c,r,g ∥2 is the squared distance contributed by group g. Level 3 predicts the root cause whose prototype is nearest: r̂ = arg minr∈Rc d(H, π c,r ). Because the distance decomposes additively by group (Equation (24)), the prediction also admits a per-group explanation, which Section 4.6 defines. At inference, Level 3 runs only after Level 2 predicts a fault category, and root-cause diagnosis is restricted to that category. The per-category cross-entropy heads are used during training to provide auxiliary gradient information for the shared encoder. At inference we use the prototype classifier because its distances decompose by group, which supports the group-level explanation in Section 4.6.

Figure 9: DEFault++ training process with shared feature processing and four loss components

4.5

Training Objective

We design a four-term training objective. Two terms are standard cross-entropy heads for fault detection and fault categorization. The other two terms target Level 3 root-cause separation. Level 3 introduces a new training formulation for fault diagnosis that combines supervised contrastive learning (Khosla et al., 2020) with prototype matching (Snell et al., 2017). The contrastive part encourages samples with the same root cause to stay close in the learned space and samples with different root causes to stay apart. The prototype-matching part then compares each sample with the learned prototype of each root cause and assigns the closest match. This formulation differs from prior DL fault detection and diagnosis techniques by using both root-cause separation and prototype-based matching in the same training objective. We test the effect of the separation loss in RQ3 (Section 5.6). DEFault++ uses a hierarchical architecture at inference (Level 1 filters before Level 2, Level 2 filters before Level 3) but trains the shared encoding pathway jointly across all three levels. We adopt joint training with hard parameter sharing (Caruana, 1997; Ruder, 2017) rather than training independent encoders per level. The dataset contains only 3,739 instances and the per-root-cause sample counts are limited (Section 3). Independent training would fragment this limited gradient information across separate encoders. Joint training is feasible because the three tasks are nested. Every Level 3 sample is also a Level 2 and Level 1 sample, and gradient directions across levels reinforce each other rather than opposing each other. The Level 2 loss encourages category-separable embeddings, and the Level 3 separation loss imposes within-category structure. Both also improve Level 1 detection, since a more structured embedding space produces a cleaner binary boundary. Each level uses a dedicated loss, and all losses are computed at the mini-batch level. Level 1 uses binary cross-entropy Ldetect over all samples in the batch. Level 2 uses class-weighted crossentropy Lcat over fault categories, applied to faulty samples only. Level 3 uses two losses jointly: per-category cross-entropy Lrc over root causes, and a root-cause separation loss Lsep that helps separate root causes within each category. Cross-entropy trains per-category heads but does not explicitly structure the embedding space for within-category separation. We therefore define Lsep with two components. The first is a contrastive term that operates on the flattened group embeddings vec(H). For each sample i with category label ci and root-cause label ri , positive pairs share both (ci , ri ), while negative pairs share ci but differ in root cause. Following supervised contrastive learning (Khosla et al., 2020), the contrastive term within category c is (c)

Lctr = −

X 1 exp(sim(hi , hj )/τ ) log P |Pc | (i,j)∈P k:ck =ci exp(sim(hi , hk )/τ )

(25)

where Pc is the set of positive pairs within category c, sim(·, ·) is cosine similarity on the normalized flattened group embeddings, and τ = 0.1 is the temperature. Cosine similarity captures the directional alignment that the contrastive objective requires (Wang & Liu, 2021; Islam et al., 2023). The prototype classifier in Equation (24) uses squared Euclidean distance instead, since prototype matching depends on spatial proximity in the grouped embedding space. We define the joint training objective LDEFault++ in Equation (26). LDEFault++ = Ldetect + α Lcat + λ Lrc + Lsep 31

(26)

where α and λ are level-specific weighting coefficients. During training, we apply the root-cause separation loss Lsep only to faulty samples within their ground-truth fault category. At inference, Level 3 uses the predicted category from Level 2. The separation loss is defined as Lsep = β Lctr + γ Lpm

(27)

where Lctr is the contrastive term from Equation (25), and Lpm is a prototype-matching loss defined as Lpm = −

1 X exp(−d(Hi , π ci ,ri )/τp ) log P |B| i∈B ′ r ∈Rc exp(−d(Hi , π ci ,r′ )/τp )

(28)

where B is the training batch, d(·, ·) is the grouped squared distance from Equation (24), Rci is the set of root causes within category ci , and τp is a temperature parameter. By converting negative prototype distances into class logits, Lpm trains H so that the group-level distance gap (Equation (29)) faithfully reflects the prediction. We set α = 1.0, λ = 1.0, β = 0.5, and γ = 0.3, where β weights the contrastive term Lctr and γ weights the prototype-matching term Lpm in Lsep . We selected all four coefficients via grid search on the inner validation fold (Section B). Training details appear in Section 5. 4.6

FPG-based Explanation

Level 3 predicts the root cause whose prototype is nearest in the grouped embedding space (Section 4.4.3). To explain that prediction, we compare the predicted prototype π c,r̂ with the nearest alternative prototype π c,r̃ , where r̃ = arg minr∈Rc \{r̂} d(H, π c,r ). The distance gap between these two prototypes decomposes additively by group: ∆g = dg (π c,r̃ ) − dg (π c,r̂ ),

g = 1, . . . , G

(29)

where dg (π c,r ) = ∥hg − π c,r,g ∥2 . A positive ∆g means group g is closer to the predicted prototype than to the nearest alternative, so it favors the predicted root cause. A negative ∆g means the group P favors the alternative. The total gap g ∆g equals the difference between the two full prototype distances, confirming that the decomposition is exact. To summarize these contributions for visualization, we normalize the positive contributions into an importance score per group: max(∆g , 0) wg = PG , g ′ =1 max(∆g ′ , 0)

g = 1, . . . , G

(30)

Groups with negative contributions receive wg = 0. The importance scores sum to one and highlight the groups that most support the predicted root cause. For structural groups (Table 7), each wg maps directly to a transformer subsystem, so a large importance identifies the subsystem that most supports the diagnosis after message passing (Wolf et al., 2024). For the cross-layer observation group (Representation drift), a large importance suggests that cross-layer representation drift is part of the diagnostic evidence, rather than a localized component effect. For model-wide context groups (Training dynamics, Validation performance), a 32

large importance indicates model-wide deviation that corroborates the component-level diagnosis but does not localize the fault to a specific subsystem. Because wg is computed from distances in the post-message-passing embedding space, structural groups reflect fault effects as they propagate across components, while model-wide context groups reflect self-transformed embeddings without cross-group mixing. The group ablation in RQ4 (Section 5.7) tests this interpretation by ablating the most important groups and measuring how much the gap decreases.

Evaluation

5.1

Research Questions

We evaluate DEFault++ on DEFault-bench using nested grouped cross-validation. Encoder and decoder architectures are evaluated separately since their feature and label spaces differ. ▶ RQ1 : How effective is DEFault++ in detecting, categorizing, and diagnosing transformer

faults? ▶ RQ2 : How does DEFault++ compare with existing DL fault-detection techniques? ▶ RQ3 : How do message passing in the Fault Propagation Graph and the root-cause separation

loss contribute to diagnostic performance? ▶ RQ4 : How faithful are the FPG-based explanations to the root-cause diagnosis?

5.2

Experiment Setup

We use nested grouped cross-validation with k = 5 outer folds. The encoder subset contains 20 model–task pairs (4 models × 5 GLUE tasks) and the decoder subset contains 12 (3 models × 4 language modeling corpora). All instances from the same model–task pair are kept in the same outer fold to prevent information leakage from correlated traces. Each encoder fold therefore holds out 4 pairs (20 / 5), and each decoder fold holds out 2 to 3 pairs (12 / 5). Per-seed measurements are already averaged into one instance per configuration (Section 3), so seed is not a separate grouping variable at this stage. The inner split uses StratifiedGroupKFold for model selection. The outer loop evaluates generalization on unseen model–task pairs, while the inner loop selects model hyperparameters. We restrict all preprocessing (scaling, thresholding, and standardization) and hyperparameter grid search to the inner training fold. This design trains 25 models per architecture (5 outer folds × 5 inner folds). We use the same outer-fold assignments for every method so that each is tested on the same held-out samples.3 Each outer fold holds out entire model–task pairs rather than random samples. As a result, a fold-level standard deviation would mostly reflect which pairs were held out in that split, rather than variability in model training. We therefore report the model–task-level results directly and include the appendix heatmap (Figure 17) to show where DEFault++ generalizes well and where performance is lower. We evaluate DEFault++ using four standard metrics. Let TP, FP, TN, and FN denote the true positive, false positive, true negative, and false negative counts, respectively. Precision (P ) and recall (R) are defined as P = 3

TP , TP + FP

TP TP + FN

Replication package: https://github.com/SigmaJahan/DEFaultplusplus-Transformer-Debugging

(31)

The F1-score is the harmonic mean of precision and recall: F1 =

2·P ·R P +R

(32)

The class distributions are imbalanced at all three levels. We therefore report Macro-F1 (Naidu et al., 2023), which averages F1 across classes:

Macro-F1 =

C 1 X F1c C c=1

(33)

where C is the number of classes and F1c is the F1-score for class c. For fault detection (Level 1), we additionally report AUROC (Bradley, 1997), which measures discriminative ability across all classification thresholds. A true positive for fault detection means that a faulty configuration is correctly identified, regardless of its fault category. A true positive for fault categorization means that the category of the fault is identified correctly, regardless of the root cause. A true positive for root-cause diagnosis requires both the Level 2 category and the Level 3 root cause to be identified correctly in the same inference path. Baselines. We compare DEFault++ against four prior techniques only on the Level 1 detection task. The baselines are evaluated on the same train/test split as DEFault++. Rule-based techniques are executed with their published rule definitions and thresholds after mapping the required observables to the closest available trace features in our dataset. AutoTrainer’s gradient-threshold rules map to our Training dynamics group. DeepDiagnosis adds rules that partially overlap with the Attention and Score groups. For baselines that require training, we train them only on the Level 1 fault-detection task. In each outer fold, the baseline is trained on the training split of DEFault-bench and tested on the held-out portion. We use the model types provided in the original replication packages (Zhang et al., 2021; Wardat et al., 2022; Cao et al., 2022; Jahan et al., 2025b). We keep the original rule thresholds unchanged. Changing them for DEFault-bench would adapt the baselines to our data, rather than evaluate the methods as prior work reported them. We do not compare Level 2 or Level 3 results since their labels do not align with our transformer component categories or root causes (see Table 9). Table 9: Baseline techniques’ coverage across all three levels Technique

Paradigm

Detection Categorization Root-Cause

AutoTrainer (Zhang et al., 2021) DeepDiagnosis (Wardat et al., 2022) DeepFD (Cao et al., 2022) DEFault (Jahan et al., 2025b)

Rule-based (5 rules) Rule-based (8 rules) Flat ensemble (KNN+DT+RF) Hierarchical RF

✓ ✓ ✓ ✓

N/A N/A N/A N/A

DEFault++

Neural hierarchical

✓

(i) AutoTrainer (Zhang et al., 2021) monitors five training symptoms (vanishing gradient, exploding gradient, dying ReLU, oscillating loss, slow convergence) via threshold rules applied 34

during training. We map each rule to the corresponding features in our dataset and apply the published thresholds without recalibration. AutoTrainer was able to produce binary detection decisions but its symptom types do not map to our fault categories. (ii) DeepDiagnosis (Wardat et al., 2022) extends AutoTrainer to eight symptom rules (adding unchanged weights, saturated activation, and out-of-range output) with a decision tree that maps detected symptoms to seven fix types. We apply the same feature-mapping strategy and thresholds. Its symptom labels likewise do not map to our fault categories. (iii) DeepFD (Cao et al., 2022) combines KNN, Decision Tree, and Random Forest classifiers with union voting to diagnose faults from runtime features. Its five diagnosis classes (e.g., incorrect learning rate, wrong activation function) do not correspond to our fault categories. We evaluate DeepFD as a binary classifier for fault detection only. (iv) DEFault (Jahan et al., 2025b) uses a hierarchical Random Forest classifier that combines static and dynamic features to detect and categorize faults across seven coarse DNN-level categories (hyperparameters, activation function, loss function, optimizer, layer configuration, regularization, weights). These categories operate at a different abstraction level. We evaluate DEFault for binary detection only.

5.3

Implementation

We implement the hierarchical model in PyTorch. We use separate MLPs for each feature group with hidden dimension h = 32 and a shared projection to e = 64 dimensions. We apply three rounds of message passing in the Fault Propagation Graph with the group-level adjacency matrix. Training uses Adam (Kingma & Ba, 2015) with learning rate 10−3 , weight decay 10−4 , and batch size 256. We train for up to 150 epochs with early stopping on the validation split. The loss weights are α = 1.0, λ = 1.0, β = 0.5, and γ = 0.3, with temperature τ = 0.1. We address fault-category and root-cause imbalance through inverse-frequency weighting in the corresponding classification losses and select a fixed set of random seeds for reproducibility. Section B (Table 27) gives the full hyperparameter configuration. The replication package (Jahan, 2026) contains the implementation, hyperparameter files, and trained models. Training loss converges by ∼epoch 100, at which point validation Detection F1 and Category F1 also stabilize (Figures 10 and 11). Training with early stopping at 150 epochs is sufficient for the three-level hierarchy.

Figure 10: Training dynamics for DEFault++ on the encoder architecture

Figure 11: Training dynamics for DEFault++ on the decoder architecture

5.4

Answering RQ1 : Effectiveness of Diagnosis

To answer RQ1 , we evaluate DEFault++ at the three diagnostic levels: fault detection (Level 1), fault categorization (Level 2), and root-cause diagnosis (Level 3). Encoder and decoder architectures are evaluated separately because their feature groups and label spaces differ. Table 10: DEFault++ performance on fault detection (Level 1) Metric

Encoder Decoder

AUROC Accuracy Macro-F1 Weighted-F1 Precision Recall

0.9660 0.9455 0.8238 0.9477 0.8014 0.8520

0.9620 0.9107 0.9088 0.9104 0.9125 0.9064

Figure 12: ROC curves for fault detection on encoder and decoder

Fault detection (Level 1). DEFault++ achieved AUROC of 0.966 on encoders and 0.962 on decoders (Table 10). Detection recall reached 0.852 on encoders and 0.906 on decoders, which indicates that the model captured most faulty configurations under the evaluated model–task splits. Figure 12 shows the corresponding ROC curves. The encoder tasks were the five GLUE classification benchmarks (Wang et al., 2019) (SST-2, QNLI, RTE, MRPC, and QQP), and the decoder tasks were the four language-modeling corpora (LAMBADA (Paperno et al., 2016), PTB (Marcus et al., 36

1993), WikiText-2 (Merity et al., 2016), and OpenWebText (Gokaslan & Cohen, 2019)) introduced in the benchmark construction (Section 3). Table 11: DEFault++ performance (overall) on fault categorization (Level 2) Metric

Encoder

Decoder

Accuracy Macro-F1 Weighted-F1 Precision Recall

0.8522 0.8497 0.8531 0.8662 0.8403

0.8842 0.8679 0.8850 0.8794 0.8649

Table 12: DEFault++ performance (per-class) on fault categorization (Level 2) Encoder

Decoder

Category

Precision

Recall

Precision

Recall

Embedding FFN Kernel KV Cache LayerNorm Masking Output Positional QKV Residual Score Variant

0.838 0.836 0.771 — 0.790 0.917 0.889 0.919 0.825 0.961 0.861 0.741

0.852 0.807 0.814 — 0.775 0.952 0.927 0.950 0.863 0.965 0.847 0.776

0.829 0.871 0.740 — 0.820 0.886 0.864 0.891 0.792 0.958 0.876 0.717

0.862 0.718 0.765 0.758 0.843 0.890 0.975 0.927 0.789 0.995 0.946 0.948

0.833 0.673 0.848 0.716 0.937 0.927 0.980 0.949 0.772 0.995 0.937 0.986

0.897 0.777 0.700 0.822 0.766 0.857 0.969 0.911 0.815 0.995 0.957 0.914

Macro avg.

0.850

0.866

0.840

0.868

0.879

0.865

Fault categorization (Level 2). DEFault++ achieved Macro-F1 of 0.850 on encoders and 0.868 on decoders (Table 11). Precision and recall remained balanced (0.866/0.840 on encoders, 0.879/0.865 on decoders), which indicates that the model did not trade one for the other. Table 12 reports the per-category breakdown. Residual and Positional were the strongest categories. Residual reached F1 of 0.961 on encoders and 0.995 on decoders, because the residual-stream feature group produced separable skip-connection measurements for the faults in this category. Positional reached 0.919 and 0.927 because positionalsensitivity features distinguished encoding faults from other categories. The weakest categories differed by architecture. On encoders, Variant (0.741) and Kernel (0.771) reported the lowest F1. On decoders, FFN (0.718) and KV Cache (0.758) were the most difficult. Variant differed sharply across architectures (F1 0.741 encoder vs. 0.948 decoder). One explanation is sample size. Encoder Variant had 24 instances (Table 5), the smallest category, while decoder Variant had 51 instances and may have benefited from cache-related discriminative features that are absent on encoders. Root-cause diagnosis (Level 3). DEFault++ achieved Macro-F1 of 0.851 on encoders and 0.867 on decoders (Tables 13 and 14). Macro-precision exceeded macro-recall on both architectures (0.941 vs. 0.793 on encoders, 0.948 vs. 0.823 on decoders). This gap indicates that DEFault++ was conservative at the root-cause level: when the model committed to a root cause, the prediction was almost always correct, but recall remained lower for categories with overlapping root-cause 37

Table 13: DEFault++ performance (overall) on root-cause diagnosis (Level 3) Metric

Encoder

Decoder

0.7953 0.8512 0.8567 0.9407 0.7932

0.8264 0.8670 0.8745 0.9475 0.8225

Accuracy Macro-F1 Weighted-F1 Precision Recall

Table 14: DEFault++ performance (per-category) on root-cause diagnosis (Level 3) Encoder

Decoder

Category

Acc.

Prec.

Rec.

Acc.

Prec.

Rec.

Embedding FFN Kernel KV Cache LayerNorm Masking Output Positional QKV Residual Score Variant

0.879 0.804 0.814 — 0.821 0.925 0.882 0.930 0.765 0.908 0.835 0.799

0.805 0.808 0.740 — 0.748 0.871 0.825 0.884 0.693 0.880 0.779 0.717

0.972 0.847 1.000 — 0.917 0.986 0.958 0.992 0.868 0.927 0.879 1.000

0.806 0.778 0.706 — 0.749 0.874 0.822 0.886 0.694 0.895 0.799 0.717

0.877 0.883 0.747 0.881 0.890 0.845 0.972 0.975 0.646 0.958 0.908 0.824

0.890 0.777 0.691 0.822 0.749 0.755 0.969 0.911 0.629 0.955 0.897 0.871

0.972 1.000 0.988 1.000 0.969 0.902 1.000 1.000 0.727 0.961 0.926 0.925

0.816 0.803 0.676 0.817 0.839 0.801 0.947 0.953 0.592 0.955 0.897 0.773

Macro avg.

0.851

0.795

0.941

0.793

0.867

0.826

0.948

0.823

feature patterns. The strongest categories for root-cause diagnosis were Positional (0.930 encoder, 0.975 decoder), Masking (0.925 encoder, 0.845 decoder), and Residual (0.908 encoder, 0.958 decoder). Positional root causes produced distinct patterns in positional-sensitivity, attention-entropy, and pre-softmax score features. Residual root causes were distinguished because skip-connection integrity and gradient-norm features produced non-overlapping distributions across root causes within that category. QKV was the weakest category, with F1 of 0.765 on encoders and 0.646 on decoders, because three of its four root causes disturbed the same projection, alignment, and update-flow measurements simultaneously. Root-cause count alone did not explain the per-category ordering. Residual remained strong with five root causes because its features were well separated, while QKV remained weakest with four root causes because most of them activated the same feature groups. This pattern suggests that per-category difficulty depended more on feature separability than on the number of root causes. Hierarchical design. The hierarchical setup links the three levels at inference. Level 1 errors prevent a configuration from reaching Level 2, and Level 2 errors prevent it from reaching Level 3. The reported Level 3 metrics therefore reflect the complete inference path rather than an isolated root-cause classifier. The precision-recall gap at Level 3 (0.941 vs. 0.793 on encoders, 0.948 vs. 0.823 on decoders) is consistent with this hierarchical filtering. Instances misclassified at Level 2 cannot be recovered at Level 3, which lowers macro-recall, while the predictions DEFault++ does commit to remain mostly correct, which keeps macro-precision high.

Summary of RQ1 : Across the three diagnostic levels, DEFault++ detected transformer faults with AUROC above 0.96 on both architectures, categorized faults with Macro-F1 of 0.850 (encoder) and 0.868 (decoder), and diagnosed root causes with Macro-F1 of 0.851 (encoder) and 0.867 (decoder), with macro-precision above 0.94 at Level 3. The strongest categories were those with distinct component-level measurements (Residual, Positional, Masking). QKV remained the hardest because most of its root causes activated overlapping feature groups.

5.5

Answering RQ2 : Comparison with Existing Techniques

To answer RQ2 , we compared DEFault++ with four prior DL fault-detection techniques: AutoTrainer (Zhang et al., 2021), DeepDiagnosis (Wardat et al., 2022), DeepFD (Cao et al., 2022), and DEFault (Jahan et al., 2025b). We restricted the comparison to Level 1 detection because the baselines do not expose label spaces that match the DEFault++ transformer categories or root causes. Comparing them at Level 2 or Level 3 would equate non-equivalent outputs. Table 15: Fault-detection comparison between DEFault++ and baselines Method

Type

DEFault++ DEFault (Jahan et al., 2025b) DeepFD (Cao et al., 2022) DeepDiagnosis (Wardat et al., 2022) AutoTrainer (Zhang et al., 2021)

Neural hierarchical Hierarchical RF Flat ensemble Rule-based (8 rules) Rule-based (5 rules)

AUROC (Enc)

M-F1 (Enc)

Acc. (Enc)

AUROC (Dec)

M-F1 (Dec)

Acc. (Dec)

0.9660 0.8610 0.8340 — —

0.8238 0.5660 0.5140 0.3380 0.2840

0.9455 0.8120 0.7810 0.7240 0.6970

0.9620 0.9030 0.8810 — —

0.9088 0.7020 0.6550 0.4820 0.4270

0.9107 0.8460 0.8190 0.7710 0.7440

Rule-based techniques produce binary decisions without probabilistic scores, so AUROC is not applicable (—).

DEFault++ outperformed all four baselines on both architectures (Table 15). On encoders, DEFault++ reached AUROC of 0.966 and Macro-F1 of 0.824. The strongest baseline at Level 1 was DEFault, which reached AUROC of 0.861 and Macro-F1 of 0.566. DeepFD reached 0.834 and 0.514. The two rule-based methods scored lowest. AutoTrainer reached Macro-F1 of 0.284 and DeepDiagnosis reached 0.338. AUROC is not applicable to AutoTrainer or DeepDiagnosis because both produce binary decisions without a probability score. On decoders, DEFault++ reached AUROC of 0.962 and Macro-F1 of 0.909. DEFault reached 0.903 and 0.702, DeepFD reached 0.881 and 0.655, AutoTrainer reached 0.427, and DeepDiagnosis reached 0.482. The learned baselines outperformed the rule-based baselines at Level 1 because they observed runtime behavior at the model level rather than checking fixed symptom thresholds. Their feature spaces were designed for general DNN faults and did not represent attention distributions, QKV alignment, cache behavior, or component-level propagation. The rule-based baselines target gradient explosion, vanishing gradients, dying ReLU, and oscillating loss. These symptoms did not cover many silent transformer faults in our benchmark, which preserved normal loss curves and finite numerical values while changing internal attention behavior. We do not report Level 2 or Level 3 comparisons. AutoTrainer and DeepDiagnosis generate generic symptom labels that do not map to any fault category in our taxonomy. DeepFD and DEFault use DNN-level fault categories that were defined independently of transformer component boundaries. A numeric comparison at those levels would equate non-equivalent labels. We report baseline coverage in Table 9 instead.

Summary of RQ2 : On Level 1 fault detection, DEFault++ outperformed all four evaluated baselines across AUROC, Macro-F1, and accuracy. The improvement over DEFault and DeepFD indicates that DNN-level diagnostic features were insufficient for many transformer faults in our benchmark. The lower scores of AutoTrainer and DeepDiagnosis indicate that fixed training-symptom rules missed faults whose evidence appeared inside attention, projection, cache, or residual behavior.

5.6

Answering RQ3 : Ablation

Ablation design. We assess the contribution of each architectural component by systematically removing it from DEFault++ and measuring the performance drop. We ablate two components: message passing in the Fault Propagation Graph and the separation loss Lsep . The separation loss applies only at Level 3. We define two variants for Levels 1 and 2 and four variants for Level 3 as follows. 1. DEFault++. The complete model with message passing in the FPG and the separation loss at Level 3. 2. −FPG. We remove message passing in the FPG while retaining Lsep at Level 3. This variant measures the contribution of propagation-informed feature interaction. 3. −Sep (Level 3 only). We remove the separation loss (β = 0, γ = 0) while retaining FPG. This variant measures the contribution of within-category root-cause discrimination. 4. −FPG−Sep. We remove both FPG and Lsep , retaining only the feature-group MLPs with projection. This variant tests the hierarchical classifier alone. To test whether the specific FPG topology matters beyond just having any graph structure, we define two additional topology variants as follows. 1. Rewired. We keep the same number of nodes and edges as the FPG but randomly reassign edge endpoints while preserving the degree distribution, following the degree-preserving rewiring procedure of Maslov and Sneppen (Maslov & Sneppen, 2002). This rewiring preserves the overall graph density and local connectivity profile while breaking the meaningful propagation links between transformer components. 2. Random. We replace the FPG with an Erdős–Rényi random graph (Erdős & Rényi, 1959) with approximately the same edge density. This replacement removes both the meaningful propagation links and the degree structure of the original FPG.

Ablation results. We examine the contribution of message passing in the Fault Propagation Graph and the separation loss through ablation. Tables 16 and 17 compare DEFault++ with and without message passing at Levels 1 and 2, where the separation loss does not apply. Table 18 compares all four variants at Level 3. We apply message passing in the Fault Propagation Graph as part of the shared encoding pathway across all three levels, following standard multi-task practice in which lower representation layers serve multiple tasks. 40

Level 1: Fault detection. We find that removing message passing in the FPG modestly reduces detection performance on both architectures (Table 16). Binary detection is a coarse task. A fault that disrupts any transformer component shifts the overall feature distribution enough for the binary classifier to separate faulty from clean configurations. At this level, the classifier only needs to separate faulty from clean configurations. We observe that message passing adds cross-group information that improves the binary boundary, but the group-level features alone already support detection. We retain message passing at Level 1 because cross-component aggregation helps detect faults with weak local deviations, and the ablation confirms that removing it hurts detection. Table 16: Ablation study of Level 1 fault detection Architecture

Variant

AUROC

Accuracy

F1w

Precision

Recall

Encoder

DEFault++ −FPG

0.966 0.921

0.946 0.922

0.824 0.782

0.948 0.924

0.801 0.752

0.852 0.816

Decoder

DEFault++ −FPG

0.962 0.949

0.911 0.892

0.909 0.890

0.910 0.892

0.912 0.890

0.906 0.891

Level 2: Fault categorization. We find that removing message passing in the FPG produces the largest drop in the entire ablation (Table 17). Categorization requires the classifier to distinguish among 11 fault categories on encoders and 12 on decoders, several of which affect overlapping feature groups. A QKV fault propagates to attention scores and the residual stream. A Score fault propagates to attention weights and onward to the residual stream, but through a narrower path than a QKV fault, which simultaneously affects both scores and values. Without message passing, we compute each group embedding independently, and the classifier can only infer crossgroup relationships from the flat projection. With message passing, we route fault evidence along the FPG propagation edges before the projection step, producing representations that already encode how the fault spreads across components. We attribute the large drop to this loss of crossgroup structure, because propagation patterns are the distinguishing evidence for categories with overlapping features. Table 17: Ablation study of Level 2 fault categorization Architecture

Variant

Accuracy

F1w

Precision

Recall

Encoder

DEFault++ −FPG

0.852 0.743

0.850 0.742

0.853 0.745

0.866 0.757

0.840 0.731

Decoder

DEFault++ −FPG

0.884 0.790

0.868 0.788

0.885 0.791

0.879 0.797

0.865 0.783

Level 3: Root-cause diagnosis. We ablate both the separation loss and message passing independently at Level 3 (Table 18). We find that removing the separation loss produces the larger drop. Root causes within the same fault category share much of the same feature distribution because they affect the same transformer component. A weight-corruption fault and a stale-parameter fault both appear in the QKV feature group with similar aggregate patterns. Cross-entropy alone does not explicitly structure the embedding space to separate these closely positioned root causes. We designed the separation loss to address this gap by pulling same-root-cause embeddings together and pushing different-root-cause embeddings apart within each category. 41

Table 18: Ablation study of Level 3 root cause diagnosis Architecture

Variant

Accuracy

F1w

Precision

Recall

Encoder

DEFault++ −FPG −Sep −FPG−Sep

0.795 0.778 0.752 0.702

0.851 0.812 0.756 0.701

0.857 0.819 0.763 0.711

0.941 0.933 0.910 0.883

0.793 0.776 0.752 0.690

Decoder

DEFault++ −FPG −Sep −FPG−Sep

0.826 0.810 0.781 0.735

0.867 0.833 0.781 0.734

0.875 0.840 0.788 0.742

0.948 0.940 0.925 0.896

0.823 0.809 0.781 0.721

We find that removing message passing in the FPG at Level 3 (while retaining the separation loss) produces a smaller drop. Message passing aggregates propagation-related evidence across groups before prototype comparison, but the root-cause discrimination problem centers on within-category separation rather than cross-group structure. We observe that removing both components together produces the largest overall drop, which confirms that message passing and the separation loss contribute complementary information at Level 3. Topology ablation. The preceding ablation compares the full FPG with no message passing at all. To test whether the specific FPG topology matters, we compare it with two alternative graph structures: Rewired (same density, shuffled edges) and Random (Erdős–Rényi graph with matched density). Table 19 reports Macro-F1 across all three levels. The results show a consistent ordering at every level and on both architectures: DEFault++ > Rewired > Random > −FPG. The FPG with the correct propagation structure outperforms both alternative topologies. Rewired edges preserve the graph density and degree distribution but break the meaningful propagation links, and the performance drops below the full FPG at every level. Random edges lose both the specific links and the degree structure of the original FPG, and performance drops further. Removing message passing entirely (−FPG) produces the lowest performance. Table 19: Topology ablation results using Macro-F1 across graph variants DEFault++

Rewired

Random

−FPG

Encoder Decoder

0.824 0.909

0.790 0.894

0.786 0.892

0.782 0.890

Encoder Decoder

0.850 0.868

0.770 0.807

0.754 0.796

0.742 0.788

Encoder Decoder

0.851 0.867

0.820 0.838

0.816 0.835

0.812 0.833

Level

Architecture

The topology effect is strongest at Level 2, where DEFault++ outperforms Rewired by 0.080 (encoder) and 0.061 (decoder) in Macro-F1. This is consistent with the earlier finding that fault categorization depends most on cross-group propagation structure. At Level 1 and Level 3, the topology differences are smaller because binary detection is a coarser task and root-cause diagnosis depends more on within-category separation than on cross-group structure.

Summary of RQ3 : Both FPG message passing and the separation loss contributed to DEFault++ performance. Removing FPG message passing reduced Macro-F1 most at Level 2 (from 0.850 to 0.742 on encoders and from 0.868 to 0.788 on decoders), where cross-component propagation patterns distinguish categories with overlapping symptoms. Removing the separation loss reduced Level 3 Macro-F1 most (from 0.851 to 0.756 on encoders and from 0.867 to 0.781 on decoders), where the model must separate root causes within the same category. The topology ablation showed that the specific FPG structure, not graph connectivity alone, explains the categorization gain (DEFault++ vs. Rewired: +0.080 encoder, +0.061 decoder at Level 2).

5.7

Answering RQ4 : Explanation Faithfulness

To answer RQ4 , we evaluated whether the feature groups identified by DEFault++ supported the predicted root cause. We used the prototype-based explanation from Level 3. For each instance, DEFault++ compares the predicted root-cause prototype with the nearest alternative prototype and decomposes the distance gap by feature group (Equation (29)).

Figure 13: Prototype-label agreement during training

Prototype-label agreement during training. DEFault++ trains with two complementary objectives at Level 3. The cross-entropy heads classify root causes from logits, while the prototypematching loss (Equation (28)) brings each sample closer to the correct prototype and farther from alternatives within the same fault category. We use the prototype classifier for explanation because the distance to a prototype decomposes group by group (Equation (29)), so each feature group receives its own contribution score. Cross-entropy logits produce a single prediction vector and do not offer this per-group breakdown. Figure 13 shows the training dynamics. CE-prototype agreement, defined as the fraction of samples for which the cross-entropy head and the prototype classifier predicted the same root cause, increased steadily over training alongside the cross-entropy objective and the root-cause prototype F1. The cross-entropy F1 rose faster in early epochs because logit-based supervision provided a direct gradient, while the prototype-matching loss required the embedding clusters to separate first. The two curves converged by mid-training, which indicates that the cross-entropy and prototype43

matching objectives produced consistent predictions despite optimizing different loss surfaces. Only the prototype classifier is used at inference.

Figure 14: Group ablation analysis for the FPG-based explanation

Group-ablation check. We tested the explanation through group ablation. For each sample, we identified the two groups with the largest importance scores (Equation (30)). The importance scores are normalized to sum to one, and the distribution is typically concentrated on a few groups. In our experiment, the top two groups captured a substantial share of the total score. We zeroed out these two groups and compared the drop in the prototype margin (the gap between the predicted and the nearest alternative prototype) with the drop obtained by zeroing two randomly chosen groups. Removing the most important groups reduced the gap more than removing random groups across all evaluated categories on both architectures (Figure 14). The most important groups therefore carried the diagnostic information that separated the predicted root cause from the nearest alternative. This result is a model-consistency check, not causal attribution. The benchmark does not provide independent feature-group ground truth. The ablation supports internal faithfulness of the explanation to the prototype decision, but it does not establish that the highlighted groups capture the true causal mechanism of the original fault. The real-world evaluation in Section 6 provides an additional check for selected cases. For each reproduced bug, we compare the feature groups highlighted by DEFault++ with the components changed by the source bug fix. This checks whether the diagnostic trace is consistent with the code change, but it does not prove that the highlighted components are the true cause of the fault. Summary of RQ4 : The FPG-based explanations are internally faithful to the prototype-based root-cause decision under the evaluated ablation. Removing the most important feature groups reduced the prototype margin more than removing random groups. This evidence supports internal faithfulness, but it does not establish external causal attribution because independent feature-group ground truth is unavailable.

Real-world Fault Evaluation

We use real-world GitHub issues to evaluate whether DEFault++ generalizes beyond the synthetic faults from our mutation-based benchmark. Following prior work (Wardat et al., 2021; Morovati et al., 2024; Jahan et al., 2025a), we collected 20 transformer-related bug-fix issues from open-source repositories and successfully reproduced 11 of them. For each reproduced issue, we compared the 44

faulty version with the fixed version using the same feature extraction technique used in the experimental evaluation. DEFault++ detected 8 of the 11 faults, identified the correct fault category for 8, and identified the root cause for 4 (see Table 20). We report each result at the same three diagnostic levels used in the main evaluation. Fault detection (FD) checks whether DEFault++ marks the faulty version as faulty. Fault categorization (FC) checks whether it assigns the fault to the correct transformer category. Root-cause diagnosis (RC) checks whether the predicted cause matches the cause confirmed by the bug fix of the source issue. The evidence column reports the feature group or feature that best reflects the faulty behavior. Table 20: Real-world evaluation of DEFault++ on 11 transformer faults Source

Fault category DEFault++ result DEFault++ evidence FD FC

jax#23349

Masking

✓

pytorch#103082

Masking

✓

transformers#19045Positional

✓

✗

transformers#17886Positional

✓

✗

ai-edge-torch#6 nsa-pytorch#20

QKV KV Cache

✓ ✓

✓ ✗

transformers#37574KV Cache

✗

transformers#36096Score

Crash*

–

pytorch#116333 Kernel transformers#35896Variant

✗ ✓

✗ ✗

diffusers#11903

✓

QKV

Attention: padding attention mass Attention: future attention mass Positional: positional sensitivity Attention: inter-head cosine similarity Score: pre-softmax score KV Cache: cache-related behavior –

Diagnostic relation/scope

Attention feature group Attention feature group Forward sequential propagation (M1) Forward sequential propagation (M1) Simultaneous propagation (M2) Generation-time cache update path Generation-time cache behavior

Runtime failure before feature extraction Backend stride validation attention en- Cross-layer propagation (M4)

– Attention: tropy Optimization: projection QKV projection/update path update ratio

Summary: FD - 8/11; FC - 8/11 categories; RC - 4/11 root causes. FD = fault detection; FC = fault categorization; RC = root-cause diagnosis. Crash* indicates an explicit runtime failure before feature extraction.

We observed two masking faults in which the faulty versions assigned attention to tokens that should have been masked. In jax#23349, attention flowed to padding tokens; in pytorch#103082, attention flowed to future tokens. DEFault++ captured these faults through padding attention mass and future attention mass, respectively, and both values returned to zero after the fixes. We also observed the QKV fault in diffusers#11903 through a near-zero QKV update ratio. In that issue, LoRA updated stale projection modules while the forward pass used the fused QKV projection, hence the updated parameters did not affect the model output. We also found four faults where DEFault++ detected the fault through feature groups connected in the Fault Propagation Graph (FPG). In transformers#19045 and transformers#17886, the Positional faults changed how the model used token positions, which then affected attention behavior. DEFault++ therefore assigned higher importance to positional sensitivity and head similarity. In ai-edge-torch#6, the checkpoint loader assigned QKV projection weights incorrectly. Pre-softmax attention scores depend on the query and key projections, so the incorrect weights changed the score distribution. DEFault++ captured this effect through the Score feature group, which the FPG connects to QKV projection by simultaneous propagation (M2). The diagnosis therefore surfaced the QKV fault through score-level evidence, not only through QKV-alignment features. 45

Similarly, in transformers#35896, sliding-window attention was applied to the wrong layers. This changed the attention distribution at those layers through a cross-layer dependency (M4), which DEFault++ captured through per-layer attention entropy. Across these four cases, DEFault++ identified the faulty component through a feature group connected to it in the FPG. This suggests that the FPG-based explanation can help interpret real-world faults when the fault affects connected feature groups. The KV-cache fault exposes a limitation of DEFault++. In nsa-pytorch#20, DEFault++ correctly predicted the faulty status and the fault category (i.e., KV Cache). However, it incorrectly predicted the root cause as cache invalidation, while the root cause is update synchronization (i.e., from ground truth labels of source issue). The cache update path was incorrect. Thus, the current token’s key-value vectors were not synchronized with the attention and prediction computation. DEFault++ could not diagnose three real-world faults across all three levels. In transformers#37574, DEFault++ missed the fault as the faulty behavior appeared during autoregressive generation, where cached key-value states from earlier decoding steps interact with the current token. DEFault++ captures cache-related behavior during fine-tuning, but it does not trace each cached decoding step during generation, which may miss abnormal behavior that appears only at generation time. In pytorch#116333, the fault came from backend stride validation rather than transformer-layer behavior. DEFault++ traces model-level fine-tuning dynamics. It does not capture low-level kernel-dispatch constraints. Detecting this fault would require system-level information beyond the model trace used by DEFault++. Finally, in transformers#36096, the faulty flex_attention implementation returned an attention tensor with an invalid rank. DEFault++ could not extract diagnostic features since the run failed before producing a valid model trace. We therefore treat this case as not applicable to DEFault++, rather than as a diagnostic failure.

(a) Hierarchical diagnostic output Fault Detection

Fault Categorization

Root Cause Diagnosis

(confidence: 0.98)

Faulty

Normal training loss, abnormal projection-update behavior

QKV

(confidence: 0.91)

Deviations in QKV alignment, projection update, attention entropy

Stale-parameter

(confidence: 0.88)

Near-zero update ratio for q_proj/v_proj, fused path active

(b) Feature-group importance (wg ) QKV alignment

45%

Optimization Attention

28%

73% in QKV alignment and optimization

12%

Score

Other

Figure 15: DEFault++ diagnosis and feature-group importance for the stale QKV fusion fault in Listing 2.1

We use diffusers#11903 as a representative example of a silent QKV fault, which was also used in the motivating example (see Section 2). In this case, training completed normally, but LoRA updated projection parameters that were no longer used by the forward pass. The faulty implementation fused the query, key, and value projections into to_qkv, while the original projection modules remained accessible to LoRA. Figure 15 shows DEFault++’s three-level diagnosis and the feature-group importance scores for this fault. Level 1: Fault detection. DEFault++ classified the case as faulty, even though we found that the training loss curve decreased normally. The strongest signal came from the optimization feature group, where the update ratios for the stale projection modules were near zero relative to the active fused path. Level 2: Fault categorization. DEFault++ assigned the fault to the QKV category. We found that QKV alignment feature group contributed the most to this decision, while the attention score feature group provided secondary evidence. Level 3: Root-cause diagnosis. DEFault++ identified the root cause as a stale projection update path. The QKV alignment and Optimization feature groups together explain 73% of the normalized feature importance (Figure 15). This aligns with the repair decision because the fault affects both the projection path and the parameter updates. The repair ensures that LoRA updates the projection module used in the forward pass rather than the stale modules left after QKV fusion.

Developer Study

We evaluated whether DEFault++ diagnoses help developers choose correct repair actions for transformer faults. Participants completed four debugging scenarios derived from reproduced realworld GitHub issues. For each scenario, they selected a repair action either from baseline materials alone or from the same materials plus the DEFault++ diagnosis. We measured repair-action accuracy, confidence, self-reported time, and perceived usefulness. 7.1

Study Design

We used a within-subjects, counterbalanced design with two conditions (Wohlin et al., 2012). In the baseline condition, participants received a scenario description, observed behavior, and logs with metrics. In the DEFault++ assisted condition, participants received the same materials plus the DEFault++ diagnosis: Level 1 fault status, Level 2 fault category, Level 3 localized root cause, and supporting feature-group evidence. Each participant completed all four scenarios, two per condition. We counterbalanced condition assignment across two survey forms. This scenario-based evaluation follows prior developer studies of DL debugging techniques (Yuan et al., 2021; Manke et al., 2025). We derived the four scenarios from reproduced real-world GitHub issues (see Section 6), each representing a distinct transformer fault category (Table 21). For one scenario (S3), the output of DEFault++ was partially correct (see Table 23). We included this scenario to test whether the partial diagnosis would still help developers select the correct repair action. Each scenario asked participants to choose one repair action from four options: the correct repair from the source issue, two incorrect repairs targeting related components, and one unrelated hyper47

parameter change. The task measured whether participants could translate the available diagnostic evidence into the correct repair direction. Table 21: Developer-study scenarios and correct repair actions Scenario

Source Issue

Fault Category Correct Repair Action

S1: Cached decoding visibility

pytorch #103082

Masking

S2: Position-dependent scoring drift

transformers #19045 Positional

S3: Incremental state update mismatch† nsa-pytorch #20

S4: Local-attention layer assignment

KV Cache

transformers #35896 Variant

Fix the causal mask for cached decoding so the new token attends to the correct cached key positions. Recompute relative-position distances from the true token positions during cached decoding. Write the current token’s key-value vectors into the cache before computing attention and prediction scores. Fix the layer-selection condition so local/sliding-window attention is applied to the intended layers.

† In the assisted condition, DEFault++ correctly predicted the faulty status and KV Cache fault category, but incorrectly

predicted the root cause as cache invalidation rather than update synchronization.

The primary outcome was repair-action correctness, scored against the ground-truth fix from the source issue. We also collected self-reported confidence on a 5-point scale and self-reported time as a perceived-effort measure. After completing all scenarios, participants rated the clarity, usefulness, and practical value of the DEFault++ diagnosis on 5-point Likert scales and answered two preference questions comparing the baseline and assisted conditions. 7.2

Participants

The Dalhousie University Research Ethics Board 4 reviewed and approved the developer study. Eligibility required experience with transformer architectures in at least one of training, fine-tuning, inference, debugging, or evaluation (Yuan et al., 2021; Manke et al., 2025). We recruited 21 practitioners through graduate student mailing lists, academic networks, and direct outreach. Participation was anonymous, voluntary, uncompensated, and hosted through Microsoft Forms. Table 22: Developer study participant demographics (N = 21) Characteristic

Value

ML/DL experience

Less than 1 year 1–2 years 3–5 years More than 5 years

1 6 10 4

Prior transformer debugging Yes No Role (multi-select)

15 6

Research assistant 10 Graduate student 11 Industry practitioner 9

https://www.dal.ca/research-and-innovation/support-for-researchers/responsible-conduct-researc h/human-ethics.html

7.3

Results

We report descriptive statistics for each scenario and condition, following prior vignette-based debugging studies such as UMLAUT (Yuan et al., 2021) and KUnit (Manke et al., 2025). As each repair-action response is selected from four options and each participant sees only two scenarios per condition, we treat the per-condition difference as an effect-size estimate rather than as the basis for a hypothesis test. Table 23: Repair accuracy and confidence by scenario and condition Scenario

Condition

Correct

Accuracy

Confidence

S1 (Masking) S1 (Masking)

Baseline Assisted

12 9

8/12 9/9

66.7% 100.0%

3.75 4.11

S2 (Positional) S2 (Positional)

Baseline Assisted

9 12

4/9 10/12

44.4% 83.3%

2.78 3.75

S3 (KV Cache) S3 (KV Cache)

Baseline Assisted

12 9

7/12 8/9

58.3% 88.9%

3.50 4.11

S4 (Variant) S4 (Variant)

Baseline Assisted

9 12

5/9 8/12

55.6% 66.7%

3.33 4.08

Total Total

Baseline Assisted

42 42

24/42 35/42

57.1% 83.3%

3.38 4.00

Repair accuracy. Across the four scenarios, participants selected the correct repair action in 24 of 42 baseline responses (57.1%) and 35 of 42 DEFault++ assisted responses (83.3%), which is an increase of 26.2% (see Table 23). The largest improvement occurred for the positional fault, where baseline accuracy was lowest. The smallest improvement occurred for the variant fault, where the scenario already provided a clearer symptom-to-repair path. Under the assisted condition with DEFault++, no participant selected the unrelated hyperparameter option (see Figure 16). Confidence. Participants also reported higher confidence under the assisted condition (M = 4.00, SD = 0.83) than under the baseline condition (M = 3.38, SD = 1.08). Confidence remained higher for correct responses than incorrect responses in both conditions, suggesting that DEFault++ increased confidence without eliminating the distinction between correct and incorrect repair choices. Participant feedback. Participants rated the diagnostic output positively. Twenty participants (95%) agreed or strongly agreed that the output from DEFault++ was easy to understand (M = 4.14, SD = 0.48), and 19 (90%) agreed or strongly agreed that it helped them decide what repair to try first (M = 4.29, SD = 0.96). The same proportion reported that they would use a tool like DEFault++ in practice (M = 4.10, SD = 0.89). 15 participants (71%) preferred the assisted condition for debugging decision support. Within-subjects comparison. At the participant level, seven participants improved under the assisted condition, 14 showed no change, and none declined. Thus, no participant performed worse with DEFault++ assistance than without it.

(a) Repair accuracy by scenario and condition

(b) Likert agreement ratings

Figure 16: Developer study results

7.4

Discussion

The results suggest that DEFault++ is most useful when symptoms alone do not clearly indicate the repair. The largest improvement happened for the positional fault (S2), where participants had the weakest baseline performance. The smallest improvement occurred for the variant fault (S4), where the repair direction was clearer from the scenario description. This pattern supports the intended role of DEFault++ since it is designed to help developers translate internal fault evidence into a repair direction when the observable behavior is ambiguous. For the KV-cache fault (S3) with partially correct outcome from DEFault++, participants saw a correct fault/category prediction but an incorrect root-cause prediction. The assisted accuracy for S3 suggests that category-level and cache-related evidence can still help participants choose a plausible repair direction. Since only one of the four scenarios included a partially incorrect diagnosis, the study does not provide enough evidence to determine whether incorrect root-cause diagnoses are generally harmless. 50

The incorrect baseline responses were mostly related but wrong repair choices, rather than unrelated hyperparameter changes. This suggests that participants engaged with the scenarios but lacked enough diagnostic information to distinguish between relevant transformer components. The assisted condition provided that missing diagnostic information. Moreover, the perception ratings reinforce the repair-accuracy result. Most participants found the DEFault++ output understandable and useful for deciding what repair to try first. This result is consistent with related developer studies where UMLAUT helped participants find and fix more faults (Yuan et al., 2021), and KUnit participants reported that mocks were useful for independent component testing (Manke et al., 2025).

Limitations and Future Directions

The Fault Propagation Graph (FPG) assumes that faults always propagate along the deterministic edges derived from the forward and backward pass. In practice, successive matrix multiplications, normalization layers, and nonlinear activations can reduce or eliminate a fault’s effect before it reaches a downstream component. The current FPG does not model this reduction. Future work can run a fault-injection study to test whether downstream feature groups change as the FPG predicts. The FPG also assumes a standard transformer block structure and does not model architecture-specific parameter-tying shortcuts, such as the tied input-embedding and output-head weights in GPT-2-style decoders (see Table 6). Architectures that differ from the standard transformer block layout may introduce propagation paths that the current graph does not represent. Future work can extend the graph with such architecture-specific edges and with relation-typed message passing that distinguishes the three mechanism classes. Our benchmark uses single-fault configurations, with one fault injected per training run. This design lets us connect the observed behavior to the injected fault, which is necessary for assigning the Level 2 category and the Level 3 root-cause label. It is also consistent with prior mutationtesting studies for deep learning, including DeepCrime and DeepMutation++. This restriction differs from DEFault (Jahan et al., 2025b), which considers compound faults in DNN programs by training a separate binary classifier for each fault category. That design can detect more than one fault category in the same run. We do not use the same strategy here because transformer fault categories can affect the same internal components. For example, a QKV fault can affect the Score and Residual groups, and a Score fault can affect Attention weights. As Section 4.2 shows, these dependencies make it difficult for independent per-category classifiers to separate co-occurring faults reliably. Extending DEFault++ to compound faults would require multi-label diagnosis with explicit co-occurrence modeling, which is outside the scope of this work. The benchmark also covers seven models and nine tasks and it does not include larger-scale architectures (e.g., mixture-of-experts models, multimodal models with cross-attention). Extending the evaluation to these architectures remains future work. Our developer study uses four scenarios, only one of which includes a partially incorrect DEFault++ diagnosis. The study therefore does not systematically evaluate how developers respond to diagnostic errors. Future work should vary the severity and type of incorrect diagnoses to measure when developers appropriately rely on, question, or reject the output. 51

Threats to Validity

Threats to internal validity (Wohlin et al., 2012) relate to experimental errors. The main threat is information leakage among configurations sharing the same model–task pair. We reduce this threat through nested grouped cross-validation that keeps all instances from the same model–task pair in the same outer fold, fits all preprocessing and hyperparameter selection only on the inner-training fold, and applies the same outer-fold assignments across all compared methods. A second threat is the joint training objective. Shared encoder means that changes in one loss can affect the representation available to the others. In particular, the separation loss targets root-cause discrimination and may guide the shared representation in ways that do not benefit Level 1 or Level 2. To reduce this threat, we apply the separation loss only to faulty samples within their ground-truth fault category (Section 4.5), which restricts its influence to within-category structure rather than the global representation. We also test the per-level impact through the ablation in RQ3 . Threats to construct validity (Smith, 2005) relate to whether the benchmark and the diagnostic outputs reflect the intended diagnosis problem. The benchmark is produced by mutation operators rather than by naturally occurring faults (Jahan et al., 2026; Humbatova et al., 2021). We reduce this threat by deriving each operator from a real transformer fault root cause: the attention-internal categories come from the 555-fault attention taxonomy (Jahan et al., 2026), and the non-attention categories come from prior DNN fault taxonomies (Humbatova et al., 2020; Islam et al., 2019) that catalog DL faults from open-source projects. Class imbalance across categories and root causes is another construct-validity threat, which we address through Macro-F1 reporting and inverse-frequency class weighting (Buda et al., 2018). The mutation-killing decision adds further construct threats specific to DEForm. Our test is a one-sided sign-flip permutation test with n = 5 matched seeds, in contrast to DeepCrime’s generalized linear model with n = 20 (Humbatova et al., 2021). Five seeds are the smallest design that admits an exact one-sided test at α = 0.05, so the attainable p-value floor is 1/25 ≈ 0.031, and subtle faults whose effect size falls below this floor may be missed. To reduce the impact of this floor, we retain surviving mutants as Level 1 negative examples rather than discarding them, so the model still sees cases whose measured effect does not pass the killing test. The operator set is also not exhaustive: new attention variants, caching schemes, and kernel implementations continue to appear, and DEForm’s 45 operators cover root causes that were reported at the time the benchmark was built rather than every possible failure mode (Humbatova et al., 2021; Islam et al., 2019). To reduce the impact of an evolving fault landscape, we link mutation operators to the attention fault taxonomy (Jahan et al., 2026) and to prior DNN fault taxonomies (Humbatova et al., 2020; Islam et al., 2019) so that new operators can be added as those taxonomies are extended, without changing the diagnostic technique. Severity and layer indices are sampled according to a uniform distribution, which prevents overrepresentation of any specific magnitude or depth. We record severity and layer index as labels on every instance and report per-category mutation scores in Table 4, which makes any sampling skew visible to readers and to follow-up work. The trivial-mutant risk identified by Humbatova et al. (2021) also applies to DEForm; we sample 52

Figure 17: Mutation scores per (model, task) pair under isKilled at α = 0.05. Cell values give the fraction of injected configurations killed by the task-performance criterion

severity uniformly across {low, medium, high} rather than concentrating on aggressive faults, although trivial configurations cannot be eliminated without per-operator parameter tuning at scale. Finally, the FPG-based explanation is derived from the same model that produces the diagnosis, so the importance scores may reflect model behavior rather than the underlying causal mechanism. We test the explanation through the group-ablation check in RQ4 and the trace-to-code consistency checks in the real-world evaluation (Section 6). These tests check consistency within our evaluation, but they do not provide independent causal evidence. We therefore frame the explanation as evidence supporting the prediction rather than as causal attribution. Threats to external validity (Findley et al., 2021) relate to the generalizability of our findings. The dataset covers seven transformer models and nine tasks (Section 3.3), which may not represent the full range of transformer architectures and deployment settings. To reduce model–task overfitting, we use nested grouped cross-validation. All instances from the same model–task pair are assigned to the same outer fold, so each held-out fold contains model–task pairs that were not used during training or hyperparameter selection. Since each outer fold holds out entire model–task pairs rather than random samples, fold-level variation mainly reflects which pairs were held out. We therefore report model–task-level results and visualize where DEFault++ generalizes well and where performance is lower in the per-(model, task) mutation-score heatmap shown in Figure 17. Encoder pairs cluster at high mutation scores, with DistilBERT reaching 100% on four of five GLUE tasks and the lowest encoder pair being BERT–SST-2 at 51%. Decoder pairs are more variable, with LAMBADA producing the lowest scores across all three decoders (25–32%). The LAMBADA gap is consistent with that corpus’s higher seed-to-seed perplexity variance on the long-context wordprediction task (Paperno et al., 2016). Existing baselines do not present a label space compatible with our fault categories, so we restrict direct comparison to Level 1 detection and report baseline coverage separately in Table 9. 53

The developer study (Section 7) carries two further external-validity threats. First, the sample is modest (N = 21); we use a within-subjects, counterbalanced design where each participant completes tasks under both baseline and DEFault++-assisted conditions, which reduces between-participant variation, although participants do not solve the same scenario in both conditions. Second, participants evaluated vignette-based scenarios rather than using DEFault++ in a live debugging environment. We base each vignette in reproduced real-world issues, use the same fault categories and diagnostic outputs as the main evaluation, and isolate the diagnostic signal from tool-usability confounds. This choice improves experimental control but may not fully capture the time pressure, context switching, code navigation, and iterative hypothesis testing of real debugging work, so we treat self-reported time as a perceived-effort measure rather than as an objective task-duration measure.

Related Work

Fault Taxonomies. Empirical studies of Deep Learning (DL) programs have classified faults by root cause (Humbatova et al., 2020; Islam et al., 2019), covering training, hyperparameter, and layer-configuration faults across feed-forward, convolutional, and recurrent architectures. These taxonomies were proposed before the wide adoption of attention-based architectures and do not include transformer-specific fault patterns. The attention fault taxonomy (Jahan et al., 2025a; 2026) addresses this gap by classifying 555 attention-related faults into seven categories and 25 root-cause mechanisms. DEFault++ combines these attention categories with five non-attention categories drawn from the prior DNN taxonomies above (see Section 3.1). Table 24: Comparison of the DEFault-bench mutation process with existing mutation techniques Dimension

DeepMut.++

DeepCrime

DeepFD

PyTorchFI

DEFault-bench

Fault granularity Layer/param Source (AST) Program Bit/tensor Transformer unit Mechanism count 17a 24 impl.b 5 types Bit-flip 12 categories Architecture coverage FFNN/CNN/RNN Generic DNN Generic DNN CNN (infer.) Encoder + decoder Attention mechanisms ✗ ✗ ✗ ✗ ✓ Behavior change ✗ ✗ ✗ ✗ ✓ Decision basis Accuracy GLM + d Accuracy SDC rate Accuracy + sign-flip Noise control ✗ ✓c ✗ ✗ ✓ Scale (reported) Varies 1,760 52 programs 107 + inj. 3,739 Labeled traces ✗ ✗ ✓ ✗ ✓ a 8 source-level + 9 model-level mechanisms (Hu et al., 2019). b 35 defined, 24 implemented from real faults (Humbatova et al.,

2021). c DeepCrime uses GLM with n=20 retrainings and Cohen’s d for statistical killing. ✓ = supported; ✗ = not supported.

Mutation Testing for Deep Learning. Mutation testing is a well-established technique in software engineering for assessing test quality, typically by introducing small, controlled changes to a program and checking whether existing tests detect them (see Table 24). Several researchers have adapted this idea for deep learning. DeepMutation (Ma et al., 2018a) and DeepMutation++ (Hu et al., 2019) introduced mutation operators that modify training code, model structure, or learned parameters, and use mutation scores to quantify test adequacy for neural models. A fundamental challenge in this setting is training stochasticity. The same injected change can appear harmful under one random seed and negligible under another, and seed-induced variance can even rival the effect size of subtle faults (Mosbach et al., 2021). Probabilistic mutation testing addresses this by treating mutation killing as a statistical decision over repeated runs (Tambon 54

et al., 2023). DeepCrime (Humbatova et al., 2021) adopts a similar strategy, performing multiple retrainings and applying statistical tests with effect-size thresholds to determine whether a mutation has a significant impact. A smaller number of studies use injected faults explicitly for fault diagnosis. DeepFD (Cao et al., 2022) injects common fault types into deep learning programs and trains classifiers on runtime features to identify fault categories. Subsequent work extends this idea to larger corpora of faulty programs through data-driven localization (Wardat et al., 2023). These approaches remain limited for transformer architectures for two reasons. First, they target generic model and training faults (e.g., loss misconfiguration, incorrect layer numbers) rather than faults within the transformer architecture. Second, these works rely on high-level model performance (e.g., accuracy drop) and standard runtime features that do not capture the internal structure of attention-based models. Debugging Techniques for Deep Learning Programs. Recent work has framed DL fault diagnosis as a classification problem using runtime information collected during training. DeepFD (Cao et al., 2022) extracts features from loss curves, gradients, weight statistics, and activation patterns, and trains a classifier to identify common training faults such as an incorrect loss function, learning rate, or activation function. DEFault (Jahan et al., 2025b) extends this idea into a hierarchical classifier that first detects whether a program is faulty and then assigns the fault to one of seven training or model fault categories. DEFault++ inherits this hierarchical structure, but changes the diagnostic target and the evidence source. It replaces architecture-agnostic DNN features with measurements taken from transformer components and adds root-cause diagnosis within each predicted category. As a result, DEFault++ can distinguish faults in QKV projection, masking, attention scoring, residual connections, and other transformer-specific components, whereas DeepFD and DEFault report generic DNN fault categories. Table 25: Comparison of fault diagnosis techniques for DL/DNN programs Technique

Target

AutoTrainer (Zhang et al., 2021) DNN ✓ Partial (5) UMLAUT (Yuan et al., 2021) DNN ✓ Partial DeepDiagnosis (Wardat et al., 2022) DNN ✓ ✓ (8) DeepLocalize (Wardat et al., 2021) DNN NaN/Inf only ✗ DeepFD (Cao et al., 2022) DNN ✓ ✓ (5) DEFault (Jahan et al., 2025b) DNN ✓ ✓ (7) ATTNChecker (Liang et al., 2025) Transformer NaN/Inf only ✗ AtPatch (Weng et al., 2026) Transformer Map outlier ✗ FT-Transformer (Dai et al., 2025) Transformer Soft errors ✗ DEFault++ (Ours)

Transformer

✓

✓ (12)

Attn-Aware Explanation

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓

Rule-based Rule-based Decision tree Numerical trace Feature attribution SHAP ✗ ✗ ✗

✓ (45)†

✓

FPG+prototype

FD = fault detection; FC = fault categorization; RC = root-cause diagnosis. † DEFault++ covers 45 root causes in total. Of these, 40 apply to encoder models and all 45 apply to decoder models because KV Cache faults are decoder-specific.

A second body of work uses predefined rules or localization heuristics. The closest neighbors to DEFault++ in this family are AutoTrainer (Zhang et al., 2021), UMLAUT (Yuan et al., 2021), and DeepDiagnosis (Wardat et al., 2022), all of which monitor training-time symptoms such as exploding gradients, vanishing gradients, and oscillating loss. Numerical-instability detectors such as DeepLocalize (Wardat et al., 2021) and GRIST (Yan et al., 2021) look for NaN or Inf values, while shape and invariant checkers such as Tensfa (Wu et al., 2021) and TFCheck (Ben Braiek & Khomh, 2023) validate tensor shapes and training-process invariants. A second cluster operates at 55

finer granularity: MODE (Ma et al., 2018b) and DeepFault (Eniser et al., 2019) localize faults at the neuron level, Apricot (Zhang & Chan, 2019) repairs faulty DNNs by adjusting model weights, and DeepSeer (Wang et al., 2023) supports interactive RNN debugging through learned state abstraction. None of these techniques observe the runtime behavior of attention components or distinguish among transformer-specific fault categories. As a result, transformer faults that preserve normal loss curves, valid tensor shapes, and finite numerical values fall outside their detection scope, which is the gap DEFault++ targets. A third body of work uses static analysis to find DL faults before or independent of training. Structural and numerical analyzers (NeuraLint (Nikanjam et al., 2021), DEBAR (Zhang et al., 2020), NerdBug (Jafarinejad et al., 2021)) check meta-model graphs, tensor abstractions, and API usage. Symbolic and backend-comparison analyzers (DeepCheck (Gopinath et al., 2019), CRADLE (Pham et al., 2019)) test trained networks or compare backends to locate library-level bugs, and BugLab (Allamanis et al., 2021) self-supervises bug detection on general source code. FL4Deep (Morovati et al., 2024) occupies a middle position: it combines static and dynamic information through a knowledge graph and ranks candidate root causes over that graph. These techniques answer a different debugging question and therefore are not direct baselines for DEFault++. Static and library-level analyzers find faults that show up in program text, symbolic constraints, API usage, tensor abstractions, or backend inconsistencies. DEFault++ targets faults that leave the transformer program structurally valid and numerically stable but change how attention components behave during training. A transformer can pass type checks, shape checks, and backend consistency checks while still producing abnormal attention behavior. The graph representations in NeuraLint and FL4Deep also serve a different purpose from the FPG: NeuraLint encodes program structure for pre-training checks, and FL4Deep encodes a static-plus-dynamic knowledge graph for ranking. In contrast, the FPG encodes how a fault in one transformer component reaches other components during training. We therefore treat these techniques as complementary and evaluate DEFault++ against dynamic techniques that use the same training-time input. Transformer-Specific Fault Analysis. The closest existing techniques to DEFault++ analyze transformer-internal behavior, but each targets a narrower failure mode than the taxonomy in Figure 3. ATTNChecker (Liang et al., 2025) flags NaN or Inf values during attention computation, AtPatch (Weng et al., 2026) localizes attention-map anomalies at the patch level, FTTransformer (Dai et al., 2025) addresses hardware soft-error tolerance for vision transformers, and Bug Attention Probe (Stein et al., 2025) applies attention probing for code understanding. None of them produces a fault category or a root cause across the transformer block. DEFault++ instead covers the full taxonomy and predicts faults at three levels: detection, category, and root cause. A separate line of transformer research uses input perturbation to study robustness and behavioral invariance (Jones et al., 2020; Ribeiro et al., 2020; Clark et al., 2019; Voita et al., 2019), including DeepTest (Tian et al., 2018) for autonomous-driving DNNs. These techniques target behavioral failures exposed by perturbed inputs, whereas DEFault++ targets implementation faults that persist during normal training without input perturbation. Recent LLM-assisted debuggers such as SoapFL (Qin et al., 2025), LLM4FL (Rafi et al., 2024), ChatDBG (Levin et al., 2025), and BugReAct (Islam et al., 2026) reason over source code, execution logs, or bug reports to localize bugs in general software. They are complementary to DEFault++: they read software artifacts after a failure is reported, while DEFault++ reads

training-time traces from transformer components and predicts a transformer-specific category and root cause. Explainability for Fault Diagnosis. Diagnostic output is more actionable when it exposes the evidence behind the prediction. Existing explanation methods fall into two families. Post-hoc methods compute explanations after a model has produced a decision. SHAP (Lundberg & Lee, 2017), DiCE (Mothilal et al., 2020), and Anchors (Ribeiro et al., 2018) return feature attributions, counterfactual examples, or rule-based explanations from a trained classifier. DEFault (Jahan et al., 2025b) used this style of explanation in its hierarchical diagnosis. Inherently interpretable methods build the explanation into the model itself. Prototype networks (Snell et al., 2017) represent each class as a learned prototype and explain a prediction through similarity to that prototype. Concept-bottleneck models (Koh et al., 2020) route predictions through human-readable intermediate concepts. DEFault++ follows the inherently interpretable line. Its diagnosis identifies which runtime feature groups distinguish the predicted root cause from the nearest alternative, and thus the explanation comes from the same model that produced the diagnosis rather than from a separate analysis applied afterward.

Conclusion

We presented DEFault++, a transformer-specific hierarchical diagnostic technique for detecting, categorizing, and localizing faults in transformer models. DEFault++ extends the hierarchical diagnosis idea introduced for FFNNs, CNNs, and RNNs by DEFault (Jahan et al., 2025b) to transformer architectures, and it reports which feature groups provide the main evidence for each diagnosis. DEFault++ organizes the diagnosis around a Fault Propagation Graph that encodes how faults propagate between transformer components. To support both training and evaluation, we constructed DEFault-bench, a benchmark of 3,739 labeled instances produced by DEForm, our transformer-specific mutation technique. On DEFault-bench, DEFault++ achieves an AUROC above 0.96 for detection, a Macro-F1 above 0.85 for categorization, and a hierarchical root-cause Macro-F1 above 0.85 on both encoder and decoder architectures. In a developer study with 21 practitioners across four debugging scenarios, the repair-action accuracy was 83.3% with DEFault++ assistance and 57.1% without it, and 90% of participants indicated that they would use such a tool in practice. Future work includes extending DEFault++ to compound faults, larger-scale architectures (e.g., mixture-of-experts and multimodal models), and live debugging environments. The replication package, including DEFault-bench, the DEForm operators, and the trained models, is publicly available at https://github.com/SigmaJahan/DEFaultplusplus-Transformer-Debugging.

Acknowledgements The construction of DEFault-bench required approximately 18,600 GPU-hours, executed on NVIDIA A100 (40 GB) and NVIDIA H100 (80 GB) GPUs provided through the Digital Research Alliance of Canada (formerly Compute Canada). We gratefully acknowledge the Digital Research Alliance of Canada for the computational resources that made this work possible.

References Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, and Purushotham Kamath. Keyformer: KV cache reduction through key tokens selection for efficient generative inference. In Proceedings of the 7th MLSys Conference (MLSys), 2024. 57

Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. Self-supervised bug detection and repair. In Advances in Neural Information Processing Systems, volume 34, pp. 27865–27876, 2021. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. Houssem Ben Braiek and Foutse Khomh. Testing feedforward neural networks training programs. ACM Transactions on Software Engineering and Methodology, 32(4):1–61, 2023. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Samuel Pittman, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. arXiv preprint arXiv:2104.00006, 2021. Andrew P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1145–1159, 1997. doi: 10.1016/S0031-3203(96)00142-2. Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249–259, 2018. Jialun Cao, Meiziniu Li, Xiao Chen, Ming Wen, Yongqiang Tian, Bo Wu, and Shing-Chi Cheung. Deepfd: Automated fault diagnosis and localization for deep learning programs. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 573–585, 2022. Rich Caruana. Multitask learning. Machine Learning, 28(1):41–75, 1997. Gautam Chakrabarti, Vinod Grover, Bastiaan Aarts, Xiangyun Kong, Manjunath Kudlur, Yuan Lin, Jaydeep Marathe, Mike Murphy, and Jian-Zhong Wang. Cuda: Compiling and optimizing for a gpu platform. Procedia Computer Science, 9:1910–1919, 2012. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP, pp. 276–286, 2019. Huangliang Dai, Shixun Wu, Jiajun Huang, Zizhe Jian, Yue Zhu, Haiyang Hu, and Zizhong Chen. Ft-transformer: Resilient and reliable transformer with end-to-end fault tolerant attention, 2025. arXiv:2504.02211v2. 58

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, pp. 16344–16359, 2022. Alana de Santana Correia and Esther Luna Colombini. Attention, please! a survey of neural attention models in deep learning. Artificial Intelligence Review, 55(8):6037–6124, 2022. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 4171–4186, 2019. Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In Proceedings of the International Conference on Machine Learning (ICML), pp. 2793–2803, 2021. Hasan Ferit Eniser, Simos Gerasimou, and Alper Sen. Deepfault: Fault localization for deep neural networks. In International Conference on Fundamental Approaches to Software Engineering, pp. 171–189, Cham, 2019. Springer International Publishing. Pál Erdős and Alfréd Rényi. On random graphs I. Publicationes Mathematicae Debrecen, 6:290–297, 1959. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. nature, 542(7639):115–118, 2017. Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 55–65, 2019. Michael G Findley, Kyosuke Kikuta, and Michael Denly. External validity. Annual Review of Political Science, 24(1):365–393, 2021. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi, Heng-Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, András Orbán, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, HyunJeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd Al Merey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura 59

Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornakov, Nithya Attaluri, Iñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Xavier Garcia, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogozińska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo-yiin Chang, Paul Komarek, Ross McIlroy, Mario Lučić, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Iinuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, 60

Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víctor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojciech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz Kępa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang, Richie Feng, Milad Gholami, Kevin Ling, Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Israel, Francesco Pongetti, Chih-Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ó Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Lee, Wojciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya Pranavi Potharaju, Eileen O’Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Baetu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Ähdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojciech Rzadkowski, Fiona Mac61

intosh, Roopali Vij, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kärrman, Paweł Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang, Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikołaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Duc Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Sertan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi PontTuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir 62

Levine, Ariel Stolovich, Rebeca Santamaria-Fernandez, Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Põder, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszcz, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette-Choo, Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Klimczak-Plucińska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M., Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vít Listík, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirnschall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, 63

Ben Bariach, Laura Weidinger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. Aaron Gokaslan and Vanya Cohen. OpenWebText corpus, 2019. http://Skylion007.github.io /OpenWebTextCorpus. Phillip Good. Permutation, Parametric, and Bootstrap Tests of Hypotheses. Springer, 3rd edition, 2005. Divya Gopinath, Mengshi Zhang, Kaiyuan Wang, Ismet Burak Kadron, Corina Pasareanu, and Sarfraz Khurshid. Symbolic execution for importance analysis and adversarial generation in neural networks. In Proceedings of the IEEE International Symposium on Software Reliability Engineering (ISSRE), pp. 313–322. IEEE, 2019. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 1321–1330. PMLR, 2017. Qiang Hu, Lei Ma, Xiaofei Xie, Bing Yu, Yang Liu, and Jianjun Zhao. Deepmutation++: A mutation testing framework for deep learning systems. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1158–1161. IEEE, 2019. Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. Taxonomy of real faults in deep learning systems. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 1110–1121, 2020. Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. Deepcrime: mutation testing of deep learning systems based on real faults. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pp. 67–78, 2021. Ashraful Islam, Benjamin Lundell, Harpreet Sawhney, Sudipta N Sinha, Peter Morales, and Richard J Radke. Self-supervised learning with local contrastive loss for detection and semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5624–5633, 2023. Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. A comprehensive study on deep learning bug characteristics. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 510– 520, 2019. Niful Islam, Ragib Shahriar Ayon, Deepak George Thomas, Shibbir Ahmed, and Mohammad Wardat. When agents fail: A comprehensive study of bugs in llm agents with automated labeling. arXiv preprint arXiv:2601.15232, 2026. Foad Jafarinejad, Krishna Narasimhan, and Mira Mezini. Nerdbug: automated bug detection in neural networks. In Proceedings of the 1st ACM International Workshop on AI and Software Testing/Analysis, pp. 13–16, 2021. Sigma Jahan. DEFault++ replication package. https://github.com/sigmaJahan/DEFaultPlu sPlus, 2026. Available at: https://github.com/sigmaJahan/DEFaultPlusPlus. 64

Sigma Jahan, Saurabh Singh Rajput, Tushar Sharma, and Mohammad Masudur Rahman. Taxonomy of faults in attention-based neural networks, 2025a. URL https://arxiv.org/abs/2508 .04925. Sigma Jahan, Mehil B Shah, Parvez Mahbub, and Mohammad Masudur Rahman. Improved Detection and Diagnosis of Faults in Deep Neural Networks Using Hierarchical and Explainable Classification . In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 2944–2956, 2025b. doi: 10.1109/ICSE55347.2025.00224. URL https://doi.ieeecomputersociety.org/10.1109/ICSE55347.2025.00224. Sigma Jahan, Saurabh Rajput, Tushar Sharma, and Mohammad Masudur Rahman. Why attention fails: A taxonomy of faults in attention-based neural networks. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), 2026. Erik Jones, Robin Jain, and Percy Liang. Automatically finding bugs in a classifier: A systematic approach. arXiv preprint arXiv:2003.02907, 2020. Hans Kamp and Barbara Partee. Prototype theory and compositionality. Cognition, 57(2):129–191, 1995. Prannay Khosla, Yonglong Tian, Huiwen Wang, Ce Liu, Phillip Isola, Dilip Krishnan, Yao Tian, Tatsunori B. Hashimoto, Alan Yuille, Quoc V. Le, Cordelia Schmid, and Trevor Darrell. Supervised contrastive learning. In Advances in Neural Information Processing Systems, 2020. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015. Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 5338–5348. PMLR, 2020. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning (ICML), pp. 3519–3529. PMLR, 2019. Kyla H. Levin, Nicolas van Kempen, Emery D. Berger, and Stephen N. Freund. ChatDBG: Augmenting debugging with large language models. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2025. doi: 10.1145/3729355. Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3538–3545, 2018. Yuhang Liang, Xinyi Li, Jie Ren, Ang Li, Bo Fang, and Jieyang Chen. ATTNChecker: Highlyoptimized fault tolerant attention for large language model training. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (PPoPP), 2025. 65

Shuai Lin, Chen Liu, Pan Zhou, Zi-Yuan Hu, Shuojia Wang, Ruihui Zhao, Yefeng Zheng, Liang Lin, Eric Xing, and Xiaodan Liang. Prototypical graph contrastive learning. IEEE transactions on neural networks and learning systems, 35(2):2747–2758, 2022. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Joshi Mandar, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017. Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. Deepmutation: Mutation testing of deep learning systems. In Proceedings of the IEEE International Symposium on Software Reliability Engineering (ISSRE), pp. 100–111. IEEE, 2018a. Shiqing Ma, Yingqi Liu, Wen-Chuan Lee, Xiangyu Zhang, and Ananth Grama. Mode: automated neural network model debugging via state differential analysis and input selection. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 175–186, 2018b. Abdulrahman Mahmoud, Neeraj Aggarwal, Alex Nobbe, Jose Rodrigo Sanchez Vicarte, Sarita V. Adve, Christopher W. Fletcher, Iuri Frosio, and Siva Kumar Sastry Hari. Pytorchfi: A runtime perturbation tool for dnns. In 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops, DSN Workshops 2020, Valencia, Spain, June 29 - July 2, 2020, pp. 25–31. IEEE, 2020. doi: 10.1109/DSN-W50199.2020.00014. URL https://doi.org/ 10.1109/DSN-W50199.2020.00014. Ruchira Manke, Mohammad Wardat, Foutse Khomh, and Hridesh Rajan. Mock deep testing: Toward separate development of data and models for deep learning. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, ICSE ’25, pp. 2970–2982. IEEE Press, 2025. ISBN 9798331505691. doi: 10.1109/ICSE55347.2025.00220. URL https://doi.org/10.1109/ICSE55347.2025.00220. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330, 1993. Sergei Maslov and Kim Sneppen. Specificity and stability in topology of protein networks. Science, 296(5569):910–913, 2002. Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. Introduces the WikiText datasets. Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019. 66

Evan Miller. Attention is off by one. https://www.evanmiller.org/attention-is-off-by-one .html, 2023. Blog post. Mohammad Mehdi Morovati, Amin Nikanjam, and Foutse Khomh. Fault localization in deep learning-based software: A system-level approach. arXiv preprint arXiv:2411.08172, 2024. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=nzpLWnVAyah. Ramaravind K. Mothilal, Amit Sharma, and Chenhao Tan. DiCE: Diverse counterfactual explanations for machine learning classifiers. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT*), pp. 607–617. ACM, 2020. doi: 10.1145/3351095.3372850. Gireen Naidu, Tranos Zuva, and Elias Mmbongeni Sibanda. A review of evaluation metrics in machine learning algorithms. In Computer science on-line conference, pp. 15–25. Springer, 2023. Amin Nikanjam, Houssem Ben Braiek, Mohammad Mehdi Morovati, and Foutse Khomh. Automatic fault detection for deep learning programs using graph transformations. ACM Transactions on Software Engineering and Methodology, 31(1):1–27, 2021. NVIDIA. Nvidia deep learning profiler (dlprof) user guide, 2021. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Denis Paperno, German Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Roberto Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of ACL 2016, 2016. Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, and Lin Tan. Cradle: cross-backend validation to detect and localize bugs in deep learning libraries. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 1027–1038. IEEE, 2019. Yihao Qin, Shangwen Wang, Yiling Lou, Jinhao Dong, Kaixin Wang, Xiaoling Li, and Xiaoguang Mao. SoapFL: A standard operating procedure for LLM-based method-level fault localization. IEEE Transactions on Software Engineering, 51(4):1173–1187, 2025. doi: 10.1109/TSE.2025.3 543187. Ketai Qiu, Niccolò Puccinelli, Matteo Ciniselli, and Luca Di Grazia. From today’s code to tomorrow’s symphony: The ai transformation of developer’s routine by 2030. ACM Transactions on Software Engineering and Methodology, 34(5), 2025. ISSN 1049-331X. doi: 10.1145/3709353. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 2019. Md Nakhla Rafi, Dong Jae Kim, Tse-Hsun Chen, and Shaowei Wang. A multi-agent approach to fault localization via graph-based retrieval and reflexion. arXiv preprint arXiv:2409.13642, 2024. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model-agnostic explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4902–4912. ACL, 2020. 67

Lucas Roquet, Fernando Fernandes dos Santos, Paolo Rech, Marcello Traiola, Olivier Sentieys, and Angeliki Kritikakou. Cross-layer reliability evaluation and efficient hardening of large vision transformers models. In Proceedings of the 61st ACM/IEEE Design Automation Conference, pp. 1–6, 2024. Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606–610. IEEE, 2007. Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. Gregory T Smith. On construct validity: issues of method and measurement. Psychological assessment, 17(4):396, 2005. Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017. Adam Stein, Arthur Wayne, Aaditya Naik, Mayur Naik, and Eric Wong. Where’s the bug? attention probing for scalable fault localization. arXiv preprint arXiv:2502.13966, 2025. Florian Tambon, Foutse Khomh, and Giuliano Antoniol. A probabilistic framework for mutation testing in deep neural networks. Information and Software Technology, 155:107129, 2023. doi: 10.1016/j.infsof.2022.107129. URL https://www.sciencedirect.com/science/article/abs/ pii/S0950584922002385. Also available as arXiv:2208.06018 [cs.SE]. Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. DeepTest: Automated testing of deepneural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering (ICSE), pp. 303–314, Gothenburg, Sweden, 2018. ACM. doi: 10.1145/31 80155.3180220. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998–6008, 2017. Elena Voita, Jean-Baptiste Talbot, Rico Sennrich Yandex Research Ivan Titov University of Edinburgh Moiseev, and Rico Sennrich. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 5797–5808, 2019. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of ICLR 2019, 2019. Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2495–2504, 2021. Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, and Tianyi Zhang. Deepseer: Interactive rnn explanation and debugging via state abstraction. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). ACM, 2023. doi: 10.1145/3544548.3580852. 68

Mohammad Wardat, Wei Le, and Hridesh Rajan. Deeplocalize: Fault localization for deep neural networks. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 251–262. IEEE, 2021. Mohammad Wardat, Breno Dantas Cruz, Wei Le, and Hridesh Rajan. Deepdiagnosis: automatically diagnosing faults and recommending actionable fixes in deep learning programs. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 561–572, 2022. Mohammad Wardat, Breno Dantas Cruz, Wei Le, and Hridesh Rajan. An effective data-driven approach for localizing deep learning faults. arXiv preprint arXiv:2307.08947, 2023. Kun Wei, Zhe Xu, and Cheng Deng. Compress to one point: Neural collapse for pre-trained model-based class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 21465–21473, 2025. Shihao Weng, Yang Feng, Jincheng Li, Yining Yin, Xiaofei Xie, and Jia Liu. Atpatch: Debugging transformers via hot-fixing over-attention. arXiv preprint arXiv:2601.21695, 2026. Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. Experimentation in Software Engineering. Springer, 2012. Tom Nuno Wolf, Fabian Bongratz, Anne-Marie Rickmann, Sebastian Pölsterl, and Christian Wachinger. Keep the faith: Faithful explanations in convolutional neural networks for casebased reasoning. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 5921–5929, 2024. Dangwei Wu, Beijun Shen, Yuting Chen, He Jiang, and Lei Qiao. Tensfa: detecting and repairing tensor shape faults in deep learning systems. In Proceedings of the IEEE International Symposium on Software Reliability Engineering (ISSRE), pp. 11–21. IEEE, 2021. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research. PMLR, 2020. URL http://proceedings.mlr.pres s/v119/xiong20b.html. Ming Yan, Junjie Chen, Xiangyu Zhang, Lin Tan, Gan Wang, and Zan Wang. Exposing numerical bugs in deep learning via gradient back-propagation. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 627–638, 2021. Zhuang Yuan, Antoine Tixier, Aashish Rastogi, Hung Bui, Ben Ramos, Raymond Toth, and Anup Rao. Umlaut: Debugging deep learning models using dynamic analysis. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1080–1091, 2021. Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pp. 40770–40803. PMLR, 2023a. 69

Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yuren Zhang, and Joshua M. Susskind. Stabilizing transformer training by preventing attention entropy collapse. In Proceedings of the International Conference on Machine Learning (ICML), pp. 40770– 40803, 2023b. Hao Zhang and W. K. Chan. Apricot: A weight-adaptation approach to fixing deep learning models. In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 376–387, San Diego, CA, USA, 2019. IEEE. doi: 10.1109/ASE.2019.000 43. Xiaoyu Zhang, Juan Zhai, Shiqing Ma, and Chao Shen. Autotrainer: An automatic dnn training problem detection and repair system. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pp. 359–371. IEEE, 2021. Yuhao Zhang, Luyao Ren, Liqian Chen, Yingfei Xiong, Shing-Chi Cheung, and Tao Xie. Detecting numerical bugs in neural network architectures. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 826–837, 2020.

DEFault++ Dataset Construction Hyperparameters

Table 26 lists the full fine-tuning hyperparameters used to construct DEFault-bench (Section 3). Encoder and decoder configurations are presented side by side. Table 26: Encoder and decoder fine-tuning hyperparameters used to construct DEFault-bench

Parameter

Encoder

Decoder

Epochs Batch size Optimizer Learning rate Weight decay ϵ Max gradient norm Warmup ratio Schedule Precision Max sequence length Seeds

5 16 AdamW 2 × 10−5 0.01 10−8 1.0 0.06 Linear with warmup fp16 (mixed) — {42, 123, 456, 789, 101112}

5 8 AdamW 5 × 10−5 0.01 10−8 1.0 0.06 Linear with warmup fp16 (mixed) 512 {42, 123, 456, 789, 101112}

DEFault++ Parameters

Table 27 lists the hyperparameter configuration for the DEFault++ neural hierarchical model. Settings are organized in three column groups (architecture, optimization, and loss/temperature/evaluation) for compactness. 70

Table 27: Hyperparameter configuration for the DEFault++ neural hierarchical model Architecture Embedding dimension Group hidden dimension GNN message-passing layers Dropout

Optimization 64 32 3 0.1

Loss / Temperature / Evaluation

Learning rate Weight decay Batch size Max epochs Early stopping patience

0.001 1 × 10−4 256 150 20

α (detection) β (category) γ (prototype) λ (root cause) βsibling (contrast.) Prototype temperature τ Sibling contrastive τs Cross-validation folds Fold strategy

1.0 0.5 0.3 1.0 0.5 0.1 0.1 5 GroupKFold

Record · ID 146921 · SHA-256 cd7a1de29f3006fc
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.