Evaluation-Strategy Gap in Fault Diagnosis of Deep Learning Programs Sigma Jahan
arXiv:2606.26492v1 [cs.SE] 25 Jun 2026
Faculty of Computer Science Dalhousie University, Halifax, Canada [email protected] Abstract—Deep Learning (DL) programs can fail during training for many reasons, and diagnosing the cause is a costly and time-consuming maintenance task. Techniques for diagnosing such failures are commonly assessed using within-program crossvalidation, which may be inadequate for deployment settings involving previously unseen programs. It is therefore necessary to assess how performance differs across these settings and to identify the causes of any performance gap in established fault diagnosis techniques for DL. We investigate this gap using DynFault, a corpus of 5,542 fault-injected training traces from 38 real-world DL programs. We found a gap of 0.190 in balanced accuracy for existing fault diagnosis techniques between within-program evaluation and holding out whole programs. We also found the gap comes from program-level structure in the features, which led us to examine two runtime feature sets, curvature features and optimizer features, and their behavior on unseen programs. We found that curvature features are useful for instability detection on unseen programs, while optimizer and activation features help only on programs seen during training. Index Terms—deep learning, dynamic analysis, fault diagnosis, generalization, program-held-out evaluation
performance of fault diagnosis for entirely new DL programs. Systematic evaluation on excluded groups is common practice in cross-project defect prediction [11], [12] as well as grouped cross-validation for clustered data [13], [14]. However, similar evaluation strategies have not been widely adopted for fault diagnosis techniques that learn from runtime data, leading to two major gaps, as follows. The impact of evaluation strategy on fault diagnosis remains unclear. We have limited systematic evidence on how much the performance of existing fault diagnosis techniques changes under different evaluation strategies, since previous work often overlooks these evaluation setups. A decline in performance could indicate that the fault signal does not transfer or that within-program evaluation partly rewards a classifier for recognizing the program rather than the fault [9]. These explanations imply different causes for the observed decline, which cannot be identified from the performance drop alone. Thus, more comprehensive evaluation and analysis are warranted to understand how the performance of fault diagnosis I. Introduction techniques reflects the overall diagnostic capability. The value of runtime features remains unclear across Deep Learning (DL) programs are prone to training-time failures caused by configuration errors, implementation bugs, evaluation strategies. Existing fault diagnosis techniques differ and numerical instability [1], [2]. Diagnosing these failures is in terms of the runtime metrics collected to learn fault challenging, as DL programs can vary widely in architecture, patterns [6], [7], [8], [15], [16]. Richer instrumentation is data set, and training configuration. The loss landscape, the often necessary to collect these metrics, which can be costly. dynamics of the gradients and the convergence behavior of However, it remains unclear whether the choice of evaluation one program can differ substantially from another [3]. When a strategy affects the usefulness of these runtime metrics. Thus, training run diverges, overfits, or shows inconsistent training it is difficult to weigh the cost of richer logging against its and validation behavior, practitioners must diagnose the root likely benefit for unseen programs. To address these gaps, we study DynFault, a corpus of cause from available information (e.g., runtime metrics, code snippets, model outputs), often under tight time and compute 5,542 fault-injected training traces from 38 real-world DL budgets [4], [5]. Fault diagnosis techniques based on dynamic programs to compare two ways of splitting the data. In withinanalysis learn to identify such failures by observing runtime program evaluation, runs are split without regard to which signals (e.g., loss trajectories, gradient statistics, and optimizer program they came from. In program-held-out evaluation, all runs from a given program stay together, so a tested program states) [6], [7], [8], [9]. Existing fault diagnosis techniques often report strong results never appears in training. We examine three diagnostic tasks, under cross-validation within the program, and several also (a) fault-type classification, (b) catastrophic-instability triage, validate on a limited number of real-world fault benchmarks. and (c) training/validation mismatch detection. In each task, For these techniques, each program produces characteristic we also compare two runtime feature configurations, first-order patterns in runtime metrics, so a classifier trained in one optimizer and activation features, and second-order curvature set of programs can learn to identify those programs instead features built from Hessian-vector products (HVPs). In short, of faults [10]. Within-program splits put the same programs we ask the following research questions. in training and testing, and fixed benchmarks report one ▶ RQ1 : How large is the within-program vs. programaggregate score. Neither approach measures the impact on the held-out gap for fault-type diagnosis, and what causes
it? We measure the gap on the four-class fault-type task and then run four controls (program-identity prediction, label permutation, per-program normalization, and distributionlevel shift analysis). These controls trace the gap to programlevel structure in the features. ▶ RQ2 : When does catastrophic instability occur, and do
curvature features detect it on unseen programs? We find that instability occurs at initialization, before substantial training has taken place, and that curvature features improve detection on new programs at every observation window. They provide an instability signal that transfers. ▶ RQ3 : Do optimizer features generalize for train-
ing/validation mismatch detection? We find that optimizer and activation features improve mismatch detection within a program but lose much of that benefit on new programs. The extra logging adds accuracy that does not transfer, and within-program evaluation hides the loss. We release DynFault and the full evaluation framework as a replication package [17]. Table I positions our study within existing fault diagnosis research and shows that program-heldout evaluation is rarely reported for learning-based techniques. TABLE I Fault diagnosis techniques for DL programs and their evaluation scope Technique
Prog-held-out
Split
DeepFD [7] kNN/DT/RF 5 types Runtime Qi et al. [9] RF 5 types Runtime, coverage AutoTrainer [15] Rules 5 problems Runtime DeepDiagnosis [6] Rules + DT 8 symptoms Runtime DEFault [8] Hierarchical RF 7 types Runtime, static
Approach
× × — — ×
Within Within Within Within Within
Our study
✓
Both
LR
Categories Signals
4–6 types
Runtime (CF, OF)
✓ evaluated under a program-held-out split, × not reported, — not applicable (rule-based detectors). Categories are the distinct fault types, training problems, or symptoms a technique reports. Signals: runtime = per-epoch training metrics (e.g., loss, accuracy, gradient, weight). Coverage = neuron-coverage metrics. Static = code-structure features. CF & OF = curvature and optimizer feature sets. Parenthesized numbers are reported feature counts. Approach: kNN, DT = decision tree, RF = random forest, LR = logistic regression.
II. Background and Corpus A. Dynamic Analysis Techniques for Fault Diagnosis Dynamic analysis techniques learn from per-epoch scalar metrics of a training run, such as loss, accuracy, and gradient statistics, to classify fault types or recommend corrective actions [6], [7], [8], [9]. Existing work evaluates under withinprogram splits and, in several cases, on separate real-world fault benchmarks, but does not isolate how performance changes on a previously unseen program. Our study addresses this dimension between programs by separately characterizing two maintenance scenarios. Within-program reuse applies a model to new runs of a program already in the training data, and cross-program deployment applies it to a program not seen during training. B. The DynFault Corpus DynFault is a corpus of 5,542 labeled training traces constructed for this study. Each trace is a per-epoch CSV
of scalar training metrics produced by a mutation-injected run of one of 38 distinct real-world DL programs (FFNN, RNN, and CNN architectures). Each program is identified by its Stack Overflow post identifier (so_id). Baseline of the program and the mutation protocol. 38 DL programs are real-world examples curated from Stack Overflow [8]. We inject faults using the extended set of mutation operators, covering seven major fault categories based on empirical studies of real DL bugs [1], [8], [18], [19]. These categories are hyperparameter, loss function, activation, layer configuration, optimizer, weight initialization, and regularization. We classify each mutant as killed or survived using the Generalized Linear Model (GLM) criterion [20] from previous work [8], [19]. Mutants that are killed are assigned a fault label that reflects their mutation category. The Weight-initialization mutants did not produce killed runs, thus six of the seven categories appear in our labeled corpus. Runtime feature configurations. Prior diagnosis techniques use standard metrics such as loss, accuracy, and gradient statistics [6], [7], [8]. DynFault also includes curvature metrics derived from Hessian-vector products (HVPs) [21], motivated by evidence that Hessian-derived metrics can localize training instability and fault sources in DL programs [22]. The optimizerfeature configuration adds optimizer control-state and activation extrema metrics that gradient norms alone do not capture. We record the two configurations separately and treat them as alternative runtime feature designs (Section II-C). Scope. We use DynFault to measure how evaluation strategy and runtime feature set jointly determine diagnostic performance and generalizability (Table II). Mutation injection gives every trace a known label and a reproducible fault condition. This controlled design isolates evaluation bias and limits our conclusions to the injected-fault corpus. TABLE II Corpus composition of DynFault Item
Count
Notes
Total trace files Non-empty runs Crash-before-epoch Distinct programs CF runs OF runs Unstable runs
5,542 4,980 562 38 4,175 1,367 219
One CSV per mutated run At least one epoch row recorded Header only, no epoch data Unique program identifiers Curvature feature set Optimizer feature set Loss > 106 , 4.4% of non-empty
Fault categories (runs / programs) Activation Hyperparameter Layer Loss function Optimization Regularization
90 1,357 2,802 991 267 35
2 programs 22 programs 38 programs 14 programs 14 programs 1 program
CF = curvature feature set (HVP-based), OF = optimizer & activation feature set.
C. Two Runtime Feature Sets The runtime feature configuration determines what a trace contains and which failure modes can be diagnosed. Adding
HVP logging or extreme activation recording can make a previously trained diagnostic model inapplicable, since the model expects a different set of input features. We study two configurations present in DynFault. Curvature features. Curvature proxies are derived from the Hessian of the loss with respect to model parameters, H =
∇2θ L(θ)
Group
Metric
CF OF
Performance
Training loss Validation loss Training accuracy Validation accuracy
✓ ✓ ✓ ✓
✓ ✓ ✓ ✓
Curvature
Mean HVP (µHVP ) HVP standard deviation (σHVP )
✓ ✓
— —
Gradient
Mean gradient Gradient standard deviation Gradient extrema (min, median, max)
— — —
✓ ✓ ✓
Learning rate (ηt ) Optimizer / activation Activation mean (µact ) Activation standard deviation (σact )
— — —
✓ ✓ ✓
— — —
✓ ✓ ✓
(1)
where L is the training loss and θ denotes the parameter vector. Forming H explicitly is prohibitive for large networks, so practical implementations use Pearlmutter’s R-operator [23], a differential operator that propagates directional derivatives through the computation graph. The R-operator computes HVP in O(|θ|) time via one additional forward-backward pass. Hv = ∇θ (∇θ L)⊤ v (2) From the per-layer HVP vectors we extract scalar summaries, the mean µHVP and standard deviation σHVP , together with per-layer HVP norms ∥Hv∥2 . These curvature metrics are used in curvature-aware optimization [24] and have been linked to training instability at the edge of stability, where the largest Hessian eigenvalue hovers near 2/η [25], [26], [27]. Prior work shows that they localize fault sources in attention-based architectures more effectively than gradient norms alone [22]. Optimizer features. Optimizer features capture first-order gradient statistics and control-state metrics. Global gradient norm, sX ∂L 2 ∥∇θ L∥2 = (3) ∂θi i
TABLE III Runtime metrics per feature configuration
System resources
CPU usage GPU memory usage System memory usage
✓ recorded, — not recorded. CF: 6 metrics (4 performance + 2 curvature). OF: 15 metrics (4 performance + 11 optimizer and system). HVP = Hessian-vector product.
materially distort the fault-category distribution. Some analyses use configuration-specific subsets. Curvature-feature (CF) and optimizer-feature (OF) traces were collected for different sets of programs. When comparing absolute performance across CF and OF, we either (i) restrict to the subset of programs that appear under both feature configurations (overlapcontrolled comparison), or (ii) compare within-configuration generalizability gaps under the same feature configuration.
summarizes the overall magnitude of the update at each training step. In addition, we record the current learning rate ηt , B. Diagnostic Tasks per-layer activation statistics (mean µact , standard deviation 1) Fault-type classification: We predict the mutation cateσact ), gradient extrema (max |∇θ L|, min |∇θ L|), and system gory of a trace at three granularities: (i) six-class fault type, (ii) resource metrics. These metrics are available in current training four-class core fault type, and (iii) three-class fault family. This debuggers and monitoring dashboards [16], [28]. Table III task is the primary subject of RQ1. Program-held-out evaluation lists the runtime metrics recorded under each configuration. requires training examples for each class in every fold, so we Section III-C describes the summary statistics derived from treat the four-class task as the primary cross-program estimate each metric. (Section IV-A). Table III lists the global runtime channels shared across the 2) Catastrophic instability triage: We label a trace as two feature configurations. The instability analyses additionally unstable if the training or validation loss exceeds 106 at any include per-layer and profile summaries derived from these logged epoch. This threshold captures unambiguous numerical channels. The feature counts in Table VI therefore exceed the divergence. Under standard cross-entropy losses, a value of 106 channel counts shown here. corresponds to predicting near-zero probability for the true class across batches, a condition that is generally not recoverable in III. Methodology practice. A sensitivity check at thresholds 103 and 104 shows We outline the diagnostic tasks, feature extraction procedures, that the epoch-0 concentration result holds (Section IV-B). and evaluation protocols in Fig. 1. 3) Training/validation performance mismatch detection (T/Vmismatch): We label a stable trace as a T/V-mismatch case A. Analysis Subsets if the per-epoch boolean flag acc_gap_too_big is true in All learning-based analyses require epoch-level metrics. We at least one epoch of the observed window. The flag is set at therefore restrict to non-empty traces (Table II) that contain each epoch e when the absolute training–validation accuracy at least one logged epoch row. The 562 crash traces fall gap meets or exceeds 0.1 (10%): predominantly in Layer (355) and Hyperparameter (131), the two largest categories, with the rest in Loss (46), Optimization acc_gap_too_bige = 1 |acctrain,e − accval,e | ≥ 0.1 . (19), and Activation (11). Their exclusion does not appear to (4)
RESULT
EVALUATION SETUP
Two evaluation splits train
DynFault
Study design
5,542 traces 38 programs
3 diagnostic tasks 2 feature sets (cf, of)
Key finding test fold
Within-program
Program-held-out
unseen
each dot = a run · each colour = a program
balanced accuracy
DATA
gap chance
within
held-out
Fig. 1. Study design overview
This threshold and computation rule originate from the DEFault corpus construction protocol [8] and are applied identically (see our replication package [17]). To isolate mismatch effects from catastrophic failures, we restrict this task to stable traces only (i.e., no loss explosion in any epoch). C. Feature Extraction For each scalar metric channel ze observed over the first k logged epochs e ∈ {0, . . . , k − 1}, we extract two summary statistics: the window mean z̄k and the ordinary least-squares slope β̂k , Pk−1 k−1 (e − ē) (ze − z̄k ) 1X z̄k = ze , β̂k = e=0 , (5) Pk−1 2 k e=0 e=0 (e − ē) where ē = (k − 1)/2. For a trace with d scalar channels, the feature vector at observation window k is xk ∈ R2d formed by concatenating the mean and slope for each channel. When k=1, the slope term is undefined, so we use only the mean statistics at k=1, giving d features. We consider observation windows k ∈ {1, 3, 5, 10, 20, 50}. We standardize features using statistics from the training split only and clip values to [±1012 ]. Missing channels are imputed with column medians computed on the training split. We use logistic regression with class-balanced weights and a one-vs-rest formulation for multi-class tasks. This setting provides a controlled linear probe of the feature representation under each evaluation protocol, keeping classifier capacity fixed while comparing within-program and program-held-out splits. Although logistic regression is linear, it can still capture program-level differences when those differences are present in the features. We use it to test whether the evaluation gap appears even without a high-capacity classifier. For comparisons with existing fault diagnosis techniques [7], [8], we report their original Random Forest classifiers alongside logistic regression (Section IV-C). On the same DEFault feature set, the gap widens from 0.088 under logistic regression to 0.34 under random forest (Fig. 13), so the logistic-regression gap we report for RQ1 is the conservative estimate. Logistic regression also enables the per-fold coefficient analysis in Fig. 11 that a random-forest ensemble would mask. D. Evaluation Strategies 1) Within-program evaluation: We perform stratified Kfold cross-validation over individual runs (K=5). Runs from
the same program may appear in both training and test folds. This protocol measures within-program reuse, where a trained diagnostic model is applied to new runs of a program already represented in training data. 2) Program-held-out evaluation: We perform grouped Kfold cross-validation over runs (K=5) with program identifiers as the grouping variable, so that each test fold contains only programs entirely absent from the corresponding training fold. This protocol measures cross-program deployment, where a model is applied to a program not seen during training. For the cluster-based curvature triage heuristic (Algorithm 1), we additionally report a stricter leave-one-program-out (LOPO) evaluation that holds out each program in turn. We use LOPO because the heuristic is trained without labels and produces a single interpretable “terminate” action, for which per-program precision is directly interpretable. 3) Metrics and uncertainty: For multi-class tasks we report balanced accuracy. Unless otherwise noted, “accuracy” refers to balanced accuracy throughout this paper. For binary tasks we report ROC-AUC and PR-AUC. PR and ROC curves pool outof-fold predicted probabilities across folds. We compute 95% confidence intervals via a cluster bootstrap over programs (with replacement) [29], preserving within-program dependence. 4) Evaluation-strategy gap: Let Mwithin (k) and Mheldout (k) denote the value of a metric M obtained under within-program and program-held-out evaluation, respectively, for the same classifier and feature vector xk . The evaluation-strategy gap is ∆M (k) = Mwithin (k) − Mheldout (k) .
(6)
A positive ∆M (k) indicates that within-program evaluation reports a higher figure than program-held-out evaluation at k observed epochs. E. Evaluation-Strategy Controls We use three classifier-based controls to identify the source of the within-program advantage. 1) Program-identity prediction: We predict the program identifier from early-epoch features. High accuracy indicates that feature vectors encode program-level structure beyond fault semantics. 2) Within-program label permutation: We randomly permute fault labels within each program, preserving each program’s label frequencies, and re-evaluate under within-program
0.25 0.17 0.33
Cross-prog.
Gap
0.469 [0.343, 0.545] 0.279 [0.232, 0.319] 0.190 [0.053, 0.258] 0.468 [0.310, 0.500] 0.178 [0.160, 0.281] 0.290 [0.091, 0.325] 0.518 [0.446, 0.595] 0.391 [0.346, 0.438] 0.128 [0.043, 0.206]
Shaded row is the primary four-class estimate. Values are balanced accuracy with 95% bootstrap confidence interval in brackets. Chance = 1/(number of classes). The six-class row is reported for completeness.
cross-validation. Above-chance performance after permutation indicates that within-program evaluation can succeed using program-level structure even when fault labels carry no discriminative information. 3) Per-program normalization (analysis-only control): As an analysis-only control (i.e., not deployable in practice), we z-normalize features within each program (i.e., subtract the per-program mean and divide by the per-program standard deviation) before any cross-validation split, since this uses unlabeled information from the held-out programs. If withinprogram performance drops substantially while program-heldout performance changes little, the evaluation gap is driven by program-level feature statistics rather than transferable fault signal. IV. Study Findings A. Evaluation-Strategy Gap Answering RQ1 : The within-program vs. program-held-out gap and its cause To answer RQ1 , we measure the within-program and program-held-out balanced accuracy for fault-type classification, and then apply four controls to identify what produces any difference between them. Table IV reports accuracy at k=5, a representative earlyepoch window. The full analysis across all k values is in our replication package [17]. We treat the four-class task (Layer, Hyperparameter, Loss, Optimization) as the primary estimate because each class appears in at least 14 programs, making program-held-out evaluation well-defined. The six-class task adds Activation (2 programs) and Regularization (1 program), so some program-held-out folds lack training signal for those labels. We report it for completeness and interpret its nearchance cross-program accuracy accordingly. This near-chance result (0.178 ≈ 0.17 chance) reflects the sparse program coverage for these two categories rather than a genuine absence of transferable signal. On the four-class task, within-program accuracy is 0.469 [0.343, 0.545] and cross-program accuracy is 0.279 [0.232, 0.319], a gap of 0.190 [0.053, 0.258]. In our experiments, the program-held-out score is only slightly higher than the 0.25 chance baseline, and its 95% CI includes chance-level performance. We therefore interpret this result as limited cross-program fault-type discrimination in this corpus, rather than deployment-ready diagnosis. The main empirical finding is the drop from within-program to program-held-out
0.5
0.5
0.4
0.4
Balanced accuracy
4 6 3
Within-prog.
0.3 0.2 0.1 0.0
Random CV Group CV (by so_id) Chance (1/4) 1 3 5
10
0.3 0.2 0.1
20
Epoch budget k (first k epochs)
0.0
50
(a) Four-class fault type
Random CV Group CV (by so_id) Chance (1/6) 1 3 5
10
20
50
Epoch budget k (first k epochs)
(b) Six-class fault type
Fig. 2. Evaluation-strategy gap across observation windows k
(a) Program-identity predictability. Fig. 3a shows that 38class program-identity accuracy reaches 0.62 at k=5 (chance is 0.026), rising monotonically with k. We observed that a diagnostic model operating on these features can identify the program independently of fault labels, suggesting that programlevel structure contributes to within-program CV performance. 1.0
0.8 0.7 0.6 0.5
Top-1 accuracy Balanced accuracy Chance (1/38)
0.4 0.3 0.2
0.6 0.4 0.2
0.1 0.0
Permuted (random CV) Permuted (group CV) Observed random CV (0.468) Observed group CV (0.178) Chance (1/6)
0.8
ECDF
Fault type Fault type Fault family
Classes Chance
Balanced accuracy
Task
evaluation. Our control experiments suggest that this drop is partly explained by program-specific patterns in the runtime features. Fig. 2 shows that this separation between within-program and program-held-out evaluation holds across all observation windows k on both task granularities. The primary four-class task preserves a positive gap at every k. On the six-class task the gap ranges from approximately 0.13 at k=1 to 0.29 at k=50, with the lower bound exceeding zero throughout (the minimum CI lower bound is 0.044, at k=1), so the gap is not explained by the observation window alone. We next use three classifier-based controls and one distribution-level analysis to identify its source.
Score
TABLE IV Evaluation-strategy gap at k=5
1 3 5
10
20
Epoch budget k (first k epochs)
(a) Predict program identity
50
0.0
0.20
0.25
0.30
0.35
Balanced accuracy (k = 5)
0.40
0.45
(b) Within-program accuracy
Fig. 3. Program-level structure controls at k=5
(b) Within-program label permutation. Fig. 3b shows that permuting fault labels within each program keeps withinprogram CV accuracy well above chance. Across 25 permutations, median permuted accuracy is 0.41, versus a 0.17 chance baseline (6-class uniform) and 0.47 for the unpermuted case. We found that a classifier achieves accuracy well above chance on within-program splits even when fault labels carry no discriminative information, suggesting that program-level structure in the feature space contributes to this performance independently of fault-label content. (c) Per-program normalization. Table V shows the normalization control. After z-normalizing features within each program, program-identity accuracy drops from 0.617 to 0.096 and within-program fault-type accuracy from 0.468 to 0.325,
while cross-program accuracy remains similar (0.178 to 0.158), (accuracy 0.41 vs. chance 0.17), and removable by per-program indicating that the within-program drop reflects removal of normalization without affecting cross-program performance. program-level shortcuts rather than loss of generalizable signal. Within-program evaluation remains valid as a measure of reuse performance, and the two strategies capture different signals TABLE V and should be interpreted accordingly. The remaining questions Per-program normalization control at k=5 examine which runtime features and diagnostic tasks remain effective under the harder cross-program standard. Metric
Setting
Fault-type accuracy
within cross
Raw
∆
Normalized
0.468 [0.310, 0.500] 0.325 [0.229, 0.473] −0.143 0.178 [0.160, 0.281] 0.158 [0.129, 0.269] −0.020
Program-identity accuracy within
0.617
RQ1 Summary. We found that within-program fault-type accuracy exceeds cross-program accuracy by 0.190 at k=5. Four converging controls confirm that program-level feature structure, rather than fault signal, accounts for this gap. These results show that within-program evaluation substantially overstates diagnostic ability.
0.096 −0.521
Values are balanced accuracy with 95% confidence interval in brackets (program-identity is plain accuracy). ∆ = normalized − raw.
1.75
211
1.0
1.50
MMD2 (RBF kernel)
1.25
1.00
0.75
0.50
0.25
0.00
Program (so_id)
Fig. 4. Pairwise MMD2 between program feature distributions at k=5
All four controls point to the same explanation. The withinprogram CV advantage is attributable to program-level structure in the feature space rather than transferable fault semantics. This structure is visible at the distribution level (median pairwise MMD2 0.716), predictable by a classifier (62% programidentity accuracy), exploitable even under permuted labels
Unstable runs (count)
200 150
epoch 0: 96.35% of 219
100 50 0 0
3
5
1
7
First instability epoch
(a) Timing of first loss explosion
ROC-AUC (grouped by so_id)
70818269 64151679 31556268 76132850 68716219 70848143 61553510 73146829 70178206 68061611 44164749 48934338 73148058 67649606 64522751 50306988 66907936 72328867 52782432 50481178 48385830 66468006 52448552 72965428 68323793 67169344 72795591 34311586 67155624 70217712 46642627 66840108 64634468 66952606 31880720 69906416 37624102 67590787
70818269 64151679 31556268 76132850 68716219 70848143 61553510 73146829 70178206 68061611 44164749 48934338 73148058 67649606 64522751 50306988 66907936 72328867 52782432 50481178 48385830 66468006 52448552 72965428 68323793 67169344 72795591 34311586 67155624 70217712 46642627 66840108 64634468 66952606 31880720 69906416 37624102 67590787
Program (so_id)
(d) Distribution-level domain shift. The controls above use classifier accuracy as a proxy for program-level structure. B. Runtime Feature Generalizability For a distribution-level analysis, we compute the pairwise Maximum Mean Discrepancy (MMD, RBF kernel with median Answering RQ2 : Instability timing and curvature-based heuristic bandwidth) [30] between the feature distributions of detection on unseen programs To answer RQ2 , we first determine when catastrophic all 38 programs at k=5 using the four shared performance instability occurs during training, and then test whether adding channels (loss and accuracy, mean and slope). The median curvature features improves its detection on unseen programs. 2 pairwise MMD across all 703 program pairs is 0.716, with Instability timing. Fig. 5a shows the distribution of the first a mean of 0.774 and a range of 0.0 to 1.93. Under the epoch of loss explosion across all 219 unstable runs. Of these, 2 RBF kernel with median heuristic bandwidth, MMD is 0 96% (211 of 219) explode at epoch 0 and 98% (214) by epoch 1. for identical distributions and increases without a fixed upper We found that this concentration holds across the instability bound as distributions diverge. A value of 0.716 therefore 3 threshold. At thresholds 10 and 104 , 99.0% and 91.9% of reflects substantial inter-program separation. Fig. 4 shows the runs, respectively, first explode at epoch 0 (vs. 96.3% heatmap sorted by mean domain shift. The large inter-program unstable 6 at 10 ). Lowering the threshold increases the flagged set (11.9% distances suggest that programs occupy distinct feature-space 3 of runs at 10 and 6.9% at 104 , vs. 4.4% at 106 ), so 106 is regions, consistent with the program-identity prediction in the most conservative threshold. These results indicate that, control (a). in our dataset, instability monitoring is most informative at initialization, before substantial training compute is spent.
0.9
0.8
0.7
0.6 base base+curv 0.5 1 3 5
10
20
50
Epoch budget k (first k epochs)
(b) Curvature feature benefit
Fig. 5. Instability characterization (a) 96% of unstable runs explode at epoch 0 (b) adding curvature features improves cross-program detection at every k
Curvature feature addition. Table VI reports instability detection under program-held-out evaluation for baseline features and baseline combined with curvature features. At k=1, adding curvature features improves ROC-AUC from 0.918 [0.707, 0.996] to 0.977 [0.930, 0.995]. The loss-ablated variants suggest that this gain is not explained solely by rising loss values. In our study, curvature provided an additional early indicator of instability. PR-AUC confidence intervals are wide at the 4.4% positive rate, because cluster-bootstrap resampling can produce folds with very few unstable runs, so ROC-AUC is the primary metric. Figure 5b shows curvature benefit across all k.
TABLE VI Cross-program instability detection
Feature set
k
#Features
Baseline Baseline
1 5
Baseline + CF Baseline + CF
TABLE VII Curvature cluster patterns
PR-AUC
19 38
0.918 [0.707, 0.996] 0.967 [0.906, 0.996]
0.776 [0.466, 0.956] 0.768 [0.635, 0.954]
1 5
22 43
0.977 [0.930, 0.995] 0.986 [0.948, 0.999]
0.896 [0.700, 0.977] 0.894 [0.752, 0.991]
No-loss baseline No-loss baseline
1 5
17 34
0.850 [0.566, 0.966] 0.954 [0.888, 0.988]
0.547 [0.181, 0.911] 0.601 [0.345, 0.899]
No-loss + CF No-loss + CF
1 5
20 39
0.896 [0.743, 0.966] 0.950 [0.865, 0.980]
0.756 [0.312, 0.916] 0.684 [0.325, 0.930]
Program-held-out evaluation, values with 95% confidence interval in brackets. ROC-AUC and PR-AUC = area under the ROC & precision–recall curves.
Cross-program risk stratification. Fig. 6 reports instability prevalence by quintile of mean HVP and gradient standard deviation at epoch-0. We observed that both metrics exhibit a consistent risk gradient across quintiles, which suggests that they may support a cross-program triage heuristic without a trained classifier. Fig. 7 shows the underlying pattern, where unstable runs exhibit uniformly elevated HVP magnitude across all network layers (median ≈0.7 log units above stable runs), suggesting that loss-surface curvature at initialization may be a whole-network property rather than a layer-specific artifact.
0 1 2 3 4 5 6
1346 121 367 100 536 933 234
19 9 8 4 6 17 8
0.5 100.0 0.8 10.0 2.2 1.9 3.4
49.1 3.3 28.6 0.0 0.9 50.2 8.5
6.1 × 10−1 5.4 × 106 6.4 × 10−1 9.2 × 103 1.9 × 10−1 5.2 × 10−2 4.0 × 101
curvature-feature runs by epoch-0 curvature geometry (HVP magnitude, gradient standard deviation, per-layer profile summaries) using k-means. Average silhouette coefficients peak at K=2 (0.742), indicating a dominant split between highcurvature and low-curvature runs. For the phenotype analysis we fit a finer K=7 partition over all 3,637 curvature-feature runs (Table VII, Fig. 8), which resolves outcome variation inside the low-curvature majority while keeping the extreme high-curvature cluster. The three largest low-curvature clusters (C0, C4, C5) share near-zero instability rates but differ sharply in mismatch prevalence. Table VII summarizes each cluster and Fig. 8 shows the PCA embedding. 4
C0 C1 C2 C3
0.175
0.175
0.100 0.075
0.125 0.100
0.050
0.050
0.025
0.025
0.000
0.000
Q1
Q2
Q3
Q4
Within-program quintile (Q1=lowest, Q5=highest)
Q5
(a) Mean HVP quintiles
1
0.075
PC2
Instability rate
0.125
Q1
Q2
Q3
Q4
Within-program quintile (Q1=lowest, Q5=highest)
Q5
(b) Gradient std quintiles
Fig. 6. Instability rate by epoch-0 quintile
0 1 2 3 2
log10 (|HVP norm|) (epoch 0, resampled)
C4 C5 C6 unstable
2
0.150
0.150
1.4 × 10−2 6.8 × 105 8.8 × 10−2 1.0 × 104 1.5 × 10−2 1.4 × 10−2 1.8 × 101
K=7 partition of 3,637 epoch-0 curvature-feature runs. n = runs in cluster, ∇-std = gradient standard deviation, HVP = Hessian-vector product. Low-curvature clusters (C0, C4, C5) show near-zero instability but differ sharply in mismatch prevalence.
3
0.200
Instability rate
n Programs Unstable (%) Mismatch (%) Median HVP Median ∇-std
Cluster
ROC-AUC
0
2
4
PC1
6
8
10
12
1.00 0.75
Fig. 8. PCA of epoch-0 curvature features by cluster (K=7)
0.50 0.25 0.00 0.25 0.50
Stable (n = 3458) Unstable (n = 179)
0.75 0.0
0.2
0.4
0.6
Normalized layer depth
0.8
1.0
Fig. 7. Layer-wise HVP magnitude at epoch 0
Curvature-based triage rule. We also examined whether epoch-0 curvature geometry can group runs into interpretable categories before training proceeds. We cluster the 3,637
Zero-false-positive early-termination rule. Cluster C1 (n=121, median mean HVP 5.37×106 , 9 programs) is 100% unstable. Treating C1 membership as a termination signal achieves precision 1.00, recall 0.676, and F1 0.807 on the 179 unstable curvature-feature runs in the full in-sample evaluation. Of these, 121 runs can be terminated at epoch 0 with zero false positives. Evaluation protocol distinction. The supervised instability models in Table VI use program-held-out grouped 5-fold crossvalidation (Section III-D). The triage heuristic below uses a stricter leave-one-program-out (LOPO) protocol because it is unsupervised and produces a single binary “terminate” action for which per-program precision is directly interpretable. Grouped 5-fold provides aggregate metrics with bootstrap
1.0
Algorithm 1. LOPO Curvature Triage (i)
confidence intervals. LOPO characterizes per-program precision of the decision rule. We assess out-of-sample precision using the LOPO triage evaluation in Algorithm 1. Across the 24 programs that have curvature-feature traces (of 38 total), which contain 179 unstable curvature-feature runs, the cross-program aggregate achieves precision 1.00 (TP = 10, FP = 0) and recall 0.056 (10 of 179, FN = 169). We observed that the zerofalse-positive property holds out-of-sample, since no stable run was assigned to a high-HVP cluster in our evaluation. LOPO recall is low because the extreme-HVP signature driving C1 (∼5 × 106 ) appears concentrated in a few programs. When C1-contributing programs are held out, their unstable runs are assigned to the next-highest-HVP cluster rather than detected. The rule therefore provides reliable precision across programs but limited recall outside the programs that formed C1. Both figures must be reported together. In-sample recall 0.676 and F1 0.807 characterize the rule within a known program subset, and cross-program recall 0.056 characterizes its generalizability. Precision-recall operating points. The low LOPO recall represents one operating point on a precision-recall trade-off. Fig. 9 shows PR curves under program-held-out evaluation for baseline and baseline+CF classifiers at k=1 and k=5. At k=1, the baseline+CF classifier achieves 0.90 precision at 0.96 recall, indicating that supervised curvature-based models can achieve both high precision and high recall in this setting. The LOPO cluster rule (star) operates at the extreme high-precision end (precision 1.00, recall 0.056), while the in-sample cluster rule (diamond) lies at precision 1.00 with recall 0.676. The operating point depends on the relative cost of false positives vs. missed instabilities. Mismatch-prevalence variation. Clusters C0 and C5 (n=1346 and n=933) exhibit mismatch prevalences of 49% and 50%, while C4 (n=536) has 0.9%. All three have instability rates
0.8 Precision
Input: Epoch-0 CF vectors {x0 }, program labels {p(i) }, instability labels {y (i) }, clusters K Output: LOPO precision and recall TP ← 0; FP ← 0; FN ← 0; foreach unique program p ∈ P do (i) Dtrain ← {x0 : p(i) ̸= p}; (i) Dtest ← {x0 : p(i) = p}; {µ1 , . . . , µK } ← k-means(Dtrain , K); c∗ ← arg maxj µj [mean-HVP] // highest mean-HVP centroid; (i) foreach (x0 , y (i) ) ∈ Dtest do (i) j ∗ ← arg minj ∥x0 − µj ∥2 ; ∗ ∗ if j = c then terminate; if y (i) = 1 then TP ← TP + 1; else FP ← FP + 1; else continue; if y (i) = 1 then FN ← FN + 1; TP TP return , ; TP + FP TP + FN
0.6 0.4 Baseline, k=1 Baseline + CF, k=1 Baseline, k=5 Baseline + CF, k=5 LOPO triage (P=1.00, R=0.056) In-sample triage (P=1.00, R=0.68)
0.2 0.0
0.0
0.2
0.4
prevalence = 0.049
Recall
0.6
0.8
1.0
Fig. 9. Instability precision–recall curves under program-held-out evaluation
below 2.3%. This 55× difference in mismatch prevalence, invisible at K=2, suggests that T/V-mismatch monitoring may be needed for some low-curvature program types but not others. These cluster patterns suggest that epoch-0 assignment may help prioritize monitoring. Programs resembling C0 or C5 showed high mismatch prevalence, whereas those resembling C4 did not. RQ2 Summary. We found that most instability originates at initialization, with 96% of explosive runs occurring at epoch 0. Incorporating curvature-based features improves crossprogram detection (ROC-AUC 0.918 → 0.977 at k=1). The unsupervised LOPO triage rule provides a high-precision but low-recall operating point, while supervised curvature models offer adjustable precision–recall trade-offs.
Answering RQ3 : Optimizer features for mismatch detection on unseen programs To answer RQ3 , we compare optimizer features against a loss-and-accuracy baseline for mismatch detection, under both evaluation strategies, on an overlap-controlled program subset and on the full subsets. Overlap-controlled comparison (primary). The two configurations cover different programs and mismatch prevalences, so we compare them on an overlap-controlled program subset. We therefore interpret absolute CF-vs-OF differences cautiously and focus on within-configuration gaps for the full subsets. Feature availability differs across configurations, so the stablerun sets are not identical. Table VIII reports the results. We observed that curvature features add little discrimination (within-program ROC-AUC 0.816 to 0.816, cross-program 0.730 to 0.731). Optimizer features substantially improve withinprogram detection (0.899 to 0.971) but slightly reduce crossprogram performance relative to the baseline model (0.826 to 0.782). The gap between within and held for the +OF condition is 0.189. Full-dataset analysis. Table IX reports the same analysis on the full program subsets. The CF subset has 3,278 stable runs across 22 programs at prevalence 0.37, and the OF subset
TABLE VIII Overlap-controlled T/V-mismatch detection (k=5)
n
Config Features Setting
π ROC-AUC
PR-AUC
CF
Baseline Baseline +CF +CF
within cross within cross
1317 .50 0.816 [0.618, 0.981] 0.855 [0.659, 0.973] 1317 .50 0.730 [0.475, 0.959] 0.773 [0.441, 0.921] 1317 .50 0.816 [0.618, 0.981] 0.856 [0.661, 0.974] 1317 .50 0.731 [0.476, 0.959] 0.774 [0.442, 0.921]
OF
Baseline Baseline +OF +OF
within cross within cross
410 .42 0.899 [0.704, 0.987] 0.907 [0.637, 0.988] 410 .42 0.826 [0.596, 0.970] 0.830 [0.491, 0.978] 410 .42 0.971 [0.860, 0.999] 0.957 [0.817, 1.000] 410 .42 0.782 [0.434, 0.988] 0.803 [0.444, 0.987]
n = stable runs, π = positive-class prevalence.
has 1,256 stable runs across 26 programs at prevalence 0.51. The two subsets cover different programs, so we focus on within-configuration generalizability gaps. Curvature features add negligible discrimination (within-program 0.799 to 0.799, cross-program 0.738 to 0.738). Optimizer features improve within-program detection (0.874 to 0.897) but degrade substantially under program-held-out evaluation (0.858 to 0.612), a generalizability gap of 0.285. The larger gap in the full dataset relative to the overlap-controlled subset (0.285 vs. 0.189) reflects sensitivity to program composition, as the two subsets cover different programs with different mismatch prevalences (Table IX). TABLE IX T/V-mismatch detection at k=5 on full dataset (stable runs only)
n
Config Features Setting
π ROC-AUC
PR-AUC
CF
Baseline Baseline +CF +CF
within cross within cross
3278 .37 0.799 [0.657, 0.918] 0.794 [0.611, 0.914] 3278 .37 0.738 [0.577, 0.871] 0.632 [0.410, 0.825] 3278 .37 0.799 [0.657, 0.919] 0.794 [0.611, 0.914] 3278 .37 0.738 [0.577, 0.872] 0.632 [0.411, 0.825]
OF
Baseline Baseline +OF +OF
within cross within cross
1256 .51 0.874 [0.681, 0.972] 0.922 [0.709, 0.987] 1256 .51 0.858 [0.671, 0.949] 0.892 [0.667, 0.974] 1256 .51 0.897 [0.728, 0.979] 0.916 [0.703, 0.987] 1256 .51 0.612 [0.371, 0.846] 0.566 [0.267, 0.873]
n = stable runs, π = positive-class prevalence. Optimizer features show a cross-program generalizability gap of 0.285.
1.0
0.8
ROC AUC
ROC AUC
0.9
0.8 0.7 0.6
0.7
Base | Random CV Base | Program-held-out +OF | Random CV +OF | Program-held-out
0.4 3
2
1
0
1
2
Logistic regression coefficient (5-fold random CV, k = 5) Fig. 11. Optimizer-feature logistic regression coefficients across withinprogram CV folds
Feature-group ablation. To identify which optimizer-feature groups are most associated with the cross-program degradation, Fig. 12 reports T/V-mismatch ROC-AUC at k=5 under programheld-out evaluation after removing each feature group in turn from the full OF channel set (1,256 stable runs, 26 programs). We found that removing activation statistics leads to the largest cross-program improvement (0.627 to 0.696, gap reduction from 0.270 to 0.195), suggesting that per-layer activation scales are a primary source of program-specific shortcut patterns. Removing system resource metrics also narrows the gap (0.270 to 0.236). In contrast, removing gradient statistics or learning rate has minimal effect on cross-program performance. The baselineonly condition (i.e., no OF channels) achieves the smallest gap (0.062) but retains cross-program ROC-AUC of 0.812, suggesting that loss and accuracy metrics alone may provide the most reliable mismatch detection across programs.
5
10
Observed epochs k (log scale)
20
50
All OF channels
= 0.270
System resources
= 0.236
Gradient stats
= 0.265
Activation stats
= 0.195
0.6 0.5
0.5 1
train acc slope std activation mean mean activation mean train acc mean gradient min mean val acc mean std grad mean val loss mean mean grad mean val loss slope gpu memory utilization mean cpu utilization mean adjusted lr mean val acc slope train loss mean memory usage mean train loss slope gradient median mean gradient max mean
1.0
Base | Random CV Base | Program-held-out +CF | Random CV +CF | Program-held-out
0.9
program-specific patterns rather than shared fault semantics. This pattern appears consistent with shortcut learning [10], where classifiers rely on correlations that hold in-distribution but not under shift. This interpretation is consistent with our cross-program results, where activation and optimizer features degraded on unseen programs with different characteristic scales.
1
3
5
10
Observed epochs k (log scale)
20
50
(b) Optimizer features (OF)
Learning rate
Fig. 10. T/V-mismatch ROC-AUC across observation windows
Baseline only (no OF)
(a) Curvature features (CF)
Fig. 11 shows that the linear model’s optimizer-feature coefficients vary substantially across within-program CV folds, which suggests that these features are being used through
= 0.254
Within-program Cross-program
= 0.062
0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95
ROC-AUC
Fig. 12. Feature-group ablation for T/V-mismatch detection at k=5
C. Re-evaluation of Existing Techniques To assess whether the evaluation-strategy gap generalizes beyond our linear methodology, we reimplement the feature extraction code of four existing dynamic-analysis techniques (DEFault [8], DeepFD [7], AutoTrainer [15], and DeepDiagnosis [6]) and evaluate them on DynFault under both settings on the four-class fault-type task at k=5. We follow the feature definitions in the original papers [6], [7], [8], [15] and use their provided replication packages. We reproduced the reported accuracy values to within 3% [17]. Learning-based techniques. DEFault features (42 features) and DeepFD features (160 features) are evaluated with both Random Forest (RF) and Logistic Regression (LR). Fig. 13 reports the results. With their original RF classifiers, both techniques achieve within-program accuracy above 0.65 but cross-program accuracy drops to 0.32, a gap of 0.34. This gap is larger than the 0.190 observed with logistic regression in RQ1, which suggests that the more expressive RF classifier captures more program-specific structure. Under LR, DEFault features show a gap of 0.088 (within 0.357, cross 0.270), while DeepFD features show a near-zero gap at near-chance performance, which suggests that DeepFD’s 160 statistical features are largely non-discriminative under a linear model. Rule-based techniques. AutoTrainer and DeepDiagnosis are rule-based detectors that identify the presence of training problems (e.g., vanishing gradient, dying ReLU) but do not classify the underlying fault type. AutoTrainer’s five rules trigger on 94.2% and DeepDiagnosis’s eight symptom detectors on 100% of traces in DynFault, indicating that the traces contain the monitored runtime metrics. The rules fire on all fault types without distinction, so they cannot produce fault-type classification metrics, and the evaluation-strategy distinction does not apply to fixed rules. We found that the evaluation-strategy gap is not an artifact of our linear methodology. It is larger, not smaller, with the higher-capacity RF classifiers used by existing techniques. In our experiments, the gap therefore appears to reflect the crossprogram generalization challenge itself rather than a specific classifier or feature set. D. Discussion and Implications Our results indicate that evaluation strategy and runtime feature design interact, jointly influencing both measured and actual diagnostic performance. Under within-program CV, we observed that all feature sets benefit from program-specific structure, so their apparent performance levels converge. Only under program-held-out evaluation did the generalizability differences between feature designs become visible in our experiments. This interaction has direct implications for feature
0.8 0.7 Balanced Accuracy
RQ3 Summary. We found that optimizer features substantially improve within-program T/V-mismatch detection but degrade under program-held-out evaluation (generalizability gap up to 0.285), driven primarily by per-layer activation statistics. These results reveal an observability–generalizability trade-off introduced by richer runtime features that is not visible under within-program evaluation.
0.68
Within-program Cross-program
0.68
0.6 0.5
Δ=0.35
0.4 0.3 chance
0.33
Δ=0.35
0.36 0.27
Δ=0.09
0.33 Δ≈0
0.26
0.26
0.2 0.1 0.0
DEFault (RF)
DEFault (LR)
DeepFD (RF)
DeepFD (LR)
Fig. 13. Within- vs. cross-program accuracy for DEFault and DeepFD at k=5
design. Optimizer features improve within-program mismatch detection (ROC-AUC 0.90 to 0.97 on the overlap-controlled subset), but this improvement degrades under program-heldout evaluation (0.78), exposing program-specific shortcuts. Curvature features, by contrast, provided a generalizable instability signal under both strategies in our experiments. We observed that features that appear similarly effective within a program pool may differ substantially when program-specific structure cannot be exploited. The appropriate instrumentation therefore depends on the task. Loss and accuracy features are often sufficient for mismatch detection, whereas curvature features are worth their additional cost when early instability must not be missed. We derive the following implications from these findings. 1. Align validation with deployment. Cross-program claims require program-held-out evaluation [11], [14], since withinprogram results may overestimate cross-program performance. Reporting program-identity predictability serves as an indicator of program-specific shortcuts. 2. Evaluate feature configurations separately. Additional logging channels can introduce shortcuts detectable only under grouped evaluation [31]. Each feature configuration should be evaluated under program-held-out splits before deployment. Our results indicate that per-program normalization reduces program-identity predictability without degrading cross-program performance, suggesting that normalizationinvariant designs may improve transfer. 3. Monitor instability at initialization. In our dataset, 96% of instability occurred at epoch 0, and curvature features generalized under program-held-out evaluation, supporting a high-precision early-termination rule, while supervised models offer adjustable precision–recall in this setting. V. Threats to Validity External validity. Our findings derive from 38 programs in DynFault with mutation-injected faults. Mutation enables controlled experimentation and the fault categories are grounded in empirical bug taxonomies [2], [19]. Mutants may not capture the full diversity of real-world training failures. We mitigate
optimism by adopting program-held-out evaluation, which Crime [19] design mutation operators grounded in empirical explicitly tests generalization to unseen programs, and we treat bug taxonomies. A recent study shows that pre-training mutants replication on real-fault corpora as future work to confirm that better match real faults than post-training mutants under the gap magnitudes hold beyond mutation-injected faults. coupling and behavioral-similarity criteria [37], supporting Construct validity. Fault-type labels correspond to the applied the construct validity of fault-injected corpora. DynFault mutation operators. The T/V-mismatch label relies on the corpus- follows this approach and uses mutation-injected runs to enable provided acc_gap_too_big flag with a fixed threshold [8]. controlled measurement of generalization across programs. The We fixed this threshold to match the DEFault corpus construc- need to align validation with deployment is well established in tion protocol and did not tune it per architecture. RNNs, CNNs, cross-project defect prediction [11], [12], [38], grouped crossand FFNNs differ in expected training/validation gap dynamics, validation for clustered data [13], and operational ML splitting so threshold sensitivity across architecture types should be decisions [14], [31], [39]. Separately, shortcut learning [10] exevaluated in future work. Alternative operationalization may plains why rich runtime features can degrade under distribution produce different absolute metrics, but the central observation shift, and curvature-aware analyses [23], [24], [25], [26], [27] that within-program evaluation reflects program-specific feature motivate curvature-derived signals as transferable instability indicators. Our study connects these threads by measuring the structure does not depend on the particular threshold. Internal and conclusion validity. We use linear models to evaluation-strategy gap for DL fault diagnosis and showing that reduce the risk that classifier capacity dominates the observed evaluation protocol and feature design interact to determine effects. Confidence intervals are computed via a cluster both measured and actual diagnostic performance. bootstrap over programs [29]. With 38 clusters, uncertainty VII. Conclusion is non-trivial, so we report full intervals and avoid overIn this work, we studied whether dynamic-analysis fault interpreting small differences. The CF and OF configurations diagnosis techniques for DL training failures generalize to were collected on partially different program subsets. To reduce programs that were not seen during training. Using DynFault, this confound, we report both overlap-controlled comparisons we found that fault-type diagnosis achieved higher balanced on shared programs and within-configuration generalizability accuracy under within-program evaluation than under programgaps on the full subsets. Absolute CF-vs-OF performance levels held-out evaluation, with a gap of 0.190. We also observed should therefore be interpreted cautiously. a similar performance gap across four existing fault diagnoVI. Related Work sis techniques. Further analysis indicates that program-level Empirical studies of DL bugs characterize common root structure in the runtime features explains much of this gap, causes and failure symptoms, and provide the taxonomies that while some fault-related information still generalizes across diagnosis and mutation operators build on [1], [2], [18], [32]. programs. Our feature analysis shows that gains from richer Building on these taxonomies, dynamic-analysis techniques runtime features do not always transfer to unseen programs. learn from runtime training metrics to detect and diagnose Curvature features helped detect instability on unseen programs. failures. DeepDiagnosis [6] and AutoTrainer [15] implement Optimizer and activation features improved training/validation symptom detectors and repair heuristics for training problems. mismatch detection mainly within the same program, with much DeepFD [7] and DEFault [8] frame diagnosis as supervised smaller gains on unseen programs. These findings suggest learning over aggregated runtime metrics, allowing fault-type that within-program evaluation can overestimate diagnostic classification and localization. Qi et al. [9] further augment performance and make richer logging appear more useful than runtime metrics with coverage-derived metrics. Across this line it is for cross-program diagnosis. Future work should assess of work, evaluations measure within-program reuse and, in whether these patterns hold on real-world faults and design several cases, detection on real-world fault benchmarks. The methods that help diagnostic models focus on fault-related cross-program generalization challenge, where the diagnostic behavior rather than program-specific runtime patterns. model encounters a program with different architecture, dataset, References and training dynamics, has not been separately measured. [1] M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive Additional techniques support interactive diagnosis and prestudy on deep learning bug characteristics,” in Proceedings of the 27th ACM Joint European Software Engineering Conference and execution checks. Cockpit [28] and UMLAUT [16] expose Symposium on the Foundations of Software Engineering (ESEC/FSE), gradient distributions, curvature-related signals, and program 2019, pp. 510–520. doi: 10.1145/3338906.3338955 structure to support human-in-the-loop debugging. DeepLocal[2] N. Humbatova et al., “Taxonomy of real faults in deep learning systems,” in Proceedings of the ACM/IEEE 42nd International Conference on ize [33] uses dynamic analysis of value propagation between Software Engineering (ICSE), 2020, pp. 1110–1121. doi: 10.1145/ layers to localize faults in DNNs. Hessian-derived metrics have 3377811.3380395 also been used to localize fault sources in attention models [22]. [3] H. Li et al., “Visualizing the loss landscape of neural nets,” in Advances in Neural Information Processing Systems (NeurIPS), 2018. Static analyses for numerical bugs [34] and design-by-contract [4] R. Zhang et al., “An empirical study on program failures of deep approaches for DL APIs [35] aim to prevent certain failures learning jobs,” in Proceedings of the ACM/IEEE 42nd International before execution. Conference on Software Engineering (ICSE), 2020, pp. 1159–1170. doi: 10.1145/3377811.3380362 Many diagnosis studies rely on mutation-injected faults to obtain labeled data at scale. DeepMutation [36] and Deep-
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
M. J. Islam, R. Pan, G. Nguyen, and H. Rajan, “Repairing deep neural networks: Fix patterns and challenges,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE), 2020, pp. 1135–1146. doi: 10.1145/3377811.3380378 M. Wardat, B. D. Cruz, W. Le, and H. Rajan, “Deepdiagnosis: Automatically diagnosing faults and recommending actionable fixes in deep learning programs,” in Proceedings of the 44th International Conference on Software Engineering (ICSE), 2022, pp. 561–572. doi: 10.1145/3510003.3510071 J. Cao, Y. Lu, M. Wen, and S. Cheung, “Deepfd: Automated fault diagnosis and localization for deep learning programs,” in Proceedings of the 44th International Conference on Software Engineering (ICSE), 2022, pp. 573–585. doi: 10.1145/3510003.3510099 S. Jahan, M. B. Shah, P. Mahbub, and M. M. Rahman, “Improved detection and diagnosis of faults in deep neural networks using hierarchical and explainable classification,” in Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE), 2025, pp. 2944–2956. doi: 10.1109/ICSE55347.2025.00224 X. Qi, T. Zhu, and Y. Li, “Coverage-enhanced fault diagnosis for deep learning programs: A learning-based approach with hybrid metrics,” Information and Software Technology, vol. 173, p. 107 488, 2024. doi: 10.1016/j.infsof.2024.107488 R. Geirhos et al., “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, pp. 665–673, 2020. doi: 10.1038/s42256020-00257-z T. Zimmermann et al., “Cross-project defect prediction: A large scale experiment on data vs. domain vs. process,” in Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, 2009, pp. 91–100. doi: 10.1145/1595696.1595713 C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto, “An empirical comparison of model validation techniques for defect prediction models,” IEEE Transactions on Software Engineering, vol. 43, no. 1, pp. 1–18, 2017. doi: 10.1109/TSE.2016.2584050 D. R. Roberts et al., “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure,” Ecography, vol. 40, no. 8, pp. 913–929, 2017. doi: 10.1111/ecog.02881 Y. Lyu et al., “An empirical study of the impact of data splitting decisions on the performance of AIOps solutions,” ACM Transactions on Software Engineering and Methodology, vol. 30, no. 4, pp. 1–38, 2021. doi: 10.1145/3447876 X. Zhang, J. Zhai, S. Ma, and C. Shen, “AutoTrainer: An automatic DNN training problem detection and repair system,” in Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 2021, pp. 359–371. doi: 10.1109/ICSE43902.2021.00043 E. Schoop, F. Huang, and B. Hartmann, “UMLAUT: Debugging deep learning programs using program structure and model behavior,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI), 2021. doi: 10.1145/3411764.3445538 S. Jahan, Replication Package for the Evaluation Strategy Gap Study, https : / / github . com / SigmaJahan / Evaluation - Strategy - Gap - Study, Accessed: 2026-06-24, 2026. Y. Yang, T. He, Z. Xia, and Y. Feng, “A comprehensive empirical study on bug characteristics of deep learning frameworks,” Information and Software Technology, p. 107 004, 2022. doi: 10.1016/j.infsof. 2022.107004 N. Humbatova, G. Jahangirova, and P. Tonella, “Deepcrime: Mutation testing of deep learning systems based on real faults,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2021, pp. 67–78. doi: 10.1145/3460319.3464825 J. A. Nelder and R. W. Wedderburn, “Generalized linear models,” Journal of the Royal Statistical Society Series A: Statistics in Society, vol. 135, no. 3, pp. 370–384, 1972. B. Ghorbani, S. Krishnan, and Y. Xiao, “An investigation into neural net optimization via hessian eigenvalue density,” in Proceedings of the International Conference on Machine Learning (ICML), 2019, pp. 2232–2241. S. Jahan and M. M. Rahman, “Can hessian-based insights support fault diagnosis in attention-based models?” In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering (FSE), New York, NY, USA: Association for Computing Machinery, 2025, pp. 676–680. doi: 10.1145/3696630.3728522
[23]
[24]
[25] [26]
[27] [28]
[29]
[30] [31]
[32]
[33]
[34]
[35]
[36]
[37]
[38]
[39]
B. A. Pearlmutter, “Fast exact multiplication by the hessian,” Neural Computation, vol. 6, no. 1, pp. 147–160, 1994. doi: 10.1162/neco. 1994.6.1.147 J. Martens, “Deep learning via hessian-free optimization,” in Proceedings of the 27th International Conference on Machine Learning (ICML), 2010, pp. 735–742. J. M. Cohen et al., “Gradient descent on neural networks typically occurs at the edge of stability,” arXiv preprint arXiv:2103.00065, 2021. S. Arora, Z. Li, and A. Panigrahi, “Understanding gradient descent on the edge of stability in deep learning,” in Proceedings of the 39th International Conference on Machine Learning (ICML), PMLR, 2022, pp. 948–1024. J. Gilmer et al., “A loss curvature perspective on training instability in deep learning,” arXiv preprint arXiv:2110.04369, 2021. F. Schneider, F. Dangel, and P. Hennig, “Cockpit: A practical debugging tool for the training of deep neural networks,” in Advances in Neural Information Processing Systems (NeurIPS), 2021. A. C. Cameron, J. B. Gelbach, and D. L. Miller, “Bootstrap-based improvements for inference with clustered errors,” The Review of Economics and Statistics, vol. 90, no. 3, pp. 414–427, 2008. doi: 10.1162/rest.90.3.414 A. Gretton et al., “A kernel two-sample test,” Journal of Machine Learning Research, vol. 13, no. 25, pp. 723–773, 2012. D. Sculley et al., “Hidden technical debt in machine learning systems,” in Advances in Neural Information Processing Systems, 2015, pp. 2503– 2511. Y. Zhang et al., “An empirical study on TensorFlow program bugs,” in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2018, pp. 129–140. doi: 10.1145/3213846.3213866 M. Wardat, W. Le, and H. Rajan, “DeepLocalize: Fault localization for deep neural networks,” in Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 2021, pp. 251–262. doi: 10.1109/ICSE43902.2021.00034 Y. Zhang et al., “Detecting numerical bugs in neural network architectures,” in Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2020. doi: 10 . 1145 / 3368089 . 3409720 S. Ahmed et al., “Design by contract for deep learning apis,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2023. doi: 10.1145/3611643.3616247 L. Ma et al., “Deepmutation: Mutation testing of deep learning systems,” in Proceedings of the IEEE 29th International Symposium on Software Reliability Engineering (ISSRE), 2018, pp. 100–111. doi: 10.1109/ ISSRE.2018.00021 Z. Ahmed, P. Makedonski, and J. Grabowski, “An empirical study of the realism of mutants in deep learning,” arXiv preprint arXiv:2512.16741, 2025. S. Herbold, A. Trautsch, and J. Grabowski, “A comparative study to benchmark cross-project defect prediction approaches,” IEEE Transactions on Software Engineering, vol. 44, no. 9, pp. 811–833, 2018. doi: 10.1109/TSE.2017.2724538 S. Amershi et al., “Software engineering for machine learning: A case study,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE-SEIP), 2019. doi: 10.1109/ICSE-SEIP.2019.00042