Deep4ge: DNN Training Trajectories for Fault Detection and Diagnosis Sigma Jahan
arXiv:2607.12868v1 [cs.SE] 14 Jul 2026
Dalhousie University, Canada [email protected]
Abstract—Deep learning systems often fail due to subtle implementation faults that alter training behavior. Recent work has studied how to detect and diagnose such failures from changes observed across training epochs. However, the software engineering community still lacks a public dataset of per-epoch training runs with documented fault history, feature extraction details, and clear reuse support for fault detection and diagnosis tasks. We present Deep4ge, a controlled benchmark of 14,227 training runs generated from 59 adapted TensorFlow/Keras deep neural network (DNN) programs collected from Stack Overflow. We generated faulty variants using 27 source-code transformations that introduce known faults across seven categories. The dataset contains 9,845 faulty runs and 4,382 correct baseline runs. For each run, we record 4 evaluation metrics and 26 features that measure training behavior at every epoch. These features capture weights, gradients, activations, accuracy and loss trends, learning rate, and hardware use. Deep4ge supports binary fault detection, multiclass fault diagnosis, and early fault prediction from partial training runs. We release the dataset and faultinjection framework at https://doi.org/10.5281/zenodo.20337241.
to run deterministically, apply these transformations, and log measurements at every epoch to form training trajectories. Deep4ge contains 14,227 training runs across seven fault categories and three architecture families. As a controlled benchmark, it can be extended by adding new transformations, programs, or features and regenerating the associated metadata. In summary, we make the following contributions: A labeled dataset of 14,227 per-epoch DNN training runs across seven fault categories. • An open-source fault-injection framework that applies these transformations and logs the resulting training runs with validation scripts. • Proof-of-concept fault diagnosis baselines on the dataset.
•
II. DATASET C ONSTRUCTION A. Program Selection
Index Terms—deep learning, mutation testing, fault diagnosis, training dynamics
I. I NTRODUCTION Deep neural network (DNN) components are now built into software systems, making training-time failures a major concern. Unlike traditional faults that produce explicit errors, Fig. 1. Workflow of Deep4ge construction DNN faults often appear as silent symptoms during training. Deep4ge is based on developer-posted code. We collected These symptoms include unstable optimization, vanishing or 59 TensorFlow/Keras DNN programs from Stack Overflow exploding gradients, and poor generalization. Debugging such (SO) that include the training code. We applied three criteria to faults therefore requires observing how training evolves over select programs. First, the post must contain a complete model time rather than examining only the source code or final output. definition with data loading, compilation, and training. Second, This methodological shift warrants datasets that record training the program must be repairable to run deterministically. Third, behavior throughout the learning process. the architecture must be in scope. We then manually fixed Recent work has proposed approaches for debugging DNN each program to ensure it runs successfully and instrumented systems, including AutoTrainer [1], UMLAUT [2], DeepLocalit to record per-epoch features during training. To reduce nonize [3], DeepDiagnosis [4], and DeepFD [5]. These approaches deterministic variability across runs, we fixed random seeds at are typically evaluated on a limited number of programs the framework and operator level and ensured consistent dataset or on study-specific injected faults, and the corresponding splits and initialization settings during execution. The seeding training runs are often not released in a form that supports setup allows the dataset to be regenerated under the same reuse or systematic re-evaluation. Mutation frameworks such configurations, supporting comparisons across fault categories. as DeepCrime [6] apply source-code transformations derived Collection date and identifiers. The artifact records the SO from real faults, but they mainly support mutation analysis and question identifier for each program and the collection date. operator evaluation. As a result, support for studying trainingEach program corresponds to a distinct SO question ID. time faults in deep learning systems remains limited. In this work, we address this gap with Deep4ge. We collect Architecture coverage. The dataset includes feedforward, TensorFlow/Keras programs from Stack Overflow, adapt them convolutional, and recurrent neural networks.
Licensing and attribution. The artifact stores program where d and p are the effect size and significance value metadata and links each program to its originating SO question, computed from AN and AM . We use Cohen’s d (β = 0.5), with license and attribution information included in the release. a Generalized Linear Model for the p-value (α = 0.05), and n = 15. A mutant that the criterion does not kill is likely B. Fault Injection equivalent to the original program, so we drop it and do Deep4ge focuses on faults in the model and training not include its runs in the release. For an operator O with code. Starting from each corrected program, we generate configuration space C, let K(O, T ) ⊆ C be the configurations faulty variants by applying one of 27 source-level mutation of O killed by T , and let TrainS denote the training set used operators to each variant. The operators are grouped into to select configurations. The mutation score is seven fault categories. The operator set combines a subset |K(O, T ) ∩ K(O, TrainS)| MS (O, T ) = . (2) of DeepCrime operators with additional layer-level operators |K(O, TrainS)| introduced in Deep4ge. From DeepCrime, we retained operators covering training configuration, activation, loss, optimizer, We retained a mutation operator only if it was killable on weight initialization and bias, and regularization. We excluded at least one program of the same architecture. The operator operators that modify training data or process-level control, as set achieved mutation scores of 0.85 (FNN), 0.87 (CNN), these are outside our scope. The Deep4ge framework adds the and 0.83 (RNN). Finally, we normalize filenames, convert full set of layer-related mutation operators not supported in logged boolean values, and discard runs that did not complete a single epoch. A validation script checks consistency across all prior work [6]. runs. Our replication package includes the construction scripts, The framework implements 27 mutation operators covering allowing practitioners to reapply operators, retrain models, and seven model and training fault categories represented in the regenerate the dataset from scratch. literature [6], [7]. Each released run is generated using one operator and receives that operator’s category in manifest.csv. Algorithm 1 Deep4ge construction loop A seed program can contribute separate variants from multiple Require: Programs P, operators O, retrainings n, seeds S 1: D ← ∅ ▷ collected runs categories, but each run has one category label. The diagnosis 2: for each program P ∈ P do benchmark is therefore a seven-class, single-label task and does 3: train P for n retrainings, log per-epoch values not evaluate multilabel diagnosis. 4: append the n correct baseline runs to D C. Training and Feature Extraction We train each mutated program and its correct baseline. Each run logs, per epoch, an epoch index, 4 metrics (training and validation loss and accuracy), and 26 dynamic features. These comprise four weight measures (large-weight count, unchangedmean and unchanged-standard-deviation flags, and NaN count), four accuracy/loss trends (gap, oscillation, decrease, and increase), four activation measures (dying ReLU, saturation, mean, and standard deviation), and ten gradient measures (vanishing/exploding flags, NaN count, mean, standard deviation, minimum, maximum, and median layer norms, and per-weight mean and standard deviation). The remaining four record learning rate, CPU use (%), peak GPU memory (MB), and system-memory use (%). GPU memory is zero when no GPU is available. We apply each operator by rewriting the program’s abstract syntax tree, which also inserts the logging callback (see Fig. 2).
5: for each operator O ∈ O do 6: M ← apply O to P 7: train M for n retrainings with seeds S, log per-epoch values 8: if IS K ILLED(P, M, T ) then ▷ Eq. 1 9: append the n faulty runs to D, label by category of O 10: end if 11: end for 12: end for 13: standardize, clean, and validate D 14: build the metadata indexing every run in D 15: return D and its metadata
III. DATASET S TRUCTURE AND ACCESS Dataset organization. The dataset contains three top-level items: metadata, 59 adapted Stack Overflow programs, and training logs of faulty and correct runs. Schema. Each entry in manifest.csv identifies the subset, SO ID, operator-derived fault category, run number, epoch count, and file path. Each run records, per epoch, an epoch index, four evaluation metrics, and 26 dynamic features. The data dictionary is included in the artifact.
D. Mutation Validation We follow the killing criterion of DeepCrime [6], which builds on the statistical mutation-killing notion of Jahangirova and Tonella [8] and is also adopted in DEFault [9]. Let AN and AM be the accuracy distributions of the original network N and a mutant M over n retrainings on the test set T . The mutant is killed by T when the difference between the two distributions is statistically significant with at least a medium effect: IS K ILLED(N, M, T ) ⇐⇒ d ≥ β ∧ p < α,
(1)
TABLE I DATASET SUMMARY Property Total training runs Faulty runs Correct baseline runs Unique Stack Overflow IDs Total epoch records Median epochs / run
TABLE II FAULT COUNTS BY ARCHITECTURE Value Arch.
#programs
#runs
#faulty
#correct
CNN FNN RNN
13 16 30
2,683 3,792 7,752
2,037 2,555 5,253
646 1,237 2,499
14,227 9,845 4,382 59 719,560 50
Availability. The dataset is released under CC BY 4.0 and the mutation framework under MIT, archived on Zenodo with
(a) Seed training fragment
(b) AST operator application
from CustomCallback import EnhancedLoggingCallback tree = ast.parse(source) model = Sequential() tree = ModifySavePath(suffix).visit(tree) model.add(Dense(4, input_dim=2, tree = ModifyCallbackFilename(csv_path).visit(tree) kernel_initializer="glorot_uniform")) OpClass = OPERATOR_CLASSES[op_code] model.add(Activation("sigmoid")) tree = OpClass().visit(tree) model.add(Dense(1)) ast.fix_missing_locations(tree) model.add(Activation("sigmoid")) code = compile(tree, "<ast>", "exec") model.compile(loss="mse", exec(code, {"__name__": "__main__"}) optimizer=Adam(learning_rate=1e-3), metrics=["accuracy"]) # example operator class class ChangeLearningRate(MutationOperator): cb = EnhancedLoggingCallback(train_dataset, def visit_Call(self, call): callback_filename) self.generic_visit(call) model.fit(x_train, y_train, epochs=50, batch_size=16, if is_compile_call(call): rewrite learning_rate validation_data=(x_test, y_test), callbacks=[cb]) return call
Fig. 2. AST-based mutation and feature logging (per-epoch)
TABLE IV M UTATION OPERATOR TAXONOMY BY FAULT CATEGORY
TABLE III S CHEMA SAMPLE SHOWING METADATA & PER - RUN FIELDS Type
filename subset so_id fault_category is_faulty run_number num_epochs csv_path
string enum string enum bool int int path
epoch train_loss, val_loss train_acc, val_acc Feature fields
Description Metadata Per-run record filename buggy or correct Stack Overflow question ID Operator category, or none True if the run is faulty Index within a program–operator pair Epochs recorded in the run Path to the per-run record
Per-run record (epoch-wise) int Epoch index float Training and validation loss float Training and validation accuracy num/bool 26 dynamic features (see data dictionary)
a citable DOI. Each run traces to its originating SO post and operator through the metadata. The release is version v1.0.0 (commit 3f30d85e4f81), with the repository at https://github.com/SigmaJahan/deep4ge and the archived release at https://doi.org/10.5281/zenodo.20337241. IV. DATASET C HARACTERIZATION
Median training loss (log scale)
Field
Category
Operators
Logs
Exemplar fault
Hyperparameter Loss Weight Layer Optimization Activation Regularization
HBS, HLR, HNE, HDB FLC WCI, WAB, WRB LKS, LCF, . . . , LCO (10) OCH, OCG ACH, ARM, AAL RCD, RAW, RCW, RRW
2,571 2,148 1,985 1,335 1,240 312 254
Bad learning rate Wrong loss function Bad weight init Wrong kernel size Optimizer swapped ReLU removed Dropout set to 1.0
Total
27 operators
9,845
+4,382 correct
2 × 10
10
6 × 10
0
0
−1
0 2 4 6 8 10 12 14 16 18 Deep4ge contains 14,227 training runs (9,845 faulty and Epoch 4,382 correct baselines) spanning 719,560 epoch records (median 50 epochs per run). The dataset covers seven fault Hyperparameter (n=2,571) Optimization (n=1,240) categories and three architectures (see Table IV). Activation (n=312) Loss (n=2,148) Regularization (n=254) Weight (n=1,985) Training dynamics. Within Deep4ge, the median trainingcorrect (n=4,382) Layer (n=1,335) loss trajectories form three broad behavioral groups (Fig. 3). Some categories clearly diverge from the correct baseline. The Activation trajectory does not converge within the first 18 Fig. 3. Median per-epoch training loss by fault category on a log scale (first 18 epochs) epochs, and the Loss trajectory settles on a different numerical scale. Others start high but converge within the displayed V. D ETECTION AND D IAGNOSIS B ENCHMARKS range. Hyperparameter faults recover within about seven epochs, Deep4ge enables a wide range of studies on training-time while Weight-initialization faults take until around epoch 13. Layer, Optimization, and Regularization faults closely track faults. As a proof-of-concept, we report baselines on two tasks: the baseline and are the hardest to distinguish from loss fault detection, which classifies runs as faulty or correct, and alone. These groupings show that the injected fault categories fault diagnosis, which assigns one of seven fault categories represented in Deep4ge differ in both when and how their to faulty runs. These baselines serve as a starting point and median loss deviates, which the final epoch alone cannot illustrate the analyses supported by the dataset. capture. Layer mutations are concentrated in CNN and RNN Experimental setup. We compare two feature represenprograms, where the applicable operators target structural tations. The final-epoch representation uses the 30 values choices such as kernel size and LSTM layer count (Table VIII). recorded at the last epoch. The trajectory representation
TABLE V D ETECTION & DIAGNOSIS : FINAL - EPOCH VS . TRAJECTORY FEATURES Task
Features
Model
F1
Macro-F1
Bal.Acc
MCC
Detection Detection Detection Detection
final-epoch final-epoch final-epoch final-epoch
Always-faulty Stratified LogReg RandForest
0.818 0.693 0.611 0.816
0.409 0.497 0.553 0.508
0.500 0.497 0.599 0.540
0.000 -0.006 0.184 0.150
Diagnosis Diagnosis Diagnosis
final-epoch final-epoch final-epoch
Stratified LogReg RandForest
0.141 0.214 0.355
0.141 0.214 0.355
0.141 0.258 0.422
-0.003 0.165 0.346
Detection Detection Detection Detection
trajectory trajectory trajectory trajectory
Always-faulty Stratified LogReg RandForest
0.818 0.693 0.693 0.825
0.409 0.497 0.583 0.553
0.500 0.497 0.599 0.568
0.000 -0.006 0.185 0.227
Diagnosis Diagnosis Diagnosis
trajectory trajectory trajectory
Stratified LogReg RandForest
0.141 0.257 0.381
0.141 0.257 0.381
0.141 0.320 0.465
-0.003 0.207 0.401
categories (Table VI). Weight-related faults achieve the highest performance, followed by Hyperparameter and Loss. In contrast, Optimization and Activation faults are the most difficult to diagnose. Fig. 4 shows that Activation faults are frequently misclassified as Optimization, while Optimization faults are often misclassified as Layer. These patterns suggest that some fault categories may exhibit similar training dynamics. Deep4ge enables further analysis of such patterns and supports future work on more discriminative features for difficult fault categories. TABLE VI FAULT DIAGNOSIS PERFORMANCE
TABLE VII C ROSS - ARCHITECTURE DETECTION
Fault Category
Setting
Per-class F1
activation
0.000
FNN CNN RNN in-distribution
F1
Bal.Acc
MCC
0.823 0.827 0.820 0.825
0.563 0.566 0.559 0.568
0.210 0.218 0.198 0.227
t
gh
n
io
ei
w
io n
re
gu
la riz at
ss
iz at
lo
tim
op
r
la ye
m
ar a
rp
hy
pe
ac tiv
at io
et e
n
r
True
0.523 summarizes each of the 30 features over the full run using six hyperparameter 0.384 statistics (mean, standard deviation, minimum, maximum, last layer loss 0.520 TABLE VIII value, and linear slope), yielding 180 features. We evaluate optimization 0.075 FAULTY RUNS BY ARCHITECTURE Logistic Regression (LR) and Random Forest (RF) [10] using regularization 0.462 5-fold group-aware cross-validation, grouping runs by SO weight 0.700 Category FNN CNN RNN program identifier so that runs from the same program do Hyperparameter 684 304 1,583 Loss 586 516 1,046 not appear in both training and test sets. This prevents leakage Weight 434 484 1,067 across runs from the same program and measures transfer to Layer 460 282 593 Optimization 270 360 610 held-out Stack Overflow programs. We report F1, Macro-F1, Activation 24 0 288 balanced accuracy, and the Matthews correlation coefficient Regularization 97 91 66 (MCC) [11], [12], along with 95% bootstrap confidence intervals computed over 1,000 resamples. We consider alwaysfaulty and stratified-random classifiers as trivial baselines. 1.0 0.00 0.05 0.06 0.35 0.11 0.00 0.42 activation Final-epoch features. The final-epoch features provide limited separation between faulty and correct runs. With final0.8 0.00 0.46 0.14 0.14 0.03 0.06 0.17 hyperparameter epoch features, Random Forest achieves an F1 score of 0.816 on detection (Table V). Most runs in the dataset are faulty (9,845 0.00 0.10 0.40 0.13 0.05 0.10 0.22 layer 0.6 of 14,227), so a classifier that labels all runs as faulty achieves 0.00 0.14 0.11 0.49 0.01 0.01 0.24 loss an F1 score of 0.818. Random Forest performs comparably to the trivial baseline in terms of F1. To reduce sensitivity 0.4 0.00 0.23 0.23 0.15 0.05 0.14 0.20 optimization to class imbalance, we consider MCC and balanced accuracy, which equal 0 and 0.5 respectively for the trivial classifier. 0.00 0.01 0.06 0.00 0.03 0.88 0.02 regularization 0.2 On these metrics, Random Forest improves modestly over the trivial baseline (MCC 0.150, balanced accuracy 0.540), while 0.00 0.01 0.00 0.00 0.01 0.00 0.98 weight Logistic Regression achieves a higher MCC (0.184). 0.0 All-epoch features. Trajectory features provide a stronger basis than final-epoch representations. Using the full training trajectory instead of only the last epoch increases Random ForPredicted est detection MCC from 0.150 to 0.227, with non-overlapping confidence intervals ([0.132, 0.169] versus [0.209, 0.245]). Fig. 4. Fault-diagnosis confusion matrix (normalized) Detection performance also improves in AUROC from 0.649 to 0.716 and in AUPRC from 0.804 to 0.842. The same pattern Cross-architecture transfer. We train RF detection models appears in diagnosis results, where Random Forest Macro- on two architecture families and test on the held-out third F1 increases from 0.355 to 0.381 and MCC increases from (Table VII). The in-distribution model achieves an MCC of 0.346 to 0.401. Both models use per-feature statistics with 0.227. Cross-architecture models achieve MCCs of 0.210 no hyperparameter tuning. Under this evaluation, Deep4ge’s (FNN), 0.218 (CNN), and 0.198 (RNN), suggesting that per-epoch trajectories support higher detection and diagnosis performance loss from held-out architectures is small but performance than final-epoch information alone. nonzero and is slightly larger for RNN. This larger gap is Per-category diagnosis. With per-epoch feature trajectories, consistent with differences between gradient flow across time we observe variation in diagnosis performance across fault steps in recurrent models and layer-wise propagation in FNN
and CNN architectures [13], although our evaluation does not isolate this mechanism. Overall, some fault-detection features transfer across the three evaluated architecture families, while others remain architecture-specific. Early detection. We evaluate whether fault detection is feasible from features extracted at a single training epoch. We repeat RF detection using only the feature values from epoch k, where k ∈ {1, 2, 5, 10, 15, 20, 25, 30, 40, 50}. MCC ranges from 0.14 to 0.19 across all values of k, compared to 0.227 for the in-distribution model that uses the full trajectory (Fig. 5). No single epoch outperforms the others, and all single-epoch models underperform the full-trajectory model. Under this evaluation, Deep4ge’s per-epoch trajectories support higher detection performance than single-epoch information alone.
Detection metric
0.8 0.6
source programs support replication and extension of existing techniques. Limitations: Deep4ge targets TensorFlow/Keras FNN, CNN, and RNN programs. Transformers fall outside its scope. Stack Overflow programs may not reflect industrial development practices, and mutation-derived faults may not represent naturally occurring faults. The results therefore do not establish generalization to industrial programs or real-fault distributions. Correct baselines were validated for execution and training, not semantic correctness. Hardware-related features depend on the execution environment and may vary across platforms. VIII. C ONCLUSION Deep4ge provides a controlled, mutation-based benchmark of 14,227 DNN training runs from adapted Stack Overflow programs. In our proof-of-concept baselines, per-epoch trajectory summaries produced higher MCC than final-epoch features for both fault detection and diagnosis. The dataset is publicly archived to support replication and future extension within its documented scope. R EFERENCES
0.4 F1 Balanced Accuracy MCC
0.2 1
5
10 15 20 25 30 40 Decision epoch k (single-epoch features)
50
Fig. 5. Single-epoch fault detection by decision epoch (RF, group-aware CV)
VI. R ELATED W ORK Deep4ge complements DEFault, our prior work on hierarchical and explainable DNN fault detection and diagnosis [9]. DEFault used the dataset to evaluate detection and diagnosis techniques. This paper documents its construction and characterizes how per-epoch trajectories vary across fault categories and architectures. To the best of our knowledge, Deep4ge is the first public dataset that links adapted DNN programs with mutation-derived categories and corresponding per-epoch trajectories. Mutation-testing work, including DeepMutation [14], DeepMutation++ [15], and DeepCrime [6], primarily focuses on operators and mutation analysis. Debugging techniques such as AutoTrainer [1], UMLAUT [2], DeepLocalize [3], DeepDiagnosis [4], and DeepFD [5] evaluate detection strategies on less reusable datasets. Deep4ge instead links source programs, mutation-derived categories, and per-epoch features in one reusable benchmark. VII. U SE C ASES AND L IMITATIONS Use cases: Deep4ge supports early fault detection from partial runs, comparison of detection and diagnosis techniques across categories and architectures, feature-importance analysis, and cross-architecture studies. Its trajectories, labels, and
[1] X. Zhang, J. Zhai, S. Ma, and C. Shen, “AutoTrainer: An automatic DNN training problem detection and repair system,” in ICSE. IEEE, 2021, pp. 359–371. [2] E. Schoop, F. Huang, and B. Hartmann, “UMLAUT: Debugging deep learning programs using program structure and model behavior,” in CHI. ACM, 2021, pp. 1–16. [3] M. Wardat, W. Le, and H. Rajan, “DeepLocalize: Fault localization for deep neural networks,” in ICSE. IEEE, 2021, pp. 251–262. [4] M. Wardat, B. D. Cruz, W. Le, and H. Rajan, “DeepDiagnosis: Automatically diagnosing faults and recommending actionable fixes in deep learning programs,” in ICSE. IEEE, 2022, pp. 561–572. [5] J. Cao, M. Li, X. Chen, M. Wen, Y. Tian, B. Wu, and S. Cheung, “DeepFD: Automated fault diagnosis and localization for deep learning programs,” in ICSE. IEEE, 2022, pp. 573–585. [6] N. Humbatova, G. Jahangirova, and P. Tonella, “DeepCrime: Mutation testing of deep learning systems based on real faults,” in ISSTA. ACM, 2021, pp. 67–78. [7] M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” in ESEC/FSE. ACM, 2019, pp. 510–520. [8] G. Jahangirova and P. Tonella, “An empirical evaluation of mutation operators for deep learning systems,” in ICST. IEEE, 2020, pp. 74–84. [9] S. Jahan, M. B. Shah, P. Mahbub, and M. M. Rahman, “Improved detection and diagnosis of faults in deep neural networks using hierarchical and explainable classification,” in ICSE. IEEE, 2025, pp. 2944–2956. [10] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001. [11] B. W. Matthews, “Comparison of the predicted and observed secondary structure of T4 phage lysozyme,” Biochimica et Biophysica Acta (BBA) – Protein Structure, vol. 405, no. 2, pp. 442–451, 1975. [12] D. Chicco and G. Jurman, “The advantages of the Matthews correlation coefficient (mcc) over F1 score and accuracy in binary classification evaluation,” BMC Genomics, vol. 21, no. 1, p. 6, 2020. [13] S. Grossberg, “Recurrent neural networks,” Scholarpedia, vol. 8, no. 2, p. 1888, 2013. [14] L. Ma, F. Zhang, J. Sun, M. Xue, B. Li, F. Juefei-Xu, C. Xie, L. Li, Y. Liu, J. Zhao, and Y. Wang, “DeepMutation: Mutation testing of deep learning systems,” in ISSRE. IEEE, 2018, pp. 100–111. [15] Q. Hu, L. Ma, X. Xie, B. Yu, Y. Liu, and J. Zhao, “DeepMutation++: A mutation testing framework for deep learning systems,” in ASE. IEEE, 2019, pp. 1158–1161.