TensorMutation: Learning Weight-Space Mutations of Deep Learning Models on Neural Graphs

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

1

TensorMutation: Learning Weight-Space Mutations of Deep Learning Models on Neural Graphs

arXiv:2611.09650v1 [cs.SE] 6 Nov 2026

Zaheed Ahmed , Philip Makedonski , and Jens Grabowski

Abstract—Mutation testing for deep learning (DL) faces a cost– realism trade-off. The pre-training approach produces realistic but computationally expensive mutants, whereas existing posttraining operators modify trained models efficiently but produce mutants that less closely resemble real faults. This study asks whether the trade-off is necessary: can the weight-space effect of a fault-informed pre-training mutation operator be predicted from a trained model, without retraining it? We answer this question with TensorMutation, which represents a trained model as a neural graph and uses a mutation-conditioned graph neural network (GNN) to predict parameter changes. The GNN is trained offline and reused across unseen models so that TensorMutation can generate executable mutants without retraining. We construct a corpus of 25 252 original–mutant pairs from 71 multilayer perceptron (MLP) classifiers using 11 fault-informed pre-training operators, including 25 newly curated subjects. TensorMutation partially reproduces the behavioral effects of pretraining mutations on nine held-out subjects, and direct weightspace supervision significantly improves weight-space fidelity. On 12 real-fault subjects, TensorMutation achieves the highest median behavioral similarity among the evaluated approaches. The improvement over the post-training tool DeepMutation++ is statistically significant (Holm-adjusted p = 0.018), whereas the differences from the pre-training tool DeepCrime and the norm-matched random control are not significant. These results show that fault-informed pre-training mutations leave learnable structure in weight space that transfers to unseen models. Index Terms—Post-training mutation, weight-space learning, mutation testing, deep neural networks, neural graphs, graph neural networks.

I. I NTRODUCTION

M

UTANTS are commonly used as proxies for real faults to evaluate the effectiveness of test suites [1]. In deep learning (DL), their use extends beyond serving as test targets. Mutants have been employed to evaluate automated program repair [2], [3], robustness and adversarial testing [4]–[8], test generation, improvement, and prioritization [9]–[11], data cleaning [12], [13], fault localization [14]–[16], evaluation of metamorphic testing techniques [17], and several other testing and debugging tasks [18]–[20]. Across these applications, the usefulness of mutation-based evaluation depends on mutants being representative of faults that occur in real DL systems. At the same time, mutant generation must remain computationally practical if mutation testing is to be applied at scale. In DL, mutation approaches are commonly divided into pretraining and post-training mutation according to the stage at which the mutation operator is applied [21], [22]. Pre-training Zaheed Ahmed, Philip Makedonski, and Jens Grabowski are with the Institute of Computer Science, University of Göttingen, Germany (e-mail: zaheed.ahmed@informatik.uni-goettingen.de; makedonski@informatik.unigoettingen.de; grabowski@informatik.uni-goettingen.de).

mutation injects faults into the training program or training data and then retrains the model to obtain a mutant [21], [23]. Post-training mutation instead applies mutation operators directly to an already trained model, typically by perturbing its parameters or architecture [4], [21], [24], [25]. The cost of pre-training mutation becomes particularly high when multiple independently trained instances of each mutant are required to account for the stochastic nature of DL training, as in the statistical killing criterion [26] used by DeepCrime [23], [27]. In a preliminary controlled comparison on LeNet-5/MNIST, Abbasishahkoo et al. [28] reported that mutation analysis required less than 49 minutes with DeepMutation++ [4], compared with more than 123 hours with DeepCrime, a difference of more than 150 times. Although this comparison involved only one modest-sized subject, it illustrates the substantial computational overhead introduced by retrainingbased mutation. At the same time, our previous empirical study [29] found DeepCrime mutants to resemble real faults more closely than DeepMutation++ mutants. Established posttraining approaches avoid retraining through manually defined transformations of trained models rather than transformations learned from fault-informed pre-training mutations. Together, these findings motivate an approach that approximates the effects of fault-informed pre-training mutations while generating mutants without retraining. This study asks whether pre-training mutations leave learnable structure in the parameters of trained models. A pretraining mutation changes the training process and can therefore alter the learned parameters of the resulting model. Shah et al. [30] found systematic differences in the weights and biases of models trained with data faults, providing evidence that data faults can leave detectable parameter-space signatures. We therefore investigate whether the parameter changes induced by fault-informed pre-training mutation operators can be predicted for previously unseen models. The idea follows from our earlier finding for conventional programs that even chains of manually defined mutations fail to reproduce most real faults, which led us to propose learning mutations from real faults instead [31]. We call this approach TensorMutation because its learned transformation mutates the trained model’s weight and bias tensors. TensorMutation represents a trained model as a neural graph [32] and conditions a graph neural network (GNN) on the requested mutation specification through Feature-wise Linear Modulation (FiLM) [33] to predict parameter changes. It learns this transformation from pairs of original models and their pre-training mutants generated by the fault-informed operators of DeepCrime [23]. To generate a mutant without re-

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

training, TensorMutation applies the predicted changes directly to the trained model and performs any structural modifications required by the specified mutation. Thus, mutant generation is post-training, while the learned transformation is grounded in the effects of pre-training mutation. The GNN is trained beforehand on a corpus of such pairs, so mutant generation for unseen models requires only GNN inference and mutant reconstruction. Ideally, TensorMutation would learn directly from reproducible real bug-fix pairs. However, obtaining such pairs at the scale required for training remains difficult in DL. Jahangirova et al. [34] were able to reproduce only 54% of the faults considered from existing DL fault datasets, and Morovati et al. [35] found that only about 3% of the machine-learning-related bugs they investigated from GitHub and Stack Overflow were reproducible. Moreover, real faults lack a shared vocabulary: there is no agreed way for a user to request a particular fault type when generating a mutant. Faultinformed mutation operators provide a practical alternative. Each operator represents a fault type documented in empirical studies and provides an explicit specification for generating that type of fault. The same operator can also be applied across subjects and configurations to produce many training pairs. TensorMutation therefore learns from mutants generated by fault-informed pre-training operators rather than directly from real faults. These mutants form synthetic bug-fix pairs, where the original model serves as the fixed version and the mutant as the faulty version, while preserving the fault type represented by the operator. We collect these pairs at scale into the corpus released with this work. In this study, a subject is a TensorFlow/Keras program that trains a multilayer perceptron (MLP) classifier on its corresponding dataset. We evaluate TensorMutation across 71 subjects and 11 DeepCrime operators, with nine subjects held out for testing, and on 12 real-fault subjects. The evaluation asks four research questions (RQs) that build on one another. RQ1 and RQ2 examine how faithfully TensorMutation reproduces the corresponding pre-training mutants on unseen subjects, in behavior and in weight space. RQ3 then asks whether behavioral fidelity and weight-space fidelity are related. Finally, RQ4 investigates how closely TensorMutation mutants resemble real faults compared with mutants generated by DeepCrime, DeepMutation++, and norm-matched random perturbations, which preserve the magnitude of the predicted change but randomize its direction. We additionally examine, as a design ablation, whether conditioning the GNN on the source model’s training configuration improves the learned transformation. The main contributions of this work are: • We introduce TensorMutation, a learned post-training mutation approach that predicts the weight-space effects of fault-informed pre-training mutations and generates executable mutants without retraining. • We provide empirical evidence that mutation-induced parameter changes contain learnable structure that transfers to unseen subjects and that direct weight-space supervision improves weight-space fidelity. On 12 real-fault subjects, TensorMutation achieves significantly higher behavioral similarity to real faults than DeepMutation++.

2

We release a corpus of 25 252 original–mutant pairs from 71 subjects and 11 fault-informed pre-training operators, including 25 newly curated MLP subjects, to support research on testing and debugging of DL systems. The remainder of the paper is organized as follows. Section II reviews DL mutation testing and weight-space learning literature in relation to this work. The TensorMutation approach is presented in Section III. Section IV describes the study design and its four RQs. The results for the RQs, followed by the source-context design ablation, are reported in Section V. Section VI discusses the findings and their implications, and Section VII examines threats to validity. Finally, Section VIII concludes the study and outlines future directions. •

II. R ELATED W ORK This study lies at the intersection of two independent fields, mutation testing of DL systems and weight-space learning. We review each in turn and then position our work. A. Mutation Testing of DL Systems The literature on mutation testing of DL systems relevant to our work can be grouped along two axes: how mutants are generated and how realistic they are, and how the cost of mutation testing can be reduced. 1) Mutation Approaches: Mutation analysis was carried over to neural networks in 2018 by MuNN [24] and DeepMutation [21]. DeepMutation distinguished mutations applied before training from those applied directly to trained models, where a mutant costs a perturbation rather than a training run. DeepMutation++ [4] extended the post-training operators to recurrent networks. A systematic empirical evaluation of these operators [26] found that their effectiveness depends strongly on configuration, with some configurations yielding mutants that are trivial or impossible to kill. Recently, MuFF [25] introduced manually defined weight- and neuroninhibitor operators together with an automated stability check to obtain mutants that are stable across runs and sensitive to test-set quality. Unlike these approaches, DeepCrime [23], [27] grounds its operators in empirical studies of real faults [36]– [38]. Because those faults occur in the training program or data, DeepCrime is a pre-training technique: every mutant is retrained, and killing is decided statistically over repeated trainings to absorb nondeterminism. The assumption that mutants stand in for real faults was carried into DL without experimental verification. Our empirical comparison [29] introduced a statistical framework for coupling strength and behavioral similarity to real faults and found that DeepCrime mutants showed significantly stronger coupling and higher behavioral similarity than DeepMutation++ mutants. Realism therefore lies on the retraining side of this trade-off, the side that supplies the corpus TensorMutation learns from. 2) Cost Reduction Techniques: A separate line of work accelerates DL mutation testing without changing how mutants are generated. Some techniques execute fewer test inputs, for example by sampling inputs near the decision boundary [39].

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

Others generate or execute fewer mutants. Operator groups can be reduced based on the decision-boundary changes induced by their mutants [40], while quality-based criteria have been used to prioritize mutation operators [41], [42] and operator configurations [43]. Strongly subsuming second-order mutants can further replace multiple first-order mutants [44]. Clustering neurons or mutants with similar weights also reduces the number of mutants generated or tested [45]. Xue et al. make each execution cheaper by detecting behavioral deviations earlier through weak mutation [46]. All of these reduce how many inputs or mutants are executed, or how much each execution costs; none reduces the cost of producing a pre-training mutant, which is the cost of retraining. TensorMutation instead targets the cost of producing each mutant by replacing retraining with GNN inference and is complementary to these techniques. B. Weight-Space Learning Treating trained neural networks as a new data modality has emerged as a field of its own, organized into understanding the symmetries of parameter space, representing models as embeddings, and generating or transforming weights. We refer readers to Han et al. [47] for a general overview and confine ourselves here to the two strands TensorMutation builds on. 1) Weight-Space Representation: The idea behind weightspace representation is that properties of a trained network can be inferred from its parameters. Network accuracy can be predicted from weight statistics without evaluating the network on input data [48], properties of the training setup can be classified from weights [49], and self-supervised hyperrepresentations learned over model zoos capture both [50]. These findings suggest, but do not establish, that faults introduced during training may also leave learnable signatures in the resulting weights, which is the possibility TensorMutation investigates. Weight-space representations need to consider symmetry, since networks that differ only by a consistent permutation of hidden neurons compute the same function [51], [52]. Early equivariant architectures explicitly encoded permutation symmetries for specific network families [53], [54]. Graph-based approaches represent trained neural networks as graphs and process them with permutationequivariant GNNs [32], [55]. TensorMutation adopts this graph representation. 2) Weight-Space Editing: Weight editing changes the parameters of an existing model and can be expressed as θ̃ = θ + ∆(θ), where ∆ is a mapping over the weights [47]. Existing post-training operators that perturb parameters are hand-designed instances of such a mapping. In the weightspace learning literature, ∆ is instead learned from data, and TensorMutation adopts this formulation. Permutationequivariant metanetworks have been used to edit implicit neural representations directly in weight space [32], [54], [55], and interpretable edit directions have been identified in the weight space of customized diffusion models [56]. Two findings from weight generation and reconstruction are relevant to learning such edits. Meynent et al. [57] show that combining structural and behavioral supervision improves

3

neural network weight reconstruction. TensorMutation investigates a related principle for fault-induced transformations: its joint objective combines parameter-delta and behavioral supervision to approximate a specified mutation, rather than to preserve the original model’s behavior. Some generative weight models largely reproduce or interpolate their training checkpoints rather than generalize beyond them [58], which motivates our evaluation on subjects held out from training. A closely related direction is weight forecasting: NiNo predicts future weights from recent parameter states using a GNN over neuron connectivity [59]. This shows that training-related parameter updates can be learned, but does not establish that changes caused by training faults are similarly predictable, which is the question TensorMutation investigates. To our knowledge, no prior work in weight-space learning learns to change a trained model the way a specified fault would, and no DL mutation approach learns its operators as weight-space transformations. TensorMutation connects the two fields: it learns the weight-space effect of a fault-informed pre-training mutation and applies it to a trained model to generate mutants without retraining. III. T ENSOR M UTATION A PPROACH This section first defines the learning problem and provides an overview of TensorMutation. It then describes the main components of the approach. A. Problem Formulation and Overview Let Mo = (Ao , θo ) denote a trained original model, where Ao is its architecture and θo its trainable parameters. A mutation specification m = (operator, configuration) identifies both the pre-training mutation and the concrete setting in which it is applied. For example, the operator may change the learning rate, while the configuration gives the target value. A pre-training mutation tool realizes m by retraining the model under the mutated configuration. This produces a mutant Mm = (Am , θm ) and the corresponding parameter change ∆θm = θm −θo . TensorMutation learns to approximate this parameter change directly from the trained model and the requested mutation. It uses a GNN to process the model as a neural graph because message passing follows the model’s neuron–connection structure and can operate on subjects with different numbers of nodes and edges. Formally, it learns the conditional transformation c m, fϕ : (G(Mo , m), m) 7−→ ∆θ

(1)

where G(Mo , m) is the mutation-aware neural graph, ϕ dec m is the notes the trainable parameters of the GNN, and ∆θ predicted parameter change. The specification m appears twice because it provides local mutation information in the graph G(Mo , m) where applicable (Section III-C) and conditions the GNN on which transformation to predict (Section III-D). The transformation is learned from a corpus of (Mo , m, Mm ) training tuples as described in Sections III-B–III-G. The mutant Mm provides two kinds of supervision: its behavior on fixed probe inputs, which is the learning target, and its parameter change ∆θm , which can additionally supervise the prediction

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

4

Mutation Specification m = (operator, configuration)

e.g., m = (remove activation function, layer 1)

1. Trained Original Model Mo Architecture Ao Dense (128)

⋮

ReLU

WL

⋮

෢ ෢ 𝑚 = {Δ𝑊, ෢ Δ𝑏} Δ𝜃

෢ Weight deltas Δ𝑊

෢ Bias deltas Δ𝑏

γ, β

෢1 Δ𝑊

෢1 Δ𝑏

Dense (128)

Mutate Ao

W₁

b₁

6. Mutant Reconstruction

෢𝑚) ෡ m = ℛ(Am, θo + Δ𝜃 M

ReLU (Removed) Dense (64)

⋮

ReLU

WL

⋮

bL

⋮

෢𝐿 Δ𝑏

(Message Passing)

෢𝑚 Predicted deltas Δ𝜃 Parameters θo

⋮ ෢𝐿 Δ𝑊

5. Deterministic Mutations Architecture Am

4. Weight-Space Editing

FiLM

Graph Neural Network fϕ

Node features: bias, layer position, weight statistics, activation old/new, ... Edge features: weight, weight statistics, ...

Dense (10)

3. Graph Neural Network Encode (m)

… … …

bL

⋮

(e.g., new value, specific layer, …)

…

b₁

ReLU

Dense (64)

Configuration

(e.g., remove activation function, change epochs, ...)

2. Weight-Space Representation

Parameters θo W₁

Operator

Apply predicted deltas to parameters and combine with (potentially modified) architecture.

෡m 7. Generated Mutant Model 𝐌 Architecture Am Dense (128)

෠m Parameters θ Ŵ₁

෠ 𝑏₁

ReLU (Removed) Dense (64)

⋮

ReLU

ŴL

⋮

⋮

Dense (10)

Dense (10)

⋮

𝑏෠ L

Fig. 1. Mutant-generation workflow of TensorMutation. Given a trained original model Mo and a mutation specification m, TensorMutation constructs a mutation-aware neural graph, uses the trained GNN fϕ to predict parameter changes, applies deterministic architectural changes when required, and reconstructs cm without retraining. the executable mutant M

(Section III-F). We train the GNN beforehand, and at generation time a user supplies Mo and m. TensorMutation then c m to θo together applies the predicted parameter change ∆θ with any deterministic structural modification required by m. The mutant is therefore produced without retraining. Fig. 1 summarizes the complete mutant-generation process of TensorMutation. TensorMutation takes the trained original model Mo and the mutation specification m as input (Stage 1). It next transforms Mo into the mutation-aware neural graph described in Section III-C (Stage 2). The graph carries local mutation information where applicable, while the mutation specification m is supplied separately as the graph-level condition. The neural graph is then processed by the GNN fϕ , which is conditioned on m as described in Section III-D (Stage 3). This conditioning allows the same GNN to handle different mutation operators and configurations. Instead of predicting the complete set of mutant parameters, the GNN predicts c m defined in (5) the mutation-induced parameter change ∆θ (Stage 4). For operators that require an explicit architectural change, TensorMutation applies the deterministic mutation mechanics described in Section III-H (Stage 5). These mechanics modify the architecture as required by m while retaining the original parameters as the base for the predicted edits. If no structural change is required, the original architecture is retained. Stage 6 combines the predicted parameter changes with the architecture produced by the deterministic branch. TensorMutation then reconstructs the executable mutant. The cm is shown in Stage 7. resulting mutant M

B. Training Corpus Construction The GNN of TensorMutation is trained on pairs of original models and their pre-training mutants. For each original model, a mutation operator and configuration are applied to its training program, which is then trained to obtain the mutant. The resulting pair, together with its specification m, forms one training tuple (Mo , m, Mm ) and captures the effect of a known pre-training mutation. A key requirement is neuron-to-neuron correspondence between the two models. Hidden neurons can be permuted without necessarily changing the function represented by a network. Direct comparison of independently ordered models could therefore confuse the mutation effect with arbitrary neuron permutations. We preserve positional correspondence between the original and mutant models and verify it before the pair is used for learning (Section IV-D). Thus, a weight wij or bias bj at a given position refers to the same connection or neuron in both models, o m wij ↔ wij ,

boj ↔ bm j ,

(2)

where superscripts o and m denote the original and mutated models, respectively. This correspondence provides a valid basis for constructing the supervision targets for weight and bias deltas. C. Neural-Graph Representation Each aligned original–mutant pair is converted into one supervised neural-graph sample. Following the neural-graph formulation of Kofinas et al. [32], the original model provides

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

5

the graph structure and input features. Neurons are represented as nodes and trainable connections as directed edges. The corresponding mutant is used to construct the supervision targets. For a node vj and an edge eij from node vi to node vj , these targets are the mutation-induced parameter differences o ∆bj = bm j − bj ,

m o ∆wij = wij − wij .

(3)

For bias removal, the absent mutant bias is taken as zero, so ∆bj = −boj . Each node is therefore associated with an input feature vector xj and a target ∆bj , while each edge has an input feature vector xij and a target ∆wij . Fig. 2 shows one such sample together with its mutation specification m, whose conditioning of the GNN through FiLM is described in Section III-D. The node representation has 21 dimensions and combines original-model information with localized mutation information. It includes the bias value, layer position and width, incoming and outgoing weight statistics, activation identities before and after the requested mutation, bias and dropout information, and an indicator identifying the targeted layer. The 10-dimensional edge representation includes the original weight, normalized weight statistics, its position within the layer weight matrix, layer position, and adjacent-layer width information. The feature dimensions remain fixed while the numbers of nodes and edges vary with the network architecture, allowing the same GNN to process models of different depths and widths. D. Mutation Conditioning Each neural-graph sample is associated with the mutation specification m defined in Section III-A. The graph representation captures the local information needed to identify where the requested mutation acts. In parallel, the mutation specification provides a graph-level condition that tells the GNN which transformation to predict. TensorMutation encodes the mutation specification into a conditioning vector zm . The encoding contains a learned operator representation together with operator-specific configuration information. For numeric mutations, the configuration captures the change in the requested value. Categorical mutations encode the requested target value, while activation mutations encode the activation transition. The conditioning vector is applied through FiLM [33]. It maps zm to affine parameters γ(zm ) and β(zm ) and uses them to modulate the hidden state of each node, h′j = γ(zm ) ⊙ hj + β(zm ).

(4)

The same graph-level condition is therefore applied to all nodes belonging to the sample, while their individual graph features preserve the local structure and state of the original model. The learned operator representation is also provided directly to the node, message-passing, update, and edgeprediction computations, providing operator identity independently of the FiLM modulation. For readability, Fig. 2 draws the FiLM block for only one node. The same modulation is applied to every node of the sample.

Mutation-Aware Neural-Graph Representation (A Training Sample from the Original–Mutant Pair Corpus) Mutation specification m = (operator, configuration) Operator

Configuration

(e.g., remove activation function, …)

(e.g., new value, specific layer, …)

Edge information

FiLM (conditions

eij: xij | Δwij xij: edge feature vector (e.g., weight, weight statistics, layer index, adjacent-layer widths, …) Δwij: supervision target

node hidden states)

…

vi

… vj

… …

Node information vj: xj | Δbj xj: node feature vector (e.g., bias, activation old/new, layer flag, weight statistics, …) Δbj: supervision target

Fig. 2. Mutation-aware neural-graph representation used by TensorMutation. Each original–mutant pair yields one supervised graph sample. The original model provides the node and edge input features, while the corresponding parameter differences provide the bias- and weight-delta supervision targets. The mutation specification is associated with the graph and conditions GNN processing through FiLM.

E. Delta-Prediction Model TensorMutation uses a message-passing GNN fϕ [60] to predict the parameter changes associated with the requested mutation. Node features are first mapped to hidden representations. The GNN then performs two message-passing steps with a hidden dimension of 128. For each edge, separate forward and reverse messages are computed and aggregated at the corresponding nodes. This bidirectional propagation allows each node representation to incorporate information from both its incoming and outgoing neighborhoods. The mutation condition modulates the node hidden states through FiLM as described above. After message passing, separate node and edge heads produce the parameter deltas. The node head predicts a bias delta c j for each node vj . The edge head combines the conditioned ∆b hidden representations of the source and destination nodes d ij for with the edge representation to predict a weight delta ∆w each edge eij . FiLM conditioning therefore also influences the weight predictions through the conditioned endpoint representations. Together, the node- and edge-level predictions form the parameter transformation n o c m = ∆W cm . d m , ∆b ∆θ (5) The predicted edge residuals are rescaled using the rootmean-square magnitude of the corresponding source-layer weights, while bias deltas are predicted directly without layerwise scaling. F. Learning Objective TensorMutation is trained to reproduce the behavioral effect of the corresponding pre-training mutant on a fixed set of inputs, which we call probes. For each training subject, the probes are a fixed sample of up to 512 inputs drawn once

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

6

from the subject’s test split. The same probes are used for the original model and all of its mutants to provide behavioral supervision during GNN training. For each training sample, the predicted deltas are applied to the original model and the resulting mutant is evaluated differentiably on the probes. The behavioral objective combines two signals. First, the output distribution of the predicted mutant is matched to that of the reference mutant using the Kullback–Leibler (KL) divergence [61] over all probes. Second, on the probes that the original model classifies correctly, the predicted mutant is trained to reproduce the reference mutant’s kill behavior. A small regularization term discourages unnecessarily large parameter changes. The behavioral objective is Lbeh = LKL + Lkill + λr Lreg ,

(6)

where each L denotes a loss term minimized during training, Lreg is an L2 penalty on the predicted weight and bias deltas, and λr controls its contribution. Behavioral agreement alone does not require the predicted and reference mutants to change their parameters in the same way. The objective therefore also includes direct weightspace supervision. It aligns the directions of the predicted and reference parameter deltas,   d m , ∆Wm , LW = 1 − cos ∆W (7)   c m , ∆bm . Lb = 1 − cos ∆b (8) The full objective is L = Lbeh + λW LW + λb Lb ,

(9)

where λW and λb control the contributions of the weight- and bias-delta directional terms, respectively. When the reference delta is non-zero, the weight-space terms constrain the direction of the predicted delta, but not its magnitude. The scale of a predicted weight delta is adjusted by the layer-wise rescaling described above, while its magnitude is otherwise shaped by the behavioral objective and regularization. When a reference delta is zero, its direction is undefined. The corresponding weight-space term therefore penalizes a non-zero prediction instead of applying cosine alignment. Separately, we retain no-op training pairs whose target parameter transformation is zero. These pairs serve as identity anchors, allowing the model to learn the conditional zero transformation. G. Training the GNN The components described above form the GNN training loop shown in Fig. 3. Each GNN variant is trained once on the training partition of the corpus (Section IV-D). For each training tuple (Mo , m, Mm ), the GNN receives the neural graph of the original model Mo together with the mutation specification m. The reference mutant Mm is used only to provide the training targets. Its probe outputs and kill behavior provide the behavioral targets, while its parameter differences from Mo provide the weight-space targets when these are c m . These used. The GNN predicts the parameter deltas ∆θ deltas are added to the original parameters, and the resulting

Neural graph of Mo and mutation specification m

Training tuple (Mo, m, Mm) from the corpus

GNN fϕ two message-passing steps, FiLM conditioning on m

GNN Parameters ϕ

Mm

predict deltas Apply predicted deltas to Mo and evaluate on probes (differentiable)

update ϕ

Reference mutant Mm probe outputs, kills, ∆θm

predicted mutant behavior Optimizer (AdamW)

gradient

Loss: behavioral only or with weight-space supervision

targets

Fig. 3. Offline training of the TensorMutation GNN. The trained GNN is subsequently reused unchanged in the mutant-generation workflow of Fig. 1.

model is evaluated differentiably on the probes. The loss is then backpropagated to update the GNN parameters ϕ. The GNN can be trained with the behavioral objective in (6) alone or with the full objective in (9). The variants evaluated in this study, together with the optimization and checkpointselection settings, are described in Section IV-E. After training, the selected GNN is fixed and used for mutant generation as shown in Fig. 1. H. Deterministic Mutation Mechanics Some mutation requests contain an inference-visible change that is known directly from the mutation specification. TensorMutation applies such changes deterministically rather than requiring the GNN to infer them. This is required for the activation operators. Changing an activation function replaces the activation of the targeted layer with the requested one. Removing an activation function replaces it with the identity function, implemented as a linear activation. When the requested activation already matches the effective activation of the targeted layer, the request is a no-op. In this case, the architecture is left unchanged and the predicted parameter deltas are set to zero, which guarantees an exact identity transformation rather than relying on the learned approximation. For the remaining operators, reconstruction retains the original architecture. Training-configuration mutations and bias removal are represented entirely through the predicted parameter deltas. I. Mutant Reconstruction After the predicted parameter changes and any required architectural modifications are available, TensorMutation combines them to construct the executable mutant. The reconstruction corresponds to Stages 6 and 7 of Fig. 1. Formally,   cm , cm = R Am , θo + ∆θ M (10) where R(·) denotes model reconstruction. The resulting muc m , giving the executable tant parameters are θbm = θo + ∆θ b c mutant Mm = (Am , θm ). The same reconstruction procedure is also used to generate two auxiliary variants for controlled evaluation. The oracle

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

7

2. Held-Out Evaluation

1. Corpus and Model Training

3. Real-Fault Evaluation

Mutation Corpus (Sections IV-B to IV-D)

Generate Mutants (Section IV-F)

Real-Fault Benchmark (Section IV-D)

• • • •

• • •

• • •

71 MLP subjects from different datasets 11 pre-training operators of DeepCrime 5 seed-matched instances per configuration 25 252 original–mutant pairs

Candidates: TM-B and TM-J mutants Reference: DeepCrime mutant Control: norm-matched random perturbations

Real fault and its fixed version 13 candidate subjects: 6 test + 7 defect4ML 12 behaviorally active subjects retained

RQ1: Behavioral Learnability

Subject-Level Split (Section IV-D) •

Subjects with identical architectures stay together Partition Subjects Model Pairs Training 55 19 161 Validation 7 2454 Test 9 3637

Train TensorMutation (Section IV-E) • • •

TM-B: behavioral loss TM-J: behavioral and weight-space loss Checkpoint: best validation IoU

• • •

TM mutant vs. DeepCrime mutant Measures: IoU, CS Test: Wilcoxon on IoU, TM vs. random control

RQ2: Weight-Space Fidelity • • •

TM delta vs. DeepCrime delta Measures: cosine, log-norm ratio, normalized L2 Test: Wilcoxon on cosine and L2, TM-B vs. TM-J

RQ3: Weight–Behavior Relationship • • •

Cosine (RQ2) vs. IoU (RQ1) Measure: Spearman ρ, within and across subjects Reported descriptively

Compared Approaches (Section IV-F) • • • •

TensorMutation (TM-J) DeepCrime (pre-training) DeepMutation++ (post-training) Norm-matched random control

RQ4: Real-Fault Similarity • • •

Each mutant vs. the real fault Measures: IoU, mass ratio Test: Wilcoxon on IoU, TM-J vs. each comparator

Aggregation and Statistics (Sections IV-G and IV-H) • •

Unless stated otherwise, measures are summarized by medians: configuration → operator → subject. The subject is the unit of statistical inference. Paired comparisons use exact two-sided Wilcoxon signed-rank tests with rank-biserial r. The three RQ4 comparisons are Holm-corrected.

Fig. 4. Overview of the TensorMutation study design. The study constructs and splits the mutation corpus, trains TensorMutation, evaluates behavioral and weight-space fidelity on held-out subjects, and compares the generated mutants with real faults. TM-B and TM-J denote TensorMutation trained with the behavioral objective only and with the full objective, respectively (Section IV-E).

replaces the predicted deltas with the observed parameter deltas ∆θm of the corresponding pre-training mutant. The norm-matched random variant replaces the learned deltas with random directions while matching the node- and edge-delta norms separately. Their roles in the evaluation are described in Section IV-F. IV. S TUDY D ESIGN In this section, we describe how we set up our experiments to evaluate TensorMutation. We present the RQs, subject collection, mutation operators, corpus preparation, model configuration, comparison methods, evaluation measures, statistical analysis, and design ablations. Fig. 4 summarizes the experimental design used to evaluate TensorMutation. A. Research Questions To evaluate the learnability and effectiveness of TensorMutation, we organize the study around four RQs. RQ1 (Behavioral Learnability): Can the behavioral effects of pre-training mutations be learned as post-training weightspace transformations that generalize to unseen subjects? RQ1 tests the central idea behind TensorMutation: whether a learned parameter transformation can reproduce the behavior of the corresponding pre-training mutant on subjects not seen during training. On held-out subjects, we measure the kill-profile similarity of each TensorMutation mutant to its corresponding DeepCrime mutant. We compare these scores with those of norm-matched random counterparts, which retain the magnitude of the TensorMutation prediction but randomize its direction. This comparison tests whether the learned direction contributes to behavioral fidelity beyond perturbation magnitude alone.

RQ2 (Weight-Space Fidelity): How faithfully does TensorMutation reproduce mutation-induced parameter changes, and does explicit weight-space supervision improve this fidelity? RQ2 examines how closely the predicted parameter changes match those produced by the corresponding DeepCrime mutant on held-out subjects. We measure their agreement in direction, magnitude, and normalized error using cosine similarity, log-norm ratio, and normalized L2 distance. We compare training with the behavioral objective in (6) against the full objective in (9), which adds direct weight-space supervision. This shows whether explicit weight-space supervision improves parameter fidelity and whether it affects behavioral fidelity. RQ3 (Weight–Behavior Relationship): How does weightspace fidelity relate to behavioral fidelity, and does this relationship differ within and across subjects? This question asks whether more accurate parameter transformations also lead to more similar mutant behavior. We therefore relate the weight-space fidelity measures from RQ2 to the behavioral fidelity measured in RQ1. The relationship is examined using Spearman correlation both within subjects and across subjects. This distinction shows whether the association holds only among mutations of the same model or also across different models. RQ4 (Real-Fault Similarity): To what extent does the behavior of TensorMutation mutants resemble that of real faults, and how does this resemblance compare with existing mutation approaches and controls? This question moves beyond reproducing pre-training mutants and asks whether TensorMutation mutants resemble the behavior of real faults. We evaluate TensorMutation on the 12 behaviorally active real-fault subjects defined in Section IV-D

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

and measure the kill-profile similarity of each mutant to the corresponding real fault. TensorMutation is compared with DeepCrime, DeepMutation++, and the norm-matched random control, whose roles are described in Section IV-F. We further analyze behavioral severity and direction to explain the observed differences. B. Subject Collection Our study focuses on MLP classifiers implemented in TensorFlow/Keras. We build the development corpus from subjects previously reproduced in our empirical study of mutant realism [29] and supplement them with a new collection of MLP subjects. The previous study reproduced real faults from CleanML [12], DeepFD [14], DeepLocalize [62], and defect4ML [35]. For TensorMutation development, we consider the reproduced subjects from CleanML, DeepFD, and DeepLocalize whose models are MLP classifiers. We further exclude three subjects whose neural graphs exceed one million edges. Their graphs contain between 6.9 and 12.7 million edges, more than ten times the largest retained subject (0.67 million), and exceeded the memory and time budget of GNN training. This results in 19 CleanML, 17 DeepFD, and 10 DeepLocalize subjects. The fixed version of each retained subject provides the original model from which pre-training mutants are generated. To increase the diversity available for cross-subject learning, we additionally curated TFEX-MLP (TensorFlow Examples of MLPs), a set of 25 MLP classifiers. Its subjects are sourcetraceable TensorFlow/Keras programs collected from independently authored tutorials, books, repositories, and other public resources. The original architecture and training semantics are retained as closely as possible, while fixed data partitions and compatibility adaptations support reproducible execution and mutation generation. The resulting development corpus contains 71 subjects. The eligible defect4ML subjects serve only as an additional real-fault benchmark for RQ4. C. Mutation Operators We generate the mutation corpus with the pre-training mutation tool DeepCrime [23]. It provides 24 mutation operators for both the training program and the training data. These operators were derived from three studies of real faults [36]– [38]. This study focuses on the training program because nearly all published datasets of real faults concern faults in the DL program. A systematic survey [34] identified 16 studies with such datasets, and all except CleanML concern program faults. We selected the 11 DeepCrime operators that mutate the training program, preserve the neuron correspondence of (2), and apply to our subjects. We excluded eight of DeepCrime’s 19 training-program operators for three reasons. Changing the weight initialization was excluded because training then starts from an independent draw of weights, so the correspondence in (2) no longer holds. Removing the validation set was excluded because it left the trained parameters unchanged on 13 of the 17 subjects where it applied, so it offers few nontrivial training pairs. The other six were excluded because they applied to fewer than ten or to none of our subjects:

8

adding an activation function, changing or removing weight regularization, changing the early-stopping patience, adding a bias, and changing gradient clipping. Table I lists the selected operators with the fault type each one emulates and the configuration space used to generate mutants. The configuration spaces follow DeepCrime’s defaults. The selected operators target fault types frequently observed in real DL programs. Among the 67 program faults reproduced in prior work [29], 54 (80.60%) contain at least one fault type emulated by a selected operator. Cao et al. [14] reported that activation function, number of epochs, learning rate, loss function, and optimizer faults collectively account for 73.55% of the fault types recorded in their benchmark, counting each fault type once per program. D. Corpus Preparation a) Mutation corpus: For each subject, we train five instances of the original model and generate the applicable DeepCrime mutants from the configurations in Table I. Original and mutant programs use matched per-instance seeds and deterministic GPU operations. Each mutant is paired with its seed-matched original, and pairs with non-finite parameters are discarded. Before constructing the parameter-delta targets, we check the saved models for pair completeness and positional tensor-shape compatibility, accounting for expected structural changes such as bias removal. Identity mutants, which request the model’s existing configuration, produce exactly zero deltas under this setup. To prevent architectural overlap across partitions, subjects with identical architectures are grouped before splitting. The split was fixed before TensorMutation training and yields 55 training, seven validation, and nine held-out test subjects, with 19 161, 2454, and 3637 retained original–mutant pairs, respectively. Test subjects, including their test inputs, are excluded from TensorMutation training and model selection, and validation subjects are used only for checkpoint selection. Only the training partition is used to construct the vocabularies required by TensorMutation. b) Real-fault benchmark: Real faults are used only for evaluation. The benchmark contains 13 subjects with a real fault and its fixed version: the six test subjects from CleanML, DeepFD, and DeepLocalize, and seven MLP classifiers from defect4ML. A subject is included only if at least one input kills the real fault in three or more of its five seed-matched instances (Section IV-G). This yields 12 subjects, which we call behaviorally active. E. Model Configuration The GNN model uses a hidden dimension of 128 and two message-passing steps. Training uses AdamW with a learning rate of 10−3 and weight decay of 10−4 for up to 200 epochs. The behavioral loss uses up to 512 probe inputs per subject, with λr = 10−3 . We evaluate the validation set every five epochs and retain the checkpoint with the highest validation IoU on the probes (Section IV-G). We train two otherwise identical variants: TensorMutation-behavioral (TMB) uses only the behavioral loss (6), whereas TensorMutationjoint (TM-J) adds direct weight-space supervision (9) with

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

9

TABLE I S ELECTED D EEP C RIME M UTATION O PERATORS WITH FAULT T YPES AND C ONFIGURATION S PACES ID

Operator

Fault type

Configuration space

HBS Change batch size Suboptimal batch size {b/4, b/2, 2b, 4b} HLR Decrease learning rate Suboptimal learning rate (∼ 0, lr], binary search with precision lr/10 HNE Change number of epochs Suboptimal number of epochs [1, e], binary search with precision e/10 HDB Disable data batching Data batching required Batch size set to the training-set size; only if batching is used ACH Change activation function Wrong type of activation function Each layer with a non-linear activation × each other non-linear activation ARM Remove activation function Missing activation function Each layer with a non-linear activation RAW Add weights regularization Excess weights regularization† Each layer without regularization × {L1, L2, L1L2} RCD Change dropout rate Excessively high/low dropout rate† Each dropout layer × four rates from (0, 1] excluding the original value WRB Remove bias from a layer Bias needed in a layer Each layer with a bias LCH Change loss function Wrong selection of loss function Each loss other than the original that is compatible with the label encoding OCH Change optimization function Wrong optimization function Each optimizer other than the original b, lr, and e denote the subject’s original batch size, learning rate, and number of epochs. Activation, loss, and optimizer candidates are taken from DeepCrime’s default Keras lists [23]. Fault types are leaves of the taxonomy of Humbatova et al. [38]. † RAW and RCD were derived from Zhang et al. [36] and Islam et al. [37], respectively. As these studies define no taxonomy leaves, their fault type labels follow Xie et al. [63].

λW = λb = 0.1. TM-J is the primary model and the one evaluated against real faults in RQ4. F. Comparison Methods TensorMutation is compared with three references, each serving a distinct role. DeepCrime [23] is the pre-training reference: its mutants provide the target transformations for RQ1 to RQ3 and a comparator against real faults in RQ4. We use DeepMutation++ [4] as the manually defined posttraining comparator because it has a public implementation and is peer reviewed, whereas MuNN [24] has no public implementation and MuFF [25] was not peer reviewed at the time of this study. For each subject, we apply the applicable DeepMutation++ feedforward operators using the default configuration and a relative accuracy threshold of 0.9. We generate five mutant variants per operator and original instance and keep only complete groups, so some DeepMutation++ comparisons use fewer than five instances. DeepMutation++ therefore differs from TensorMutation in operator coverage and configuration granularity, so RQ4 compares real-fault similarity across approaches. Norm-matched random perturbations serve as the direction control. For each TensorMutation prediction, we generate three perturbations that separately match the predicted node- and edge-delta norms while randomizing their directions. All comparisons use the same seed-matched original model instances. As an integrity check, we reconstruct oracle mutants by applying the true DeepCrime deltas through the TensorMutation reconstruction pipeline. The pre-specified integrity criterion requires their kill indicators to match those of the corresponding DeepCrime mutants exactly. The criterion held for all 3637 test pairs and for 4449 of the 4450 real-fault pairs. The single exception arises from a numerical near-tie between the two highest class scores on one input. G. Evaluation Measures a) Behavioral measures: All behavioral measures use the full test split T of each subject and follow the definitions of [29]. An input is a killing input (KI) for a model instance if its seed-matched original classifies it correctly and the instance misclassifies it [23]. The killing probability KP (t) of input

t is the fraction of the available seed-matched instances for which P t is a killing input. The behavioral mass of a candidate is t∈T KP (t), and its mass ratio is this mass divided by that of the reference, computed per configuration and summarized hierarchically like all other measures. For DeepMutation++, which yields five variants per original instance, we form five evaluation sets by grouping the j-th variant from each seedmatched original. We compute the measures for each set and average them. The three norm-matched random perturbations are handled analogously. We measure the behavioral similarity between a candidate C and a reference R as P min(KPC (t), KPR (t)) IoU = P t∈T , (11) t∈T max(KPC (t), KPR (t)) where the reference is the corresponding DeepCrime configuration in RQ1 and the real fault in RQ4. IoU is bounded by the ratio of the smaller to the larger behavioral mass, so it penalizes both report coupling P missed and excess kills. We P strength (CS), t∈T min(KPC (t), KPR (t))/ t∈T KPC (t), with CS defined as zero when the candidate mass is zero. We use CS descriptively but not as the primary cross-method measure because its interpretation depends on the candidate’s behavioral mass. We therefore use coupling only for this directional measure and similarity for IoU. b) Weight-space measures: For RQ2, we compare the c with the DeepCrime delta ∆θ over all predicted delta ∆θ aligned weights and biases, using cosine similarity for direcc tion, the log-norm ratio log10 (∥∆θ∥/∥∆θ∥) for magnitude, c − ∆θ∥/∥∆θ∥ for overall and the normalized L2 distance ∥∆θ error. c) Weight–behavior relationship: For RQ3, we use Spearman rank correlation to relate weight-space cosine similarity (RQ2) to behavioral IoU (RQ1). We examine this relationship within subjects at the configuration level and across subjects using subject-level summaries. Unless stated otherwise, measures are summarized hierarchically: configuration values are aggregated by the median within each operator, operator medians by the median within each subject, and subject medians by the median across subjects. Configurations whose reference has zero behavioral

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

10

mass are excluded from the reported behavioral summaries, and configurations with a zero reference delta are excluded from weight-space summaries.

TABLE II B EHAVIORAL SIMILARITY TO THE CORRESPONDING D EEP C RIME MUTANTS ON THE TEST SUBJECTS

IoU

CS

H. Statistical Analysis

Subject

TM-B Rand. B TM-J Rand. J TM-B TM-J

The subject is the unit of statistical inference because configurations and operators are repeated observations nested within a subject rather than independent samples. For paired subject-level comparisons, we use two-sided exact Wilcoxon signed-rank tests on the hierarchical subject summaries, omitting zero differences, and report the rank-biserial correlation [64] as the effect size. For RQ1, we compare TM-B and TM-J with their corresponding norm-matched random controls using subject-level IoU summaries across the nine test subjects. For RQ2, TMB and TM-J are paired across the same nine subjects using subject-level cosine similarity and normalized L2 error. For RQ3, we report the relationship descriptively: the median of the per-subject Spearman correlations at the configuration level for the within-subject analysis, and the Spearman correlation over the nine subject summaries for the across-subject analysis. For RQ4, TM-J is paired separately with DeepCrime, DeepMutation++, and the norm-matched random control using subject-level IoU over the 12 real-fault subjects. These three comparisons form the family of primary RQ4 tests, and we control its family-wise error rate with the Holm correction [65] at α = 0.05.

cleanml018 cleanml019 deepfd030 deepfd050 deeplocalize030 deeplocalize039 tfex027 tfex028 tfex029

0.246 0.297 0.162 0.178 0.091 0.216 0.247 0.438 0.081

0.153 0.243 0.190 0.164 0.109 0.186 0.258 0.406 0.192

0.232 0.299 0.179 0.248 0.099 0.235 0.124 0.380 0.060

0.118 0.214 0.202 0.087 0.085 0.094 0.295 0.411 0.126

0.483 0.482 0.163 0.228 0.355 0.284 0.274 0.495 0.084

Median

0.216

0.190

0.232

0.126

0.284 0.340

We examine whether source-training context improves cross-subject transfer. The motivation is that the effect of a mutation may depend on the source setting; for example, halving a learning rate of 0.1 differs from halving one of 0.001. TM-J conditions on the trained neural graph and requested mutation only, and we construct two context-augmented variants. TM-J-Narrow additionally receives only the source value associated with the requested training-setting mutation. TM-JFull receives the available source learning rate, number of epochs, batch size, optimizer, and loss function for every mutation. Both variants inject this context through FiLM using the same context width. All other model, training, split, and evaluation settings are kept consistent with TM-J. We treat this comparison as exploratory rather than an additional confirmatory RQ. V. R ESULTS This section reports the results for the four RQs and the sourcecontext design ablation. All values are hierarchical summaries as defined in Section IV-G, and reported values are medians unless stated otherwise. A. RQ1: Behavioral Learnability Table II reports the behavioral similarity between TensorMutation and the corresponding DeepCrime mutants on the nine test subjects. TM-B achieves a median IoU of 0.216 and TM-J a median of 0.232. Subject-level IoU ranges from

5/4 0.570 0.244

5/4 0.652 0.200

Rand.: norm-matched random control. CS: coupling strength (Section IV-G), reported descriptively; paired tests compare IoU only.

1.0 TM-B

Operator-level median IoU

I. Design Ablations

Wins/losses vs. control Wilcoxon p Rank-biserial r

0.483 0.441 0.229 0.316 0.411 0.340 0.181 0.494 0.061

TM-J

0.8

0.6

0.4

0.2

0.0

HBS HLR HNE HDB ACH ARM RAW RCD WRB LCH OCH (9) (9) (9) (9) (8) (9) (9) (3) (9) (9) (9)

Fig. 5. Operator-level median IoU of TM-B and TM-J with the corresponding DeepCrime mutants across the test subjects. Numbers in parentheses give the number of test subjects to which each operator applies.

0.081 to 0.438 for TM-B and from 0.060 to 0.380 for TMJ. Both variants therefore reproduce part of the behavioral effect of pre-training mutations on unseen subjects, although fidelity varies substantially across subjects. TM-B and TM-J mutants have greater behavioral mass than the corresponding DeepCrime mutants on six and seven of the nine test subjects, respectively. The median candidate-to-reference behavioralmass ratio is 1.850 for TM-B and 1.366 for TM-J. Fig. 5 shows the results per mutation operator. Operatorlevel median IoU ranges from 0.058 (RCD) to 0.309 (HNE) for TM-B and from 0.136 (RAW) to 0.274 (HNE) for TM-J. Changing the number of epochs achieves the highest median IoU for both variants. TM-J achieves a higher median than TM-B on six operators and a lower one on five, so direct weight-space supervision does not change behavioral fidelity uniformly. The norm-matched random controls achieve median

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

IoUs of 0.190 for TM-B and 0.126 for TM-J. Both medians favor TensorMutation, most clearly for TM-J. However, each TensorMutation variant exceeds its corresponding control on only five of the nine subjects, and neither paired difference is statistically significant (TM-B: p = 0.570; TM-J: p = 0.652). Behavioral similarity alone therefore does not establish that the learned perturbation direction is responsible for the observed fidelity. RQ2 examines the direction directly in weight space, and RQ4 examines its contribution against real faults. Answer to RQ1: TensorMutation reproduces part of the behavioral effect of pre-training mutations on unseen subjects, with median IoUs of 0.216 for TM-B and 0.232 for TM-J. Fidelity varies across subjects and operators, and both variants usually produce greater behavioral mass than their DeepCrime references. The norm-matched random comparison does not show a consistent subject-level advantage for the learned direction.

11

TABLE III W EIGHT- SPACE FIDELITY TO THE CORRESPONDING D EEP C RIME DELTAS ON THE TEST SUBJECTS

Cosine Subject

Norm. L2

TM-J

TM-B

TM-J

TM-B TM-J

cleanml018 0.146 0.422 cleanml019 0.282 0.532 deepfd030 0.020 0.115 deepfd050 0.080 0.227 deeplocalize030 0.030 0.156 deeplocalize039 0.037 0.179 tfex027 0.165 0.508 tfex028 −0.004 0.125 tfex029 0.360 0.481

0.046 −0.114 0.130 −0.175 0.156 −0.111 0.383 0.536 −0.274

−0.151 −0.226 −0.171 −0.479 −0.154 −0.371 0.407 0.218 −0.189

1.377 1.009 1.717 1.074 1.777 1.124 2.430 3.543 0.986

Median

0.046

−0.171 1.377 1.030

0.080

TM-J better Wilcoxon p Rank-biserial r

0.227

9/9 0.004 1.000

1.030 0.925 1.125 0.983 1.117 0.988 2.129 1.895 0.878

9/9 0.004 1.000

Log-norm ratio: log10 of the predicted-to-reference delta norm; negative values indicate smaller predicted deltas. No test is reported for it, since calibration is best at zero rather than at either extreme.

B. RQ2: Weight-Space Fidelity

1.0 TM-B

Operator-level median cosine

Table III compares the predicted parameter deltas with those of the corresponding DeepCrime mutants. With the behavioral objective alone, the predicted deltas are only weakly aligned with the reference deltas, with a median cosine similarity of 0.080. Adding direct weight-space supervision raises the median to 0.227. TM-J improves cosine similarity on all nine subjects (p = 0.004, rank-biserial r = 1.000). Fig. 6 shows that the improvement also holds across mutation operators. TM-B’s operator-level median cosine similarity is at most 0.039 for six operators and negative for three of them. TM-J’s median is at least as high as TM-B’s for all 11 operators, and TM-J is higher on 84 of the 92 subject-operator combinations. Normalized L2 error also decreases on all nine subjects, from a median of 1.377 to 1.030 (p = 0.004, rank-biserial r = 1.000). Since this measure divides the prediction error by the norm of the reference delta, predicting no change at all yields exactly one. TM-B’s error exceeds this level on eight of the nine subjects, whereas TM-J’s median error is 1.030 and its error falls below one on four subjects. The larger gain is therefore in direction rather than in exact reconstruction of the full delta. Magnitude calibration shows a different pattern. The median log-norm ratio is 0.046 for TM-B and −0.171 for TM-J. TMB’s ratio is positive on five subjects and negative on four, whereas TM-J’s is negative on seven. A ratio of −0.171 corresponds to predicted deltas about two thirds the norm of the reference. TM-J therefore tends to predict smaller parameter changes than DeepCrime, while TM-B shows no consistent direction of error. Weight-space supervision does not reduce median behavioral fidelity. TM-J has a slightly higher median IoU than TM-B (0.232 versus 0.216, Table II), but it is higher on only five of the nine subjects and lower on four. The same mixed picture holds across operators, where TM-J has the higher median on six operators and the lower one on five (Fig. 5). The consistent gain in direction therefore does not carry over uniformly into behavior. RQ3 examines how directional and behavioral fidelity are related.

TM-B

Log-norm ratio

TM-J

0.8 0.6 0.4 0.2 0.0 −0.2 −0.4

HBS HLR HNE HDB ACH ARM RAW RCD WRB LCH OCH (9) (9) (9) (9) (8) (9) (9) (3) (9) (9) (9)

Fig. 6. Operator-level median cosine similarity of TM-B and TM-J with the corresponding DeepCrime deltas across the test subjects. Numbers in parentheses give the number of test subjects to which each operator applies.

Answer to RQ2: Direct weight-space supervision improves the direction and normalized error of the predicted deltas on every test subject, without reducing the median behavioral fidelity. Median cosine similarity increases from 0.080 to 0.227, while normalized L2 error decreases from 1.377 to 1.030, close to the error of predicting no change. Magnitude remains imperfectly calibrated: TM-B’s signed errors vary across subjects, while TM-J tends to predict smaller deltas than the reference. C. RQ3: Weight–Behavior Relationship We next examine whether greater directional fidelity in weight space is associated with greater behavioral fidelity. The analysis aligns the 697 configurations listed in Table IV for each TensorMutation variant and relates cosine similarity to

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

TABLE IV S PEARMAN CORRELATION BETWEEN CONFIGURATION - LEVEL COSINE SIMILARITY AND I O U ON THE TEST SUBJECTS

12

TABLE V R EAL - FAULT SIMILARITY ON THE 12 SUBJECTS IoU with real fault

Spearman ρ Subject

Configurations

TM-B

TM-J

53 54 119 87 120 55 67 60 82

0.242 0.339 0.404 0.360 0.412 0.515 −0.122 −0.228 0.038

0.241 0.202 0.151 0.573 0.119 0.552 0.288 −0.136 0.251

Median Positive

0.339 7/9

0.241 8/9

Across subjects

−0.100

−0.083

cleanml018 cleanml019 deepfd030 deepfd050 deeplocalize030 deeplocalize039 tfex027 tfex028 tfex029

The across-subject row correlates the subject-level IoU and cosine similarity of Tables II and III.

behavioral IoU using Spearman correlation. Within subjects, the relationship is positive for most subjects. The median of the nine subject-level correlations is ρ = 0.339 for TMB and ρ = 0.241 for TM-J, and the correlation is positive on seven of the nine subjects for TM-B and on eight for TM-J. TM-J has the lower median but the more consistent sign; its only negative correlation occurs on tfex028, which is also negative for TM-B. Thus, within a given subject, configurations whose predicted parameter changes are more closely aligned with the corresponding DeepCrime changes tend to reproduce their behavioral effects more closely. The associations are weak to moderate, indicating that weightspace direction is only partly associated with variation in behavioral fidelity. The relationship does not persist across subjects. Using the hierarchical subject summaries from RQ1 and RQ2, the correlation between subject-level cosine similarity and IoU is ρ = −0.100 for TM-B and ρ = −0.083 for TMJ. Subjects with higher overall directional fidelity therefore do not systematically exhibit higher behavioral fidelity. These results also clarify the effect observed in RQ2. Although direct weight-space supervision substantially increases cosine similarity for TM-J, it does not produce a stronger configuration-level association between cosine similarity and behavioral IoU. Improved directional reconstruction and behavioral resemblance are therefore related, but they are not interchangeable measures. Answer to RQ3: Better directional fidelity in weight space tends to correspond to better behavioral fidelity within a subject, with median Spearman correlations of 0.339 for TM-B and 0.241 for TM-J. This relationship does not persist across subjects, where the correlations are −0.100 and −0.083, respectively. Weight-space fidelity is therefore associated with behavioral fidelity within subjects but does not explain differences in behavioral fidelity across subjects.

DC

Subject

TM-J

DC

Rand. DM++

KI

ceiling

defect4ml038 defect4ml061 defect4ml067 defect4ml068 defect4ml069† defect4ml100 cleanml018 cleanml019 deepfd030† deepfd050 deeplocalize030 deeplocalize039

0.248 0.172 0.257 0.000 0.000 0.202 0.173 0.138 0.143 0.195 0.189 0.225

0.008 0.105 0.175 0.000 0.000 0.132 0.128 0.114 0.333 0.184 0.308 0.221

0.117 0.116 0.135 0.000 0.000 0.122 0.094 0.090 0.347 0.115 0.154 0.126

0.010 0.131 0.057 0.000 0.000 0.065 0.059 0.034 0.219 0.086 0.041 0.107

486 120 7041 500 12 2786 241 4702 7 2167 85 3028

0.008 0.131 0.250 0.000 0.000 0.214 0.331 0.225 0.707 0.513 0.515 0.478

Median

0.181 0.130 0.117

0.058

DC: DeepCrime; Rand.: norm-matched random control; DM++: DeepMutation++. KI: total kills of the real fault summed over its five instances. DC ceiling: upper bound on DeepCrime’s IoU set by the ratio of the smaller to the larger behavioral mass. † Fewer than 50 real-fault kills.

D. RQ4: Real-Fault Similarity We now evaluate TM-J as the primary variant of TensorMutation, whose directional fidelity exceeds TM-B’s on every test subject with no loss in median behavioral fidelity (Section V-B). Table V reports the subject-level IoU between each mutation approach and the real fault on the 12 subjects, and Fig. 7 shows the distributions beneath these summaries. TM-J achieves the highest descriptive median IoU (0.181), followed by DeepCrime (0.130), the norm-matched random control (0.117), and DeepMutation++ (0.058). Two subjects, defect4ml068 and defect4ml069, yield zero IoU for every approach. Both have degenerate test sets: the 200 test inputs of defect4ml068 contain only two distinct inputs, and defect4ml069 has only seven test inputs. Table VI reports the paired subject-level comparisons. TM-J exceeds DeepMutation++ on nine of the ten subjects with a nonzero difference, with a median paired difference of 0.112. This is the only comparison that remains significant after Holm correction (pHolm = 0.018, rank-biserial r = 0.927). Against the norm-matched random control, TM-J also wins on nine of the ten non-tied subjects (median difference 0.067), but the difference is not significant after correction (pHolm = 0.168). The single loss against both comparators occurs on deepfd030, whose real fault causes only seven kills in total. Against DeepCrime, TM-J wins on eight of the ten non-tied subjects, but the median paired difference is only 0.018 and the comparison is not significant (pHolm = 0.322). The positive descriptive margin over DeepCrime is concentrated in the defect4ML subjects. Among the six subjects from CleanML, DeepFD, and DeepLocalize, TM-J is higher on four, by 0.003 to 0.045, whereas DeepCrime is higher on deepfd030 and deeplocalize030, the two with the fewest real-fault kills. To examine why the approaches differ, we analyze the behavioral severity of their mutants on the ten subjects with at least 50 real-fault kills. This floor keeps the analysis from resting on very sparse real-fault behavior: it excludes deepfd030 and defect4ml069, which have only seven and 12 kills, and any

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

13

1.0

IoU with real-fault behavior

TensorMutation (TM-J)

DeepCrime

Norm-Matched Random

DeepMutation++

0.8

0.6

0.4

0.2

39

de

ep

lo

ca

liz

e0 liz ca

de

ep

lo

e0

30

0 ep de

de

ep

fd 0

fd

30

05

†

19 cl

ea

nm

l0

18 nm ea cl

ct fe de

de

l0

l1 4m

l0 t4 m fe c

ct fe de

00

†

69

68 4m

l0

67 fe

de

ct

4m

l0

61 l0 4m fe

de

ct

fe de

ct

4m

l0

38

0.0

Fig. 7. IoU with the real fault on the 12 subjects. Boxes show the IoU of individual mutants, and black diamonds show the subject summaries of Table V, on which the tests are based. † Subjects with fewer than 50 real-fault kills.

TABLE VI PAIRED SUBJECT- LEVEL COMPARISONS OF TM-J WITH EACH APPROACH Comparator

W/L/T

Median ∆

p

pHolm

r

DM++ Random DeepCrime

9/1/2 9/1/2 8/2/2

0.112 0.067 0.018

0.006 0.084 0.322

0.018 0.168 0.322

0.927 0.636 0.382

W/L/T: wins, losses, and ties of TM-J across subjects. ∆: IoU of TM-J minus IoU of the comparator, with the median taken over all 12 subjects.

TABLE VII S OURCE - CONTEXT ABLATION Test subjects

Real faults

Variant

IoU

Cosine

IoU

TM-J TM-J-Full TM-J-Narrow

0.232 0.223 0.170

0.227 0.214 0.195

0.181 0.140 0.155

E. Does Source-Training Context Help? floor between 13 and 85 would exclude the same two subjects. Since IoU is bounded by the ratio of the two behavioral masses (Section IV-G), Table V also reports the resulting ceiling for DeepCrime. Computed per configuration as IoU divided by the ceiling, DeepCrime reaches a median of 0.753 of its ceiling on the five defect4ML subjects; on defect4ml038, for example, its IoU and ceiling are both 0.008. On the five subjects from CleanML, DeepFD, and DeepLocalize, it reaches only 0.479; on deepfd050, for example, it attains 0.184 against a ceiling of 0.513. There, DeepCrime’s similarity is limited more by which inputs kill its mutants than by their mass. Relative to the real fault, TM-J carries more behavioral mass than DeepCrime on seven of the ten subjects and less on two, with one tie; the medians are 0.348 and 0.237, and the difference is not significant (p = 0.250, rank-biserial r = 0.467). Both medians are below one, so the mutants of both approaches are typically milder than the real fault. Answer to RQ4: TensorMutation (TM-J) achieves the highest median IoU with real faults (0.181), compared with 0.130 for DeepCrime, 0.117 for the norm-matched random control, and 0.058 for DeepMutation++. Its advantage over DeepMutation++ is statistically significant after Holm correction. Its paired differences from DeepCrime and the random control are positive but not statistically significant.

Table VII compares TM-J with the two source-context variants defined in Section IV-I. Neither variant improves on TM-J in any of the three measures. On the test subjects, TM-J has the highest behavioral IoU and cosine similarity, followed by TM-J-Full and then TM-J-Narrow. Against real faults, the order of the two variants reverses, but both remain below TMJ. The two variants therefore rank differently on fidelity to DeepCrime and on similarity to real faults. VI. D ISCUSSION Our key findings support one central conclusion: the behavioral effects of pre-training mutations can be learned as posttraining weight-space transformations, and these transformations transfer to models never seen in training. The resulting mutants show a higher median behavioral similarity to real faults than the pre-training reference, with no significant paired difference, and a significantly higher similarity than the posttraining comparator. Two limits come with this conclusion. The learned direction adds less to the behavioral similarity than the medians suggest, once the magnitude that the model itself predicts is accounted for. That magnitude is also not calibrated against the reference. Below, we explain the reasons behind these results, discuss what they mean for realistic post-training mutation, and draw implications for testing and debugging of DL systems.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

A. Reasons for the Observed Results We were initially surprised that TM-J reaches a higher median real-fault IoU than DeepCrime, the pre-training reference from which it learns. The difference is not statistically significant, so we do not interpret this as TensorMutation surpassing its reference. Still, the descriptive pattern requires interpretation, and the behavioral-severity analysis provides a partial explanation. For two kill profiles, IoU is upperbounded by the ratio of their smaller to larger behavioral mass. For the five defect4ML subjects in the severity analysis, DeepCrime often operates close to this ceiling, indicating that limited behavioral mass constrains its attainable similarity. By contrast, DeepCrime remains substantially below the ceiling on the subjects from CleanML, DeepFD, and DeepLocalize, so behavioral severity alone cannot explain the remaining gap. DeepCrime’s search strategy helps explain why its mutants are not arbitrarily severe. DeepCrime searches for configurations that are killable but not trivial to kill, and for rangebased parameters it uses binary search to identify such configurations [23]. List-based parameters are instead searched exhaustively. This design therefore avoids trivially strong mutants rather than maximizing behavioral severity. Real faults are not calibrated to any such threshold. On four of the five defect4ML subjects, the mass-imposed ceilings of 0.008 to 0.250 mean that the real faults carry roughly 4 to 120 times the behavioral mass of DeepCrime’s mutants. Why the benchmark faults are this severe, our data cannot say. Our hypothesis is selection: faults with visible behavioral effects are more likely to be noticed and reported. TensorMutation, in turn, tends to carry more behavioral mass than its DeepCrime reference. On the nine held-out subjects, TM-J has greater behavioral mass than the corresponding DeepCrime mutants on seven subjects, with a median mass ratio of 1.366. This is a behavioral calibration error against the reference. Whether the additional mass helps against real faults depends on where it falls. Mass on inputs that also kill the real fault raises the overlap, whereas mass on other inputs enlarges the union and lowers IoU, and even aligned mass stops helping once TM-J becomes more severe than the real fault. Where TM-J remains less severe than the real fault, the additional mass raises the maximum IoU it can attain. Consistent with this, TM-J carries more behavioral mass relative to the real fault than DeepCrime on seven of the ten subjects in the severity analysis, although the paired difference is not statistically significant. Which inputs the mass falls on is a matter of direction. RQ2 establishes that TensorMutation learns mutation-induced parameter direction, RQ3 shows that better directional fidelity tends to accompany better behavioral fidelity within subjects, and RQ4 provides supporting evidence against the norm-matched random control. This account also implies that excess behavioral mass can become a disadvantage when the real fault itself is mild. That pattern appears in the data. Among the ten non-tied TMJ–DeepCrime comparisons, DeepCrime exceeds TM-J only on deepfd030 and deeplocalize030, the two subjects with the smallest real-fault kill counts. We stress that this excess behavioral mass was found, not designed. A direct follow-up

14

would rescale DeepCrime’s parameter deltas while preserving their direction and then re-evaluate behavioral mass and IoU, helping to separate the contributions of perturbation magnitude and direction. The norm-matched random control isolates the contribution of directional information more directly. It preserves TensorMutation’s predicted perturbation magnitude while randomizing its direction, so its IoU captures the behavioral overlap obtainable at the same perturbation budget without the learned direction. A systematic margin over this control is therefore evidence that the learned direction contributes information beyond perturbation magnitude. On this basis we consider the direction claim established in parameter space. In real-fault behavior it is supported but not statistically established. B. Learned Mutation and Mutant Realism Our earlier work on the competent programmer hypothesis found that even chains of mutations built from manually defined operators fail to reproduce most real faults, and it proposed learning mutations from real faults instead [31]. That proposal was made for conventional programs. TensorMutation explores this direction for DL by learning the effects of existing mutation operators derived from real-fault studies. The learned transformation does not add a new operator. It reproduces the effect of an existing one without the cost of retraining. Our results show that these effects exhibit learnable structure that transfers to previously unseen models. What the results support is narrower than what one might hope for. Realism here means overlap of kill profiles with real faults. This criterion rewards a match in severity. If real-fault benchmarks favor faults with visible effects, as we hypothesized above, then mild mutants are penalized under it. Mutants also serve a second role, as test goals, where resistant and hard-to-kill mutants are the valuable ones. DeepCrime’s search targets exactly such mutants. Resistance and realism have been found to be largely independent qualities of a mutant [43]. We make no claim about test-suite adequacy. The two roles may therefore need differently calibrated generators. The preceding paragraphs asked what the learned transformation produces. The source-context ablation asks what it needs as input for better generalization. Supplying the source model’s training metadata did not improve either tested variant over TM-J. One possible explanation is that the trained parameters already encode much of the outcome of training, so the metadata is largely redundant with the neural graph. Another is that the FiLM pathway we tested does not transfer the supplied context to unseen subjects. Our experiment cannot separate these explanations. The defensible conclusion is narrower. For the tested variants, conditioning on the trained graph and the requested mutation transferred better than also supplying the examined metadata. Two of the three real-fault comparisons are not statistically significant, so what we conclude from RQ4 depends on how much weight they are given. The medians consistently favor TensorMutation, but a non-significant paired difference establishes neither that TensorMutation exceeds DeepCrime nor that the two are equivalent. We read the descriptive

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

pattern as suggestive. A more cautious reader would treat the comparison with DeepCrime as inconclusive. Our evidence is consistent with that reading as well. Neither reading changes the key result: TensorMutation is significantly more similar to real faults than the existing post-training mutation tool DeepMutation++.

C. Implications For research on mutant realism, the main implication is methodological. Studies of weight-space mutations that compare kill-profile overlap should include a norm-matched random control. It keeps the predicted magnitude and randomizes only the direction, so it indicates how much of the overlap the learned direction adds. In our case the answer is less than the medians suggest, which is exactly why the control belongs in the evaluation. For the evaluation of weight-space methods, RQ3 shows that directional fidelity is associated with behavioral fidelity within subjects, but this relationship does not extend across subjects. Behavioral evaluation therefore remains necessary, particularly when comparing results across subjects. For testing and debugging research, TensorMutation generates mutants conditioned on fault-informed mutation specifications. Each mutant retains its requested operator and configuration, allowing evaluations to distinguish, for example, learning-rate mutations from activation-function mutations. Post-training perturbation tools also have identifiable operators, but these describe how the parameters are perturbed. TensorMutation’s operators correspond to developer-fault categories documented in empirical studies of real DL faults. This provides a basis for studying fault-type-specific testing and debugging behavior. However, RQ1 establishes only partial reproduction of the corresponding DeepCrime effects; fidelity to individual real-fault types requires further evaluation. For practice, TensorMutation replaces per-mutant retraining with GNN inference and mutant reconstruction. The trained GNN can be reused across supported models. Although the statistical killing criterion still requires multiple instances per configuration, generating these mutants no longer requires retraining each one. For the design of learned mutation, our results highlight the importance of calibration. Although direct weight-space supervision improves directional fidelity, smaller parameter changes do not necessarily produce smaller behavioral effects. Future approaches should therefore distinguish parameter magnitude from behavioral severity and consider how both affect similarity to the intended reference. Weight-space learning also suggests applications beyond mutation. Our corpus pairs original models with synthetic pre-training mutants, providing data that could be used to supervise fault-to-fix transformations. Such transformations could support learned repair, while mutationinduced parameter changes may provide signatures for investigating fault localization. Structural differences would require additional handling, and whether transformations learned from synthetic faults transfer to real faults remains untested. We consider this a promising direction for future work.

15

VII. T HREATS TO VALIDITY We discuss validity threats following the classification of Wohlin et al. [66]. a) Construct validity: We measure mutant realism through behavioral similarity to real faults using kill-profile overlap. This captures the dimension of realism that mutation testing generally evaluates. Other downstream tasks may also require that a mutant and a real fault share the same root cause, which our measure does not capture. Such correspondence is plausible, because the operators that supervise learning represent developer mistakes. However, RQ1 and RQ2 show only partial fidelity to the corresponding DeepCrime mutants in behavior and in weight space. b) Internal validity: We identify three threats concerning the learning targets, the learning objective, and the data split. First, the parameter deltas require position-by-position neuron correspondence between each original–mutant pair in the training set. We generate each pair with matched per-instance seeds and deterministic GPU operations, so that a pair differs only by the mutation. We verify each pair before building the targets, and identity mutants yield exactly zero deltas under this setup. Second, the results depend on the learning objective. We compare the behavioral objective alone with the joint objective under an identical recipe, but the coefficients were not tuned and each final variant was trained once. Third, the same model can appear in more than one fault dataset. We group subjects with identical architectures before splitting, keeping exact-model duplicates within the same partition. c) Conclusion validity: Our statistical analysis treats subjects rather than individual mutants as independent observations. We use hierarchical aggregation and apply Holm correction to the three primary RQ4 comparisons. However, the nine held-out subjects and 12 real-fault subjects limit statistical power, so non-significant differences cannot be interpreted as equivalence. d) External validity: Our evaluation covers MLP classifiers and 11 selected pre-training operators, each representing a fault type. Generalization to different network families and other fault types remains untested. VIII. C ONCLUSION This paper introduced TensorMutation, a learned post-training mutation approach that predicts the weight-space changes induced by pre-training mutations without retraining individual mutants. Our evaluation shows that these parameter changes exhibit learnable structure that transfers to previously unseen models. Direct weight-space supervision significantly improves directional fidelity, which is associated with behavioral fidelity within subjects but not across subjects. In the real-fault evaluation, TensorMutation achieves a higher median behavioral similarity than DeepCrime, although the difference is not statistically significant. It achieves significantly higher similarity than DeepMutation++, the evaluated posttraining comparator. Supplying the source model’s training configuration as additional input did not improve the evaluated behavioral similarity or weight-space fidelity.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

These findings demonstrate the potential of learning faultinformed weight-space transformations for post-training mutation. Two limitations remain. The norm-matched random control also achieves substantial real-fault similarity, so the additional behavioral contribution of the learned direction is not statistically established in the primary comparison. Parameter magnitude and behavioral severity are also distinct quantities, and their calibration remains an open problem. Future work will investigate controlled calibration, generalization to other network families, and the use of our paired corpus to learn fault-to-fix transformations for repair and fault localization. We also plan to develop TensorMutation into a reusable mutantgeneration tool for research on DL testing and debugging. DATA AND C ODE AVAILABILITY The visualization notebook and the result CSV files underlying the reported analyses are available at https://doi.org/10.5281/ zenodo.23318504. The full replication package will be made publicly available upon acceptance. ACKNOWLEDGMENT The authors used Claude and ChatGPT during the preparation of this manuscript to assist with language refinement, clarity, presentation of the content, and software implementation tasks. All scientific ideas, decisions, experimental designs, analyses, interpretations, and conclusions remain the work of the authors, who take full responsibility for the content. R EFERENCES [1] R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser, “Are mutants a valid substitute for real faults in software testing?” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ser. FSE 2014. New York, NY, USA: Association for Computing Machinery, Nov. 2014, pp. 654–665. [Online]. Available: https://dl.acm.org/doi/10.1145/2635868.2635929 [2] H. Wu, Z. Li, Z. Cui, and J. Zhang, “A Mutation-based Approach to Repair Deep Neural Network Models,” in 2021 8th International Conference on Dependable Systems and Their Applications (DSA). Yinchuan, China: IEEE, Aug. 2021, pp. 730–731. [Online]. Available: https://ieeexplore.ieee.org/document/9622974 [3] J. Sohn, S. Kang, and S. Yoo, “Arachne: Search-Based Repair of Deep Neural Networks,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 4, pp. 85:1–85:26, May 2023. [Online]. Available: https://dl.acm.org/doi/10.1145/3563210 [4] Q. Hu, L. Ma, X. Xie, B. Yu, Y. Liu, and J. Zhao, “DeepMutation++: A Mutation Testing Framework for Deep Learning Systems,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). San Diego, CA, USA: IEEE, Nov. 2019, pp. 1158–1161. [Online]. Available: https://ieeexplore.ieee.org/document/ 8952248/ [5] R. Lin, Q. Zhou, B. Wu, and X. Nan, “Robustness evaluation for deep neural networks via mutation decision boundaries analysis,” Information Sciences, vol. 601, pp. 147–161, Jul. 2022. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0020025522003541 [6] M. Méndez, M. Benito-Parejo, and M. G. Merayo, “Testing the Robustness of Machine Learning Models Through Mutations,” in Advances in Computational Collective Intelligence, N.-T. Nguyen, B. Franczyk, A. Ludwig, M. Núñez, J. Treur, G. Vossen, and A. Kozierkiewicz, Eds. Cham: Springer Nature Switzerland, 2024, pp. 308–320. [7] J. Wang, G. Dong, J. Sun, X. Wang, and P. Zhang, “Adversarial sample detection for deep neural network through model mutation testing,” in Proceedings of the 41st International Conference on Software Engineering, ser. ICSE ’19. Montreal, Quebec, Canada: IEEE Press, May 2019, pp. 1245–1256. [Online]. Available: https: //dl.acm.org/doi/10.1109/ICSE.2019.00126

16

[8] M. V. Pour, Z. Li, L. Ma, and H. Hemmati, “A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding,” in 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). Porto de Galinhas, Brazil: IEEE Computer Society, Apr. 2021, pp. 36–46. [Online]. Available: https://www.computer.org/ csdl/proceedings-article/icst/2021/683600a036/1tRP9PPnyj6 [9] Z. Wang, H. You, J. Chen, Y. Zhang, X. Dong, and W. Zhang, “Prioritizing Test Inputs for Deep Neural Networks via Mutation Analysis,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). Madrid, Spain: IEEE, May 2021, pp. 397–409. [Online]. Available: https://ieeexplore.ieee.org/document/ 9402064 [10] V. Riccio, N. Humbatova, G. Jahangirova, and P. Tonella, “DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). Melbourne, Australia: IEEE Computer Society, Nov. 2021, pp. 355–367. [Online]. Available: https://www.computer. org/csdl/proceedings-article/ase/2021/033700a355/1AjT9eVtxbG [11] H. Deokuliar, R. S. Sangwan, Y. Badr, and S. M. Srinivasan, “Improving Testing of Deep-Learning Systems,” Communications of the ACM, vol. 67, no. 3, pp. 44–48, Mar. 2024. [Online]. Available: https://dl.acm.org/doi/10.1145/3633311 [12] P. Li, X. Rao, J. Blase, Y. Zhang, X. Chu, and C. Zhang, “CleanML: A Study for Evaluating the Impact of Data Cleaning on ML Classification Tasks,” in 2021 IEEE 37th International Conference on Data Engineering (ICDE). Chania, Greece: IEEE, Apr. 2021, pp. 13–24. [Online]. Available: https://ieeexplore.ieee.org/document/9458702 [13] M. Abdelaal, C. Hammacher, and H. Schöning, “REIN: A Comprehensive Benchmark Framework for Data Cleaning Methods in ML Pipelines,” in Proceedings of the 26th International Conference on Extending Database Technology (EDBT). OpenProceedings.org, Mar. 2023, pp. 499–511. [Online]. Available: https://openproceedings.org/2023/conf/edbt/3-paper-49.pdf [14] J. Cao, M. Li, X. Chen, M. Wen, Y. Tian, B. Wu, and S.-C. Cheung, “DeepFD: automated fault diagnosis and localization for deep learning programs,” in Proceedings of the 44th International Conference on Software Engineering, ser. ICSE ’22. New York, NY, USA: Association for Computing Machinery, May 2022, pp. 573–585. [Online]. Available: https://dl.acm.org/doi/10.1145/3510003.3510099 [15] A. Ghanbari, D.-G. Thomas, M. A. Arshad, and H. Rajan, “Mutation-based Fault Localization of Deep Neural Networks,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). Luxembourg City, Luxembourg: IEEE Computer Society, Sep. 2023, pp. 1301–1313. [Online]. Available: https://www.computer.org/csdl/proceedings-article/ase/2023/ 299600b301/1SBGryufqzS [16] M. A. Raza and M. Wardat, “Graph neural network for fault localization in sequence-based models,” Empirical Software Engineering, vol. 30, no. 5, p. 119, Jun. 2025. [Online]. Available: https://doi.org/10.1007/ s10664-025-10666-6 [17] X. Xie, J. W. K. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen, “Testing and validating machine learning classifiers by metamorphic testing,” Journal of Systems and Software, vol. 84, no. 4, pp. 544–558, Apr. 2011. [Online]. Available: https://doi.org/10.1016/j.jss.2010.11.920 [18] Z. Wang, M. Yan, J. Chen, S. Liu, and D. Zhang, “Deep learning library testing via effective model generation,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Virtual Event USA: ACM, Nov. 2020, pp. 788–799. [Online]. Available: https://dl.acm.org/doi/10.1145/3368089.3409761 [19] A. Ghanbari, “Decomposition of Deep Neural Networks into Modules via Mutation Analysis,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2024. New York, NY, USA: Association for Computing Machinery, Sep. 2024, pp. 1669–1681. [Online]. Available: https://dl.acm.org/doi/10.1145/3650212.3680390 [20] G. Jahangirova, A. Stocco, and P. Tonella, “Quality Metrics and Oracles for Autonomous Vehicles Testing,” in 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). Porto de Galinhas, Brazil: IEEE, Apr. 2021, pp. 194–204. [Online]. Available: https://ieeexplore.ieee.org/document/9438556 [21] L. Ma, F. Zhang, J. Sun, M. Xue, B. Li, F. Juefei-Xu, C. Xie, L. Li, Y. Liu, J. Zhao, and Y. Wang, “DeepMutation: Mutation Testing of Deep Learning Systems,” in 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE). Memphis, TN, USA: IEEE Computer Society, Oct. 2018, pp. 100–111.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

[Online]. Available: https://www.computer.org/csdl/proceedings-article/ issre/2018/832100a100/17D45W1Oa1G [22] Z. Ahmed and P. Makedonski, “Exploring the Fundamentals of Mutations in Deep Neural Networks,” in Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems, ser. MODELS Companion ’24. New York, NY, USA: Association for Computing Machinery, Sep. 2024, pp. 227–233. [Online]. Available: https://dl.acm.org/doi/10.1145/3652620.3687426 [23] N. Humbatova, G. Jahangirova, and P. Tonella, “DeepCrime: mutation testing of deep learning systems based on real faults,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. Virtual Denmark: ACM, Jul. 2021, pp. 67–78. [Online]. Available: https://dl.acm.org/doi/10.1145/3460319.3464825 [24] W. Shen, J. Wan, and Z. Chen, “MuNN: Mutation Analysis of Neural Networks,” in 2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C). Lisbon, Portugal: IEEE, Jul. 2018, pp. 108–115. [Online]. Available: https://ieeexplore.ieee.org/document/8431960 [25] J. Kim, N. Humbatova, G. Jahangirova, S. Yoo, and P. Tonella, “MuFF: Stable and Sensitive Post-training Mutation Testing for Deep Learning,” Jan. 2025, arXiv:2501.09846 [cs]. [Online]. Available: http://arxiv.org/abs/2501.09846 [26] G. Jahangirova and P. Tonella, “An Empirical Evaluation of Mutation Operators for Deep Learning Systems,” in 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). Porto, Portugal: IEEE, Oct. 2020, pp. 74–84. [Online]. Available: https://ieeexplore.ieee.org/document/9159089 [27] N. Humbatova, G. Jahangirova, and P. Tonella, “DeepCrime: from Real Faults to Mutation Testing Tool for Deep Learning,” in 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). Melbourne, Australia: IEEE, May 2023, pp. 68–72. [Online]. Available: https: //ieeexplore.ieee.org/document/10172493/ [28] A. Abbasishahkoo, M. Dadkhah, L. Briand, and D. Lin, “TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep Neural Networks,” IEEE Transactions on Software Engineering, vol. 50, no. 12, pp. 3307–3329, Dec. 2024. [Online]. Available: https://ieeexplore.ieee.org/document/10720834 [29] Z. Ahmed, P. Makedonski, and J. Grabowski, “An Empirical Study of the Realism of Mutants in Deep Learning,” Dec. 2025, arXiv:2512.16741 [cs]. [Online]. Available: http://arxiv.org/abs/2512.16741 [30] M. B. Shah, M. M. Rahman, and F. Khomh, “Towards understanding the impact of data bugs on deep learning models in software engineering,” Empirical Software Engineering, vol. 30, no. 6, p. 168, Sep. 2025. [Online]. Available: https://doi.org/10.1007/s10664-025-10717-y [31] Z. Ahmed, E. Schwass, S. Herbold, F. Trautsch, and J. Grabowski, “A new perspective on the competent programmer hypothesis through the reproduction of real faults with repeated mutations,” Software Testing, Verification and Reliability, vol. 34, no. 3, p. e1874, May 2024. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/stvr.1874 [32] M. Kofinas, B. Knyazev, Y. Zhang, Y. Chen, G. J. Burghouts, E. Gavves, C. G. M. Snoek, and D. W. Zhang, “Graph Neural Networks for Learning Equivariant Representations of Neural Networks,” in Proceedings of the 12th International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=oO6FsMyDBt [33] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville, “FiLM: Visual Reasoning with a General Conditioning Layer,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, pp. 3942–3951, Apr. 2018. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/11671 [34] G. Jahangirova, N. Humbatova, J. Kim, S. Yoo, and P. Tonella, “Real faults in deep learning fault benchmarks: How real are they?” Oct. 2026, arXiv:2412.16336 [cs.SE]. [Online]. Available: http://arxiv.org/abs/2412.16336 [35] M. M. Morovati, A. Nikanjam, F. Khomh, and Z. M. J. Jiang, “Bugs in machine learning-based systems: a faultload benchmark,” Empirical Software Engineering, vol. 28, no. 3, p. 62, Apr. 2023. [Online]. Available: https://doi.org/10.1007/s10664-023-10291-1 [36] Y. Zhang, Y. Chen, S.-C. Cheung, Y. Xiong, and L. Zhang, “An empirical study on TensorFlow program bugs,” in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2018. New York, NY, USA: Association for Computing Machinery, Jul. 2018, pp. 129–140. [Online]. Available: https://dl.acm.org/doi/10.1145/3213846.3213866 [37] M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference

17

and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2019. New York, NY, USA: Association for Computing Machinery, Aug. 2019, pp. 510–520. [Online]. Available: https://dl.acm.org/doi/10.1145/3338906.3338955 [38] N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ser. ICSE ’20. New York, NY, USA: Association for Computing Machinery, May 2020, pp. 1110–1121. [Online]. Available: https://dl.acm.org/doi/10.1145/3377811.3380395 [39] W. Shen, Y. Li, Y. Han, L. Chen, D. Wu, Y. Zhou, and B. Xu, “Boundary sampling to boost mutation testing for deep learning models,” Information and Software Technology, vol. 130, p. 106413, Feb. 2021. [Online]. Available: https://www.sciencedirect.com/science/ article/pii/S0950584920301737 [40] L.-C. Feng, X.-Y. Wang, S.-Y. Zhang, R.-Z. Gao, and Z.-H. Zhao, “Mutation Operator Reduction for Cost-effective Deep Learning Software Testing via Decision Boundary Change Measurement,” Journal of Internet Technology, vol. 23, no. 3, pp. 601–610, May 2022. [Online]. Available: https://jit.ndhu.edu.tw/article/view/2705 [41] Y. Wang, Z. Zhang, Y. Yao, and Z. Huang, “A Fine-Grained Evaluation of Mutation Operators for Deep Learning Systems: A Selective Mutation Approach,” in Proceedings of the 14th Asia-Pacific Symposium on Internetware. Hangzhou China: ACM, Aug. 2023, pp. 123–133. [Online]. Available: https://dl.acm.org/doi/10.1145/3609437.3609453 [42] Z. Zhang, Y. Wang, Y. Yao, Z. Wang, and Z. Huang, “A fine-grained evaluation of mutation operators to boost mutation testing for deep learning systems,” Empirical Software Engineering, vol. 30, no. 3, p. 63, Feb. 2025. [Online]. Available: https: //doi.org/10.1007/s10664-025-10613-5 [43] Z. Ahmed, E. C. Dapaah, P. Makedonski, and J. Grabowski, “QualityDriven Selective Mutation for Deep Learning,” in International Conference on Evaluation and Assessment in Software Engineering (EASE ’26). Glasgow, United Kingdom: ACM, Jun. 2026, to be published. [Online]. Available: http://arxiv.org/abs/2604.22640 [44] Y. Li, W. Shen, T. Wu, L. Chen, D. Wu, Y. Zhou, and B. Xu, “How higher order mutant testing performs for deep learning models: A finegrained evaluation of test effectiveness and efficiency improved from second-order mutant-classification tuples,” Information and Software Technology, vol. 150, p. 106954, Oct. 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950584922000994 [45] L. Lyons and A. Ghanbari, “On Accelerating Deep Neural Network Mutation Analysis by Neuron and Mutant Clustering,” in 2025 IEEE Conference on Software Testing, Verification and Validation (ICST). Naples, Italy: IEEE, Mar. 2025, pp. 267–278. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10989014 [46] Y. Xue, Z. Zhang, C. Liu, S. Chen, and Z. Huang, “DeepWeak: Weak Mutation Testing for Deep Learning Systems,” in 2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS). Cambridge, United Kingdom: IEEE, Jul. 2024, pp. 49–60. [Online]. Available: https://ieeexplore.ieee.org/document/10684584 [47] X. Han, Z. Wang, B. Zhao, B. Zhang, J. Li, D. Borth, R. Yu, H. Maron, Y. Ye, L. Yin, and F. Neri, “A Survey of Weight Space Learning: Understanding, Representation, and Generation,” Mar. 2026, arXiv:2603.10090 [cs]. [Online]. Available: http://arxiv.org/abs/2603. 10090 [48] T. Unterthiner, D. Keysers, S. Gelly, O. Bousquet, and I. Tolstikhin, “Predicting Neural Network Accuracy from Weights,” Apr. 2021, arXiv:2002.11448 [stat]. [Online]. Available: http://arxiv.org/abs/2002. 11448 [49] G. Eilertsen, D. Jönsson, T. Ropinski, J. Unger, and A. Ynnerman, “Classifying the Classifier: Dissecting the Weight Space of Neural Networks,” in ECAI 2020. IOS Press, 2020, pp. 1119–1126. [Online]. Available: https://ebooks.iospress.nl/doi/10.3233/FAIA200209 [50] K. Schürholt, D. Kostadinov, and D. Borth, “Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction,” in Advances in Neural Information Processing Systems, vol. 34. Curran Associates, Inc., 2021, pp. 16 481–16 493. [Online]. Available: https://openreview.net/forum?id=F1D8buayXQT [51] R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur, “The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks,” in International Conference on Learning Representations, Apr. 2022. [Online]. Available: https://openreview.net/ forum?id=dNigytemkL [52] S. Ainsworth, J. Hayase, and S. Srinivasa, “Git Re-Basin: Merging Models modulo Permutation Symmetries,” in The Eleventh International

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021

Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=CQsmMYmlP5T [53] A. Navon, A. Shamsian, I. Achituve, E. Fetaya, G. Chechik, and H. Maron, “Equivariant Architectures for Learning in Deep Weight Spaces,” in Proceedings of the 40th International Conference on Machine Learning. PMLR, Jul. 2023, pp. 25 790–25 816. [Online]. Available: https://proceedings.mlr.press/v202/navon23a.html [54] A. Zhou, K. Yang, K. Burns, A. Cardace, Y. Jiang, S. Sokota, J. Z. Kolter, and C. Finn, “Permutation Equivariant Neural Functionals,” in Advances in Neural Information Processing Systems, vol. 36. Curran Associates, Inc., 2023, pp. 24 966–24 992. [Online]. Available: https://doi.org/10.52202/075280-1085 [55] D. Lim, H. Maron, M. T. Law, J. Lorraine, and J. Lucas, “Graph Metanetworks for Processing Diverse Neural Architectures,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=ijK5hyxs0n [56] A. Dravid, Y. Gandelsman, K.-C. Wang, R. Abdal, G. Wetzstein, A. A. Efros, and K. Aberman, “Interpreting the Weight Space of Customized Diffusion Models,” in Advances in Neural Information Processing Systems 37. Vancouver, BC, Canada: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024, pp. 137 334–137 371. [Online]. Available: http://www.proceedings.com/079017-4363.html [57] L. Meynent, I. Melev, K. Schürholt, G. Kauermann, and D. Borth, “Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction,” in ICLR 2025 Workshop on Neural Network Weights as a New Data Modality, Apr. 2025. [Online]. Available: https://openreview.net/forum?id=APsHrpqO3W [58] B. Zeng, Y. Yin, Z. Xu, and Z. Liu, “Generative Modeling of Weights: Generalization or Memorization?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2026, pp. 41 974–41 984. [59] B. Knyazev, A. Moudgil, G. Lajoie, E. Belilovsky, and S. LacosteJulien, “Accelerating Training with Neuron Interaction and Nowcasting Networks,” in The Thirteenth International Conference on Learning Representations, Apr. 2025. [Online]. Available: https://openreview.net/ forum?id=cUFIil6hEG [60] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural Message Passing for Quantum Chemistry,” in Proceedings of the 34th International Conference on Machine Learning. PMLR, Jul. 2017, pp. 1263–1272. [Online]. Available: https://proceedings.mlr. press/v70/gilmer17a.html [61] S. Kullback and R. A. Leibler, “On Information and Sufficiency,” The Annals of Mathematical Statistics, vol. 22, no. 1, pp. 79–86, Mar. 1951. [Online]. Available: https://projecteuclid. org/journals/annals-of-mathematical-statistics/volume-22/issue-1/ On-Information-and-Sufficiency/10.1214/aoms/1177729694.full [62] M. Wardat, W. Le, and H. Rajan, “DeepLocalize: Fault Localization for Deep Neural Networks,” in Proceedings of the 43rd International Conference on Software Engineering, ser. ICSE ’21. Madrid, Spain: IEEE Press, May 2021, pp. 251–262. [Online]. Available: https://dl.acm.org/doi/10.1109/ICSE43902.2021.00034 [63] H. Xie, Z. Deng, Y. Lei, M. Li, M. Yan, and D. Lo, “Mutants Will Tell: Statistical Mutation-Based Multiple Fault Localization for Deep Learning Programs,” IEEE Transactions on Software Engineering, vol. 52, no. 3, pp. 939–953, Mar. 2026. [Online]. Available: https://ieeexplore.ieee.org/document/11359137 [64] D. S. Kerby, “The Simple Difference Formula: An Approach to Teaching Nonparametric Correlation,” Comprehensive Psychology, vol. 3, p. 11.IT.3.1, Jan. 2014. [Online]. Available: https://journals. sagepub.com/doi/full/10.2466/11.IT.3.1 [65] S. Holm, “A Simple Sequentially Rejective Multiple Test Procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979. [Online]. Available: https://www.jstor.org/stable/4615733 [66] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering, 2nd ed. Berlin, Heidelberg: Springer, 2024. [Online]. Available: https://link.springer. com/10.1007/978-3-662-69306-3

18

Related documents

• Provably Lossless Acceleration of DNN Mutation Testing via Memoization#386936
• CryoAtlas: A Large Curated Dataset and Unified Benchmark for Cryo-EM Atomic Model Building#1534586
• DIFFNAS: Predictor-Free Neural Architecture Search via Diffusion Model and Zero-Cost Evaluation#1535139
• DIFFNAS: Predictor-Free Neural Architecture Search via Diffusion Model and Zero-Cost Evaluation#1514937
• Rethinking Layer-wise Model Merging through Chain of Merges#1535483