arXiv CS open access

BUMP: Brief Unexpected Mid-Interpolation Peak

Melinda Flóra Kiss et al. · arxiv_cs

arXiv CS · Papers

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

BUMP: Brief Unexpected Mid-Interpolation Peak

Melinda Flóra Kiss 1 2 Márton Muntag 1 Adrián Csiszárik 1 Dániel Varga 1

arXiv:2611.08182v1 [cs.LG] 5 Nov 2026

Abstract

linearly interpolating between two models. It is empirically found to occur near the midpoint of the interpolation loss curve, plotting the loss as a function of the λ interpolation coefficient.

We identify an intriguing phenomenon when linearly interpolating between permutation-aligned neural networks: test accuracy briefly improves near the interpolation endpoints before dropping toward the midpoint, which we term BUMP (Brief Unexpected Mid-interpolation Peak). We observe that BUMP appears consistently across datasets, architectures, training regimes, and remains robust under substantial perturbations such as early checkpoints and extreme pruning. Moreover, controlled ablations suggest that it cannot be attributed to trivial regularization effects. While similar near-endpoint gains are visible in prior interpolation plots, they have not been explicitly documented or systematically and thoroughly analyzed. Leveraging this effect, we propose a simple, novel ensembling scheme: weight–prediction mixed ensembling that combines weight mixing with prediction-space mixing. Across our experiments, the resulting ensembling method consistently outperforms standard logit ensembling in test accuracy and yields improved calibration trade-off, highlighting complementary benefits of weight- and prediction-space mixing.

In this paper, we focus on another point along the interpolation curve. Our main observation is that near the original models, the interpolation actually leads to improved model performance. Concretely, given model A and permutationally aligned model B, the following simple recipe leads to an increase in test accuracy: mix weights linearly according to (1 − λ)A + λB, for λ ≈ 0.1 or symmetrically, λ ≈ 0.9. We refer to this effect as BUMP, coined as a backronym for Brief Unexpected Mid-Interpolation Peak. For various models, datasets, and network width multipliers, we demonstrate small, but statistically highly significant increases in test accuracy when applying such a recipe. While the gains are modest, they are highly consistent, warranting careful analysis and providing insights into merging and ensembling of deep neural networks, and motivating new model combination strategies. Our contributions are as follows:1 • We empirically demonstrate that the BUMP phenomenon appears consistently across multiple datasets, architectures, training regimes, and persists under model degradation and at early checkpoints.

1. Introduction Linear mode connectivity (LMC) has emerged as a useful lens to understand neural network loss landscapes (Frankle et al., 2020; Entezari et al., 2022). As observed by Ainsworth et al. (2023), when the weights of independently trained, heavily overparameterized neural networks are aligned by optimizing over neuron permutations, the resulting networks become linearly mode connected. The standard way to quantify mode connectivity is the loss barrier, defined as the maximal increase in loss observed when

• We formulate several plausible regularization-based hypotheses for the underlying causes of the BUMP phenomenon and empirically falsify them.

• We analyze the BUMP from multiple perspectives, including its connection to classical logit ensembling and an inspection of model predictions.

1 HUN-REN Alfréd Rényi Institute of Mathematics, Budapest, Hungary 2 Department of Computer Science, Eötvös Loránd University, Budapest, Hungary. Correspondence to: Melinda Flóra Kiss <melindafkiss@student.elte.hu>, Adrián Csiszárik <csadrian@renyi.hu>.

• We propose a novel ensembling method that combines weight mixing and logit mixing, achieving higher accuracy than both logit ensembling and the BUMP itself. 1 Code is available in the supplementary material, and will be released in a public repository upon acceptance.

This paper was carried out in parallel with the work of Beatrix Benkő (Benkő, 2026).

BUMP: Brief Unexpected Mid-Interpolation Peak

2. Related Work

3. Notations and formal definition of BUMPs

2.1. Linear mode connectivity and model re-basing

Let A and B be two models with identical architectures and d ∈ N parameters, trained from different random initializations and batch orders. Let ΘA ∈ Rd and ΘB ∈ Rd denote their converged weight vectors. With a slight abuse of notation, we identify models with their weight vectors: we write ΘA (resp. ΘB ) to denote model A (resp. B) with parameters set to ΘA (resp. ΘB ).

Linear mode connectivity (LMC) is a widely studied phenomenon that provides insight into neural network loss landscapes and generalization. An intriguing direction considers permutation symmetries, Entezari et al. (2022) conjectured that, once these symmetries are accounted for, SGD solutions from different initializations become linearly mode connected. This conjecture was empirically demonstrated in Ainsworth et al. (2023), using a simple greedy neuronmatching algorithm family dubbed Git Re-Basin, but only for networks employing Layer Normalization (Ba et al., 2016) and very wide architectures (16x–32x wider than the baseline). Jordan et al. (2022) later extended these results to networks using Batch Normalization (Ioffe & Szegedy, 2015), and to networks without normalization. Theus et al. (2025) considers a broader class of symmetries (including permutations as a special case) and provides the first demonstration of LMC for transformers.

For a model M , datapoint x ∈ D and class c ∈ C, we denote the logit vector corresponding to x by ℓM (x), and denote the logit value corresponding to x and class c by ℓM (x, c), where D denotes the (input) dataset and C the set of class labels. Given models ΘA , ΘB and coefficient λ ∈ [0, 1], we define the LogitMix model as the classical logit ensemble of ΘA and ΘB with coefficient λ. We denote the resulting logit corresponding to x by ℓLogitMix (A, B, x, λ) := λ · ℓB (x) + (1 − λ) · ℓA (x). We define the WeightMix as the weight-interpolated model M (λ, ΘA , ΘB ) with parameter vector λΘB + (1 − λ)ΘA , and denote its logits by ℓWeightMix (A, B, x, λ) := ℓM (λ,ΘA ,ΘB ) (x).

2.2. Model merging and ensembling 3.1. Loss barrier: worst performing model along the interpolation line

Ensembling and model merging both aim to combine multiple trained models to improve performance, robustness, or generalization, but they differ in the level at which models are combined.

Linear mode connectivity and the approximate convexity of the loss basin is captured through the notion of loss barrier. Definition 3.1 (Loss barrier (Frankle et al., 2020)). Given two models ΘA and ΘB with L(ΘA ) ≈ L(ΘB ), the loss barrier is maxλ∈[0,1] L((1 − λ)ΘA + λΘB ) − 12 (L(ΘA ) + L(ΘB )), where L is the utilized loss function.

In ensembling, multiple base predictors are combined into a single predictor, ranging from simple output aggregation (e.g., averaging logits or probabilities) to more general learned combinations. Some methods aggregate model probabilities (soft voting), which work well for well-calibrated and confident models (Lakshminarayanan et al., 2017). Others average logits, preserving relative confidence and often producing sharper, more accurate predictions (Tassi et al., 2022; Buchanan et al., 2023).

Two networks are considered linear mode connected if this barrier is small (near zero). Analogously, an accuracy barrier can be defined by comparing the minimum accuracy along the interpolation line to the average accuracy of ΘA and ΘB .

In model merging, models are combined by operating directly on network weights. Many approaches have been explored, Izmailov et al. (2018) introduces Stochastic Weight Averaging (SWA), which improves generalization by averaging points along an SGD trajectory using cyclical or constant learning rates. Wortsman et al. (2022) proposes model soups, where several fine-tuned models (that share a common initialization but are further trained with different hyperparameters) are weight-averaged to boost accuracy and robustness, offering a cheaper inference-time alternative to logit ensembling. Other methods use weighted combinations of models in federated learning (McMahan et al., 2017). Another line of work, such as Fisher merging, uses Fisher information to guide the merging procedure (Matena & Raffel, 2022).

Ainsworth et al. (2023) demonstrates zero-barrier linear mode connectivity between ΘA and π(ΘB ), where π(ΘB ) is obtained by applying a neuron permutation to ΘB that is computed to align it with ΘA . We call such ΘA , π(ΘB ) pairs “aligned” and refer to unpermuted ΘA , ΘB pairs as “naı̈ve”. 3.2. BUMP models: best performing model along the interpolation line In this work, we focus on the best-performing interpolated model, instead of the worst-performing one. Empirically, we consistently observe models near A and B whose accuracy exceeds that of the closer endpoint. Based on the shape of these interpolation accuracy curves (see Figure 1), we call these BUMP models. Definition 3.2 (BUMP model). Let ΘA , ΘB ∈ Rd be two 2

BUMP: Brief Unexpected Mid-Interpolation Peak

-0.5 -1.0 -1.5 -2.0 -2.5 0.00

0.5

0.0

-0.5

Modelpair seeds (0, 1) Modelpair seeds (2, 3) Modelpair seeds (4, 5) Modelpair seeds (6, 7) Modelpair seeds (8, 9) Modelpair seeds (10, 11) Modelpair seeds (12, 13) Modelpair seeds (14, 15) Modelpair seeds (16, 17) Modelpair seeds (18, 19) 0.05

0.10

-1.5 -2.0

0.15

0.20

0.25

0.30

(a) Left BUMP (aligned)

0.0

-0.5

-1.0

-2.5 0.70

0.5

Accuracy gain

0.0

Accuracy gain

0.5

Accuracy gain

0.5

0.75

0.80

0.85

0.90

0.95

-1.0 -1.5 -2.0 -2.5 0.00

1.00

(b) Right BUMP (aligned)

0.0

-0.5

0.10

-1.0 -1.5 -2.0

0.15

0.20

0.25

(a) Left BUMP (naı̈ve)

0.30

-2.5 0.70

0.75

0.80

0.85

0.90

0.95

1.00

(b) Right BUMP (naı̈ve)

Figure 1. The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-10 test set, models are ResNet-20, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.

Figure 2. The two BUMPs when interpolating between models without alignment, the accuracies are measured on the CIFAR-10 test set, models are ResNet-20, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.

models and let λ ∈ [0, 1]. We call the WeightMix model M (λ, ΘA , ΘB ) a BUMP model if its accuracy exceeds that of the closer endpoint, i.e.,

Method “Fixed-λ”: For a model pair (ΘA , ΘB ), we fix the interpolation coefficient λ to obtain models M1 = M (λ, ΘA , ΘB ) and M2 = M (1 − λ, ΘA , ΘB ), i.e., we evaluate BUMPs at λ and 1 − λ. The λ parameter was chosen based on extensive experiments on the training dataset (determining highest accuracy BUMP locations and averaging the results). Based on these evaluations we chose λ = 0.05 for 4x-width, λ = 0.0075 for 8x-width and λ = 0.1 for 16x-width. We measure BUMP accuracy gains as acctest (M1 ) − acctest (A) and acctest (M2 ) − acctest (B).

( acc(M (λ, ΘA , ΘB )) >

acc(ΘA ), acc(ΘB ),

if λ < 12 , if λ ≥ 12 .

4. Empirical demonstration of the BUMP In this section, we provide empirical evidence that the BUMP is a consistently occurring phenomenon. We experiment on multiple datasets and architectures. We report results on CIFAR-10 Krizhevsky (2009), training 100 ResNet-20 He et al. (2016) models with different initialization seeds, for all width multiplier in {4, 8, 16}. We form 50 model pairs, and align each pair via the weight-matching algorithm of Ainsworth et al. (2023). We interpolate linearly between the aligned models using 25 equally spaced coefficients in [0, 1]. We also train Tiny-10 models on CIFAR-10, and ResNet-20 models on CIFAR-100 Krizhevsky (2009), the results can be found in Appendix D and C. The training details for both model architectures can be found in Appendix A. (For additional evidence with further models and training regimes, see Section 5.)

Method “Test”: For a model pair (ΘA , ΘB ) we search for the interpolated model with the highest accuracy on the test dataset separately on the intervals [0, 0.5) and [0.5, 1], yielding two models M1 and M2 . Again, we measure BUMP accuracy gains as acctest (M1 ) − acctest (A) and acctest (M2 ) − acctest (B). We are aware that methodologically this constitutes data contamination (as we are selecting the highest accuracy on the test set), but we include these values because the smoothness of the accuracy curves suggests that they are more indicative of the accuracy gains achievable by the method. Since the models A and B play symmetric roles, we average all 100 BUMP gains per dataset–architecture–width setting, rather than treating the first and second BUMPs separately (50 each). We report results for both methods on CIFAR-10, ResNet-20 in Table 1 (aligned pairs) and Table 2 (naı̈ve pairs). These demonstrate that BUMP gains are consistently present, and that alignment substantially amplifies the BUMP effect compared to naı̈ve pairs.

For 16x-width, we plot interpolation curves for λ ∈ (0.0, 0.3) and λ ∈ (0.7, 1.0) across 10 model pairs. Figure 1 shows clear BUMPs near both endpoints for aligned model pairs, whereas for naı̈ve pairs the BUMP is absent or much smaller (Figure 2). For width multipliers 1 and 2, BUMPs are negligible (note, that for these widths, the loss barriers remain large even after permutation). BUMPs also appear at 32x-width, but with smaller accuracy gains than at 16x-width.

5. Robustness of the BUMP phenomenon In this section, we show that the BUMP phenomenon is robust and widespread: it can be observed consistently across architectures and training regimes, and it persists under substantial degradation of the underlying models. Concretely, we provide evidence along three axes: (i) the effect is visi-

To quantify the accuracy gain of a BUMP we use multiple methods. We denote a model M ’s training and test accuracies by acctrain (M ) and acctest (M ) respectively. 3

BUMP accuracy gain (%)

Table 1. Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs. Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) trained to convergence (250 epochs). Standard deviations are denoted in parentheses.

Accuracy gain % — Aligned pairs Width 4x-width 8x-width 16x-width

Fixed-λ

Test

0.053 (±0.096) 0.118 (±0.111) 0.181 (±0.137)

0.087 (±0.081) 0.164 (±0.117) 0.227 (±0.140)

Table 2. Average accuracy gains of the BUMPs over their corresponding endpoint models on non-aligned, i.e., naı̈ve model pairs. Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) trained to convergence (250 epochs). Standard deviations are denoted in parentheses.

BUMP gain Model accuracy

1.0 0.8

Average model accuracy (%)

BUMP: Brief Unexpected Mid-Interpolation Peak 95 94 93

0.6

0.4

91 90

0.2

89 0.0

120

180

Checkpoint (epoch)

250

Figure 3. BUMP gain (solid line, left axis) and average model accuracy (dashed line, right axis) as a function of training epochs for 16x-width ResNet-20 models on CIFAR-10. Values are averages for 50 model pairs, bands represent standard deviations.

Accuracy gain % — Naı̈ve pairs Width 4x-width 8x-width 16x-width

Fixed-λ

Test

−0.041 (±0.111) −0.041 (±0.123) −0.073 (±0.147)

0.038 (±0.061) 0.051 (±0.065) 0.056 (±0.069)

Figure 3 show the results for checkpoints 60, 120, 180, and the full training of 250 epochs. We can observe that BUMPs are already present well before convergence and their magnitude is often larger at intermediate checkpoints than for fully trained models, and diminishes toward convergence. This shows that the phenomenon does not rely on late-stage optimization and can be strongest when models are still relatively inaccurate.

ble (though typically unremarked) in interpolation figures of prior works, (ii) it already appears well before full convergence along the training trajectory and is often more pronounced at earlier checkpoints than at convergence, and (iii) it survives—and frequently strengthens—under extreme sparsification via magnitude pruning. Together, these results argue against the interpretation of BUMPs as fragile interpolation artifacts that occur only under carefully tuned conditions.

5.3. BUMPs in pruned models We study how weight pruning affects the BUMP phenomenon, asking whether BUMPs persist under extreme sparsification. We apply a single round of global magnitude pruning whose purpose is not pruning optimization but deliberate performance degradation and sparsification. For the 16x-width ResNet-20 model on CIFAR-10, we prune 95% of the weights and evaluate interpolations using the same Fixed-λ protocol as in Section 4.

5.1. BUMPs in prior works After observing the BUMP phenomenon in our own experiments, we examined prior works, and found similar local performance improvements near the endpoints of aligned interpolation curves across a wide range of studies, including those involving convolutional networks and transformers. While these effects are visible in several published interpolation plots, they are not discussed and are at most briefly mentioned; in other words, the BUMP phenomenon is not explicitly explored in these works (see Figure 1 in McMahan et al. (2017), Figure 5 in Ainsworth et al. (2023) and Figure 4 in Theus et al. (2025)).

We consider two approaches: (i) pruning first, followed by alignment via weight permutation, and (ii) alignment first, followed by pruning. In both cases, BUMPs remain clearly present. When pruning is applied before alignment, the average Fixed-λ BUMP gain is 1.54% ± 1.07%; when pruning is applied after alignment, the gain is 1.60% ± 1.11%. Notably, these gains are substantially larger than those observed for fully dense, converged models, despite pruning reducing average test accuracy from approximately 94.5% to 74.6%. This demonstrates that BUMPs are robust to severe sparsification and do not rely on dense or carefully optimized model representations.

5.2. BUMPs at earlier checkpoints

Overall, these results establish the BUMP phenomenon as a systematic and robust property of neural network interpolations, persisting across architectures, training stages, and even under severe model degradation.

To test whether BUMPs depend on full convergence, we analyze interpolations between models taken from earlier checkpoints of the training process. 4

BUMP: Brief Unexpected Mid-Interpolation Peak

6. The BUMP is a result of a simple regularization effect?

0.950 0.948

0.944

Test, permuted

0.942

When seeking for an explanation for the BUMP phenomenon, a natural hypothesis is that BUMP reflects a simple form of regularization. We investigate this hypothesis from multiple perspectives, including whether the effect depends on moving toward another trained model or whether it can be reproduced by adding noise to the logits.

Accuracy

0.8

0.946

0.6

Test, permuted

0.4

0.940 0.938

0.2

0.936 0.0

0.2

0.4

0.6

0.8

1.0

(a) Distance multiplier 1

0.0

0.2

0.4

0.6

0.8

1.0

(b) Distance multiplier 5

Figure 4. Interpolation in the direction of a randomly chosen model on CIFAR-10, ResNet-20. Width multiplier is 16, the distance multiplier determines the distance of model A from the random model R compared to the distance of model A and B.

6.1. Is the direction important? In the previous experiments, we moved in parameter space from one trained model toward another. This raises the question of whether the BUMP specifically depends on moving toward another trained model. To test this, we conduct two sets of experiments.

domly “reflecting” parameters. Specifically, for each parameter pair (wa , wb ) ∈ ΘA × π(ΘB ), we set wb := 2wa − wb with probability 0.5, yielding a new model B ′ that is, coordinate-wise, equally far from A. We then interpolate between A and B ′ . Table 4 reports results averaged over all model pairs. As in the previous experiment, we observe no significant accuracy gains.

6.1.1. R ANDOM DIRECTION MODEL In this experiment, we interpolate between a trained model A and a random model R with parameter vector ΘR sampled uniformly from a hypersphere centered at A. For each aligned pair (A, π(B)) we sample R on the sphere with radius ∥ΘA − ΘB ∥2 . To measure accuracy change, we use slightly modified versions of our earlier methods. With Fixed-λ, we evaluate only the model (1 − λ)ΘA + λΘR , and for method “Test”, we search over the full interval [0, 1] to obtain the interpolating coefficient, instead of splitting into halves. Table 3 reports results for width multipliers 4, 8, 16, averaged over model pairs.

Table 4. Average accuracy differences when moving towards models with randomly reflected weights on CIFAR-10 with ResNet-20 (50 model pairs). Standard deviations are denoted in parentheses.

Accuracy gain % — Randomly reflected Width 4x-width 8x-width 16x-width

Table 3. Average accuracy differences when moving towards random directions on CIFAR-10 with ResNet-20 (50 model pairs), with standard deviations denoted in parentheses.

Fixed-λ

Test

−0.081 (±0.117) −0.094 (±0.112) −0.077 (±0.110)

0.025 (±0.042) 0.022 (±0.043) 0.028 (±0.046)

Accuracy gain % — Random direction Width 4x-width 8x-width 16x-width

Fixed-λ

Test

−0.009 (±0.065) −0.012 (±0.056) −0.002 (±0.053)

0.028 (±0.035) 0.035 (±0.040) 0.031 (±0.035)

6.2. Adding noise to the logits The hypothesis that the BUMP is a regularization effect is particularly compelling in light of the relationship between the logits produced by model A and those produced by model M1 (the closer BUMP to A). As shown in Figure 5, there is an extremely strong linear relationship across data points and labels: to a first approximation, each logit output by model M1 is simply a constant multiple of the corresponding logit from model A. This might suggest a seemingly trivial explanation, namely that such a constant rescaling alone improves performance. However, this cannot be the case, since scaling logits by a constant factor does not change the predicted labels.

We find no BUMPs when using a Fixed-λ, and the BUMPs selected by the “Test” method are negligible compared to those observed when interpolating toward π(B). Consistent with this, the interpolation line segment between A and a randomly sampled model shows only minor accuracy variation (Figure 4a). When we increase the sphere radius to 5 · ∥ΘA − ΘB ∥2 , accuracy stays nearly constant for a short range and then drops sharply (Figure 4b).

We now exclude two, successively more subtle variations of this hypothesis, namely that the logits of model M1 can be explained as a noisy affine transformation of the logits of model A.

6.1.2. R ANDOMLY REFLECTED MODEL To further test whether moving toward a trained model is essential, we construct a perturbed version of π(B) by ran5

Logits Fit: y=0.9776x+0.0001

0.9525

0.16

0.9500

0.14 0.12

0.9475

0.10

0.9450

0.08

0.9425

0.06

0.9400

0.04

0.9375

0.02

permuted weightmix permuted logitmix

0.9350 0.9325

0.0

0.2

0.4

0.00 0.6

0.8

1.0

(a) Accuracy curves

A logits

DKL(E||Mweight mix) DKL(E||Mlogit mix)

KL Divergence

30 25 20 15 10 5 0 5 10

Accuracy

M1 logits

BUMP: Brief Unexpected Mid-Interpolation Peak

0.0

0.2

0.4

0.6

0.8

1.0

(b) KL divergences

Figure 6. Left (comparing accuracy curves): The WeightMix and LogitMix accuracy curve on CIFAR-10, ResNet-20, width multiplier is 16. Right (comparing class probabilities): KullbackLeibler divergence between the class probability distributions of the WeightMix/LogitMix models and the λ = 0.5 LogitMix (Ensemble) model, CIFAR-10, ResNet-20, width multiplier is 16.

Figure 5. Relationship between logit values of A and M1 models on CIFAR-10, ResNet-20, width multiplier is 16. The red line indicates the best-fit affine relationship, highlighting the linear correspondence between the logits of the original and BUMP models.

7. Comparing BUMP to Classical Logit Ensembling In this section, we investigate how the BUMP phenomenon relates to classical logit ensembling, i.e., LogitMix. Figure 6a compares the accuracy curves of the LogitMix and WeightMix models for two 16x-width models. Although the two curves behave similarly near the endpoints, this similarity alone does not imply that the underlying mechanisms are the same. To address this question, we carry out a geometric and probabilistic analysis comparing the two models in detail.

6.2.1. H OMOSCEDASTIC AFFINE NOISE MODEL In this experiment, we model the conditional distribution of ℓM1 (x, c) given ℓA (x, c) as ℓM1 (x, c) = a · ℓA (x, c) + b + ε, with scalar parameters a, b, σ ∈ R, where ε ∼ N (0, σ 2 ). The parameters a, b are fitted by least squares, and the σ is the empirical standard deviation of ℓM1 (x, c)−(a·ℓA (x, c)+ b) on the test dataset. We then replace the true logit values ℓM1 (x, c) by samples drawn from the fitted conditional distribution and evaluate classification accuracy using these synthetic logits. If model M1 differed from model A only by an affine transformation with homoscedastic noise, sampling from this model would be expected to preserve or improve accuracy. However, we observe no statistically significant improvement, see Table 16 in Appendix B.

7.1. Inspecting the angles We directly compare the logits of the WeightMix and LogitMix models. More precisely, we measure the angle between the logits for the same inputs. If the two mixing procedures created identical logits, this angle would be zero. For a model pair (A, B), g ∈ Z+ and a datapoint x, let vℓ (x) := ℓLogitMix (A, B, x, 1) − ℓLogitMix (A, B, x, 0), vw,i (x) := ℓWeightMix (A, B, x, i/g) − ℓWeightMix (A, B, x, 0), for i = 1, . . . , g. (Note, that since we are comparing the angles, and LogitMix logits all lie on the line segment determined by the logits corresponding to models A, B, we may choose one of the LogitMix logit differences as a reference.) For each i, we compute the angle between vℓ (x) and vw,i (x), average over test datapoints, and then average over the 50 model pairs. Figure 7 reports the mean angle as a function of λ = i/g. (For i = g, we recover model B, so the angle is necessarily 0◦ .) The large angles for small i indicate that WeightMix and LogitMix models produce substantially different logits near model A.

6.2.2. H ETEROSCEDASTIC AFFINE NOISE MODEL We next consider a more flexible model in which the variance of the noise depends on the logit value of model A. Specifically, we set ℓM1 (x, c) = a · ℓA (x, c) + b + ε, where ε ∼ N (0, σ(ℓA (x, c))2 ), allowing the noise to vary as a function of ℓA (x, c). The parameters a, b are again fitted by least squares, while σ is estimated by binning the values of ℓA (x, c), estimating the empirical variance within each bin and fitting a polygonal line on these values. We used 10 bins in our experiments. Again, we replace the true logit values with samples and we observe significantly worse results in terms of accuracy, see Table 16 in Appendix B.

7.2. Probability-Space Proximity to the Logit Ensemble To compare weight mixing and logit ensembling at the granularity of predicted class-distributions, we take the λ = 0.5 6

BUMP: Brief Unexpected Mid-Interpolation Peak

Angle (degrees)

For an input x, let ℓB (x) denote the logits of model ΘB , and let ℓM (λ) (x) denote the logits of M (λ, ΘA , ΘB ). We define two variants:

Width multiplier 4 Width multiplier 8 Width multiplier 16

70 60 50 40

ℓWPME-Logit (x; λ, β) = (1 − β) ℓM (λ) (x) + β ℓB (x), pWPME-Prob (x; λ, β) = (1 − β) softmax ℓM (λ) (x) + β softmax ℓB (x) .

30 20 10 0 0.0

0.2

0.4

0.6

0.8

1.0

The prediction of WPME–Logit is obtained by applying the softmax to ℓWPME-Logit .

Figure 7. The mean angles across the test datapoints for g = 24, CIFAR-10, ResNet-20, width multiplier is 16. Values are averages for 50 model pairs.

Setting β = 0 yields predictions from the WeightMix model M (λ, ΘA , ΘB ), and β = 1 yields predictions from the endpoint model B. Thus β controls the trade-off between the WeightMix model and the original predictor at endpoint B, rather than directly averaging the logits or weights of model A and B.

LogitMix model as a reference, denoted by E (for Ensemble), and measure with the Kullback–Leibler divergence how closely WeightMix and LogitMix track it. Figure 6b shows the results. Near the endpoints A and B, the divergence decreases (even for WeightMix models), indicating that the interpolation path moves toward E rather than away from it. However, Figure 6b also makes clear that WeightMix approaches E only modestly compared to LogitMix: the predictive distributions remain substantially farther from E across λ. One possible interpretation is that both procedures move toward E simply because the ensemble distribution concentrates better predictions, and many reasonable perturbations from A or B may initially improve agreement with it. The quantitative gap suggests that weight mixing is not equivalent to classical logit ensembling, instead, it induces a distinct transformation of the predictive distribution.

WPME accuracy outperforms baselines. Figure 8 shows accuracy heatmaps obtained by sweeping the weight mixing coefficient λ and the prediction mixing coefficient β. Across both checkpoints, WPME exhibits broad regions of high accuracy that exceed those achieved by classical logit mixing or the weight mixing (i.e., the BUMP). (Note, that classical logit mixing corresponds to the horizontal line λ = 0, while weight mixing to the vertical line β = 0 on the heatmap.) The accuracy landscape is smooth and unimodal, indicating that weight-space interpolation and predictionspace interpolation provide complementary benefits rather than requiring precise hyperparameter tuning.

A natural question arises: is weight mixing still merely an approximation of classical ensembling, or does it provide complementary information that can be exploited beyond standard logit mixing?

WPME improves the accuracy–calibration trade-off. As shown in Table 5, WPME-logit achieves the highest accuracy at both checkpoints while modestly improving calibration relative to LogitMix; WPME-prob attains near-SoftMaxMix calibration with slightly higher accuracy (where SoftmaxMix simply linearly averages predicted probabilities, p(x) = (1 − β) softmax(ℓA (x)) + β softmax(ℓB (x))).

8. A novel ensembling method: combining weight and logit mixing In this section, we introduce Weight–Prediction Mixed Ensembling (WPME), a novel ensembling method that jointly combines models in weight space and prediction space, and surpasses the accuracy of both classical logit ensembling and the BUMP itself. Classical ensembling combines model predictions (logits or probabilities) to reduce variance, but it does not leverage information directly contained in the model parameters. Our novel ensembling scheme demonstrates that weight mixing provides complementary information to output-space aggregation and can be exploited to achieve higher generalization performance.

Validation protocol To avoid tuning mixing coefficients on the test set, we split the original test data into a validation split and a held-out test split. For each method, we select the hyperparameters (λ, β) that maximize validation-set accuracy, and then report accuracy, NLL, and ECE on the held-out test split. In summary, WPME demonstrates that weight mixing is not merely an approximation of logit ensembling, but a complementary ingredient that can be leveraged for improved accuracy and calibration.

Definition 8.1 (Weight–Prediction Mixed Ensembling (WPME)). Given two models ΘA , ΘB and coefficients λ, β ∈ R, we form the WeightMix model M (λ, ΘA , ΘB ). 7

BUMP: Brief Unexpected Mid-Interpolation Peak Table 5. Comparison of Weight–Prediction Mixed Ensembling with baseline methods on CIFAR-10, ResNet-20, width-multiplier 16: accuracy (Acc.), negative log-likelihood (NLL), and expected calibration error (ECE) values for different ensembling methods of two models at two different stages of model training (epoch 60 and 250). Reported values are sample means ± standard deviations (SD). Best results within each checkpoint are shown in bold. Model checkpoints: @60 epochs

A (orig) B (orig) LogitMix SoftMaxMix WeightMix WPME-logit WPME-prob

NLL ↓

ECE ↓

0.89972 (±0.00871) 0.90011 (±0.01077) 0.92369 (±0.00591) 0.92160 (±0.00607) 0.91268 (±0.00593) 0.92605 (±0.00513) 0.92424 (±0.00567)

0.30911 (±0.03136) 0.31030 (±0.04081) 0.22944 (±0.01954) 0.24270 (±0.01824) 0.25942 (±0.01875) 0.21918 (±0.01612) 0.23245 (±0.01576)

0.02789 (±0.00650) 0.02893 (±0.00905) 0.01374 (±0.00385) 0.01990 (±0.00482) 0.01387 (±0.00514) 0.01015 (±0.00297) 0.02124 (±0.00682)

A (orig) B (orig) LogitMix SoftMaxMix WeightMix WPME-logit WPME-prob

0.8

0.910

0.6

0.4

0.900

0.4

Accuracy

0.905 0.895

0.2

0.890 0.2

0.4

0.6

0.8

1.0

(a) Checkpoint at epoch 60

0.950

0.8

0.6

0.945 0.940

0.0 0.0

0.930 0.2

0.4

0.6

0.8

ECE ↓ 0.03313 (±0.00256) 0.03315 (±0.00243) 0.02598 (±0.00222) 0.01496 (±0.00214) 0.03144 (±0.00225) 0.02555 (±0.00227) 0.01499 (±0.00209)

Average number of datapoints

0.935 0.2

NLL ↓ 0.23465 (±0.01756) 0.23454 (±0.01448) 0.18520 (±0.01103) 0.16739 (±0.00897) 0.22169 (±0.01343) 0.18137 (±0.00991) 0.16484 (±0.00905)

Table 6. Comparing model M1 with model A on CIFAR-10, ResNet-20 with method Fixed-λ. Numbers are averages calculated for 50 model pairs.

1.0 0.915

Acc. ↑ 0.94949 (±0.00349) 0.94968 (±0.00308) 0.95790 (±0.00262) 0.95742 (±0.00288) 0.95162 (±0.00309) 0.95848 (±0.00261) 0.95816 (±0.00292)

Method

Accuracy

1.0

0.0 0.0

Model checkpoints: @250 (full training)

Acc. ↑

Method

1.0

(b) Checkpoint at convergence

Figure 8. Accuracy over (λ, β) for WPME obtained by interpolating model parameters in weight-space (λ) followed by logitspace mixing (β) with endpoint model B. Results are shown for a partially trained model (left) and the fully trained model (right), CIFAR-10, ResNet-20, width multiplier is 16.

Width

A #1

A #2

Other

4x-width 8x-width 16x-width

9909.72 9893.92 9880.44

85.88 99.12 110.80

4.40 6.96 8.76

and the closer endpoint. We form two main categories: correct → incorrect denotes those datapoints, where model A predicts the true label and M1 does not, and incorrect → correct denotes those datapoints, where model M1 predicts the true label and A does not. (Note, that these datapoints characterize the accuracy gain of the BUMP.) In each category, we count test datapoints where M1 ’s top #1 prediction matches B’s #1, B’s #2 prediction, or neither, and then average across the 50 pairs. Thus, we quantify how the farther model influences the BUMP’s prediction, and whether it “corrects” or “ruins” the predictions of the closer model. For each model pair, we count datapoints in these overall six categories, and then average across the 50 model pairs. As a control, we also introduce an independently chosen model C for each pair to compare whether the BUMP’s predictions are more likely to match B, than those of C. Results are shown in Table 7. Overall, we observe only minor differences between alignment with B and alignment with C in prediction. We therefore hypothesize that B contributes by providing an “informed” regularization and sharpening A’s decision function through the interpolation direction, as opposed to causing M1 to explicitly inherit B’s predicted labels. We leave the further exploration of this for future work.

9. Exploring the relationship between the predictions of the interpolated and the original models Here, we analyze the relationship between the BUMP model’s predictions and those of the original models. Comparing the BUMP with the closer model For each model pair (A, B), we count test datapoints where M1 ’s top #1 prediction matches A’s #1, A’s #2 prediction (second largest probability), or neither, and then average across the 50 pairs. Results are shown in Table 6. In most cases M1 agrees with A, when it does not, its prediction typically matches A’s second choice, while the remaining cases are negligible. This suggests that BUMPs primarily arise from small, local changes in the decision boundary: the interpolated model M1 usually preserves A’s top-1 decision, and when it deviates it most often swaps to A’s runner-up rather than producing completely different predictions. Comparing the BUMP with the farther model It is natural to assume that the accuracy gain of M1 relative to A reflects information contributed by the other model B. In this section, we test this hypothesis. We restrict to test datapoints, where the truthiness of the prediction (whether the predicted label matches the true label) differs for the BUMP

10. Conclusion We thoroughly analyzed BUMP, an accuracy peak near the endpoints of interpolation between permutation-aligned 8

BUMP: Brief Unexpected Mid-Interpolation Peak Table 7. Comparing the predictions of model M1 with predictions from model B and random chosen model C on CIFAR-10, 16xwidth ResNet-20 (50 model pairs) with method Fixed-λ. In each column, we present the average number of datapoints.

In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 3259–3269. PMLR, 2020. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016. doi: 10.1109/CVPR.2016.90.

Average number of datapoints with changed truthiness

B C

correct → incorrect

incorrect → correct

Other

10.30 10.80

18.94 18.18

11.02 11.28

40.72 39.26

13.78 15.04

5.92 6.12

Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 448–456. PMLR, 2015.

models that has not been explicitly discussed or analyzed in prior work. Leveraging BUMP, we proposed WPME, a simple hybrid of weight- and prediction-space mixing that consistently outperforms classical logit ensembling. Overall, our results demonstrate that aligned weight space offers practical opportunities for improved model combination.

Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, 2018. URL https://arxiv.org/abs/1803.05407.

Impact Statement

Jordan, K., Sedghi, H., Saukh, O., Entezari, R., and Neyshabur, B. REPAIR: REnormalizing permuted activations for interpolation repair, 2022. URL https: //arxiv.org/abs/2211.08403.

This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

References

Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, pp. 6402–6413, 2017.

Ainsworth, S., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=CQsmMYmlP5T.

Matena, M. and Raffel, C. Merging models with fisherweighted averaging. Advances in Neural Information Processing Systems, 35:17703–17716, 2022.

Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization, 2016. URL https://arxiv.org/abs/1607. 06450.

McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. y. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, pp. 1273–1282. PMLR, 2017.

Benkő, B. Quadratic effects of linear interpolation between permutation-aligned neural networks. Neural Networks, 203:109074, 2026. doi: 10.1016/j.neunet.2026.109074. Buchanan, E. K., Pleiss, G., Wang, Y., and Cunningham, J. P. The effects of ensembling on long-tailed data. In NeurIPS 2023 Workshop: Heavy Tails in Machine Learning, 2023. URL https://openreview.net/pdf? id=l4GYs60kre. Workshop paper.

Tassi, C. R. N., Gawlikowski, J., Fitri, A. U., and Triebel, R. The impact of averaging logits over probabilities on ensembles of neural networks. In Proceedings of the 2022 Workshop on Artificial Intelligence Safety (AISafety 2022), co-located with IJCAI-ECAI 2022, volume 3215 of CEUR Workshop Proceedings, 2022. URL https: //ceur-ws.org/Vol-3215/19.pdf.

Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022. URL https: //openreview.net/forum?id=dNigytemkL.

Theus, A., Cabodi, A., Anagnostidis, S., Orvieto, A., Singh, S. P., and Boeva, V. Generalized linear mode connectivity for transformers, 2025. URL https://arxiv.org/ abs/2506.22712. NeurIPS 2025 (oral).

Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. 9

BUMP: Brief Unexpected Mid-Interpolation Peak

Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965–23998. PMLR, 2022. URL https://proceedings.mlr.press/ v162/wortsman22a.html.

BUMP: Brief Unexpected Mid-Interpolation Peak

A. Training details Tiny-10 Tiny-10 is a simple non-residual convolutional network where convolutional, normalization, and activation layers are following each other. Table 8 contains the precise architecture of the variant we were using in our experiments. We used Layer Normalization (Ba et al., 2016) to follow the methodology of Ainsworth et al. (2023). We trained our models for 300 epochs, we used SGD with momentum 0.9. The initial learning rate was set to 0.01, which was then divided by 10 at the third and the two thirds of the training. The batch size was 100. We used weight decay with value 10−4 . Table 8. The Tiny-10 architecture we used.

Tiny-10 Layers

Name

3 × 3 conv. 16-LN-ReLu 3 × 3 conv. 16-LN-ReLu 3 × 3 conv. 32-LN-ReLu 3 × 3 conv. 32-LN-ReLu 3 × 3 conv. 32-LN-ReLu 3 × 3 conv. 64-LN-ReLu 3 × 3 conv. 64-LN-ReLu 1 × 1 conv. 64-LN-ReLu Global average pooling Dense Logits

Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 8

ResNet-20 We used the ResNet-20 architecture of He et al. (2016), but with LayerNorms instead of BatchNorms. We trained our models for 250 epochs, we used SGD with momentum 0.9. We applied a weight decay regularization term of 10−4 , and used cosine decay schedule with linear warmup: the learning rate was initialized at 10−6 and linearly increased to 10−1 over an epoch and after a cosine decay schedule was used for the rest of the training. The batch size was 100. Data augmentation

The following data augmentation was performed during training for both models:

• Random resizes of the image with scale factors between 0.8 and 1.2. • Random 32 × 32 pixel crops. • Random horizontal flips. • Random rotations between ±30◦ .

B. Additional results for CIFAR-10 with ResNet-20 models In this section, we present additional tables and figures to further support our experiments described in Sections 5, 6, 7, 8. B.1. Earlier checkpoints and pruning We report accuracy gains of the BUMPs in tables analogous to Table 1 and Table 2. Results are shown for checkpoints at epoch 60 (Table 9), epoch 120 (Table 10) and epoch 180 (Table 11). Table 12 summarizes the average accuracies across the 50 model pairs (A, B) at epochs 60, 120, 180 and 250 (full training). We also examine pruning effects: BUMP accuracy gains when pruning is applied before alignment are given in Table 13 and gains when pruning is applied after alignment are shown in Table 14. For 16x-width and 8x-width we pruned 95% of the weights, and for 4x-width we pruned 90% of the weights. Finally, Table 15 reports the average accuracies of the 50 pruned model pairs. B.2. Noise models We present the results for our homoscedastic and heteroscedastic noise model experiments in Table 16. 11

BUMP: Brief Unexpected Mid-Interpolation Peak Table 9. Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs. Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) at epoch 60. Standard deviations are denoted in parentheses.

Accuracy gain % — Aligned pairs at epoch 60 Width 4x-width 8x-width 16x-width

Fixed-λ

Test

0.346 (±0.223) 0.577 (±0.252) 0.731 (±0.309)

0.663 (±0.528) 1.009 (±0.549) 1.188 (±0.590)

Table 10. Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs. Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) at epoch 120. Standard deviations are denoted in parentheses.

Accuracy gain % — Aligned pairs at epoch 120 Width 4x-width 8x-width 16x-width

Fixed-λ

Test

0.291 (±0.178) 0.448 (±0.191) 0.614 (±0.261)

0.506 (±0.365) 0.732 (±0.380) 0.970 (±0.487)

Table 11. Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs. Results are reported on the CIFAR-10 test set using aligned ResNet-20 pairs (100 models) at epoch 180. Standard deviations are denoted in parentheses.

Accuracy gain % — Aligned pairs at epoch 180 Width 4x-width 8x-width 16x-width

Fixed-λ

Test

0.223 (±0.148) 0.358 (±0.173) 0.513 (±0.187)

0.354 (±0.250) 0.526 (±0.300) 0.755 (±0.348)

Table 12. Average model accuracy for CIFAR-10, ResNet-20 at different training epochs, comparing Model A and Model B for various model widths (100 models).

Average Accuracy Width

At epoch 60 A B

4x-width 8x-width 16x-width

0.8729 0.8869 0.8980

0.8736 0.8858 0.8935

At epoch 120 A B

At epoch 180 A B

At epoch 250 A B

0.8966 0.9081 0.9135

0.9126 0.9206 0.9251

0.9351 0.9418 0.9450

0.8945 0.9069 0.9131

0.9138 0.9196 0.9249

0.9356 0.9412 0.9456

Table 13. Average accuracy gains of the BUMPs over their corresponding endpoint models when pruning is applied before alignment. Results are reported on the CIFAR-10 test set using ResNet20 pairs (100 models) trained to convergence (250 epochs). Standard deviations are denoted in parentheses.

Table 14. Average accuracy gains of the BUMPs over their corresponding endpoint models when pruning is applied after alignment. Results are reported on the CIFAR-10 test set using ResNet20 pairs (100 models) trained to convergence (250 epochs). Standard deviations are denoted in parentheses.

Accuracy gain % — Pruned before alignment

Accuracy gain % — Pruned after alignment

Width

4x-width 8x-width 16x-width

Fixed-λ

Test

1.592 (±0.946) 2.359 (±1.640) 1.536 (±1.072)

3.317 (±2.705) 6.071 (±4.958) 2.171 (±1.819)

4x-width 8x-width 16x-width

Fixed-λ

Test

1.933 (±1.057) 2.296 (±1.635) 1.603 (±1.108)

4.775 (±3.487) 6.300 (±5.133) 2.550 (±2.066)

BUMP: Brief Unexpected Mid-Interpolation Peak Table 15. Average model accuracies for CIFAR-10, ResNet-20 across widths, comparing original and pruned variants of Model A and Model B (100 models).

Average Accuracies Width

Model A

Model B

Pruned Model A

Pruned Model B

4x-width 8x-width 16x-width

0.9351 0.9418 0.9450

0.9356 0.9412 0.9456

0.6195 0.4613 0.7433

0.6235 0.4652 0.7493

Table 16. Accuracy gains (mean, std, p-value) for homoskedastic and heteroskedastic noise models across widths.

Homoskedastic

Heteroskedastic

Width

mean

std

mean

std

4x-width 8x-width 16x-width

-0.0001 -0.0001 -0.0002

0.0006 0.0007 0.0006

0.146 0.22 0.0186

-0.0005 -0.0007 -0.0009

0.0007 0.0009 0.0010

1.27e-05 7.4e-07 2.27e-07

B.3. Further figures to explore the Weight-Prediction Mixed Ensembling

0.8

0.0425 ECE

0.0400

0.4

0.0375 0.0350

0.2

0.0325 0.2

0.4

0.6

0.8

1.0

0.040

0.8

0.0450

0.6

0.0 0.0

1.0

0.0475

0.0300

0.035

0.6

0.030

0.4

0.025

0.2

0.020

0.0 0.0

ECE

1.0

0.2

0.4

0.6

0.8

1.0

0.015

Figure 9. Expected calibration error (ECE; 15 bins) over (λ, β) for probability-space hybrid ensembling, where weight-interpolated models are combined with endpoint predictions after softmax. Lower values indicate improved calibration.

C. Results for CIFAR-100 with ResNet-20 models In this section, we present the figures and tables for experiments on the CIFAR-100 dataset with ResNet-20 models. For each width multiplier in {4, 8, 16} we train 50 models, form 25 model pairs, and align them via the weight-matching algorithm of Ainsworth et al. (2023). We used a similar methodology to quantify accuracy gain as in Section 4, but with λ = 0.04 for 4x-width, λ = 0.07 for 8x-width, λ = 0.08 for 16x-width with the Fixed-λ method. C.1. The BUMPs for aligned and naı̈ve model pairs We present figures analogous to Figures 1 and 2, together with tables analogous to Tables 1, and 2. For the 16x-width setting, we plot interpolation curves for λ ∈ (0.0, 0.3) and λ ∈ (0.7, 1.0) across 10 model pairs. Figure 10 shows clear BUMPs near both endpoints for aligned model pairs, whereas for naı̈ve pairs the BUMP is absent or much 13

BUMP: Brief Unexpected Mid-Interpolation Peak

0.5

Accuracy gain

0.5 0.0

-0.5

0.0

-0.5

-1.0 -1.5 -2.0 -2.5 0.00

0.05

0.10

-1.0 -1.5 -2.0

0.15

0.20

0.25

-2.5 0.70

0.30

(a) Left BUMP (aligned)

0.75

0.80

0.85

0.90

0.95

1.00

(b) Right BUMP (aligned)

Figure 10. The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-100 test set, models are ResNet-20, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.

smaller (Figure 11). We report results for both methods quantifying accuracy gain of a BUMP in Table 17 (aligned pairs) and Table 18 (naı̈ve pairs). Table 17. Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs. Results are reported on the CIFAR-100 test set using aligned ResNet-20 pairs (50 models) trained to convergence (250 epochs). Standard deviations are denoted in parentheses.

Table 18. Average accuracy gains of the BUMPs over their corresponding endpoint models on non-aligned, i.e., naı̈ve model pairs. Results are reported on the CIFAR-100 test set using aligned ResNet-20 pairs (50 models) trained to convergence (250 epochs). Standard deviations are denoted in parentheses.

Accuracy gain % — Aligned pairs

Accuracy gain % — Naı̈ve pairs

Width 4x-width 8x-width 16x-width

Fixed-λ

Test

0.201 (±0.197) 0.226 (±0.184) 0.319 (±0.188)

0.254 (±0.218) 0.264 (±0.169) 0.366 (±0.173)

Width 4x-width 8x-width 16x-width

Fixed-λ

Test

0.092 (±0.289) −0.037 (±0.290) −0.007 (±0.218)

0.156 (±0.203) 0.140 (±0.154) 0.178 (±0.142)

C.2. A geometric and probabilistic analysis of model mixing In this section, we present the results for the experiments described in Section 7 for CIFAR-100 and ResNet-20. Figure 12 depicts the mean angles between the WeightMix and LogitMix logits. Figure 13a shows the accuracy curves, and Figure 13b shows the KL-divergences between the Weightmix and LogitMix models.

BUMP: Brief Unexpected Mid-Interpolation Peak

0.5

Accuracy gain

0.5 0.0

-0.5 -1.0 -1.5 -2.0 -2.5 0.00

0.0

-0.5

0.10

-1.0 -1.5 -2.0

0.15

0.20

0.25

-2.5 0.70

0.30

(a) Left BUMP (naı̈ve)

0.75

0.80

0.85

0.90

0.95

1.00

(b) Right BUMP (naı̈ve)

Figure 11. The two BUMPs when interpolating between models without alignment, the accuracies are measured on the CIFAR-100 test set, models are ResNet-20, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.

Angle (degrees)

Width multiplier 4 Width multiplier 8 Width multiplier 16

70 60 50 40 30 20 10 0 0.0

0.2

0.4

0.6

0.8

1.0

Figure 12. The mean angles across the test datapoints for g = 24, CIFAR-100, ResNet-20 (50 models).

BUMP: Brief Unexpected Mid-Interpolation Peak 1.0

DKL(E||Mweight mix) DKL(E||Mlogit mix)

KL Divergence

0.76 0.74

Accuracy

0.8

0.72

0.6

0.70

0.4

0.68 0.66

0.2

0.64

permuted weightmix permuted logitmix

0.62 0.0

0.2

0.4

0.0 0.6

0.8

1.0

0.0

0.2

(a) Accuracy curves

0.4

0.6

0.8

1.0

(b) KL divergences

Figure 13. Left (comparing accuracy curves): The WeightMix and LogitMix accuracy curve on CIFAR-100, ResNet-20, width multiplier is 16. Right (comparing class probabilities): Kullback-Leibler divergence between the class probability distributions of the WeightMix/LogitMix models and the λ = 0.5 LogitMix (classicle ensemble) model, CIFAR-100, ResNet-20.

D. Results for CIFAR-10 with Tiny-10 models In this section, we present the figures and tables for experiments on the CIFAR-10 dataset with Tiny-10 models. For each width multiplier in {4, 8, 16} we train 50 models, form 25 model pairs, and align them via the weight-matching algorithm of Ainsworth et al. (2023). We used a similary methodology to quantify accuracy gain as in Section 4, but with λ = 0.003 for 4x-width, λ = 0.01 for 8x-width, λ = 0.015 for 16x-width in the Fixed-λ method. D.1. The BUMPs for aligned and naı̈ve model pairs We present figures analogous to Figures 1 and 2, together with tables analogous to Tables 1, and 2. For the 16x-width setting, we plot interpolation curves for λ ∈ (0.0, 0.3) and λ ∈ (0.7, 1.0) across 10 model pairs. Figure 14 shows clear BUMPs near both endpoints for aligned model pairs, whereas for naı̈ve pairs the BUMP is absent or much smaller (Figure 15). We report results for both methods quantifying accuracy gain of a BUMP in Table 19 (aligned pairs) and Table 20 (naı̈ve pairs). Table 19. Average accuracy gains of the BUMPs over their corresponding endpoint models on aligned model pairs. Results are reported on the CIFAR-10 test set using aligned Tiny-10 pairs (50 models) trained to convergence (300 epochs). Standard deviations are denoted in parentheses.

Table 20. Average accuracy gains of the BUMPs over their corresponding endpoint models on non-aligned, i.e., naı̈ve model pairs. Results are reported on the CIFAR-10 test set using aligned Tiny-10 pairs (50 models) trained to convergence (300 epochs). Standard deviations are denoted in parentheses.

Accuracy gain % — Aligned pairs

Accuracy gain % — Naı̈ve pairs

Width 4x-width 8x-width 16x-width

Fixed-λ

Test

0.013 (±0.024) 0.040 (±0.054) 0.054 (±0.056)

0.106 (±0.122) 0.144 (±0.131) 0.178 (±0.115)

Width 4x-width 8x-width 16x-width

Fixed-λ

Test

0.027 (±0.038) 0.075 (±0.087) 0.050 (±0.059)

0.077 (±0.112) 0.112 (±0.133) 0.093 (±0.081)

D.2. A geometric and probabilistic analysis of model mixing In this section, we present the results for the experiments described in Section 7 for CIFAR-10 and Tiny-10. Figure 16 depicts the mean angles between the WeightMix and LogitMix logits. Figure 17a shows the accuracy curves, and Figure 17b shows the KL-divergences between the Weightmix and LogitMix models.

BUMP: Brief Unexpected Mid-Interpolation Peak

0.5

Accuracy gain

0.5 0.0

-0.5 -1.0 -1.5 -2.0 -2.5 0.00

0.0

-0.5

0.10

-1.0 -1.5 -2.0

0.15

0.20

0.25

-2.5 0.70

0.30

(a) Left BUMP (aligned)

0.75

0.80

0.85

0.90

0.95

1.00

(b) Right BUMP (aligned)

Figure 14. The two BUMPs when interpolating between aligned models, the accuracies are measured on the CIFAR-10 test set, models are Tiny-10, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.

0.5

Accuracy gain

0.5 0.0

-0.5 -1.0 -1.5 -2.0 -2.5 0.00

0.0

-0.5

0.10

-1.0 -1.5 -2.0

0.15

0.20

0.25

-2.5 0.70

0.30

(a) Left BUMP (naı̈ve)

0.75

0.80

0.85

0.90

0.95

1.00

(b) Right BUMP (naı̈ve)

Figure 15. The two BUMPs when interpolating between models without alignment, the accuracies are measured on the CIFAR-10 test set, models are Tiny-10, showing 10 model pairs. Width multiplier is 16. We shifted the plots so that all accuracies of the original models are 0.

BUMP: Brief Unexpected Mid-Interpolation Peak

Width multiplier 4 Width multiplier 8 Width multiplier 16

Angle (degrees)

80 70 60 50 40 30 20 10 0 0.0

0.2

0.4

0.6

0.8

1.0

Figure 16. The mean angles across the test datapoints for g = 24, measured on CIFAR-10, model is Tiny-10 (50 models).

0.40

KL Divergence

0.90

Accuracy

0.30 0.25

0.89

0.20

0.88

0.15 0.10

0.87 0.86

DKL(E||Mweight mix) DKL(E||Mlogit mix)

0.35

0.05

permuted weightmix permuted logitmix 0.0

0.2

0.4

0.00 0.6

0.8

1.0

0.0

(a) Accuracy curves

0.2

0.4

0.6

0.8

1.0

(b) KL divergences

Figure 17. Left (comparing accuracy curves): The WeightMix and LogitMix accuracy curve on CIFAR-10 with model Tiny-10, width multiplier is 16. Right (comparing class probabilities): Kullback-Leibler divergence between the class probability distributions of the WeightMix/LogitMix models and the λ = 0.5 LogitMix (classical logit ensemble) model, dataset is CIFAR-10, model is Tiny-10, width multiplier is 16.

BUMP: Brief Unexpected Mid-Interpolation Peak

Analyze & explore related research in Conceptio

Related documents