Conceptio › Archive › arXiv CS

arXiv CSopen access

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

Nghia Bui 1 Lijing Wang 1 1.0

Successful

0.9

Fine-tuning pretrained models has become a standard approach to adapting pretrained knowledge to improve the accuracy on new sparse, imbalance datasets. However, issues arise when optimization falls into a collapsed state, where the model gets stuck, leading to degraded performance and unstable training. One possible reason for this is the cancellation of gradients across training examples. To address this problem, we propose a novel algorithm, dynamic scaled gradient descent (DSGD), that directly modifies the gradients returned by training examples, specifically, scaling down the gradients of correctly classified examples using a dynamic scaler. This strategy offers both theoretical and empirical advantages in improving training stability. Experiments on a variety of benchmark datasets, spanning multiple tasks and large pretrained models, demonstrate that our method consistently reduces performance variance and surpasses the accuracy of existing approaches.

Accuracy

arXiv:2604.27987v1 [cs.LG] 30 Apr 2026

Abstract

Degenerated

Collapsed

0.8 0.7 0.6 0.5 COPA

MultiRC

CoLA

RTE

I-CIFAR100

Figure 1. Training failures (either collapsed state or degenerate solution) on various tasks when fine-tuning pretrained models across 10 random seeds. Each task uses the same hyperparameters, with only the random seed varying between runs.

training runs, a process that is both computationally prohibitive and impractical for real-time applications. Training stability is a well-documented challenge, particularly during fine-tuning, where pretrained representations become sensitive to factors such as weight initialization, data ordering, and hyperparameter choices. Prior work has studied this phenomenon in NLP (Mosbach et al., 2020; Zhang et al., 2020; Dodge et al., 2020; Phang et al., 2018; Wang et al., 2023; Nishida et al., 2025; Hidey et al., 2022; Bui et al., 2025) and computer vision (Summers & Dinneen, 2021; Pecher et al., 2024), attributing instability to distribution shifts between pretraining and downstream data, as well as optimization nondeterminism in deep learning pipelines (Picard, 2021; Hardt et al., 2016; Mosbach, 2023; Zhang et al., 2020).

1. Introduction The blossom of deep learning in recent years has led to the success of large pretrained models (LPMs), such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), LLaMA (Touvron et al., 2023) in natural language processing (NLP) and ViT in computer vision (Dosovitskiy, 2020), where they can be finetuned for a wide range of downstream tasks with remarkable performance. Despite their success, fine-tuning LPMs often suffers from high sensitivity to random seeds (Dodge et al., 2020; Mosbach et al., 2020; Zhang et al., 2020; Cao et al., 2019), a phenomenon known as training instability, causing significant performance variation across runs (Bethard, 2022; Hua et al., 2021; Wang et al., 2023; Bui et al., 2025), as shown in Figure 1. As a result, obtaining reliable performance requires multiple

To reduce performance variation induced by random seeds, few methods have been proposed. While ensembles and their variations (Wang et al., 2023; 2020; Nishida et al., 2025; Izmailov et al., 2018; Nishida et al., 2025) are theoretically grounded and often the most effective at lowering variance, they are computationally (either training time or storage) prohibitive for large-scale deployment. Datacentric solutions like increasing dataset size (Dodge et al., 2020) are often impractical, and optimization guidelines such as extended training (Mosbach et al., 2020) still result in high variance. Noise-based techniques (Hua et al., 2021; Wu et al., 2022) typically seek to improve generalization and stability of LPMs in fine-tuning by injecting noise but lack a targeted mechanism to directly stabilize the fine-tuning optimization process and can increase divergence. Consequently, stabilizing fine-tuning of LPMs remains largely a

Department of Data Science, New Jersey Institute of Technology, New Jersey, USA. Correspondence to: Lijing Wang <[email protected]>. Preprint. May 1, 2026.

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

process of trial and error, with the core issue of seed-induced variance still unresolved.

result in either a high-performing solution, a degenerate solution, or a collapsed state, depending on the optimization trajectory and loss landscape.

In this work, we attribute fine-tuning instability to gradient conflicts across training examples. Under certain random seeds, conflicting gradients arising from parameter initialization or data ordering may cancel each other, causing premature convergence even when training error remains high. Consequently, a model fails to learn meaningful patterns, resulting in drastically different outputs across runs. Although no prior work has been directly working on solving gradient cancellation across training examples, the gradient conflict resolvers from multitask learning (Sener & Koltun, 2018; Yu et al., 2020; Liu et al., 2021; Chen et al., 2020) were proposed to reduce inter-task gradient conflict through projection or weighting schemes; however, these methods require expensive separate backward passes and rely on detecting significant negative cosine similarity between gradients, failing when gradients are orthogonal or small-magnitude.

Definition 2.2. Degenerate Solution: A suboptimal outcome of optimization where the model produces varied predictions but fails to learn meaningful decision boundaries, resulting in poor generalization performance despite non-trivial predictive behavior. Definition 2.3. Collapsed State: A pathological failure mode of optimization where the model’s predictions saturate to a single class label for all inputs, representing a severe form of model collapse with zero predictive variance. Definition 2.4. Gradient Cancellation: A phenomenon occurring during optimization where the expected gradient of training examples within a batch can be approximately zero, leading to no effective update.

We propose a Dynamic Scaled Gradient Descent (DSGD) algorithm for classification that explicitly targets optimization instability during fine-tuning. We adaptively scale gradients from correctly classified examples to alleviate gradient cancellation, leading to improved convergence and fine-tuning stability under seed-induced randomness. We also provide theoretical guarantee and empirical evidence to validate the proposed algorithm. DSGD is a drop-in replacement for standard optimizers: no additional backward passes, no architectural changes, no task-specific tuning, and negligible overhead. To our knowledge, DSGD is the first method to provably and empirically avoid seed-induced collapsed failure during gradient-based learning.

Phenomenon: As shown in Figure 1, when fine-tuning a LPM, certain random seeds can lead to a training failure, either a collapsed state (red crosses) or a degenerate solution (blue crosses). This sensitivity results in significant performance variance across different runs. Underlying phenomenon: To investigate the reasons for a training failure, we compute the pairwise cosine similarity matrix of [CLS] token representations from a random subset of training examples after fine-tuning. The results are shown in Figure 2a for a collapsed state and 2b for a successful run. We can see that in a collapsed state, the [CLS] representations lose their class-discriminative structure and converge to nearly identical vectors, resulting in uniformly high similarity scores. This indicates a failure to learn meaningful feature separation.

Major Contributions: (1) identifies gradient conflicts between correctly and wrongly classified examples as the key cause of a failed run during fine-tuning; (2) proposes DSGD, a lightweight and proactive optimization algorithm that dynamically downscales the gradient contributions of correctly classified examples within each mini-batch to mitigate gradient conflicts; (3) provides a formal theoretical motivation for DSGD via gradient decomposition; (4) proves DSGD ensures convergence to better stationary points and a tighter stability upper bound than standard gradient descent, formally guaranteeing improved accuracy and stability; (5) demonstrates consistent and significant improvements in accuracy and stability over baselines across 14 diverse NLP and vision tasks, confirming broad applicability.

Furthermore, in Figure 2c and 2d, we analyze the gradient norm (blue curve) of the within-batch training examples and the cosine similarity (green curve) between the gradients of correctly and wrongly classified examples. Consistent with the observation of (Mosbach et al., 2020) that gradient norms are markedly lower in failed runs, our results show a similar trend (i.e., close to 0 in most iterations). This phenomenon can be attributed to gradient cancellation. Cosine similarity reveals substantial gradient conflicts in the failed run, where 90% of iterations yield negative values and over 30% fall below -0.5 (as low as -0.85). In contrast, successful run exhibits milder conflicts, with most similarities above -0.25 and only a few below -0.5. This consistent opposition in failed runs results in gradient cancellation during the update step of parameters. This widespread cancellation in the early training can lead to a collapsed state or a degenerate solution.

2. Phenomenon Definition 2.1. Convergence: The state during training when model parameters stabilize, with the loss function exhibiting negligible further decrease. Convergence may

0.9975 0 1 1 0 0 1 1 0 1 0

0.9970

Label

0 1 1 0 0 1 1 0 0 1

Label

(a) Collapsed state.

(b) Successful run.

1.0 0.8 0.6 0.4 0.2

100

0.0

1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00

Cosine Sim Grad Norm

0.8 0.6 0.4 0.2

Number of Iteration

1.0

Gradient Norm

0.9980

Cosine Sim Grad Norm

Cosine Similarity

0.9985

1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00

Gradient Norm

Label

0.9990

1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00

Cosine Similarity

0.9995

Label

0 1 0 1 1 0 0 1 1 0

1.0000

1 0 0 1 1 0 0 1 1 0

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

100

0.0

Number of Iteration

(d) Successful run.

Figure 2. (a) and (b) are cross-cosine similarity of [CLS] token representations for 10 randomly selected examples (5 per class) when fine-tuning RoBERTa-large on COPA (a binary classification task), in a collapsed state (random seed 132) and a successful run (random seed 42), respectively. Axes list the corresponding labels. Representations in collapsed states become uniformly similar, while successful runs show higher within-class than between-class similarity. (c) and (d) show gradient norms of all training examples and cosine similarities between gradients of correctly and incorrectly classified training examples within a batch in a failed run and successful run respectively, measured by mean of 10 randomly selected dense layers over training iterations.

3. Proposed Solution: DSGD

where GW and GC are the total gradients from BW and BC respectively. The dynamic scaling factor γt ∈ [0, τ ] is defined as a function: γt = ζ(t, T, τ ), where t denotes the current training iteration, T is the total number of training iterations, and τ ≤ 1 is a hyperparameter that controls the maximum allowed scaling. A smaller τ imposes a stricter upper bound, thereby more aggressively downscaling the gradient contribution from correctly classified examples. Various γt functions are implemented in Section 5.1.

Motivation: We consider a learning in the context of minibatch training, where within a batch the distribution of classes of examples can be highly imbalanced. This frequently occurs either due to inherent dataset imbalance or simply the randomness of mini-batch sampling (Dodge et al., 2020). In such a scenario, models can collapse to a trivial solution where predictions are dominated by the majority class within the batch. In this collapsed state, gradients from the correctly classified examples form a dominant signal that overwhelms and cancels out the crucial error signal from misclassified minority examples. This prevents the model from learning meaningful features for minority classes. This will repeat for each batch training leading to the model trapped in the collapsed state. To prevent this, we must break the gradient cancellation, especially early in training.

4. Theoretical Foundations In this section, we analyze the theoretical properties of DSGD. We prove that, under mild assumptions, our method ensures that the optimization process does not stall at poor solutions but instead converges to a more desirable stationary point. Moreover, we prove that our method provides a tighter stability bound than standard gradient descent (GD). Proofs of the theorems in this section can be found in Appendix D.

Solution: Assume that we are given N data points in the training dataset S: z1 , z2 , . . . zN where zi = (xi , yi ) drawn i.i.d from an unknown distribution D, where xi ∈ X is an input and yi ∈ Y is the corresponding one-hot encoded label. Let fθ : X → Y be a model parameterized by θ ∈ Θ, and let L(θ, z) be the loss function. Let θt denote model parameters at training iteration t, η the learning rate, and B ⊂ D a batch of m examples. The standard gradient 1 descent (GD) update at iteration t is θt+1 = θt − η m G, Pm where G = i=1 ∇θ Li denotes the total gradient over the entire batch. We define BW and BC as the subset of wrongly and correctly classified examples in B respectively. We introduce a Dynamical Scaled Gradient Descent (DSGD) algorithm that reduces the influence of correctly classified examples within each batch. Our modified update rule is as follows:

θt+1 = θt − η

1 (GW + γt GC ) , m

4.1. Theoretical Motivation Consider a single layer neural network, let pi = σ(oi ) where oi = ωxi + b denote the output of the model, where θ = {ω, b} are learnable parameters and σ(·) is the softmax function. L is the linear combination loss decomposed as: L = LC + LW ,

(2)

where LC and LW denote the loss of BC and BW , respectively. Assume the model parameter is updated using stochastic GD (SGD) with batch size m = 1 (Hardt et al., 2016) and S is linear separable. We begin by establishing key properties of the two loss components, LW and LC . Definition 4.1 (Zero Training Error). A classifier parametrized by θ is said to attain zero training error on S = {(xi , yi )}N i=1 if ∀i, arg max{yi } = arg max{pi }.

(1) 3

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

Lemma 4.2 (Property of LW ). If gradient-based optimization converges to a stationary point of LW , then the resulting classifier attains zero training error (which does not require zero training loss). Consequently, optimizing LW leads to a parameter θ∗ for which every training example is correctly classified.

Theorem 4.5 (Tighter stability bound of DSGD). Assume L is H-Lipschitz. Let B = {B1 , B2 , . . . , BT } and B̃ = {B̃1 , B̃2 , . . . , B̃T } be two mini-batch sequences to update the learnable parameters. Let θT and θ̃T denote the parameters obtained after T steps using UDSGD on B and B̃, respectively, and similarly let ψT and ψ̃T denote those obtained using USGD . Initialization is identical across update rules: θ0 = ψ0 and θ̃0 = ψ̃0 . Let ϵ and ϵ′ denotes the stability bound for DSGD and SGD, i.e., ∀z ∼ D, ϵ ≥ E[|L(θT , z) − L(θ̃T , z)|] and ϵ′ ≥ E[|L(ψT , z) − L(ψ̃T , z)|]. Then the stability bound of DSGD is tighter than that of SGD, i.e., ϵ ≤ ϵ′ .

Lemma 4.3 (Property of LC ). Convergence to a stationary point of LC does not guarantee zero training error. The resulting classifier may misclassify some training examples. In lemma 4.2 and 4.3 We can rewrite LC and LW in term of weighted sums of individual training point loss to ensure the continuity over the training data partitions regardless they are empty or not.

Based on the proof of Theorem 4.5 in Appendix D.6, divergence is bounded by a function of the Lipschitz constant H, learning rate η, and the total training iteration T . While a smaller η tightens the stability bound, it also slows convergence; a smaller T usually results in poor predictive performance due to under-training. Our method DSGD improves stability without sacrificing convergence speed and accuracy.

Remark: The arguments in Lemma 4.2 and Lemma 4.3 depend only on per-example first-order optimality and thus extend straightforwardly to multi-layer neural networks. Corollary 4.4 (Motivation for Gradient Modulation). Since optimizing LW guarantees zero training error (Lemma 4.2) while optimizing LC does not (Lemma 4.3), it is theoretically preferable to retain the full gradient information from LW while selectively modulating the influence of gradients from LC .

4.3. Convergence Analysis Theorem 4.6 (Convergence of DSGD). Assume that BW and BC are not empty and LW is M -smooth. Let {θt } be 1 generated by DSGD update rule θt+1 = θt − η m (GW + γt GC ) with η > 0. If (1) 0 < γt ̸= γt+1 ≤ τ ; (2) m ∥GW ∥2 ≥ τ 2 ∥GC ∥2 for all t; and (3) 0 < η < M , then LW will strictly decrease every 2 consecutive iterations under the new update rule or L and LW will converge to stationary points.

Based on Corollary 4.4, we propose DSGD to prioritize LW while controllably incorporating LC . 4.2. Stability Analysis We analyze the model’s stability under variations in initialization and stochastic mini-batch ordering induced by random seeds. Let θ and θ̃ represent model parameters trained from two different random initializations θ0 ̸= θ̃0 , using the same training dataset S but presented in a different stochastic order (i.e., sampled differently per iteration). The stability of a learning algorithm with respect to the random seed is quantified by the expected divergence: Ez∼D |L(θ, z) − L(θ̃, z)| , (3)

Note that we schedule γ dynamically hence γt ̸= γt+1 moreover ∥GW ∥2 ≥ τ 2 ∥GC ∥2 is a mild condition, since theoretically τ can be taken arbitrarily small. In our paper, we conduct sensitivity analysis on τ and discuss the results in Section 5.5. According to Theorem 4.6, DSGD provides a provable escape mechanism from the collapsed state, guaranteeing progress where standard GD would stall. Remark: When BW or BC is empty, DSGD is equivalent to GD and converges at a rate of O(1/T ). From the proof of Theorem D.7, DSGD stops updating only when ∥GW ∥ = ∥GC ∥ = 0, indicating convergence to a stationary point. In contrast, GD stops when ∥GW + GC ∥ = 0 indicating either a convergence or a gradient cancellation (∥GW ∥ > 0 and ∥GC ∥ > 0) which makes GD learn ineffectively.

A smaller divergence indicates greater robustness to randomness in initialization and mini-batch ordering. We aim to prove that the stability of DSGD has a tighter upper bound than SGD. We consider general update rules of the form U : Θ → Θ that map a point θ ∈ Θ in the parameter space to another point U (θ). Let UDSGD and USGD denote update rules of DSGD and SGD respectively, i.e., USGD (θ) = 1 1 θ −η m G and UDSGD (θ) = θ −η m (GW +γGC ). In this paper, we compare UDSGD and USGD in terms of stability, i.e., Ez∼D [|L(θ, z) − L(θ̃, z)|] and Ez∼D [|L(ψ, z) − L(ψ̃, z)|] where θ is optimized by using DSGD and ψ is updated using SGD. With a fixed random seed and model architecture, initialization is identical across update rules, i.e., θ0 = ψ0 and θ̃0 = ψ̃0 .

5. Experiments We conducted our experiments on various tasks including sequence and image classifications to show the effectiveness of our approach. Additional results on multi-task learning and more analysis can be found in Appendix F. Key experi4

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications MultiRC

COPA

RTE

BoolQ

MRPC

CoLA

ACC(↑)

STD(↓)

FFT FocalLoss LNSR NoisyTune PCGrad DSGD

74.05 ± 14.50 73.38 ± 5.73 66.07 ± 11.46 63.62 ± 10.72 62.03 ± 8.70 84.01 ± 1.31

74.70 ± 15.71 80.40 ± 7.59 82.00 ± 12.58 *55.00 ± 0.00* 69.60 ±19.11 88.10 ± 3.21

76.17 ± 15.44 77.29 ± 1.70 77.44 ± 15.91 70.65 ± 15.53 81.05 ± 10.08 83.61 ± 1.59

83.20 ± 7.39 81.22 ± 0.46 73.41 ± 11.86 71.04 ± 11.46 83.22 ± 7.40 85.69 ± 0.20

84.90 ± 5.33 86.52 ± 6.50 88.36 ± 1.55 86.91 ± 6.62 86.67 ± 6.52 88.38 ± 1.17

78.95 ± 8.46 83.82 ± 5.20 80.75 ± 8.03 80.67 ± 8.01 81.80 ± 6.84 84.92 ± 1.05

78.66 81.10 78.00 71.31 77.39 85.78

11.13 4.53 10.23 8.72 9.77 1.42

SWA ENS (×3) ENS (×5) ENS (×7) ENS (×9) DSGD (×9)

65.92 ± 11.27 82.77 ± 8.99 80.19 ± 12.12 85.72 ± 0.81 86.11 ± 0.23 85.44 ± 0.16

87.70 ± 4.06 82.90 ± 4.72 84.80 ± 3.43 85.10 ± 2.13 86.40 ± 1.17 90.80 ± 0.79

78.45 ± 13.61 79.86 ± 3.01 80.51 ± 2.58 79.89 ± 1.92 81.16 ± 1.06 84.77 ± 0.41

80.36 ± 9.61 85.86 ± 0.35 86.13 ± 0.14 86.20 ± 0.15 86.31 ± 0.08 86.57 ± 0.08

88.19 ± 1.21 86.74 ± 1.40 87.25 ± 0.90 87.82 ± 0.50 88.04 ± 0.32 88.53 ± 0.28

82.46 ± 7.06 78.24 ± 7.86 79.76 ± 7.36 80.77 ± 6.17 83.54 ± 0.75 86.18 ± 0.21

80.51 82.82 82.79 84.85 85.26 87.04

7.80 4.35 4.71 1.94 0.60 0.32

MultiRC

COPA

RTE

BoolQ

MRPC

CoLA

ACC(↑)

STD(↓)

FFT FocalLoss LNSR NoisyTune PCGrad DSGD

62.50 ± 0.80 56.94 ± 0.49 65.33 ± 4.46 60.85 ± 0.93 60.76 ± 3.24 67.17 ± 4.70

79.40 ± 6.08 76.00 ± 7.72 82.10 ± 3.14 *55.00 ± 0.00* 76.40 ± 2.95 83.30 ± 2.06

79.49 ± 1.83 49.39 ± 2.97 74.90 ± 2.70 74.69 ± 2.92 73.83 ± 2.95 81.55 ± 1.22

73.98 ± 3.85 *62.17 ± 0.00* 66.81 ± 1.24 67.64 ± 1.01 67.77 ± 1.21 75.43 ± 3.55

86.69 ± 1.05 *68.38 ± 0.00* 82.89 ± 3.15 84.76 ± 2.08 82.84 ± 3.39 87.52 ± 0.82

84.07 ± 0.83 *69.13 ± 0.00* 78.67 ± 3.65 80.91 ± 0.44 80.09 ± 1.62 84.75 ± 0.59

77.69 63.66 75.12 70.64 73.61 79.95

2.41 1.86 3.06 1.23 2.56 2.16

SWA ENS (×3) ENS (×5) ENS (×7) ENS (×9) DSGD (×9)

62.53 ± 1.01 63.44 ± 0.42 64.10 ± 0.33 63.79 ± 0.32 64.20 ± 0.16 69.11 ± 0.88

55.50 ± 5.51 81.10 ± 4.79 77.60 ± 9.09 81.30 ± 7.18 83.70 ± 3.56 83.90 ± 3.28

76.17 ± 1.47 81.91 ± 1.06 82.78 ± 0.83 83.14 ± 0.30 82.96 ± 0.28 83.54 ± 0.42

67.29 ± 1.64 76.12 ± 2.90 76.52 ± 1.76 76.57 ± 0.72 76.64 ± 0.55 78.09 ± 0.19

82.62 ± 3.79 87.65 ± 0.83 87.75 ± 0.66 88.48 ± 0.54 88.26 ± 0.69 89.26± 0.34

79.95 ± 2.02 84.94 ± 0.76 85.25 ± 0.34 85.11 ± 0.33 85.30 ± 0.23 85.60 ± 0.24

70.67 79.20 79.00 79.73 80.18 81.58

2.57 1.79 2.17 1.56 0.91 0.89

RoBERTa-large

Llama-3.2-1B

Table 1. Comparison of DSGD with baselines on 6 sequence classification benchmark datasets, evaluated using ACC (%) and STD. Results are averaged over 10 runs, ∗∗ indicates all collapsed states. The last two columns show the average ACC and STD of 6 NLP tasks. The best performance (based on accuracy) is in bold, and the second best is underlined.

mental settings are described below. More details, including but not limited to dataset statistics, training settings, and baseline implementation, can be found in Appendix E.

fine-tuning variance on GLUE and SuperGLUE benchmarks in our preliminary experiment and previous work (Mosbach et al., 2020; Wang et al., 2023). Baselines. We compare DSGD against (1) fully fine-tuning (FFT) using the training guidelines from (Mosbach et al., 2020; Zhang et al., 2020); (2) FFT with focal loss (Lin et al., 2017) (FocalLoss); noise injection approaches: (3) LNSR (Hua et al., 2021) and (4) NoisyTune (Wu et al., 2022); gradient conflict resolver: (5) PCGrad (Yu et al., 2020); ensemble methods: (6) bagging ensemble of N single learners based on majority voting (ENS (×N )); (7) stochastic weight averaging (SWA) (Izmailov et al., 2018).

5.1. Implementation Dynamic Scaler γ. We implemented a linear scheduler: γt = ζ(t, T, τ ) =

t τ, T

(4)

with τ = 1. In the sensitivity analysis (Section 5.5), we compared this scheduler with two alternatives: a descending t linear function γ = τ 1 − t T and a cosine scheduler γt = t τ 1 − cos π , along with various static γ values and 2 T different choices of τ .

Overall Performance. In Table 1, we report ACC and STD across 10 random seeds for 6 NLP tasks. Compared to single-learner baselines, DSGD consistently achieves substantially higher ACC and lower STD across task-model pairs. Using RoBERTa-large, DSGD achieves an average +7.12% improvement in ACC (78.66% vs. 85.78%) and an average -9.71 reduction in STD (11.13 vs. 1.42) over FFT. Except FocalLoss which we will discuss separately, all other baselines underperform FFT in average ACC while keep similar high STD. For instance, on MultiRC, DSGD achieved 84.01% ACC with 1.31 STD, significantly outperforming FFT (74.05% ACC with 14.50 STD), while other baselines (PCGrad 62.03% ± 8.70, LNSR 66.07%

Metrics. Accuracy (ACC) and standard deviation (STD) are used as evaluation metrics, with the mean accuracy and variance derived from 10 arbitrary random seeds. 5.2. Sequence Classification Datasets and Models. We conducted experiments on six tasks (MultiRC, COPA, RTE and BoolQ from SuperGLUE (Wang et al., 2019); and MRPC and CoLA from GLUE (Wang et al., 2018)) using encoder-only Roberta-large (Liu et al., 2019) and decoder-only Llama-3.2-1B (Touvron et al., 2023), both show high 5

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

± 11.46, and NoisyTune 63.62% ± 10.72) failed to learn an effective discriminator while keeping similar high variance. Similar trends were observed in RTE and BoolQ by Llama-3.2-1B.

to report the best performance in Table 1. Its inconsistent performance across tasks suggest that tuning is non-trivial. DSGD introduces a task-agnostic hyperparameter τ that generalizes well across all task-model pairs in our experiments.

The noise injection approaches, LNSR and NoisyTune, were not able to mitigate the seed-induced instability while also hurting accuracy compared to FFT. The gap was most evident on MultiRC, RTE, and BoolQ with RoBERTa-large, and RTE, BoolQ, MRPC, and CoLA with Llama-3.2-1B. One possible reason is that noise injection adds randomness everywhere, which fails to stabilize the sensitive parameter directions that cause divergence and may even increase it by creating more varied starting points. The added noise may destroy some of the transferrable knowledge leading to reduced accuracy. Instead, our method DSGD ensures escape from the collapsed state regardless of the starting points, leading to consistently improved accuracy and stability.

Ensemble Performance. Despite their high computational cost, ensembling techniques are considered as the most effective methods to mitigate random failures in the deep learning domain as they can improve the average accuracy and consistency with theoretical guarantee (Wang et al., 2020; 2023; Summers & Dinneen, 2021; Nishida et al., 2025). To compare our method with ensemble, we applied simple bagging ensemble of FFT in size 3, 5, 7, and 9, as well as an ensemble of DSGD in size 9, and show our results in Table 1. We observe that it exhibits increasing improvements in accuracy and variance as the ensemble size grows. However, with RoBERTa-large a single DSGD consistently outperforms most ensemble sizes across 6 tasks, achieving an average accuracy of 85.78% with 1.42 STD, surpassing even the largest ensemble ENS (×9) at 85.26% with 0.6 STD. With Llama-3.2-1B, DSGD also outperforms the strong ensemble configuration ENS (×7) in terms of average accuracy. In addition, the ensemble of DSGD performs the best across various tasks and models compared to the ensemble of FFT.

PCGrad was proposed to solve gradient conflict in multitask learning which we adapted to our problem. However, there are still collapsed states observed despite its lower accuracy compared to FFT in most tasks (see Figure 9,10,11). Specifically, by RoBERTa-large, the performance got reduced significantly on MultiRC (62.03%) and COPA (69.06%) compared with FFT (74.05% and 74.70%). Although the ACC got improved for the other four tasks, the STD still kept high. By Llama-3.2-1B, the ACC of all tasks degraded using PCGrad with increased STD, indicating that PCGrad failed to improve either accuracy or stability. The reason could be that PCGrad only acts when gradients have significant negative cosine similarity, but a collapsed state can happen even with orthogonal gradients or small-magnitude gradients. In contrast, DSGD actively adjusts gradients based on learning outcomes that do not depend on the detection of actual gradient cancellation, making it a more robust and proactive defense against task collapse and seedinduced instability.

Wang et al. (2023) shows that the variance of an LPM can be decomposed into optimization variance and sampling variance. In over-parameterized settings, optimization variance vanishes as the ensemble size increases while sampling variance remains unchanged, implying an upper bound on variance reduction that strongly depends on ensemble size. This is consistent with our empirical results. For instance, with RoBERTa-large, we observe no failed runs only once the ensemble size exceeds task-specific thresholds (e.g., 5 for MultiRC, 3 for MRPC, and 7 for CoLA), with similar behavior observed for Llama-3.2-1B. This threshold is the minimum ensemble size needed to avoid failed runs and is related to the failure rate of individual components on each task. However, because ensemble components are sampled randomly, there is no deterministic rule to guarantee failure-free ensembles purely based on size. The threshold should therefore be interpreted as an empirical estimate rather than a strict guarantee. In addition, an ensemble classifier may not necessarily be better than the performance of the best component (Figure 8). The ”best” FFT component can itself be a failure run selected by random chance, which caps the ensemble’s potential, and if all components fail, the ensemble fails as well. In contrast, DSGD as a single learner is free of these issues. More importantly, compared to an ensemble of size N , our method requires only 1/N training time of the ensemble. More computational cost analysis is discussed in Section 5.4.

FocalLoss performs well using RoBERTa-large (81% ACC 4.53 STD on average) but also shows high instability on COPA (7.59 STD) and MRPC (6.5 STD). However, the best performance across 10 random runs is relatively low compared to other baselines (see Figure 9). It also shows universal training failures across diverse benchmarks when using Llama-3.2-1B (e.g., BoolQ, MRPC, and CoLA). This happens because FocalLoss reshapes the gradient field via confidence-dependent curvature (Section E.3), effectively suppressing gradients from high-confidence examples. As training progresses and most examples become high-confidence, this reduces the effective batch size and leads to unstable updates. A similar effect is observed for DSGD with very small τ ≈ 0. See Section 5.5 for more discussion. Furthermore, FocalLoss relies on a task-sensitive hyperparameter which we have performed exhaustive search

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

Accuracy

Beyong bagging ensemble, we evaluated SWA which averages nearby checkpoints within a local optimization basin. SWA fails to stabilize performance when the model converges to a collapsed state, as nearby checkpoints remain trapped in the same suboptimal solution. Empirically, SWA fails to mitigate instability in several NLP tasks, such as exhibiting high variances of 11.27 on MultiRC, 13.61 on RTE, 9.61 on BoolQ using RoBERTa-large.

0.85 0.80 0.75 0.70 0.65 0.60

DSGD DSGD x9 FFT FFT x3 2000

4000

FFT x5 FFT x7 FFT x9 FocalLoss 6000

Training Time Per Epoch (s)

LNSR NoisyTune PCGrad SWA 8000

Figure 3. Training time per epoch of each methods on MultiRC. Size of the dot denotes standard deviation.

5.3. Image Classification To assess DSGD for image classification, we design our experiments with the following configurations and provide justification of our settings in Appendix Section E.5.

large-scale Llama-3.2-1B. PCGrad requires 2 backward passes to compute the gradient of correctly and wrongly classified groups separately hence the training time is nearly doubled. Bagging ensemble achieves both better accuracy and variance at the expense of linearly increasing computational cost and storage space with ensemble size, while SWA requires extra storage space to save nearby checkpoints yet delivers inconsistent performance gains compared to FFT. For some tasks, such as RTE and MRPC, DSGD outperforms even the largest ensemble (ENS (×9)) in accuracy while matching its low standard deviation, yet requires no extra time or storage, achieving the best overall performance, similar trends are observed across other tasks.

Datasets and Models and Baselines. Following (Cao et al., 2019), we created imbalanced versions of CIFAR-10 and CIFAR-100 by reducing training samples per class while preserving the original validation sets. We consider two imbalance ratios (100:1 and 50:1) and two imbalance patterns: long-tailed (exponential decay) (Cui et al., 2019) and step imbalance (Azizzadenesheli et al., 2019), resulting in four distinct datasets per benchmark. We used Vision Transformer-based model ViT-base (Dosovitskiy, 2020) as the pretrained model and adapted sequence classification baselines for image classification tasks. Overall Performance. Table 2 presents single-learner results across eight datasets. On the more challenging 100:1 imbalance ratio, PCGrad fails to improve model performance in terms of both accuracy and variance, FocalLoss exhibits substantially high variance on CIFAR10:step (14.22 STD) and CIFAR-100:long-tailed (10.70 STD), while LNSR and NoisyTune yield inconsistent results that improving some tasks but degrading others. In contrast, DSGD consistently enhances both accuracy and stability over FFT in a large margin, outperforming all baselines. With the easier 50:1 imbalance, FFT already yields strong accuracy and low STD. Most baselines still provide improvements, except NoisyTune and FocalLoss on CIFAR10:long-tailed which underperform FFT. DSGD achieves the best or the second best performance in most tasks. Overall, existing baselines show inconsistent gains over FFT, whereas DSGD delivers consistent improvements across tasks, achieving superior accuracy and stability.

5.5. Sensitivity Analysis and Discussion In this section we analyze multiple aspects of DSGD using RoBERTa-large on COPA. Scaler γ. The superior of dynamic scaler is theoretically proved in Theorem 4.6 and 4.5. In our experiment, a linearly ascending scaler consistently outperforms existing methods in both accuracy and stability. To further explore the influence of the scaling strategy, we compare static scalers γ ∈ {0.1, 0.3, 0.5, 0.7, 0.9} with two dynamic alternatives: a linear descending function and a cosine scheduler. The results shown in Figure 4a show that compared to dynamic scalers, using a static one exhibits larger performance variations and we can observe collapsed states for γ above 0.1. While γ = 0.1 avoids collapse, it achieves lower average accuracy. On the other hand, dynamic scalers show consistently improved accuracy and stability over FFT, i.e., no collapsed states. Among them, the linearly ascending scaler and cosine scaler perform similarly best. These findings align with the experimental results in Section 5 and the theoretical analysis in Section 4.

5.4. Computational Cost Figure 3 compares accuracy, standard deviation, and training time across all methods using RoBERTa-large on MultiRC. We observe that although baseline methods can mitigate instability for some tasks, they come with significant trade-offs in efficiency, performance, or both. Single-learner baselines like LNSR, NoisyTune, and FocalLoss require no additional time or storage compared to FFT, yet they underperform in both accuracy and variance, especially using

Dynamic Scaler Upper Bound τ . The hyperparameter τ , introduced in Section 3, governs the maximum scaling amplitude; lower values produce a more aggressive downscaling effect. To investigate the impact of τ , we vary τ ∈ {0, 0.1, 0.3, 0.5, 1}. As shown in Figure 4b, larger τ yields better mean accuracy and lower variance. With any τ ≤ 1, the performance is consistently higher and more 7

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications ViT-base Ratio FFT FocalLoss LNSR NoisyTune PCGrad DSGD

CIFAR-10: long-tailed 100 50 88.47 ± 1.77 89.32 ± 0.68 88.05 ± 1.58 87.88 ± 1.67 86.63 ± 2.44 89.23 ± 0.95

CIFAR-10: step 100 50

96.46 ± 0.44 95.29 ± 3.58 97.27 ± 0.14 88.94 ± 9.39 96.85 ± 0.19 97.15 ± 0.17

83.27 ± 2.13 70.31 ± 14.22 84.03 ± 0.98 68.86 ± 15.40 82.05 ± 3.57 84.18 ± 0.66

CIFAR-100: long-tailed 100 50

96.09 ± 0.51 97.15 ± 0.57 96.90 ± 0.25 96.80 ± 0.24 96.84 ± 0.15 96.95 ± 0.42

69.55 ± 2.71 64.68 ± 10.70 64.90 ± 1.29 68.05 ± 3.75 68.13 ± 1.47 70.33 ± 1.18

CIFAR-100: step 100 50

85.53 ± 0.48 87.20 ± 2.21 87.90 ± 0.28 87.89 ± 0.24 87.02 ± 0.37 87.72 ± 0.31

56.73 ± 1.58 57.19 ± 2.49 57.26 ± 0.78 57.17 ± 1.11 50.01 ± 1.18 59.91 ± 0.44

84.30 ± 0.48 86.60 ± 4.17 87.72 ± 0.22 87.89 ± 0.24 87.24 ± 0.37 87.93 ± 0.37

Table 2. Comparison with baselines on CIFAR-10 and CIFAR-100 with various imbalance settings.

0.7 0.6 0.5 0.4

0.8

Accuracy

0.8

0.7 0.6

(a) Various scaler γ functions.

0.8 0.7 0.6

LoRA FFT IA3

0.5

0.5 0.1 0.3 0.5 0.7 0.9 1.0 Asc Dsc Cos

1.0

0.9

0.0

0.1

0.3

0.5

1.0

FFT

(b) Various τ settings.

Epoch

AdaLoRA Ours 8

Gradient Norm

0.9

DSGD grad norm Failed run grad Norm

0.8 0.6 0.4 0.2 0.0

100

Number of Iteration

120

(d) Gradient comparison.

Figure 4. Sensitivity analysis conducted using RoBERTa-large on COPA dataset.

stable than FFT (γ = 1). This occurs because larger τ incorporates more signals from correctly classified examples, analogous to increasing the effective batch size, thereby boosting performance and updating stability. Hence, we set τ = 1 in all experiments.

relatively lower variance than FFT throughout. Nevertheless, collapsed states and degenerate solutions still occur, suggesting that reducing the number of trainable parameters can mitigate instability at beginning but its effect diminishes as training progresses. In contrast, DSGD exceeds FFT after the third epoch and maintains the best accuracy throughout training while continuously decreasing the variance. These findings align with Theorems 4.6 and 4.5, confirming that DSGD ensures superior convergence and stability.

Practical Guidelines. FocalLoss and DSGD with configurations like descending linear γ or small τ (0 or 0.1) lead to lower accuracy and higher variance compared to DSGD with an ascending linear or cosine γ with τ = 1. As the model gains confidence, the former settings apply near-zero scaling to ”easy” examples, drastically reducing the effective batch size. This results in unstable updates and degraded performance. Conversely, the later settings retain more gradient signal throughout training, ensuring more stable and effective optimization.

Mitigation of Gradient Cancellation. As discussed in Section 2 and shown in Figures 2c and 2d, failure runs arise from gradient conflicts between correctly and incorrectly classified examples within a batch. DSGD alleviates this issue by dynamically downscaling the gradients of correctly classified examples. As shown in Figure 4d, the resulting gradient norm (orange curve) increases substantially over training compared to the original failed run (blue curve), indicating that DSGD effectively mitigates gradient cancellation. Figure 7 shows more examples.

Size of Trainable Parameters. To investigate whether fine-tuning a smaller set of trainable parameters can improve model stability, in Figure 4c, we compare DSGD against several parameter-efficient fine-tuning (PEFT) methods (Han et al., 2024) including LoRA (Hu et al., 2022), AdaLoRA (Zhang et al., 2023), and IA3 (Liu et al., 2022), which are designed to improve fine-tuning efficiency by updating only a subset of model parameters. Compared to FFT (∼350M learnable parameters), all PEFT methods LoRA (∼787.5K), AdaLoRA (∼6.2M), IA3 (∼173K), and our method DSGD (∼350M) initially exhibit lower accuracies, which gradually rise at different rates over epochs. All methods except LoRA surpass FFT after epoch 7. LoRA exhibits the weakest performance, with accuracy significantly lower than FFT. While the variance of FFT and PEFT methods grows with training time, PEFT approaches maintain

6. Conclusion This work investigates the seed-induced instability of finetuning LPMs on classification tasks. To mitigate this issue, we propose DSGD, a modified gradient descent algorithm that directly downscales gradients from correctly classified training examples to reduce gradient conflicts within minibatches. Through both theoretical analysis and extensive empirical evaluation, we demonstrate that DSGD achieves significantly greater stability and higher accuracy than existing baselines. 8

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

Limitation: Our approach focuses on correcting the cumulative effects of seed-induced randomness during the optimization process, including not only parameter initialization and data ordering, but also dropout masks, optimizer state initialization, and hardware-level non-determinism which collectively contribute to run-to-run variance. However, a limitation of DSGD is that it is designed specifically for gradient-based fine-tuning. It does not address other sources of instability, such as prompt formatting in prompt tuning (He et al., 2024) and in-context example selection in in-context learning (Gupta et al., 2023).

computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186, 2019. Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., and Smith, N. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020. Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.

Impact Statement

Gao, Y., Herold, C., Yang, Z., and Ney, H. Revisiting checkpoint averaging for neural machine translation. arXiv preprint arXiv:2210.11803, 2022.

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Gupta, S., Gardner, M., and Singh, S. Coverage-based example selection for in-context learning. arXiv preprint arXiv:2305.14907, 2023. Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024.

References Azizzadenesheli, K., Liu, A., Yang, F., and Anandkumar, A. Regularized learning for domain adaptation under label shifts. arXiv preprint arXiv:1903.09734, 2019.

Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pp. 1225–1234. PMLR, 2016.

Bethard, S. We need to talk about random seeds. arXiv preprint arXiv:2210.13393, 2022. Buda, M., Maki, A., and Mazurowski, M. A. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249–259, 2018.

He, J., Rungta, M., Koleczek, D., Sekhon, A., Wang, F. X., and Hasan, S. Does prompt formatting have any impact on llm performance? arXiv preprint arXiv:2411.10541, 2024.

Bui, N., Savova, G., and Wang, L. Assessing the macro and micro effects of random seeds on fine-tuning large language models, 2025. URL https://arxiv.org/ abs/2503.07329.

Hidey, C., Liu, F., and Goel, R. Reducing model churn: Stable re-training of conversational agents. In Lemon, O., Hakkani-Tur, D., Li, J. J., Ashrafzadeh, A., Garcia, D. H., Alikhani, M., Vandyke, D., and Dušek, O. (eds.), Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 14–25, Edinburgh, UK, September 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. sigdial-1.2. URL https://aclanthology.org/ 2022.sigdial-1.2/.

Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T. Learning imbalanced datasets with label-distributionaware margin loss. Advances in neural information processing systems, 32, 2019. Chen, Z., Ngiam, J., Huang, Y., Luong, T., Kretzschmar, H., Chai, Y., and Anguelov, D. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33: 2039–2050, 2020.

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.

Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S. Classbalanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268–9277, 2019.

Hua, H., Li, X., Dou, D., Xu, C., and Luo, J. Noise stability regularization for improving bert fine-tuning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3229–3241, 2021.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for

Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to 9

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.

22883–22895, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-889176-335-7. doi: 10.18653/v1/2025.findings-emnlp. 1246. URL https://aclanthology.org/2025. findings-emnlp.1246/.

Lee, C., Cho, K., and Kang, W. Mixout: Effective regularization to finetune large-scale pretrained language models. arXiv preprint arXiv:1909.11299, 2019.

Pecher, B., Cegin, J., Belanec, R., Simko, J., Srba, I., and Bielikova, M. Fighting randomness with randomness: Mitigating optimisation instability of fine-tuning using delayed ensemble and noisy interpolation. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 11005–11044, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp. 644. URL https://aclanthology.org/2024. findings-emnlp.644/.

Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980–2988, 2017. Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflictaverse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878– 18890, 2021. Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. Few-shot parameter-efficient finetuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35: 1950–1965, 2022.

Phang, J., Févry, T., and Bowman, S. R. Sentence encoders on stilts: Supplementary training on intermediate labeleddata tasks. arXiv preprint arXiv:1811.01088, 2018.

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.

Picard, D. Torch. manual seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision. arXiv preprint arXiv:2109.08203, 2021.

Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.

Sener, O. and Koltun, V. Multi-task learning as multiobjective optimization. Advances in neural information processing systems, 31, 2018.

Madhyastha, P. S. and Jain, R. On model stability as a function of random seed. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 929–939, 2019.

Summers, C. and Dinneen, M. J. Nondeterminism and instability in neural network optimization. In International Conference on Machine Learning, pp. 9913–9922. PMLR, 2021.

Mosbach, M. Analyzing pre-trained and fine-tuned language models. In Elazar, Y., Ettinger, A., Kassner, N., Ruder, S., and A. Smith, N. (eds.), Proceedings of the Big Picture Workshop, pp. 123–134, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.bigpicture-1. 10. URL https://aclanthology.org/2023. bigpicture-1.10/. Mosbach, M., Andriushchenko, M., and Klakow, D. On the stability of fine-tuning bert: Misconceptions. Explanations, and Strong Baselines. arXiv, 2020.

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pp. 353–355, 2018. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.

Nesterov, Y. Introductory Lectures on Convex Optimization: A Basic Course. Springer Publishing Company, Incorporated, 1 edition, 2014. ISBN 1461346916. Nishida, Y., Isonuma, M., and Oda, Y. Instability in downstream task performance during LLM pretraining. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp.

Wang, L., Ghosh, D., Gonzalez Diaz, M., Farahat, A., Alam, M., Gupta, C., Chen, J., and Marathe, M. Wisdom of the ensemble: Improving consistency of deep learning 10

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

models. Advances in Neural Information Processing Systems, 33:19750–19761, 2020. Wang, L., Li, Y., Miller, T., Bethard, S., and Savova, G. Two-stage fine-tuning for improved bias and variance for large pretrained language models. In Rogers, A., BoydGraber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15746– 15761, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long. 877. URL https://aclanthology.org/2023. acl-long.877/. Wu, C., Wu, F., Qi, T., and Huang, Y. Noisytune: A little noise can help you finetune pretrained language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 680–685, 2022. Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33: 5824–5836, 2020. Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for parameter-efficient finetuning. arXiv preprint arXiv:2303.10512, 2023. Zhang, T., Wu, F., Katiyar, A., Weinberger, K. Q., and Artzi, Y. Revisiting few-sample bert fine-tuning. arXiv preprint arXiv:2006.05987, 2020.

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

A. Related Works Fine-tuning instability remains an open challenge, with prior work focusing on diagnosis rather than efficient solutions. (Devlin et al., 2019; Dodge et al., 2020; Lee et al., 2019) identified two potential reasons for the observed instability: catastrophic forgetting and small size of the fine-tuning datasets. Thus increasing training size will be helpful in improving stability. However, collecting and annotating new data is super challenging in some domains. Thus, this method is infeasible in many domain tasks and its effectiveness is not guarantied. (Mosbach et al., 2020) attributes failed runs to unsuccessful optimization. They stated that observed fine-tuning instability is caused by optimization difficulties that lead to vanishing gradients. They proposed guidelines for improving stability of fine-tuning BERT which are using small learning rates with bias correction optimizer (like ADAM) and increase the number of iterations considerably and train to (almost) zero training loss. We followed their guidelines in FFT in our experiment which still show significant performance variance. While these works provide guidelines in improving stability of fine-tuning, they are either infeasible or less effective in real-world scenarios. Ensemble methods offer the most effective mitigation, providing the greatest improvements in performance and reductions in variance compared to experimental setup enhancements or data augmentation, as supported by numerous theoretical analyses (Wang et al., 2020; 2023). (Wang et al., 2020) studies model instability by defining consistency of a learning model in the context of periodic retraining of deployed models where the outputs from successive generations of the models might not agree on the correct labels assigned to the same input. It proposes dynamic snapshot ensemble to improve the consistency with theoretical supports. However, this method requires sequencial training of those components which is highly inefficient in time-intensive scenarios. Later (Wang et al., 2023) decomposes high performance variance of LPMs in fine-tuning into variance due to sampling and variance due to optimization. They perform theoretical and empirical analysis using bagging ensemble to decrease variance due to optimization. However, ensemble suffers from high computational cost as discussed in Section 5.4. Weighted averaging (Izmailov et al., 2018; Gao et al., 2022; Madhyastha & Jain, 2019; Nishida et al., 2025) is a technique that averaging nearby checkpoints to produce the final model which is empirically and theoretically shown efficiency in reducing model variations in instability tasks by avoiding saddle points in optimization landscape (Madhyastha & Jain, 2019). (Nishida et al., 2025) attributes model stability varies across different runs due to the non-determinism factors such as weight initialization, dropout, stochastic data shuffling. They investigate two post-hoc checkpoint integration methods: checkpoint averaging and ensemble, to reduce performance volatility. However, these methods require huge storage space to save those checkpoints which prohibits LPMs from deploying in real-time systems. Noise-based techniques (Hua et al., 2021; Wu et al., 2022) typically seek to improve generalization and stability of LPMs in fine-tuning by injecting noise but lack a targeted mechanism to directly stabilize the fine-tuning optimization process. The noise injection methods add randomness to everywhere which does not stabilize the sensitive directions in the parameter space that cause seed divergence but can actually increase divergence by starting from more varied points. The added noise may destroy some of the transferrable knowledge leading to reduced accuracy. Instead, our method attributes the cause of fine-tuning instability as the gradient conflicts between training examples and proposes a solution that directly targets the optimality during fine-tuning. It ensures escape from the collapsed state regardless of starting points to obtain better convergence and a tighter stability bound, leading to consistently better accuracy and variance. Despite no method has been proposed to solve gradient cancellation in fine-tuning, gradient conflicts resolvers are proposed with the aim of resolving the cancellation in multitask learning where the objective functions of individual tasks conflicts to other making the learning inefficient (Sener & Koltun, 2018; Yu et al., 2020; Liu et al., 2021; Chen et al., 2020). We adapted these methods to solve gradient cancellation in multilabel fine-tuning learning. However, they acts when gradients have significant negative cosine similarity (gradient cancellation), but collapsed state can happen even with orthogonal gradients or small-magnitude gradients for one task. In addition, these methods require two backward passes to compute the gradient separately and therefore nearly doubles the training time. In contrast, our method DSGD actively adjusts gradients based on learning outcomes that do not depend on the detection of actual gradient cancellation, making it a more robust and proactive defense against task collapse and seed-induced instability.

B. Preliminaries Definition B.1. (M-Lipschitz function). A function f is M-Lipschitz if ∀θ ∈ Θ and ∀z ∈ D: ∥∇f (θ, z)∥ ≤ M which also implies that: |f (θ, z) − f (θ′ , z)| ≤ M ∥θ − θ′ ∥. 12

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

Definition B.2. A function f ∈ CLk,α (Rn ) i.e. derivatives up to order k exist and are continuous. All k-th order derivatives satisfied: |∇k f (x) − ∇k f (y)| ≤ L∥x − y∥α . Definition B.3. A function f : Rn → R is L-smooth ⇔ f ∈ CL1,1 (Rn ).

C. Lemmas 1,1 Lemma C.1. (Descent lemma (Nesterov, 2014)) Assume that f is M -smooth i.e. f ∈ CM (Rn ), for any x, y ∈ Rn we have: M |f (y) − f (x) − ⟨f ′ (x), y − x⟩| ≤ ∥y − x∥2 . (5) 2 Lemma C.2. If L is H-Lipschitz, then DSGD and SGD updates are bounded:

∥θ − UDSGD (θ)∥ ≤ ϱηH and ∥ψ − USGD (ψ)∥ ≤ ηH. 1 where ϱ = m

(6)

i vi and vi = 1 when zi is wrongly classified according to θ and vi = γ otherwise.

Lemma C.3. Assume that L is H-Lipschitz. Fix an arbitrary sequence of updates UDSGD,1 , UDSGD,2 , . . . , UDSGD,T on sampled m-size mini-batch sequence B1 , B2 , . . . , BT and another sequence ŨDSGD,1 , ŨDSGD,2 , . . . , ŨDSGD,T on sampled m-size mini-batch sequence B̃1 , B̃2 , . . . , B̃T . Let θ0 and θ̃0 be the starting points in Θ and θ0 ̸= θ̃0 . Define ∆t = ∥θt − θ̃t ∥ where θt , θ̃t are defined recursively through: |Bt |

θt+1 = UDSGD,t (θt ) = θt − ηt

1 X vi ∇L(θt , zi ), m i=1 |B̃t |

(7)

1 X ṽi ∇L(θ̃t , zi ), θ̃t+1 = ŨDSGD,t (θ̃t ) = θ̃t − ηt m i=1 where vi = 1 if zi is wrongly classified according to θt and vi = γt otherwise, similarly with ṽi . Then we have the recurrence relation for DSGD: ∆0 = ∥θ0 − θ̃0 ∥, 1 where ϱt = m

P|Bt |

1 i=1 vi and ϱ̃t = m

∆t+1 ≤ ∆t + (ϱt + ϱ̃t )ηt H.

(8)

P|B̃t |

i=1 ṽi . For SGD, ∀i : vi = ṽi = 1:

∆′0 = ∥ψ0 − ψ̃0 ∥,

∆′t+1 ≤ ∆′t + 2ηt H.

(9)

In our analysis, when the random seed and model architecture are fixed, the initialization remains identical across different update rules, i.e., θ0 = ψ0 and θ̃0 = ψ̃0 .

D. Proofs D.1. Proof of Lemma 4.2 Proof. For a cross-entropy loss with softmax outputs, the gradients with respect to Θ and b are ∂L X = (pi − yi )x⊤ i , ∂ω i

∂L X = (pi − yi ). ∂b i

At a stationary point of L, a batch size m = 1 gives us: ∇θ L = 0 ⇐⇒ ∀i, pi = yi . Now consider LW , the loss restricted to misclassified samples. If LW has converged, then every misclassified example (xi , yi ) ∈ W satisfies (pi − yi ) = 0 =⇒ pi = yi , 13

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

which contradicts the definition of misclassification unless the example is now correctly classified. Thus all samples previously in W must satisfy arg max(pi ) = arg max(yi ) at convergence. For all correctly classified samples (xi , yi ) ∈ C, we already have arg max(pi ) = arg max(yi ) by definition. Therefore, at convergence of LW , every training example is correctly classified, which implies zero training error (i.e., 100% training accuracy), though the training loss itself may not be zero. D.2. Proof of Lemma 4.3 Proof. If LC has converged, then for all correctly classified samples (xi , yi ) ∈ C :

pi = yi .

However, this condition applies only to C and does not constrain the misclassified set W . If |W | > 0, then there exists some (xj , yj ) ∈ W such that arg max(pj ) ̸= arg max(yj ), implying strictly positive training error. Thus, convergence of LC does not guarantee zero training error unless W is empty. D.3. Proof of Lemma C.1 Proof. For all x, y ∈ Rn we have: Z 1 f (y) = f (x) +

⟨f ′ (x), τ (y − x), y − x⟩dτ

(10)

Z 1

′

= f (x) + ⟨f (x), y − x)⟩ +

′

⟨f (x + τ (y − x)) − f (x), y − x⟩dτ. 0

Hence: |f (y) − f (x) − ⟨f ′ (x), y − x⟩| = |

Z 1

⟨f ′ (x + τ (y − x)) − f ′ (x), y − x⟩dτ |

Z 1 ≤

|⟨f ′ (x + τ (y − x)) − f ′ (x), y − x⟩|dτ

(11)

Z 1 ≤

′

∥f (x + τ (y − x)) − f (x)∥ · ∥y − x∥dτ 0

Z 1 ≤

τ L · ∥y − x∥2 dτ =

M ∥y − x∥2 . 2

D.4. Proof of Lemma C.2 Proof. Based on Lemma 3.3 in (Hardt et al., 2016), consider a mini-batch B (|B| = m), by H-Lipschitz condition we have ∥∇L(θ)∥ ≤ H, ∥∇L(ψ)∥ ≤ H and γ ∈ (0, τ ] and τ ≤ 1 hence:    X  |B| |B|  1 |B|  1 X 1 X  ∥θ − U (θ)∥ = ∥η vi ∇L(θ, zi )∥ ≤ η vi ∥∇L(θ, zi )∥ ≤ η  vi   H ≤ ηH. m i=1 m i=1  m i=1  | {z }

(12)

ϱ≤1

where vi = 1 when zi is truly classified and vi = γt otherwise. We also have: |B| |B| 1 X 1 X ∥ψ − Ũ (ψ)∥ = ∥η ∇L(ψ, zi )∥ ≤ η ∥∇L(ψ, zi )∥ ≤ ηH. m i=1 m i=1

This completes the proof. 14

(13)

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

D.5. Proof of Lemma C.3 Proof. Based on Lemma C.2 that Ut and Ũt for DSGD are ϱt ηt H and ϱ̃t ηt H-bounded, then by the triangle inequality, we have: ∆t+1 = ∥Ut (θt ) − Ũt (θ̃t )∥, ≤ ∥θt − θ̃t ∥ + ∥Ut (θt ) − θt − Ũt (θt ) + θ̃t ∥,

(14)

≤ ∆t + ∥Ut (θt ) − θt ∥ + ∥Ũt (θ̃t ) − θ̃t ∥, ≤ ∆t + (ϱt + ϱ̃t )ηt H, For SGD case: vi = 1 for all cases from this the proof is completed. D.6. Proof of Theorem 4.5 Proof. By the assumption the loss function L is H-Lipschitz for every example z ∼ D, we have: E[|L(θT , z) − L(θ̃T , z)|] ≤ HE[∥θT − θ̃T ∥] = HE[∆T ],

(15)

Using Lemma C.3, we have ∆t+1 ≤ ∆t + (ϱt + ϱ̃t )ηt H we have: E[∆t+1 ] ≤ E[∆t ] + ηHE[(ϱt + ϱ̃t )]

(16)

where ηt and υt are functions of training iteration t which are independent of the data sequences. Applying this recursively and using Equation 15 we obtain: E[|L(θT , z) − L(θ̃T , z)|] ≤ HE[∆T ], ≤ HE[∆T −1 ] + H 2 E[(ϱT −1 + ϱ̃T −1 )]ηT −1 , T −1 X

≤ HE[∆T −2 ] + H 2

E[(ϱt + ϱ̃t )]ηt ,

(17)

t=T −2

≤ ..., ≤ HE[∆0 ] + H 2

T −1 X

E[(ϱt + ϱ̃t )]ηt = ϵ.

t=0

Similarly, for SGD, using Lemma C.3 we have ∆t+1 ≤ ∆t + 2ηt H, we can obtain SGD’s stability upper bound as: E[|L(ψT , z) − L(ψ̃T , z)|] ≤ HE[∆′T ], ≤ HE[∆′T −1 ] + 2ηT −1 H 2 , ≤ HE[∆′T −2 ] + 2

T −1 X

ηt H 2 ,

(18)

t=T −2

≤ ..., ≤ HE[∆′0 ] + 2H 2

T −1 X

ηt = ϵ′ .

t=0

Equations 17 and 18 hold for any z ∼ D, we analyze a model’s stability under different initializations and stochastic mini-batch ordering introduced by different random seeds. In our analysis, given a fixed random seed and a fixed model architecture, the initialization is the same for different update rules, i.e., θ0 = ψ0 and θ̃0 = ψ̃0 , which also means ∆0 = ∆′0 . Note that ϱt ∈ (0, τ ] and ϱ̃t ∈ (0, 1], then E[(ϱt + ϱ̃t )] ≤ 2. Thus, the right side of Eq. 17 satisfies: HE[∆0 ] + H 2

T −1 X

E[(ϱt + ϱ̃t )]ηt ≤ H∥θ0 − θ̃0 ∥ + 2H 2

t=0

{z ϵ

T −1 X

ηt = H∥ψ0 − ψ̃0 ∥ + 2H 2

t=0

}

T −1 X

{z ϵ′

Thus, based on Equation 17, 18, and 19, the stability bound of DSGD is lower or equals to that of SGD. 15

ηt .

t=0

}

(19)

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

D.7. Proof of Theorem 4.6 Proof. We consider y = θt+1 as the updated parameters at (t + 1)-th iteration, x = θt denotes the previous parameters, f (·) = LW (·) as the loss function. Since ∇LW is M -Lipschitz, from Lemma C.1, it follows in particular that: LW (θt+1 ) ≤ LW (θt ) + ⟨∇LW (θt ), θt+1 − θt ⟩ +

M ∥θt+1 − θt ∥2 . 2

(20)

1 We update the parameters using a modified gradient step: θt+1 = θt − η m (γt GC + GW ) where G = GC + GW denotes the original gradient. We have:

M ∥θt+1 − θt ∥2 , 2 M 1 1 ∥ − η (GW + γt GC )∥2 , ≤ LW (θt ) + ⟨GW , −η (GW + γt GC )⟩ + m 2 m 1 M η2 ∥GW + γt GC ∥2 , ≤ LW (θt ) − η ⟨GW , GW + γt GC ⟩ + m 2m2 1 M η2 ≤ LW (θt ) − η (∥GW ∥2 + γt G⊤ ∥GW + γt GC ∥2 , W GC ) + m 2m2 M η2 η (2∥GW ∥2 + 2γt G⊤ ∥GW + γt GC ∥2 . ≤ LW (θt ) − W GC ) + 2m 2m2

LW (θt+1 ) ≤ LW (θt ) + ⟨∇LW (θt ), θt+1 − θt ⟩ +

(21)

We scale GC down by γt ∈ (0, τ ] such that ∥GW ∥2 ≥ τ 2 ∥GC ∥2 ≥ γt2 ∥GC ∥2 then: η M η2 (∥GW ∥2 + γt2 ∥GC ∥2 + 2γt GTW GC ) + ∥GW + γt GC ∥2 , 2 2m 2m 1 η M η2 ≤ LW (θt ) − − ∥GW + γt GC ∥2 , m 2 2m

LW (θt+1 ) ≤ LW (θt ) −

(22)

m then LW strictly decrease unless ∥GW + γt GC ∥2 = 0. hence if 0 < η < M 1 When ∥GW + γt GC ∥2 = 0 i.e. GW + γt GC = 0 which indicates that θt+1 = θt − η m (GW + γt GC ) = θt . Since γt is dynamic, we consider the next iteration:

1 1 (GW + γt+1 GC ) = θt − η (GW + γt+1 GC ) m m In iteration t + 1, the gradients GW and GC remain unchanged due to θt+1 = θt , thus GW + γt GC = 0 still holds. Since 2 ∥GW ∥2 ≥ τ 2 ∥GC ∥2 ≥ γt+1 ∥GC ∥2 we obtain: θt+2 = θt+1 − η

LW (θt+2 ) ≤ LW (θt+1 ) + ⟨∇LW (θt+1 ), θt+2 − θt+1 ⟩ +

M ∥θt+2 − θt+1 ∥2 , 2

M ∥θt+2 − θt ∥2 , 2 M η2 ≤ LW (θt ) − ⟨GW , η(GW + γt+1 GC )⟩ + ∥GW + γt+1 GC ∥2 , 2m2 η M η2 ≤ LW (θt ) − (2∥GW ∥2 + 2γt+1 G⊤ ∥GW + γt+1 GC ∥2 , W GC ) + 2 2m 2m 1 η M η2 ≤ LW (θt ) − − ∥GW + γt+1 GC ∥2 . m 2 2m ≤ LW (θt ) + ⟨∇LW (θt ), θt+2 − θt ⟩ +

(23)

There are two cases based on 23: (1) If ∥GW + γt+1 GC ∥2 > 0, using telescoping sums obtained from 22 we have: LW (θT +1 ) − LW (θ0 ) ≤ PT η M η2 1 −m ∥GW,t + γt GC,t ∥2 . Therefore: t=0 2 − 2m mint≤T

T X 1 1 2(LW (θ0 ) − LW (θT +1 )) ∥GW,t + γt GC,t ∥2 ≤ ∥GW,t + GC,t ∥2 ≤ , m m(T + 1) t=0 (η − M η 2 /m)(T + 1)

(24)

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

hence if LW is lower bounded, i.e., L∗W := infθ∈Θ LW > −∞ then mint≤T ∥GW,t + γt GC,t ∥2 = O(1/T ). LW decreases at a rate of O(1/T ). (2) If GW + γt+1 GC = 0, since γt+1 ̸= γt > 0 and GW + γt GC = 0, then GW = GC = 0. Hence LW , LC , and L converges to a stationary point. Corollary D.1. If GW = 0 or GC = 0 DSGD also converges at a rate of O(1/T ), the same as SGD due to DSGD now becomes SGD. i.e. 1 1 • GW = 0: UDSGD (θ) = θ − η m γGC = θ − ηγ m G hence DSGD becomes SGD with learning rate of ηγ. 1 1 GW = θ − η m G thus DSGD becomes SGD with learning rate of η. • GC = 0: UDSGD (θ) = θ − η m

E. Detailed Experiment Setup In this section we detail the settings for our experiments. E.1. Implementation We arbitrarily choose 10 random seeds (42, 52, 62, 72, 82, 92, 102, 112, 122, 132) to obtain the mean and variance performance. All single runs are conducted on 2 NVIDIA-A100-80GB GPUs. E.2. Sequence Classification We conduct our experiments on 4 tasks on Super-GLUE benchmark including: RTE, COPA, MultiRC and BoolQ data statistics is shown in Table 3. We finetune pretrained language model with different type of architecture including encoder only: RoBERTa-large, decoder only: Llama-3.2-1B. To ensure a proper baseline implementation, we refer to the configurations in (Liu et al., 2019) and replicate the state-of-the-art (SOTA) scores reported using RoBERTa-large with FFT. For Llama-3.2-1B, no established reference performance was available in the literature; we therefore fine-tune it using our own settings. Although our implementation may not achieve SOTA performance, it establishes a consistent basis for comparison across different methods. Baselines. We reimplement LNSR (Hua et al., 2021) by ourself and using our own hyper-parameters to finetune backbone models for NoisyTune (Wu et al., 2022) we use the example code snippets provided in the paper and obtain the reported hyper-parameter settings for GLUE tasks. MultiRC

RTE

COPA

BoolqQ

CoLA

MRPC

2 27243 4848

2 2490 277

2 400 100

2 9427 3270

2 8551 1043

2 3668 408

Number of Classes Training Samples Validation Samples

Table 3. Sequence classification data statistics.

MultiRC Batch Size Gradient Accumulation Learning Rate Learning Rate Scheduler Max Sequence length Epoch Number of GPUs

5 2 linear 512 6 2

RTE

COPA

BoolQ

CoLA

5 5 5 5 2 2 2 2 5E-6; 1E-5; 2E-5; 3E-5; 1E-4; 3E-4 linear linear linear linear 512 256 512 256 10 10 10 3 2 2 2 2

Table 4. Roberta and Llama hyper parameters settings.

MRPC 5 2 linear 256 3 2

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

Learning rate Init r Target r r alpha Target module

IA3

AdaLoRA

LoRA

5E-3 -

5E-3 64 16 query, value

3E-4 8, 16 16 query, value

Learning Rate Batch Size Epoch Gradient Accumulation Number of GPUs

CIFAR10

CIFAR100

1E-3 128 5 2 2

Figure 6. ViT settings.

Figure 5. PEFT settings.

E.3. Settings & Implementations We follow (Liu et al., 2019; Hu et al., 2022) settings to finetune RoBERTa and Llama on GLUE and SuperGLUE benchmarks. All approaches are optimized using AdamW (Loshchilov & Hutter, 2017) optimizer. Baselines: • For PCGrad (Yu et al., 2020) we customize it for our task where we consider each group of training example in every iteration as an individual task in multi-task learning context and apply PCGrad learning procedure. The pseudo code of PCGrad is shown in Algorithm 1. • For the LNSR (Hua et al., 2021) approach, we apply a noise regularizer to the last two layers of the classifier, with λ (the weight of the regularization loss) set to 0.2 and 0.1, respectively. • For NoisyTune (Wu et al., 2022), we add uniform noise U (a, b), where λ controls the noise intensity. For the GLUE dataset, we set λ = 0.15, following (Wu et al., 2022). For other task we use the same λ setting. • SWA (Izmailov et al., 2018) maintains an average of model parameters over previous checkpoints. We tune the averaging window size in the range 3, 5 and report the best result for each task. • The settings for the PEFT methods LoRA (Hu et al., 2022), AdaLoRA (Zhang et al., 2023), and IA3 (Liu et al., 2022) are listed in Table 5. All settings follow those used in the respective reference papers. • Although the original implementation of Focal Loss (Lin et al., 2017) suggests default hyperparameters of α = 0.25 and γ = 2, the modified loss for a single data point zi is defined as: L(zi ) = −αt (1 − pt )γ log(pt ) where pt denotes the predicted probability of the ground-truth class. These configurations frequently induced training divergence or collapsed states in our preliminary evaluations. Consequently, we performed an extensive hyperparameter search for α and γ tailored to each task and pretrained architecture. The results reported herein represent the optimal performance achieved after this exhaustive tuning process. DSGD: The pseudo code of DSGD is stated in Algorithm 2. E.4. Multi-task learning For the multi-task learning experiments, we train the model on the joint training data from all tasks and evaluate it on the validation set. The model settings are the same as those used in single-task learning, as shown in Table 4, and the model is trained for 8 epochs. E.5. Image classification To assess our method for image classification, we first identify tasks affected by seed-induced variance. Initial experiments on balanced CIFAR-10/100 using Resnet-34, ResNet50, EfficientNet-b0, and ViT-base revealed minimal variance across all models. We then created imbalanced versions following (Cao et al., 2019), finding that CNN-based models exhibit significantly lower performance but comparably small variance relative to ViT-based models, which is consistent with prior work on long-tailed recognition. Thus, in our experiment, we conduct experiments on imbalanced datasets using ViT-base. 18

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

Algorithm 1 PCGrad for single task learning Require: Model parameter θ = θ0 ; training dataset distribution D; mini-batch B of m examples; loss function loss(·). 1: for i from 1 to T do 2: Sample a mini-batch training examples from D 3: Let W ← ∅ and C ← ∅ 4: for j from 1 to B do 5: if f (θi ) equals yi then 6: C ← C + {zi } 7: else 8: W ← W + {zi } 9: end if 10: end for 11: Let LW be None,PLC be None 12: Compute LW =P loss(zi ) where i ∈ W 13: Compute LC = loss(zi ) where i ∈ C 14: if LW is None and LC is not None then 1 ∇LC 15: θi+1 ← θi − η |C| 16: else if LW is not None and LC is None then 1 17: θi+1 ← θi − η |W | ∇LW 18: else 19: if cos(∇LW , ∇LC ) ≥ 0 then 1 (∇LW + ∇LC ) 20: θi+1 ← θi − η m 21: else ∇LW ·∇LC 1 W ·∇LC 22: θi+1 ← θi − η m ∇LW + ∇LC − ∇L ∇L − ∇L C W ∥∇LC ∥ ∥∇LW ∥ 23: end if 24: end if 25: end for 26: Return θT

Datasets. CIFAR-10 and CIFAR-100 contain 50000 training points and 10000 validation images of size 32 × 32 with 10 and 100 classes respectively. To create their imbalance version, Cao et al. reduce the number of training examples per class and keep the validation set unchanged. There are two types of imbalance: long-tailed (Cui et al., 2019) and step imbalance (Buda et al., 2018) Imbalance type Long-Tailed Step

Ratio

CIFAR-10

CIFAR-100

100 50 100 50

12406 36223 25250 37500

10847 36029 25250 37500

Validation samples

10000

Table 5. Image classification data statistics.

We follow (Dosovitskiy, 2020) recommendation on finetuning ViT on CIFAR-10 and CIFAR-100 datasets using a learning rate of 1E − 3 and a batch size of 256 detailed setup can be found in Table 6.

F. Additional Results F.1. Multi-tasks Learning We also conduct experiments in multi-task learning settings, where multiple tasks are learned jointly using RoBERTa-large as a shared backbone model with task-specific classification heads for each individual task. More 19

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

Algorithm 2 DSGD Require: Model parameter θ = θ0 ; training dataset distribution D; mini-batch B of m examples; scaler factor τ ; scaler function ζ(t, τ, T ); loss function loss. 1: for t from 1 to T do 2: γt = ζ(t, τ, T ) 3: Sample a mini-batch training examples zi = (xi , yi ) : i ∈ 1, . . . , m from D 4: Let W ← ∅ and C ← ∅ 5: for j from 1 to m do 6: if f (θi , xi ) equals yi then 7: C ← C + {zi } 8: else 9: W ← W + {zi } 10: end if 11: end for 12: LW ← 0, LC ←P 0 13: Compute LW =P loss(zi ) where i ∈ W 14: Compute LC = loss(zi ) where i ∈ C 1 15: θi+1 ← θi − η m (∇LW + γt ∇LC ) 16: end for 17: Return θT Roberta-large

FFT PCGrad LNSR NoisyTune SWA DSGD

MRPC + RTE

MRPC + CoLA

RTE + CoLA

MRPC

RTE

MRPC

CoLA

RTE

CoLA

85.22 ± 8.91 88.63 ± 0.67 81.18 ± 11.03 84.78 ± 8.87 79.68 ± 9.97 88.14 ± 1.20

76.03 ± 12.41 83.25 ± 1.99 69.75 ± 14.69 75.02 ± 12.68 65.56 ± 14.62 81.73 ± 1.56

77.33 ± 10.64 76.67 ± 10.70 88.01 ± 6.93 70.80 ± 6.76 79.63 ± 10.86 88.95 ± 0.80

77.30 ± 8.63 75.66 ± 8.44 83.74 ± 5.17 71.74 ± 5.46 79.01 ± 8.54 85.21 ± 0.57

68.30 ± 16.05 80.94 ± 10.10 79.24 ± 11.54 68.16 ± 14.94 76.79 ± 12.55 83.10 ± 2.56

80.51 ± 7.88 83.99 ± 5.26 83.59 ± 5.12 81.14 ± 6.86 85.54 ± 1.04 85.35 ± 0.47

ACC(↑)

STD(↓)

77.44 81.52 80.91 75.27 77.68 85.41

10.75 6.19 9.08 9.26 9.59 1.19

Table 6. Comparison with single-learner baselines on multi-tasks learning settings.

specifically, the result on three multi-task learning problem are shown in Table 6. Results. From the aggregated results on RoBERTa-large across MRPC, RTE, and CoLA, DSGD clearly achieves the best overall performance, with the highest average accuracy (85.41) and by far the lowest standard deviation (1.19). This indicates not only strong accuracy but also excellent stability across runs, which is particularly important for small and sensitive GLUE tasks like RTE and CoLA. In contrast, most baselines exhibit either lower mean performance, high variance, or both.

0.6 0.4 0.2 50

0.0

100

Cosine Sim Grad Norm

0.4 0.2 50

100

150

200

Number of Iteration

1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00

Cosine Similarity

0.8 0.6

0.8

0.6 0.4 0.2 0

0.0

100

0.0

(d) Original failed run.

DSGD grad norm Failed run grad Norm

0.6 0.4 0.2 0.0

Number of Iteration

(b) Stable run corrected by DSGD. 1.0

Gradient Norm

Cosine Similarity

(a) Original failed run. 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00

1.0

Cosine Sim Grad Norm

1.0

0.8

0.6 0.4 0.2 0

100

150

Number of Iteration

200

100

Number of Iteration

120

1.0

Gradient Norm

Number of Iteration

1.0

Gradient Norm

Cosine Sim Grad Norm

Gradient Norm

0.8

1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00

Gradient Norm

1.0

Cosine Sim Grad Norm

Cosine Similarity

1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00

Gradient Norm

Cosine Similarity

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

0.0

DSGD grad norm Failed run grad Norm

0.6 0.4 0.2 0.0

(e) Stable run corrected by DSGD.

100

150

Number of Iteration

200

(f) Gradient Comparison.

0.9

0.8

Accuracy

Figure 7. Gradient norm comparison for RoBERTa-large on COPA (seed 132 and seed 456). (a), (d) Original failed runs. (b), (e) Stable runs corrected by DSGD. In the failed run, cosine similarity reveals severe gradient conflicts. Negative values occur in 90% of iterations, with 30% falling below -0.5. Applying DSGD does not significantly change these directions, as it only scales the magnitudes of correctly classified examples. However, gradient norms increase significantly indicating that DSGD effectively mitigates gradient cancellation. Figures (c) and (e) show the gradient norm comparisons.

0.7 Ensemble Acc Mean Acc

0.6 0.5 0.4

0.7 Ensemble Acc Mean Acc

0.6 0.5

#Run

0.4

9 10

(a) ENS (×3) components.

#Run

9 10

(b) ENS (×5) components.

Figure 8. Bagging ensemble collapsed and degenerated solution proportions in individual run on COPA dataset.

0.9

0.90

0.8

0.85

0.7 0.6 0.5

FFT FocalLoss LNSR NoisyTune PCGrad

Methods

0.4

Ours

0.80 0.75 0.70 0.65

FFT FocalLoss LNSR NoisyTune PCGrad

Methods

(a) MultiRC.

0.60

Ours

(b) RTE.

0.85

0.90

0.9

0.85

0.8

0.80 0.75 0.70

Accuracy

0.95

0.65

FFT FocalLoss LNSR NoisyTune PCGrad Ours

Methods

0.90

Accuracy

0.90 0.85 0.80 0.75 0.70 0.65 0.60 0.55 0.50

Accuracy

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

0.80 0.75

0.6

0.70 0.65

FFT FocalLoss LNSR NoisyTune PCGrad Ours

Methods

FFT FocalLoss LNSR NoisyTune PCGrad

Methods

(d) COLA.

0.7

0.5

Ours

FFT FocalLoss LNSR NoisyTune PCGrad

Methods

(e) MRPC.

Ours

(f) COPA.

Figure 9. Accuracies of 10 random runs by Roberta-Large.

0.70

Accuracy

0.65 0.60 0.55 0.50

FFT FocalLoss PCGrad LNSR NoisyTune Ours

0.90 0.85 0.80 0.75 0.70 0.65 0.60 0.55 0.50 0.45

Methods

0.80 0.75 0.70 0.65 0.60 0.55

FFT FocalLoss PCGrad LNSR NoisyTune Ours

Methods

(a) MultiRC.

(b) RTE.

0.90

0.85

0.80 0.75 0.70 0.65

0.80 0.75 0.70

FFT FocalLoss PCGrad LNSR NoisyTune Ours

Methods

(d) COLA.

0.65

FFT FocalLoss PCGrad LNSR NoisyTune Ours

Methods

0.90

Accuracy

0.85

Accuracy

0.75

Accuracy

0.80

0.80 0.75 0.70

FFT FocalLoss PCGrad LNSR NoisyTune Ours

0.65

Methods

(e) MRPC. Figure 10. Accuracies of 10 random runs by Llama-3.2-1B.

FFT FocalLoss PCGrad LNSR NoisyTune Ours

Methods

(f) COPA.

0.95

0.9

0.90

0.8

0.85 0.80 0.75

0.85 0.80 0.75 0.70

FFT FocalLoss PCGrad LNSR NoisyTune Ours

Methods

FFT FocalLoss PCGrad LNSR NoisyTune Ours

0.80

FFT FocalLoss PCGrad LNSR NoisyTune Ours

Methods

(e) CIFAR-100 Long-tailed 0.01.

0.65

0.90

0.60

0.88

0.55 0.50 0.45 0.40

FFT FocalLoss PCGrad LNSR NoisyTune Ours

0.90

Accuracy

0.80 0.75 0.70 0.65 0.60 0.55 0.50 0.45 0.40

Methods

(d) CIFAR-10 Step 0.5.

0.4

FFT FocalLoss PCGrad LNSR NoisyTune Ours

(b) CIFAR-10 Long-tailed 0.5.

Accuracy FFT FocalLoss PCGrad LNSR NoisyTune Ours

0.6

Methods

(a) CIFAR-10 Long-tailed 0.01. 0.99 0.98 0.97 0.96 0.95 0.94 0.93 0.92 0.91 0.90

0.7

0.5

Accuracy

0.70

Accuracy

0.95

Accuracy

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

(f) CIFAR-100 Long-tailed 0.5.

0.86 0.84 0.82 0.80

FFT FocalLoss PCGrad LNSR NoisyTune Ours

Methods

FFT FocalLoss PCGrad LNSR NoisyTune Ours

Methods

(g) CIFAR-100 Step 0.01.

(h) CIFAR-100 Step 0.5.

Figure 11. Accuracies of 10 random runs by ViT-base.

Record · ID 146939 · SHA-256 66b0de03baf7f812
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.