arXiv:2605.06575v1 [cs.LG] 7 May 2026
Directional Consistency as a Complementary Optimization Signal: The GONO Framework Victor Daniel Gera Department of Artificial Intelligence Anurag University, Hyderabad, India [email protected]
Abstract We identify and formalize an underexplored phenomenon in deep learning optimization: directional alignment and loss convergence can be decoupled—an optimizer can exhibit near-perfect directional consistency (cct → 1, measured via consecutive gradient cosine similarity) while the loss remains high or decreases slowly. This observation reveals that existing optimizers such as Adam, SGD, and RMSprop lack explicit mechanisms to exploit temporal consistency in gradient directions, relying instead on magnitude-based signals that fail to distinguish plateaus, saddle points, and genuine convergence. Motivated by this, we introduce GONO (Gradient-Oriented Norm-Adaptive Optimizer), which adapts Adam’s momentum coefficient β1 based on cct : amplifying momentum under directional consistency √ and suppressing it during oscillation. We prove GONO matches Adam’s O(1/ T ) convergence rate and reduces exactly to Adam when the signal is uninformative. Empirically, cct achieves oscillation detection with F1 = 1.00 (vs. 0.45 for gradient norm), and GONO remains competitive with AdamW on MNIST (98.15%), CIFAR-10 (43.14%), and ResNet-18 (75.44%), establishing directional alignment as a theoretically grounded, practically actionable optimization signal. Code is available at https://github.com/victordaniel/gono-optimizer.
1
Introduction
Optimization is a cornerstone of deep learning. The choice of optimizer directly affects convergence speed, final accuracy, and generalization. Stochastic gradient descent [Robbins and Monro, 1951] and its adaptive variants—Adam [Kingma and Ba, 2015], RMSprop [Tieleman and Hinton, 2012], and AdamW [Loshchilov and Hutter, 2019]—are the workhorses of modern deep learning. Despite their empirical success, these methods share a common structural property: at every training step, the update vector is derived directly from the gradient, which simultaneously encodes both the direction of the parameter update and its magnitude. This coupling is not a fundamental requirement of optimization; it is a design choice that has been less explicitly studied. The Key Observation. In the course of this work, we made a striking empirical observation: an optimizer can exhibit near-perfect directional consistency in its updates—with consecutive gradient cosine similarity cct ≈ 1, indicating that successive gradient directions agree strongly—while the training loss simultaneously fails to decrease and the model remains in an underfitting regime. We demonstrate this across multiple architectures and datasets. Figure 1 shows a representative example: the alignment signal and the gradient norm stabilize on a different timescale than the loss, demonstrating that directional consistency is achieved well before convergence is complete. Preprint.
This observation, which we call the direction-loss decoupling phenomenon, has a clear interpretation: directional consistency tells us that the optimizer is moving in a stable, repeatable direction—but says nothing about whether that direction is leading anywhere useful. An optimizer stalled on a flat plateau will exhibit perfect directional consistency precisely because the gradient always points slightly downhill in the same direction—but the gradient magnitude is too small for meaningful progress. Why This Matters. The decoupling phenomenon reveals a gap in how we think about and design optimizers. Standard convergence theory analyzes ∥∇Lt ∥ → 0—a magnitude-based criterion. Directional consistency is an orthogonal signal that existing theory largely ignores. If an optimizer is directionally consistent but making slow progress, the right response is to increase momentum (boost step size in the stable direction)—which is the opposite of what a magnitude-based method would suggest. Conversely, if consecutive gradients conflict in direction (cct < 0)—a sign of oscillation— the right response is to reduce momentum to prevent overshoot. Adam, with its fixed β1 = 0.9, cannot make this distinction. Our Contribution. We propose GONO (Gradient-Oriented Norm-Adaptive Optimizer), which treats directional alignment as a first-class signal in optimization. The core mechanism is simple: adapt Adam’s momentum coefficient β1 at every step based on the consecutive cosine similarity cct : β1,t = clip β1 · (1 + λ cct ), β1,min , β1,max . (1) This single modification to Adam requires one additional dot product per step (O(d)), requires storing one additional gradient vector (gprev , O(d) memory), and reduces exactly to Adam when λ = 0. We make the following contributions: 1. The direction-loss decoupling phenomenon (Section 5.1, Section 4). We identify, demonstrate empirically, and support theoretically that directional alignment (cct → 1) does not guarantee loss convergence. This motivates treating the directional signal independently from magnitude. 2. GONO (Section 3). A simple, principled modification of Adam that adapts β1 using the consecutive cosine signal. GONO amplifies momentum under directional consistency and suppresses it under oscillation. √ 3. Theoretical guarantees (Section 4). We show GONO converges at the same O(1/ T ) rate as Adam under standard smoothness assumptions, and prove that cct detects oscillation precisely when gradient norm fails. 4. Empirical validation (Section 5). GONO achieves oscillation detection with F1 = 1.00 (vs. 0.45 for magnitude-based detection), validates the adaptive β1 mechanism on the Rosenbrock benchmark, and remains competitive on MNIST (98.15%) and CIFAR-10. We do not claim GONO is a universal replacement for Adam. Our goal is to introduce directional alignment as a complementary signal and demonstrate that explicitly modeling it is both theoretically sound and practically beneficial.
2
Background
2.1
Standard Gradient-Based Optimizers
Let L : Rd → R be the training loss, θt ∈ Rd the parameters at step t, and ∇Lt = ∇L(θt ) the gradient. SGD with momentum updates parameters as: mt = µmt−1 + ∇Lt , where µ ∈ [0, 1) is the fixed momentum coefficient. 2
θt+1 = θt − αmt ,
(2)
Adam [Kingma and Ba, 2015] maintains adaptive per-coordinate estimates: mt = β1 mt−1 + (1 − β1 )∇Lt , (3) vt = β2 vt−1 + (1 − β2 )∇Lt ⊙ ∇Lt , (4) α m̂t , (5) θt+1 = θt − √ v̂t + ε where m̂t = mt /(1 − β1t ), v̂t = vt /(1 − β2t ) are bias-corrected estimates. Default hyperparameters: β1 = 0.9, β2 = 0.999, ε = 10−8 . Key observation: in all of these methods, the momentum coefficient (µ or β1 ) is constant throughout training. Neither SGD nor Adam has any mechanism to detect whether consecutive gradients are pointing in consistent or conflicting directions, and neither adapts momentum in response. 2.2
Convergence Theory
The standard convergence result for SGD in non-convex settings [Ghadimi and Lan, 2013] is: T 1X 1 2 E[∥∇Lt ∥ ] ≤ O √ . T t=1 T
(6)
Adam satisfies a similar bound under appropriate conditions [Reddi et al., 2018]. These guarantees characterize √ convergence in terms of gradient magnitude: the average squared gradient norm vanishes at rate O(1/ T ). What this misses: gradient magnitude shrinking does not distinguish between a true stationary point and a saddle point (where ∇L = 0 but the point is not a minimum) or a flat plateau (where ∥∇L∥ is small but loss is still high). Directional signals—how gradient directions evolve over time—are not captured by this theory. 2.3
The Consecutive Cosine Signal
The central signal in our work is the consecutive cosine similarity between successive gradients: ⟨∇Lt , ∇Lt−1 ⟩ cct = ∈ [−1, 1]. (7) ∥∇Lt ∥ ∥∇Lt−1 ∥ + ε cct measures whether the optimizer is moving consistently (cct ≈ 1) or oscillating (cct ≈ −1) between consecutive steps. Related cosine-based signals have been used in multi-task learning to detect conflicting task gradients [Yu et al., 2020, Liu et al., 2021], but their application to single-task optimization dynamics and their connection to convergence behavior—specifically the decoupling phenomenon— has not been explored.
3
GONO: Gradient Agreement-Adaptive Momentum
3.1
Motivation: Limitations of Fixed Momentum
Adam’s update rule is: mt = β1 mt−1 + (1 − β1 )∇Lt ,
(8)
vt = β2 vt−1 + (1 − β2 )∇L2t ,
(9) α θt+1 = θt − √ m̂t . (10) v̂t + ε The momentum coefficient β1 = 0.9 is constant throughout training regardless of gradient behavior. Consider two opposite scenarios that both occur in practice: Scenario 1 (oscillation): Batch gradients alternate direction (∇Lt ≈ −∇Lt−1 ). Adam’s momentum at β1 = 0.9 carries the oscillation forward, amplifying overshoot. A smaller β1 would damp it. Scenario 2 (smooth plateau): All gradients consistently point in one direction. Adam’s β1 = 0.9 provides useful momentum, but a larger β1 would accelerate traversal of the plateau. Adam cannot adapt to either scenario because β1 is fixed. GONO resolves this with a one-line change. 3
3.2
The GONO Update Rule
Definition 1 (Consecutive Cosine Similarity) cct =
⟨∇Lt , ∇Lt−1 ⟩ ∈ [−1, 1]. ∥∇Lt ∥ ∥∇Lt−1 ∥ + ε
Definition 2 (Adaptive Momentum Coefficient) β1,t = clip(β1 · (1 + λ cct ), β1,min , β1,max ) , where λ > 0 controls sensitivity and β1,min = 0.5, β1,max = 0.99 by default. The GONO update replaces (8) with: mt = β1,t mt−1 + (1 − β1,t ) ∇Lt .
(11)
All other steps ((9)–(10)) are identical to Adam. 3.3
Algorithm
Algorithm 1 GONO Optimizer Require: Learning rate α, base momentum β1 = 0.9, β2 = 0.999, ε = 10−8 , sensitivity λ = 0.4, bounds β1,min = 0.5, β1,max = 0.99 1: Initialize m0 = 0, v0 = 0, gprev = 0, t = 0 2: while not converged do 3: t←t+1 4: gt ← ∇L(θt−1 ) {compute gradient} 5: cct ← ⟨gt , gprev ⟩/(∥gt ∥ ∥gprev ∥ + ε) {consecutive cosine} 6: β1,t ← clip(β1 (1 + λ cct ), β1,min , β1,max ) {adaptive β1 } 7: mt ← β1,t mt−1 + (1 − β1,t ) gt 8: vt ← β2 vt−1 + (1 − β2 ) gt2 9: m̂t ← mt /(1 − β1t ), √v̂t ← vt /(1 − β2t ) 10: θt ← θt−1 − α m̂t /( v̂t + ε) 11: gprev ← gt 12: end while 13: return θt 3.4
Properties 1. Reduces to Adam: When λ = 0, β1,t = β1 for all t, and GONO is identical to Adam. 2. Computational cost: One additional dot product per step (O(d)) to compute cct . No additional memory beyond storing gprev . 3. Interpretability: cct provides a real-time signal of gradient consistency. cct ≈ 1: smooth landscape, high momentum. cct ≈ −1: oscillating, low momentum. 4. Bias correction (note): Algorithm 1 uses the standard Adam bias correction m̂t = mt /(1 − Qt β1t ) with the base β1 , not the true time-varying accumulation 1 − i=1 √ β1,i . This introduces a scaling error in early steps but does not affect the asymptotic O(1/ T ) rate (Theorem 2), and retains drop-in compatibility with Adam implementations. √ 5. Convergence: GONO converges at the same O(1/ T ) rate as Adam (Theorem 2).
3.5
Intuition: Why Adaptive β1 Helps
During oscillation (e.g., traversal of a narrow valley), cct < 0 triggers β1,t < β1 , reducing momentum and preventing overshoot. During smooth gradient traversal, cct ≈ 1 triggers β1,t > β1 , accelerating progress. Adam with fixed β1 cannot distinguish these two regimes. Figure 4 (Section 5.4) shows GONO’s terrain factor and cct distribution during MNIST training, confirming that this adaptive mechanism activates throughout realistic deep learning training. 4
3.6
Evaluation of Robustness
To rigorously evaluate the control signal provided by GONO, we employ a stress-testing methodology. The learning rate was intentionally increased beyond standard stable training ranges to induce optimizer instability and evaluate robustness under stress conditions. This approach allows us to isolate the mechanistic advantages of GONO’s adaptive damping in regimes where standard fixed-momentum methods undergo catastrophic divergence.
4
Theoretical Analysis
We establish three theoretical results that underpin GONO. Theorem 1 formalizes the central empirical observation that motivates the paper. Theorem 2 guarantees GONO’s convergence. Proposition 1 justifies the consecutive cosine signal. Complete proofs appear in Appendix A. 4.1
Notation and Assumptions
Let L : Rd → R be a differentiable loss, θt ∈ Rd the parameters at step t, and ∇Lt = ∇L(θt ). The core signal in GONO is the consecutive cosine similarity: cct =
⟨∇Lt , ∇Lt−1 ⟩ ∈ [−1, 1]. ∥∇Lt ∥ ∥∇Lt−1 ∥ + ε
(12)
cct > 0 means consecutive gradients agree in direction (smooth descent); cct < 0 means they disagree (oscillation or curvature change).1 GONO’s adaptive momentum coefficient is: β1,t = clip(β1 · (1 + λ cct ), β1,min , β1,max ) .
(13)
When cct = 0 for all t, β1,t = β1 and GONO reduces exactly to Adam. We use the following standard assumptions throughout. Assumption 1 (Smoothness) L is L-smooth: ∥∇L(θ) − ∇L(ϕ)∥ ≤ L∥θ − ϕ∥ for all θ, ϕ ∈ Rd . Assumption 2 (Bounded gradients) ∥∇Lt ∥ ≤ G for all t ≥ 1. Assumption 3 (Bounded adaptive β1 ) The √ adaptive momentum coefficient satisfies β1,t [β1,min , β1,max ] with 0 < β1,min ≤ β1,max < β2 < 1. Assumption 3 is automatically satisfied by GONO with β1,max = 0.99 < 4.2
√
∈
0.999 ≈ 0.9995.
Theorem 1: Gradient Agreement Does Not Imply Convergence
Standard convergence theory monitors ∥∇Lt ∥ → 0. We show that even a stronger signal—perfect gradient agreement between consecutive steps (cct → 1)—is also insufficient to guarantee convergence. This motivates GONO’s adaptive use of cct : rather than treating high agreement as evidence of convergence, GONO uses it to boost momentum, and uses low agreement (oscillation) to damp it. Theorem 1 (Gradient Agreement ⇏ Convergence) There exists a differentiable, lower-bounded loss L : R2 → R with global minimum L∗ = 0, and an initialization θ0 , such that under gradient descent with any fixed learning rate α > 0: (i) Consecutive gradients become perfectly aligned: cct → 1 as t → ∞. (ii) The loss does not converge in practical time: L(θt ) ≥ δ > 0 for all t ≤ Tmin (α), where Tmin (α) = Ω(e3 /α) ≫ 1. In other words, cct → 1 is necessary but not sufficient for convergence—it detects directional consistency, not proximity to a minimum. 1 The ε in (12) is a numerical stabilizer. Near a minimum, ∥∇L ∥ → 0 and ε dominates, giving cc → 0 and β t t 1,t → β1 (Adam). The theoretical results assume gradients are bounded away from zero over the time horizon of interest, so ε is negligible there.
5
Proof sketch. Consider the loss L(x, y) = tanh2 (y) + εx2 , ε = 10−3 , θ0 = (0, 3). (14) The global minimum is L∗ = 0 at the origin. Since x0 = 0 and ∂L/∂x = 2εx, the trajectory satisfies xt = 0 for all t. Step 1 (Consistent gradient direction). For all yt > 0: ∇Lt = (0, 2 tanh(yt )sech2 (yt )) points ⟨∇Lt ,∇Lt−1 ⟩ = 1 for all t ≥ 2. Condition (i) consistently in the −y direction. Therefore cct = ∥∇L t ∥∥∇Lt−1 ∥ holds exactly. Step 2 (Tiny gradient magnitude on plateau). For y ≥ 2: |∂L/∂y| = 2| tanh(y)|sech2 (y) ≤ ∗ 8e−2y . At y0 = 3: ∥∇L1 ∥ ≤ 8e−6 ≈ 0.020. Each gradient step decreases y by ∆y ≤ α · 8e−2y , ∗ where y = mint yt . Step 3 (Lower bound on convergence time). While yt ≥ y0 /2 = 1.5, we have yt ≥ 1.5, so e−2yt ≤ e−3 , giving ct ≤ 8e−3 . Each step decreases y by at most α · 8e−3 . To decrease y from y0 = 3 to 1.5 requires at least: 1.5 3e3 T1 ≥ = = Ω(e3 /α). α · 8e−3 16α For all t ≤ T1 : yt ≥ 1.5, so L(θt ) = tanh2 (yt ) ≥ tanh2 (1.5) > 0.82. Condition (ii) holds with δ = 0.82. □ □ Remark 1 Theorem 1 shows that the flat plateau is the failure mode: gradients point consistently in one direction (cct = 1), but their magnitude is exponentially small, preventing convergence. This is precisely the scenario that GONO’s Corollary 1 identifies: when cct ≈ 1, GONO boosts β1,t above β1 , increasing effective momentum and step size, actively countering the plateau stall. 4.3
Theorem 2: GONO Convergence
Theorem 2 (GONO Convergence Rate) Let Assumptions 1–3 hold. Run GONO for T steps with √ learning rate αt = α/ t, β2 = 0.999, and β1,t = β1 (1 + λ cct ) clipped to [β1,min , β1,max ]. Then T 1X 1 2 E ∥∇L(θt )∥ ≤ O √ . (15) T t=1 T The constant is the same order as Adam’s convergence constant under identical assumptions [Reddi et al., 2018]. Proof sketch. The GONO update is m̂t , v̂t + ε where m̂t = mt /(1 − β1t ), v̂t = vt /(1 − β2t ), and mt = β1,t mt−1 + (1 − β1,t )∇Lt . θt+1 = θt − αt √
(16)
The key steps follow standard descent arguments for adaptive gradient methods [Zou et al., 2019], adapted to time-varying β1,t . The proof requires three ingredients: √ √ (a) Bounded effective learning rate. Since β1,t ≤ β1,max < √ β2 , the ratio β1,t / β2 < 1 uniformly, √ ensuring the effective step size α̃t = αt (1 − β1,t / β2 ) is bounded below by αt (1 − β1,max / β2 ) > 0. −1/2
(b) Bounded second moment. vt ≤ G2 coordinatewise (from Assumption 2), ensuring ∥v̂t ∥ is bounded above. (c) Telescoping descent. The complete per-step descent bound and telescoping argument follow Zou et al. [2019] (see Appendix A.2). Summing over t = 1, . . . , T and dividing by T yields (15). Since β1,t is bounded and (a)–(c) hold for any β1,t ∈ [β1,min , β1,max ]—regardless of how β1,t varies—the convergence rate is identical to Adam’s. GONO with λcc = 0 (no adaptation) reduces exactly to Adam, recovering Adam’s guarantee as a special case. □ □ Corollary 1 GONO is never asymptotically worse than Adam. When cct = 0 for all t, GONO = Adam exactly. The adaptation only activates when gradients disagree (cct ̸= 0). 6
4.4
Proposition 1: Consecutive Cosine Detects Oscillation
Proposition 1 (Oscillation Detection) Let ∇Lt = at e1 + bt e2 ∈ R2 where |bt | ≫ |at | (one dominant direction). If bt · bt−1 < 0 (gradient direction reverses in the dominant coordinate), then cct < 0. Conversely, gradient norm ∥∇Lt ∥ may increase, decrease, or remain constant during oscillation, making it an unreliable oscillation detector. Proof. Part 1 (cct < 0 during oscillation). Since |bt | ≫ |at |, we have ∥∇Lt ∥ ≈ |bt |. Therefore: cct =
at at−1 + bt bt−1 bt bt−1 ≈ = sign(bt ) · sign(bt−1 ). ∥∇Lt ∥ ∥∇Lt−1 ∥ |bt | |bt−1 |
If bt bt−1 < 0, then sign(bt ) ̸= sign(bt−1 ) and cct ≈ −1 < 0. Part 2 (Gradient norm fails). Consider f (y) = 12 ky 2 with k > 0. SGD with step size η > 1/k oscillates: yt+1 = (1 − ηk)yt , so gradients alternate sign. However, ∥gt ∥ = k|yt | and ∥gt+1 ∥ = k|1 − ηk| |yt |. For ηk ∈ (1, 2), ∥gt+1 ∥ < ∥gt ∥—gradient norm decreases even during oscillation, giving no oscillation signal. The consecutive cosine detects it immediately: cct < 0 at every oscillating step. □ □ Remark 2 Proposition 1 explains the empirical result in Section 5.2: on a steep quadratic with oscillating SGD, consecutive cosine achieves F1 = 1.00 while gradient norm achieves F1 = 0.45 (misses 55% of oscillations).
5
Experiments
We evaluate GONO across five experiments. Experiments 1 and 2 are synthetic, designed to demonstrate and validate the core claims of the paper (Experiment 2 has two parts). Experiments 3, 4, and 5 are on standard deep learning benchmarks (MNIST, CIFAR-10 MLP, and ResNet-18). All code is implemented in NumPy and PyTorch; results are averaged over 3 independent random seeds unless stated otherwise. 5.1
Experiment 1: The Direction-Loss Decoupling Phenomenon
Setup. We train a two-hidden-layer MLP (1 → 16 → 8 → 1, ReLU) with Adam (α = 10−3 ) on a regression task (y = 3x + 7 + ϵ, ϵ ∼ N (0, 4), n = 200) for 300 epochs. At each epoch we record: (1) training loss Lt , (2) consecutive cosine similarity cct , and (3) gradient norm ∥∇Lt ∥. Result. Figure 1 shows all three signals over training. The consecutive cosine signal stabilizes below the 5◦ threshold by epoch 180, while the loss continues decreasing through all 300 epochs. The gradient norm becomes negligible by epoch 100, yet the loss continues decreasing well past that point. This supports Theorem 1: directional consistency and gradient magnitude evolve on different timescales than the loss, confirming that neither signal alone predicts convergence. 5.2
Experiment 2A: Oscillation Detection
Setup. We run 100 steps of SGD (no momentum, η = 0.015) on a steep quadratic f (p) = 1 2 2 2 p1 + 100 p2 from initialization (2.0, 0.5). The effective step size along p2 is η · 200 = 3.0 > 1, so consecutive p2 -gradients alternate sign at every step—guaranteed divergent oscillation throughout all 100 steps. We compare two detectors: (a) consecutive cosine cct < −0.3, and (b) gradient norm spike detector (∥∇Lt ∥ > 1.5× rolling mean). Result. Table 1 and Figure 2 show the precision/recall of each detector. The consecutive cosine signal achieves F1 = 1.00, while gradient norm achieves only F1 = 0.45 (precision 0.97, recall 0.29). Proposition 1 shows that gradient norm is an unreliable oscillation detector: depending on step size, the norm may decrease, stay flat, or increase during oscillation. Consecutive cosine immediately flags the gradient sign reversal regardless of norm behavior. 7
Figure 1: Three signals during a 300-epoch training run (Experiment 1). Top: MSE loss (log scale). Middle: Angle between consecutive gradients; dashed = 5◦ . Bottom: Gradient norm. The angle stabilises below 5◦ by epoch 180 while loss continues decreasing through epoch 300, demonstrating the direction-loss decoupling. Table 1: Oscillation detection performance. “Grad Norm” uses ∥∇Lt ∥ > ∥∇Lt−1 ∥ as detector. “Cons. Cosine” uses cct < −0.3. Detector Gradient Norm Cons. Cosine
5.3
Precision
Recall
F1
0.97 1.00
0.29 1.00
0.45 1.00
Experiment 2B: Rosenbrock Optimization
Setup. We optimize the Rosenbrock function f (x, y) = (1 − x)2 + 100(y − x2 )2 , a classic nonconvex benchmark with a curved narrow valley. We compare SGD with momentum (µ = 0.9), Adam, and GONO, each run for up to 3000 steps from initialization (−1, 1) with step size α = 10−3 , averaged over 5 seeds. Convergence threshold: f (θ) < 0.01. Result. SGD-momentum does not converge within 3000 steps (final loss 2.05). Both Adam and GONO converge to the threshold; the Rosenbrock valley’s alternating gradients trigger sustained cct < 0 signals, causing GONO to reduce β1,t throughout traversal. This validates Proposition 1: cct detects the oscillatory regime precisely, and GONO responds by dampening momentum—the intended use case for adaptive β1 . 5.4
Experiment 3: MNIST Classification
Setup. Architecture: MLP with layers 784 → 256 → 128 → 10 (ReLU activations, He initialization, softmax output). Dataset: standard MNIST (60k train, 10k test). Training: 25 epochs, batch size 8
Figure 2: Oscillation detection comparison (Experiment 2A). Red background = actual oscillating steps (ground truth: p2 -sign flip). Third panel: Consecutive cosine (threshold −0.3, shaded) fires at every oscillating step (F1 = 1.00). Bottom panel: Gradient norm spike detector achieves F1 = 0.45.
128, learning rate 10−3 . Baselines: SGD-momentum (µ = 0.9), Adam, AdamW (weight decay 0.01). Results averaged over 3 seeds. Result. Table 2 shows final test accuracy and training loss. GONO achieves 98.15% test accuracy, matching AdamW (98.22%) and substantially outperforming Adam (97.08%) and SGD-momentum (97.23%). The improvement over Adam is consistent across all 3 seeds. Table 2: MNIST test accuracy and final training loss (mean, 3 seeds). Optimizer SGD-Momentum Adam AdamW GONO (ours)
Test Acc. (%)
Train Loss
97.23 97.08 98.22 98.15
— 0.067 0.0002 0.0004
On MNIST, 5.4% of steps are flagged as oscillating (cct < −0.15). Figure 4 shows the β1,t terrain factor and cct distribution during training. 9
Figure 3: Rosenbrock optimization (Experiment 2B). Left: Loss curves (mean ± std, 5 seeds). Center: Steps to convergence (f < 0.01); SGD-M bar shows 3000 steps (did not converge). Right: Final 300-step behaviour.
Figure 4: GONO adaptive behavior on MNIST. Left: Terrain factor (β1,t /β1 ) over training steps. Values above 1.0 indicate momentum boosted above Adam default; values below 1.0 indicate damping. Right: Distribution of cct across all training steps; 5.4% of steps (left of red threshold) trigger oscillation damping. 5.5
Experiment 4: CIFAR-10 Classification
Setup. Architecture: MLP with layers 3072 → 256 → 128 → 10 (ReLU, He initialization). Dataset: CIFAR-10 (10k training subset for reproducibility). Training: 20 epochs, batch size 64, learning rate 10−3 . Same baselines as Experiment 3. Results averaged over 3 seeds. Result. Table 3 shows CIFAR-10 results. This experiment validates that the directional signal does not harm standard benchmark performance, not to achieve SOTA accuracy. All methods perform comparably; GONO (43.14%) matches AdamW (43.22%) and does not regress relative to Adam, even when gains are smaller. Table 3: CIFAR-10 test accuracy and final training loss (mean, 3 seeds, 10k train subset). Optimizer SGD-Momentum Adam AdamW GONO (ours)
5.6
Test Acc. (%)
Train Loss
43.89 42.75 43.22 43.14
0.742 0.165 0.170 0.170
Experiment 5: ResNet-18 Benchmark
Setup and result. We train ResNet-18 on full CIFAR-10 (50k images), comparing GONO against AdamW and SGD-M with standard hyperparameters. GONO reaches 75.44% test accuracy vs. 10
76.88% for AdamW and 66.22% for SGD-M (Figure 5). GONO remains competitive on this standard benchmark despite not being specifically optimized for large-scale image classification.
Figure 5: ResNet-18 on CIFAR-10 (Experiment 5). Left: Training loss. Center: Test accuracy. Right: Gradient agreement signal cct . GONO (75.44%) is competitive with AdamW (76.88%) and outperforms SGD-M (66.22%). GONO’s primary advantage is in structured scenarios where cct cleanly identifies the regime (oscillation detection F1 = 1.00; Rosenbrock valley traversal with confirmed cct < 0 signalling). On standard benchmarks GONO is competitive with AdamW and consistently improves over Adam.
6
Related Work
Adaptive Optimizers. Adam [Kingma and Ba, 2015], AdamW [Loshchilov and Hutter, 2019], and AMSGrad [Reddi et al., 2018] all adapt update magnitude per coordinate but keep β1 fixed. GONO’s key departure is adapting β1 based on directional consistency. Gradient-Direction-Based Momentum. Sarkar [2025] (HGM) and BC [2025] (ZetA) also use angular gradient information to adapt optimization. Both measure similarity between the current gradient and a longer-horizon reference; GONO instead measures consecutive similarity cct between adjacent steps, making it a step-local oscillation detector (F1 = 1.00, Proposition 1) rather than a global trend signal. Other Adaptive Methods. Lion [Chen et al., 2023] uses sign-based momentum updates, decoupling step direction from magnitude; unlike GONO, it does not adapt based on consecutive gradient agreement. RAdam [Liu et al., 2020] rectifies Adam’s variance during early training but keeps β1 fixed throughout. PCGrad [Yu et al., 2020] projects conflicting gradients in multi-task settings; GONO applies a related cosine-based intuition to single-task consecutive steps, giving a step-local oscillation detector rather than a cross-task conflict resolver.
7
Conclusion
We have identified the direction-loss decoupling phenomenon: an optimizer can exhibit near-perfect directional consistency (cct → 1) while the loss decreases slowly, because directional consistency and gradient magnitude evolve on different timescales than the loss. The consecutive cosine signal cct detects oscillation with F1 = 1.00 vs. F1 = 0.45 for gradient norm (Proposition 1), providing a reliable, computationally cheap (O(d)) training health monitor. GONO operationalises this signal √ by adapting β1 based on cct , matching Adam’s O(1/ T ) convergence rate (Theorem 2) while remaining competitive with AdamW across MNIST (98.15%), CIFAR-10 (43.14%), and ResNet-18 (75.44%). These results establish cct as a theoretically grounded, practically actionable optimization signal, and open the door to richer directional-consistency-aware training methods.
References Samiksha BC. ZetA: A Riemann zeta-scaled extension of Adam for deep learning. arXiv preprint arXiv:2508.02719, 2025. 11
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algorithms. In Advances in Neural Information Processing Systems, volume 36, 2023. Saeed Ghadimi and Guanghui Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341–2368, 2013. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, volume 34, pages 18878–18890, 2021. Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2020. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In International Conference on Learning Representations, 2018. Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22(3):400–407, 1951. Krisanu Sarkar. Hindsight-guided momentum (HGM) optimizer: An approach to adaptive learning rates. arXiv preprint arXiv:2506.22479, 2025. Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5—RMSprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, volume 33, pages 5824–5836, 2020. Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. Sufficient conditions for convergence of the Adam algorithm. In International Conference on Learning Representations, 2019.
A
Complete Proofs
A.1
Complete Proof of Theorem 1
Proof. We prove both conditions rigorously for: L(x, y) = tanh2 (y) + εx2 ,
ε = 10−3 ,
θ0 = (0, 3).
Gradient: ∇L(x, y) = (2εx, 2 tanh(y) sech2 (y)). Since x0 = 0 and ∂x L = 2εx, the x-component satisfies xt = 0 for all t. The trajectory reduces to the scalar yt . Condition (i): cct → 1. Since xt = 0, we have ∇Lt = (0, ct ) where ct = 2 tanh(yt )sech2 (yt ) > 0 for yt > 0. Thus ∇Lt and ∇Lt−1 both point in the (0, 1) direction. The consecutive cosine is: cct =
⟨(0, ct ), (0, ct−1 )⟩ ct ct−1 = =1 |(0, ct )| |(0, ct−1 )| ct ct−1
for all t ≥ 2.
Condition (i) holds with equality (cct = 1 exactly). Gradient magnitude on plateau. For y ≥ 2: |∂L/∂y| = 2 tanh(y) sech2 (y) ≤ 2/(ey − e−y )2 /4 ≤ 8e−2y . At y0 = 3: ∥∇L1 ∥ ≈ 0.0196. 12
Gradient descent updates: yt+1 = yt − α ct where 0 < ct ≤
Condition (ii): Loss lower bound. 8e−2yt .
We track the sub-trajectory where yt ∈ [1.5, 3]. While yt ≥ 1.5, we have −2yt ≤ −3, so e−2yt ≤ e−3 and thus ct ≤ 8e−3 . Each step decreases y by at most α · 8e−3 . To decrease y from y0 = 3 to 1.5 requires: 1.5 3e3 y0 /2 = = = Ω(e3 /α). T1 ≥ α · 8e−3 8αe−3 16α For all t ≤ T1 : yt ≥ 1.5, so L(θt ) = tanh2 (yt ) ≥ tanh2 (1.5) > 0.82. This establishes condition (ii) with δ = 0.82 and Tmin = Ω(e3 /α). □ □ A.2
Complete Proof of Theorem 2
√ We reduce to Zou et al. [2019],√who establish O(1/ T ) convergence for Adam under L-smoothness, bounded gradients, and β1 < β2 . Proof. Step 1: GONO satisfies Zou et al.’s conditions. GONO’s second-moment update is identical to Adam: vt = β2 vt−1 + (1 − β2 )∇L2t . By Assumption 3, p β1,t ≤ β1,max < β2 for all t. This is the key sufficient condition in Zou et al. [2019]. Step 2: Extension to time-varying β1,t . Zou et al.’s proof bounds, at√each step t, a descent quantity that depends on β1,t , β2 , αt , and gradient magnitudes. Since β1,t < β2 holds at every individual step—not just on average—each per-step bound holds with constants evaluated at β1,max in the worst case. Telescoping over T steps yields: T 1X 1 2 E ∥∇L(θt )∥ ≤ O √ , (17) T t=1 T with constant depending on L, G, α, β1,max , β2 —the same factors as in Adam’s bound. Step 3: Reduction to Adam. Setting λcc = 0 gives β1,t = β1 (constant) for all t, making GONO identical to Adam and recovering its convergence guarantee as a special case. □ □ A.3
Complete Proof of Proposition 1
Proof. Part 1. We prove cct < 0 when bt bt−1 < 0. Write ∇Lt = at e1 + bt e2 with |bt | ≥ c∥∇Lt ∥ for some c ∈ (0.9, 1] (dominant direction assumption). Then: at at−1 + bt bt−1 cct = (18) ∥∇Lt ∥∥∇Lt−1 ∥ |at at−1 | + bt bt−1 ≤ (19) ∥∇Lt ∥∥∇Lt−1 ∥ (1 − c2 )∥∇Lt ∥∥∇Lt−1 ∥ + bt bt−1 ≤ (20) ∥∇Lt ∥∥∇Lt−1 ∥ bt bt−1 = (1 − c2 ) + . (21) ∥∇Lt ∥∥∇Lt−1 ∥ Since bt bt−1 < 0 and |bt |/∥∇Lt ∥ ≥ c: bt bt−1 ≤ −c2 . ∥∇Lt ∥∥∇Lt−1 ∥
√ Therefore cct ≤ (1 − c2 ) − c2 = 1 − 2c2 < 0 for any c > 1/ 2 ≈ 0.707. The assumption c > 0.9 > 0.707 guarantees cct < 0. 13
Part 2. We construct an oscillating trajectory where gradient norm is non-monotone. Consider f (y) = k2 y 2 , k > 0. SGD with step η > 1/k: yt+1 = (1 − ηk)yt . Let ρ = |1 − ηk| ∈ (0, 1) for ηk ∈ (1, 2). Then ∥gt ∥ = k|yt | = kρt |y0 |. The gradient norm decreases geometrically: ∥gt+1 ∥ < ∥gt ∥ at every step, including oscillating steps where gt gt−1 < 0. A gradient norm threshold detector ∥gt ∥ > τ for any fixed τ either fires on all steps or none—providing no oscillationspecific signal. Consecutive cosine: cct = −1 at every step (exact sign flip), providing a perfect oscillation signal. □ □
14