RODE: A Radial-Orthogonal Decoupled Engine for Optimization

RODE: A R ADIAL -O RTHOGONAL D ECOUPLED E NGINE FOR O PTIMIZATION A P REPRINT

arXiv:2608.21024v1 [cs.LG] 21 Aug 2026

Guoxiang Xu Bince Qu Qi Sun* Cheng Zhuo* Zhejiang University, Hangzhou, China {12647046,12621175}@zju.edu.cn

*

Correspondence: {qisunchn,czhuo}@zju.edu.cn

August 24, 2026

A BSTRACT Modern neural network training increasingly uses matrix-aware optimizers, yet their conditioned matrix step is typically added directly to the weight, jointly changing its norm and direction. This interaction matters because the current norm determines angular motion, while directional learning can drive norm growth and thereby alter later steps. We introduce RODE, which gives the radial and directional components separate update rules and step sizes. RODE explicitly updates the matrix Frobenius norm through a scalar radial rule, while its directional channel performs Newton– Schulz-conditioned updates in the tangent space. Controlled GPT-2 interventions show gains from both direct norm control and RODE’s directional update. Across two language-modeling and two image-classification tasks, RODE outperforms both Muon variants in every direct comparison and ends with lower full-model norms. At 1.5B scale, using the learning rate transferred directly from the Qwen2-style LM sweep, RODE lowers loss from 4.145 to 3.346 and final global norm from 11964 to 2183 relative to Muon RMS, with fixed-radius RODE improving further. For Qwen3.5-9B full-parameter fine-tuning, all six optimizers use the same tuning budget and the same formal-training and evaluation settings; RODE outperforms both Muon variants on all four evaluation tasks and attains the highest mean on GSM8K and MATH-500. Thus, decoupling radial and directional dynamics offers a more effective and controllable approach to matrix optimization. Keywords Optimization · Matrix parameters · Newton-Schulz iteration · Radial-directional optimization · Deep learning

1

Introduction

Modern neural networks are trained largely by updating weight matrices. Adam and AdamW adapt individual coordinates [Kingma and Ba, 2015, Loshchilov and Hutter, 2019], whereas matrix-aware optimizers exploit twodimensional structure; Muon, for example, applies Newton–Schulz iterations to momentum [Jordan et al., 2024, Higham, 2008]. Despite different constructions, most ultimately add a matrix-valued step to the current weight, Wt+1 = Wt + ∆Wt .

(1)

We refer to this operation as an additive matrix update. An additive matrix update has two geometrically distinct effects. Write Wt = ρt Ut ,

ρt = ∥Wt ∥F ,

Ut =

Wt , ρt

∥Ut ∥F = 1,

(2)

where Ut is direction and ρt is scale. In normalized or scale-invariant blocks, rescaling may preserve the represented function but still change the angular motion of the next update [van Laarhoven, 2017, Hoffer et al., 2018, Heo et al., 2021, Kosson et al., 2024]; elsewhere, scale can also affect the function directly. Thus norm and direction play different roles.

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Let Qt denote a matrix update produced by a conditioner such as Muon, so that Wt+1 = Wt − ηQt . Then ∥Wt − ηQt ∥2F = ρ2t − 2ηρt ⟨Ut , Qt ⟩F + η 2 ∥Qt ∥2F , ⊥ 2 The component parallel to Ut changes Frobenius norm to first order, while Q⊥ t changes direction by η∥Qt ∥F /ρt +O(η ). Even if Qt is tangent, the quadratic term grows the squared norm. Directional learning can therefore increase norm and reduce the angle realized by later absolute steps. Muon’s update is generally not tangent to Ut , so one conditioned step mixes both effects. Weight decay can oppose growth, but only through an optimizer- and schedule-dependent balance [Kosson et al., 2024, Wang and Aitchison, 2025]. Explicit radial control instead makes this choice visible and tunable.

We introduce RODE, a Radial-Orthogonal Decoupled Engine for matrix optimization. RODE extracts radial and directional signals, gives them different update rules and learning rates, and retains Newton–Schulz conditioning for direction. It uses tangent projections and a spherical directional realization, while a scalar rule updates radius. Radius evolution and directional learning thus become separately adjustable, while momentum across changing tangent spaces introduces a residual interaction that is accounted for in our analysis (Appendix D). RODE retains matrix-aware conditioning while making norm and directional dynamics explicitly controllable. We evaluate its components through controlled interventions, its overall optimization performance across four language and vision tasks, hyperparameter transfer at 1.5B scale, and full-parameter fine-tuning at 9B scale. Our contributions are: • We introduce RODE, a matrix optimizer that reformulates additive matrix optimization in radial–directional coordinates, turning weight scale and direction from implicitly coupled consequences of an update into separately controllable optimization variables. • First, RODE introduces an explicit radial optimization channel. It extracts the radial gradient of each managed matrix and updates its Frobenius norm with a dedicated scalar rule and learning rate, preventing norm evolution from being an uncontrolled by-product of directional learning. • Second, RODE develops an orthogonal directional engine for matrix-aware optimization. It reprojects momentum onto the current tangent space, applies projected Newton–Schulz conditioning, and realizes the resulting update as an on-sphere directional step with an independent learning rate. This construction preserves matrix-aware conditioning while decoupling directional progress from the current parameter norm. Our analysis further accounts for the residual interaction induced by momentum across changing tangent spaces and provides convergence guarantees under explicit assumptions (Appendices D and E). • We validate both the mechanism and the resulting optimizer empirically. Controlled interventions separately identify the gains from explicit norm control and the directional engine, while experiments across language modeling and image classification, together with the 1.5B hyperparameter-transfer study, demonstrate consistent improvements over Muon variants with substantially more controlled norm growth. In Qwen3.5-9B full-parameter fine-tuning, all six optimizers use the same tuning budget and formal-training and evaluation settings; RODE outperforms both Muon variants on all four evaluation tasks and attains the highest mean on GSM8K and MATH-500.

2

Related Work

RODE connects two often separate lines of work: matrix-valued update directions and parameter-scale control. Adaptive and matrix-aware optimizers. Adam(W) combines momentum with coordinate-wise second moments [Kingma and Ba, 2015, Loshchilov and Hutter, 2019]; Adafactor factorizes that state [Shazeer and Stern, 2018], and AdEMAMix mixes fast and slow momenta [Pagliardini et al., 2025]. Shampoo uses Kronecker-factored statistics [Gupta et al., 2018]; SOAP updates Adam-like statistics in the preconditioner’s eigenbasis [Vyas et al., 2025]; and MARS-M adapts variance reduction to matrix updates [Yuan et al., 2025, Liu et al., 2025a]. Muon applies short Newton–Schulz iterations to momentum [Jordan et al., 2024], with later work studying its large-scale recipe [Liu et al., 2025b]. Muon adds its conditioned step directly; RODE gives radial and matrix-conditioned directional signals different rules. Scale, direction, and manifold methods. Weight normalization exposes magnitude and direction in the model parameterization [Salimans and Kingma, 2016]. For scale-invariant normalized weights, the norm controls the effective learning rate: update-induced norm growth slows subsequent directional change, and weight decay can counteract this effect [van Laarhoven, 2017, Hoffer et al., 2018, Roburin et al., 2022]. AdamP removes norm-increasing radial momentum components to prevent premature effective-step decay [Heo et al., 2021]. Rotational Equilibrium shows that update-induced growth and weight decay can settle into an equilibrium norm and angular update, and that controlling 2

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

(a) Muon-style update

(b) RODE: radial-directional coordinates

Orthogonalize the update

Condition direction; optimize scale explicitly

A P REPRINT

RADIAL CHANNEL momentum Mt

scalar gradient ρ gt

Newton--Schulz Qt ≈ Polar(Mt)

Wt = ρtUt ‖Ut‖F = 1

gradient split ρ (gt , gt⟂ )

additive step Wt + 1 = Wt − ηQt

Wt

NS step + tangent projection Qt̂

on-sphere step Ut + 1

direction moves on ‖U‖F = 1

both change

Scale remains implicit

Wt + 1 = ρt + 1Ut + 1

TANGENT CHANNEL reproject + momentum Mt⟂

‖Wt + 1‖F

scalar update ρt + 1

Scale is an optimizer coordinate, not a side effect

Figure 1: Muon conditions momentum and adds the resulting matrix step directly to Wt , so its effects on norm and direction are jointly determined. RODE exposes a scalar radial signal and a tangent-projected directional signal, assigns them separate step sizes, and reconstructs the managed matrix from the resulting ρt+1 and Ut+1 . rotation explicitly recovers important benefits of that balance [Kosson et al., 2024]. The appropriate AdamW decay also changes with model and dataset scale and can determine whether learning-rate transfer remains valid [Wang and Aitchison, 2025]. Together, these results establish weight norm as a controller of optimization dynamics, not merely a regularization statistic. LARS/LAMB scale additive steps by parameter norms [You et al., 2017, 2020]; Fromage uses layer-relative updates [Bernstein et al., 2020]; and Nero performs angle-controlled projected updates per neuron [Liu et al., 2021]. RODE instead exposes one matrix-wide radius as an optimizer coordinate and applies its orthogonalizing kernel to tangentprojected momentum. Riemannian optimization methods based on tangent-space updates and manifold-preserving maps motivate RODE’s constrained directional update [Absil et al., 2008, Bonnabel, 2013, Bécigneul and Ganea, 2019]. Recent radius–direction methods. Hyperball wraps a base optimizer by constraining each managed matrix to the sphere defined by its initial Frobenius norm and normalizing the base-optimizer update, so the learning rate directly sets the relative update length [Wen et al., 2026]. The radius remains fixed throughout training. Magnitude–Direction Decoupling also maintains a fixed-norm directional factor, while scale is represented by separately optimized row and column gains; the directional factor can be updated by Adam, Muon, or another base optimizer [Hägele et al., 2026]. In its default form, RODE learns one matrix-wide scalar radius directly from the radial gradient. Its directional channel applies projected Newton–Schulz conditioning to tangent-projected momentum. The radial and directional channels use independent learning rates.

3

Method

RODE decomposes the current signal into radial and directional components, gives them different rules and step sizes, and retains matrix conditioning for direction. Figure 1 contrasts this interface with a direct additive update. We describe one matrix; other parameters use AdamW. 3.1

Radial–tangent gradient decomposition

Let gt be a stochastic estimator of Gt = ∇f (Wt ), and let ⟨A, B⟩F = tr(A⊤ B). We use Pt to denote the orthogonal projection onto the tangent space at the current direction Ut . Specifically, Pt (X) = X − ⟨X, Ut ⟩F Ut ,

gtρ = ⟨gt , Ut ⟩F ,

gt⊥ = Pt (gt ).

(3)

Then ⟨gt⊥ , Ut ⟩F = 0, so gtρ and gt⊥ are the instantaneous radial and tangent components of the current gradient. Under the parameterization Wt = ρt Ut , the gradient with respect to the directional coordinate U is ρt gt⊥ . RODE omits this scale factor when forming the directional signal, so that the directional update is not directly scaled by the current matrix norm. Since a positive scalar does not change the normalized polar direction used by the Newton–Schulz conditioner, ηdir can directly control the directional step. 3

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

3.2

A P REPRINT

Radial update

Let Bt−1 denote the raw momentum buffer stored from the previous step, and recall that Pt projects a matrix onto the tangent space at the current direction Ut . The radial channel updates only the matrix Frobenius norm, using the current gradient component parallel to Ut . RODE first reprojects the historical momentum onto the current tangent space and then adds the current gradient: et , Ut ⟩F = gtρ , st = ⟨B

et = βPt (Bt−1 ) + gt , B ρt+1 = max{ρt − ηρ st , κfloor ρt }.

(4)

Because Pt (Bt−1 ) lies in the tangent space at Ut , it is orthogonal to Ut and therefore contributes no radial component. Hence the radial signal satisfies st = gtρ and is determined only by the current gradient. The scalar learning rate ηρ directly controls the change in the matrix Frobenius norm. The multiplicative floor prevents a single update from collapsing the radius or crossing zero; our analysis considers the interior regime in which this safeguard is inactive. 3.3

Tangent momentum and conditioning

RODE forms the tangent momentum by projecting the updated raw buffer, et ), Mt⊥ = Pt (B

(5)

et for the next step. Because the direction Ut changes during training, radial content retained in the while storing Bt = B raw buffer can enter a future tangent space after reprojection. Appendix D bounds this cross-step residual. For wide matrices, RODE first transposes the matrix and initializes X0 =

Mt⊥ . ∥Mt⊥ ∥F

(6)

It then applies the projected Newton–Schulz iterations Ak = Xk⊤ Xk ,   Xk+1 = Pt 3.4445Xk + Xk {−4.7750Ak + 2.0315A2k } .

(7)

Newton–Schulz iterations approximate matrix sign and polar factors [Higham, 2008]. Each polynomial step can introduce a component parallel to Ut , so RODE projects the result back onto the current tangent space after every iteration. After K iterations, we denote the resulting conditioned tangent kernel by b t = XK . Q

(8)

Its Frobenius norm is used in the directional update below. 3.4

Spherical directional realization

Let b t ∥F . qt = ∥Q

(9)

For qt > 0, define the normalized tangent direction Dt⊥ and rotation angle θt as b t /qt , Dt⊥ = Q

θt = ηdir qt ,

Ut+1 = Ut cos θt − Dt⊥ sin θt ,

Wt+1 = ρt+1 Ut+1 .

(10)

Because Dt⊥ lies in the tangent space at Ut , we have ⟨Dt⊥ , Ut ⟩F = 0, which gives ∥Ut+1 ∥F = 1 in exact arithmetic. The implementation renormalizes Ut+1 for numerical stability and handles the qt = 0 case separately; Appendix A gives pseudocode. 3.5

Optional scalar damping

RODE-SNR optionally damps the directional step using a running estimate of the tangent-gradient scale. We maintain a scalar state vt that accumulates the recent magnitude of gt⊥ : vt = βvt−1 + ∥gt⊥ ∥F + ϵ. 4

(11)

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

The damping factor compares the current tangent-momentum norm with this reference scale:   ∥Mt⊥ ∥F , γmin , 1 . γt = clip vt

A P REPRINT

(12)

RODE-SNR then uses θt = ηdir γt qt , (13) ⊥ so γt changes only the directional step magnitude and leaves Dt unchanged. Default RODE sets γt = 1; we report RODE-SNR as an optional, separately tuned variant.

4

Theory

Appendix D analyzes the implemented raw-buffer recurrence in the nondegenerate interior regime, where the radial floor is inactive and the projected tangent kernel remains nonzero. The analysis assumes smoothness, bounded stochastic noise and iterates, local tangent-gradient regularity, bounded rotation angles, and positive alignment between the projected Newton–Schulz kernel and the tangent momentum. For the full gradient Gt = ∇f (Wt ), let Gρt = ⟨Gt , Ut ⟩F and G⊥ t = Pt (Gt ) denote its radial and tangent components. Under the stated assumptions, an exact expansion of the displacement Wt+1 − Wt separates radial descent, tangent descent, geodesic curvature, and cross-step leakage from the raw momentum buffer. A Lyapunov function combining f (Wt ) with the tangent-momentum tracking error yields lim sup T →∞

T 1 X  ρ 2 E (Gt ) ≤ ϵρ , T t=1

lim sup T →∞

T 1 X  ⊥ 2 E ∥Gt ∥F ≤ ϵ⊥ . T t=1

(14)

The explicit expressions for ϵρ and ϵ⊥ are given in Theorem 1; they depend on the radial and directional step sizes, the stochastic-noise levels, and the constants in the stated assumptions. These bounds cover both gradient components even though the stored raw momentum need not remain perfectly tangent as the direction Ut changes. Appendix E separately studies the first-order approximation Wt+1 = Wt − ηHt ,

(15)

where Ht is the effective update direction and η > 0 is its step size. The approximation assumes that Ht has uniformly bounded Frobenius norm and remains positively aligned with the full gradient: ⟨Gt , Ht ⟩F ≥ α∥Gt ∥2F ,

α > 0,

(16)

where α quantifies the minimum degree of alignment. The objective is also assumed to satisfy the Polyak–Lojasiewicz condition [Karimi et al., 2016],
1 ∥Gt ∥2F ≥ µ f (Wt ) − f ∗ , µ > 0, (17) 2 where µ is the PL constant and f ∗ = inf W f (W ) is the optimal objective value. If 0 < 2ηαµ < 1, smoothness gives
f (Wt ) − f ∗ ≤ (1 − 2ηαµ)t f (W0 ) − f ∗ + O(η). (18) Thus the aligned first-order model converges linearly to a neighborhood whose size is controlled by η. Its contraction factor 1 − 2ηαµ depends jointly on the alignment strength α and the PL constant µ.

5

Experiments

We test optimization quality and norm behavior, separate radial and directional contributions, then examine scale, modality, components, and systems cost. 5.1

Experimental questions and setup

We study two WikiText-103 [Merity et al., 2017] decoders: standard GPT-2 [Radford et al., 2019] for 8000 steps and a randomly initialized Qwen2-style language model for 4000 steps. The latter has 114M parameters and uses RoPE, RMSNorm, grouped-query attention, and SwiGLU [Yang et al., 2024]; it is a locally instantiated architecture rather than an official Qwen checkpoint. Vision experiments train ResNet-50 [He et al., 2016] on CIFAR-100 [Krizhevsky, 2009] and ImageNet-1K [Deng et al., 2009]. For a separate scaling study, we scale the Qwen2-style architecture to 5

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

a 1.5B-parameter decoder and train it for 20000 steps on FineWeb [Penedo et al., 2024]. We additionally conduct a full-parameter fine-tuning benchmark on Qwen3.5-9B [Qwen Team, 2026]; all six optimizers use the same tuning budget and formal-training and evaluation settings. All reported optimization-quality comparisons use three seeds. In language pretraining, RODE manages twodimensional non-embedding, non-head matrices; in vision it manages flattened multi-dimensional weights. Remaining parameters use AdamW. A within-task formal comparison shares the model, data, horizon, seed set, and checkpoint rule; optimizer-specific rates are selected by Optuna or transferred as marked. The original vision suite reuses standard-GPT-2 settings, added baselines reuse Qwen2-style-LM settings, and the 1.5B test reuses Qwen2-style-LM settings without another sweep. Appendix B specifies routing, search spaces, schedules, and selected values. Evaluation metrics and all tabulated values use raw logs. Table 1: Cross-task optimization performance on language modeling and image classification benchmarks. Results are reported as mean ± sample standard deviation over three seeds. Vision reports best accuracy on the official CIFAR-100 test set and the ImageNet-1K validation set. † Rates transferred from the Qwen2-style LM; ‡ rates transferred from GPT-2. Best available mean per column is bold. Optimizer

GPT-2 loss ↓

AdamW 3.1792 ± 0.0173 Muon Original 3.0295 ± 0.0001 Muon RMS 3.0265 ± 0.0027 SGD – RODE 2.9774 ± 0.0011 RODE-SNR 2.9724 ± 0.0557 SOAP 2.9522 ± 0.0022† MARS-M 2.9856 ± 0.0009† AdEMAMix 3.2088 ± 0.0029†

5.2

Qwen2-style LM loss ↓ CIFAR acc. ↑ ImageNet acc. ↑ 3.0503 ± 0.0071 2.8185 ± 0.0065 3.0205 ± 0.0103 – 2.7731 ± 0.0013 2.7493 ± 0.0011 2.7684 ± 0.0008 2.8885 ± 0.0040 2.9807 ± 0.0026

68.66 ± 0.29‡ 62.49 ± 0.46‡ 71.49 ± 0.67‡ 64.98 ± 0.05‡ 71.11 ± 0.14‡ 64.48 ± 0.14‡ 72.77 ± 0.24 70.25 ± 0.08 73.75 ± 0.29‡ 69.74 ± 0.47‡ 73.64 ± 0.56‡ 70.49 ± 0.09‡ 73.03 ± 0.03† – 66.26 ± 0.63† – 65.57 ± 0.35† –

Optimization quality and norm control

RODE shows consistent gains across all four main benchmarks (Table 1). Compared with the stronger Muon variant on each task, it reduces loss by 0.0491 on GPT-2 and 0.0454 on the Qwen2-style LM, while improving accuracy by 2.36 points on CIFAR-100 and 4.48 points on ImageNet-1K. Its performance also remains competitive against the broader optimizer set: SOAP gives the best GPT-2 result, RODE-SNR leads on the Qwen2-style LM and ImageNet-1K, and RODE leads on CIFAR-100. All comparisons are made within task, with tuning and transfer sources indicated in Table 1. The norm trajectories reveal a complementary aspect of RODE’s behavior. Among non-SGD methods, a RODE variant attains the lowest observed final full-model norm on every task; on GPT-2, CIFAR-100, and ImageNet-1K, the two lowest values are both obtained by RODE variants (Table 2). Muon RMS finishes at 2.5–5.7× the norm of RODE. This pattern aligns with the motivation for explicit norm control: for scale-invariant weights, parameter norm governs effective directional progress, and sustained update-induced norm growth can progressively reduce subsequent angular steps [van Laarhoven, 2017, Heo et al., 2021, Kosson et al., 2024]. Across these experiments, improved task performance under RODE is accompanied by substantially more controlled norm evolution. Full-model norm includes all trainable parameters; Appendix C.2 gives the exact definition. Figure 2 shows the GPT-2 trajectories, with the Qwen2-style LM results reported in Appendix Figure 5. 5.3

Separating norm control from directional conditioning

Lower norms alone do not establish an optimization benefit, so Table 3 introduces the radial and directional interventions sequentially. Fixing the radii of Muon-managed matrices lowers validation loss from 3.027 to 3.008. Replacing Muon’s additive directional update with RODE’s tangent directional update further lowers the loss to 2.977, while full RODE with learned radii reaches the same level. These interventions show a measurable benefit from explicit norm control and a further gain from RODE’s directional update. Muon RMS and full RODE use learning rates selected from their respective GPT-2 sweeps; Muon + NormProj inherits the Muon RMS settings, and fixed-radius RODE inherits the full RODE settings. 6

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Table 2: Final observed full-model weight norm at the end of training (three-seed mean ± sample SD). Values include every trainable parameter, including tensors delegated to AdamW, and are comparable only within a task. A dash indicates that the task–optimizer combination was not run; values are descriptive rather than ranked. Optimizer AdamW Muon Original Muon RMS SGD RODE RODE-SNR SOAP MARS-M AdEMAMix

GPT-2

Qwen2-style LM

413.6 ± 3.0 1187.9 ± 0.8 1074.4 ± 0.8 – 315.6 ± 6.6 269.1 ± 0.3 1440.1 ± 1.9 2220.0 ± 8.0 454.0 ± 1.5

711.1 ± 8.7 1314.7 ± 1.9 1722.4 ± 2.9 – 683.5 ± 2.6 695.8 ± 4.1 1046.9 ± 2.6 1417.6 ± 3.8 695.6 ± 2.4

CIFAR-100

355.9 ± 0.7 2049.5 ± 34.7 1302.6 ± 0.6 4889.3 ± 0.5 1142.0 ± 1.2 4547.4 ± 6.9 29.8 ± 0.7 56.6 ± 0.1 348.6 ± 0.3 801.8 ± 27.6 272.8 ± 0.1 320.6 ± 0.1 1136.0 ± 1.7 – 23540.5 ± 470.3 – 8375.5 ± 974.9 –

GPT-2 on WikiText-103 RODE RODE-SNR SOAP MARS-M

GPT-2 on WikiText-103 Muon RMS Muon Original AdamW AdEMAMix

2000

4.5 Weight norm

Validation loss

5.0

ImageNet-1K

4.0 3.5

RODE RODE-SNR SOAP MARS-M

Muon RMS Muon Original AdamW AdEMAMix

1500 1000 500

3.0

00

1000 2000 3000 4000 5000 6000 7000 8000 Epoch / evaluation step

2000 4000 6000 Epoch / evaluation step

8000

Figure 2: Standard GPT-2/WikiText-103 over 8000 steps (three-seed mean ± sample SD): validation loss (left) and observed full-model weight norm (right). The norm aggregates all trainable parameters, including those delegated to AdamW; both RODE variants show substantially less growth than the Muon runs. Table 3: Sequential GPT-2 norm–direction intervention. Three-seed values are mean ± sample SD; ∆ is relative to Muon RMS.

5.4

Condition

Direction update

Radius policy

Val. loss ↓

Muon RMS Muon + NormProj RODE, fixed ρ Full RODE

additive Muon additive Muon RODE tangent stack RODE tangent stack

implicit fixed by projection fixed learned scalar

3.027 ± 0.003 3.008 ± 0.001 2.977 ± 0.002 2.977 ± 0.001

∆ vs. Muon – −0.019 −0.050 −0.050

Hyperparameter transfer at 1.5B scale

We train a Qwen2-style 1.5B decoder for 20000 FineWeb steps, transferring the smaller model’s rates without a new sweep (Table 4). With the transferred rates, RODE achieves lower loss and a substantially lower global norm than Muon RMS. Fixed-radius RODE improves both measures further. Figure 3 shows the trajectories; Appendices C.3 and B give the cache construction and complete protocol. Table 4: 1.5B scale transfer with rates selected on the smaller Qwen2-style LM (three seeds, mean ± sample SD). Optimizer RODE, fixed ρ RODE Muon RMS

Best cache loss ↓

Final cache loss ↓

Final observed global norm

3.13670 ± 0.00406 3.34568 ± 0.01108 4.14465 ± 0.01800

3.13670 ± 0.00406 3.34571 ± 0.01109 4.14478 ± 0.01806

1743 ± 7 2183 ± 2 11964 ± 32

7

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

a

b

Fixed-cache Eval Loss

4.75

10000

4.25

8000

4.00

Norm

Loss

Weight Norm

12000

4.50

3.75

Muon RMS (n=3) RODE (n=3) RODE (fixed radius) (n=3)

6000 4000

3.50 3.25

A P REPRINT

Muon RMS (n=3) RODE (n=3) RODE (fixed radius) (n=3)

5k

10k Step

2000 15k

0

20k

0

5k

10k Step

15k

20k

Figure 3: Qwen2-style 1.5B/FineWeb scale transfer over 20000 steps (three-seed mean ± sample SD): unsmoothed fixed-cache loss and observed global weight norm. Fixed-radius RODE freezes managed matrix radii; parameters delegated to AdamW can still change the global norm.

5.5

Full-parameter fine-tuning at 9B scale

We full-parameter fine-tune Qwen3.5-9B on GSM8K [Cobbe et al., 2021]. Each of the six optimizers receives the same tuning budget of 20 trials of 800 steps, followed by three 3000-step runs with distinct seeds; the minimum-validation-loss checkpoint from each run is evaluated on GSM8K, MATH-500 [Hendrycks et al., 2021a], MMLU-STEM [Hendrycks et al., 2021b], and MMLU-Pro Math [Wang et al., 2024], as reported in Table 5. Appendix B gives the complete training protocol, search spaces, selected learning rates, parameter routing, and task definitions. Table 5: Qwen3.5-9B full-parameter fine-tuning and task evaluation (three-seed mean ± sample SD). All task columns report accuracy in percent. Each optimizer is independently tuned, and downstream evaluation uses the minimumvalidation-loss checkpoint from each run. Best mean per column is bold. Optimizer

Best val. loss ↓

GSM8K ↑

MATH-500 ↑ MMLU-STEM ↑ MMLU-Pro Math ↑

AdamW 0.3490 ± 0.0005 79.58 ± 0.52 44.67 ± 0.70 Muon Original 0.3497 ± 0.0005 84.23 ± 0.80 48.60 ± 3.47 Muon RMS 0.3455 ± 0.0005 84.03 ± 0.98 53.40 ± 1.51 Adafactor 0.3435 ± 0.0021 80.87 ± 2.18 54.33 ± 1.94 SOAP 0.3434 ± 0.0011 81.68 ± 1.98 48.13 ± 0.83 RODE 0.3492 ± 0.0022 85.47 ± 0.46 54.53 ± 0.95

75.53 ± 0.36 75.77 ± 0.91 76.24 ± 0.91 77.63 ± 0.14 76.22 ± 0.54 76.75 ± 1.14

40.07 ± 0.65 39.97 ± 1.35 41.77 ± 1.98 43.35 ± 0.86 41.40 ± 1.04 41.87 ± 1.26

RODE outperforms both Muon variants on all four evaluation tasks. Across all six optimizers, it achieves the highest mean on GSM8K and MATH-500, while Adafactor leads on MMLU-STEM and MMLU-Pro Math. Appendix Figure 7 provides the full tuning and validation-loss trajectories. 5.6

Transfer to computer vision

With transferred hyperparameters, RODE reaches 73.75 ± 0.29 best accuracy on CIFAR-100 and 69.74 ± 0.47 on ImageNet-1K, exceeding both Muon variants; RODE-SNR has the highest ImageNet-1K mean. RODE’s final norms are 348.6 ± 0.3 and 801.8 ± 27.6, versus 1142.0 ± 1.2 and 4547.4 ± 6.9 for Muon RMS. Thus the joint accuracy and norm behavior transfers beyond language models (Figure 4); Appendix C gives norm trajectories. 5.7

Component necessity

As shown in Table 6, all six Qwen2-style-LM variants follow the same 4000-step protocol. Structural variants inherit RODE’s learning rates, while RODE-SNR is independently tuned. Removing projected Newton–Schulz conditioning or directional learning substantially degrades performance, confirming that both are important components of RODE. 8

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

ResNet-50 on CIFAR-100

ResNet-50 on ImageNet-1K 0.7

0.7

0.6 Validation accuracy

0.6 Test accuracy

A P REPRINT

0.5 0.4 0.3

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

0.2 0.1 0.0 0

20

40 60 80 Epoch / evaluation step

0.5 0.4 0.3

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

0.2 0.1 0.0 0

100

20

40 60 80 Epoch / evaluation step

100

Figure 4: ResNet-50 test accuracy on CIFAR-100 (left) and validation accuracy on ImageNet-1K (right), each over three seeds. Table 6: Three-seed 114M component study (mean ± sample SD). Structural ablations inherit RODE rates and are not retuned; RODE-SNR is an independently tuned diagnostic. ∆ is relative to full RODE.

5.8

Variant

Intervention

Full RODE RODE-SNR w/o Newton–Schulz w/o geodesic RODE, radius fixed Radial-only control

Full engine Add tangent-gradient ratio damping Remove projected NS conditioning Use normalized retraction Hold radius fixed Hold direction fixed

Final val. ↓

∆

2.7731 ± 0.0013 2.7493 ± 0.0011 3.1249 ± 0.0113 2.7736 ± 0.0016 2.7745 ± 0.0017 3.8715 ± 0.0023

– −0.0238 +0.3518 +0.0005 +0.0014 +1.0984

Systems cost

The radial coordinate adds computation but no matrix-sized state. On the Qwen2-style-LM benchmark, RODE stores 611 MB, equal to Muon and 33% below AdamW; shape-batched fusion raises throughput from 0.59× to 0.72× AdamW, versus 0.77× for Muon RMS. On ResNet-50, fused RODE stores 102.4 MB and reaches 0.64×, versus 0.70× for Muon RMS. Table 7 gives controlled measurements; Appendix A.2 details the implementation.

6

Discussion

What RODE contributes. RODE combines a learned matrix-radius policy, a separately tuned directional policy, and matrix conditioning retained for direction. It turns two effects jointly determined by direct matrix addition into visible optimizer choices. Why radius control matters. An additive step makes norm both an outcome of directional learning and a determinant of later angular progress. Growth can therefore suppress directional motion, while weight decay regulates the interaction only indirectly [van Laarhoven, 2017, Hoffer et al., 2018, Kosson et al., 2024]. RODE gives the radial signal its own rule and removes the explicit 1/ρt factor from directional step size. Its goal is not to minimize norm, but to keep unintended norm growth from silently governing directional optimization. What the evidence supports. RODE reduces observed norm growth while improving over both Muon variants in every main language-modeling and vision comparison. The GPT-2 intervention attributes gains to radial control and then the directional engine; in the 1.5B hyperparameter-transfer study, learned-radius RODE improves both loss and global norm, and fixed radius improves further. At 9B scale, each of the six optimizers is tuned with the same budget, then trained and evaluated under the same settings. RODE outperforms both Muon variants on GSM8K, MATH-500, MMLU-STEM, and MMLU-Pro Math; among all six optimizers, it attains the highest mean on GSM8K and MATH-500. Together, these results support explicit norm control as an effective mechanism in matrix optimization. Limitations. RODE currently applies its radial–directional update only to matrix-valued parameters, while the remaining parameters are delegated to AdamW. The additional tangent projections and directional updates introduce extra computational overhead. We evaluate full-parameter fine-tuning of Qwen3.5-9B and training of a randomly 9

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Table 7: Compute and memory benchmark. Measurements use a single RTX 5090 D v2 with bf16 autocast: the Qwen2-style LM uses micro-batch 16 and sequence length 512, while ResNet-50 uses batch 64 and 224 × 224 inputs. Each of three runs has 8 warm-up and 25 timed steps; Step and Opt. are medians of the three per-run medians. Peak is PyTorch peak allocated memory, State is exact optimizer-state tensor storage, Opt. includes step() and gradient clearing, and throughput is relative to AdamW. Task

Optimizer

State (MB)

Peak (GB)

Step (ms)

9.0 37.7 39.1 42.1 68.1 23.1 81.2 47.9

1.00× 0.78× 0.77× 0.74× 0.64× 0.86× 0.59× 0.72×

3.27 41.3 2.5 3.26 59.0 20.2 3.26 58.9 20.2 3.71 74.0 34.7 3.28 201.6 161.8 3.37 58.2 18.6 3.17 89.5 50.4 3.17 64.8 24.7

1.00× 0.70× 0.70× 0.56× 0.20× 0.71× 0.46× 0.64×

Qwen2-style LM AdamW Muon Original Muon RMS SOAP MARS-M AdEMAMix RODE (loop) RODE (fused)

913.0 10.92 102.0 611.0 10.62 131.3 611.0 10.62 132.4 2643.2 12.67 137.3 913.0 10.96 158.9 1369.6 11.38 118.9 611.0 10.62 172.3 611.0 10.63 142.2

ResNet-50

204.5 102.4 102.4 630.3 204.5 306.7 102.4 102.4

AdamW Muon Original Muon RMS SOAP MARS-M AdEMAMix RODE (loop) RODE (fused)

Opt. Throughput (ms) (vs. AdamW)

initialized Qwen2-style 1.5B model, but the current study covers a limited set of language and vision architectures and training budgets; broader scales and model families remain to be explored. The theoretical convergence guarantees and rates are established under idealized regularity and alignment assumptions.

7

Conclusion

Direct additive matrix updates couple weight-norm evolution with directional progress, allowing norm growth to alter subsequent optimization dynamics. RODE separates these effects by treating the matrix Frobenius norm and the matrixconditioned direction as distinct optimization coordinates with independent update rules and step sizes. Controlled interventions isolate gains from both radial control and directional learning, while experiments across language modeling and image classification show consistent improvements over Muon variants together with substantially more controlled norm growth. The 1.5B hyperparameter-transfer study further shows that these gains persist when learning rates are transferred from a smaller model, with fixed-radius RODE improving further. In Qwen3.5-9B full-parameter fine-tuning, RODE outperforms both Muon variants on all four evaluation tasks and leads the six-optimizer comparison on GSM8K and MATH-500. These results support radial–directional decoupling as an effective and practical design strategy for matrix-aware optimization.

References Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan.github.io/posts/muon/, 2024. Accessed August 17, 2026. Nicholas J. Higham. Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics, 2008. doi:10.1137/1.9780898717778. Twan van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350, 2017. URL https://arxiv.org/abs/1706.05350. 10

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: Efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, volume 31, 2018. Byeongho Heo, Sanghyuk Chun, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Gyuwan Kim, Youngjung Uh, and Jung-Woo Ha. AdamP: Slowing down the slowdown for momentum optimizers on scale-invariant weights. In International Conference on Learning Representations, 2021. Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational equilibrium: How weight decay balances learning across neural networks. In International Conference on Machine Learning, volume 235, pages 25333–25369, 2024. Xi Wang and Laurence Aitchison. How to set AdamW’s weight decay as you scale model and dataset size. In International Conference on Machine Learning, volume 267, pages 62222–62250, 2025. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, volume 80, pages 4596–4604, 2018. Matteo Pagliardini, Pierre Ablin, and David Grangier. The AdEMAMix optimizer: Better, faster, older. In International Conference on Learning Representations, 2025. Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, volume 80, pages 1842–1850, 2018. Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. SOAP: Improving and stabilizing Shampoo using Adam for language modeling. In International Conference on Learning Representations, 2025. Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. MARS: Unleashing the power of variance reduction for training large models. In International Conference on Machine Learning, volume 267, pages 73553– 73587, 2025. Yifeng Liu, Angela Yuan, and Quanquan Gu. MARS-M: When variance reduction meets matrices, 2025a. URL https://arxiv.org/abs/2510.21800. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025b. URL https://arxiv.org/abs/2502.16982. Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, volume 29, 2016. Simon Roburin, Yann de Mont-Marin, Andrei Bursuc, Renaud Marlet, Patrick Pérez, and Mathieu Aubry. Spherical perspective on learning with normalization layers. Neurocomputing, 487:66–74, 2022. doi:10.1016/j.neucom.2022.02.021. Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks, 2017. URL https: //arxiv.org/abs/1708.03888. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations, 2020. Jeremy Bernstein, Arash Vahdat, Yisong Yue, and Ming-Yu Liu. On the distance between two neural networks and the stability of learning. In Advances in Neural Information Processing Systems, volume 33, 2020. Yang Liu, Jeremy Bernstein, Markus Meister, and Yisong Yue. Learning by turning: Neural architecture aware optimisation. In International Conference on Machine Learning, volume 139, pages 6748–6758, 2021. P.-A. Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008. Silvère Bonnabel. Stochastic gradient descent on Riemannian manifolds. IEEE Transactions on Automatic Control, 58 (9):2217–2229, 2013. doi:10.1109/TAC.2013.2254619. Gary Bécigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. In International Conference on Learning Representations, 2019. Kaiyue Wen, Xingyu Dang, Kaifeng Lyu, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them II: Hyperball optimization. arXiv preprint arXiv:2606.16899, 2026. URL https://arxiv.org/abs/ 2606.16899. Alexander Hägele, Alejandro Hernández-Cano, Atli Kosson, and Martin Jaggi. Improving neural network training by decoupling the magnitude and direction of weight vectors. arXiv preprint arXiv:2606.25971, 2026. URL https://arxiv.org/abs/2606.25971. 11

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak–łojasiewicz condition. In European Conference on Machine Learning and Knowledge Discovery in Databases, pages 795–811, 2016. doi:10.1007/978-3-319-46128-1_50. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Technical Report, 2019. URL https://cdn.openai.com/ better-language-models/language-models.pdf. An Yang et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. URL https://arxiv.org/abs/ 2407.10671. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. doi:10.1109/CVPR.2016.90. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report, 2009. URL https: //www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. doi:10.1109/CVPR.2009.5206848. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, volume 37, 2024. doi:10.52202/079017-0970. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id= qwen3.5. Accessed August 21, 2026. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021a. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/ hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021b. URL https://openreview.net/forum?id=d7KBjmI3GmQ. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems, volume 37, 2024. doi:10.52202/079017-3018. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track.html. HuggingFaceFW. FineWeb dataset card, 2025. URL https://huggingface.co/datasets/HuggingFaceFW/ fineweb. Version 1.2.0; accessed July 31, 2026.

12

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A

Complete update and fused implementation

A.1

Per-matrix update

A P REPRINT

Input: parameter Wt , gradient gt , raw buffer Bt−1 , learning rates ηρ , ηdir , momentum β, floor κfloor , NS steps K, and safeguards ϵ, ϵNS . 1. If ∥Wt ∥F > ϵ, form ρt = ∥Wt ∥F , Ut = Wt /ρt , and Pt (X) = X − ⟨X, Ut ⟩F Ut . et = βPt (Bt−1 ) + gt . 2. Set B et , Ut ⟩F and ρt+1 = max{ρt − ηρ st , κfloor ρt }. 3. Set st = ⟨B ⊥ et ) = B et − st Ut . 4. Set Mt = Pt (B 5. Transpose Mt⊥ and Ut if needed so rows ≥ columns; initialize X0 = Mt⊥ /∥Mt⊥ ∥F in the nonzero branch. 6. For k = 0, . . . , K − 1, set h i Xk+1 = Pt 3.4445Xk + Xk {−4.7750Xk⊤ Xk + 2.0315(Xk⊤ Xk )2 } . b t = XK . Undo any transpose and set Q ⊥ b b t /qt , θt = ηdir qt , and Ut+1 = Ut cos θt − Dt⊥ sin θt . Renormalize Ut+1 . 7. For qt = ∥Qt ∥F > 0, set Dt = Q et . 8. Write Wt+1 = ρt+1 Ut+1 and store Bt = B

Algorithm 1: Default RODE update for one matrix parameter Zero parameter norms and zero projected kernels take safeguarded no-direction branches. RODE-SNR changes θt to ηdir γt qt . A.2

Shape-batched implementation

The per-parameter form of RODE iterates over matrices in Python, issuing many small kernels for the split, projection, Newton–Schulz conditioning, and geodesic step. Table 7 shows 0.59× AdamW throughput on the Qwen2-style-LM configuration. The fused implementation removes the Python loop without changing the per-matrix computation. Within a parameter group, parameters are bucketed by shape; the B matrices of a common shape m × n are stacked into a single tensor W ∈ RB×m×n , and likewise their gradients and momentum buffers. Each step of the engine then becomes a batched operation that broadcasts over the leading dimension: • Polar split. ρ = ∥W∥F and U = W/ρ use a Frobenius norm reduced over the last two dimensions, giving ρ ∈ RB×1×1 . • Tangent momentum. The projection PU (X ) = X − ⟨X , U⟩ U uses a batched inner product (an elementwise product summed over the matrix dimensions, kept as B × 1 × 1). The fused code first forms the raw buffer f ← βPU (Mold ) + G, then removes the radial component for the directional engine: M⊥ ← PU (M) f = M f βPU (Mold ) + PU (G). The raw buffer M is what is written back; the next step begins by reprojecting it to the then-current tangent space. • Newton–Schulz. The quintic iteration is batched matrix multiplication: A = X ⊤ X and X ← 3.4445 X + X (−4.7750 A + 2.0315 A2 ) are all batched bmms, with the tangent reprojection applied batched after each iteration. Transposition (the m < n case) is uniform within a bucket. • Geodesic step. cos and sin of the per-matrix angle θ ∈ RB×1×1 are applied elementwise, and the updated matrices are written back in bulk with torch._foreach_copy_. Matrices of different shapes form separate buckets. Loop and batched implementations agree to bf16 numerical tolerance (9 × 10−4 in a five-step check). Batching raises Qwen2-style-LM throughput from 0.59× to 0.72× AdamW (Table 7), compared with 0.77× for Muon RMS at equal state memory. RODE performs additional projection and geodesic operations in this comparison.

B

Optimizer Definitions and Hyperparameters

This appendix gives the exact optimizer routing, sweep budgets, and selected hyperparameters used in the reported experiments. We write η2d/dir for the matrix-parameter learning rate; for RODE-family optimizers this is the angular 13

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Table 8: Operational baseline definitions used in the code. Matrix weight decay is the task-level –wd; non-matrix AdamW uses weight decay 9 × 10−4 unless the row states otherwise. Optimizer

Matrix-parameter update

AdamW

AdamW on matrix parameters AdamW on all remaining parameters Muon, original LR scaling AdamW Muon, RMS-matched LR AdamW scaling Radial update + tangent AdamW Newton–Schulz geodesic step RODE Add tangent-gradient AdamW ratio damping SOAP on matrix parameters AdamW

Muon Original Muon RMS RODE RODE-SNR SOAP MARS-M AdEMAMix SGD

MARS-M on matrix parameters AdEMAMix on matrix parameters SGD on matrix parameters

Non-matrix update

AdamW AdEMAMix on all remaining parameters SGD on all remaining parameters

Fixed optimizer constants betas (0.9, 0.95) momentum 0.95; matrix wd 0 momentum 0.95; matrix wd 0 momentum 0.9; NS steps 5; κfloor = 10−6 ; ϵ = 10−12 RODE constants; ratio clamp [0.01, 1] betas (0.95, 0.95), precondition frequency 10, max precondition dim 10000 momentum 0.95, NS steps 5, γ = 0.025, approximate mode betas (0.9, 0.999, 0.9999), α = 2.0 momentum 0.9

learning rate ηdir , while for non-RODE matrix optimizers it is the ordinary matrix-parameter learning rate. We write ηρ for RODE’s radial learning rate and η1d for the non-matrix learning rate. A dash denotes an unused parameter. Unless stated otherwise, language runs use two-dimensional non-embedding, non-head parameters as the matrix set, with all remaining parameters routed to the listed non-matrix optimizer. Vision runs route all parameters with ndim >= 2 to the matrix optimizer; for Muon baselines, convolutional tensors are reshaped to matrices before the Muon step. Hyperparameter selection protocol. For each sweep group in Table 9, all optimizers in that group use the listed number of Optuna trials, training horizon, data source, model configuration, evaluation protocol, and log-uniform search spaces. The Qwen2-style sweeps evaluate once at the end of the sweep horizon and select the trial with the lowest validation loss. The standard GPT-2 sweep selects the trial with the lowest mean training loss over the final 30 steps of its 300-step protocol. After selection, the chosen values are fixed for the formal multi-seed runs. When a row is marked as transferred in Tables 10–15, the target task is not re-swept: the hyperparameters are copied from the stated source sweep before running the target-task experiment. CIFAR-100, ImageNet-1K, and the Qwen2-style 1.5B FineWeb experiment are transfer evaluations, not task-specific tuning studies. The separate Qwen3.5-9B sweep and training protocol is specified below and in Table 13.

14

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Table 9: Learning-rate sweep budgets and search spaces. All sweep ranges are log-uniform. The selected values from these sweeps are reported in the task-specific hyperparameter tables below. The added baselines are swept on the Qwen2-style LM/WikiText-103 benchmark and then transferred to standard GPT-2 and CIFAR-100. Source

Optimizer

Budget

Search space

GPT-2 GPT-2 GPT-2 GPT-2

AdamW Muon Original Muon RMS RODE/RODE-SNR

40×300 40×300 40×300 40×300

Qwen2-style Qwen2-style Qwen2-style

AdamW 30×500 Muon Origi- 30×500 nal/Muon RMS RODE/RODE-SNR 30×500

η : [10−4 , 10−2 ]; weight decay [10−4 , 10−1 ] η2d/dir : [10−4 , 2×10−1 ]; η1d : [10−5 , 10−2 ] η2d/dir : [10−5 , 5×10−2 ]; η1d : [10−5 , 10−2 ] η2d/dir : [10−4 , 5×10−1 ]; ηρ : [10−5 , 10−1 ]; η1d : [10−5 , 10−2 ] η2d/dir , η1d : [3×10−5 , 3×10−3 ] η2d/dir : [5×10−3 , 3×10−1 ]; η1d : [3×10−5 , 3×10−3 ]

Qwen2-style Qwen2-style Qwen2-style

SOAP MARS-M AdEMAMix

30×500 30×500 30×500

η2d/dir : [10−4 , 5×10−2 ]; ηρ : [10−6 , 5×10−3 ]; η1d : [3×10−5 , 3×10−3 ] η2d/dir : [3×10−5 , 10−2 ]; η1d : [3×10−5 , 3×10−3 ] η2d/dir : [3×10−5 , 3×10−2 ]; η1d : [3×10−5 , 3×10−3 ] η2d/dir , η1d : [3×10−5 , 3×10−3 ]

Table 10: Standard GPT-2/WikiText-103 hyperparameters for the 8000-step main benchmark. Runs use sequence length 512, effective batch size 128, micro-batch size 16, bf16 autocast, evaluation every 500 steps, 100 warmup steps, and cosine decay. Optimizer AdamW Muon Original Muon RMS RODE RODE-SNR SOAP MARS-M AdEMAMix

η2d/dir

ηρ

η1d

−4

Matrix wd −4

−4

4.483×10 – 4.483×10 9.077×10 1.264×10−2 – 1.314×10−5 0 1.806×10−3 – 9.274×10−4 0 6.432×10−4 1.502×10−2 5.012×10−3 0 2.413×10−3 1.288×10−4 2.311×10−4 0 2.900×10−3 – 2.591×10−3 10−2 4.097×10−3 – 2.693×10−3 10−2 2.237×10−4 – 1.779×10−3 10−2

1D wd

Source −4

9.077×10 9×10−4 9×10−4 9×10−4 9×10−4 9×10−4 9×10−4 9×10−4

GPT-2 sweep GPT-2 sweep GPT-2 sweep GPT-2 sweep GPT-2 sweep Qwen transfer Qwen transfer Qwen transfer

Table 11: Qwen2-style LM/WikiText-103 hyperparameters for the aligned 4000-step main benchmark and component ablation. The model has 114M parameters. All runs use sequence length 512, effective batch size 128, micro-batch size 16, bf16 autocast, the official validation split, evaluation every 200 steps, 200 warmup steps, and cosine decay. This locally instantiated architecture uses a separate protocol from Table 10. Optimizer AdamW Muon Original Muon RMS RODE RODE-SNR SOAP MARS-M AdEMAMix

η2d/dir

ηρ

η1d

−4

Matrix wd 1D wd Source −3

1.544×10 – 2.988×10 2.257×10−2 – 2.897×10−3 5.160×10−3 – 2.913×10−3 1.206×10−3 2.438×10−3 2.915×10−3 4.544×10−3 3.698×10−5 2.974×10−3 2.900×10−3 – 2.591×10−3 −3 4.097×10 – 2.693×10−3 −4 2.237×10 – 1.779×10−3

15

10−2 0 0 0 0 10−2 10−2 10−2

9×10−4 Qwen sweep 9×10−4 Qwen sweep 9×10−4 Qwen sweep 9×10−4 Qwen sweep 9×10−4 Qwen sweep 9×10−4 Qwen sweep 9×10−4 Qwen sweep 9×10−4 Qwen sweep

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Table 12: Qwen2-style 1.5B/FineWeb transfer hyperparameters. This three-seed experiment reuses learning rates selected on the smaller Qwen2-style LM without a new sweep. Runs use streaming FineWeb sample-100BT, sequence length 1024, effective batch size 128, micro-batch size 2 per GPU, bf16 parameters/autocast, 20000 optimizer steps, evaluation every 500 steps, checkpointing every 100 steps, 200 warmup steps, cosine decay, and torch.compile(default). The fixed evaluation cache is identical across optimizers. η2d/dir

ηρ

η1d

Matrix wd

1D wd

RODE, fixed ρ

1.206×10−3

0

2.915×10−3

0

9×10−4

64

RODE

1.206×10−3 2.438×10−3 2.915×10−3

0

9×10−4

64

Muon RMS

5.160×10−3

0

9×10−4

64

Optimizer

–

2.913×10−3

Accum. Implementation fused RODE, fixed radius fused RODE, chunk 2 PyTorch Muon RMS

Qwen3.5-9B full-parameter fine-tuning. All six optimizers start from the same non-Base Qwen3.5-9B checkpoint and use the same materialized GSM8K split: 7099 training examples and 374 validation examples. Each optimizer receives 20 Optuna trials of 800 steps. The Adafactor, Muon Original, and SOAP studies enqueue three, four, and four fixed candidate settings, respectively, before continuing with log-uniform sampling; these candidates count toward the same 20-trial budget. Validation is run every 100 steps on all 374 examples, and the trial with the lowest observed validation loss supplies the learning rates for the formal runs. Formal training uses three distinct seeds for 3000 steps, sequence length 1024, micro-batch size 1, gradient accumulation 8, BF16 full-parameter training, gradient checkpointing, gradient clipping at norm 1.0, and zero weight decay. The schedule applies 3% linear warmup followed by cosine decay. Evaluation is run every 100 steps, and the checkpoint with the minimum validation loss is saved for downstream evaluation. The parameter routing in this experiment differs from the language-model default above: every trainable two-dimensional tensor, including any embedding and output-head matrices, belongs to the matrix set. Muon Original, Muon RMS, SOAP, and RODE update this set with their matrix rule and route all remaining tensors to AdamW with betas (0.9, 0.95). AdamW and Adafactor each update all trainable parameters with one optimizer. Here Adafactor uses an explicit learning rate with relative-step scaling, parameter scaling, and warmup initialization disabled. SOAP uses betas (0.9, 0.95), precondition frequency 100, maximum precondition dimension 1024, and disables dimension merging, one-dimensional preconditioning, and gradient normalization. For downstream evaluation, every run loads its minimum-validation-loss checkpoint. GSM8K (1319 examples) and MATH-500 (500 examples) use greedy generation with at most 512 new tokens and normalized exact-answer matching. MMLU-STEM (3153 examples) and MMLU-Pro Math (1351 examples) use a fixed zero-shot multiple-choice prompt: each option letter is scored by its summed sequence log-likelihood, and the highest-scoring option is selected. Table 5 reports accuracy over each complete materialized evaluation set.

Table 13: Qwen3.5-9B learning-rate searches and selected values. Every optimizer receives 20 trials of 800 steps; all searched ranges are log-uniform. “Same” denotes a single learning rate applied to all trainable parameters. Optimizer AdamW Muon Original Muon RMS Adafactor SOAP RODE

Selected η2d/dir Selected ηρ Selected η1d

Search space −6

−5

η : [3×10 , 8×10 ] η2d/dir : [10−4 , 5×10−2 ]; η1d : [2×10−6 , 10−4 ] η2d/dir : [3×10−6 , 10−4 ]; η1d = 2×10−5 fixed η : [10−6 , 10−4 ] η2d/dir : [10−5 , 8×10−3 ]; η1d : [2×10−6 , 2×10−4 ] η2d/dir : [2×10−6 , 3×10−5 ]; ηρ : [10−6 , 5×10−5 ]; η1d : [3×10−6 , 5×10−5 ]

16

2.402×10−5 2.869×10−4

– –

Same 5.846×10−6

1.971×10−5 8.000×10−6 2.118×10−5

– – –

2.000×10−5 Same 3.020×10−6

6.259×10−6

2.919×10−6 5.592×10−6

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Table 14: ResNet-50/CIFAR-100 hyperparameters. Runs use 100 epochs, image size 224, effective batch size 128, three distinct seeds, and epoch-wise cosine annealing. The implemented scheduler is CosineAnnealingLR(T_max=epochs) with no warmup stage. We report best test accuracy. No CIFAR-specific sweep is used: the original suite uses standardGPT-2-selected settings, while the added baselines use settings selected on the Qwen2-style LM. η2d/dir

Optimizer AdamW Muon Original Muon RMS RODE RODE-SNR SOAP MARS-M AdEMAMix SGD

ηρ

η1d

Matrix wd 1D wd Source

4.483×10−4 – 4.483×10−4 1.264×10−2 – 1.314×10−5 −3 1.806×10 – 9.274×10−4 −4 −2 6.432×10 1.502×10 5.012×10−3 −3 −4 2.413×10 1.288×10 2.311×10−4 −3 2.900×10 – 2.591×10−3 −3 4.097×10 – 2.693×10−3 2.237×10−4 – 1.779×10−3 1.0×10−1 – 1.0×10−1

10−2 0 0 0 0 10−2 10−2 10−2 5×10−4

9×10−4 GPT-2 transfer 9×10−4 GPT-2 transfer 9×10−4 GPT-2 transfer 9×10−4 GPT-2 transfer 9×10−4 GPT-2 transfer 9×10−4 Qwen transfer 9×10−4 Qwen transfer 9×10−4 Qwen transfer 5×10−4 standard

Table 15: ResNet-50/ImageNet-1K hyperparameters. Runs use 100 epochs, image size 224, effective batch size 256, three distinct seeds, and epoch-wise cosine annealing without an implemented warmup stage. Table 1 reports, for each seed, the best validation accuracy observed over 100 epochs. The added SOAP/MARS-M/AdEMAMix baselines are not run on ImageNet-1K. Optimizer AdamW Muon Original Muon RMS RODE RODE-SNR SGD

η2d/dir

ηρ −4

η1d

4.483×10 – 1.264×10−2 – 1.806×10−3 – 6.432×10−4 1.502×10−2 2.413×10−3 1.288×10−4 1.0×10−1 –

Matrix wd −4

4.483×10 1.314×10−5 9.274×10−4 5.012×10−3 2.311×10−4 1.0×10−1

−2

10 0 0 0 0 5×10−4

1D wd −4

9×10 9×10−4 9×10−4 9×10−4 9×10−4 5×10−4

Source GPT-2 transfer GPT-2 transfer GPT-2 transfer GPT-2 transfer GPT-2 transfer standard

Table 16: Ablation protocols. Each experiment uses three distinct seeds. Experiment

Optimizers

Shared training protocol

Hyperparameter policy

GPT-2 norm–direction intervention

Muon RMS, Muon + NormProj, RODE with fixed radius, full RODE

RODE component ablation

RODE, RODE-SNR, fixed-radius control, radial-only control, w/o Newton–Schulz, w/o geodesic

standard GPT-2/WikiText-103; 8000 steps; effective batch 128; 100 warmup steps; cosine decay; three distinct seeds Qwen2-style LM/WikiText-103; 114M parameters; 4000 steps; effective batch 128; 200 warmup steps; cosine decay; three distinct seeds

Muon + NormProj inherits Muon RMS rates; the fixed-radius control inherits RODE rates and holds each matrix radius at initialization RODE-SNR uses its independently swept row in Table 11; structural variants inherit RODE’s learning rates

17

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

C

A P REPRINT

Additional Experimental Diagnostics and Curves

This appendix collects supporting diagnostics and full trajectories that complement the main results. The same plotting code and color mapping are used as in the main figures.

C.1

Qwen2-style LM validation trajectories

Full trajectory RODE RODE-SNR SOAP MARS-M

RODE RODE-SNR SOAP MARS-M

3.3

4.5

Validation loss

Validation loss

5.0

Late-stage detail

3.4

Muon RMS Muon Original AdamW AdEMAMix

4.0 3.5

Muon RMS Muon Original AdamW AdEMAMix

3.2 3.1 3.0 2.9 2.8

3.0

2.72000

500 1000 1500 2000 2500 3000 3500 4000 Training step

2500

3000 Training step

3500

4000

Figure 5: Qwen2-style LM/WikiText-103 validation trajectories over 4000 steps (three-seed mean ± sample SD): full trajectory (left) and late-stage detail (right).

C.2

Metric definitions

Diagnostics include every trainable parameter, including parameters delegated to AdamW. For the diagnostic parameter set Pdiag , 1/2

 ∥Wt ∥global = 

X

∥pt ∥2F 

,

(19)

p∈Pdiag

1/2

 ∥∆Wt ∥global = 

X

∥pt+1 − pt ∥2F 

.

(20)

p∈Pdiag

The update ratio divides the latter by ∥Wt ∥global + 10−12 . Gradient norm is the analogous root-sum-square immediately before the optimizer step. Only high-frequency training, gradient, and update diagnostics are EMA-smoothed for visualization; validation, accuracy, weight norm, schedules, and tabulated values use raw logs.

C.3

1.5B scale-transfer protocol

The 1.5B runs use sequence length 1024, effective batch size 128, 200 warmup steps, cosine decay, and learning rates selected on the smaller Qwen2-style LM. Fixed-radius RODE uses the same directional and delegated-parameter rates as full RODE, sets ηρ = 0, and holds each managed matrix’s radius at initialization. All three methods use the same three distinct seeds for 20000 steps. The fixed evaluation cache contains 1024 length-1024 sequences built from FineWeb sample-10BT; document-level exclusion from the sample-100BT training stream is not enforced [HuggingFaceFW, 2025]. Every optimizer is evaluated on the identical cache. Figure 3 reports cache loss and global norm; Figure 6 reports dense training diagnostics. Raw per-seed scalars and provenance are archived with the paper. 18

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

a

b

Training Loss

Gradient L2 Norm

Muon RMS (n=3) RODE (n=3) RODE (fixed radius) (n=3)

Norm (log scale)

10

Loss

A P REPRINT

8

6

101 Muon RMS (n=3) RODE (n=3) RODE (fixed radius) (n=3)

100

4 0

5k

10k Step

15k

20k

0

5k

10k Step

15k

20k

Figure 6: Qwen2-style 1.5B/FineWeb training diagnostics over 20000 steps (three seeds per method): EMA-smoothed training loss (left) and gradient norm (right). Gradient norm uses a logarithmic axis. C.4

Qwen3.5-9B tuning and training diagnostics Qwen3.5-9B full SFT on GSM8K

Validation loss

0.46

RODE SOAP Muon RMS

Qwen3.5-9B hyperparameter sweeps

Muon Original AdamW Adafactor

Best validation loss so far

0.48

0.44 0.42 0.40 0.38 0.36

RODE SOAP Muon RMS

2.0

Muon Original AdamW Adafactor

1.5

1.0

0.5

0.34 500

1000

1500 2000 Update step

2500

3000

1

5

10 15 Completed Optuna trials

20

Figure 7: Qwen3.5-9B full-parameter fine-tuning diagnostics for the six optimizers in Table 5. Left: validation loss over the 3000-step formal runs (three-seed mean ± sample SD). Right: cumulative best objective over the 20 800-step Optuna trials for each optimizer; each trial’s objective is its minimum validation loss. Both panels use raw, unsmoothed values. GPT-2/WikiText-103 dynamics. Figure 8 shows the training loss, update step size, and gradient norm for the GPT-2 runs; the GPT-2 weight norm is in the main text (Figure 2). ResNet-50 evaluation loss and weight norm. Figures 9 and 10 show test loss for CIFAR-100, validation loss for ImageNet-1K, and observed full-model weight norm for both datasets.

19

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

GPT-2 on WikiText-103 Muon RMS Muon Original AdamW AdEMAMix

RODE RODE-SNR SOAP MARS-M

8 Update step size

Training loss

GPT-2 on WikiText-103

RODE RODE-SNR SOAP MARS-M

10

A P REPRINT

8 6

6

Muon RMS Muon Original AdamW AdEMAMix

4 2

4 0

2000 4000 6000 Epoch / evaluation step

0

8000

0

2000 4000 6000 Epoch / evaluation step

8000

GPT-2 on WikiText-103 RODE RODE-SNR SOAP MARS-M

Gradient L2 norm

5 4

Muon RMS Muon Original AdamW AdEMAMix

3 2 1 00

2000 4000 6000 Epoch / evaluation step

8000

Figure 8: GPT-2/WikiText-103 training dynamics (three seeds): training loss, update step size, and gradient norm.

ResNet-50 on CIFAR-100

8 6 5

Validation loss

7 Test loss

ResNet-50 on ImageNet-1K 5

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

4 3

4 3 2

2 1

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

0

20

40 60 Epoch / evaluation step

80

100

0

20

40 60 80 Epoch / evaluation step

100

Figure 9: ResNet-50 test loss on CIFAR-100 (left) and validation loss on ImageNet-1K (right), shown as three-seed mean ± sample SD.

20

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

ResNet-50 on CIFAR-100

5000

A P REPRINT

ResNet-50 on ImageNet-1K

1200 4000 RODE RODE-SNR Muon RMS Muon Original AdamW SGD

800 600 400

Weight norm

Weight norm

1000

3000 2000

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

1000

200 00

00

10000 20000 30000 Epoch / evaluation step

100000 200000 300000 400000 500000 Epoch / evaluation step

Figure 10: Observed full-model weight norms on CIFAR-100 (left) and ImageNet-1K (right), shown as three-seed mean ± sample SD. RODE directly updates managed-matrix radii; these curves additionally include parameters delegated to AdamW.

ResNet-50 on CIFAR-100

ResNet-50 on CIFAR-100 7

0.6 RODE RODE-SNR Muon RMS Muon Original AdamW SGD

0.4 0.2 0.0 0

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

6

0.8 Training loss

Training accuracy

1.0

20

40 60 80 Epoch / evaluation step

5 4 3 2 1 0

100

0

ResNet-50 on CIFAR-100 80 60

ResNet-50 on CIFAR-100 RODE RODE-SNR Muon RMS Muon Original AdamW SGD

0.025 Update / weight ratio

Gradient L2 norm

100

40 20 00

10000 20000 30000 Epoch / evaluation step

0.020 0.015

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

0.010 0.005 0.000 0

10000 20000 30000 Epoch / evaluation step

10000 20000 30000 Epoch / evaluation step

Figure 11: Additional ResNet-50/CIFAR-100 curves: training accuracy, training loss, gradient norm, and update-toweight ratio.

21

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

ResNet-50 on ImageNet-1K

1.0

ResNet-50 on ImageNet-1K 7

0.2

40

Gradient L2 norm

Training loss

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

30

20

40 60 80 Epoch / evaluation step

5 4 3 2 1 0

100

0

ResNet-50 on ImageNet-1K

100000 200000 300000 400000 500000 Epoch / evaluation step

ResNet-50 on ImageNet-1K 0.0175

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

0.0150 Update / weight ratio

Training accuracy

0.6

0.0 0

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

6

0.8

0.4

A P REPRINT

20 10

0.0125 0.0100

RODE RODE-SNR Muon RMS Muon Original AdamW SGD

0.0075 0.0050 0.0025

00

0.0000 0

100000 200000 300000 400000 500000 Epoch / evaluation step

100000 200000 300000 400000 500000 Epoch / evaluation step

Figure 12: Additional ResNet-50/ImageNet-1K curves from the exported per-step diagnostics: three-seed means with sample-standard-deviation bands for training accuracy, training loss, gradient norm, and update-to-weight ratio. Main ImageNet validation results are reported in Table 1 and Figure 4.

22

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

D

A P REPRINT

Detailed Global Convergence Proof

This appendix expands the convergence-neighborhood statement in Section 4. We derive the geodesic displacement, split radial and tangent descent, bound the tangent tracking error, and combine the potential and kinetic inequalities with a Lyapunov function. D.1

Notation and algorithmic equations

Let f (W ) : Rm×n → R be the objective for a matrix parameter. For clarity assume m ≥ n; if m < n, the implementation transposes the matrix before applying Newton–Schulz and transposes the result back. The Frobenius inner product is p (21) ⟨A, B⟩F = tr(A⊤ B), ∥A∥F = ⟨A, A⟩F . At iteration t, RODE writes ρt = ∥Wt ∥F ,

Wt = ρt Ut ,

Ut =

Wt , ρt

∥Ut ∥F = 1.

(22)

Let Ft denote the history immediately before sampling the stochastic gradient at step t, and write Et [·] = E[· | Ft ]. Let Gt = ∇f (Wt ) denote the full gradient and let gt denote a stochastic gradient satisfying Et [gt ] = Gt . We decompose both into radial and tangent components: Gρt = ⟨Gt , Ut ⟩F ,

ρ G⊥ t = Gt − Gt Ut ,

(23)

gtρ = ⟨gt , Ut ⟩F ,

gt⊥ = gt − gtρ Ut .

(24)

Thus Wt , Ut , ρt , and Gt are fixed under Et . Unsubscripted E denotes total expectation; below we pass from conditional one-step inequalities to total expectations using the tower property. The radial update is ρt+1 = ρt − ηρ gtρ .

(25)

The implementation uses the guarded scalar update ρt+1 = max{ρt − ηρ gtρ , κfloor ρt }.

(26)

The proof below analyzes the interior regime in which this floor is inactive. If the floor activates, the radial channel becomes a projected one-dimensional step and the radial descent inequality must be replaced by the corresponding projected-gradient inequality; we do not claim that branch in this appendix. The implemented optimizer stores a raw buffer rather than only a tangent buffer: Bt = βPt (Bt−1 ) + gt ,

⊥ Mraw,t = Pt (Bt ).

(27)

At the current step this gives the same tangent direction as projecting the newly formed buffer. Across steps, however, the radial part retained in Bt can re-enter the next tangent space through Pt+1 (Bt ). Write ⊥ Bt = Mraw,t + at Ut ,

at = ⟨Bt , Ut ⟩F .

(28)

⊥ ⊥ = βPt+1 (Mraw,t ) + gt+1 + rt+1 ,

(29)

rt+1 = βat Pt+1 (Ut ).

(30)

Then the next projected raw tangent buffer obeys ⊥ Mraw,t+1 = Pt+1 (Bt+1 )

with the cross-step radial leakage residual The implementation stores the unnormalized momentum above. For the tracking argument we analyze its normalized counterpart ⊥ Mt⊥ = (1 − β)Mraw,t , r̄t+1 = (1 − β)rt+1 , (31) so that ⊥ ⊥ Mt+1 = βPt+1 (Mt⊥ ) + (1 − β)gt+1 + r̄t+1 .

(32)

This rescaling does not change the Newton–Schulz direction because the engine starts from M/∥M ∥F . The implemented raw buffer therefore induces an additional cross-step leakage residual; below we bound its squared contribution and absorb it into the curvature-order residual. From here on, Mt⊥ denotes the normalized analysis momentum, and 23

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

the appendix analyzes the nondegenerate, inactive-floor raw-buffer branch rather than the zero-norm safeguard or active-floor branches. The projected Newton–Schulz engine starts from X0 =

Mt⊥ ∥Mt⊥ ∥F

(33)

in the nondegenerate branch and, matching the default implementation, applies tangent reprojection after each quintic polynomial step:
Yk+1 = 3.4445Xk + Xk −4.7750Xk⊤ Xk + 2.0315(Xk⊤ Xk )2 , (34) Xk+1 = Yk+1 − ⟨Yk+1 , Ut ⟩F Ut ,

k = 0, . . . , K − 1.

(35)

After K projected steps, define b t = XK . Q

(36)

Define its actual post-projection norm b t ∥F . qt = ∥Q

(37) bt Projection removes the component parallel to Ut , so qt is generally different from n. The proof assumes that Q retains a constant fraction of the polar-target alignment with Mt⊥ and has nondegenerate projected norm, so that for some cNS ∈ (0, 1], b t ⟩F ≥ cNS ∥Mt⊥ ∥∗ , ⟨Mt⊥ , Q 0 < qmin ≤ qt ≤ qmax < ∞. (38) Define the unit tangent direction bt Q , ⟨Dt⊥ , Ut ⟩F = 0. (39) Dt⊥ = qt Default RODE uses no scalar damping, i.e. γt = 1 and γmin = 1. The proof also covers any optional scalar damping satisfying 0 < γmin ≤ γt ≤ 1, including the RODE-SNR ablation. The geodesic angle is √

θt = ηdir γt qt .

(40)

Ut+1 = Ut cos θt − Dt⊥ sin θt , Wt+1 = ρt+1 Ut+1 .

(41) (42)

Finally,

D.2

Assumptions

Assumption 1 (Smoothness). For all matrices X, Y , L ∥Y − X∥2F . (43) 2 Assumption 2 (Unbiased noise with bounded variance). The stochastic gradient is unbiased, and the radial and tangent noises are bounded: f (Y ) ≤ f (X) + ⟨∇f (X), Y − X⟩F +

Et [gtρ − Gρt ] = 0,

Et [(gtρ − Gρt )2 ] ≤ σρ2 ,

(44)

Et [gt⊥ − G⊥ t ] = 0,

2 2 Et [∥gt⊥ − G⊥ t ∥F ] ≤ σ⊥ .

(45)

Assumption 3 (Bounded physical states). There exist constants such that 0 < ρmin ≤ ρt ≤ ρmax ,

|Gρt | ≤ Gmax ,

∥Mt⊥ ∥F ≤ Mmax ,

|at | ≤ Amax

for some Mmax , Amax > 0.

(46)

Here at = ⟨Bt , Ut ⟩F is the radial component retained in the implemented raw buffer. Assumption 4 (Local tangent-gradient regularity). Let Pt+1 (X) = X − ⟨X, Ut+1 ⟩F Ut+1 . For the small geodesic moves considered here,   ⊥ 2 2 2 E ∥Pt+1 (G⊥ (47) t ) − Gt+1 ∥F ≤ L E[∥∆Wt ∥F ], where ∆Wt = Wt+1 − Wt . Assumption 5 (Aligned projected Newton–Schulz kernel). After the finite projected Newton–Schulz iterations, the constant-step engine returns a tangent kernel satisfying, for some cNS ∈ (0, 1], b t ⟩F ≥ cNS ∥Mt⊥ ∥∗ , ⟨Mt⊥ , Q

b t ∥F ≤ qmax < ∞. 0 < qmin ≤ ∥Q

(48)

This assumption isolates the optimizer dynamics from finite-iteration polynomial approximation errors, projectioninduced alignment loss, and reduced-precision effects, and excludes degenerate steps where tangent projection nearly annihilates the conditioned kernel. The exact polar identity is recovered in the ideal case cNS = 1. 24

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Assumption 6 (Bounded geodesic angle). There exists θmax ∈ (0, π/4] such that ηdir ≤

θmax . qmax

(49)

Since γt ≤ 1, this gives θt ≤ θmax and ensures the global lower bound sin(θt ) ≥ kθt , D.3

√ 2 2 sin(θmax ) ≥ . k= θmax π

(50)

Raw-buffer leakage bound

The geodesic update gives Ut+1 = Ut cos θt − Dt⊥ sin θt ,

⟨Ut , Dt⊥ ⟩F = 0.

(51)

Hence ⟨Ut , Ut+1 ⟩F = cos θt and ∥Pt+1 (Ut )∥2F = 1 − ⟨Ut , Ut+1 ⟩2F = sin2 θt .

(52)

Using θt = ηdir γt qt , γt ≤ 1, and qt ≤ qmax , ∥Pt+1 (Ut )∥F ≤ θt ≤ ηdir qmax .

(53)

Therefore the normalized raw-buffer leakage residual satisfies 2 2 2 ∥r̄t+1 ∥2F ≤ (1 − β)2 β 2 A2max qmax ηdir =: Kleak ηdir .

D.4

(54)

Exact displacement expansion and second-order bound

The physical displacement is ∆Wt = Wt+1 − Wt = ρt+1 (Ut cos θt − Dt⊥ sin θt ) − ρt Ut .

(55)

Collecting the orthogonal basis terms Ut and Dt⊥ gives ∆Wt = (ρt+1 cos θt − ρt )Ut − ρt+1 sin θt Dt⊥ .

(56)

Because ⟨Ut , Dt⊥ ⟩F = 0 and both directions have unit Frobenius norm, ∥∆Wt ∥2F = (ρt+1 cos θt − ρt )2 + ρ2t+1 sin2 θt = ρ2t+1 cos2 θt − 2ρt+1 ρt cos θt + ρ2t + ρ2t+1 sin2 θt = ρ2t+1 + ρ2t − 2ρt+1 ρt cos θt = (ρt+1 − ρt )2 + 2ρt+1 ρt (1 − cos θt ).

(57) (58) (59) (60)

Using 1 − cos θ ≤ θ2 /2, ∥∆Wt ∥2F ≤ (ρt+1 − ρt )2 + ρt+1 ρt θt2 .

(61)

Substituting ρt+1 − ρt = −ηρ gtρ and θt = ηdir γt qt , 2 ∥∆Wt ∥2F ≤ ηρ2 (gtρ )2 + ρt+1 ρt ηdir γt2 qt2 .

(62)

By Assumptions 3–5, γt ≤ 1, and qt ≤ qmax , 2 2 ∥∆Wt ∥2F ≤ ηρ2 (gtρ )2 + ρ2max qmax ηdir .

(63)

Taking conditional expectation and using Et [(gtρ )2 ] = (Et [gtρ ])2 + Vart (gtρ ) ≤ (Gρt )2 + σρ2 ,

(64)

we obtain the second-order smoothness contribution 2 L L Lρ2max qmax L 2 Et [∥∆Wt ∥2F ] ≤ ηρ2 (Gρt )2 + ηρ2 σρ2 + ηdir . 2 2 2 2

25

(65)

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

D.5

A P REPRINT

First-order descent: radial part

The first-order term in smoothness is ⟨Gt , ∆Wt ⟩F .

(66)

Using Gt = Gρt Ut + G⊥ t and the displacement expansion, ⊥ ⟨Gt , ∆Wt ⟩F = Gρt (ρt+1 cos θt − ρt ) − ρt+1 sin θt ⟨G⊥ t , Dt ⟩F .

(67)

For the radial component, Iρ = Gρt (ρt+1 cos θt − ρt )

(68)

= Gρt (ρt+1 − ρt ) + Gρt ρt+1 (cos θt − 1)

(69)

= −ηρ Gρt gtρ − ρt+1 Gρt (1 − cos θt ).

(70)

Taking conditional expectation, the first term becomes Et [−ηρ Gρt gtρ ] = −ηρ Gρt Et [gtρ ] = −ηρ (Gρt )2 .

(71)

For the curvature contamination term, −ρt+1 Gρt (1 − cos θt ) ≤ |ρt+1 Gρt |(1 − cos θt ) ≤ ρmax Gmax

θt2

2 2 Gmax ρmax qmax 2 ηdir . ≤ 2

(72) (73) (74)

The last line uses θt = ηdir γt qt , γt ≤ 1, and the projected-kernel bound qt ≤ qmax . Therefore Et [Iρ ] ≤ −ηρ (Gρt )2 + D.6

2 Gmax ρmax qmax 2 ηdir . 2

(75)

First-order descent: tangent part

The tangent component in (67) is ⊥ I⊥ = −ρt+1 sin θt ⟨G⊥ t , Dt ⟩F .

(76)

⊥ ⊥ ⊥ ⊥ b Insert the identity G⊥ t = Mt − (Mt − Gt ) and use Dt = Qt /qt :

sin θt b t ⟩F + ρt+1 sin θt ⟨Mt⊥ − G⊥ b ⟨Mt⊥ , Q t , Qt ⟩F qt qt sin θt sin θt b ≤ −ρt+1 cNS ∥Mt⊥ ∥∗ + ρt+1 ⟨Mt⊥ − G⊥ t , Qt ⟩F , qt qt

I⊥ = −ρt+1

(77) (78)

where the second line uses Assumption 5; the negative coefficient turns the alignment lower bound into an upper bound on I⊥ . By Assumption 6, sin θt ≥ kθt for the negative term. The remaining inner product can have either sign, so we first upper-bound it by its absolute value and then use sin θt ≤ θt . Since θt = ηdir γt qt , the actual post-projection norm cancels: b I⊥ ≤ −kρt+1 ηdir cNS γt ∥Mt⊥ ∥∗ + ρt+1 ηdir γt ⟨Mt⊥ − G⊥ (79) t , Qt ⟩F . We next lower bound the possibly damped and finite-NS-aligned nuclear norm. Since γt ≥ γmin , ∥M ∥∗ ≥ ∥M ∥F , and ∥Mt⊥ ∥F ≤ Mmax , cNS γmin cNS γt ∥Mt⊥ ∥∗ ≥ cNS γmin ∥Mt⊥ ∥F ≥ ∥Mt⊥ ∥2F . (80) Mmax Using ρt+1 ≥ ρmin and defining kcNS ρmin γmin αdir = , (81) Mmax the main tangent term is bounded by −kρt+1 ηdir cNS γt ∥Mt⊥ ∥∗ ≤ −αdir ηdir ∥Mt⊥ ∥2F . 26

(82)

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

For the tracking-error term, Cauchy-Schwarz gives ⊥ ⊥ b b ρt+1 ηdir γt ⟨Mt⊥ − G⊥ t , Qt ⟩F ≤ ρmax ηdir ∥Mt − Gt ∥F ∥Qt ∥F

(83)

≤ ρmax qmax ηdir ∥Mt⊥ − G⊥ t ∥F .

(84)

Applying Young’s inequality ab ≤ a /2 + b /2 with a = ρmax qmax ηdir and b = ∥Mt⊥ − G⊥ t ∥F , 2

2

1 1 2 2 2 2 ρmax qmax ηdir + ∥Mt⊥ − G⊥ t ∥F . 2 2

(85)

1 2 1 2 2 2 Et [I⊥ ] ≤ −αdir ηdir Et [∥Mt⊥ ∥2F ] + Et [∥Mt⊥ − G⊥ t ∥F ] + ρmax qmax ηdir . 2 2

(86)

≤ Taking conditional expectation,

D.7

Potential-energy inequality

By smoothness, L Et [∥∆Wt ∥2F ]. 2 Substituting (65), (75), and (86), then taking total expectation and using the tower property, we obtain   L 2 E[f (Wt+1 )] − E[f (Wt )] ≤ − ηρ − ηρ E[(Gρt )2 ] − αdir ηdir E[∥Mt⊥ ∥2F ] 2 1 L 2 2 2 2 + E[∥Mt⊥ − G⊥ σ η , t ∥F ] + C1 ηdir + 2 2 ρ ρ where 1 L 1 2 2 2 C1 = ρ2max qmax + ρ2max qmax + Gmax ρmax qmax . 2 2 2 If ηρ < 2/L, then ηρ − L2 ηρ2 > 0. Et [f (Wt+1 )] − f (Wt ) ≤ Et [⟨Gt , ∆Wt ⟩F ] +

D.8

(87)

(88) (89)

Tangent tracking-error dynamics

Define the tangent tracking error 2 Et⊥ = E[∥Mt⊥ − G⊥ (90) t ∥F ]. ⊥ Here Mt is the normalized analysis momentum defined in Appendix D; the implemented projected raw tangent buffer ⊥ Mraw,t differs from it only by the positive scalar (1 − β)−1 before Newton–Schulz normalization. For the actual

raw-buffer recurrence, the normalized analysis momentum obeys ⊥ ⊥ Mt+1 = βPt+1 (Mt⊥ ) + (1 − β)gt+1 + r̄t+1 . ⊥ Subtract G⊥ t+1 and add/subtract Pt+1 (Gt ): ⊥ ⊥ ⊥ ⊥ ⊥ Mt+1 − G⊥ t+1 = βPt+1 (Mt − Gt ) + β(Pt+1 (Gt ) − Gt+1 ) ⊥ + (1 − β)(gt+1 − G⊥ t+1 ) + r̄t+1 .

(91)

(92)

Let xt = βPt+1 (Mt⊥ − G⊥ t ),

⊥ yt = β(Pt+1 (G⊥ t ) − Gt+1 ),

(93)

⊥ zt = (1 − β)(gt+1 − G⊥ t+1 ),

wt = r̄t+1 .

(94)

Using   1 (∥y∥2 + ∥z∥2 + ∥w∥2 ), ∥x + y + z + w∥ ≤ (1 + δ)∥x∥ + 3 1 + δ 2

2

(95)

we obtain ⊥ 2 Et+1 ≤ β 2 (1 + δ)E[∥Pt+1 (Mt⊥ − G⊥ t )∥F ]   1 ⊥ 2 + 3β 2 1 + E[∥Pt+1 (G⊥ t ) − Gt+1 ∥F ] δ   1 2 + 3(1 − β)2 1 + σ⊥ δ   1 +3 1+ E[∥r̄t+1 ∥2F ]. δ

27

(96)

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Orthogonal projection is non-expansive: 2 ⊥ ⊥ 2 ∥Pt+1 (Mt⊥ − G⊥ t )∥F ≤ ∥Mt − Gt ∥F .

(97)

  1+β 1−β =β < β, β 2 (1 + δ) = β 2 1 + 2β 2

(98)

Choose δ = (1 − β)/(2β). Then

and

    1 2β 1+β 6 3 1+ =3 1+ =3 ≤ . δ 1−β 1−β 1−β By Assumption 4, the displacement bound, and (54), ⊥ Et+1 ≤ βEt⊥ +

6Kleak 2 6β 2 L2 2 2 2 E[ηρ2 (gtρ )2 + ρ2max qmax ηdir ] + 6(1 − β)σ⊥ + η . 1−β 1 − β dir

(99)

(100)

Using E[(gtρ )2 ] ≤ E[(Gρt )2 ] + σρ2 , 6β 2 L2 2 2 η E[(Gρt )2 ] + C2 ηdir 1−β ρ 6β 2 L2 2 2 2 σ η , + 6(1 − β)σ⊥ + 1−β ρ ρ

⊥ Et+1 − Et⊥ ≤ − (1 − β)Et⊥ +

where C2 = D.9

2 6Kleak 6β 2 L2 ρ2max qmax + . 1−β 1−β

(101)

(102)

Lyapunov synthesis and convergence neighborhood

Define the Lyapunov function Φt = E[f (Wt )] + c⊥ Et⊥ .

(103)

Choose

1 , (104) 1−β so that the Et⊥ coefficient after combining (88) and c⊥ times (101) becomes 1 1 − c⊥ (1 − β) = − . (105) 2 2 The radial coefficient becomes   L L 6β 2 L2 2 6β 2 L2 2 − ηρ − ηρ2 + c⊥ ηρ = −ηρ + ηρ2 + η (106) 2 1−β 2 (1 − β)2 ρ   6β 2 L2 L = −ηρ 1 − ηρ − η (107) ρ . 2 (1 − β)2 For sufficiently small ηρ , the parenthesized term is positive. Denote the resulting positive constant by α bρ > 0. 2 Combining all constant terms, including the raw-buffer leakage contribution c⊥ C2 ηdir , into Kcurv , Kn⊥ , and Knρ gives 1 Φt+1 − Φt ≤ − α bρ ηρ E[(Gρt )2 ] − αdir ηdir E[∥Mt⊥ ∥2F ] − Et⊥ 2 2 2 + Kcurv ηdir + Kn⊥ σ⊥ + Knρ σρ2 ηρ2 . (108) Summing from t = 1 to T ,  T  X 1 ⊥ ρ 2 ⊥ 2 2 ⊥ 2 ρ 2 2 ΦT +1 − Φ1 ≤ −b αρ ηρ E[(Gt ) ] − αdir ηdir E[∥Mt ∥F ] − Et + Kcurv ηdir + Kn σ⊥ + Kn σρ ηρ . (109) 2 t=1 c⊥ =

Since f is lower bounded and Et⊥ ≥ 0, there is a finite Φinf with Φt ≥ Φinf . Moving the nonnegative descent terms to the left and using ΦT +1 ≥ Φinf gives the finite-time bound  T  1X 1 Φ1 − Φinf 2 2 α bρ ηρ E[(Gρt )2 ] + αdir ηdir E[∥Mt⊥ ∥2F ] + Et⊥ ≤ Kcurv ηdir + Kn⊥ σ⊥ + Knρ σρ2 ηρ2 + . (110) T t=1 2 T Taking lim supT →∞ removes the last term. 28

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Theorem 1 (Interior-regime decoupled convergence neighborhood). Under Assumptions 1–6, sufficiently small ηρ , inactive radial floors, and the nondegenerate raw-buffer recurrence above, the time-averaged radial gradient, normalized tangent momentum, and tangent tracking error admit the following asymptotic upper bounds: T

lim sup T →∞

2 1X Kcurv ηdir K⊥ σ 2 Kρ E[(Gρt )2 ] ≤ + n ⊥ + n ηρ σρ2 =: ϵρ , T t=1 α b ρ ηρ α b ρ ηρ α bρ

(111)

T

K⊥ σ 2 Kρ ηρ2 2 1X Kcurv lim sup E[∥Mt⊥ ∥2F ] ≤ ηdir + n ⊥ + n σ , αdir αdir ηdir αdir ηdir ρ T →∞ T t=1

(112)

T

T →∞

1X ⊥ 2 2 E ≤ 2Kcurv ηdir + 2Kn⊥ σ⊥ + 2Knρ σρ2 ηρ2 . T t=1 t

T



lim sup

(113)

Consequently, 1X 2 E[∥G⊥ lim sup t ∥F ] ≤ O(ηdir ) + O T →∞ T t=1

2 σ⊥ ηdir

 +O

ηρ2 2 σ ηdir ρ

! =: ϵ⊥ .

(114)

bρ ηρ , αdir ηdir , and 1/2, respectively. For the final true Proof. The first three bounds are obtained by dividing (110) by α tangent-gradient bound, use 2 ⊥ ⊥ ⊥ 2 ⊥ 2 ⊥ ⊥ 2 ∥G⊥ t ∥F = ∥Mt − (Mt − Gt )∥F ≤ 2∥Mt ∥F + 2∥Mt − Gt ∥F ,

(115)

then take expectation and substitute the tangent-momentum and tracking-error bounds above.

E

PL Recursion for an Aligned First-Order Model

This appendix analyzes the first-order model Wt+1 = Wt − ηHt for RODE’s effective displacement. It is separate from the exact raw-buffer recurrence proved in Appendix D: finite geodesic curvature, unequal coordinate stepsizes, the radial floor, and finite projected Newton–Schulz error must be absorbed into Ht or controlled as residual terms before this model applies. Within this scope, boundedness, effective alignment, smoothness, and the PL condition yield linear convergence to a stepsize-controlled neighborhood. E.1

Additional assumptions for PL analysis

Assumption 7 (PL smooth setting). The objective is L-smooth and satisfies the Polyak-Lojasiewicz condition 1 ∥∇f (Wt )∥2F ≥ µ(f (Wt ) − f ∗ ). (116) 2 Assumption 8 (Bounded effective direction). The full effective update direction Ht satisfies ∥Ht ∥F ≤ G. Assumption 9 (Local reprojected tangent-gradient regularity). For the momentum history used in the PL argument, let Gtan denote the tangent gradient from step i reprojected into the current tangent space at Ut , and let Gtan = G⊥ t . i There is a constant Ltan such that ∥Gtan − Gtan ∥F ≤ Ltan ∥Wi − Wt ∥F . (117) i This local condition accounts for both ordinary gradient variation and projection drift from moving tangent spaces; it is not implied by Euclidean L-smoothness alone. Assumption 10 (Bounded tangent gradient and local polar stability). Along the trajectory considered in the PL argument, ∥Gtan ∥F ≤ Gmax . Moreover, the projected polar map is locally Lipschitz around the tangent gradients: for the momentum tracking errors considered below, ∥Pt (Polar(Gtan + E)) − Pt (Polar(Gtan ))∥F ≤ Cpol ∥E∥F .

(118)

This assumption excludes degenerate points where the polar factor is unstable. Since Pt is non-expansive, the displayed condition follows from the corresponding local Lipschitz bound for the unprojected polar map; the global-neighborhood proof above instead isolates polar accuracy through Assumption 5. Assumption 11 (Strong directional alignment). There exists α > 0 such that ⟨Gt , Ht ⟩F ≥ α∥Gt ∥2F .

(119)

The next two lemmas give sufficient alignment conditions for a projected-polar first-order model. They do not establish the condition for every finite-step implementation branch; Theorem 2 assumes effective alignment directly. 29

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

E.2

A P REPRINT

Radial alignment

Lemma 1 (Radial alignment). The radial update direction has positive alignment with the radial gradient. Proof. Write Wt = ρt Ut . The radial gradient vector is Grad = ⟨Gt , Ut ⟩F Ut .

(120)

The ideal radial descent direction is this gradient component; the outer PL recursion applies the scalar stepsize separately: Hrad = Grad . (121) Therefore ⟨Grad , Hrad ⟩F = ∥Grad ∥2F .

(122)

⟨Grad , Hrad ⟩F ≥ Krad ∥Grad ∥2F .

(123)

Let Krad = 1. Then

E.3

Tangent alignment

Lemma 2 (Tangent alignment in the projected-polar model). Under Assumptions 8–10 and sufficiently small learning rate, the projected-polar tangent step in the first-order model remains positively aligned with the true tangent gradient. The same result holds for scalar damping bounded below by a positive constant. Proof. Let Gtan denote the tangent component of Gt at the current direction Ut . The tangent update is based on momentum. To show that momentum preserves alignment, use the projection-based momentum form of the EMA: Gtan denotes a past tangent gradient reprojected into the current tangent space, and i t X

wi Gtan i ,

(124)

(1 − β)β t−i , i ≥ 1, βt, i = 0.

(125)

Mt =

i=0

where, with initialization at step 0,  wi = The weights sum to one: t X

t X

! + βt

(126)

= (1 − β t ) + β t = 1. P Define the momentum tracking error Et = Mt − Gtan . Because i wi = 1,

(127)

wi =

i=0

(1 − β)β

t−i

i=1

Et = =

t X i=0 t X

wi Gtan − i

t X

wi Gtan

(128)

i=0

wi (Gtan − Gtan ). i

(129)

i=0

By the triangle inequality, ∥Et ∥F ≤

t X

wi ∥Gtan − Gtan ∥F . i

(130)

i=0

Using Assumption 9, ∥Gtan − Gtan ∥F ≤ Ltan ∥Wi − Wt ∥F . i 30

(131)

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

If each update has norm at most G and learning rate η, then t X

∥Wi − Wt ∥F =

ηHk

k=i+1 t X

≤

(132) F

η∥Hk ∥F

(133)

k=i+1

≤ (t − i)ηG. Thus ∥Et ∥F ≤ Ltan Gη

t X

(134)

wi (t − i).

(135)

i=0

Let S=

t X

wi (t − i) = tβ t + (1 − β)

i=0

For A =

Pt−1

k=1 kβ

k

t−1 X

kβ k .

(136)

k=1

, A = β + 2β 2 + · · · + (t − 1)β t−1 ,

(137)

βA = β 2 + 2β 3 + · · · + (t − 2)β t−1 + (t − 1)β t .

(138)

(1 − β)A = β + β 2 + · · · + β t−1 − (t − 1)β t

(139)

and Subtracting, t−1

=

β(1 − β 1−β

)

− (t − 1)β t .

(140)

Substituting into S, β − βt − tβ t + β t 1−β β β − β t+1 < . = 1−β 1−β

S = tβ t +

Therefore ∥Et ∥F ≤ Ltan Gη

β = C1 η. 1−β

(141) (142)

(143)

Now consider the projected polar factor. Let Ot = Pt (Polar(Mt )) = Pt (Polar(Gtan + Et )),

Og = Pt (Polar(Gtan )).

(144)

By Assumption 10, the polar factor is locally stable, so Ot = Og + ∆o ,

∥∆o ∥F ≤ Cpol ∥Et ∥F ≤ C1 Cpol η.

(145)

The first-order tangent contribution to the effective direction is Htan = γt Ot , which is tangent by construction because of the projection Pt . For default RODE, γt = 1; for an optional damped variant, assume 0 < γmin ≤ γt ≤ 1.

(146)

⟨Gtan , Htan ⟩F = γt (⟨Gtan , Og ⟩F + ⟨Gtan , ∆o ⟩F ) .

(147)

Then

For the projected polar target, since Gtan lies in the current tangent space, ⟨Gtan , Og ⟩F = ⟨Gtan , Polar(Gtan )⟩F = ∥Gtan ∥∗ ≥ ∥Gtan ∥F .

(148)

|⟨Gtan , ∆o ⟩F | ≤ ∥Gtan ∥F ∥∆o ∥F ≤ C1 Cpol η∥Gtan ∥F .

(149)

For the perturbation term,

31

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

Hence ⟨Gtan , Htan ⟩F ≥ γt (1 − C1 Cpol η)∥Gtan ∥F ≥ γmin (1 − C1 Cpol η)∥Gtan ∥F .

(150) (151)

If ∥Gtan ∥F ≤ Gmax , then ∥Gtan ∥F ≥ Therefore ⟨Gtan , Htan ⟩F ≥

∥Gtan ∥2F . Gmax

(152)

γmin (1 − C1 Cpol η) ∥Gtan ∥2F . Gmax

(153)

For η < 1/(C1 Cpol ), define Ktan =

γmin (1 − C1 Cpol η) > 0. Gmax

(154)

This proves tangent alignment. E.4

Orthogonal synthesis

Lemma 3 (Alignment synthesis in the first-order model). The radial and tangent alignment bounds imply Assumption 11. Proof. Decompose Gt = Grad + Gtan ,

Ht = Hrad + Htan .

(155)

⟨Gtan , Hrad ⟩F = 0,

(156)

The radial and tangent subspaces are orthogonal, so ⟨Grad , Htan ⟩F = 0, and ∥Gt ∥2F = ∥Grad ∥2F + ∥Gtan ∥2F .

(157)

⟨Gt , Ht ⟩F = ⟨Grad , Hrad ⟩F + ⟨Gtan , Htan ⟩F

(158)

≥ Krad ∥Grad ∥2F + Ktan ∥Gtan ∥2F .

(159)

Thus

Let K = min{Krad , Ktan } > 0. Then ⟨Gt , Ht ⟩F ≥ K(∥Grad ∥2F + ∥Gtan ∥2F ) = K∥Gt ∥2F .

(160)

Taking α = K proves the claim. E.5

PL linear convergence

Theorem 2 (PL recursion for an aligned first-order model). Assume smoothness, the PL condition, bounded updates, directional alignment for the first-order update Wt+1 = Wt − ηHt , and 0 < 2ηαµ < 1. Then this aligned effective update satisfies Kerr η f (Wt ) − f ∗ ≤ (1 − 2ηαµ)t (f (W0 ) − f ∗ ) + , (161) 2αµ where Kerr = L2 G2 . Proof. Consider the first-order effective update Wt+1 = Wt − ηHt .

(162)

By L-smoothness, f (Wt+1 ) ≤ f (Wt ) + ⟨∇f (Wt ), Wt+1 − Wt ⟩F + = f (Wt ) − η⟨Gt , Ht ⟩F +

32

L 2 η ∥Ht ∥2F . 2

L ∥Wt+1 − Wt ∥2F 2

(163) (164)

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

A P REPRINT

L 2 G . 2

(165)

Using ∥Ht ∥F ≤ G, f (Wt+1 ) ≤ f (Wt ) − η⟨Gt , Ht ⟩F + Kerr η 2 , Directional alignment gives Therefore

Kerr =

−η⟨Gt , Ht ⟩F ≤ −ηα∥Gt ∥2F .

(166)

f (Wt+1 ) ≤ f (Wt ) − ηα∥Gt ∥2F + Kerr η 2 .

(167)

f (Wt+1 ) − f ∗ ≤ f (Wt ) − f ∗ − ηα∥Gt ∥2F + Kerr η 2 .

(168)

∥Gt ∥2F ≥ 2µ(f (Wt ) − f ∗ ).

(169)

∗

Subtract f from both sides:

The PL condition implies Thus

f (Wt+1 ) − f ∗ ≤ f (Wt ) − f ∗ − 2ηαµ(f (Wt ) − f ∗ ) + Kerr η 2 ∗

2

= (1 − 2ηαµ)(f (Wt ) − f ) + Kerr η .

(170) (171)

∗

Let ∆t = f (Wt ) − f and ρ = 2ηαµ. The recursion is ∆t+1 ≤ (1 − ρ)∆t + Kerr η 2 .

(172)

Expanding the first few steps makes the geometric structure explicit: ∆1 ≤ (1 − ρ)∆0 + Kerr η 2 ,

(173)

2

2

2

∆2 ≤ (1 − ρ) ∆0 + (1 − ρ)Kerr η + Kerr η , 3

2

(174)

2

2

2

∆3 ≤ (1 − ρ) ∆0 + (1 − ρ) Kerr η + (1 − ρ)Kerr η + Kerr η .

(175)

By induction, ∆t ≤ (1 − ρ)t ∆0 + Kerr η 2

t−1 X

(1 − ρ)i .

(176)

i=0

Since

t−1 X 1 1 − (1 − ρ)t < , (1 − ρ)i = ρ ρ i=0

(177)

we obtain ∆t ≤ (1 − ρ)t ∆0 +

Kerr η 2 ρ

= (1 − 2ηαµ)t (f (W0 ) − f ∗ ) +

(178) Kerr η . 2αµ

(179)

This proves linear convergence to an O(η) neighborhood for the aligned first-order model. It does not extend the PL rate automatically to every branch of the finite implementation; Appendix D provides the separate convergence-neighborhood guarantee for the exact raw-buffer update under its stated assumptions.

33

Related documents

• RODE: A Radial-Orthogonal Decoupled Engine for Optimization#662204
• RODE: A Radial-Orthogonal Decoupled Engine for Optimization#661154
• Nonlinear Optimization#67412
• Bridging Spherical Black-Box Optimizers#307034
• Nonsmooth Optimization via Orthogonalized Momentum#1110366