arXiv CS open access

Approximate Muon with low-rank adapters

2026 · arxiv_cs

arXiv CS · Papers · 2026

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Approximate Muon with low-rank adapters

Ben Anson University of Bristol School of Mathematics ben.lja@proton.me

Conor Houghton University of Bristol School of Engineering Mathematics and Technology conor.houghton@bristol.ac.uk

arXiv:2608.14492v1 [cs.LG] 14 Aug 2026

Edward Milsom University of Bath Department of Computer Science em846@bath.ac.uk

Abstract The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficient finetuning (PEFT). One potential reason is that the most common PEFT method, LoRA, does not naturally combine with Muon since it is not mathematically possible to orthogonalize the weight update given by a low-rank parameterization. In this paper, we address this issue by approximating the solution to a relaxed Muon objective in the low-rank setting via linearization and then least-squares. We provide an efficient implementation that uses matmul operations only, as opposed to more complex linear algebra decomposition routines. Our method, sMuon (small Muon), performs favourably across SFT and a ReLoRA pretraining experiment. While results are model- and eval-dependent, we find overall that using Muon for low-rank fine-tuning provides moderate performance improvements.

Introduction

Muon [Jordan et al., 2024b] is a neural network optimizer which orthogonalizes updates at each layer, equalizing all singular values. This can be interpreted as steepest descent under the spectral norm [Bernstein and Newhouse, 2024b], thereby controlling the worst-case change to the layer as a linear operator, rather than viewing it as a flattened vector. Muon has demonstrated excellent results in the pretraining of LLMs, often outperforming AdamW, leading to its increasing adoption [Liu et al., 2025, Shah et al., 2025, Kimi Team, 2025, Team et al., 2026, Xu et al., 2026]. However, Muon is less commonly used for fine-tuning (FT), particularly PEFT (Parameter-Efficient Fine-Tuning, which reduces compute resource requirements). There are various reasons for this, including the fact that Muon FT has been shown to perform better with Muon-pretrained base models [Liu et al., 2025, Qu et al., 2026], which are currently less common than AdamW-pretrained models. Perhaps the most glaring issue is that Muon does not straightforwardly combine with LoRA [Hu et al., 2021], the dominant PEFT method. In principle, Muon requires that we orthogonalize the descent step for the whole layer, but this is not generally realizable in the LoRA parameterization. Some approximation is therefore inevitable. The simplest and most common workaround is ‘per-factor Muon’, which ignores the geometry of the whole layer and orthogonalizes changes to B and A separately [Qu et al., 2026, Janson et al., 2026, Kang et al., 2026]. Other works retain the whole layer’s geometry and introduce an approximation elsewhere, with Cesista et al. [2026] relaxing the Muon objective using the triangle inequality, and Bogachev et al. [2026] recasting the problem as Riemannian optimization on a low-rank manifold. Preprint.

1.0 0.8 0.6

σ /σ1

However, we feel that alternative approaches that more closely fit with Muon’s original ‘steepest descent under the spectral norm’ motivation are still underexplored.

0.4 In this paper, we linearize the low-rank Muon 0.2 objective to find the optimal descent step within the available parameter space. This solution ef0.0 10 20 30 40 50 60 fectively orthogonalizes the full layer’s gradient Singular Value Index and projects it onto the appropriate low-rank AdamW LoRA-Muon sMuon (ours) subspace. Empirically, we find this retains a Per-factor Muon Riemannion more uniform spectrum than alternatives such as per-factor Muon (Fig. 1) and LoRA-Muon [Cesista et al., 2026]. Furthermore, unlike Rie- Figure 1: Spectra of the weight change for difmannion [Bogachev et al., 2026], our approach ferent LoRA optimizers. We show the average avoids expensive matrix decompositions. While singular values of the weight updates (attention our solution to the low-rank Muon problem is weight, layer 6 of 12), during ReLoRA pretraining clean (Alg. 1), a naive implementation is compu- (r = 32). We normalize by σ 1 , and denote the tationally inefficient, as it requires orthogonal- LoRA rank by the dashed grey line. izing a large matrix. To address this, we introduce a mathematically equivalent formulation (Alg. 2) that is both fast and memory-efficient. By relying entirely on matrix multiplications, we are able to avoid the overhead from QR and SVD. Finally, we show that our optimizer, sMuon, achieves competitive performance across a range of commonsense and coding SFT benchmarks, as well as in ReLoRA pretraining experiments. i

Our contributions are as follows: • We derive a method for training LoRA adapters with Muon by taking a first-order approximation of the spectral-norm-regularized steepest descent objective. The method is available in closed form, and requires less optimizer state than AdamW. • We provide a matmul-only algorithm of the method, sMuon, that is efficient for modern hardware, with comparable speed to other Muon methods such as per-factor Muon and LoRA-Muon, while being significantly faster than Riemannion for larger LoRA rank. • We run SFT experiments on four base models, three AdamW-pretrained and one Muonpretrained, over eleven benchmarks to compare low-rank Muon optimizers. While results are model-dependent, we find that our optimizer gives strong performance, performing best on the Muon-pretrained model with the top accuracy in six out of eleven tasks with Moonlight-16B-A3B. • In the ReLoRA pretraining setting, we illustrate that Muon LoRA methods are generally stronger than AdamW LoRA. Specifically we find that our method attains the lowest final validation loss, tying with Riemannion, but being ∼ 30% faster.

Related Work

Muon and spectral steepest descent Muon orthogonalizes the gradient descent step for each weight matrix, so that the singular values of weight updates are equal [Jordan et al., 2024b]. This update is exactly steepest descent under the spectral norm [Bernstein and Newhouse, 2024a, Bernstein, 2025]. Muon has been shown to scale to billion- and trillion-parameter models for LLM pretraining, with considerable efficiency gains over AdamW [Liu et al., 2025, Kimi Team, 2025]. Muon for fine-tuning and optimizer mismatch Although Muon excels at pretraining, it is used far less for fine-tuning. Liu et al. [2025] note that fine-tuning an AdamW-pretrained model with Muon (or vice-versa) degrades performance. Qu et al. [2026] suggest that the differing implicit biases of Adam and Muon are to blame, but that constraining the update (e.g. via LoRA) narrows the gap between Muon and AdamW fine-tuning. Muon’s increasing adoption makes this question more important than ever, and so in this paper we evaluate on both AdamW- and Muon-pretrained base models. 2

Geometry-aware and Muon-based LoRA optimizers The most closely related works are Cesista et al. [2026] and Bogachev et al. [2026]; these both attempt to approximate a low-rank solution to the spectral-regularized gradient descent problem. LoRA-Muon [Cesista et al., 2026] is derived by approximating the spectral steepest-descent optimization problem using the triangle inequality applied to the spectral norm regularizer, and then minimizing to first order. We similarly linearize (we expect second-order terms to be suitably small), but we avoid the triangle inequality, as this is not necessarily tight under the spectral norm. For their ‘Riemannion’ optimizer, Bogachev et al. [2026] present the problem as Riemannian optimization on a constrained low-rank manifold. While some key ideas of this method are similar to our method (projecting onto the low-rank space), Riemannian optimization also involves a ‘retract’ step. One issue with their retraction is requiring a truncated SVD, which cannot be efficiently replaced with an efficient matmul-only approximation. In practice this leads to Riemannion being slower than alternatives, especially for larger LoRA rank. By contrast, we obtain the step using least squares on the orthogonalized gradient direction, avoiding expensive manifold constraints that don’t appear to benefit model performance. The PoLoRA [Ghosh et al., 2026] method is again similar to our work, but can be understood as a version of LoRA-Muon with additional preconditioning. LoRA-RITE [Yen et al., 2025] and Riemannian-preconditioned LoRA [Zhang and Pilanci, 2024] attempt to keep the adapters invariant to parameterization. Another popular approach is to simply apply Muon to B and A independently [Qu et al., 2026, Janson et al., 2026, Mo et al., 2025]. This ignores the geometry of the linear layer and causes the update to depend on the adapter parameterization, whereas our update is first-order parameterization-invariant (see App. A.3). Appendix C has a more detailed comparison of the various approaches. Low-rank optimization beyond LoRA fine-tuning Some works, such as Spectron [Janson et al., 2026], LORO [Mo et al., 2025], GaLore [Zhao et al., 2024], and LoRA-Pre [Wang et al., 2026] target efficient low-rank pretraining, either through stabilization, or optimizer state compression. He et al. [2025] accelerate Muon itself with a low-rank orthogonalization routine that exploits the low-rank structure of the gradient, but operate on full dense weight matrices rather than in the LoRA adapter setting. In contrast to all of these, our goal is a simple, principled, memory-efficient Muon update for the composite LoRA adapter ∆W = BA, rather than the full base weight W0 .

Background

3.1

Muon

Given a gradient matrix GW ∈ Rd1 ×d2 for weight matrix W ∈ Rd1 ×d2 , Muon selects the step direction δW ⋆ ∈ Rd1 ×d2 through a regularized minimization under the spectral norm ∥ · ∥2 (largest singular value) [Bernstein and Newhouse, 2024a]: λ δW ⋆ = arg min ⟨GW , δW ⟩ + ∥δW ∥22 . (1) 2 δW The solution, assuming GW is full rank, is proportional to − msign(GW ), where msign(X) = U V ⊤ for singular value decomposition X = U ΣV ⊤ , thereby orthogonalizing the gradient (or, in practice, the momentum). The solution is not unique for low-rank GW , though we use the convention that msign(X) = U sign(Σ)V ⊤ , so that non-zero singular values are promoted, and the rest remain equal to zero. 3.2

LoRA

LoRA [Hu et al., 2021] is a parameter-efficient method for fine-tuning neural networks. LoRA adds adapters in the form ∆W = BA to the pretrained base weight W0 of a layer, with B ∈ Rd1 ×r and A ∈ Rr×d2 where r ≪ d1 , d2 . Typically B is initialized to zero to ensure that, initially, ∆W = 0. The factors B, A are usually chosen by minimizing the fine-tuning loss with the AdamW optimizer. We instead seek to find the analogue to Muon’s whitened update in the LoRA setting.

Method

Let ∆W = BA be the adapter with gradient G∆W . Our goal is to find updates δB, δA for the factors B, A such that the implied update δ∆W to the adapter ∆W has an approximately flat spectrum. We 3

emphasize that simply orthogonalizing the factor updates δB, δA does not imply that δ∆W itself is orthogonal. Under spectral steepest descent, the optimal adapter update δ∆W ⋆ is given by 2 δ∆W ⋆ = arg min G∆W , δ∆W + λ2 δ∆W 2 . (2) δ∆W

If r = min(d1 , d2 ) then ∆W is full rank and this is an identical problem to Eq. (1), which is solved by whitening the gradient [Bernstein and Newhouse, 2024a], i.e., δ∆W ⋆ = −η msign(G∆W ),

(3)

where proportionality constants have been absorbed into η. However, we do not get the same solution under a low-rank constraint r < min(d1 , d2 ). We have, δ∆W = (B + δB)(A + δA) − BA = δB A + B δA + δB δA,

(4)

but generally it is not possible to find δB, δA that satisfy δ∆W = −η msign(G∆W ). The problem here is that the change δ∆W has rank at most 2r, but msign(G∆W ) is usually full rank. We proceed by writing Eq. (2) in terms of the adapter matrices, 2 ⋆ ⋆ λ (δB , δA ) = arg min G∆W , δB A + B δA + δB δA + 2 δB A + B δA + δB δA 2 . (5) δB,δA

This is a non-linear optimization problem due to the second-order terms, and it has no known closedform solution. For small updates δB, δA, we approximate by dropping these second-order terms, giving 2 (δB ⋆ , δA⋆ ) = arg min G∆W , δB A + B δA + λ2 δB A + B δA 2 . (6) δB,δA

Since it is generally not possible to achieve the rank-unconstrained minimum, we instead try to get as close as possible. We solve a least-squares problem for the linearized update map (overloading the notation δB ⋆ , δA⋆ for convenience), (δB ⋆ , δA⋆ ) = arg min

δ∆W ⋆ − (δB A + B δA)

δB, δA

2 . F

(7)

Taking gradients of the objective w.r.t. δB and δA and setting to zero, we have, 0 = −2(δ∆W ⋆ − δB A − B δA)A⊤ , ⊤

⋆

0 = −2B (δ∆W − δB A − B δA).

(8a) (8b)

Assuming B and A have full rank, right-multiplying Eq. (8a) by (AA⊤ )−1 and left-multiplying Eq. (8b) by (B ⊤ B)−1 gives δ∆W ⋆ A† = δB AA† + B δA A† = δB + BδA A† , †

⋆

†

B δ∆W = B δB A + B B δA = B δB A + δA,

(9a) (9b)

where B † = (B ⊤ B)−1 B ⊤ and A† = A⊤ (AA⊤ )−1 are the Moore-Penrose inverses satisfying B † B = I = AA† . Note that the minimizer is not unique: if (δB, δA) is a minimizer of Eq. (7), then so is (δB + BS, δA − SA) since, δB A + B δA = (δB + BS)A + B (δA − SA).

(10)

Rearranging Eq. (9b), and setting S = −B † δB means that δA⋆ = B † δ∆W ⋆

(11a)

is one particular minimizer, and then substituting into Eq. (9a) gives the corresponding δB ⋆ , δB ⋆ = (I − BB † )δ∆W ⋆ A† . 4

(11b)

Algorithm 1: sMuon Input: B ∈ Rd1 ×r , A ∈ Rr×d2 ; momenta MB , MA ; lr η; weight decay λ // Low-rank momentum † ⊤ † † ⊤ 1 H ← (B ) MA +(I −BB ) MB (A ) // Compute step † 2 δA ← B msign(H) † † 3 δB ← (I − BB ) msign(H) A // Update parameters (split wd) √ 4 s← 1 − λη η 5 B ← s B − s δB η 6 A ← s A − s δA // Project momentum ⊤ 7 MB ← H A ⊤ 8 MA ← B H

4.1

Algorithm 2: Efficient/matmul-only sMuon Input: Same as Alg. 1; jitter ϵ > 0 ⊤ 1 SB ← invroot(B B + ϵI) ⊤ 2 SA ← invroot(AA + ϵI) // Factor H = BX + Y A 2 3 X ← SB MA 2 2 ⊤ 2 4 Y ← MB SA − BSB (B MB SA ) // Bases for col(H), row(H) 5 U2 ← msign(Y ) ⊤ 6 V1 ← msign(A ) ⊤ ⊤ ⊤ 7 V2 ← msign(X − V1 (V1 X )) // 2r × 2r core SB MA V1 SB MA V2 8 Ω ← msign (U2⊤ Y )(AV1 ) 0 // Compute step ⊤ ⊤ 9 δA ← SB (Ω11 V1 + Ω12 V2 ) 10 δB ← U2 Ω21 SA // Update parameters √ 1 − λη 11 s ← η ′ 12 B ← sB − s δB, η ′ 13 A ← sA − s δA // Project momentum ′⊤ ′⊤ 14 MA ← (B B)X + (B Y )A ⊤ ′ ′⊤ 15 MB ← B(XA ) + Y (AA ) ′ ′ 16 B ← B , A ← A

Low-rank gradients

While the solution (δB ⋆ , δA⋆ ) given by Eq. (11) is simple, it is not efficiently computed. The δ∆W ⋆ = −η msign(G∆W ) term requires orthogonalization of a large matrix (remember that G∆W ∈ Rd1 ×d2 ), which we would like to avoid. Fortunately, we do not actually require the full gradient G∆W , and can instead use the low-rank gradient, H := G∆W − (I − PB ) G∆W (I − PA ) = PB G∆W + (I − PB )G∆W PA ,

(12)

where PB = BB † and PA = A† A are projections onto the column space of B and row space of A respectively. The low-rank gradient H is the projection of the gradient onto the subspace of weight updates that the current LoRA adapter can express. The two gradients H and G∆W are first-order equivalent, in the sense that for any δB, δA, ⟨G∆W , δB A + BδA⟩ = ⟨H, δB A + BδA⟩. Thus when linearizing the Muon objective, we can replace G∆W with H to obtain, (δB, δA) = arg min ⟨H, δB A + B δA⟩ + λ2 ∥δB A + B δA∥22 .

(13)

(14)

δB,δA

In particular, this means we use δ∆W ⋆ = −η msign(H). We show in the next section how this can be exploited to compute the update efficiently. 4.2

Efficient, matmul-only computation

We now show how to compute the descent steps δB ⋆ and δA⋆ efficiently. One of the principal advantages of our method versus Riemannion is that it permits a matmul-only solution — Riemannion requires truncated SVDs during the retraction step, which makes parallelization/batching of computation slower (see Table 4). In particular, the most expensive operations required are d × r matmuls, which are easily optimized with deep learning libraries. 5

The gradients for B and A are GB = G∆W A⊤ and GA = B ⊤ G∆W respectively; the low-rank gradient of Eq. (12) can therefore be written as, H = B (B ⊤ B)−1 GA + (I − PB ) GB (AA⊤ )−1 A, | {z } | {z } =: X

(15)

=: Y

where X ∈ Rr×d2 and Y ∈ Rd1 ×r . Assuming B, A, X, Y have rank r, we have rank(H) ≤ 2r. The key identity we exploit is as follows. Let U ∈ Rd1 ×2r , V ∈ Rd2 ×2r be any matrices with orthonormal columns, such that the column space of U contains the column space of H and the column space of V contains the row space of H, then1 , msign(H) = U msign(U ⊤ HV ) V ⊤ .

(16)

Eq. (16) lets us convert a d1 × d2 orthogonalization into a 2r × 2r orthogonalization, and we will show shortly that instantiating H ∈ Rd1 ×d2 can be avoided. To choose U and V , note that since H = BX + Y A, its column space satisfies col(H) ⊆ col(B) + col(Y ), while its row space satisfies row(H) ⊆ row(A) + row(X). We therefore construct orthonormal bases for these two spaces as, U = U1 U2 , V = V1 V2 , (17) with each block a polar factor (and therefore orthonormal columns) in the appropriate space, U1 = B(B ⊤ B)−1/2 , ⊤

⊤ −1/2

V1 = A (AA )

U2 = Y (Y ⊤ Y )−1/2 , ,

⊤

−1/2

V2 = Z(Z Z)

(18a) ⊤

where Z = (I − PA )X .

(18b)

We have U1⊤ U2 = 0 since B ⊤ Y = B ⊤ (I − PB )GB (AA⊤ )−1 = 0, and V1⊤ V2 = 0 since AZ = A(I − PA )X ⊤ = 0. So U and V have orthonormal columns, satisfying the conditions for Eq. (16). Let SB = (B ⊤ B)−1/2 and SA = (AA⊤ )−1/2 . We calculate U ⊤ HV efficiently, ⊤ U1 B U1⊤ Y XV1 XV2 U ⊤ HV = AV1 AV2 U2⊤ B U2⊤ Y −1 SB 0 XV1 XV2 = 0 U2⊤ Y AV1 0 SB GA V1 SB GA V2 = , (U2⊤ Y )(AV1 ) 0 which avoids instantiating H directly. Writing its orthogonalization in blocks of size r × r, Ω11 Ω12 ⊤ Ω := msign(U HV ) = , Ω21 Ω22

(19)

(20)

we now calculate the final factor updates δB ⋆ and δA⋆ . Substituting msign(H) = U ΩV ⊤ into the previous expressions for δB ⋆ and δA⋆ (Eq. 11) gives, δA⋆ = −ηB † U ΩV ⊤ = −ηSB Ω11 V1⊤ + Ω12 V2⊤ , (21a) where we used B † U1 = SB and B † U2 = 0. Similarly, δB ⋆ = −η(I − PB )U ΩV ⊤ A† = −ηU2 Ω21 SA ,

(21b)

using (I − PB )U1 = 0, (I − PB )U2 = U2 , V1⊤ A† = SA , and V2⊤ A† = 0. All components of δB ⋆ and δA⋆ can be computed with matmuls, matrix inverse roots, and msign. In practice, both the r × r inverse square roots and the msign’s can be evaluated with matrix polynomial iterations [Amsel et al., 2026, Zhang et al., 2026], which are themselves sequences of matrix products. 1 To see why Eq. (16) is true, note that since U U ⊤ and V V ⊤ act as the identity on the column and row spaces of H, we have H = U (U ⊤ HV )V ⊤ . Let P, Σ, Q⊤ = SVD(U ⊤ HV ). Then H = (U P )Σ(V Q)⊤ is a valid, albeit truncated, SVD of H (U P and V Q have orthonormal columns), so msign(H) = (U P )(V Q)⊤ = U (P Q⊤ )V ⊤ = U msign(U ⊤ HV ) V ⊤ .

Table 1: Per-eval accuracy (%), grouped by base model. Each column is an optimizer (best run by validation loss); per row the best optimizer is bolded and cells are shaded green (best) to white (worst). Eval

AdamW

Per-factor Muon

LoRA-Muon

Riemannion

sMuon (ours)

Qwen2.5-3B Piqa Winogrande OBQA Hellaswag BoolQ Arc-C MMLU

81.7 62.3 78.4 81.7 67.7 83.3 64.2

82.1 61.2 79.2 78.7 66.5 83.6 64.9

81.7 61.4 76.0 76.9 65.6 82.9 65.0

82.7 61.0 77.4 77.6 67.7 83.3 64.9

81.7 61.5 77.0 80.3 67.4 83.6 64.9

Llama-3.2-3B Piqa Winogrande OBQA Hellaswag BoolQ Arc-C MMLU

77.9 51.0 67.2 63.7 63.1 67.9 54.4

77.1 52.0 68.4 64.9 63.4 66.2 54.8

74.2 53.8 63.4 58.9 63.8 60.5 52.7

76.9 54.9 69.4 64.3 64.1 64.9 54.9

76.6 52.4 66.8 63.3 63.3 63.2 54.4

DeepSeek-V2-Lite Piqa 79.2 Winogrande 58.2 68.0 OBQA 74.4 Hellaswag BoolQ 64.6 Arc-C 70.6 MMLU 55.4

76.7 58.8 70.8 71.3 64.1 64.9 54.7

78.2 59.7 69.0 71.2 64.6 70.6 53.1

75.5 55.8 61.2 65.4 64.7 67.2 54.1

79.2 58.2 70.8 76.4 65.2 67.6 53.5

Moonlight-16B-A3B Piqa 84.7 Winogrande 59.4 OBQA 81.0 Hellaswag 77.9 BoolQ 68.4 Arc-C 85.6 MMLU 66.1

84.5 61.2 81.2 80.5 67.1 81.9 65.9

85.6 63.3 81.2 84.7 69.8 84.3 65.6

84.4 64.0 78.6 82.3 68.5 83.3 66.2

86.2 65.8 82.4 85.9 69.5 87.0 64.5

4.3

Algorithm

We combine the above updates with split weight decay [Cesista et al., 2026] to match the weight decay dynamics of full-parameter training to first order, and momentum projection (inspired by momentum transport of Bogachev et al. [2026], see App. A.3 for details) to ensure reparameterization invariance, yielding the sMuon optimizer for LoRA adapters. We provide Alg. 1 as a pedagogical version, while Alg. 2 incorporates the efficient matmul-only computations from the previous section. We also discuss a more numerically stable variant in Appendix B. One cause for concern is the practice of initializing the adapter ∆W = BA = 0, and then computing inverse roots of B ⊤ B and AA⊤ . We found that, despite initializing ∆W = 0, calculating inverse roots of AA⊤ and B ⊤ B was not an issue when using jitter and a semi-orthonormal B, A = 0 initialization. 7

Table 2: Code pass@1 (%), grouped by base model. Each column is an optimizer (best run by validation loss); per row the best optimizer is bolded and cells are shaded green (best) to white (worst). Eval

AdamW

Per-factor Muon

LoRA-Muon

Riemannion

sMuon (ours)

Qwen2.5-3B HumanEval HumanEval+ MBPP MBPP+

54.9 48.8 65.9 57.1

20.1 17.1 69.3 60.6

43.3 39.0 68.0 58.7

31.1 25.6 65.9 56.9

51.8 43.9 62.7 53.4

Llama-3.2-3B HumanEval HumanEval+ MBPP MBPP+

22.0 19.5 43.1 35.4

18.3 16.5 35.7 30.4

29.9 25.0 42.1 34.1

30.5 26.8 46.3 38.1

29.9 27.4 45.2 38.6

DeepSeek-V2-Lite 36.6 HumanEval HumanEval+ 28.7 52.6 MBPP MBPP+ 45.5

29.3 23.8 53.7 45.0

22.0 18.3 48.4 43.4

23.2 19.5 49.5 43.7

26.8 23.2 51.3 44.7

Moonlight-16B-A3B HumanEval 49.4 HumanEval+ 42.7 MBPP 71.7 60.1 MBPP+

57.9 48.8 71.2 60.1

53.7 47.0 71.2 61.4

40.2 35.4 68.8 58.7

54.9 48.8 71.2 60.3

Table 3: ReLoRA pretraining: final validation loss at each optimizer’s optimal learning rate, where the optimal LR minimises the validation loss at the final logged step. Lower is better; the best optimizer is bolded and cells are shaded green (best) to white (worst).

Final val loss Optimal LR

AdamW

Per-factor Muon

LoRA-Muon

Riemannion

sMuon (ours)

3.83 0.001

3.70 0.006

3.68 0.001

3.66 0.0006

Experiments

We compare sMuon to regular LoRA with AdamW, per-factor Muon (LoRA with Muon applied to each adapter weight B and A individually), LoRA-Muon [Cesista et al., 2026], and Riemannion [Bogachev et al., 2026]. We evaluated on standard commonsense [Clark et al., 2018, 2019, Zellers et al., 2019, Mihaylov et al., 2018, Bisk et al., 2020, Sakaguchi et al., 2021] and coding [Chen et al., 2021, Austin et al., 2021] benchmarks after training with SFT. We experimented with four models, Llama3.2-3B [Grattafiori et al., 2024], Qwen2.5-3B [Qwen Team, 2025], DeepSeek-V2-Lite [DeepSeek-AI, 2024], and Moonlight-16B-A3B [Liu et al., 2025]. The first two models are dense models; the last two are sparse MoE models. The Moonlight model is of particular note because it was pretrained with Muon, whereas the others are all AdamW-pretrained. We also tested the methods in a pretraining setting using ReLoRA [Lialin et al., 2023] which periodically merges and refreshes the adapter. We tuned the learning rate for each method and model, with example learning rate sweeps for the commonsense tasks shown in Appendix E. While our results are both model- and eval-dependent, sMuon performs strongly overall, especially when combined with the Moonlight-16B-A3B base model, a Muon-pretrained model (Tables 1 and 2). sMuon also achieved the joint-best validation loss in the ReLoRA pretraining setting (Table 3). We 8

attribute performance gains, particularly over AdamW and per-factor Muon, to more accurately approximating the Muon objective. Experiment details LoRA-Muon and Riemannion were chosen as baselines in addition to the simpler AdamW and per-factor Muon baselines as these are the most similar to our method. Unfortunately, we could not find implementations for LoRA-Muon or Riemannion, so we implemented these ourselves. For Riemannion and our method, we use an A = 0 and B orthonormal initialization, and for the remaining methods we used a standard initialization (B = 0, A random Gaussian). We scale the learning rate to approximately match AdamW RMS for all Muon optimizers (for details see App. A.1). For all experiments, we used weight decay 0.01, momentum 0.9, inserted LoRA adapters into all hidden layers (except the router for models with MoE). We used GH200 for all experiments except to produce Table 4 for which we used A100. We used bf16 to store weights and optimizer states, but used fp32 to compute the step for Riemannion, sMuon, and LoRA-Muon. For the ReLoRA experiments [Lialin et al., 2023], we started with the modded-nanogpt codebase at ‘record 8’ [Jordan et al., 2024a], which trains a 162M parameter transformer from scratch (using features including QK-norm, ReLU2 activations) on the FineWeb dataset [Penedo et al., 2024], and evaluates on a 10M token validation set. We initialize LoRA adapters with rank 64, and train for a total of 3000 steps; every 500 steps we merge the adapters into the base weights and reinitialize the adapters; we use a cosine decay learning rate schedule, though after each merge+reinitialization, we also reset the learning rate to zero and warm back up to the target learning rate over 50 steps (this is also known as a jagged cosine schedule). We swept learning rates in 3e-4, 6e-4, 1e-3, 3e-3, 6e-3, 1e-2. For the HumanEval [Chen et al., 2021] and MBPP [Austin et al., 2021] coding evals, we fine-tuned for 3 epochs on completing Python functions from the CodeAlpaca dataset [Chaudhary, 2023]. For the MMLU [Hendrycks et al., 2021] eval, we fine-tuned on a 50/50 mixture of FLAN [Longpre et al., 2023] and MMLU training subset. For the remaining evals, we trained on a mixture of FLAN and CommonsenseQA [Talmor et al., 2019], and training subset for each eval. All SFT evals were 0-shot. For SFT training runs, we used α = 32, r = 16, gradient clipping (clipping at 1), and we used roughly 216 tokens per batch. Muon-based LoRA is strongest on the Muon-pretrained model We originally hypothesized that Muon-based LoRA methods might perform better with Moonlight, since it is a Muon-pretrained model. We do find some evidence for this when aggregating over all benchmarks in Tables 1 and 2: AdamW only achieves top performance on one of the eleven tasks, while sMuon is best for six. Some methods are markedly below others on certain tasks (e.g. per-factor Muon with Llama-3.2-3B on HumanEval, Table 2). This could be due to optimizer instabilities or noise in the benchmarks or selection procedure. Benefits of Muon pretraining remain with ReLoRA The ReLoRA experiment (Table 3) removes optimizer mismatch entirely by training from scratch. The separation between optimizer families is clear. Every Muon-based method achieves lower validation loss than LoRA+AdamW. One explanation for the strong Muon performance is amplification of the small but real signal in the long tail of the gradient’s spectrum [Wang et al., 2025]. Efficiency gain from the matmul-only implementation Table 4 compares step times for the different optimizers. AdamW is the fastest; the only computation required by AdamW is elementwise, meaning that data communication is likely to be the largest overhead. Per-factor Muon is the next fastest, because computation is dominated by two msign(·) computations (that can be batched). Both LoRA-Muon and sMuon require more matmuls than per-factor Muon, but the compute time is still negligible compared to the forward+backward pass time per step of ∼ 1.4s. On the other hand, computing the Riemannion step is much slower, adding ∼ 50% overhead to the forward+backward pass when r = 64, and even longer for r = 128. The computation is slow because Riemannion requires thin QRs and 2r × 2r SVDs, which add overhead for larger r. Note that the retraction step in Riemannion specifically requires truncated SVD, which cannot be replaced by a more efficient Newton-Schulz algorithm. We found that compiling the core computation (i.e. the matmuls), and batching the computation leads to big efficiency gains with all the Muon methods except Riemannion. 9

Table 4: All Muon optimizers are slightly slower than AdamW, but Riemannion adds huge overhead at larger rank. We show mean time for LoRA adapter optimizer step (ms, 2 s.f.), as well as the proportion (%) of the total step time. Figures were obtained while training a 12 layer GPT model with dmodel = 768. We ran these timings on an A100, calculating the mean time over 25 steps, with different ranks r ∈ {24 , 25 , 26 , 27 }, batch size 128, 4 gradient accumulation steps. Step time, ms (% total) ↓ AdamW Per-factor Muon LoRA-Muon Riemannion sMuon (ours) Fwd+bwd, ms

r = 16

r = 32

r = 64

r = 128

0.66 (0.047%) 0.59 (0.041%) 0.65 (0.046%) 0.78 (0.054%) 1.4 (0.10%) 1.6 (0.11%) 2.0 (0.14%) 3.6 (0.25%) 4.5 (0.32%) 4.8 (0.33%) 6.3 (0.44%) 10 (0.70%) 90 (6.0%) 300 (17%) 700 (33%) 1700 (54%) 8.7 (0.62%) 8.9 (0.62%) 12 (0.80%) 20 (1.4%) 1400

1432

1429

1459

Conclusion

We considered optimizing low-rank adapters using an approximate form of Muon. The resulting step is available in closed form, is invariant to the adapter parameterization (B, A) 7→ (BS, S −1 A) and is cheap, costing O(r2 (d1 + d2 ) + r3 ) time and O(r max(d1 , d2 )) memory, using less optimizer state than LoRA+AdamW. Our implementation, sMuon, has empirically flatter spectra than per-factor Muon and out-performs other methods on the Moonlight Muon-pretrained model. In ReLoRA pretraining every Muon-based method beats AdamW, with sMuon and Riemannion the strongest. In general, by respecting the geometry of the full layer, rather than treating B and A as unrelated matrices, we often benefit, and there is little extra cost.

References Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The polar express: Optimal matrix sign methods and their application to the muon algorithm. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, editors, International Conference on Learning Representations, volume 2026, 2026. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Jeremy Bernstein. Deriving muon, 2025. deriving-muon.

URL https://jeremybernste.in/writing/

Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024a. Jeremy Bernstein and Laker Newhouse. arXiv:2410.21265, 2024b.

Modular duality in deep learning.

arXiv preprint

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020. arXiv:1911.11641. Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko, Denis Bobkov, Vera Soboleva, Aibek Alanov, and Maxim Rakhuba. Lora meets riemannion: Muon optimizer for parametrizationindependent low-rank adapters. In International Conference on Learning Representations, volume 2026, pages 82190–82215, 2026. Franz Louis Cesista, Katherine Crowson, Cédric Simal, and Stella Biderman. Lora-muon: Spectral steepest descent on the low-rank manifold. arXiv preprint arXiv:2606.12921, 2026. Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https: //github.com/sahil280114/codealpaca, 2023. 10

Mark Chen, Jerry Tworek, Heewoo Jun, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, 2019. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. Nikhil Ghosh, Tetiana Parshakova, and Robert M Gower. Polora: A preconditioned orthogonalized lora optimizer. arXiv preprint arXiv:2607.17620, 2026. Luc Giraud, Julien Langou, Miroslav Rozložník, and Jasper van den Eshof. Rounding error analysis of the classical gram-schmidt orthogonalization process. Numerische Mathematik, 2005. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Chuan He, Zhanwang Deng, and Zhaosong Lu. Low-rank orthogonalization for large-scale matrix optimization with applications to foundation model training. arXiv preprint arXiv:2509.11983, 2025. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. International Conference on Learning Representations (ICLR), 2021. arXiv:2009.03300. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. Paul Janson, Edouard Oyallon, and Eugene Belilovsky. Stabilizing native low-rank llm pretraining. arXiv preprint arXiv:2602.12429, 2026. Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, Boza Vlado, Jiacheng You, Franz Cesista, Braden Koszarsky, and Grad62304977. modded-nanogpt: Speedrunning the NanoGPT baseline. https://github.com/KellerJordan/modded-nanogpt, 2024a. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan. github. io/posts/muon, 2024b. Changmin Kang, Jihun Yun, Baekrok Shin, Yeseul Cho, and Chulhee Yun. Uniform spectral growth and convergence of muon in lora-style matrix factorization. arXiv preprint arXiv:2602.06385, 2026. Kimi Team. Kimi k2: Open agentic intelligence, 2025. URL https://arxiv.org/abs/2507. 20534. Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. ReLoRA: High-rank training through low-rank updates. arXiv preprint arXiv:2307.05695, 2023. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025. 11

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, and Adam Roberts. The Flan collection: Designing data and methods for effective instruction tuning. International Conference on Machine Learning (ICML), 2023. arXiv:2301.13688. Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017.

Decoupled weight decay regularization.

arXiv preprint

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018. doi: 10.18653/v1/D18-1260. Zhanfeng Mo, Long-Kai Huang, and Sinno Jialin Pan. Parameter and memory efficient pretraining via low-rank riemannian optimization. In International Conference on Learning Representations, volume 2025, pages 7907–7936, 2025. Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), 2024. arXiv:2406.17557. Xingyu Qu, Peigeng Huang, and Samuel Horvath. Can muon fine-tune adam-pretrained models? arXiv preprint arXiv:2605.10468, 2026. Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2025. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 2021. arXiv:1907.10641. Ishaan Shah, Anthony M Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, et al. Practical efficiency of muon for pretraining. arXiv preprint arXiv:2505.02222, 2025. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of NAACL-HLT, 2019. arXiv:1811.00937. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y Charles, et al. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang, Zhuoran Yang, Mingyi Hong, and Vincent YF Tan. Muon outperforms adam in tail-end associative memory learning. arXiv preprint arXiv:2509.26030, 2025. Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Taming momentum: Rethinking optimizer states through low-rank approximation. arXiv preprint arXiv:2602.24283, 2026. Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient milliontoken context intelligence. arXiv preprint arXiv:2606.19348, 2026. Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar. LoRA done RITE: Robust invariant transformation equilibration for LoRA optimization. In International Conference on Learning Representations, 2025. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, July 2019. Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned lora for fine-tuning foundation models. arXiv preprint arXiv:2402.02347, 2024. 12

Jack Zhang, Noah Amsel, Berlin Chen, and Tri Dao. Gram newton-schulz, 2026. URL https: //dao-ailab.github.io/blog/2026/gram-newton-schulz/. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024.

Other algorithmic details

We discuss some pre-existing techniques that we incorporated into sMuon. A.1

Match AdamW RMS

Inspired p by Liu et al. [2025], we adopt a learning rate adjustment. The full-parameter scaling is 0.2 max(din , dout ), with the overall update, including weight decay, becoming p x ← x − ληx − 0.2η max(din , dout ) · δx. (22) To match this in the LoRA setting, we used x ← x − ληx − 0.2η

p din dout /r · δx.

(23)

We loosely justify this heuristic: assume the weight update δ∆W is orthogonalized. The rank of δ∆W is at most 2r (it can be written p as the difference of two rank √ r matrices), and therefore 2 ∥δ∆W ∥ ≤ 2r. So ∥δ∆W ∥ ≤ 2r/(d d ). We absorb the 2 factor, and use a scale of rms in out p F 0.2 din dout /r. A.2

Split weight decay

If we apply weight decay to B and A separately, this gives different dynamics compared to fullparameter training. Concretely, At+1 = (1 − ηλ)At − η δA, Bt+1 = (1 − ηλ)Bt − η δB =⇒ δ∆W = (1 − ηλ)Bt − η δB (1 − ηλ)At − η δA − Bt At = − 1 − (1 − ηλ)2 Bt At − η(1 − ηλ) Bt δA + δB At + O(η 2 ), which is not recognizable in terms of full-parameter dynamics. We adopt the split weight √ decay of Cesista et al. [2026] to preserve the first-order weight decay dynamics. Setting s = 1 − λη, LoRA factors update as, At+1 = s At − ηs δA,

(24)

Bt+1 = s Bt − ηs δB.

(25)

Expanding the resulting weight change δ∆W = Bt+1 At+1 − Bt At gives, δ∆W = sBt − ηs δB sAt − ηs δA − Bt At 2

= s2 Bt At − η(δB At + Bt δA) + ηs2 δB δA − Bt At 2 = (s − 1)Bt At − η(δB At + Bt δA) + ηs2 δB δA = −λη ∆Wt − η(δB At + Bt δA) +O(η 2 )

decay term

}

gradient term

(26) (27) (28) (29)

}

which we identify as having the dynamics of decoupled weight decay [Loshchilov and Hutter, 2017]. A.3

Invariance to parameterization

In practice, we orthogonalize a momentum rather than a raw gradient. The natural choice for momentum accumulation is MA ← βMA + (1 − β)GA and MB ← βMB + (1 − β)GB , and to reconstruct the effective momentum by substituting them for GA , GB in the equation factorizing H, H̄t = (Bt† )⊤ MA,t + (I − PBt ) MB,t (A†t )⊤ .

(30)

Here, GB = G∆W A⊤ and GA = B ⊤ G∆W are the gradients of the adapter matrices B and A respectively. Consider two training runs, (Bt , At ) and (Bt′ , A′t ), that differ only in their parameterizations at each step, (Bt′ , A′t ) = (Bt St , St−1 At ). (31) ′ ′ ′ Since ∆Wt = Bt At = Bt At = ∆Wt , we want the optimizer to take equivalent steps throughout training, regardless of St . For this, it is necessary that H̄t′ = H̄t . 14

Without transporting momentum buffers The adapter gradients in the primed run are G′A,k = (Bk′ )⊤ G∆W,k = Sk⊤ GA,k and G′B,k = G∆W,k (A′k )⊤ = GB,k Sk−⊤ . Unrolling the EMA, we have, ′ MA,t =

X (1 − β)β t−k Sk⊤ GA,k ,

′ MB,t =

k≤t

(1 − β)β t−k GB,k Sk−⊤ .

(32)

k≤t

Thus, the ‘primed’ run’s effective momentum is, H̄t′ =

h i (1 − β)β t−k (Bt† )⊤ St−⊤ Sk⊤ GA,k + (I − PBt ) GB,k Sk−⊤ St⊤ (A†t )⊤ ,

(33)

k≤t

which differs from H̄t by the factors St−⊤ Sk⊤ and Sk−⊤ St⊤ . These are the identity only when Sk = St , so in general H̄t′ ̸= H̄t . With momentum transport We fix the issue by re-deriving the buffers from H̄ against the factors as they stand at the end of the step, MA ← B ⊤ H̄,

MB ← H̄A⊤ .

(34)

Geometrically, this projects H̄ onto the space in which our updates, δB A + B δA, exist. The momenta at the next step are then, ⊤ MA,t+1 = Bt+1 β H̄t + (1 − β)G∆W,t+1 , MB,t+1 = β H̄t + (1 − β)G∆W,t+1 A⊤ t+1 .

(35) (36)

′ ′ ⊤ Assuming H̄t′ = H̄t , we have MA,t+1 = (Bt+1 )⊤ β H̄t + (1 − β)G∆W,t+1 = St+1 MA,t+1 and −⊤ ′ similarly MB,t+1 = MB,t+1 St+1 . By further substitution, † −1 ′ H̄t+1 = St+1 Bt+1

⊤

−⊤ ⊤ St+1 MA,t+1 + (I − PBt+1 ) MB,t+1 St+1 A†t+1 St+1

⊤

† −⊤ ⊤ −⊤ ⊤ = (Bt+1 )⊤ St+1 St+1 MA,t+1 + (I − PBt+1 ) MB,t+1 St+1 St+1 (A†t+1 )⊤ | {z } | {z } I

= H̄t+1 . So long as we project H̄ into the momentum buffers at every step, the two runs will orthogonalize the same matrix, and therefore the realized steps will agree.

Alternative stable algorithm

See Alg. 3, a mathematically equivalent but more stable alternative to Alg. 2 from the main text. In particular, we perform two orthogonalization steps [Giraud et al., 2005] to help preserve orthogonality to machine precision, and include relative jitter to help stability for larger r. Another difference between Alg. 2 and 3 is that we use slightly different U2 , V2 . Alg. 3 uses U2 = msign(Ỹ ), where 2 2 Y = Ỹ SA , and V2 = msign(Z̃), where Z = Z̃SB . This is possible because the property we need is col(Y ) = col(Ỹ ), and similar for Z/Z̃, which is true when SB and SA are invertible (we assume this to be the case). One can also consider additional interventions, such as replacing msign(X) = X(X ⊤ X)−1/2 for X ∈ Rd×r with X invroot(X ⊤ X). The latter is slightly faster for small r (since X ⊤ X ∈ Rr×r ), but we found the benefits to be small in practice. 15

Algorithm 3: Stable sMuon Input: Same as Alg. 1; jitter ϵ > 0 (default to 10−4 ) // Inverse roots ⊤ 1 DB ← B B 2 ϵB ← ϵ max(tr(DB )/r, 1) 3 SB ← invroot(DB + ϵB I) ⊤ 4 DA ← AA 5 ϵA ← ϵ max(tr(DA )/r, 1) 6 SA ← invroot(DA + ϵA I) // Bases 2 7 X ← SB MA ⊤ 8 V1 ← msign(A ) 2 ⊤ 9 Ỹ ← MB − B SB (B MB ) 2 ⊤ 10 Ỹ ← Ỹ − B SB (B Ỹ ) // two orthog. steps 2 11 U2 ← msign(Ỹ ), Y ← Ỹ SA ⊤ ⊤ ⊤ 12 Z̃ ← MA − V1 (V1 MA ) ⊤ 13 V2 ← msign(Z̃ − V1 (V1 Z̃)) // two orthog. steps SB (MA V1 ) SB (MA V2 ) 14 C ← (U2⊤ Y )(AV1 ) 0 15 Ω ← msign(C) // Step direction 1 ⊤ 16 P ← 2 AV1 + (AV1 ) // symmetric matrix 1 2 ⊤ 2 // symmetric matrix 17 T ← 2 P SA + (P SA ) ⊤ ⊤ 18 δA ← SB (Ω11 V1 + Ω12 V2 ) 19 δB ← U2 Ω21 T // Update parameters √ 20 s ← 1 − λη η ′ 21 B ← sB − s δB η ′ 22 A ← sA − s δA // Project momentum ′⊤ ′⊤ 23 MA ← (B B)X + (B Y )A ⊤ ′ ′⊤ 24 MB ← B(XA ) + Y (AA ) ′ 25 B ← B ′ 26 A ← A

Comparison of Muon-based LoRA optimizers

Every method that combines Muon with LoRA has the same obstruction: Muon requires a step along msign(G∆W ), but a gradient step for a LoRA adapter is constrained inside the set T = {Z : (I −PB )Z(I −PA ) = 0}. The ideal step msign(G∆W ) is not in general within this set, thus different methods make different approximations. We compare the different methods in Table 5.

Table 5: Comparison of the Muon-based LoRA optimizers, all written in the convention ∆W = BA of this paper. ΠT (·) is the orthogonal projector onto the realizable step directions. Method

δ∆W

Momentum

Weight decay on ∆W

Expensive operations

Per-factor Muon [Qu et al., 2026, Kang et al., 2026]

−η msign(MB )A + B msign(MA ) + O(η 2 )

Standard per-factor EMAs, i.e. accumulate GB in MB (MB ← βMB + (1−β)GB ), likewise for MA

Decoupled, per factor: ∆W 7→ (1 − λη)2 ∆W

2 orthogonalizations

Spectron [Janson et al., 2026]

As above with η replaced by ρ = η/(∥B∥2 +∥A∥2 +1); only ∥δ∆W ∥2 ≤ η is controlled, no direction matched

Standard per-factor EMAs

Decoupled, per factor

2 orthogonalizations, 2 power iterations to estimate ∥B∥2 , ∥A∥2

Split:√let s = 1 − λη, and A ← sA − η/s δA (same for B). Overall gives −λη ∆W + O(η 2 ) (Sec. A.2)

The same 2 orthogonalizations, 2 r × r inverse square roots of (B ⊤ B)−1/2 and (AA⊤ )−1/2

LoRA-Muon − η2 msign(PB G∆W ) + Standard per-factor EMAs [Cesista et al., msign(G∆W PA ) + O(η 2 ) 2026]

Riemannion [Bogachev et al., 2026]

SVDr ∆W − η ΠT (msign(H)) − ∆W

Ours

−η(I −PB ) msign(H)PA − Standard per-factor EMAs, but let Split ηPB msign(H) + O(η 2 ) H = (B † )⊤ MA + (I − PB )MB (A† )⊤ . After the gradient step, set MB = HA⊤ , MA = B ⊤ H

Prior to retraction, 4 thin QRs, 2 SVDs (2r×2r) A and B semi-orthonormal. A is decayed via Accumulate (I − BB ⊤ )GB in MB , and accumulate GA in MA . −ηγ A; B is not Let decayed. H = BMA +(I −BB ⊤ )MB A. After the retraction, set MB = (I − BB ⊤ )HA⊤ , MA = B ⊤ H 2 r×r inverse square roots, 4 orthogonalizations (d × r, worst case)

Optimality of the sMuon step

We show in Lemma 1 that our particular choice of δA and δB (a) is, to first order, as close as possible to the target step −η msign(H), and (b) while the minimum is not unique in terms of steps (δB, δA), it is unique in terms of the first-order change to the whole linear layer. Lemma 1. Let B ∈ Rd1 ×r have full column rank and A ∈ Rr×d2 have full row rank. Let PB = BB † and PA = A† A, let G∆W be the adapter gradient, and let H = G∆W − (I − PB ) G∆W (I − PA ). Write δ∆W = (B + δB)(A + δA) − BA and δ∆W ⋆ = −η msign(H). Consider updates of the form δB = O(η), δA = O(η), then ∥δ∆W ⋆ − δ∆W ∥F = ∥η msign(H) + δB A + BδA∥F + O(η 2 ),

(37)

1. The choice δA = −η B † msign(H), δB = −η (I − PB ) msign(H) A† minimizes ∥δ∆W ⋆ − δ∆W ∥F to first order. 2. The first-order minimizer is not unique. If (δB, δA) is a minimizing pair, then so is (δB − BS, δA + SA) with S ∈ Rr×r , provided that S = O(η). Every first-order minimizer realizes the same overall update, δ∆W = −η PB msign(H) + (I − PB ) msign(H)PA + O(η 2 ). (38) Proof. Since B † B = I = AA† ∈ Rr×r , we have (I − PB )B = 0 and A(I − PA ) = 0. Write M = msign(H), δB = ηb, δA = ηa with a, b bounded independently of η. Then δ∆W ⋆ − δ∆W = −η (M + bA + Ba) + O(η 2 ) 17

and the reverse triangle inequality gives Eq. (37). It therefore suffices to minimize the leading coefficient ∥M + bA + Ba∥F . Any X ∈ Rd1 ×d2 admits the decomposition X = PB XPA + PB X(I − PA ) + (I − PB )XPA + (I − PB )X(I − PA ), a sum of four mutually Frobenius-orthogonal terms. We have (I − PB )(bA + Ba)(I − PA ) = 0, so ∥M + bA + Ba∥2F ≥ ∥(I − PB ) M (I − PA )∥2F . Crucially, the RHS is independent of the choice of b, a. We get equality when the other three blocks of the decomposition are zero, which they are for M + bA + Ba, PB (M + bA + Ba)PA + PB (M + bA + Ba)(I − PA ) + (I − PB )(M + bA + Ba)PA = PB (M + bA + Ba) + (I − PB )(M + bA + Ba)PA = PB M − PB M + (I − PB )M PA − (I − PB )M PA = 0, where we substituted the choice of Part 1, bA = −(I − PB )M A† A = −(I − PB )M PA ,

Ba = − BB † M = −PB M,

and used PB (I − PB ) = 0, (I − PB )2 = I − PB , PA2 = PA , etc.. Since ∥δB∥F , ∥δA∥F = O(η), this pair lies in the domain and attains equality, so it minimizes the leading coefficient, and hence ∥δ∆W ⋆ − δ∆W ∥F to first order. For Part 2, equality holds only when the three blocks vanish. Since the fourth block of bA + Ba is identically zero by (I − PB )B = 0 and A(I − PA ) = 0, the three conditions force bA + Ba = − PB M − (I − PB )M PA

for every minimizing pair.

Consequently, every first-order minimizer satisfies δ∆W = η (bA + Ba) + δB δA = −η PB M + (I − PB )M PA + O(η 2 ), which is Eq. (38). Finally, if S = O(η), the pair (δB − BS, δA + SA) remains O(η) and satisfies (δB − BS) A + B (δA + SA) = δB A + B δA, so it has the same leading coefficient and is again a first-order minimizer.

Learning rate curves

We show learning rate curves for the commonsense runs in Figures 2 (overall commonsense accuracy) and 3 (val. loss). We swept different learning rates for different optimizers, since we found that the optimal learning rate varies between optimizers. The accuracy figures shown in the main text are chosen by taking the optimal learning rate according to loss on a held-out validation set.

Qwen2.5-3B

Llama-3.2-3B 0.65 0.60 0.55 0.50 0.45 0.40 0.35

accuracy

0.7 0.6 0.5 0.4

DeepSeek-V2-Lite

Moonlight-16B-A3B 0.8

accuracy

0.7

0.6

0.5

0.4 0.3

0.4 10 6

10 5

10 4

learning rate

Per-factor Muon AdamW

0.3

10 3

LoRA-Muon

10 6

10 5

Riemannion

10 4

learning rate

10 3

sMuon (ours)

Figure 2: Common-sense accuracy against learning rate for each base model. Each curve is one optimizer; accuracy is the overall common-sense benchmark accuracy at the final training step.

cross-entropy

Qwen2.5-3B

Llama-3.2-3B

0.28

0.34

0.26

0.32 0.30

0.24

0.28

0.22

0.26

cross-entropy

DeepSeek-V2-Lite

Moonlight-16B-A3B 0.30

0.30

0.28

0.26

0.24

0.24 10 6

10 5

10 4

learning rate

Per-factor Muon AdamW

10 3

10 6

LoRA-Muon

Riemannion

10 5

10 4

learning rate

10 3

sMuon (ours)

Figure 3: Validation loss against learning rate for each base model, over the same runs as Figure 2.

Approximate Muon with low-rank adapters

Analyze & explore related research in Conceptio

Related documents