Preprint
M UON AS A R ESIDUAL C ONNECTION Hao Huang College of Computer Science and Technology Zhejiang University Hangzhou, Zhejiang, China [email protected]
arXiv:2607.01124v1 [cs.LG] 1 Jul 2026
A BSTRACT Muon has recently emerged as one of the most effective optimizers for training large neural networks, yet its empirical success has been explained from several different perspectives. In this paper, we propose a simple mechanistic interpretation: Muon can be understood as an implicit residual connection during training. Specifically, orthogonalizing the update can sacrifice some immediate gradient fidelity while improving representation preservation for downstream layers. We study this trade-off in controlled linear optimization settings, where Muon can learn representations that are slower to fit a local target but easier for downstream layers to exploit. Our results suggest a conceptual explanation for Muon and a design perspective for optimizers that balance local descent with downstream usability.1
1
I NTRODUCTION
In recent years, Muon has attracted considerable attention as a highly effective optimizer for deep neural networks, delivering state-of-the-art performance on a wide range of architectures, from CNNs to LLMs. Muon is built upon a simple idea—orthogonalizing the updates of matrix parameters. Yet, despite its simplicity, its empirical success has inspired a surprisingly diverse collection of sophisticated explanations, including spectral steepest descent, sharpness, curvature, and long-tail learning. A natural question arises before introducing yet another explanation of Muon: why does Muon need a new explanation at all? Rather than viewing these explanations as mutually exclusive, we begin from a broader perspective on scientific explanation. As emphasized in the philosophy of science, a scientific phenomenon often admits multiple valid explanations, each highlighting different causal mechanisms, levels of abstraction, or explanatory purposes. Different explanations are valuable not only because they improve our understanding of a phenomenon, but also because they inspire different directions for future research. For example, interpreting Muon through the lens of spectral-norm steepest descent naturally motivates the exploration of optimizers based on alternative matrix norms. In contrast, explaining Muon’s success through curvature suggests incorporating richer second-order information into optimizer design. From this perspective, a new explanation is valuable not merely as another interpretation, but as a new source of hypotheses and algorithmic design principles. Our goal is therefore not to replace existing explanations, but to introduce a new explanatory perspective that complements them. Specifically, we argue that Muon can be understood as an implicit residual connection, providing a simple and intuitive framework for reasoning about how optimization can affect downstream representation usability.
1
Code is available at https://github.com/huanghao-sss/muon_interpretation.
1
Preprint
2
BACKGROUND
2.1
T HE M UON O PTIMIZER
Muon is a recently proposed optimizer that has attracted considerable attention due to its remarkable empirical success, from benchmarks such as NanoGPT and CIFAR-10 to frontier large language models including Kimi K2 and DeepSeek V4. At its core, Muon replaces the standard parameter update W ← W + ∆W with an orthogonalized update W ← W + Orth(∆W ), where Orth(∆W ) denotes the orthogonal matrix closest to ∆W . 2.2
E XISTING I NTERPRETATIONS OF M UON
Existing interpretations of Muon emphasize several complementary aspects of its behavior. The spectral-norm steepest-descent view interprets the orthogonalized update as the steepest descent direction under the matrix spectral norm (Bernstein & Newhouse, 2024), while the trust-region view derives the Muon step as the exact solution of a non-Euclidean trust-region subproblem (Kovalev, 2025). Other work views Muon with decoupled weight decay as a stochastic Frank–Wolfe method on a spectral-norm ball, thereby emphasizing the constrained optimization trajectory rather than a single-step update direction (Sfyraki & Wang, 2025). A further line of work connects Muon to rare directions and long-tail learning, showing that spectral normalization can prevent dominant gradient modes from overwhelming weaker but important directions (Vasudeva et al., 2025; Wang et al., 2025). Curvature-based explanations argue that Muon can achieve comparable first-order progress to Adam while incurring a smaller second-order curvature penalty (Wang et al., 2026). The spectral Wasserstein-flow perspective places Muon within a continuous-time optimal-transport framework for spectrally normalized optimization (Peyré, 2026). Finally, another interpretation emphasizes batch gradient alignment and directional descent potential, arguing that exact orthogonalization is not uniquely responsible for Muon’s behavior (Shumaylov et al., 2026). We provide a more detailed summary of these interpretations in Appendix A. These existing interpretations offer useful perspectives. Nevertheless, we seek an explanation of a different nature—one that is more intuitive and accessible, especially for readers without extensive mathematical training. Rather than invoking external mathematical frameworks, we aim for a mechanistic account grounded solely in common knowledge and basic principles familiar to the deep learning community. Specifically, we investigate the hypothesis that Muon can be understood as an implicit residual connection, reducing the question of why Muon works well to the question of why residual connections work well. 2.3
R ESIDUAL C ONNECTIONS IN D EEP L EARNING
Residual connections have become one of the most fundamental architectural components in modern deep learning since their introduction in ResNets (He et al., 2016b). The basic idea is to add the input of a block directly to its output: xl+1 = xl + Block(xl ), where the identity shortcut carries xl forward unchanged while Block applies the usual nonlinear transformation. This simple modification substantially alleviates optimization difficulties in very deep networks and has enabled the successful training of models with hundreds or even thousands of layers (He et al., 2016b). Residual connections have been studied from a variety of theoretical perspectives. Existing explanations include identity-mapping views, which argue that residual blocks make it easier for layers to represent or stay close to the identity mapping (He et al., 2016a); path-ensemble views, which interpret ResNets as ensembles of many paths of different effective depths and attribute their trainability to short gradient-carrying paths (Veit et al., 2016); and dynamical-systems views, in which very deep residual networks are regarded as discretizations of ordinary differential equations (Chen et al., 2018). Today, residual connections constitute a core design principle across convolutional networks and Transformers. In this paper, we emphasize a mechanistic role of residual connections: they pass both xl and Block(xl ) to downstream layers. In other words, downstream layers need not rely exclusively 2
Preprint
on the transformed representation, but can choose, mix, ignore, or amplify information from the preserved upstream representation and the newly transformed representation (Longon, 2024).
3
M UON AS AN I MPLICIT R ESIDUAL C ONNECTION
3.1
O RTHOGONAL U PDATES AS R ESIDUAL C ONNECTION
Residual connections explicitly transform the representation as x ← x + Block(x). When Block is a linear layer, this is equivalent to x ← (I + W )x, or, equivalently, W ← W + I. More generally, the identity matrix is not essential: any orthogonal matrix R can equally serve as the skip connection, W ← W + R. During optimization, the weight matrix evolves according to W ← W + ∆W. Muon replaces the update by its orthogonalized counterpart, W ← W + Orth(∆W ), where Orth(∆W ) denotes the nearest orthogonal matrix to ∆W . From this perspective, residual connections explicitly add an orthogonal transformation to the network weights, whereas Muon implicitly injects an orthogonal transformation through the optimization trajectory. This is intended as a mechanistic explanation rather than a strict mathematical equivalence between Muon and residual architectures. The key analogy is that both mechanisms create an option for downstream computation. A residual block explicitly exposes both the preserved upstream representation and the transformed representation to later layers. Muon, by contrast, creates this option through the training dynamics: the update must still serve as a descent direction for the current parameter, but its orthogonalized component also tends to preserve and reshape the representation in a way that remains easy for downstream layers to use. Thus, Muon’s update can be understood as serving two coupled roles: preserving gradient fidelity for the current optimization step while promoting representation preservation for downstream layers. 3.2
T HE G RADIENT F IDELITY–R EPRESENTATION P RESERVATION T RADE - OFF
Orthogonalizing the update inevitably modifies its original direction. On the one hand, remaining closer to the raw update preserves a larger component along the gradient direction, leading to a greater first-order decrease in the objective, namely higher gradient fidelity. On the other hand, as discussed in the previous section, orthogonal updates implicitly inject residual transformations into the network weights, allowing representations learned by earlier layers to be propagated to deeper layers with less distortion and resulting in stronger representation preservation. Muon can therefore be viewed as balancing these two competing objectives. This perspective is complementary to previous trade-off interpretations of Muon. For example, recent work argues that Muon balances batch gradient alignment and directional descent potential (Shumaylov et al., 2026). In contrast, we interpret Muon as balancing gradient fidelity, which favors immediate loss reduction, and representation preservation, which favors preserving representations for subsequent layers through residual transformations. 3
Preprint
4
A T WO -P HASE L INEAR M ODEL
4.1
P ROBLEM S ETUP AND M OTIVATION
Gradient backpropagation is the standard paradigm for training deep neural networks. It decomposes the optimization of the overall loss into layer-wise subproblems by propagating gradients through the chain rule, indicating how changing the parameters of each layer affects the final objective. An alternative paradigm is target propagation (Lee et al., 2015), which directly assigns each layer a desired target representation rather than a gradient direction. Although the optimization mechanisms differ, both approaches share the same underlying philosophy: decomposing a global optimization problem into a collection of local subproblems. Our experiment is motivated by this viewpoint, although we do not implement target propagation itself. Instead, we use it as a conceptual framework for analyzing different optimization strategies. Suppose a target transformation A is assigned to a linear layer with weight matrix W . The local optimization problem is W → A, which is achieved through the additive update W ← W + ∆W. Under this local objective, SGD follows the steepest descent direction and is therefore expected to approach A faster than Muon, whose update is first orthogonalized before being applied. If this local objective were the only objective of interest, SGD would naturally be preferred. This immediately raises two questions. Is the local objective itself the one that ultimately matters? Even if it is, must it necessarily be realized by the current layer? To answer these questions, consider two consecutive linear layers with weights W1 and W2 . Realizing the target through the current layer corresponds to the additive update W1 ← W1 + ∆W1 . Alternatively, the same target transformation may be realized by adapting the downstream layer so that W2 W1 → A. Updating the second layer, W2 ← W2 + ∆W2 , induces ∆(W2 W1 ) ≈ (∆W2 )W1 , which corresponds to a multiplicative update of W1 . This suggests that the layer-wise objective can be redistributed: part of the additive update that would otherwise be performed by the current layer can be amortized into multiplicative updates in downstream layers. Motivated by this observation, we present an experiment to demonstrate one possible mechanism: although Muon performs slower additive updates than SGD at the current layer, it can induce faster downstream multiplicative updates, leading to faster overall optimization than SGD. We consider a two-layer linear network ŷ = W2 W1 x, where the target transformation is a Gaussian random matrix A ∈ Rd×d ,
i.i.d.
Aij ∼ N (0, 1/d),
d = 64.
We compute the singular value decomposition A = U ΣV ⊤ , and construct one valid factorization W1⋆ =
√
√ W2⋆ = U Σ,
ΣV ⊤ , 4
Preprint
Phase 1 optimizer SGD Muon
∥W1 − W1⋆ ∥F 0.0312 0.0447
W1 flatness 0.4301 0.4389
Phase 2 steps 13436 13321
total steps 14036 13921
final error 0.049999 0.049999
Table 1: Two-phase linear experiment. Distances are normalized by ∥A∥F . Spectrum flatness is ∥W1 ∥2F /∥W1 ∥22 divided by d (an effective-rank proxy; higher means a flatter singular spectrum). SGD reaches a smaller Phase 1 local error, while Muon’s W1 is flatter and requires fewer downstream SGD steps in Phase 2. which satisfies W2⋆ W1⋆ = A. The first factor W1⋆ serves as the local target of the first layer, while the overall objective remains realizing the global transformation A. The experiment consists of two phases. In Phase 1, we freeze W2 and optimize W1 toward the local target 1 min ∥W1 − W1⋆ ∥2F . W1 2 We compare SGD and Muon under identical learning rates and optimization budgets. In Phase 2, the learned W1 is frozen, while W2 is randomly reinitialized and optimized using SGD in both settings by minimizing 1 min ∥W2 W1 − A∥2F . W2 2 Since Phase 2 uses exactly the same optimizer, hyperparameters, and initialization strategy in both experiments, the only difference is the representation learned by W1 during Phase 1. Consequently, any difference in convergence speed directly reflects how easily the learned representation can be exploited by downstream layers. The executed experiment uses d = 64. Phase 1 runs for 600 steps with learning rate 0.2. Phase 2 uses SGD with learning rate 0.02 and stops when ∥W2 W1 − A∥F ≤ 0.05. ∥A∥F Distances in Table 1 are normalized by ∥A∥F . 4.2
P HASE 1: A NCHORING W1
Phase 1 intentionally favors SGD by evaluating only the local objective L1 (W1 ) =
1 ∥W1 − W1⋆ ∥2F . 2
Since SGD follows the steepest descent direction of this objective, whereas Muon first orthogonalizes the update before applying it, we expect SGD to approach W1⋆ more rapidly. The result in Table 1 confirms this expectation. After the same 600 Phase 1 steps, SGD reaches a normalized reconstruction error of 0.0312, while Muon reaches 0.0447. Thus Muon sacrifices local optimization efficiency on this layer-wise subproblem. 4.3
P HASE 2: C OMPOSING W2 W1 = A
Phase 2 evaluates whether the representation learned during Phase 1 facilitates downstream optimization. The learned W1 is frozen, while W2 is optimized from scratch by minimizing L2 (W2 ) =
1 ∥W2 W1 − A∥2F . 2 5
Preprint
Unlike Phase 1, both optimization paths now use exactly the same SGD optimizer. Therefore, any difference in convergence speed must originate solely from the representation encoded by W1 , rather than from differences between optimization algorithms. Interestingly, although Muon converges more slowly in Phase 1, it requires fewer optimization steps in Phase 2. With SGD-trained W1 , the downstream layer needs 13436 steps to reach the threshold; with Muon-trained W1 , it needs 13321 steps, saving 115 downstream steps. This indicates that the representation learned by Muon is easier for the downstream layer to utilize despite being farther from the local target W1⋆ . 4.4
W HY O RTHOGONAL W1 ACCELERATES P HASE 2
The behavior observed in Phase 2 admits a simple theoretical explanation. With W1 fixed, the downstream objective is L(W2 ) =
1 ∥W2 W1 − A∥2F , 2
∇W2 L = (W2 W1 − A)W1⊤ .
Letting Et = W2,t W1 − A, a single SGD step implies Et+1 = Et (I − ηW1⊤ W1 ). Thus, downstream convergence is governed by the spectrum of W1⊤ W1 , rather than only by how close W1 is to the local target W1⋆ ; see Appendix C.1 for the singular-mode derivation. A flatter singular spectrum and smaller condition number make the downstream problem better conditioned. This explains how Muon can sacrifice local optimization accuracy in Phase 1 while enabling faster downstream optimization in Phase 2. The same spectral argument also connects this experiment to linear probing: a better-conditioned fixed representation makes downstream linear prediction easier. We discuss this connection in Appendix B.2.
5
τ S CHEDULING
5.1
F ROM J OINT U PDATES TO S EGMENT A LTERNATION
The two-phase experiment above deliberately separates representation learning from downstream adaptation. We next ask whether the same trade-off remains visible when both layers are trained end-to-end on the composition loss. To this end, we use a simple τ schedule. Consider the same two-layer linear model ŷ = W2 W1 x, optimized directly using the composition loss 2
e 1 , W2 ) = ∥W2 W1 − A∥F . L(W ∥A∥2F The parameter τ controls how long the other layer is frozen: τ = 0, joint updates of W1 and W2 at every step; τ ≥ 1, τ steps updating only W1 , followed by τ steps updating only W2 . Here τ sets the segment length of the alternating updates. Larger τ makes the W1 -only and W2 -only phases more pronounced. Throughout this section, both methods optimize the same composition loss from the same initialization. The downstream layer W2 is always updated using SGD. The only difference is the optimizer used for W1 : the SGD path uses SGD for both layers, while the Muon path uses Muon for W1 and SGD for W2 . 6
Preprint
5.2
E MPIRICAL B EHAVIOR OF THE τ S CHEDULE
We run the sweep with d = 64, learning rate 0.02, and stop when ∥W2 W1 − A∥F ≤ 0.05. ∥A∥F The number of optimization steps required to reach this threshold is reported below. schedule joint (τ = 0) τ =1 τ = 10 τ = 50 τ = 200 τ = 600
SGD on W1 13054 26109 26122 26165 26221 26553
Muon on W1 11535 23064 23077 23064 23352 21975
steps saved 1519 3045 3045 3101 2869 4578
Muon reaches the target loss faster for every schedule tested, including the fully joint case τ = 0; the full log-scale trajectories and early linear-scale behavior are shown together in Figure 1. The alternating schedules require roughly twice as many total steps as joint training because only one factor is updated at each step, but the comparison within each schedule remains matched: the two paths differ only in the W1 optimizer. The advantage of Muon is particularly visible for larger segment lengths, where the downstream effect of the learned representation has more time to appear within each W2 segment. 5.3
S EGMENT- WISE DYNAMICS OF THE τ S CHEDULE
The colored segments in Figure 1 separate two effects. Blue segments update only W1 while keeping W2 fixed, exposing the local W1 optimization problem; red segments update only W2 while keeping W1 fixed, testing how easily the representation produced by W1 can be used downstream. This distinction supports the same mechanism as the two-phase experiment. Early blue segments often favor SGD, reflecting Muon’s gradient-fidelity cost on the local subproblem. Muon’s advantage appears more clearly once downstream adaptation has occurred: red segments can exploit the representation produced by the Muon trajectory, and the improved downstream layer can then make later W1 subproblems easier. Detailed loss-gap and replay diagnostics are provided in Appendix B.1. Overall, Muon may sacrifice immediate decrease on a layer-wise subproblem while producing representations that make downstream optimization easier.
7
Preprint
Figure 1: Composition loss under the τ schedule. Top: full trajectories on a logarithmic scale. Bottom: early-step zoom on a linear scale. Solid curves use SGD for W1 , while dashed curves use Muon for W1 ; W2 is always trained with SGD. Colors indicate which layer is updated at each step: joint updates in green, W1 -only segments in blue, and W2 -only segments in red.
8
Preprint
6
D ISCUSSION
6.1
I MPLICATIONS FOR O PTIMIZER D ESIGN
The central design implication is that an optimizer need not be judged only by its immediate loss decrease on each layer-wise subproblem. An update can also shape the representation that downstream layers will later consume. From this perspective, Muon suggests a class of optimizers that explicitly balance gradient fidelity, which favors local descent, and representation preservation, which favors downstream usability. A natural direction is to make this balance adaptive, for example by controlling the strength of orthogonalization or spectral flattening according to layer spectra, conditioning, or downstream progress. This view also suggests when Muon-like behavior should be most valuable. The payoff should be larger when downstream layers have enough capacity and training time to exploit a betterconditioned representation, and when optimization involves long chains of composed transformations. Conversely, when the main objective is nearly local, or when downstream adaptation is weak, the cost of reduced gradient fidelity may dominate. 6.2
L IMITATIONS AND F UTURE W ORK
Our experiments are deliberately controlled and linear. They demonstrate one possible mechanism by which Muon’s slower local additive updates can be offset by faster downstream multiplicative adaptation, but they do not constitute a systematic study of the conditions under which this payoff appears in modern nonlinear networks. In particular, future work should test how the effect depends on depth, residual architecture, layer type, data imbalance, optimizer hyperparameters, and the amount of downstream adaptation available. Another limitation is that our residual-connection analogy is mechanistic rather than architectural. Residual connections provide an option by reparameterizing the forward computation so that both xl and Block(xl ) are explicitly available, whereas Muon provides a related option indirectly through the optimization trajectory. Muon does not insert an explicit skip path into the network, and real residual connections usually span an entire block rather than a single linear layer. Thus, the analogy is intended to highlight a shared option-like role for downstream computation, not to claim that the two mechanisms are interchangeable.
7
C ONCLUSION
We presented a mechanistic interpretation of Muon as an implicit residual connection during optimization. In this view, orthogonalized updates trade some immediate gradient fidelity for representation preservation, making the resulting representations easier for downstream layers to use. Controlled two-phase and τ -schedule linear experiments support this possibility: Muon can be slower on a local layer-wise target yet faster end-to-end because its learned representation improves downstream optimization. This perspective complements existing spectral, trust-region, curvature, and long-tail explanations, and suggests optimizer designs that reason explicitly about the payoff between local descent and downstream representation usability.
R EFERENCES Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024. Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, volume 31, 2018. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pp. 630–645, 2016a. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016b. 9
Preprint
Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non-euclidean trust-region optimization. arXiv preprint arXiv:2503.12645, 2025. Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. arXiv preprint arXiv:1412.7525, 2015. André Longon. Interpreting the residual stream of ResNet18. arXiv preprint arXiv:2407.05340, 2024. Gabriel Peyré. Muon dynamics as a spectral wasserstein flow. arXiv preprint arXiv:2604.04891, 2026. Maria-Eleni Sfyraki and Jun-Kun Wang. Lions and muons: Optimization via stochastic frank–wolfe. arXiv preprint arXiv:2506.04192, 2025. Zakhar Shumaylov, Nathael Da Costa, Peter Zaika, Bálint Mucsányi, Alex Massucco, Yoav Gelberg, Carola-Bibiane Schönlieb, Yarin Gal, and Philipp Hennig. Muon is not that special: Random or inverted spectra work just as well. arXiv preprint arXiv:2605.11181, 2026. Bhavya Vasudeva, Puneesh Deora, Yize Zhao, Vatsal Sharan, and Christos Thrampoulidis. How Muon’s spectral design benefits generalization: A study on imbalanced data. arXiv preprint arXiv:2510.22980, 2025. Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. In Advances in Neural Information Processing Systems, volume 29, 2016. Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang, Zhuoran Yang, Mingyi Hong, and Vincent Y. F. Tan. Muon outperforms Adam in tail-end associative memory learning. arXiv preprint arXiv:2509.26030, 2025. Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Dirk Bergemann, and Zhuoran Yang. Why Muon outperforms Adam: A curvature perspective. arXiv preprint arXiv:2606.04662, 2026.
A
D ETAILED E XISTING I NTERPRETATIONS OF M UON
For completeness, we provide a more detailed summary of the existing interpretations of Muon discussed in the main text. A.1
S TEEPEST D ESCENT UNDER THE S PECTRAL N ORM
Bernstein & Newhouse (2024) interpret Muon as steepest descent under the matrix spectral norm. Let G = ∇f (W ) and define Orth(G) = U V ⊤ ,
G = U ΣV ⊤ .
The steepest descent direction under a norm ∥ · ∥ is obtained by solving D ∈ arg min ⟨G, D⟩. ∥D∥≤1
When ∥ · ∥ is the matrix spectral norm ∥ · ∥2 , its dual norm is the nuclear norm ∥ · ∥∗ . Therefore, min ⟨G, D⟩ = − max ⟨G, D⟩ = −∥G∥∗ .
∥D∥2 ≤1
∥D∥2 ≤1
Using the singular value decomposition G = U ΣV ⊤ , the maximizer is D = U V ⊤ = Orth(G), and hence the steepest descent direction is D∗ = − Orth(G). Equivalently, the associated non-Euclidean gradient direction, obtained from the duality mapping, is G# = ∥G∥∗ Orth(G), which leads to the update W ← W − η∥G∥∗ Orth(G). 10
Preprint
A.2
N ON -E UCLIDEAN T RUST-R EGION M ETHODS
Rather than viewing Muon as steepest descent, Kovalev (2025) derive it exactly as the solution of a non-Euclidean trust-region subproblem, min ⟨G, ∆⟩,
s.t. ∥∆∥2 ≤ η,
∆
where ∥·∥2 denotes the matrix spectral norm. Exploiting the duality between the spectral and nuclear norms, the optimal update admits the closed-form solution ∆∗ = −η Orth(G), which coincides exactly with the Muon update W ← W − η Orth(G). Unlike the steepest-descent interpretation, this trust-region formulation recovers Muon without the additional nuclear-norm scaling factor. Although both interpretations identify the orthogonalized gradient as the update direction, they differ in the treatment of the step magnitude. The steepest-descent interpretation introduces an additional factor ∥G∥∗ arising from the duality mapping of the spectral norm, whereas the trust-region formulation recovers the Muon update exactly with a fixed trust-region radius. A.3
S PECTRAL -N ORM -C ONSTRAINED U PDATES WITH W EIGHT D ECAY
Recent work argues that Muon combined with decoupled weight decay should be viewed as optimizing under an implicit spectral-norm constraint (Sfyraki & Wang, 2025). Unlike the previous interpretations, which focus on the update direction of a single optimization step, this perspective emphasizes the constrained optimization trajectory rather than the update direction of a single step. The central idea originates from the Frank–Wolfe algorithm, which maintains the iterates inside a prescribed convex set throughout optimization. At each iteration, Frank–Wolfe first solves the linear minimization oracle (LMO) S = arg min⟨G, S⟩, S∈C
where C is the feasible set, and then moves the current iterate toward the selected extreme point, W ← (1 − γ)W + γS. When the feasible set is the spectral-norm ball C = {W : ∥W ∥2 ≤ r}, the solution of the LMO is S = −r Orth(G). Consequently, Frank–Wolfe naturally selects the orthogonalized gradient as the target point and updates the weight by interpolating between the current weight matrix and this extreme point. Building on this observation, Sfyraki & Wang (2025) show that Muon with decoupled weight decay is exactly a stochastic Frank–Wolfe method on a spectral-norm ball, with weight decay keeping the trajectory inside the feasible region and the orthogonalized gradient selecting the extreme point to approach. As a result, Muon is interpreted as optimizing under an implicit spectral-norm constraint rather than merely following an orthogonalized gradient direction at each step. A.4
R ARE D IRECTIONS AND L ONG -TAIL L EARNING
Another line of work connects Muon to rare directions in the gradient spectrum and to learning under heavy-tailed data. By replacing the singular values of a matrix gradient with a common scale while preserving its singular vectors, orthogonalization prevents a few high-energy directions from dominating the update and gives relatively larger weight to weaker spectral components. Vasudeva et al. (2025) study this mechanism systematically through spectral gradient descent (SpecGD), the canonical form of Muon in which each update is U V ⊤ for a gradient G = U ΣV ⊤ . They show 11
Preprint
that unlike Euclidean gradient descent, which learns dominant principal components of the data first, SpecGD learns all principal components at comparable rates. On imbalanced classification problems, this spectral design yields better class-balanced loss and minority-group generalization early in training, and experiments with Muon and Shampoo confirm the same trend in practical settings. This perspective has been extended to large language models through associative memory. Wang et al. (2025) argue that Muon’s advantage over Adam is concentrated in the associativememory parameters of Transformers—notably the Value–Output attention weights and feed-forward networks—whose updates have an outer-product structure. On heavy-tailed corpora, Muon produces a more isotropic singular spectrum than Adam and therefore learns tail classes more effectively, while matching Adam on frequent head classes; their one-layer associative-memory analysis further shows that Muon maintains balanced class-wise learning under imbalance, whereas Adam can exhibit large disparities depending on the representation embeddings. A.5
C URVATURE
Wang et al. (2026) explain Muon’s empirical advantage from the perspective of the local curvature of the loss landscape. Using a second-order Taylor expansion, they show that Muon and Adam achieve comparable first-order gains but differ substantially in their second-order curvature penalties. Specifically, the one-step loss decrease can be decomposed into a gradient-alignment term and a curvature term, and Muon consistently incurs a smaller curvature penalty. This advantage is further attributed to a lower Normalized Directional Sharpness (NDS), rather than a smaller update norm. The paper further shows that data imbalance amplifies Muon’s NDS advantage and proves, on stylized quadratic problems with heterogeneous curvature, that Muon’s spectrally normalized updates distribute update energy more evenly across curvature modes, leading to lower average directional sharpness and faster optimization. A.6
S PECTRAL WASSERSTEIN F LOW
Peyré (2026) interpret Muon as the finite-particle limit of a continuous-time Spectral Wasserstein gradient flow. In the mean-field regime, optimization is formulated over probability measures equipped with a family of Spectral Wasserstein distances indexed by matrix norms. The trace norm recovers the classical quadratic Wasserstein geometry, the operator norm corresponds to Muon, and intermediate Schatten norms interpolate between the two. This interpretation establishes a unified optimal transport framework encompassing both Euclidean and spectrally normalized optimization. A.7
A LIGNMENT AND D IRECTIONAL D ESCENT P OTENTIAL
Shumaylov et al. (2026) offer a complementary and partly skeptical view of explanations that attribute Muon’s success to its exact orthogonalized spectrum. They argue that Muon’s empirical advantage is better understood through two local quantities: batch gradient alignment, which measures how well the update direction agrees with individual example gradients, and directional descent potential, which measures how much descent can be obtained along the chosen direction. From this perspective, exact orthogonalization is not uniquely responsible for Muon’s performance. Their proposed variants, including Freon and Kaon, modify the singular spectrum in random or inverted ways while retaining competitive behavior, suggesting that a broader class of spectrally shaped updates can preserve the relevant alignment–descent trade-off.
B
A DDITIONAL E XPERIMENTAL D ETAILS
B.1
S EGMENT- WISE D IAGNOSTICS OF THE τ S CHEDULE
The colored segments in Figure 1 separate two effects. Blue segments update only W1 while keeping W2 fixed, and therefore expose the local W1 optimization problem. Red segments update only W2 while keeping W1 fixed, and therefore test how easily the representation produced by W1 can be used by the downstream layer. 12
Preprint
This distinction is important. In early blue segments, SGD often decreases the composition loss more rapidly than Muon, reflecting the gradient-fidelity cost discussed earlier. The advantage of Muon appears more clearly in the red segments: its W1 trajectory can produce a representation that is easier for W2 to exploit. After such red segments improve W2 , the resulting downstream layer also makes later blue-segment W1 subproblems easier. This blue-slow, red-fast, easier-next-blue feedback loop can compensate for Muon’s per-step cost on W1 and still yield faster end-to-end convergence.
Figure 2: Loss gap LSGD − LMuon across the full τ -schedule sweep. Negative values mean the SGD path has lower loss at the same step, while positive values mean the Muon path is ahead. The colored segments mark which layer is active.
Figure 3: Early-step zoom of the loss gap in Figure 2. The zoom highlights the initial localoptimization advantage of SGD on W1 segments before Muon’s representation advantage propagates through later W2 updates. Figure 2 plots this effect directly as the loss gap LSGD − LMuon , with Figure 3 zooming in on the first 2000 steps. The early negative gaps show SGD’s local advantage on fixed-W2 blue segments, whereas later positive gaps show the accumulated benefit of the Muon trajectory once the downstream layer has adapted. Why, then, can Muon appear to improve faster than SGD even inside mid-to-late blue segments, despite still orthogonalizing its updates? The answer is path-dependent rather than a claim that Muon has become the better local optimizer on an identical state. In the τ = 200 run, Figure 4 (top) records the spectral flatness of W2 at the start of each blue segment, using the same effective13
Preprint
rank proxy as in Table 1. The Muon path typically leaves a flatter W2 spectrum before each blue segment; equivalently, the fixed-W2 subproblem faced by W1 is easier on the Muon trajectory. Muon can therefore gradually overtake SGD in later blue segments even while still sacrificing single-step speed to orthogonalization.
Figure 4: Mechanism diagnostic for τ = 200. Top: spectral flatness of W2 at the start of each W1 segment (higher means a flatter singular spectrum). The Muon path tends to leave a flatter downstream layer before each blue segment. Bottom: loss gap LSGD − LMuon ; blue portions are W1 -only segments and pale red portions are W2 -only segments. Negative gap means SGD has lower loss. The orange dashed replay, computed at each blue-segment start from the Muon-path (W1 , W2 ) with SGD used for W1 in that segment only, tests whether Muon’s mid-to-late catch-up is due to faster local optimization. The bottom panel separates two sources of blue-segment behavior. When the gap decreases during a blue segment, SGD is reducing L faster and widening its lead on the local W1 subproblem, as expected from gradient fidelity. When the gap increases during a blue segment, Muon is catching up or pulling ahead even though only W1 is being updated. The orange replay shows that this midto-late reversal is not because Muon has become locally superior on the same subproblem: if one fixes the Muon-path state at a blue-segment start and swaps Muon for SGD on W1 in that segment alone, SGD can decrease the loss even faster on that same subproblem. Muon’s mid-to-late bluesegment gains therefore reflect an easier trajectory-induced subproblem created by earlier red and blue segments, not faster local optimization on an identical downstream state. These observations extend the two-phase message to end-to-end training. Muon may sacrifice immediate decrease on a layer-wise subproblem, yet the representations produced along its trajectory can make downstream optimization easier, and that downstream improvement can feed back into the remaining upstream optimization. B.2
C ONNECTION TO L INEAR P ROBING
The second phase of our experiment can be viewed as a simplified linear probing task. Instead of directly optimizing the matrix product W2 W1 → A, consider a fixed encoder W1 that produces the representation h = W1 x, 14
Preprint
while a linear probe B is trained to recover a target transformation M . Depending on the choice of M , this includes both upstream reconstruction (M = I) and downstream prediction (M = A): LM (B) =
1 ∥BW1 − M ∥2F . 2
Let Et = Bt W1 − M. A gradient descent step gives Bt+1 = Bt − η(Bt W1 − M )W1⊤ , which implies Et+1 = Et (I − ηW1⊤ W1 ). Therefore, the convergence rate is again completely determined by the spectrum of W1 . A flatter singular spectrum leads to faster convergence of the linear probe. This establishes a direct connection between our two-phase linear model and the standard linear probing protocol widely used in representation learning. Although Muon may optimize the local objective more slowly, its tendency to produce better-conditioned representations can make them easier for downstream linear predictors to exploit.
C
P ROOFS AND D ERIVATIONS
C.1
P HASE 2 C ONVERGENCE UNDER A F IXED R EPRESENTATION
Suppose W1 = U ΣV ⊤ . Then and the error recursion
W1⊤ W1 = V Σ2 V ⊤ , Et+1 = Et (I − ηW1⊤ W1 )
implies that each singular component of the error evolves independently as si (Et+1 ) = |1 − ησi2 | si (Et ), where σi denotes the i-th singular value of W1 . Choosing α , η= σmax (W1 )2 the slowest contraction factor becomes 1−
α , κ(W1 )2
where κ(W1 ) is the condition number of W1 .
15