Blog: Survey of Optimizers

Blog: Survey of Optimizers Ruoran Xu Literature current through 28 August 2026

arXiv:2608.28557v1 [cs.LG] 28 Aug 2026

Abstract Neural-network optimization in 2025–2026 is not well described as a parade of new Adam variants. The design unit has expanded: from coordinates to matrices and layers, from a fixed training horizon to a policy over time, and from a mathematical update rule to a state representation that must survive sharding and low precision. This blog survey organizes recent optimizers and trainingoptimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-reduced optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is deliberately non-triumphal: matrix-aware methods are a real advance, but there is no context-free replacement for AdamW. Rankings change with scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the objective is tokens, FLOPs, wall-clock time, or memory. The practical consequence is a compositional view of optimizer design and a stricter evaluation protocol.

1

Blog: Survey of Optimizers

August 2026

Contents 1 The optimizer question has changed 1.1

5

How to read the evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 Foundations: what an optimizer chooses

6 6

2.1

Direction, metric, and representation . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

2.2

From Euclidean steepest descent to norm-aware updates . . . . . . . . . . . . . . . . .

7

2.3 The optimizer state as a statistical object . . . . . . . . . . . . . . . . . . . . . . . . .

7

2.4

Parameter classes are not interchangeable . . . . . . . . . . . . . . . . . . . . . . . . .

8

2.5

Five budgets, five different winners . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

3 Why AdamW remains the reference point 3.1

More careful use of gradient history

9

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

3.2 A closer look at temporal estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10

3.3 Agreement filters, resets, and robustness . . . . . . . . . . . . . . . . . . . . . . . . . .

10

3.4 ADOPT and the timing of normalization . . . . . . . . . . . . . . . . . . . . . . . . . .

11

4 The matrix-aware renaissance

12

4.1 Whitening: Shampoo, SOAP, and stable descendants . . . . . . . . . . . . . . . . . . .

12

4.2

Spectral steepest descent: what Muon is and is not . . . . . . . . . . . . . . . . . . . .

12

4.3 Adaptivity after orthogonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

4.4 What controlled comparisons actually say . . . . . . . . . . . . . . . . . . . . . . . . .

13

5 Dissecting matrix optimizers

14

5.1

Muon’s Newton–Schulz map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

5.2

Shape scaling, weight norms, and angular learning rate . . . . . . . . . . . . . . . . . .

15

5.3 Whitening is not one operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

5.4 Why hybrids proliferate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

5.5

Communication can be an optimizer operation . . . . . . . . . . . . . . . . . . . . . .

16

5.6 A decision tree inside the matrix family . . . . . . . . . . . . . . . . . . . . . . . . . .

17

6 The emerging layerwise optimizer stack 6.1

17

Equivariance as a routing principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

6.2 The cost of compositionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

7 Memory is part of the algorithm 7.1

18

Compress the statistic, not automatically the update . . . . . . . . . . . . . . . . . . .

2

18

Blog: Survey of Optimizers

7.2

August 2026

State-free and near-state-free alternatives . . . . . . . . . . . . . . . . . . . . . . . . .

8 Memory-efficient optimization as statistical compression

19 19

8.1 A concrete memory ledger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20

8.2

Projection changes coordinates over time . . . . . . . . . . . . . . . . . . . . . . . . . .

21

8.3 When state should be allocated unevenly . . . . . . . . . . . . . . . . . . . . . . . . . .

21

8.4

Memory savings alter the statistical experiment . . . . . . . . . . . . . . . . . . . . . .

21

8.5 A fixed-memory evaluation frontier . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

9 Time: schedules, averaging, and unknown horizons

22

9.1

Schedule-free does not mean time-free . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

9.2

Batch size changes the optimizer’s clock . . . . . . . . . . . . . . . . . . . . . . . . . .

23

10 Horizon management in detail

23

10.1 Cosine decay and the hidden endpoint . . . . . . . . . . . . . . . . . . . . . . . . . . .

23

10.2 Stable trunks and cooldown branches . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

10.3 Schedule-free interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

10.4 Checkpoint merging as temporal optimization . . . . . . . . . . . . . . . . . . . . . . .

24

10.5 Dual averaging and regularization schedules . . . . . . . . . . . . . . . . . . . . . . . .

25

10.6 Batch, accumulation, and the unit of time . . . . . . . . . . . . . . . . . . . . . . . . .

25

11 Low precision: state semantics matter more than bit count

25

12 Quantized optimizer dynamics

27

12.1 Blockwise dynamic quantization as the baseline . . . . . . . . . . . . . . . . . . . . . .

27

12.2 Why first and second moments need different codebooks . . . . . . . . . . . . . . . . .

27

12.3 State staleness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

12.4 Why Muon is unusually quantization tolerant . . . . . . . . . . . . . . . . . . . . . . .

28

12.5 Eliminating master weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

12.6 Dynamic precision allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

12.7 Optimizer choice can change later quantizability . . . . . . . . . . . . . . . . . . . . .

29

13 A synthesis: five claims that survive the leaderboard

29

13.1 Matrix-aware optimization is real, but second order is too coarse . . . . . . . . . . . .

29

13.2 Variance adaptation keeps returning . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

13.3 The best geometry is parameter-class dependent . . . . . . . . . . . . . . . . . . . . .

30

13.4 Memory and systems can reverse an algorithmic result . . . . . . . . . . . . . . . . . .

30

13.5 There is no universal twofold replacement for AdamW . . . . . . . . . . . . . . . . . .

30

3

Blog: Survey of Optimizers

August 2026

14 Why optimizer leaderboards disagree

30

14.1 Scale and data ratio are separate axes . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

14.2 Batch size changes both noise and systems . . . . . . . . . . . . . . . . . . . . . . . . .

31

14.3 Hyperparameter transfer can be unfair in both directions . . . . . . . . . . . . . . . .

31

14.4 The endpoint metric can hide practical value . . . . . . . . . . . . . . . . . . . . . . .

31

14.5 Reading paper-specific speedup claims . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

14.6 Evidence tiers should apply to claims, not papers . . . . . . . . . . . . . . . . . . . . .

32

14.7 A reproducible benchmark matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

33

15 A practical decision guide

34

16 A minimum credible evaluation protocol

34

17 Open problems for the next optimizer cycle

35

18 Conclusion

36

A Compact literature map

37

4

Blog: Survey of Optimizers

August 2026

Table 1: A functional map of the recent optimizer landscape. Persistent state is approximate and omits implementation-dependent master weights. Axis

Question

Temporal estimation

Which past gradients should influence the next step?

Geometry

Representative mechanisms

Fast/slow EMAs, recursive variance reduction, agreement masks, spike resets What counts as an equally sized Diagonal RMS scaling, update? Kronecker whitening, polar factors, row/column normalization

Horizon

How should progress be exposed Cosine or WSD schedules, over time? cooldowns, averaging, schedule-free interpolation

Representation

What can be stored and moved? Block statistics, low-rank moments, stateless transforms, FP8/INT8/2–4 bit states

Typical failure mode Stale or noisy history; sensitivity to β and batch size Costly state; unstable scaling; inappropriate treatment of non-matrix tensors Unknown stopping time; endpoint quality that hides pathwise progress Quantization staleness; changing-basis errors; communication bottlenecks

1 The optimizer question has changed For most practitioners, the optimizer question used to be a short menu: SGD with momentum, Adam, or AdamW. The choice mattered, but the rest of the training stack could be discussed separately. That separation is breaking down. A modern optimizer may decide how long to remember a gradient, which matrix norm defines a small update, whether a low-rank subspace should carry state, how learning-rate decay is emulated when the stopping time is unknown, and how moment estimates are encoded in 8, 4, or even fewer bits. This change is easiest to miss when every proposal is treated as one more named optimizer. The useful unit of analysis is instead an optimizer stack. For a parameter block θt and stochastic gradient gt , write schematically zt = Tt (g1:t ),

ut = Pt (zt ; St ),

bt = Qt (ut ), u

bt , ηt , λt ). θt+1 = Rt (θt , u

(1)

Here Tt is a temporal estimator such as momentum or a variance-reduced recursion; Pt sets the geometry through diagonal, Kronecker, spectral, or low-rank structure; Qt is the stored and communicated representation; and Rt includes learning-rate policy, averaging, and regularization. Two optimizers with different names can modify the same box. Conversely, a single implementation can quietly combine ideas from all four. This review focuses on work released from January 2025 through August 2026, with a small number of late-2024 precursors needed to make the lineage intelligible. It is a field guide rather than a claim of exhaustive bibliometrics. The emphasis is large-scale language-model pretraining because that is where the interaction between convergence, memory, and systems is most visible, but most mechanisms are more general.

5

Blog: Survey of Optimizers

August 2026

1.1 How to read the evidence Optimizer papers are unusually easy to compare unfairly. A result can be true within its protocol and still reverse under another horizon or batch size. Throughout this article, claims should be read at one of three levels: • Scaled evidence: experiments at roughly billion-parameter scale or long multi-billion-token horizons. This is valuable systems evidence, not universality. • Controlled evidence: careful sweeps, ablations, or multiple scales designed to isolate components. These studies may use smaller models precisely because control is expensive. • Emerging evidence: a recent preprint, limited scale, or a method whose independent replication is not yet mature. Numbers below are attributed to the corresponding study and are not presented as directly comparable league-table scores. Faster may mean fewer tokens to a validation loss, fewer FLOPs, shorter wall-clock time, or a larger feasible batch. These are different claims.

2 Foundations: what an optimizer chooses The conventional update θt+1 = θt − ηt gt hides almost every choice that matters in contemporary training. An optimizer chooses an estimator of direction, a metric for measuring the size of that direction, a temporal policy, a regularizer, a representation for persistent state, and a distribution strategy. The recent literature becomes much easier to parse when these choices are separated.

2.1 Direction, metric, and representation Let Gt ∈ Rm×n denote the gradient of a matrix-valued parameter block. A general preconditioned step can be written ∆Wt = −ηt At Φt (G1:t ) Bt , (2) where Φt is a temporal estimator and At , Bt encode left and right geometry. Several familiar optimizers are special cases. • In momentum SGD, Φt is an exponential moving average and At = Bt = I. • In Adam, the operator is diagonal in the coordinate basis; each scalar receives its own running scale. • In Shampoo, At and Bt are inverse roots of accumulated row and column covariance matrices. • In Muon, Eq. (2) is better interpreted through a constrained linear problem: the momentum matrix is mapped approximately to its polar factor, which is the steepest direction under a spectral-norm trust region. • In low-rank methods, At and Bt may be implicit projection operators, with adaptive statistics stored only in the projected coordinates. These operations can produce similar-looking normalized matrices while answering different questions. A preconditioner estimates how the objective stretches space. A normalizer enforces a chosen update magnitude. A projection decides where information is retained. A quantizer decides how accurately the retained information survives. Confusing these roles is a recurring source of overbroad claims. The distinction also clarifies the word adaptive. At least four meanings occur in current papers: 6

Blog: Survey of Optimizers

August 2026

1. coordinate adaptation: a different effective step size for each scalar, as in Adam; 2. matrix adaptation: a basis or whitening transform learned from matrix statistics; 3. temporal adaptation: a changing mixture of recent and old gradients, or resets triggered by instability; 4. resource adaptation: precision, rank, or state granularity selected according to layer and time. Calling all four adaptive without qualification obscures which failure mode is being addressed.

2.2 From Euclidean steepest descent to norm-aware updates For a differentiable objective f and a norm ∥ · ∥, the linearized steepest-descent direction solves d⋆ = arg min ⟨∇f (θ), d⟩.

(3)

∥d∥≤1

Under the Euclidean norm, d⋆ is the normalized negative gradient. Under an ℓ∞ constraint, the solution is a sign vector. For a matrix constrained in spectral norm, the solution is the negative polar factor −U V ⊤ of the gradient. This gives a common language for sign methods, Lion [Chen et al., 2023], Muon, and the norm-designed family developed by Scion [Pethick et al., 2025]. The norm view is useful, but it does not make every norm-based method a second-order optimizer. Equation (3) changes the geometry of the trust region without estimating curvature. A second-order or natural-gradient method instead uses an operator Ct intended to approximate a Hessian, generalized Gauss–Newton matrix, Fisher matrix, or gradient covariance: dt = −(Ct + λI)−1 gt .

(4)

K-FAC approximates layerwise Fisher blocks by Kronecker products [Martens and Grosse, 2015]; Shampoo builds Kronecker statistics from gradients [Gupta et al., 2018]; Sophia uses a lightweight diagonal curvature estimate [Liu et al., 2024]. Their computational compromises differ, but all try to use information beyond the instantaneous direction. −1/4

−1/4

A third category is whitening. If Lt and Rt estimate row and column second moments, Lt Gt Rt approximately balances the covariance of the transformed gradient. Whitening can resemble both preconditioning and spectral normalization, yet it additionally changes variance across time and coordinates. The 2025 component studies show that this variance-adaptation role is empirically essential [Frans et al., 2025a]. It is therefore better to ask which part of a whitening transform matters than to label the entire family second order.

2.3 The optimizer state as a statistical object Persistent state is not merely an implementation detail. It is a compressed summary of the gradient history. For an EMA st = βst−1 + (1 − β)xt , (5) the nominal memory in steps is approximately (1 − β)−1 . But the useful memory depends on the sampling unit. If a step contains B tokens, the state remembers roughly B/(1 − β) tokens. Changing the batch while holding β fixed changes the statistical estimator even if the update code is identical. State has three distinct dimensions: • support: which subspace, block, row, column, or coordinate receives a statistic; • timescale: how past observations are weighted; 7

Blog: Survey of Optimizers

August 2026

Table 2: Three questions that distinguish methods often grouped under geometric optimization. Question

Spectral / norm-based

Whitening / covariance Curvature / natural gradient

What is estimated?

Often no curvature; a direction is normalized under a chosen norm Singular values or an induced update norm Muon, Scion, SoftMuon

Historical row/column or rotated-basis second moments Covariance and variance of transformed gradients Shampoo, SOAP, SPlus, COSMOS Matrix state plus inverse roots/eigenbases

What is controlled? Representative methods Main cost

Matrix transforms and shape policy

Fisher, Hessian, Gauss–Newton, or a structured surrogate Predicted objective or distributional change K-FAC, Sophia, structured Fisher methods Curvature estimation, damping, inversion, communication

• numerical representation: the precision, scale, and quantizer used to store it. Adam-mini changes support, AdEMAMix changes timescale, and SOLO changes representation. LDAdam changes all three because the subspace itself moves. This decomposition helps explain why two methods with similar byte counts may behave very differently.

2.4 Parameter classes are not interchangeable Transformers contain several parameter classes with different shapes and symmetries: token embeddings, attention projections, gated MLP matrices, normalization scales, output heads, and sometimes mixtureof-experts routers. A matrix transform defined for a dense hidden weight does not automatically make sense for a vector or an embedding table. Practical Muon stacks therefore route many non-hidden parameters to AdamW. This routing decision can materially affect results. Embeddings often have sparse or highly non-uniform row access; output heads see high variance; normalization parameters are one-dimensional; router matrices have permutation structure tied to experts. A symmetry-compatible construction formalizes the principle that an update should commute with the transformations under which the represented function is unchanged [Lau and Su, 2026]. The result is not one universal matrix rule but a layerwise stack: two-sided spectral updates for some blocks, one-sided or row-aware transforms for others, and coordinatewise methods where the symmetry is coordinate-specific.

2.5 Five budgets, five different winners An optimizer can be evaluated under at least five budgets: 1. tokens: how many training examples reach a target loss; 2. gradient evaluations: especially relevant when probing, sharpness-aware, or multi-evaluation steps are used; 3. FLOPs: including orthogonalization, eigendecomposition, and state refresh; 4. wall-clock time: including kernels, communication, synchronization, and checkpoint recovery; 5. memory: including the opportunity to increase batch, context length, or model size. Per-step speed and per-token convergence can move in opposite directions. A matrix method may save 8

Blog: Survey of Optimizers

August 2026

20% of tokens but add 5% to step time; a low-memory method may match Adam per token yet win decisively because it permits a fourfold larger batch. Conversely, a theoretically efficient preconditioner can lose in wall-clock time when a matrix operation serializes a distributed step. For this reason, the remainder of the survey treats the optimizer as a budgeted policy. The relevant question is not only whether an update is mathematically attractive, but whether its statistics, numerical representation, and implementation convert that attraction into the resource metric that constrains the run.

3 Why AdamW remains the reference point AdamW combines exponential first and second moments with decoupled weight decay [Kingma and Ba, 2015, Loshchilov and Hutter, 2019]. Ignoring bias correction for compactness, mt = β1 mt−1 + (1 − β1 )gt ,

(6)

vt = β2 vt−1 + (1 − β2 )gt2 ,

(7)

mt . θt+1 = (1 − ηt λ)θt − ηt √ vt + ϵ

(8)

Its continuing strength is not only diagonal adaptivity. AdamW is a mature engineering contract: tensorwise behavior is predictable, sharding is routine, low-precision implementations are available, and years of recipes encode useful priors. A new method must beat this total risk, not only a loss curve. Recent work attacks three weaknesses of the Adam recipe: its temporal statistics may forget useful old gradients, its current gradient can interact awkwardly with its current denominator, and a second moment per scalar is expensive.

3.1 More careful use of gradient history AdEMAMix adds a slow exponential average to the usual fast momentum, motivated by the observation that gradients from tens of thousands of steps ago can still be useful [Pagliardini et al., 2025]. In its largest reported comparison, a 1.3B model trained on 101B tokens matched an AdamW model trained on 197B tokens. The result is striking, but it is best read as evidence that a single momentum timescale is an avoidable restriction, not as a portable twofold guarantee. MARS introduces scaled stochastic recursive momentum, using a variance-reduced gradient estimator that can be paired with AdamW-, Lion-, or Shampoo-like preconditioners [Yuan et al., 2025]. The conceptual contribution is modular: improve the temporal estimator without committing to one geometry. Cautious Optimizers takes a more local route, masking coordinates where the proposed momentum update disagrees in sign with the current gradient [Liang et al., 2026]. Cautious Weight Decay applies the same agreement intuition to regularization, decaying only weights for which decay is locally compatible with the optimizer’s update [Chen et al., 2026]. SPAM is motivated by rare gradient spikes that can be orders of magnitude larger than surrounding values [Huang et al., 2025a]. It combines spike-aware clipping with periodic resets of the momentum state. Resetting sounds crude, but it identifies a real failure mode: a contaminated EMA can remain wrong long after the triggering event. ADOPT isolates a theoretical dependency in Adam by preventing the current gradient from determining the very denominator used to normalize it; operationally, normalization and moment accumulation are reordered [Taniguchi et al., 2024]. This yields optimal convergence guarantees across a broad range of second-moment decay values. Controlled pretraining comparisons nevertheless find that empirical 9

Blog: Survey of Optimizers

August 2026

performance can still depend on β2 [Semenov et al., 2025]. The lesson is useful beyond ADOPT: removing a parameter from a theorem is not the same as removing it from tuning. Singularity-aware Adam targets a different instability: gradient chattering near non-smooth regions rather than ordinary stochastic variance [Xu et al., 2026]. Randomized directional probes test whether nearby loss changes remain coherent; when they do not, the method temporarily damps the Adam update. Unlike sharpness-aware training, which changes the update using an adversarially perturbed neighborhood [Foret et al., 2021], the probes act as a diagnostic brake while retaining Adam’s direction. This places the work beside clipping, agreement filters, and state resets as a robustness mechanism, rather than as a separate optimizer family. Its present evidence is emerging, with the clearest motivation in quantization-aware, hard-routing, and high-noise training.

3.2 A closer look at temporal estimators The first moment in Adam is often described simply as momentum, but this description hides a modeling choice: the optimizer assumes that one exponential timescale is an adequate summary of useful history. AdEMAMix replaces this with a short and a long timescale. In simplified notation, mfast = βf mfast t t−1 + (1 − βf )gt ,

(9)

mslow = βs mslow t t−1 + (1 − βs )gt , e t = mfast m + αt mslow , t t

(10) (11)

with βs close to one and a schedule for introducing the slow component [Pagliardini et al., 2025]. The mixture can retain directions that are individually too weak to dominate a short EMA. This is plausible in pretraining, where a gradient direction associated with a rare feature may recur after many unrelated batches. The benefit is not free. A very slow average can carry obsolete directions after the model enters a new regime, and its timescale interacts with batch size and training duration. A comparison that copies βs across token budgets may inadvertently compare different estimators. Reporting EMA half-lives in tokens would make such studies substantially clearer. MARS attacks variance more directly. Its recursive estimator combines current and previous stochastic gradients before feeding a chosen preconditioner [Yuan et al., 2025]. A generic recursive form is matched ct = gt + γt (gt − gt−1 ),

(12)

where the second gradient is evaluated or approximated so that the correction reduces stochastic drift. The precise MARS instances differ, but the important abstraction is that temporal variance reduction and spatial preconditioning can be designed independently. MARS-AdamW, MARS-Lion, and MARS-Shampoo are therefore a small family rather than one fixed rule.

3.3 Agreement filters, resets, and robustness Cautious optimization asks whether a momentum proposal remains locally supported by the current gradient. For a proposed update ut and gradient gt , an elementwise mask ct = I{ut ⊙ gt > 0}

(13)

removes coordinates whose signs disagree, with normalization used to preserve overall scale [Liang et al., 2026]. The intervention is intentionally minimal. It can be attached to AdamW or Lion without constructing new state. Its value is best understood as a trust filter: momentum is allowed to accelerate a direction only while current evidence has not reversed. 10

Blog: Survey of Optimizers

August 2026

This filter is not guaranteed to help every noisy coordinate. Under symmetric noise, sign disagreement may be evidence of variance rather than a true change of direction. Excessive masking can reduce the effective dimension of the step and alter weight-decay balance. The correct comparison must therefore log the active-mask fraction and the RMS of the post-mask update. Cautious weight decay applies related logic to the regularizer [Chen et al., 2026]. Decoupled decay is normally applied uniformly even when it opposes the data-gradient update. A sign-conditioned version treats regularization as another proposal whose local compatibility can be checked. This does not make weight decay an unbiased estimator of any fixed penalty; it turns decay into a state-dependent training policy. That distinction matters when interpreting generalization. SPAM focuses on rare, extreme events. Its study reports gradient spikes up to roughly three orders of magnitude above typical values in some pretraining traces [Huang et al., 2025a]. Spike-aware clipping limits the immediate damage, while periodic state reset limits how long the event contaminates the first and second moments. These two operations target different terms: • clipping bounds the observation entering the state; • reset truncates the influence of observations already stored. Sparse momentum then reduces memory by retaining state for only a subset of coordinates. The combined method spans robustness and memory, which is why a single headline such as stable Adam understates its design. Stable-SPAM extends this line to aggressive low-bit training [Huang et al., 2025b]. It tracks historical gradient norms and spike thresholds, normalizes the full matrix, and retains periodic resets. The reported result that a 4-bit run can outperform a higher-precision Adam baseline is protocol-specific, but it makes an important mechanistic point: numerical precision and gradient stability are coupled. Low precision can amplify a poorly scaled update; a stabilization rule can be more valuable than simply adding bits.

3.4 ADOPT and the timing of normalization √ Adam’s current squared gradient contributes immediately to vt and is then divided by vt . The numerator and denominator are therefore statistically coupled. ADOPT uses the previous secondmoment estimate to normalize the current gradient and updates the state afterward [Taniguchi et al., 2024]. A simplified ordering is gt , vt−1 + ϵ

(14)

mt = β1 mt−1 + (1 − β1 )ḡt ,

(15)

vt = β2 vt−1 + (1 − β2 )gt2 .

(16)

ḡt = √

This lag removes a dependency that complicates convergence analysis and yields an optimal nonconvex rate under weaker conditions. Yet the benchmark literature still finds practical sensitivity to β2 [Semenov et al., 2025]. The parameter no longer decides whether the proof works, but it still decides which temporal variance estimate is useful. This difference between a validity parameter and a performance parameter recurs throughout optimization. Damping may be required for numerical stability but still need tuning for speed. An inverse-root refresh interval may preserve convergence while changing wall-clock performance. A schedule-free method may remove the need to know the final horizon while retaining a learning-rate-like scale.

11

Blog: Survey of Optimizers

August 2026

4 The matrix-aware renaissance The most visible shift of 2025–2026 is from coordinatewise scaling to matrix-aware update geometry. Neural-network parameters are not naturally a bag of unrelated scalars: dense weights, attention projections, embeddings, experts, and convolutional kernels have row/column structure. Matrix optimizers try to exploit that structure, but they do so in materially different ways.

4.1 Whitening: Shampoo, SOAP, and stable descendants Shampoo accumulates left and right gradient statistics for a matrix Gt ∈ Rm×n and applies matrix inverse roots [Gupta et al., 2018]. In schematic form, Lt = βLt−1 + (1 − β)Gt G⊤ t ,

(17)

Rt = βRt−1 + (1 − β)G⊤ t Gt , e t ≈ L−1/4 Gt R−1/4 . G t t

(18) (19)

This can be read as Kronecker-factored whitening of historical gradients. Its costs are obvious: matrix states, eigendecompositions or inverse roots, and shape-dependent engineering. SOAP rotates the gradient into the evolving eigenbases of Shampoo-like statistics and runs an Adamstyle update there [Vyas et al., 2025]. This decomposition matters conceptually. SOAP is not merely more second order than Adam; it combines a historical matrix coordinate system with coordinatewise adaptivity inside that system. SPlus revisits the same family with stability as the design target [Frans et al., 2025b]. It uses a historical eigenbasis, an instantaneous normalization/sign-like operation, shape-aware scaling, and iterate averaging. Its authors report reaching Adam-level validation performance in an average of 44% of the steps and 62% of wall-clock time across their benchmark. What Really Matters in Matrix-Whitening Optimizers? provides the more important component-level interpretation: variance adaptation is critical, pure spectral normalization is effective but incomplete, and input-side preconditioning can recover much of SOAP’s behavior with about 41% of full SOAP’s matrix-state memory in the studied setting [Frans et al., 2025a].

4.2 Spectral steepest descent: what Muon is and is not Muon applies momentum to each eligible matrix and approximately orthogonalizes the resulting update with a short Newton–Schulz iteration [Jordan et al., 2024, Liu et al., 2025]. If Bt = µBt−1 + Gt ,

Bt = U ΣV ⊤ ,

(20)

the idealized transform is the polar factor Polar(Bt ) = U V ⊤ .

(21)

This replaces singular values by one while preserving singular directions. In the language of normconstrained steepest descent, it is the linear minimization oracle associated with a spectral-norm constraint [Bernstein and Newhouse, 2024, Pethick et al., 2025]. That description also prevents a common category error. Muon is matrix-aware, but it is not a classical second-order method: it does not estimate a Hessian, and its Newton–Schulz iteration computes an approximate polar factor rather than a Newton step. Shampoo and SOAP use historical gradient covariance structure that can act as a preconditioner; Muon normalizes the current momentum matrix 12

Blog: Survey of Optimizers

August 2026

spectrally. Putting all three under second order erases the mechanism that should determine when they work. The scalable Muon study reports matching an AdamW baseline with roughly 52% of the training FLOPs under its fitted scaling-law protocol and demonstrates a 16B-parameter mixture-of-experts run [Liu et al., 2025]. This is strong evidence that polar-style updates can survive serious scale. It is not evidence that every Muon implementation halves every training bill. Performance depends on how matrix and non-matrix tensors are separated, how shape scaling is applied, and how update RMS is aligned with the baseline. Scion gives the norm-constrained view a broader vocabulary: choose a norm appropriate to a parameter block, compute its linear minimization oracle, and design scaling rules that transfer across width [Pethick et al., 2025]. This suggests a future in which embeddings, attention matrices, experts, and normalization parameters need not share one geometry.

4.3 Adaptivity after orthogonalization Pure polar normalization deliberately discards singular-value magnitude. Several methods try to restore just enough adaptivity without returning to full SOAP: • COSMOS applies SOAP-like treatment in a leading eigensubspace and Muon-like treatment to the residual [Liu et al., 2026a]. • AdaMuon tracks elementwise second moments of orthogonalized updates and stabilizes their signs, combining spectral normalization with Adam-like variance information [Si et al., 2026]. • NorMuon normalizes rows after orthogonalization using neuron-level second moments. Its authors report 21.74% greater training efficiency than Adam and 11.31% over Muon in a 1.1B setup, while providing a sharded implementation [Li et al., 2026]. • Mousse first whitens with Shampoo-derived Kronecker statistics and then takes a polar step in the whitened coordinates. On 160M–800M language models, its study reports roughly 12% fewer steps than Muon with about 3% wall-clock overhead [Zhang et al., 2026]. Other recent preprints alter the geometry still further. MuonEq equilibrates row and column scales before orthogonalization [Chang et al., 2026]; Muown imposes explicit row-norm control to reduce spectral-norm drift [Lion et al., 2026]; Hyperball constrains Frobenius norms of both weights and updates [Wen et al., 2026a]; and a softsign relaxation interpolates between sign-like and magnitude-sensitive behavior to address terminal oscillation [Feoktistov et al., 2026]. These are emerging results, not a settled sequence of upgrades. Their collective message is more durable than any current ranking: the singular directions, row/column scales, and overall update norm are independent design choices.

4.4 What controlled comparisons actually say Two broad comparisons make the current uncertainty unusually legible. One study evaluated ten optimizers from 0.1B to 1.2B parameters and from 1× to 8× a standard data-to-parameter training ratio [Wen et al., 2026b]. Matrix-aware methods improved token efficiency by as much as roughly 1.4× at small scale, but the gap narrowed to around 1.1× at 1.2B. Muon was strongest at shorter ratios, while SOAP- and Kronecker-style methods could overtake it at longer ratios. Rankings at intermediate checkpoints did not reliably predict final rankings. A separate benchmark of eleven methods, up to 720M parameters and 50B tokens, found AdEMAMix and MARS particularly strong in its larger-model, larger-batch setup, while Muon, SOAP, and 13

Blog: Survey of Optimizers

August 2026

Table 3: Mechanism-level comparison of representative matrix-aware methods. A historical basis depends on accumulated statistics rather than only the current momentum. Method

Basis

Spectrum treatment

Variance adaptation

Persistent matrix state

Primary design goal

Muon

Current momentum singular vectors

None in basic form

Momentum matrix

Cheap spectral steepest descent

Shampoo

Historical row/column statistics Shampoo eigenbasis

Approximately sets singular values to one Inverse-root whitening

Implicit in accumulated covariance Elementwise in rotated basis Sign-like / averaging based

Left and right factors

Structured preconditioning

Factors plus Adam-style moments Historical basis and auxiliary state

Strong per-step adaptation Stable, cheaper whitening

Concentrated in important subspace

Low-rank basis and moments

Memory-quality trade-off

Elementwise second moment

Momentum and adaptive state

Neuron-wise second moment Kronecker geometry

Momentum plus row statistics Shampoo-like factors

Restore coordinate variance Balance neuron utilization Anisotropic spectral step

SOAP SPlus

Historical eigenbasis

COSMOS

Leading historical subspace plus residual Orthogonalizedupdate coordinates

AdaMuon

NorMuon

Polar update rows

Mousse

Shampoo-whitened coordinates

Adam update in rotated coordinates Instantaneous bounded normalization SOAP in leading subspace, Muon outside Polar transform plus sign stabilization Polar transform then row scaling Polar transform after whitening

distributed Muon variants could lead shorter runs [Semenov et al., 2025]. It also found schedule interactions: cosine was broadly strong, but Muon sometimes preferred a warmup-stable-decay policy. These results do not cancel one another. They expose a basic fact: An optimizer winner is a statement about a protocol, not an intrinsic property of a name. At even larger scale, a study of multi-billion-parameter models and trillion-token workloads stabilizes SOAP for large batches, matches update RMS across methods, and implements exact layerwise distributed optimizer operations [Khona et al., 2026]. SOAP and Muon outperform AdamW in the tested regimes, with a SOAP variant slightly ahead. The systems work is part of the scientific claim: an algorithm whose communication cannot be hidden or whose matrix operations are approximated differently is a different practical optimizer.

5 Dissecting matrix optimizers The names in the matrix-optimizer family are multiplying faster than the underlying mechanisms. A more stable taxonomy separates five operations: momentum formation, basis choice, singular-value transformation, variance adaptation, and output scaling.

5.1 Muon’s Newton–Schulz map Computing U V ⊤ with an exact singular value decomposition every step would be expensive. Muon uses a small fixed number of Newton–Schulz iterations to approximate the polar factor. After normalizing a matrix X0 , a generic iteration has the form Xk+1 = aXk + bXk Xk⊤ Xk + cXk Xk⊤ Xk Xk⊤ Xk , 14

(22)

Blog: Survey of Optimizers

August 2026

with coefficients chosen so singular values move rapidly toward one over a useful interval. Implementations differ in coefficients, normalization, number of iterations, and arithmetic precision. Those details affect both the approximation and kernel performance. Several practical points follow. • Rectangularity matters. For a highly rectangular matrix, the polar factor has orthonormal columns or rows, not both. Implementations often transpose the matrix so the smaller dimension is treated favorably. • Finite iterations preserve spectral information. A five-step approximation is not an exact sign of the singular values. The input condition number and stable rank influence the output. • Scale must be reintroduced. A polar factor has a shape-dependent Frobenius norm. Without a shape rule, the same global learning rate produces different RMS updates for matrices of different aspect ratios. • Non-matrix parameters need a fallback. Most stacks retain AdamW for embeddings, vectors, or selected heads, making the reported method a hybrid from the start. MuonEq exploits the second point. It normalizes rows or columns before finite-step orthogonalization, improving the spectrum seen by Newton–Schulz without building a full whitening preconditioner [Chang et al., 2026]. The interpretation is useful: equilibration is a zeroth-order surrogate for whitening, and finite-step polar computation is itself a tunable part of the optimizer.

5.2 Shape scaling, weight norms, and angular learning rate For a scale-invariant parameter block, the functional change induced by an update depends on the angle between the old and new weight matrices. If W and ∆W are approximately orthogonal in Frobenius inner product, the angular step is roughly ∆ϕ ≈

∥∆W ∥F . ∥W ∥F

(23)

Weight decay influences ∥W ∥F and therefore implicitly controls angular learning rate. This explains why optimizer comparisons can change dramatically with weight-decay conventions even when the data-gradient update is unchanged. Hyperball makes the norm control explicit. It fixes or targets the Frobenius norms of weight matrices and optimizer updates, decoupling angular progress from the equilibrium created by constant decoupled decay [Wen et al., 2026a]. Its reported 20–30% token-equivalent advantage over weight-decay baselines on models up to 1.2B is evidence that norm dynamics are not a peripheral regularization detail. Muown decomposes each matrix into row magnitudes and directions [Lion et al., 2026]. The rowmagnitude vector receives an ℓ∞ -aligned update while Muon acts on the directional component. This addresses a diagnosed drift in spectral norm driven by row magnitudes. AngularMuown subsequently shows that the unnormalized directional parameterization implicitly decays angular step size and makes that multiplier explicit [Hübler et al., 2026]. These methods recast weight decay, normalization, and scheduling as different controls over radial and angular motion.

5.3 Whitening is not one operation Suppose G has singular value decomposition U ΣV ⊤ . Exact two-sided whitening maps G toward U V ⊤ , which looks like spectral normalization. But historical whitening uses covariance estimated over time: E[GG⊤ ]

and 15

E[G⊤ G].

(24)

Blog: Survey of Optimizers

August 2026

The eigenvectors and eigenvalues of these expectations need not match those of the current gradient. The resulting transform does two things: 1. it reduces anisotropy across matrix directions; 2. it adapts to persistent variance across time. The component study of matrix-whitening methods isolates this distinction by comparing sign-like and variance-adapted pairs: Signum versus Adam, SPlus versus SOAP, and Muon versus AdaMuon [Frans et al., 2025a]. In its controlled setup, spectral accuracy alone does not explain performance; Muon can produce a tighter singular-value spread while SOAP achieves stronger per-step progress. Variance adaptation explains much of the gap. This finding changes how low-memory approximations should be designed. If the benefit came only from exact spectral normalization, approximating the polar factor would be the central task. If historical variance matters, then a cheap estimator of variance in a useful basis can be more valuable than a more accurate instantaneous polar transform. Input-side preconditioning and low-rank variance estimators become principled compromises rather than incomplete SOAP.

5.4 Why hybrids proliferate The leading-subspace strategy in COSMOS assumes that the most important covariance structure is concentrated in a small eigenspace [Liu et al., 2026a]. SOAP is used where that structure justifies adaptive state; Muon handles the residual without paying full matrix-memory cost. Mousse instead applies a full conceptual composition: first transform into a Kronecker-whitened coordinate system, then solve a spectral steepest-descent problem there [Zhang et al., 2026]. The trust region is anisotropic in the original coordinates. AdaMuon and NorMuon add variance after the polar transform at different granularities [Si et al., 2026, Li et al., 2026]. Elementwise adaptation has the greatest expressive power and highest state cost. Rowwise adaptation assumes that neurons, rather than scalar coordinates, are the meaningful units of imbalance. This granularity is a testable modeling assumption. SoftMuon changes a different component. Hard polar or sign-like updates discard magnitude and can oscillate near convergence. A temperature-controlled soft-sign map interpolates toward magnitudesensitive steps, with an adaptive quantile rule controlling temperature [Feoktistov et al., 2026]. The proposal treats terminal convergence as a geometry-relaxation problem rather than a reason to abandon norm-aware optimization.

5.5 Communication can be an optimizer operation Dion reduces distributed communication by maintaining local momentum and synchronizing compact orthonormalized update information rather than full gradients [Ahn et al., 2025]. The goal is not simply to reduce optimizer-state bytes; it changes what is exchanged between devices while retaining synchronous semantics. Such a method should be evaluated on communication volume, overlap, and the discrepancy between device-local states, not only validation loss. Large-scale SOAP and Muon implementations take another route: distribute optimizer work layer by layer, balance matrix operations across ranks, and hide communication behind model computation [Khona et al., 2026]. Exactness matters because approximating an eigenspace, delaying a factor update, or reducing orthogonalization steps can change the optimization rule. Systems comparisons should therefore state:

16

Blog: Survey of Optimizers

August 2026

• which tensors are sharded and when they are gathered; • whether inverse roots, QR factorizations, or Newton–Schulz steps are exact relative to the singledevice algorithm; • the refresh interval for matrix statistics; • the precision of factorization and state; • whether optimizer communication is exposed or overlapped.

5.6 A decision tree inside the matrix family The matrix family can be navigated through four questions. 1. Is persistent covariance information affordable? If no, start with Muon or a stateless transform. If yes, a Shampoo/SOAP-style method becomes plausible. 2. Is the horizon short or long? Pure spectral normalization is often strong early; long horizons more often reward variance adaptation. 3. Is memory or step time the binding resource? SOAP can be step-efficient but state-heavy; Muon is state-light but still needs matrix kernels; COSMOS and SPlus occupy intermediate points. 4. Can parameter classes be routed explicitly? If the implementation forces one transform onto embeddings, heads, and hidden matrices, theoretical elegance can become practical mismatch. The result is not a total order. Muon is the cleanest low-cost test of whether matrix geometry helps. SOAP is a strong test of whether historical covariance and rotated-basis adaptivity justify cost. SPlus tests whether bounded whitening and averaging capture much of that gain more stably. NorMuon and AdaMuon test the missing-variance hypothesis. Mousse and COSMOS test where expensive geometry should be concentrated.

6 The emerging layerwise optimizer stack Matrix optimization is moving from one rule per model toward one rule per parameter class. This direction is visible even in methods that do not advertise it: Muon commonly uses AdamW for non-hidden parameters; SCALE reserves momentum for the output layer; symmetry-compatible designs derive distinct transforms for embeddings and routers.

6.1 Equivariance as a routing principle Let a parameter transformation T leave the represented network function unchanged after a corresponding transformation elsewhere in the model. An optimizer update U is symmetry-compatible if U (T W, T −T G) = T U (W, G) (25) for the relevant group action. Bi-orthogonal equivariance motivates polar updates for generic matrices. Embedding rows instead admit permutation symmetries tied to token identities; mixture routers admit expert permutations; gated MLP factors share coupled rescalings. A single coordinatewise Adam rule is not invariant to all of these actions, while a single two-sided spectral rule may impose the wrong invariance. The symmetry-compatible optimizer derives one-sided spectral, row-norm, centered, and hybrid updates for these parameter classes [Lau and Su, 2026]. Its experiments on dense and sparse models support 17

Blog: Survey of Optimizers

August 2026

Table 4: Illustrative parameter-class routing for a compositional optimizer. The table states hypotheses to test, not a universal prescription. Parameter class

Candidate geometry

Reason

Critical diagnostic

Attention/MLP hidden matrices

Muon, SOAP, SPlus, NorMuon

Embedding tables

Row-aware Adam or one-sided rule

Update RMS, singular-value spread, row-norm balance Frequency-conditioned update norm

Output head

Adam-like or dedicated momentum AdamW or simple coordinate rule

Dense two-axis structure; matrix kernels amortize well Sparse/non-uniform row access; token permutation symmetry High gradient variance and direct loss coupling One-dimensional parameters; matrix geometry is artificial Expert permutation and shared-shift structure

Normalization scales/biases MoE routers

Centered row/column-aware rule

Logit-scale drift and row variance Relative update-to-weight ratio Expert-load stability and router entropy

the routing principle, but the broader research opportunity is larger: the architecture can expose parameter metadata, and the optimizer can dispatch a geometry accordingly.

6.2 The cost of compositionality A layerwise stack introduces new hyperparameters and failure surfaces: • relative learning-rate scales between parameter classes; • inconsistent weight-decay semantics; • different state precisions and sharding layouts; • update imbalance at residual branches; • harder attribution when a combined method wins. The natural engineering response is a common calibration layer. Update-RMS matching places each rule on a comparable scale. Token-based timescales align momentum across differently sharded batches. A routing table makes non-matrix fallbacks explicit. These should be treated as part of the algorithm specification.

7 Memory is part of the algorithm For a parameter count N , full-precision Adam commonly stores two moment values per parameter in addition to weights, gradients, and often master weights. At scale, the question is not simply whether to precondition, but where information deserves persistent state.

7.1 Compress the statistic, not automatically the update Adam-mini preserves the first moment but replaces many elementwise second moments with blockwise statistics chosen to follow approximate Hessian structure [Zhang et al., 2025]. It reports roughly 45–50% lower optimizer-state memory than AdamW. SlimAdam makes compression conditional: a layerwise 18

Blog: Survey of Optimizers

August 2026

signal-to-noise criterion determines when second moments can be aggregated across dimensions, with up to 98% second-moment savings reported in its experiments [Kalra et al., 2025]. APOLLO uses random low-rank auxiliary state to estimate structured learning-rate scaling; its rankone Mini variant reduces that state further [Zhu et al., 2025]. LDAdam also maintains moments in a low-dimensional subspace, but explicitly transports state when the projection changes and uses generalized error feedback [Robert et al., 2025]. That machinery addresses a subtle failure mode: a moving basis can make a perfectly reasonable EMA semantically stale. GaLore2 improves the practicality of low-rank gradient projection, reducing projection-update and sharding bottlenecks and demonstrating 7B pretraining over long token horizons [Su et al., 2025]. Alice and RACS begin from a structured Fisher approximation: Alice combines low-rank eigenspaces with Adam-like updates, while RACS uses row/column-scaled SGD with SGD-like memory [Gong et al., 2026]. Their study reports more than a twofold target-loss speedup for Alice over Adam in tested setups up to 1B parameters. A crucial distinction runs through these methods: Low-rank state is not the same as a low-rank update. One can compress the information used to choose scales while retaining a full-rank residual update. Conversely, a low-rank update can remain constrained even if its state estimate is accurate. FRUGAL makes the distinction explicit by assigning an advanced optimizer to a low-dimensional component while sending the full-rank residual through a state-free rule [Zmushko et al., 2025]. LoRAPre reinterprets an EMA as online linear regression and factorizes the momentum itself [Wang et al., 2026]. Whether either is attractive depends on how quickly the relevant subspace changes.

7.2 State-free and near-state-free alternatives SWAN uses instantaneous normalization and whitening without persistent moments [Ma et al., 2025]. The authors report SGD-level optimizer memory, roughly 50% lower end-to-end memory, and about twofold token efficiency over Adam in 350M and 1.3B experiments. SCALE uses column-normalized SGD and keeps momentum only for the output layer; its 60M–1B results report Adam-like performance at 35–45% of total training memory [Glentis et al., 2026]. These methods are appealing when optimizer state determines the largest trainable model. Their caveat is equally important: instantaneous normalization has no temporal variance estimate, so batch size, gradient noise, and tensor shape become more exposed. A fixed-memory benchmark is more informative here than a table of bytes per parameter. Extra memory can be reinvested in batch size, sequence length, activation checkpointing, or model width, and each changes the optimization problem.

8 Memory-efficient optimization as statistical compression Memory-efficient optimizers are sometimes compared by a single number: state bytes per parameter. That number is necessary but incomplete. The methods compress different statistical objects and therefore introduce different biases. A useful classification has four categories: 1. factorization: replace an elementwise tensor with row, column, block, or Kronecker factors; 2. projection: store adaptive state in a lower-dimensional subspace; 3. sparsification: retain state for selected coordinates or parameter classes; 4. elimination: use instantaneous normalization or plain SGD so no persistent state is required. 19

Blog: Survey of Optimizers

August 2026

Table 5: What is compressed by representative memory-efficient methods. Method

Compressed object

Information retained

Main risk

Adam-mini

Second moment within Hessian-informed blocks Selected dimensions of second moments Adaptive scaling in random low-rank coordinates Gradient and optimizer state in refreshed low-rank subspace Projected first/second moments Expensive optimizer on low-dimensional component All persistent moments

Full first moment; one scale per block Fine state where measured SNR is low Structured learning-rate information

Partition may not match changing curvature SNR estimate and decision overhead Projection may miss important directions

Full parameters; low-rank adaptive dynamics

SVD/refresh and subspace drift

State transported between subspaces Full-rank residual via cheap rule

Error-feedback stability and refresh cost Balancing component and residual

SlimAdam APOLLO GaLore2 LDAdam FRUGAL SWAN SCALE

Instantaneous normalization and whitening Most momentum and all Column normalization; second moments momentum at output

Batch noise and shape sensitivity Architecture-specific concentration of variance

Adafactor is an early example of factorizing a matrix second moment into row and column statistics [Shazeer and Stern, 2018]. Adam-mini uses a block partition informed by approximate Hessian structure rather than a rank-one factorization [Zhang et al., 2025]. SlimAdam observes the state trajectory and compresses only dimensions whose signal-to-noise ratio makes aggregation safe [Kalra et al., 2025]. These three methods can have similar state counts on a particular model while encoding different assumptions.

8.1 A concrete memory ledger Consider mixed-precision training with N parameters. A simplified per-parameter ledger may include: • model parameters in BF16 or FP16: 2 bytes; • gradients in BF16 or FP16: 2 bytes; • FP32 master weights: 4 bytes; • Adam first and second moments in FP32: 8 bytes. Before activations and fragmentation, this is roughly 16 bytes per parameter. Sharding changes the per-device share, but not the aggregate volume. Removing one Adam moment, quantizing both, or eliminating master weights therefore changes different portions of the ledger. The end-to-end benefit depends on the rest of the workload. If activations dominate, reducing optimizer state may not change the feasible batch. If parameters and state dominate, as in sparse mixture-ofexperts models, eliminating master weights can be decisive. Papers should report both optimizer-only and total peak memory, with and without sharding.

20

Blog: Survey of Optimizers

August 2026

8.2 Projection changes coordinates over time Let Pt ∈ Rd×r be an orthonormal basis with r ≪ d. A projected optimizer forms zt = Pt⊤ gt

(26)

and stores moments in Rr . The full-space update may be reconstructed as Pt ut . If Pt changes, the old state is expressed in the wrong coordinates. Simply keeping the vector mt−1 while replacing Pt−1 by Pt changes its meaning. LDAdam addresses this by transporting optimizer states between consecutive subspaces and applying generalized error feedback [Robert et al., 2025]. GaLore periodically recomputes a low-rank gradient basis, while GaLore2 reduces the SVD and distributed-integration bottlenecks and demonstrates much longer 7B pretraining [Zhao et al., 2024, Su et al., 2025]. The engineering advance is scientifically relevant: a projection that is refreshed too rarely becomes stale; one refreshed too often can spend its memory savings on decompositions and synchronization. APOLLO avoids expensive data-dependent subspace estimation by using random projections to approximate structured learning-rate scaling [Zhu et al., 2025]. The rank-one APOLLO-Mini result suggests that learning-rate information may be far more compressible than the gradient itself. This supports a general design rule: Use a compressed statistic to choose a full-space scale whenever possible; do not force the effective update to inherit the statistic’s rank. FRUGAL implements this rule by applying an advanced optimizer to a low-dimensional component and a stateless optimizer to the residual [Zmushko et al., 2025]. RACS reaches a similar goal through row/column scaling, while Alice learns a low-rank eigenspace for richer adaptation [Gong et al., 2026]. The methods differ in how they divide expensive and cheap information, but all reject the idea that every direction deserves an independent second moment.

8.3 When state should be allocated unevenly Optimizer state is most valuable where gradient statistics are heterogeneous and persistent. SCALE’s finding that output-layer momentum carries disproportionate value is an extreme instance [Glentis et al., 2026]. SlimAdam’s SNR rule gives a data-dependent version. Sparse SPAM retains momentum for selected coordinates. A layerwise stack can allocate: • full first and second moments to high-variance heads or embeddings; • rowwise or blockwise state to hidden matrices; • no state to stable normalization or residual parameters; • low-rank matrix statistics only where measured anisotropy justifies them. The unresolved question is whether this allocation can be predicted before training. Architecture, initialization, data mixture, and token frequency all affect compressibility. Online allocation is more robust but consumes monitoring and can create discontinuities when state is added or removed.

8.4 Memory savings alter the statistical experiment Suppose optimizer A needs half the state of optimizer B. There are at least three fair comparisons: 1. hold model and batch fixed, isolating the update rule; 21

Blog: Survey of Optimizers

August 2026

2. hold peak memory fixed and give A a larger batch or context; 3. hold hardware and wall-clock fixed, letting each implementation choose its throughput-maximizing workload. The first is a scientific ablation; the second is a resource comparison; the third is a deployment comparison. They answer different questions. Reporting only the first can understate the value of compression, while reporting only the second can attribute a batch-size effect to the optimizer. APOLLO reports a threefold throughput gain in a setting where its memory savings permit a fourfold larger batch [Zhu et al., 2025]. Adam-mini reports substantial throughput improvement in a multi-GPU setup [Zhang et al., 2025]. These are valuable end-to-end results, but the causal chain should remain explicit: smaller state → larger feasible workload → higher device utilization → wall-clock gain.

(27)

The optimizer may not be faster at an equal batch; it enables a faster training configuration.

8.5 A fixed-memory evaluation frontier The most informative output is a Pareto frontier with at least three axes: validation loss, wall-clock time, and peak memory. A fourth axis can report robustness, such as the fraction of a learning-rate sweep that remains stable. For each point, the paper should state: • rank, block size, or state-allocation policy; • refresh interval and factorization cost; • whether projection constrains the final update; • total and optimizer-only memory; • batch and sequence length enabled by the configuration. Without this information, a low-memory optimizer can look either better or worse merely because the saved memory was or was not reinvested. The correct scientific object is not bytes per parameter in isolation, but the quality-cost frontier of the complete training configuration.

9 Time: schedules, averaging, and unknown horizons The learning-rate schedule is often treated as external to the optimizer. That is increasingly untenable. Momentum defines a timescale, weight decay accumulates over time, averaging selects among iterates, and the final decay phase can change both validation loss and downstream adaptability.

9.1 Schedule-free does not mean time-free The Road Less Scheduled constructs schedule-free variants by maintaining fast and averaged iterates and combining momentum with online-to-batch conversion [Defazio et al., 2024]. Hyperparameters need not encode the final training horizon. This does not remove temporal policy; it internalizes it in interpolation and averaging. Warmup-stable-decay (WSD) and reusable cooldowns attack the same horizon problem from a checkpointing perspective [Wen et al., 2025, Hägele et al., 2024]. A stable high-rate trunk can continue without a known endpoint, and a short decay branch exposes a strong checkpoint when needed. 22

Blog: Survey of Optimizers

August 2026

Weight-space model merging shows that merging checkpoints along a trajectory can emulate some effects of learning-rate decay, with merge duration acting as an important control variable [Tian et al., 2026]. The matrix-aware versions are now arriving. SF-NorMuon combines schedule-free training with neuronnormalized spectral updates and reports competitive results for 125M and 772M models across 1×–8× horizons [Apte et al., 2026]. ScheduleFree+ reports long-horizon language-model results and a 31% advantage over a WSD baseline at a very high tokens-per-parameter ratio [Defazio, 2026]. Both are promising but still emerging. Notably, SF-NorMuon finds that applying weight decay at the fast iterate is essential, a reminder that averaging, decay, and geometry cannot always be composed naively. Weight-space optimization without decay complicates the endpoint story further: in its 1B and 8B studies, a checkpoint with worse pretraining loss can adapt better during downstream supervised fine-tuning [Yano et al., 2026]. Decay can improve the pretraining endpoint while reducing later plasticity. Therefore best final loss is not always the correct training objective.

9.2 Batch size changes the optimizer’s clock An EMA decay is normally specified per update, but examples arrive per token. If a reference run uses batch B and decay β, then a run with batch B ′ preserves approximately the same half-life in processed examples by using ′ β ′ = β B /B . (28) Small-batch experiments show that applying this conversion can make momentum SGD surprisingly competitive and that gradient accumulation may waste optimization opportunities even when it improves device utilization [Marek et al., 2025]. Before concluding that an optimizer fails at a new batch size, one should first ask whether its memory was measured in steps or in tokens.

10 Horizon management in detail A learning-rate curve is a policy over training time. It decides how long the process explores, when it reveals the progress accumulated during a high-rate phase, and whether a useful checkpoint exists before the planned endpoint. Three families now address this policy: explicit decay, stable trunks with cooldowns, and schedule-free averaging.

10.1 Cosine decay and the hidden endpoint Cosine decay specifies a horizon T : 1 πt ηt = ηmin + (ηmax − ηmin ) 1 + cos . 2 T 



(29)

The endpoint enters every step. Extending a run changes the schedule that would have been preferred from the beginning. A checkpoint at 0.5T was trained under a rate selected for a future decay and is not equivalent to the midpoint of a run planned for 2T . This path dependence complicates scaling studies and data additions. It also helps explain optimizerranking flips. A method that tolerates a high learning rate can lead during the stable portion, while a variance-adaptive method can benefit more from late decay. Comparing before the intended endpoint can favor the wrong mechanism [Wen et al., 2026b].

23

Blog: Survey of Optimizers

August 2026

10.2 Stable trunks and cooldown branches Warmup-stable-decay retains a high or constant rate for a reusable main trajectory, then starts a decay branch when a checkpoint is needed [Wen et al., 2025]. The river-valley interpretation separates motion along a flat valley from oscillation across steep walls: the stable rate makes longitudinal progress while the cooldown suppresses transverse oscillation. Reusable cooldown studies show that constant-rate training followed by a short decay can obey predictable scaling behavior and make one long trajectory support multiple endpoints [Hägele et al., 2024]. The practical benefit is option value. A project can continue the trunk when more data or compute arrives without retroactively changing all previous rates. The trunk is not fully horizon-free if its optimal peak rate depends on the intended token budget. Power Scheduler models this dependence by relating learning rate to processed tokens and batch size [Shen et al., 2024]. In its empirical scaling rule, ηopt ≈ B aT b ,

(30)

with b near a negative half in the studied regime. The resulting online schedule depends on tokens already processed rather than a fixed final horizon. The important conceptual step is to express time in tokens and to model transfer across batch and model scale.

10.3 Schedule-free interpolation Schedule-free optimization maintains multiple iterates. A fast point follows gradient information; an averaged point accumulates stability; an interpolated point is used to evaluate gradients [Defazio et al., 2024]. Although implementations differ, the generic pattern is yt = (1 − γt )zt + γt xt ,

(31)

zt+1 = zt − η g(yt ),

(32)

xt+1 = (1 − ct )xt + ct zt+1 .

(33)

The coefficients induce effective decay and averaging without encoding the final step. The method is schedule-free in the sense of horizon independence, not in the sense of constant dynamics. This semantic distinction matters when composing weight decay. Applying decay to the averaged iterate, fast iterate, or gradient-evaluation point produces different trajectories. SF-NorMuon identifies decay at the fast iterate as essential in its long-horizon spectral construction [Apte et al., 2026]. A paper that reports only a decay coefficient without its location has not fully specified the algorithm. ScheduleFree+ adds scale and batch corrections aimed at long language-model runs [Defazio, 2026]. Its largest claimed benefit appears at an extreme tokens-per-parameter ratio, which is exactly where a horizon-independent averaging policy should be most differentiated from a short cosine run. Independent large-scale replication remains important.

10.4 Checkpoint merging as temporal optimization Weight-space model merging offers another view of decay. Averaging or merging checkpoints from a trajectory can suppress high-frequency parameter noise and approximate the endpoint produced by a cooldown [Tian et al., 2026]. The merge window becomes a temporal hyperparameter: a short window behaves like local averaging; a long window can mix functionally distant models and blur useful progress. This connection unifies three practices often discussed separately: 24

Blog: Survey of Optimizers

August 2026

• stochastic weight averaging; • schedule-free online averaging; • post-hoc checkpoint merging. All choose a temporal kernel over iterates. The differences are whether the kernel affects subsequent gradients, whether it can be updated online, and how much state or checkpoint storage is required.

10.5 Dual averaging and regularization schedules SODA interprets several modern optimizers through optimistic dual averaging and proposes a wrapper with a theoretically motivated 1/k decay rule [Pethick et al., 2026]. In this view, Muon, Lion, AdEMAMix, and NAdam differ in the geometry and optimism used to map accumulated gradients into a primal update. The wrapper aims to reduce weight-decay tuning rather than replace the base optimizer. This is useful because weight decay has at least three roles: 1. regularization toward small weights; 2. control of equilibrium weight norms; 3. indirect control of angular learning rate. A 1/k schedule, constant decoupled decay, norm projection, and cautious decay implement different policies even when they produce similar final norms. Hyperball and Muown make the angular interpretation explicit, while no-decay weight-space studies ask whether preserving plasticity can matter more than minimizing pretraining loss [Wen et al., 2026a, Lion et al., 2026, Yano et al., 2026].

10.6 Batch, accumulation, and the unit of time Equation (28) preserves an EMA half-life across batch sizes, but learning rate and noise scale also change. Gradient accumulation is especially subtle. Accumulating K microbatches and taking one optimizer step is not equivalent to taking K smaller steps: • the parameters remain fixed across the K microbatches; • clipping and normalization act on the accumulated gradient; • momentum and second moments are updated once rather than K times; • matrix factors or probes are computed from a different statistic. For wall-clock efficiency, accumulation may be unavoidable. For optimizer comparison, the effective update frequency should be reported in both steps and tokens. If a method benefits from frequent state updates, a large accumulated batch can erase its advantage.

11 Low precision: state semantics matter more than bit count Quantizing weights and activations while keeping optimizer states in high precision leaves a large memory opportunity untouched. Yet moments are not ordinary activations: they are recursively updated state, and small rounding errors can alter their effective timescale. COAT stores optimizer state and selected training tensors in FP8 using dynamic range expansion and mixed-granularity scaling [Xi et al., 2025]. It reports up to 1.54× memory reduction and 1.43× 25

Blog: Survey of Optimizers

August 2026

Table 6: Horizon strategies and the question each one answers. Strategy

Endpoint assumption

Strength

Primary caveat

Cosine decay

Fixed T known in advance Stable trunk, endpoint chosen by branch Depends on processed tokens, not final T Unknown horizon

Strong mature baseline and smooth endpoint Reusable trajectory and multiple budgets Transfers time policy across token and batch scales Anytime averaged iterate with constant main scale

Earlier iterates depend on planned endpoint Peak rate and cooldown still need calibration Empirical power law may be regime-specific Averaging and decay placement are algorithmic choices Merge window, storage, and function-space mismatch

WSD/cooldown Power schedule Schedule-free

Checkpoint merging Post-hoc or periodic

Converts stored trajectory into implicit decay

speedup over a BF16 baseline. SOLO targets 2–3 bit state using logarithmic unsigned quantization and precision-specific momentum, reporting roughly 45GB of savings for a 7B model [Xu et al., 2025]. Muon has an interesting advantage in this regime. A study of 8-bit Muon finds blockwise dynamic or linear quantization sufficient to match full-precision Muon at 1.6B scale, while reducing optimizer state by 74% relative to full-precision Muon and 86% relative to FP32 AdamW [Gupta et al., 2025]. The explanation is structural: Adam’s second moment appears in a denominator, where small or poorly quantized values can cause large effective steps. Momentum used only before a polar transform is more tolerant of scale error. The less obvious problem is quantization staleness. When an EMA update is smaller than a quantization cell, rounding can return the stored value unchanged; the moment silently stops responding. Periodic state resets can restore responsiveness [Topollai and Choromanska, 2026]. ECO removes full-precision master weights and feeds weight-quantization error back into momentum, extending low-precision reasoning beyond moments [Nikdan et al., 2026]. Non-smooth transitions create another failure mode. Quantizers, hard routers, and structured sparsity can make nearby parameter perturbations produce sharply different loss responses. Singularity-aware Adam uses randomized probes to detect this local disagreement and reduces the update only while the instability persists [Xu et al., 2026]. The idea is complementary to state resets: resets remove contaminated history, whereas probing tries to recognize when the current neighborhood itself is unreliable. The additional objective evaluations make it most plausible when instability is costly enough to justify an explicit diagnostic. The design principle is broader than any format: Quantize according to the state’s topology and recurrence, not only its histogram. An unsigned logarithmic representation suits a nonnegative second moment; a blockwise signed representation suits momentum; an error-feedback channel suits a repeatedly quantized parameter. The phrase eight-bit optimizer is therefore not a complete algorithm description.

26

Blog: Survey of Optimizers

August 2026

12 Quantized optimizer dynamics Quantization maps a real state s to a finite codebook: sb = Q(s; α, C),

(34)

where α is a scale and C is a set of representable codes. For an activation used once, quantization error disappears after the layer computation. For an optimizer state, the quantized value is fed back into the next update: sbt = Q(β sbt−1 + (1 − β)xt ) . (35) The error is recursive. It can change the effective decay, create absorbing codebook cells, and interact with resets.

12.1 Blockwise dynamic quantization as the baseline Early 8-bit optimizers combine blockwise scaling, nonlinear dynamic codebooks, and special handling of unstable embeddings [Dettmers et al., 2022]. Blockwise scaling isolates outliers: one extreme coordinate changes the resolution of its block rather than an entire tensor. Dynamic codebooks devote more levels to the small values common in moment tensors. This remains a strong engineering baseline. A new low-bit optimizer should compare against a tuned blockwise implementation, not naive per-tensor linear quantization. It should also distinguish storage precision from compute precision: many kernels dequantize a state in registers, perform the arithmetic at higher precision, and requantize only for memory.

12.2 Why first and second moments need different codebooks Momentum is signed and often roughly symmetric, while Adam’s second moment is nonnegative and heavy-tailed. Their downstream use differs: ut = √

mt . vt + ϵ

(36)

An additive error in mt perturbs the numerator. An error that underestimates a small vt can greatly enlarge the effective step. This asymmetry explains why a single linear quantizer is rarely optimal for both states. SOLO identifies two low-bit pathologies: signal swamping in unsigned states and direction corruption in signed states [Xu et al., 2025]. It uses logarithmic unsigned quantization for the second moment and adjusts momentum according to precision. This is a good example of algorithm–representation co-design: the optimizer hyperparameter is changed because the storage recurrence changes. COAT instead targets a broader FP8 training stack, using dynamic range expansion for state and mixed-granularity quantization for activations [Xi et al., 2025]. Its speed benefit depends on kernels for more than the optimizer. The comparison is therefore end-to-end rather than a pure update-rule ablation.

12.3 State staleness Let ∆t = (1 − β)(xt − sbt−1 ) be the unquantized increment in Eq. (35). If |∆t | is smaller than half the local quantization interval, then Q(sbt−1 + ∆t ) = sbt−1 . (37) 27

Blog: Survey of Optimizers

August 2026

The nominal update occurs in floating point but the stored state does not move. Repeated stalling makes the effective half-life much longer than (1 − β)−1 . The state-staleness analysis models this probability and explains why resets can restore responsiveness [Topollai and Choromanska, 2026]. A reset moves the state away from an absorbing local cell and temporarily increases increments. But resetting too often destroys useful history. A theory-guided period should depend on bit width, scale policy, β, and the distribution of observations. This mechanism connects low precision to SPAM. In high precision, resets remove contamination by rare spikes. In low precision, they can also remove accumulated staleness. Stable-SPAM combines state reset with norm tracking and clipping for 4-bit training [Huang et al., 2025b]. The same operation has two different justifications, which should be diagnosed separately.

12.4 Why Muon is unusually quantization tolerant Muon stores momentum but subsequently maps it through an approximate polar transform. If quantization mostly perturbs singular values while preserving dominant singular subspaces, the polar map removes much of the scale error. Adam places the quantized second moment in a denominator, where scale error is directly converted to step-size error. The 8-bit Muon study reports parity with full-precision Muon up to 2.7B parameters and as much as 62% reduction in optimizer-state footprint in its latest experiments [Gupta et al., 2025]. It finds that simple linear blockwise quantization can work for Muon, whereas AdamW more often requires a dynamic codebook. This is not evidence that every spectral optimizer is quantization invariant. Perturbations that rotate singular subspaces or interact with finite Newton–Schulz iterations can still matter.

12.5 Eliminating master weights Mixed-precision training commonly applies an update to an FP32 master copy and then casts to the training format. If the update is applied directly to a quantized weight, ct+1 = QW (W ct + ∆Wt ), W

(38)

small updates can disappear and the quantization residual accumulates. ECO injects this residual into momentum as error feedback [Nikdan et al., 2026]. Conceptually, ft+1 = W ct + ∆Wt + et , W

(39)

ct+1 = QW (W ft+1 ), W

(40)

ft+1 − W ct+1 , et+1 = W

(41)

with the error represented through the optimizer dynamics without a full additional buffer. The method targets a different memory term from state quantization. Combining low-bit moments and no master weights can therefore compound savings, but also couples two recursive error channels.

12.6 Dynamic precision allocation Fixed bit width ignores the fact that sensitivity varies by layer, state, and training phase. STQuant allocates precision spatio-temporally, selecting influential factors and updating decisions with lower search cost [Liu et al., 2026b]. Its reported average width near five bits and large state-memory reduction suggest that uniform 8-bit storage leaves efficiency on the table. 28

Blog: Survey of Optimizers

August 2026

Table 7: Failure modes specific to low-precision optimizer state. Failure

Mechanism

Range saturation

Outlier exceeds block scale

State staleness Denominator error Direction corruption Master-weight loss

Diagnostic

Saturated-code fraction by layer EMA increment rounds to Fraction of unchanged same code state entries Small vt is under-resolved Distribution of effective learning rates Signed momentum crosses Cosine between code boundary quantized/full update Parameter update vanishes Accumulated weight after requantization residual

Mitigation Smaller blocks, dynamic range expansion Log codebook, scale update, reset Unsigned log quantization, floor, higher precision Precision-specific momentum, error feedback Master copy or ECO-style feedback

Dynamic allocation introduces a control problem. A precision transition changes the quantization lattice and can itself perturb state. The controller needs a stable sensitivity signal, a transition cost, and a budget constraint. Useful evaluations should report the precision path over time, not only its average.

12.7 Optimizer choice can change later quantizability Low-precision optimization is not limited to storing the training state. The optimizer can change the activation outliers of the final model. Outlier-Safe Pre-Training combines Muon, a single-scale normalization design, and an embedding projection to prevent extreme activation channels [Park et al., 2025]. A 1.4B model trained for one trillion tokens shows much stronger 4-bit post-training behavior than an Adam-trained comparison in that study. The causal contribution of the optimizer must be separated from the architecture changes, but the result broadens the evaluation target. A training optimizer can be judged by: • pretraining loss and time; • state memory during training; • stability under quantization-aware training; • outlier statistics and post-training quantizability. An optimizer that is slightly worse in BF16 loss but produces a substantially easier-to-quantize model may be preferable for deployment.

13 A synthesis: five claims that survive the leaderboard 13.1 Matrix-aware optimization is real, but second order is too coarse Muon’s polar normalization, Shampoo’s Kronecker whitening, SOAP’s rotated Adam, and row/columnadaptive hybrids solve different problems. Their gains across controlled and scaled studies are too consistent to dismiss. But calling all of them second order encourages incorrect transfer of intuition and hyperparameters. The relevant questions are which statistics are historical, which norm is controlled, and which information is discarded.

29

Blog: Survey of Optimizers

August 2026

13.2 Variance adaptation keeps returning Pure spectral normalization is strong, especially early in training and at shorter data ratios. Longer horizons often reward variance-aware methods, whether diagonal, rotated, rowwise, or Kroneckerfactored [Wen et al., 2026b, Frans et al., 2025a]. AdaMuon, NorMuon, COSMOS, and Mousse can all be read as attempts to recover variance or curvature information at different granularities.

13.3 The best geometry is parameter-class dependent Most Muon implementations already exempt vectors, biases, normalization scales, and often embeddings, sending them to AdamW. A symmetry-compatible optimizer makes the principle explicit: assign an update rule based on the symmetry and role of each parameter block, including embeddings, output heads, gated MLPs, and routers [Lau and Su, 2026]. One optimizer for the model may eventually sound as crude as one initialization scale for every layer.

13.4 Memory and systems can reverse an algorithmic result Matrix inverse roots, basis updates, all-gathers, state sharding, and quantization kernels determine wall-clock performance. An exact distributed SOAP that hides communication is scientifically different from an approximation that changes its preconditioner [Khona et al., 2026]. Likewise, a smaller optimizer state can enable a batch or sequence length that changes the gradient distribution. Reported convergence without system context is incomplete.

13.5 There is no universal twofold replacement for AdamW The strongest current conclusion is conditional. A well-tuned matrix-aware method can reduce tokens to target loss, sometimes substantially. A temporal method such as AdEMAMix or MARS can win in a different large-batch, long-horizon regime. AdamW can remain preferable when robustness, downstream compatibility, implementation maturity, or switching cost dominates. The honest output of the literature is a decision procedure, not a crown.

14 Why optimizer leaderboards disagree The optimizer literature contains apparent contradictions: • Muon leads some short-horizon comparisons, while SOAP or Kronecker methods lead at longer data ratios. • AdEMAMix and MARS lead one large-batch study, while matrix methods dominate another. • a method reports a large speedup over AdamW in its own paper, but a controlled benchmark recovers a smaller gap. • an intermediate checkpoint favors one optimizer and the final checkpoint reverses the order. These observations are compatible once the protocol is treated as part of the result.

14.1 Scale and data ratio are separate axes Model scale changes width, depth, matrix shapes, communication, and the relative cost of optimizer kernels. Data-to-parameter ratio changes how long the optimizer remains in early, middle, and terminal regimes. A study spanning 0.1B–1.2B parameters and 1×–8× a standard compute-optimal data ratio 30

Blog: Survey of Optimizers

August 2026

finds that matrix-optimizer gains shrink with model scale while rankings also change with training duration [Wen et al., 2026b]. This means that a single scaling curve is insufficient. At minimum, evaluation needs a grid: {model scale} × {tokens per parameter}.

(42)

A method can have a favorable exponent in one dimension and unfavorable behavior in the other. Short proxy runs can then be misleading even when the proxy model is architecturally faithful.

14.2 Batch size changes both noise and systems The comprehensive optimizer benchmark varies model size, batch, and horizon and finds different leading families across settings [Semenov et al., 2025]. Batch size changes: • stochastic gradient variance; • the token half-life of momentum; • the number of optimizer steps at a fixed token budget; • hardware utilization and communication frequency. A matrix preconditioner estimated from a large, low-noise batch may behave differently from the same code on a small batch. AdEMAMix’s slow state sees a different number of updates. A schedule specified in steps changes in token time. Every one of these can alter rankings.

14.3 Hyperparameter transfer can be unfair in both directions Blindly copying AdamW’s learning rate to a matrix optimizer can under-scale or over-scale the update. But giving every method an unconstrained search can also be misleading if the tuning budgets differ. A fair study should distinguish: 1. native tuning: best achievable result with a comparable search budget; 2. transfer tuning: performance when parameters are transferred by a published scaling rule; 3. default robustness: volume of the stable, near-optimal region. Update-RMS matching is a useful calibration, not a substitute for tuning. If an Adam step has RMS rA and a Muon step has RMS rM , scale the initial learning rate so their early RMS values match. The optimizers can still diverge later because their norm dynamics differ. Weight decay must be tuned jointly with learning rate. In scale-invariant networks, decay changes weight norms and hence angular progress. Matrix optimizers can appear fragile to decay because their update norm is more tightly controlled; Hyperball and Muown are direct responses to this interaction [Wen et al., 2026a, Lion et al., 2026].

14.4 The endpoint metric can hide practical value Final validation loss is important but not sufficient. A method may: • reach a useful target earlier but finish at the same loss; • have better final pretraining loss but worse downstream plasticity; • be token-efficient but wall-clock slower; 31

Blog: Survey of Optimizers

August 2026

Table 8: Protocol variables that can reverse an optimizer comparison. Variable

Mechanism of reversal

Minimum reporting requirement

Training horizon

Early spectral gain versus late variance adaptation; decay exposes progress Kernel-to-model cost and optimizer advantage scale differently Changes noise, momentum half-life, step count, and utilization Different methods tolerate or prefer different rate paths Controls norms and angular rate, not only regularization Non-matrix fallback can dominate embeddings and heads Quantization error affects denominators, factors, and polar maps differently Communication and factorization cost alter wall-clock order

Multiple endpoints including planned final token budget At least two scales; fit uncertainty if claiming a trend Global tokens per step, microbatch, accumulation, β conversion Method-specific schedule sweep or justified shared policy Joint learning-rate/decay sweep and weight-norm traces Exact parameter-class map and relative update RMS State/compute precision for every operation Step-time breakdown and exposed/overlapped communication

Model scale Batch size Schedule Weight decay Parameter routing Precision Systems implementation

• be slightly worse per token but enable a much larger fixed-memory model; • reduce loss spikes and failed runs without improving the median successful run. The no-decay weight-space study suggests that pretraining endpoint and later supervised adaptation can diverge [Yano et al., 2026]. Stable-SPAM emphasizes failure prevention. Low-memory methods emphasize feasible workload. A single scalar cannot rank all of these objectives.

14.5 Reading paper-specific speedup claims A claim such as 1.4 times faster should be decomposed into: speedup =

baseline resource to target . method resource to target

(43)

The resource may be tokens, steps, FLOPs, or seconds. The target may be final loss, a chosen intermediate loss, or downstream score. The baseline may use transferred rather than tuned hyperparameters. None of these choices invalidates the claim, but all determine its scope. For example, the scalable Muon study fits scaling laws and reports a large FLOP reduction in its own protocol [Liu et al., 2025]. The controlled cross-scale study finds smaller gains at its largest model [Wen et al., 2026b]. The large-scale systems study again finds both Muon and SOAP better than AdamW under update-RMS matching and very large batches [Khona et al., 2026]. The correct synthesis is not an average percentage. It is that matrix geometry remains beneficial across diverse regimes, while the magnitude is strongly protocol-dependent.

14.6 Evidence tiers should apply to claims, not papers A paper can contain scaled evidence for stability, controlled evidence for one component, and emerging evidence for generalization. Labeling the entire paper strong or weak loses this granularity. This survey uses: 32

Blog: Survey of Optimizers

August 2026

Table 9: A minimal family-level benchmark rather than a name-level tournament. Family

Representative baseline

Mechanism isolated

Required control

Coordinate adaptive

AdamW / ADOPT

Multi-timescale

AdEMAMix / MARS

β2 , decay, denominator precision Token half-life and batch size

Spectral

Muon / Scion

Whitening

SOAP / SPlus

Hybrid matrix

NorMuon / COSMOS

Low-memory

Adam-mini / GaLore2 / SWAN 8-bit AdamW / 8-bit Muon / SOLO

Elementwise variance and update ordering Longer memory or variance-reduced temporal estimator Matrix norm and polar direction Historical matrix basis and variance adaptation Granularity and placement of adaptation State support, projection, or elimination Quantized state recurrence

Low-precision

Shape scaling and update RMS Factor refresh, state cost, eigensolver Same polar implementation and fallback policy Fixed batch and fixed-memory views Same codebook budget and kernel accounting

• scaled claim: demonstrated at high parameter/token/system scale; • controlled claim: isolated through matched sweeps or ablations; • theoretical claim: proved under stated assumptions; • emerging claim: recent, narrow, or awaiting independent replication. The label attaches to the sentence. Muon has scaled evidence that polar-style updates can work in large language models, controlled evidence for some scaling choices, and still-emerging evidence for many recent variants. S-Adam has a developed stability argument for non-smooth objectives and emerging large-model evidence. STQuant has a concrete systems result but limited architecture coverage. This claim-level discipline is more informative than ranking publication status.

14.7 A reproducible benchmark matrix A compact but credible future benchmark could use: 1. two model scales separated by at least fourfold parameters; 2. two data ratios, one short and one long; 3. two batch regimes with token-time-corrected momentum; 4. cosine and an anytime schedule; 5. full-precision and one low-precision state configuration; 6. a fixed-token, fixed-FLOP, and fixed-memory view. Not every optimizer needs a full factorial search. A staged design can first tune each family on a central setting, then test transfer. The important requirement is to predeclare which comparisons isolate mechanisms and which measure complete recipes.

33

Blog: Survey of Optimizers

August 2026

Table 10: Reasonable starting points by constraint. These are experiment priorities, not universal defaults. Primary constraint Start with

What to verify first

Low switching risk

Retune learning rate, decay, and momentum timescale; measure target-loss time, not only final loss Update-RMS alignment, matrix eligibility rules, shape scaling, and non-matrix parameter treatment

Token efficiency on matrix-heavy models

Optimizer-state memory Unknown stopping time Low-precision state Small batch

Tuned AdamW; then Cautious or AdEMAMix Muon as the inexpensive matrix baseline; SOAP/SPlus or a Kronecker method for longer horizons Adam-mini or SlimAdam for Adam-like semantics; APOLLO/RACS/SWAN for aggressive reduction Schedule-free AdamW or a WSD/cooldown trunk; SF-NorMuon as an emerging matrix option A tested dynamic 8-bit AdamW implementation or 8-bit Muon Existing optimizer with token-time rescaling of β

Fixed-memory throughput, basis-refresh cost, full-rank residual behavior, and accuracy at the enlarged batch Anytime checkpoint quality, decay/averaging placement, and later fine-tuning plasticity State staleness, small-denominator behavior, loss-spike recovery, and end-to-end kernel speed Use Eq. (28); compare true microbatch updates against gradient accumulation

15 A practical decision guide A conservative experiment sequence is: 1. Establish a reproducible AdamW frontier. Sweep learning rate and weight decay jointly; log update RMS by parameter class; keep data order and precision fixed. 2. Change one optimizer axis. If testing Muon, keep the schedule conventional. If testing schedulefree training, start with AdamW geometry. If quantizing state, first match the full-precision algorithm. 3. Add the interaction only after both components work. A schedule-free matrix optimizer or low-bit SOAP changes multiple axes and is otherwise hard to diagnose. 4. Choose the budget metric before the winner. Fixed tokens answers a sample-efficiency question; fixed FLOPs includes matrix operations; time-to-target includes kernels and communication; fixed HBM includes the opportunity to enlarge the workload.

16 A minimum credible evaluation protocol Because optimizer rankings are protocol-sensitive, a credible comparison should report more than a final loss: • same initialization distribution, data mixture, token order, tokenizer, precision, and model parameterization; • optimizer-specific sweeps for learning rate, momentum/decay constants, weight decay, and schedule— 34

Blog: Survey of Optimizers

August 2026

defaults should not be transplanted blindly; • the policy for embeddings, output heads, normalization parameters, biases, and tensors that are not two-dimensional; • update RMS and weight RMS by parameter class, especially when comparing Adam-like and spectral methods; • loss at fixed tokens and FLOPs, time to a prespecified target, peak memory, optimizer-step time, and communication volume; • multiple horizons, including the actual endpoint, because rankings can flip late; • several seeds where the expected gap is small, plus instability counts and failed hyperparameter regions; • downstream adaptation, retention, or quantization quality when those are the real deployment objectives. The protocol should also state how much tuning compute each method received. An optimizer that wins only after a much larger sweep may still be worthwhile for a trillion-token run, but the search cost is part of the adoption decision.

17 Open problems for the next optimizer cycle A theory of hybrid geometry. We have useful interpretations of diagonal adaptivity, Kronecker preconditioning, and spectral steepest descent, but less guidance for choosing the granularity at which they should be combined. Why should variance be elementwise, rowwise, subspace-wise, or singular-direction-wise? Scale laws for optimizer advantage. The observed gap between matrix methods and Adam often shrinks with model size yet changes with data-to-parameter ratio [Wen et al., 2026b]. We need scaling laws that include optimizer state, batch, horizon, and tuning budget, not only parameter and token counts. Optimizer-aware parameterization. Update-RMS matching is a repair for mismatched conventions. A cleaner approach would co-design initialization, residual scaling, normalization, and optimizer geometry so learning-rate transfer is a property of the parameterization. State under changing coordinates. Low-rank projections, eigenspaces, sharding layouts, and quantizers all create coordinate systems that change over time. State transport and error feedback should become first-class primitives rather than paper-specific patches. Anytime training with downstream objectives. Schedule-free and cooldown methods optimize access to checkpoints, while no-decay results suggest that pretraining loss and later plasticity can diverge [Yano et al., 2026]. The right objective may be a path of adaptable models rather than one endpoint. Reproducible systems evidence. The field needs public, exact, distributed implementations with matched kernels and communication. Otherwise algorithmic and engineering gains remain entangled in ways that cannot be audited.

35

Blog: Survey of Optimizers

August 2026

18 Conclusion The optimizer landscape of 2025–2026 is best understood as a move from scalar update rules to composed training policies. Temporal estimation decides what history is trustworthy. Matrix geometry decides how a layer should move. Horizon policy decides when progress becomes visible. Representation decides whether the state can fit, communicate, and remain meaningful at low precision. Within that stack, the matrix-aware renaissance is the central scientific development: Muon demonstrates that a cheap polar transform can be powerful at scale, while SOAP, SPlus, and related methods show the continuing value of variance and covariance structure. Memory-reduced, schedule-free, and quantized methods are not side stories; they determine which geometry is operationally possible. There is therefore no single optimizer to replace AdamW. The more consequential transition is from choosing a brand name to designing and evaluating a stack. The winning optimizer will increasingly be the one that matches the parameter block, horizon, precision, and system budget of the run—and states those conditions clearly enough to reproduce.

36

Blog: Survey of Optimizers

August 2026

A Compact literature map Table 11: Representative works used in this review. Evidence characterizes the cited claim, not the permanent status of the method. Family

Work

Primary intervention

Evidence emphasis

Temporal

AdEMAMix [Pagliardini et al., 2025] MARS [Yuan et al., 2025] Cautious [Liang et al., 2026] SPAM [Huang et al., 2025a] Stable-SPAM [Huang et al., 2025b] ADOPT [Taniguchi et al., 2024] S-Adam [Xu et al., 2026]

Fast and slow gradient EMAs

Scaled, method-specific

Recursive variance-reduced momentum

Controlled and scaled

Gradient–momentum agreement mask

Broad empirical

Spike clipping and periodic state reset

Broad empirical

Historical norm tracking, clipping, and resets for 4-bit training

Controlled and scaled

Muon [Jordan et al., 2024, Liu et al., 2025] Scion [Pethick et al., 2025] SOAP [Vyas et al., 2025] SPlus [Frans et al., 2025b] COSMOS [Liu et al., 2026a] AdaMuon [Si et al., 2026] NorMuon [Li et al., 2026] Mousse [Zhang et al., 2026] MuonEq [Chang et al., 2026] Muown [Lion et al., 2026] AngularMuown [Hübler et al., 2026]

Approximate polar factor of momentum Scaled matrices

Adam-mini [Zhang et al., 2025]

Temporal Temporal Temporal Temporal / low precision Temporal Robustness / low precision Spectral Spectral Whitening Whitening Hybrid matrix Hybrid matrix Hybrid matrix Hybrid matrix Hybrid matrix Hybrid matrix Hybrid matrix Memory

Decouples current gradient from current Theory plus empirical denominator Randomized probes and Theory plus emerging instability-dependent damping empirical

Norm-constrained linear minimization oracles Adam in evolving Shampoo eigenbases Stable whitening, shape scaling, averaging SOAP leading subspace, Muon residual

Theory plus empirical Controlled and scaled Controlled Controlled

Second moments after orthogonalization Emerging Rowwise variance normalization after polar step Kronecker whitening before polar step

Scaled, method-specific Emerging

Row/column equilibration before polar step Explicit row-norm control

Emerging

Explicit control of angular step size

Emerging

Blockwise second moments

Scaled

Emerging

Continued on next page

37

Blog: Survey of Optimizers

August 2026

Table 11 continued from previous page

Family

Work

Primary intervention

Evidence emphasis

Memory

SlimAdam [Kalra et al., 2025] APOLLO [Zhu et al., 2025] LDAdam [Robert et al., 2025] GaLore2 [Su et al., 2025] Alice/RACS [Gong et al., 2026] SWAN [Ma et al., 2025] SCALE [Glentis et al., 2026]

SNR-guided aggregation of second moments Random low-rank auxiliary state

Controlled

Schedule-Free [Defazio et al., 2024] WSD [Wen et al., 2025] Power Scheduler [Shen et al., 2024] WSM [Tian et al., 2026] SF-NorMuon [Apte et al., 2026]

Memory Memory Memory Memory Memory Memory Horizon Horizon Horizon Horizon Horizon Low precision

COAT [Xi et al., 2025] Low precision SOLO [Xu et al., 2025] Low precision 8-bit Muon [Gupta et al., 2025] Low precision ECO [Nikdan et al., 2026] Low precision State staleness [Topollai and Choromanska, 2026] Low precision STQuant [Liu et al., 2026b] QuantizationOSP [Park et al., aware pretraining 2025]

Scaled

Projected moments with state transport Controlled Scalable low-rank gradient projection

Scaled

Low-rank eigen-Adam / row-column-scaled SGD Stateless instantaneous normalization/whitening Column-normalized SGD, minimal momentum

Controlled and scaled

Fast/averaged iterates without known endpoint Indefinite stable trunk plus decay branch Token- and batch-aware learning-rate scaling law Checkpoint merging as implicit decay

Broad empirical

Schedule-free neuron-normalized spectral update

Emerging

FP8 state and activations

Scaled

2–3 bit log-quantized states

Controlled and scaled

Blockwise momentum quantization

Scaled, method-specific Emerging

No master weights; quantization error feedback Analysis and resets for recursively quantized EMAs

Scaled Scaled

Controlled and scaled Controlled and scaled Controlled

Emerging

Layer- and time-dependent Emerging optimizer-state precision Optimizer and parameterization choices Scaled, that suppress activation outliers method-specific

References Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms. In Advances in Neural Information Processing Systems, volume 36, pages 49205–49233. Neural Information Processing Systems Foundation, 2023. doi: 10.52202/075280-2140. URL 38

Blog: Survey of Optimizers

August 2026

https://proceedings.neurips.cc/paper_files/paper/2023/hash/9a39b4925e35cf447ccba87 57137d84f-Abstract-Conference.html. Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 49069–49104. PMLR, 2025. URL https://proceedings.mlr.press/v267/pethi ck25a.html. James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2408–2417. PMLR, 2015. URL https://procee dings.mlr.press/v37/martens15.html. Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1842–1850. PMLR, 2018. URL https://proceedings.mlr.press/v80/gupta18a.html. Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=3xHDeA8Noi. Kevin Frans, Sergey Levine, and Pieter Abbeel. What really matters in matrix-whitening optimizers?, 2025a. URL https://arxiv.org/abs/2510.25000. Tim Tsz-Kit Lau and Weijie Su. Symmetry-compatible principle for optimizer design: Embeddings, LM heads, SwiGLU MLPs, and MoE routers, 2026. URL https://arxiv.org/abs/2605.18106. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. URL https://arxiv.org/abs/1412.6980. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. Matteo Pagliardini, Pierre Ablin, and David Grangier. The AdEMAMix optimizer: Better, faster, older. In International Conference on Learning Representations, 2025. URL https://openreview .net/forum?id=jj7b3p5kLY. Huizhuo Yuan, Yifeng Liu, Shuang Wu, Zhou Xun, and Quanquan Gu. MARS: Unleashing the power of variance reduction for training large models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 73553–73587. PMLR, 2025. URL https://proceedings.mlr.press/v267/yuan25f.html. Kaizhao Liang, Lizhang Chen, Bo Liu, and Qiang Liu. Cautious optimizers: Improving training with one line of code. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=zBPZeRjfgu. Lizhang Chen, Jonathan Li, Kaizhao Liang, Baiyu Su, Cong Xie, Nuo Wang Pierse, Chen Liang, Ni Lao, and Qiang Liu. Cautious weight decay. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=Gwe6gbGng5.

39

Blog: Survey of Optimizers

August 2026

Tianjin Huang, Ziquan Zhu, Gaojie Jin, Lu Liu, Zhangyang Wang, and Shiwei Liu. SPAM: Spike-aware adam with momentum reset for stable LLM training. In International Conference on Learning Representations, 2025a. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/ 7a70ad3d9c704fb9b81b5c69eda722dc-Abstract-Conference.html. Shohei Taniguchi, Keno Harada, Gouki Minegishi, Yuta Oshima, Seong Cheol Jeong, Go Nagahara, Tomoshi Iiyama, Masahiro Suzuki, Yusuke Iwasawa, and Yutaka Matsuo. ADOPT: Modified adam can converge with any β2 with the optimal rate. In Advances in Neural Information Processing Systems, volume 37, pages 72438–72474, 2024. doi: 10.52202/079017-2309. URL https://proceedi ngs.neurips.cc/paper_files/paper/2024/hash/84d286e32bbee8fa3a86ee9c50e00081-Abstr act-Conference.html. Andrei Semenov, Matteo Pagliardini, and Martin Jaggi. Benchmarking optimizers for large language model pretraining, 2025. URL https://arxiv.org/abs/2509.01440. Ruoran Xu, Borong She, Xiaobo Jin, and Qiufeng Wang. Singularity-aware optimization via randomized geometric probing: Towards stable non-smooth optimization, 2026. URL https://arxiv.org/abs/ 2605.29547. Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=6Tm1mposlrM. Tianjin Huang, Haotian Hu, Zhenyu Zhang, Gaojie Jin, Xiang Li, Li Shen, Tianlong Chen, Lu Liu, Qingsong Wen, Zhangyang Wang, and Shiwei Liu. Stable-SPAM: How to train in 4-bit more stably than 16-bit adam, 2025b. URL https://arxiv.org/abs/2502.17055. Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing shampoo using adam for language modeling. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.c c/paper_files/paper/2025/hash/e988664070e9591f93fdcf605f7dc623-Abstract-Conferenc e.html. Kevin Frans, Sergey Levine, and Pieter Abbeel. A stable whitening optimizer for efficient neural network training. In Advances in Neural Information Processing Systems, volume 38, pages 192840–192864, 2025b. doi: 10.52202/085713-5790. URL https://proceedings.neurips.cc/paper_files/paper /2025/hash/fe722dac731f46d30b043c6b0db9a3ef-Abstract-Conference.html. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://keller jordan.github.io/posts/muon/. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for LLM training, 2025. URL https://arxiv.or g/abs/2502.16982. Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology, 2024. URL https://arxiv.org/abs/2409.20325. Liming Liu, Zhenghao Xu, Zixuan Zhang, Hao Kang, Zichong Li, Chen Liang, Weizhu Chen, and Tuo Zhao. COSMOS: A hybrid adaptive optimizer for memory-efficient training of LLMs. In International Conference on Learning Representations, 2026a. URL https://proceedings.iclr.cc/paper_file s/paper/2026/file/8f81b7ffe28153e35ddfa60b35994eff-Paper-Conference.pdf. 40

Blog: Survey of Optimizers

August 2026

Chongjie Si, Debing Zhang, and Wei Shen. Adamuon: Adaptive muon optimizer. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=OpxVAH FmkL. Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. Normuon: Making muon more efficient and scalable. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=m1IRWFAMsa. Yechen Zhang, Shuhao Xing, Junhao Huang, Kai Lv, Yunhua Zhou, Xipeng Qiu, Qipeng Guo, and Kai Chen. Mousse: Rectifying the geometry of muon with curvature-aware preconditioning, 2026. URL https://arxiv.org/abs/2603.09697. Da Chang, Qiankun Shi, Lvgang Zhang, Yu Li, Ruijie Zhang, Yao Lu, Yongxiang Liu, and Ganzhao Yuan. Muoneq: Balancing before orthogonalization with lightweight equilibration, 2026. URL https://arxiv.org/abs/2603.28254. Kai Lion, Florian Hübler, Bingcong Li, Antonio Orvieto, and Niao He. Muown: Row-norm control for muon optimization, 2026. URL https://arxiv.org/abs/2605.10797. Kaiyue Wen, Xingyu Dang, Kaifeng Lyu, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them II: Hyperball optimization, 2026a. URL https://arxiv.org/abs/2606.1 6899. Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov, Amir Zainullin, and Aleksandr Beznosikov. Softsign: Smooth sign in your optimizer for better parameter heterogeneity handling, 2026. URL https://arxiv.org/abs/2605.31371. Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them. In International Conference on Learning Representations, 2026b. URL https: //proceedings.iclr.cc/paper_files/paper/2026/file/ea4f0a6ad7088c9b93f22d74c0c9e8c2 -Paper-Conference.pdf. Mikail Khona, Aditya Vavre, Boxiang Wang, Deyu Fu, Hao Wu, Mike Chrzanowski, Bryan Catanzaro, Dheevatsa Mudigere, Jeff Pool, Michael Lightstone, Mohammad Shoeybi, Mostofa Patwary, Nima Tajbakhsh, and Tijmen Blankevoort. SOAP, muon, and beyond: Pushing LLM pretraining scales, 2026. URL https://arxiv.org/abs/2607.20548. Florian Hübler, Kai Lion, Antonio Orvieto, and Niao He. Muown implicitly performs angular step-size decay, 2026. URL https://arxiv.org/abs/2606.23637. Kwangjun Ahn, Byron Xu, Natalie Abreu, and John Langford. Dion: Distributed orthonormalized updates, 2025. URL https://arxiv.org/abs/2504.05295. Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P. Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_file s/paper/2025/hash/45ae878717399e6f62d57c65f052cd46-Abstract-Conference.html. Dayal Singh Kalra, John Kirchenbauer, Maissam Barkeshli, and Tom Goldstein. When can you get away with low memory adam?, 2025. URL https://arxiv.org/abs/2503.01843. Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. APOLLO: SGD-like memory, AdamW-level performance. 41

Blog: Survey of Optimizers

August 2026

In Proceedings of Machine Learning and Systems, volume 7, 2025. URL https://proceedings.ml sys.org/paper_files/paper/2025/hash/437bc4ccafd3fc6d4289bd10940be42b-Abstract-Con ference.html. Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu, and Dan Alistarh. LDAdam: Adaptive optimization from low-dimensional gradient statistics. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/0e 6eb6f6001748b66348b9b53ad434db-Abstract-Conference.html. DiJia Su, Andrew Gu, Jane Xu, Yuandong Tian, and Jiawei Zhao. GaLore 2: Large-scale LLM pre-training by gradient low-rank projection, 2025. URL https://arxiv.org/abs/2504.20437. Wenbo Gong, Meyer Scetbon, Chao Ma, and Edward Meeds. Towards efficient optimizer design for LLM via structured fisher approximation with a low-rank extension. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026 /file/d43f4656992d7869213f8d39b5bec640-Paper-Conference.pdf. Philip Zmushko, Aleksandr Beznosikov, Martin Takáč, and Samuel Horváth. FRUGAL: Memory-efficient optimization by reducing state overhead for scalable training. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 80708–80739. PMLR, 2025. URL https://proceedings.mlr.press/v267/zmushko25a.html. Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Taming momentum: Rethinking optimizer states through low-rank approximation. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=9Q0dNBYeEY. Chao Ma, Wenbo Gong, Meyer Scetbon, and Edward Meeds. SWAN: SGD with normalization and whitening enables stateless LLM training. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 41907–41942. PMLR, 2025. URL https://proceedings.mlr.press/v267/ma25g.html. Athanasios Glentis, Jiaxiang Li, Andi Han, and Mingyi Hong. A minimalist optimizer design for LLM pretraining. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=QBJvaWGj04. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 2018. URL https://proceedings.mlr.pres s/v80/shazeer18a.html. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-efficient LLM training by gradient low-rank projection. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 61121–61143. PMLR, 2024. URL https://proceedings.mlr.press/v235/zhao2 4s.html. Aaron Defazio, Xingyu Alice Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled. In Advances in Neural Information Processing Systems, volume 37, pages 9974–10007, 2024. doi: 10.52202/079017-0320. URL https://proceedings.neurips.cc/p aper_files/paper/2024/hash/136b9a13861308c8948cd308ccd02658-Abstract-Conference.ht ml.

42

Blog: Survey of Optimizers

August 2026

Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape view. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/20 25/hash/6a1fe80a9e2dcda0b3e5fd0fd87eb097-Abstract-Conference.html. Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Advances in Neural Information Processing Systems, volume 37, pages 76232–76264, 2024. URL https: //proceedings.neurips.cc/paper_files/paper/2024/hash/8b970e15a89bf5d12542810df8eae 8fc-Abstract-Conference.html. Changxin Tian, Jiapeng Wang, Qian Zhao, Kunlong Chen, Jia Liu, Ziqi Liu, Jiaxin Mao, Wayne Xin Zhao, Zhiqiang Zhang, and Jun Zhou. WSM: Decay-free learning rate schedule via checkpoint merging for LLM pre-training. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/file/da07cfa60cc883c5ee94ba8993 83bb6d-Paper-Conference.pdf. Anuj Apte, Pranav Deshpande, Niraj Kumar, Shouvanik Chakrabarti, and Junhyung Lyle Kim. Anytime training with schedule-free spectral optimization, 2026. URL https://arxiv.org/abs/2605.23061. Aaron Defazio. Schedulefree+: Scaling learning-rate-free and schedule-free learning to large language models, 2026. URL https://arxiv.org/abs/2605.19095. Kazuki Yano, Shun Kiyono, Sosuke Kobayashi, Sho Takase, and Jun Suzuki. Pre-training LLM without learning rate decay enhances supervised fine-tuning. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=JnebU2QLdH. Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, and Micah Goldblum. Small batch size training for language models: When vanilla SGD works, and why gradient accumulation is wasteful. In Advances in Neural Information Processing Systems, volume 38, pages 165262–165287, 2025. doi: 10.52202/085713-4978. URL https://proceedings.neurips.cc/paper_files/paper /2025/hash/dabfbf500318462382aa70a95466ad85-Abstract-Conference.html. Yikang Shen, Matthew Stallone, Mayank Mishra, Gaoyuan Zhang, Shawn Tan, Aditya Prasad, Adriana Meza Soria, David D. Cox, and Rameswar Panda. Power scheduler: A batch size and token number agnostic learning rate scheduler, 2024. URL https://arxiv.org/abs/2408.13359. Thomas Pethick, Wanyun Xie, Roman Machacek, and Volkan Cevher. Optimistic dual averaging unifies modern optimizers, 2026. URL https://arxiv.org/abs/2605.11172. Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Keutzer, Jianfei Chen, and Song Han. COAT: Compressing optimizer states and activations for memory-efficient FP8 training. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_file s/paper/2025/hash/6ac807c9b296964409b277369e55621a-Abstract-Conference.html. Cong Xu, Wenbin Liang, Mo Yu, Anan Liu, Ke-Yue Zhang, Lizhuang Ma, Jianyong Wang, Jun Wang, and Wei Zhang. Pushing the limits of low-bit optimizers: A focus on EMA dynamics, 2025. URL https://arxiv.org/abs/2505.00347. Aman Gupta, Rafael Celente, Abhishek Shivanna, D. T. Braithwaite, Gregory Dexter, Shao Tang, Hiroto Udagawa, Daniel Silva, Rohan Ramanath, and S. Sathiya Keerthi. Effective quantization of muon optimizer states, 2025. URL https://arxiv.org/abs/2509.23106. 43

Blog: Survey of Optimizers

August 2026

Kristi Topollai and Anna Choromanska. Understanding quantization of optimizer states in LLM pre-training: Dynamics of state staleness and effectiveness of state resets, 2026. URL https: //arxiv.org/abs/2603.16731. Mahdi Nikdan, Amir Zandieh, Dan Alistarh, and Vahab Mirrokni. ECO: Quantized training without full-precision master weights. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=dfVm5yZZ6G. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. In International Conference on Learning Representations, 2022. URL https://open review.net/forum?id=shpkpVXzo3h. Minglu Liu, Cunchen Hu, Liangliang Xu, Fengming Tang, Ruijia Wang, and Fu Yu. STQuant: Spatiotemporal adaptive framework for optimizer quantization in large multimodal model training, 2026b. URL https://arxiv.org/abs/2604.06836. Jungwoo Park, Taewhoo Lee, Chanwoong Yoon, Hyeon Hwang, and Jaewoo Kang. Outlier-safe pre-training for robust 4-bit quantization of large language models, 2025. URL https://arxiv.or g/abs/2506.19697.

44

Related documents

• Reinforcement learning to choose optimizers#822236
• RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory#1076281
• Separating plane algorithms for convex optimization#429981
• AReaL-TIK: Stateful Agentic Optimization of Unified RL Kernels through an Optimization IR#1129739
• AReaL-TIK: Stateful Agentic Optimization of Unified RL Kernels through an Optimization IR#1108701