Preprint: 1–11, 2026
Open Problem: Is AdamW Effective Under Heavy-Tailed Noise? Dingzhi Yu1 Hongyi Tao1 Yuanyu Wan2 Luo Luo3 Lijun Zhang1
arXiv:2606.23676v1 [cs.LG] 22 Jun 2026
1
Nanjing University
YUDZ @ LAMDA . NJU . EDU . CN 221220032@ SMAIL . NJU . EDU . CN WANYY @ ZJU . EDU . CN LUOLUO @ FUDAN . EDU . CN ZHANGLJ @ LAMDA . NJU . EDU . CN 2
Zhejiang University
3
Fudan University
Abstract AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying, as empirical evidence indicates that stochastic gradient noise in LLM pretraining is typically heavy-tailed. Recent work shows that sign-based optimizers such as Lion and Muon achieve sharp heavy-tailed rates, and that AdaGrad can also converge under heavy-tailed noise. However, no rigorous convergence theory for AdamW has yet been established in this regime. Can AdamW converge under the same heavy-tailed assumptions, or does its second-moment accumulator create a genuine obstruction? We formulate this as an open problem, prove a positive weighted-metric benchmark, and give a corridor lower-bound mechanism showing how denominator memory can hide large gradients. Keywords: AdamW, heavy-tailed noise, convergence theory, LLMs, Lion and Muon
1. Introduction Heavy-tailed gradient noise is ubiquitous in modern machine learning (Simsekli et al., 2019; Zhang et al., 2020; Gurbuzbalaban et al., 2021; Battash et al., 2024). This phenomenon naturally arises in language modeling tasks (Piantadosi, 2014), as verified by a fruitful line of empirical research (Kunstner et al., 2023, 2024; Ahn et al., 2024; Kunstner and Bach, 2025; Yadav et al., 2025). Yu et al. (2026b) further validate on nanoGPT (Karpathy, 2022) pretraining that heavy-tailed noise persists across coordinates and matrix blocks, and that the noise magnitude tracks the gradient magnitude in a way not captured by classical bounded-variance models. Since AdamW (Kingma and Ba, 2015; Loshchilov and Hutter, 2019) is the main workhorse for LLM training, understanding AdamW under this heavy-tailed regime is a central theoretical question. This heavy-tailed viewpoint is sharpened by the empirical success of sign-based optimizers. Lion (Chen et al., 2023) and Muon (Jordan et al., 2024; Liu et al., 2025a) have shown consistent gains over AdamW in many practical scenarios (Zhao et al., 2025; Shah et al., 2025; Wen et al., 2025; Semenov et al., 2025), including large-scale LLM systems such as Kimi-K2, GLM-5, and DeepSeek-V4 (Kimi Team, 2025; Zeng et al., 2026; DeepSeek-AI, 2026). Yu et al. (2026b) establish a sharp theory for this phenomenon, showing that under finite pth moment heavy-tailed noise with p ∈ (1, 2], sign-based vector and matrix optimizers attain the sharp rate of O(T −(p−1)/(3p−2) ), which is tight in T (Liu and Zhou, 2025). This makes the missing AdamW baseline more important, not less. If AdamW satisfies the same guarantee, then the empirical advantage of Lion and Muon must come from other aspects that cannot be captured by classical convergence theory. If AdamW fails, then the theory would help explain why it may be dominated by Muon in heavy-tailed LLM training regimes. © 2026 D. Yu, H. Tao, Y. Wan, L. Luo & L. Zhang.
Y U TAO WAN L UO Z HANG
The open problem is therefore deliberately narrow. The √ finite-variance AdamW baseline has essentially been supplied by Li et al. (2025), who prove an O( d/T 1/4 ) ℓ1 -convergence rate under coordinate-wise bounded variance. We ask what remains true after replacing this variance assumption by a finite pth moment condition. Existing Adam-type convergence analyses usually rely on finite/affine variance, bounded gradients, or bounded coordinate ratios (Reddi et al., 2018; Zhang et al., 2022; Li et al., 2023b; Hong and Lin, 2024; Ahn and Cutkosky, 2024; Peng et al., 2025; Li 2 ] et al., 2025). Those assumptions do not address the case where E[|gt,i |p ] < ∞ for p < 2 but E[gt,i may be infinite, where gt,i is the i-th coordinate of the stochastic gradient gt . Notably, Chezhegov et al. (2025) show that Adam could suffer from worse convergence when the noise is heavy-tailed. A positive or negative AdamW theorem in this regime should therefore expose how the optimizer handles heavy-tailed noise, not only repeat the finite-variance descent algebra.
2. Setup Let f : Rd → R be differentiable and lower bounded, and write ∆ = f (x1 ) − inf x∈Rd f (x). At PB b −1 step t, AdamW receives a mini-batch {gtb }B b=1 gt . With m0 = v0 = 0, b=1 and uses gt = B mt = β1 mt−1 +(1−β1 )gt , vt = β2 vt−1 +(1−β2 )gt ⊙gt , xt+1 = (1−ηt λ)xt − √
ηt mt . (1) vt + ε t
The schedules ηt and εt may absorb the usual bias corrections. The primary open case is λ = 0. Classical stochastic analyses assume smoothness of f and finite oracle variance, for example E[∥gt − ∇f (xt )∥2 | Ft−1 ] < ∞. In that regime, recent AdamW analyses can control the deterministic AdamW terms and the stochastic error by second moments (Li et al., 2025). We instead focus on the heavy-tailed regime motivated by language-model training, where both the local curvature and the noise scale may depend on the current gradient magnitude. Following Yu et al. (2026b), let D(x) = diag(L0 + L1 ⊙ |∇f (x)|), where L0 , L1 ∈ Rd+ are coordinate-wise curvature parameters. For all pairs x, x′ satisfying the local step-size condition of the generalized model, assume ∇f (x′ ) − ∇f (x) D(x)−1 ≤ x′ − x D(x) .
(GC)
This is the vector version of the gradient-curvature (GC) condition used by Yu et al. (2026b) in their matrix analysis, and it is closely related to AdaGrad and generalized-smoothness analyses under nonuniform curvature (Faw et al., 2023; Li et al., 2023a; Liu et al., 2025b; Yu et al., 2025; Liu, 2026). For p ∈ (1, 2], σ0 ≻ 0, σ1 ⪰ 0, the stochastic gradients are captured by the coordinate-wise heavy-tailed (HT) noise model below: h i p p p b E[gtb | Ft−1 ] = ∇f (xt ), E gt,i − ∇i f (xt ) | Ft−1 ≤ σ0,i + σ1,i |∇i f (xt )|p , ∀i ∈ [d]. (HT) b and ∇ f (x ) denote the ith coordinate. Figures 1 and 2 of Yu et al. Here subscripts such as gt,i i t (2026b) provide direct empirical support for this generalized heavy-tailed model on language-model pretraining runs. Our partial results below focus on the clean specialization L1 = σ1 = 0, where we write L = L0 , σ = σ0 > 0, and D = diag(L) ≻ 0. In this specialization, we assume (GC) globally, which is the regime used in the derivations below. This standard simplification is already sufficient to isolate the AdamW-specific barriers, as the function class for L1 = σ1 = 0 is a subset of the more general class with L1 , σ1 ≻ 0.
2
O PEN P ROBLEM : A DAM W U NDER H EAVY TAILS
Remark 1 The conditions (GC) and (HT) follow Yu et al. (2026b) and align with the geometry of sign-based methods (Jiang et al., 2025; Tao et al., 2026; Yu et al., 2026a). Since Adam can be viewed as a variance-adapted SignSGD variant (Balles and Hennig, 2018; Kunstner et al., 2023; Peng et al., 2025), the two methods share related geometry (Balles et al., 2020; Liu et al., 2026); this motivates studying AdamW on the same class. One can also leverage other assumptions based on a different geometry, such as standard ∥∇f (x′ ) − ∇f (x)∥2 ≤ L2 ∥x′ − x∥2 and p E[ gtb − ∇f (xt ) 2 | Ft−1 ] ≤ σ2p , but that would typically incur explicit dimensional factors. More importantly, regardless of which curvature and noise models are considered, the same AdamWspecific technical barriers identified later in this paper would still apply, and the open problem would still be whether AdamW can overcome those barriers to match the sign-based rates.
3. Open Problem Open problem. Under (GC) and (HT), determine whether the convergence of AdamW can match the rate of sign-based methods under heavy-tailed noise in (2). More concretely, for the AdamW update defined in (1), and already for the core case λ = 0, either prove a nonasymptotic ℓ1 stationarity upper bound with the same heavy-tailed rate as sign-based methods, or construct a lower-bound instance showing that no such matching guarantee is possible for AdamW. The comparison target is the following. Under the same heavy-tailed model, Yu et al. (2026b) prove that sign-based optimizers attain the rate O(T −(p−1)/(3p−2) ), which is tight in T (Liu and Zhou, 2025). In the clean specialization L1 = σ1 = 0 from Section 2, this corresponds to T p p−1 p−1 1X − 3p−2 3p−2 3p−2 ∥σ∥1 (BT ) . (2) E[∥∇f (xt )∥1 ] ≤ O (∆ ∥L∥1 ) T t=1
This target is intentionally ambitious. It asks whether AdamW can theoretically keep pace with signbased methods once the finite-variance assumption is removed. A slower AdamW upper bound under only (HT) would still be progress, but it would not answer this comparison. Conversely, a lower bound should identify an AdamW-specific obstruction rather than merely restating that arbitrary heavy-tailed noise is difficult. The key requirement is that the theorem use only (HT), not a canonical finite variance assumption or a strong boundedness assumption. The following propositions are our partial results, proved in Appendix A. They separate a positive weighted benchmark from Adam-specific obstructions to converting weighted progress into plain ℓ1 stationarity. Proposition 2 (A positive weighted-metric benchmark) Consider AdamW with λ = 0, constant ηt = η, εt ≥ 0, β2 = β1 ∈ (0, 1), and the convention 0/0 = 0. Define at,i = √
|mt,i | , vt,i + εt
St = ⟨at , |∇f (xt )|⟩ .
Under the clean specialization of (GC) and (HT), with independent mini-batch samples, there are choices of η and β1 such that, in the large-horizon regime T (1 − β1 ) ≳ 1, T p p−1 p−1 1X − 3p−2 3p−2 3p−2 E[St ] ≤ O (∆ ∥L∥1 ) ∥σ∥1 (BT ) . T t=1
The hidden constant depends only on p. 3
Y U TAO WAN L UO Z HANG
Proposition 2 is useful for understanding how sign alignment and finite pth moments can yield the target heavy-tailed rate. For SignSGD and Lion, at ≡ 1, so St = ∥∇f (xt )∥1 , and the same proof template becomes a plain ℓ1 guarantee. However, it still does not solve the open problem. For Adam, the data-dependent weights at,i may collapse after a heavy-tailed outlier. The result also uses the equal-memory regime β2 = β1 , while practical AdamW usually has β2 ≫ β1 . The unresolved case is therefore exactly the unequal-memory regime where the denominator may remember outliers much longer than the numerator (cf. Proposition 5). The weighted guarantee isolates the missing step, which is to prove that Adam’s self-normalized weights cannot hide a large gradient, or construct an instance where they do.
4. Technical Barriers The next two propositions are barrier mechanisms isolated in this paper and proved in Appendix A. They should be read as mechanism statements rather than previously known lower bounds. Proposition 3 shows why unbiasedness alone cannot give alignment, while Proposition 5 gives a corridor lower-bound mechanism for long-memory Adam schedules. √ Let ut = mt /( vt + εt ). By the descent lemma implied by (GC), any proof with λ = 0 must control η2 f (xt+1 ) ≤ f (xt ) − ηt ⟨∇f (xt ), ut ⟩ + t ∥ut ∥2D . 2 The smoothness term depends on the size of the normalized update and is not where heavy tails enter. The heavy-tailed difficulty is the alignment term. SummingPthe descent inequality can only give a plain ℓ1 stationarity bound if the cumulative inner product Tt=1 ηt ⟨∇f (xt ), ut ⟩ dominates PT t=1 ηt ∥∇f (xt )∥1 . We therefore isolate the exact alignment deficit quantity as ET (c) =
T X
i h ηt E (c ∥∇f (xt )∥1 − ⟨∇f (xt ), ut ⟩)+ .
t=1
By definition of ET (c), it holds that T X t=1
ηt E[⟨∇f (xt ), ut ⟩] ≥ c
T X
ηt E[∥∇f (xt )∥1 ] − ET (c).
(A)
t=1
Thus (A) is a bookkeeping identity, not an extra assumption. If ET (c) is lower order, then the descent inequality gives (2) up to the smoothness term. The challenge is that, for AdamW, this deficit 2 |F depends on the self-normalizer vt built from gt2 . When p < 2, E[gt,i t−1 ] may be infinite, so the finite-variance proof strategy of replacing vt by a conditional second-moment proxy is unavailable. AdaGrad is informative only as a contrast: its monotone accumulator can be controlled under finite pth moments in recent work (Liu, 2026), whereas AdamW must additionally control the mismatch between an exponential numerator and an exponential squared-gradient denominator. Proposition 3 (Averaging is necessary) Fix any µ > 0, p ∈ (1, 2], and q < 1/2. At any onedimensional point with true gradient µ, there is an unbiased oracle distribution with finite pth noise moment such that AdamW with β1 = β2 = 0 and εt = 0 has negative expected alignment, E[µg/ |g|] = µ(2q − 1) < 0. Consequently, no AdamW theorem can prove (A) from unbiasedness alone unless it explains how mini-batching, momentum, or robustness restores alignment. 4
O PEN P ROBLEM : A DAM W U NDER H EAVY TAILS
Remark 4 When β1 = β2 = 0 and εt = 0, AdamW with λ = 0 is exactly SignSGD. Proposition 3 is therefore a warning about arbitrary unbiased heavy-tailed oracles, not about practical AdamW. Proposition 5 (A corridor lower-bound mechanism) Fix p ∈ (1, 2) and consider 1D AdamW started from x1 = 0, with λ = 0, εt = 0, β12 < β2 , and ρi = 1 − βi with ρ2 ≤ 1/2. Let Cβ be the update bound in (B). There are universal constants c, c0 , CK , Cp > 0 such that the following holds. For any µ, R, σ > 0 and δ ∈ (0, 1/2), set M = σq −1/p ≥ 2µ,
q = c0 ρ2 log(1/δ) ≤ 1/2,
K = ⌈CK /ρ2 ⌉.
There exists a C 2 , lower-bounded one-dimensional function fµ,R and an unbiased oracle satisfying ′ E[ gt − fµ,R (xt )
p
| Ft−1 ] ≤ 2σ p
such that, if T > K and K X ηt Cβ t=1
then
R
+
T i X ηt h µ (p−1)/p (2−p)/(2p) Cp + ρ1 ρ2 (log(1/δ))1/p + Cβ δ ≤ c, R σ
t=K+1
T
1X ′ (xt ) ] ≥ µ/2. E[ fµ,R T t=1
Moreover fµ,R has smoothness L ≍ µ/R, initial gap ∆ ≍ µR, and hence ∆L ≍ µ2 . Proposition 5 shows that the weighted guarantee in Proposition 2 cannot be naively upgraded by proving that Adam’s magnitude weights stay bounded away from zero. Rare negative outliers with probability q < 1/2 still leave SignSGD with positive drift, but Adam’s vt remembers their √ squares and can make |mt | / vt polynomially small. For example, ignoring logarithms, take equal p memory ρ1 = ρ2 = T −a , ηt /R ≍ ρ2 /T , and µ = σT −s . At the natural weighted-proof scale a = p/(3p − 2), one has 1/2 < a < 1 for p < 2, and there is a nonempty interval of s < (p − 1)/p for which Proposition 5 gives an Ω(µ) stationarity lower bound, while the sign-method target (2) is o(µ). The result is not a universal lower bound over all AdamW hyperparameters, but it identifies a concrete long-memory regime where weighted stationarity can coexist with failure of plain ℓ1 stationarity.
References Kwangjun Ahn and Ashok Cutkosky. Adam with model exponential moving average is effective for nonconvex optimization. In Advances in Neural Information Processing Systems 37 (NeurIPS), pages 94909–94933, 2024. Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, and Suvrit Sra. Linear attention is (maybe) all you need (to understand Transformer optimization). In International Conference on Learning Representations (ICLR), pages 16193–16205, 2024. Lukas Balles and Philipp Hennig. Dissecting Adam: The sign, magnitude and variance of stochastic gradients. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 404–413, 2018. 5
Y U TAO WAN L UO Z HANG
Lukas Balles, Fabian Pedregosa, and Nicolas Le Roux. The geometry of sign gradient descent. arXiv preprint arXiv:2002.08056, 2020. Barak Battash, Lior Wolf, and Ofir Lindenbaum. Revisiting the noise model of stochastic gradient descent. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 4780–4788, 2024. Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. In Advances in Neural Information Processing Systems 36 (NeurIPS), pages 49205–49233, 2023. Savelii Chezhegov, Klyukin Yaroslav, Andrei Semenov, Aleksandr Beznosikov, Alexander Gasnikov, Samuel Horváth, Martin Takáč, and Eduard Gorbunov. Clipping improves Adam-norm and AdaGrad-norm when the noise is heavy-tailed. In Proceedings of the 42nd International Conference on Machine Learning (ICML), pages 10269–10333, 2025. DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence, 2026. Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smoothness: A stopped analysis of adaptive sgd. In Proceedings of the 36th Conference on Learning Theory (COLT), pages 89–160, 2023. Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in SGD. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 3964– 3975, 2021. Yusu Hong and Junhong Lin. On convergence of Adam for stochastic optimization under relaxed assumptions. In Advances in Neural Information Processing Systems 37 (NeurIPS), pages 10827– 10877, 2024. Wei Jiang, Dingzhi Yu, Sifan Yang, Wenhao Yang, and Lijun Zhang. Improved analysis for signbased methods with momentum updates. arXiv preprint arXiv:2507.12091, 2025. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https: //kellerjordan.github.io/posts/muon/. Andrej Karpathy. NanoGPT, 2022. URL https://github.com/karpathy/nanoGPT. Kimi Team. Kimi K2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. Frederik Kunstner and Francis Bach. Scaling laws for gradient descent and sign descent for linear bigram models under Zipf’s law. In Advances in Neural Information Processing Systems 38 (NeurIPS), pages 144435–144467, 2025. Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be. In International Conference on Learning Representations (ICLR), 2023. 6
O PEN P ROBLEM : A DAM W U NDER H EAVY TAILS
Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why Adam outperforms gradient descent on language models. In Advances in Neural Information Processing Systems 37 (NeurIPS), pages 30106–30148, 2024. Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. Convex and non-convex optimization under generalized smoothness. In Advances in Neural Information Processing Systems 36 (NeurIPS), pages 40238–40271, 2023a. Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of Adam under relaxed assumptions. In Advances in Neural Information Processing Systems 36 (NeurIPS), pages 52166–52196, 2023b. √ Huan Li, Yiming Dong, and Zhouchen Lin. On the O( d/K 1/4 ) convergence rate of AdamW measured by ℓ1 norm. In Advances in Neural Information Processing Systems 38 (NeurIPS), pages 132360–132387, 2025. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for LLM training. arXiv preprint arXiv:2502.16982, 2025a. Yuxing Liu, Rui Pan, and Tong Zhang. AdaGrad under anisotropic smoothness. In International Conference on Learning Representations (ICLR), pages 19574–19608, 2025b. Yuxing Liu, Jianyu Wang, and Tong Zhang. Optimizer-model consistency: Full finetuning with the same optimizer as pretraining forgets less. arXiv preprint arXiv:2605.06654, 2026. Zijian Liu. Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad. arXiv preprint arXiv:2605.18694, 2026. Zijian Liu and Zhengyuan Zhou. Nonconvex stochastic optimization under heavy-tailed noises: Optimal convergence without gradient clipping. In International Conference on Learning Representations (ICLR), pages 92529–92554, 2025. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. Hanyang Peng, Shuang Qin, Yue Yu, Fangqing Jiang, Hui Wang, and Zhouchen Lin. Simple convergence proof of Adam from a sign-like descent perspective. arXiv preprint arXiv:2507.05966, 2025. Steven T Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review, 21(5):1112–1130, 2014. Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In International Conference on Learning Representations (ICLR), 2018. Andrei Semenov, Matteo Pagliardini, and Martin Jaggi. Benchmarking optimizers for large language model pretraining. arXiv preprint arXiv:2509.01440, 2025. 7
Y U TAO WAN L UO Z HANG
Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J. Shah, et al. Practical efficiency of Muon for pretraining. arXiv preprint arXiv:2505.02222, 2025. Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 5827–5837, 2019. Hongyi Tao, Dingzhi Yu, and Lijun Zhang. When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1 -norm Lower Bounds. arXiv preprint arXiv:2605.06615, 2026. Bengt von Bahr and Carl-Gustav Esseen. Inequalities for the rth absolute moment of a sum of random variables, 1 ≤ r ≤ 2. The Annals of Mathematical Statistics, 36(1):299–303, 1965. Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them. arXiv preprint arXiv:2509.02046, 2025. Robin Yadav, Shuo Xie, Tianhao Wang, and Zhiyuan Li. Provable benefit of sign descent: A minimal model under heavy-tail class imbalance. In OPT 2025: Optimization for Machine Learning, 2025. Dingzhi Yu, Wei Jiang, Hongyi Tao, Yuanyu Wan, and Lijun Zhang. Mirror descent under generalized smoothness. arXiv preprint arXiv:2502.00753, 2025. Dingzhi Yu, Rui Pan, Yuxing Liu, and Tong Zhang. StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models. arXiv preprint arXiv:2604.15416, 2026a. Dingzhi Yu, Hongyi Tao, Yuanyu Wan, Luo Luo, and Lijun Zhang. Sign-based optimizers are effective under heavy-tailed noise. arXiv preprint arXiv:2602.07425, 2026b. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chengxing Xie, Cunxiang Wang, et al. GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In Advances in Neural Information Processing Systems 33 (NeurIPS), pages 15383–15393, 2020. Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules. In Advances in Neural Information Processing Systems 35 (NeurIPS), pages 28386–28399, 2022. Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstructing what makes a good optimizer for autoregressive language models. In International Conference on Learning Representations (ICLR), pages 2830–2850, 2025.
Appendix A. Derivations Behind the Barriers All derivations in this appendix use the clean specialization L1 = σ1 = 0, so that D(x) = D = diag(L0 ) = diag(L), 8
σ = σ0 > 0.
O PEN P ROBLEM : A DAM W U NDER H EAVY TAILS
Gradient-curvature implies descent. Let s = x′ − x and D = diag(L) = diag(L0 ). Applying (GC) to x + τ s gives ∥∇f (x + τ s) − ∇f (x)∥D−1 ≤ τ ∥s∥D . Therefore, it holds that Z 1
′
⟨∇f (x + τ s) − ∇f (x), s⟩ dτ ≤
f (x ) − f (x) − ⟨∇f (x), s⟩ = 0
1 ∥s∥2D . 2
With s = −ηt ut , this gives the one-step inequality used before (A). The decoupled weight-decay case follows with s = −ηt (ut + λxt ) and extra terms involving λxt . Proof of (B).
When β12 < β2 , for one coordinate, mt = (1 − β1 )
t X
β1t−k gk ,
vt = (1 − β2 )
k=1
t X
β2t−k gk2 .
k=1
Cauchy’s inequality with weights β2t−k gives ! ! t t X X β2t−k gk2 ≤ (β12 /β2 )t−k m2t ≤ (1 − β1 )2 k=1
k=1
(1 − β1 )2 vt . (1 − β12 /β2 )(1 − β2 )
Since εt ≥ 0, this gives the deterministic update bound |ut,i | ≤ Cβ = p
1 − β1 (1 − β2 )(1 − β12 /β2 )
.
(B)
Proof of Proposition 2. Let ρ = 1 − β1 , ξt = gt − ∇f (xt ), and et = mt − ∇f (xt ). We use the von Bahr–Esseen inequality (von Bahr and Esseen, 1965). For conditionally mean-zero random variables and p ∈ [1, 2], the pth moment of their sum is bounded, up to a universal constant, by the sum of their pth moments. Applying it within a mini-batch gives E[|ξt,i |p | Ft−1 ] ≤ O(σip B 1−p ). The momentum error decomposes as et = −β1t ∇f (x1 ) +
t X
t X β1t−s+1 ∇f (xs−1 ) − ∇f (xs ) + ρ β1t−s ξs . s=1
s=2
By (GC) and (B), ∥∇f (xs ) − ∇f (xs−1 )∥1 ≤
q ∥L∥1 ∥∇f (xs ) − ∇f (xs−1 )∥D−1 ≤ ηCβ ∥L∥1 .
For the stochastic term, the weighted sequence {ρβ1t−s ξs,i }ts=1 is a martingale difference sequence. A second application of von Bahr–Esseen gives " # ! t 1−1/p X ∥σ∥ ρ 1 E ρ β1t−s ξs ≤O , 1−1/p B s=1 1
9
Y U TAO WAN L UO Z HANG
P p(t−s) because ρp ts=1 β1 ≤ O(ρp−1 ). The initial term is controlled by smoothness and lower boundedness. q q ∥∇f (x1 )∥1 ≤ ∥L∥1 ∥∇f (x1 )∥D−1 ≤ 2∆ ∥L∥1 . Thus
T
1X E[∥et ∥1 ] ≤ T t=1
! p 2∆ ∥L∥1 ηCβ ∥L∥1 ∥σ∥1 ρ1−1/p + +O . ρT ρ B 1−1/p
Now write ut = at ⊙ sign(mt ). Since 0 ≤ at,i ≤ Cβ by (B), sign mismatch implies ⟨∇f (xt ), ut ⟩ ≥ St − 2Cβ ∥et ∥1 . Combining this inequality with the descent lemma and ∥ut ∥2D ≤ Cβ2 ∥L∥1 gives, after summing from 1 to T , ! p T Cβ2 η ∥L∥1 Cβ ∥σ∥1 (1 − β1 )1−1/p Cβ ∆ ∥L∥1 1X ∆ E[St ] ≤ O + + . + T ηT 1 − β1 (1 − β1 )T B 1−1/p t=1
When β2 = β1 , we have Cβ = 1. Choosing s η≍
∆(1 − β1 ) , ∥L∥1 T
1 − β1 ≍
! 2p p ∆ ∥L∥1 B 1−1/p 3p−2 √ ∥σ∥1 T
balances the first three terms. Since σ > 0, the initial-gradient term is lower order in the usual large-horizon, noise-dominated regime and can be absorbed into the same expression by enlarging the constant. This gives Proposition 2. Proof of Proposition 3. Let g = −aµ with probability 1 − q and g = bµ with probability q, where b = (1 + (1 − q)a)/q. Then E[g] = µ(−a(1 − q) + bq) = µ. The noise moment is finite: E[|g − µ|p ] = µp ((1 − q)(a + 1)p + q(b − 1)p ) . For β1 = β2 = 0 and εt = 0, AdamW uses g/ |g| = sign(g). Hence E[µ sign(g)] = µ ((+1)q + (−1)(1 − q)) = µ(2q − 1) < 0. Proof of Proposition 5. First construct the corridor. Let h : R → [0, 1] be a smooth cutoff with h(z) = 0 for z ≤ −1, h(z) = 1 for z ≥ −1/2, and |h′ (z)| ≤ Ch . Set Z x/R fµ,R (x) = µR
h(z) dz. −∞
′ (x) ≤ µ, f ′ (x) = µ for x ≥ −R/2, f ′ (x) = 0 for x ≤ −R, and f ′′ (x) ≤ Then 0 ≤ fµ,R µ,R µ,R µ,R
Cµ/R. Thus L ≍ µ/R, ∆ = fµ,R (0) − inf x fµ,R (x) ≍ µR, and ∆L ≍ µ2 . 10
O PEN P ROBLEM : A DAM W U NDER H EAVY TAILS
We next record the stopped-displacement implication. trajectory xt+1 = xt − ηt ut PT For any + η E[u ]. Since only positive ut moves started at x1 = 0, let u+ = max{u , 0} and D = t T t t t=1 t left along the corridor, X ηs u+ (−xt )+ ≤ s. s<t
Hence, if DT ≤ R/4, Markov’s inequality gives 2E[(−xt )+ ] 1 ≤ . R 2 PT ′ (x ) = µ, so T −1 ′ On the complementary event, fµ,R t t=1 E[ fµ,R (xt ) ] ≥ µ/2. It remains to upper bound Adam’s expected positive displacement. At a point with ht = ′ (x ) ∈ [0, µ], define the oracle by g = h + ξ , where fµ,R t t t t ( −M, with probability q, ξt = qM/(1 − q), with probability 1 − q. Pr(xt < −R/2) ≤
Then E[ξt | Ft−1 ] = 0. With M = σq −1/p and q ≤ 1/2, p
p
E[|ξt | | Ft−1 ] = qM + (1 − q)
qM 1−q
p
≤ 2σ p .
Let K = ⌈CK /ρ2 ⌉, where CK is a sufficiently large absolute constant, and let At be the event that at least one negative outlier occurs in the last K oracle calls. Since q = c0 ρ2 log(1/δ) and ρ2 ≤ 1/2, choosing c0 CK large enough ensures Pr(Act ) ≤ δ for all t > K. On At , the condition M ≥ 2µ implies that one of these gradients has magnitude at least M/2. Moreover β2K is bounded below by an absolute constant, so vt ≥ cρ2 M 2 . For the numerator, mt = ρ1
t X
′ (xs ) + ρ1 β1t−s fµ,R
s=1
t X
β1t−s ξs .
s=1
The deterministic part has magnitude at most µ. By von Bahr–Esseen, # " t X (p−1)/p t−s β1 ξs ≤ Cp σρ1 . E ρ1 s=1
Therefore, for every t > K, E[u+ t ] ≤ E[|ut |] ≤ Cp
µ σ
(p−1)/p
+ ρ1
(2−p)/(2p)
ρ2
(log(1/δ))1/p + Cβ δ,
where the last term uses the deterministic bound |ut | ≤ Cβ on Act . For the first K iterates, the same deterministic bound gives E[u+ t ] ≤ Cβ . The displacement condition in the proposition is exactly the requirement that DT ≤ R/4, after adjusting the absolute constant c. The stopped-displacement argument above completes the proof.
11